What's New in Inference Engineering — Philip Kiely, Baseten
Inference gains increasingly depend on training-aware techniques: weight quantization, KV-cache management, and speculative decoding can matter more than swapping serving code.
The talk groups day-to-day inference optimization into **quantization, KV caching, and speculation**. TurboQuant compresses the KV cache to **4 bits**, while DFlash drafts **8 or 16 tokens** per window and showed more than a 3x improvement over Eagle in one B200/Qwen 38B test.
Treat inference choices as workload-specific experiments. For agent systems, measure weight quantization, cache-aware routing, offloading, sharing, and speculative acceptance rates against your actual prompts rather than adopting a paper result by default.
The talk groups day-to-day inference optimization into **quantization, KV caching, and speculation**. TurboQuant compresses the KV cache to **4 bits**, while DFlash drafts **8 or 16 tokens** per window and showed more than a 3x improvement over Eagle in one B200/Qwen 38B test. Treat inference choices as workload-specific experiments. For agent systems, measure weight quantization, cache-aware routing, offloading, sharing, and speculative acceptance rates against your actual prompts rather than adopting a paper result by default. TurboQuant proved less applicable to data-center inference than its initial attention suggested. Continuous retraining of speculators may improve acceptance by **20% to 2x**, but it requires permission to reuse traffic, substantial storage and compute, and retraining whenever the target model changes.
The talk organizes inference tuning into quantization, cache management, and speculation, but narrows headline speedups to workload-specific evidence. For agent serving, optimization decisions must jointly measure prompt shape, cache behavior, acceptance, quality, permissions, and retraining cost; promising techniques such as KV compression or learned speculators are not default upgrades.