What's New in Inference Engineering — Philip Kiely, Baseten
Inference gains increasingly depend on training-aware techniques: weight quantization, KV-cache management, and speculative decoding can matter more than swapping serving code.
The talk groups day-to-day inference optimization into **quantization, KV caching, and speculation**. TurboQuant compresses the KV cache to **4 bits**, while DFlash drafts **8 or 16 tokens** per window and showed more than a 3x improvement over Eagle in one B200/Qwen 38B test.
Treat inference choices as workload-specific experiments. For agent systems, measure weight quantization, cache-aware routing, offloading, sharing, and speculative acceptance rates against your actual prompts rather than adopting a paper result by default.
The talk groups day-to-day inference optimization into **quantization, KV caching, and speculation**. TurboQuant compresses the KV cache to **4 bits**, while DFlash drafts **8 or 16 tokens** per window and showed more than a 3x improvement over Eagle in one B200/Qwen 38B test. Treat inference choices as workload-specific experiments. For agent systems, measure weight quantization, cache-aware routing, offloading, sharing, and speculative acceptance rates against your actual prompts rather than adopting a paper result by default. TurboQuant proved less applicable to data-center inference than its initial attention suggested. Continuous retraining of speculators may improve acceptance by **20% to 2x**, but it requires permission to reuse traffic, substantial storage and compute, and retraining whenever the target model changes.
This organizes inference tuning around quantization, KV-cache management, and speculation, but narrows each to measured workload fit. It confirms that cache capacity and reuse matter for agents while adding speculative acceptance and retraining economics as separate variables. The mixed applicability of TurboQuant and traffic-reuse requirements argue against treating headline paper gains as deployment defaults.