# What's New in Inference Engineering — Philip Kiely, Baseten

Source: [AI Engineer](https://www.youtube.com/watch?v=75ckHC2LU_0)  
Feed7 permalink: https://feed7.dev/p/what-s-new-in-inference-engineering-philip-kiely-baseten-1mlen9m  
Published: 2026-09-19T17:00:27.000Z  
Trust: Source Linked (source_linked)

## Why Included

Inference gains increasingly depend on training-aware techniques: weight quantization, KV-cache management, and speculative decoding can matter more than swapping serving code.

## Source Summary

The talk groups day-to-day inference optimization into **quantization, KV caching, and speculation**. TurboQuant compresses the KV cache to **4 bits**, while DFlash drafts **8 or 16 tokens** per window and showed more than a 3x improvement over Eagle in one B200/Qwen 38B test.

## Practical Implication

Treat inference choices as workload-specific experiments. For agent systems, measure weight quantization, cache-aware routing, offloading, sharing, and speculative acceptance rates against your actual prompts rather than adopting a paper result by default.

## Agent-Ready Context

The talk groups day-to-day inference optimization into **quantization, KV caching, and speculation**. TurboQuant compresses the KV cache to **4 bits**, while DFlash drafts **8 or 16 tokens** per window and showed more than a 3x improvement over Eagle in one B200/Qwen 38B test.

Treat inference choices as workload-specific experiments. For agent systems, measure weight quantization, cache-aware routing, offloading, sharing, and speculative acceptance rates against your actual prompts rather than adopting a paper result by default.

TurboQuant proved less applicable to data-center inference than its initial attention suggested. Continuous retraining of speculators may improve acceptance by **20% to 2x**, but it requires permission to reuse traffic, substantial storage and compute, and retraining whenever the target model changes.

## Connected Context

Feed7 judgment across 823 accumulated Signals:

This organizes inference tuning around quantization, KV-cache management, and speculation, but narrows each to measured workload fit. It confirms that cache capacity and reuse matter for agents while adding speculative acceptance and retraining economics as separate variables. The mixed applicability of TurboQuant and traffic-reuse requirements argue against treating headline paper gains as deployment defaults.

- [The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI](https://feed7.dev/p/the-frontier-ai-inference-cloud-for-agents-byung-gon-gon-chun-friendliai-0b7vgoo) — FriendliAI shows how the KV-cache category becomes a system design problem through cache-local routing, distributed storage, and task-aware scheduling.
- [Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher](https://feed7.dev/p/deep-dive-on-llm-inference-at-scale-harshul-jain-audible-tanmay-sah-inde-04utjkh) — The capacity analysis supplies the premise for KV compression and offloading: long contexts and concurrency can exhaust cache memory before weights become limiting.
- [TokTier: Exact Stateful Tokenization for Agentic LLM Serving](https://feed7.dev/p/2607-29678v1-1fuv1qw) — TokTier adds tokenization as a distinct latency source, preventing optimization of caching and speculation from being mistaken for complete time-to-first-token coverage.
- [Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli](https://feed7.dev/p/weight-folding-cuda-streams-and-the-bug-that-made-my-model-speak-backwar-1vqmg44) — The CUDA stream race demonstrates the correctness consequence of low-level optimization, reinforcing the need to pair speed measurements with long-generation validation.

## Context Map

- Layer: infra
- Domains: None
- Topics: context-caching, agent-reliability

## Uncertainty

- TurboQuant proved less applicable to data-center inference than its initial attention suggested. Continuous retraining of speculators may improve acceptance by **20% to 2x**, but it requires permission to reuse traffic, substantial storage and compute, and retraining whenever the target model changes.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.