KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat
Agentic inference makes KV-cache locality and prefill pressure first-class scheduling concerns. Red Hat's case study improved latency and capacity, but P/D separation needs suitable traffic and fast networking.
Red Hat argues that multi-turn agent traffic invalidates steady-state inference assumptions: prompts fluctuate, cache residency matters, and prefill can interrupt latency-sensitive decode. LLMD combines KV-aware routing with independently scalable prefill and decode workers.
Measure cache-aware throughput, input/output ratios, time to first token, and streaming latency against your own traces. In one agentic case study with a **45:1 input/output ratio**, a **2-prefill/1-decode** setup produced **4× faster TTFT** and handled **60% more requests**.
Red Hat argues that multi-turn agent traffic invalidates steady-state inference assumptions: prompts fluctuate, cache residency matters, and prefill can interrupt latency-sensitive decode. LLMD combines KV-aware routing with independently scalable prefill and decode workers. Measure cache-aware throughput, input/output ratios, time to first token, and streaming latency against your own traces. In one agentic case study with a **45:1 input/output ratio**, a **2-prefill/1-decode** setup produced **4× faster TTFT** and handled **60% more requests**. Prefill/decode separation is not universal. It fits long-context, prefill-heavy workloads in the middle concurrency range, but KV transfer requires fast fabric; short contexts, low concurrency, strict first-token targets, or weaker networking may favor aggregated serving.
This narrows inference architecture choices to measured agent traffic rather than generic serving benchmarks. KV-aware routing and split prefill/decode can materially help long-context, prefill-heavy workloads, but only in a suitable concurrency range with fast KV transport. The case study is useful local evidence, not a default topology; prompt shape, cache residency, TTFT, streaming latency, and network cost must be observed together.