Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave
Agent requests often repeat most of their input, making KV-cache-aware routing a direct latency and cost lever for hosted coding-agent workloads.
CoreWeave describes one inference platform spanning serverless and dedicated deployments, heterogeneous GPUs, and several engines. In agentic traffic, **80–90% of input** may repeat across requests, so its router prioritizes KV-cache locality before falling back to the least-loaded target.
For a coding-agent backend, retain reusable prefixes across turns and make cache locality part of routing. Choose prefill/decode disaggregation only after measuring the workload, and schedule latency-tolerant batch work onto capacity left idle by real-time traffic.
CoreWeave describes one inference platform spanning serverless and dedicated deployments, heterogeneous GPUs, and several engines. In agentic traffic, **80–90% of input** may repeat across requests, so its router prioritizes KV-cache locality before falling back to the least-loaded target. For a coding-agent backend, retain reusable prefixes across turns and make cache locality part of routing. Choose prefill/decode disaggregation only after measuring the workload, and schedule latency-tolerant batch work onto capacity left idle by real-time traffic. The talk presents platform design choices rather than controlled comparisons. Cache behavior varies by customer, provisioned throughput needs a known traffic profile, and disaggregation is explicitly not economical for every workload.
This confirms repeated prefixes as a concrete routing input for coding-agent infrastructure and places cache locality inside a broader platform spanning engines, GPUs, and deployment modes. It narrows the design choice by treating prefill/decode separation and provisioned capacity as traffic-dependent, while suggesting idle real-time capacity can absorb latency-tolerant batch work.