Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave
Agent requests often repeat most of their input, making KV-cache-aware routing a direct latency and cost lever for hosted coding-agent workloads.
CoreWeave describes one inference platform spanning serverless and dedicated deployments, heterogeneous GPUs, and several engines. In agentic traffic, **80–90% of input** may repeat across requests, so its router prioritizes KV-cache locality before falling back to the least-loaded target.
For a coding-agent backend, retain reusable prefixes across turns and make cache locality part of routing. Choose prefill/decode disaggregation only after measuring the workload, and schedule latency-tolerant batch work onto capacity left idle by real-time traffic.
CoreWeave describes one inference platform spanning serverless and dedicated deployments, heterogeneous GPUs, and several engines. In agentic traffic, **80–90% of input** may repeat across requests, so its router prioritizes KV-cache locality before falling back to the least-loaded target. For a coding-agent backend, retain reusable prefixes across turns and make cache locality part of routing. Choose prefill/decode disaggregation only after measuring the workload, and schedule latency-tolerant batch work onto capacity left idle by real-time traffic. The talk presents platform design choices rather than controlled comparisons. Cache behavior varies by customer, provisioned throughput needs a known traffic profile, and disaggregation is explicitly not economical for every workload.
This turns repeated agent prefixes into a concrete platform-routing input across heterogeneous engines, GPUs, and deployment modes. It reinforces stateful, cache-local scheduling while narrowing prefill/decode separation and provisioned capacity to traffic-dependent choices. The idle-capacity batch pattern adds a utilization consequence, but the talk does not establish comparative gains across platforms.