# Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave

Source: [AI Engineer](https://www.youtube.com/watch?v=cQQbJqvZkpo)  
Feed7 permalink: https://feed7.dev/p/vertical-mobility-inference-from-mvp-to-trillion-parameter-workloads-sit-11qvtzr  
Published: 2026-09-19T16:30:34.000Z  
Trust: Source Linked (source_linked)

## Why Included

Agent requests often repeat most of their input, making KV-cache-aware routing a direct latency and cost lever for hosted coding-agent workloads.

## Source Summary

CoreWeave describes one inference platform spanning serverless and dedicated deployments, heterogeneous GPUs, and several engines. In agentic traffic, **80–90% of input** may repeat across requests, so its router prioritizes KV-cache locality before falling back to the least-loaded target.

## Practical Implication

For a coding-agent backend, retain reusable prefixes across turns and make cache locality part of routing. Choose prefill/decode disaggregation only after measuring the workload, and schedule latency-tolerant batch work onto capacity left idle by real-time traffic.

## Agent-Ready Context

CoreWeave describes one inference platform spanning serverless and dedicated deployments, heterogeneous GPUs, and several engines. In agentic traffic, **80–90% of input** may repeat across requests, so its router prioritizes KV-cache locality before falling back to the least-loaded target.

For a coding-agent backend, retain reusable prefixes across turns and make cache locality part of routing. Choose prefill/decode disaggregation only after measuring the workload, and schedule latency-tolerant batch work onto capacity left idle by real-time traffic.

The talk presents platform design choices rather than controlled comparisons. Cache behavior varies by customer, provisioned throughput needs a known traffic profile, and disaggregation is explicitly not economical for every workload.

## Connected Context

Feed7 judgment across 823 accumulated Signals:

This confirms repeated prefixes as a concrete routing input for coding-agent infrastructure and places cache locality inside a broader platform spanning engines, GPUs, and deployment modes. It narrows the design choice by treating prefill/decode separation and provisioned capacity as traffic-dependent, while suggesting idle real-time capacity can absorb latency-tolerant batch work.

- [KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat](https://feed7.dev/p/kv-cache-aware-routing-and-p-d-disaggregation-on-kubernetes-yuchen-fama-0u2jkju) — Red Hat provides supporting case-study evidence for KV-aware routing and clarifies the concurrency and network conditions under which prefill/decode separation is worthwhile.
- [The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI](https://feed7.dev/p/the-frontier-ai-inference-cloud-for-agents-byung-gon-gon-chun-friendliai-0b7vgoo) — FriendliAI extends the same locality principle from individual routing decisions to hierarchical KV storage and whole-task, agent-aware scheduling.
- [AI Gateway logs now have a dedicated page](https://feed7.dev/p/ai-gateway-logs-1272t5j) — Request-level routing, latency, token, and fallback logs provide part of the observability needed to validate whether locality policies and heterogeneous targets improve real workloads.

## Context Map

- Layer: infra
- Domains: coding
- Topics: context-caching, gateways, cloud-agents

## Uncertainty

- The talk presents platform design choices rather than controlled comparisons. Cache behavior varies by customer, provisioned throughput needs a known traffic profile, and disaggregation is explicitly not economical for every workload.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.