The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI
Agent-serving systems should optimize whole-task latency, preserving shared prefixes across tool calls and routing later steps toward the GPU that already holds their KV cache.
Agent work alternates model calls with tools, grows its context, and often branches into subagents. Consecutive steps share large prefixes, so FriendliAI centers its stack on **prefix caching**, hierarchical and distributed KV storage, cache-aware routing, and agent-aware scheduling.
Measure end-to-end task completion rather than treating every call as independent. Keep related steps cache-local, balance that locality against hot spots, and use task context when deciding preemption, eviction, or speculative prefill.
Agent work alternates model calls with tools, grows its context, and often branches into subagents. Consecutive steps share large prefixes, so FriendliAI centers its stack on **prefix caching**, hierarchical and distributed KV storage, cache-aware routing, and agent-aware scheduling. Measure end-to-end task completion rather than treating every call as independent. Keep related steps cache-local, balance that locality against hot spots, and use task context when deciding preemption, eviction, or speculative prefill. The talk’s speed, cost, and quality comparisons are provider-reported and do not include enough methodology here for independent evaluation. Agent-aware scheduling is also presented as an emerging optimization area, not a settled recipe for every workload.
This strengthens the case that agent inference should be optimized as a stateful, end-to-end workload rather than isolated requests. It extends prior cache-aware routing evidence with hierarchical KV storage and task-aware scheduling, while keeping the same qualification: locality must be balanced against load and validated on real traffic, not accepted from provider comparisons alone.