AI EngineerVideoSource Linked

The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

Agent-serving systems should optimize whole-task latency, preserving shared prefixes across tool calls and routing later steps toward the GPU that already holds their KV cache.

AI Engineer · Sep 19, 2026

Open Source Open Markdown Open JSON

Source Summary

Agent work alternates model calls with tools, grows its context, and often branches into subagents. Consecutive steps share large prefixes, so FriendliAI centers its stack on **prefix caching**, hierarchical and distributed KV storage, cache-aware routing, and agent-aware scheduling.

Practical Implication

Measure end-to-end task completion rather than treating every call as independent. Keep related steps cache-local, balance that locality against hot spots, and use task context when deciding preemption, eviction, or speculative prefill.

Agent-Ready Context

Agent work alternates model calls with tools, grows its context, and often branches into subagents. Consecutive steps share large prefixes, so FriendliAI centers its stack on **prefix caching**, hierarchical and distributed KV storage, cache-aware routing, and agent-aware scheduling.

Measure end-to-end task completion rather than treating every call as independent. Keep related steps cache-local, balance that locality against hot spots, and use task context when deciding preemption, eviction, or speculative prefill.

The talk’s speed, cost, and quality comparisons are provider-reported and do not include enough methodology here for independent evaluation. Agent-aware scheduling is also presented as an emerging optimization area, not a settled recipe for every workload.

Connected Context · Feed7 Judgment

This strengthens the view that agent inference is a stateful workflow rather than a sequence of independent requests: shared prefixes, growing context, and branches make KV locality and task-aware scheduling central. It also narrows the claim to a workload-dependent systems hypothesis, since locality competes with load balancing and the provider-reported gains lack enough methodology for independent adoption decisions.

Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeaveCoreWeave independently reinforces repeated agent prefixes as a routing signal and places cache locality within broader choices about engines, deployment modes, and provisioned capacity.KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red HatRed Hat supplies a concrete implementation qualification: KV-aware routing and prefill/decode separation depend on concurrency, prefill pressure, network speed, and observed cache residency rather than forming a default topology.TokTier: Exact Stateful Tokenization for Agentic LLM ServingTokTier identifies repeated tokenization as a complementary bottleneck, showing that strong KV-cache reuse alone does not guarantee low time to first token for growing agent transcripts.Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, MetaMeta’s account extends task-aware scheduling beyond cache locality to priorities, retries, accumulated cost, hardware, and latency budgets, clarifying the additional state a completed-task optimizer must consider.

Context Map

infracoding#context-caching#cloud-agents#agent-reliability

Uncertainty

The talk’s speed, cost, and quality comparisons are provider-reported and do not include enough methodology here for independent evaluation. Agent-aware scheduling is also presented as an emerging optimization area, not a settled recipe for every workload.