# The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

Source: [AI Engineer](https://www.youtube.com/watch?v=Hvb2LfMH58c)  
Feed7 permalink: https://feed7.dev/p/the-frontier-ai-inference-cloud-for-agents-byung-gon-gon-chun-friendliai-0b7vgoo  
Published: 2026-09-19T18:00:03.000Z  
Trust: Source Linked (source_linked)

## Why Included

Agent-serving systems should optimize whole-task latency, preserving shared prefixes across tool calls and routing later steps toward the GPU that already holds their KV cache.

## Source Summary

Agent work alternates model calls with tools, grows its context, and often branches into subagents. Consecutive steps share large prefixes, so FriendliAI centers its stack on **prefix caching**, hierarchical and distributed KV storage, cache-aware routing, and agent-aware scheduling.

## Practical Implication

Measure end-to-end task completion rather than treating every call as independent. Keep related steps cache-local, balance that locality against hot spots, and use task context when deciding preemption, eviction, or speculative prefill.

## Agent-Ready Context

Agent work alternates model calls with tools, grows its context, and often branches into subagents. Consecutive steps share large prefixes, so FriendliAI centers its stack on **prefix caching**, hierarchical and distributed KV storage, cache-aware routing, and agent-aware scheduling.

Measure end-to-end task completion rather than treating every call as independent. Keep related steps cache-local, balance that locality against hot spots, and use task context when deciding preemption, eviction, or speculative prefill.

The talk’s speed, cost, and quality comparisons are provider-reported and do not include enough methodology here for independent evaluation. Agent-aware scheduling is also presented as an emerging optimization area, not a settled recipe for every workload.

## Connected Context

Feed7 judgment across 823 accumulated Signals:

This strengthens the case that agent inference should be optimized as a stateful, end-to-end workload rather than isolated requests. It extends prior cache-aware routing evidence with hierarchical KV storage and task-aware scheduling, while keeping the same qualification: locality must be balanced against load and validated on real traffic, not accepted from provider comparisons alone.

- [Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave](https://feed7.dev/p/vertical-mobility-inference-from-mvp-to-trillion-parameter-workloads-sit-11qvtzr) — CoreWeave independently supports the premise that repeated agent prefixes make KV-local routing a practical latency and cost lever.
- [KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat](https://feed7.dev/p/kv-cache-aware-routing-and-p-d-disaggregation-on-kubernetes-yuchen-fama-0u2jkju) — Red Hat supplies case-study evidence for cache-aware scheduling while clarifying that disaggregation depends on concurrency, prompt shape, and network performance.
- [TokTier: Exact Stateful Tokenization for Agentic LLM Serving](https://feed7.dev/p/2607-29678v1-1fuv1qw) — TokTier identifies repeated tokenization as a complementary bottleneck, showing that KV-cache locality alone does not guarantee low time to first token.
- [Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta](https://feed7.dev/p/operating-distributed-inference-systems-at-scale-nishant-gupta-naman-ahu-0l3qmbh) — Meta reinforces the shift toward workflow-aware orchestration by likewise requiring scheduling to account for KV state, retries, hardware, and completed-task cost.

## Context Map

- Layer: infra
- Domains: coding
- Topics: context-caching, cloud-agents, agent-reliability

## Uncertainty

- The talk’s speed, cost, and quality comparisons are provider-reported and do not include enough methodology here for independent evaluation. Agent-aware scheduling is also presented as an emerging optimization area, not a settled recipe for every workload.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.