# Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

Source: [AI Engineer](https://www.youtube.com/watch?v=7c9FSUVcXR0)  
Feed7 permalink: https://feed7.dev/p/operating-distributed-inference-systems-at-scale-nishant-gupta-naman-ahu-0l3qmbh  
Published: 2026-09-19T15:00:10.000Z  
Trust: Source Linked (source_linked)

## Why Included

Large agent workloads turn inference into an orchestration problem: scheduling must account for workflow state, KV cache, hardware, retries, and cost per completed task.

## Source Summary

Meta frames agent capacity as users multiplied by calls and tokens: roughly **1 call** for a chatbot turn, **10–20** for copilots, **50** for research agents, and potentially thousands for autonomous workflows. Routing decisions also affect cache hits, batching, GPU use, and autoscaling.

## Practical Implication

Design the scheduler around whole workflows, not isolated requests. Track model state, GPU type, HBM headroom, KV cache, tenant priority, latency budget, prior workflow spend, and retry cost; optimize for cost per completed task rather than cost per token.

## Agent-Ready Context

Meta frames agent capacity as users multiplied by calls and tokens: roughly **1 call** for a chatbot turn, **10–20** for copilots, **50** for research agents, and potentially thousands for autonomous workflows. Routing decisions also affect cache hits, batching, GPU use, and autoscaling.

Design the scheduler around whole workflows, not isolated requests. Track model state, GPU type, HBM headroom, KV cache, tenant priority, latency budget, prior workflow spend, and retry cost; optimize for cost per completed task rather than cost per token.

These are operating principles, not benchmarked prescriptions. Workflow-aware priority can reduce wasted compute, but tight coupling also expands the control plane’s failure surface, and cold pools cannot absorb rerouted traffic until models and caches warm up.

## Connected Context

Feed7 judgment across 823 accumulated Signals:

This elevates workflow state into the inference scheduler: priority, cache locality, retries, accumulated spend, and latency budgets should be optimized per completed task rather than per request or token. It reinforces long-horizon harness work while adding a systems consequence: more agent calls amplify capacity demands and make control-plane resilience and warm-pool readiness part of agent reliability.

- [The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI](https://feed7.dev/p/the-frontier-ai-inference-cloud-for-agents-byung-gon-gon-chun-friendliai-0b7vgoo) — Both favor whole-task latency and preserving KV-cache locality across tool calls; this Signal expands that requirement to GPU state, tenant priority, retry cost, and autoscaling.
- [Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning](https://feed7.dev/p/scaling-to-long-horizons-ross-taylor-chengxi-taylor-general-reasoning-0jwtg4d) — Long-horizon agents create the many-call, stateful workloads this scheduler is meant to support, connecting context and reward trade-offs to concrete GPU-capacity and priority decisions.
- [AI Agents Are Just Distributed Systems Now — Salman Munaf, TikTok](https://feed7.dev/p/ai-agents-are-just-distributed-systems-now-salman-munaf-tiktok-1v4yc47) — Workflow-aware scheduling can reduce wasted retries, but unknown tool outcomes still require persisted steps, idempotency, and reconciliation; infrastructure optimization does not replace execution semantics.
- [What we’ve learned building cloud agents](https://feed7.dev/p/cloud-agent-lessons-06dh9iq) — Cursor’s durable orchestration evidence reinforces treating an agent run as a persistent workflow, while this Signal adds the inference-capacity and cache consequences of doing so at scale.

## Context Map

- Layer: agent
- Domains: None
- Topics: harness-engineering, agent-reliability, cloud-agents

## Uncertainty

- These are operating principles, not benchmarked prescriptions. Workflow-aware priority can reduce wasted compute, but tight coupling also expands the control plane’s failure surface, and cold pools cannot absorb rerouted traffic until models and caches warm up.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.