Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta
Large agent workloads turn inference into an orchestration problem: scheduling must account for workflow state, KV cache, hardware, retries, and cost per completed task.
Meta frames agent capacity as users multiplied by calls and tokens: roughly **1 call** for a chatbot turn, **10–20** for copilots, **50** for research agents, and potentially thousands for autonomous workflows. Routing decisions also affect cache hits, batching, GPU use, and autoscaling.
Design the scheduler around whole workflows, not isolated requests. Track model state, GPU type, HBM headroom, KV cache, tenant priority, latency budget, prior workflow spend, and retry cost; optimize for cost per completed task rather than cost per token.
Meta frames agent capacity as users multiplied by calls and tokens: roughly **1 call** for a chatbot turn, **10–20** for copilots, **50** for research agents, and potentially thousands for autonomous workflows. Routing decisions also affect cache hits, batching, GPU use, and autoscaling. Design the scheduler around whole workflows, not isolated requests. Track model state, GPU type, HBM headroom, KV cache, tenant priority, latency budget, prior workflow spend, and retry cost; optimize for cost per completed task rather than cost per token. These are operating principles, not benchmarked prescriptions. Workflow-aware priority can reduce wasted compute, but tight coupling also expands the control plane’s failure surface, and cold pools cannot absorb rerouted traffic until models and caches warm up.
This elevates workflow state into the inference scheduler: priority, cache locality, retries, accumulated spend, and latency budgets should be optimized per completed task rather than per request or token. It reinforces long-horizon harness work while adding a systems consequence: more agent calls amplify capacity demands and make control-plane resilience and warm-pool readiness part of agent reliability.