Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta
Large agent workloads turn inference into an orchestration problem: scheduling must account for workflow state, KV cache, hardware, retries, and cost per completed task.
Meta frames agent capacity as users multiplied by calls and tokens: roughly **1 call** for a chatbot turn, **10–20** for copilots, **50** for research agents, and potentially thousands for autonomous workflows. Routing decisions also affect cache hits, batching, GPU use, and autoscaling.
Design the scheduler around whole workflows, not isolated requests. Track model state, GPU type, HBM headroom, KV cache, tenant priority, latency budget, prior workflow spend, and retry cost; optimize for cost per completed task rather than cost per token.
Meta frames agent capacity as users multiplied by calls and tokens: roughly **1 call** for a chatbot turn, **10–20** for copilots, **50** for research agents, and potentially thousands for autonomous workflows. Routing decisions also affect cache hits, batching, GPU use, and autoscaling. Design the scheduler around whole workflows, not isolated requests. Track model state, GPU type, HBM headroom, KV cache, tenant priority, latency budget, prior workflow spend, and retry cost; optimize for cost per completed task rather than cost per token. These are operating principles, not benchmarked prescriptions. Workflow-aware priority can reduce wasted compute, but tight coupling also expands the control plane’s failure surface, and cold pools cannot absorb rerouted traffic until models and caches warm up.
This extends whole-task optimization into fleet scheduling: agent demand, cache locality, GPU memory, tenant priority, retries, and accumulated workflow cost must be considered together rather than reduced to request count or token price. It reinforces stateful serving and long-horizon systems work, while adding a warning that workflow-aware coordination can create a larger failure surface and that rerouting is limited by model and cache warm-up.