Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI
OpenAI replaced opaque feedback-driven inference routing with explicit global optimization plus fast local guards, making latency, capacity, and failure behavior easier to reason about.
OpenAI’s earlier router adjusted engine weights from smoothed performance signals relative to the fleet average. The newer design uses a **global control plane** to publish optimized weight snapshots while each cluster’s **local data plane** routes requests without a synchronous control-plane call.
Builders operating multiple inference pools should separate slow global optimization from fast request handling. Optimize end-to-end latency using demand, network distance, engine capacity, health, TTFT, and TBOT, while preserving local guards for rapidly changing failures.
OpenAI’s earlier router adjusted engine weights from smoothed performance signals relative to the fleet average. The newer design uses a **global control plane** to publish optimized weight snapshots while each cluster’s **local data plane** routes requests without a synchronous control-plane call. Builders operating multiple inference pools should separate slow global optimization from fast request handling. Optimize end-to-end latency using demand, network distance, engine capacity, health, TTFT, and TBOT, while preserving local guards for rapidly changing failures. A farther engine can beat a nearby overloaded one, but production signals are noisy and capacity can collapse. The design therefore still needs engine penalties, dynamic retry budgets, and load shedding; the talk does not provide a universal policy or measured gain.
This turns gateway observability into a concrete routing architecture: slow fleet-wide optimization publishes weights, while clusters retain fast local decisions during failures. It narrows adaptive routing claims by requiring noisy latency, capacity, health, and distance signals to be balanced with penalties, retries, and shedding rather than treated as a universal policy.