Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI
OpenAI replaced opaque feedback-driven inference routing with explicit global optimization plus fast local guards, making latency, capacity, and failure behavior easier to reason about.
OpenAI’s earlier router adjusted engine weights from smoothed performance signals relative to the fleet average. The newer design uses a **global control plane** to publish optimized weight snapshots while each cluster’s **local data plane** routes requests without a synchronous control-plane call.
Builders operating multiple inference pools should separate slow global optimization from fast request handling. Optimize end-to-end latency using demand, network distance, engine capacity, health, TTFT, and TBOT, while preserving local guards for rapidly changing failures.
OpenAI’s earlier router adjusted engine weights from smoothed performance signals relative to the fleet average. The newer design uses a **global control plane** to publish optimized weight snapshots while each cluster’s **local data plane** routes requests without a synchronous control-plane call. Builders operating multiple inference pools should separate slow global optimization from fast request handling. Optimize end-to-end latency using demand, network distance, engine capacity, health, TTFT, and TBOT, while preserving local guards for rapidly changing failures. A farther engine can beat a nearby overloaded one, but production signals are noisy and capacity can collapse. The design therefore still needs engine penalties, dynamic retry budgets, and load shedding; the talk does not provide a universal policy or measured gain.
This turns gateway observability into a two-level routing architecture: global optimization can use fleet-wide latency and capacity signals without putting the control plane on every request path, while local guards handle fast failures. It complements task-level model selection rather than replacing it, and confirms that fallbacks, retries, penalties, and shedding remain necessary because optimized weights cannot make noisy or collapsing capacity predictable.