# Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

Source: [AI Engineer](https://www.youtube.com/watch?v=sOB3HSiG8vo)  
Feed7 permalink: https://feed7.dev/p/routing-llm-inference-in-production-from-engine-signals-to-policy-qianru-0ug6kw6  
Published: 2026-09-19T15:30:04.000Z  
Trust: Source Linked (source_linked)

## Why Included

OpenAI replaced opaque feedback-driven inference routing with explicit global optimization plus fast local guards, making latency, capacity, and failure behavior easier to reason about.

## Source Summary

OpenAI’s earlier router adjusted engine weights from smoothed performance signals relative to the fleet average. The newer design uses a **global control plane** to publish optimized weight snapshots while each cluster’s **local data plane** routes requests without a synchronous control-plane call.

## Practical Implication

Builders operating multiple inference pools should separate slow global optimization from fast request handling. Optimize end-to-end latency using demand, network distance, engine capacity, health, TTFT, and TBOT, while preserving local guards for rapidly changing failures.

## Agent-Ready Context

OpenAI’s earlier router adjusted engine weights from smoothed performance signals relative to the fleet average. The newer design uses a **global control plane** to publish optimized weight snapshots while each cluster’s **local data plane** routes requests without a synchronous control-plane call.

Builders operating multiple inference pools should separate slow global optimization from fast request handling. Optimize end-to-end latency using demand, network distance, engine capacity, health, TTFT, and TBOT, while preserving local guards for rapidly changing failures.

A farther engine can beat a nearby overloaded one, but production signals are noisy and capacity can collapse. The design therefore still needs engine penalties, dynamic retry budgets, and load shedding; the talk does not provide a universal policy or measured gain.

## Connected Context

Feed7 judgment across 823 accumulated Signals:

This turns gateway observability into a concrete routing architecture: slow fleet-wide optimization publishes weights, while clusters retain fast local decisions during failures. It narrows adaptive routing claims by requiring noisy latency, capacity, health, and distance signals to be balanced with penalties, retries, and shedding rather than treated as a universal policy.

- [AI Gateway logs now have a dedicated page](https://feed7.dev/p/ai-gateway-logs-1272t5j) — Per-request latency, routing, and fallback records provide evidence needed to evaluate the global policy and local guards described here, but the router adds the control loop that logging alone lacks.
- [TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI](https://feed7.dev/p/2607-22465v1-1g7nw7j) — TRACE-Router learns from task outcomes and keeps an agent task on one backend, whereas this design optimizes fleet routing from serving signals; together they expose a policy choice between task consistency and engine-level adaptation.
- [How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs](https://feed7.dev/p/how-web-data-infrastructure-powers-the-next-generation-of-ai-patricija-z-1ta4xzh) — The retrieval case broadens end-to-end latency beyond inference engines, implying that a router optimizing user-visible latency may need evidence from downstream browser, proxy, parsing, and retry stages.

## Context Map

- Layer: infra
- Domains: None
- Topics: gateways, observability, agent-reliability

## Uncertainty

- A farther engine can beat a nearby overloaded one, but production signals are noisy and capacity can collapse. The design therefore still needs engine penalties, dynamic retry budgets, and load shedding; the talk does not provide a universal policy or measured gain.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.