{
  "schema_version": "1.1",
  "id": "s8:https://www.youtube.com/watch?v=sOB3HSiG8vo",
  "slug": "routing-llm-inference-in-production-from-engine-signals-to-policy-qianru-0ug6kw6",
  "url": "https://feed7.dev/p/routing-llm-inference-in-production-from-engine-signals-to-policy-qianru-0ug6kw6",
  "title": "Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI",
  "why_included": "OpenAI replaced opaque feedback-driven inference routing with explicit global optimization plus fast local guards, making latency, capacity, and failure behavior easier to reason about.",
  "summary": "OpenAI’s earlier router adjusted engine weights from smoothed performance signals relative to the fleet average. The newer design uses a **global control plane** to publish optimized weight snapshots while each cluster’s **local data plane** routes requests without a synchronous control-plane call.",
  "practical_implication": "Builders operating multiple inference pools should separate slow global optimization from fast request handling. Optimize end-to-end latency using demand, network distance, engine capacity, health, TTFT, and TBOT, while preserving local guards for rapidly changing failures.",
  "agent_context": "OpenAI’s earlier router adjusted engine weights from smoothed performance signals relative to the fleet average. The newer design uses a **global control plane** to publish optimized weight snapshots while each cluster’s **local data plane** routes requests without a synchronous control-plane call.\n\nBuilders operating multiple inference pools should separate slow global optimization from fast request handling. Optimize end-to-end latency using demand, network distance, engine capacity, health, TTFT, and TBOT, while preserving local guards for rapidly changing failures.\n\nA farther engine can beat a nearby overloaded one, but production signals are noisy and capacity can collapse. The design therefore still needs engine penalties, dynamic retry budgets, and load shedding; the talk does not provide a universal policy or measured gain.",
  "source": {
    "name": "AI Engineer",
    "url": "https://www.youtube.com/watch?v=sOB3HSiG8vo",
    "published_at": "2026-09-19T15:30:04.000Z"
  },
  "source_class": "video",
  "content_type": "Video",
  "layer": "infra",
  "domains": [],
  "topics": [
    "gateways",
    "observability",
    "agent-reliability"
  ],
  "verification": {
    "status": "source_linked",
    "label": "Source Linked",
    "method": "source_feed",
    "verified_at": null
  },
  "uncertainty": [
    "A farther engine can beat a nearby overloaded one, but production signals are noisy and capacity can collapse. The design therefore still needs engine penalties, dynamic retry budgets, and load shedding; the talk does not provide a universal policy or measured gain."
  ],
  "connected_context": {
    "meaning": "This turns gateway observability into a concrete routing architecture: slow fleet-wide optimization publishes weights, while clusters retain fast local decisions during failures. It narrows adaptive routing claims by requiring noisy latency, capacity, health, and distance signals to be balanced with penalties, retries, and shedding rather than treated as a universal policy.",
    "corpus_size": 823,
    "generated_at": "2026-09-20T09:13:02.255Z",
    "connections": [
      {
        "title": "AI Gateway logs now have a dedicated page",
        "source_name": "Vercel",
        "source_url": "https://vercel.com/changelog/ai-gateway-logs",
        "feed7_url": "https://feed7.dev/p/ai-gateway-logs-1272t5j",
        "reason": "Per-request latency, routing, and fallback records provide evidence needed to evaluate the global policy and local guards described here, but the router adds the control loop that logging alone lacks."
      },
      {
        "title": "TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2607.22465v1",
        "feed7_url": "https://feed7.dev/p/2607-22465v1-1g7nw7j",
        "reason": "TRACE-Router learns from task outcomes and keeps an agent task on one backend, whereas this design optimizes fleet routing from serving signals; together they expose a policy choice between task consistency and engine-level adaptation."
      },
      {
        "title": "How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=1UmZHb_E_SM",
        "feed7_url": "https://feed7.dev/p/how-web-data-infrastructure-powers-the-next-generation-of-ai-patricija-z-1ta4xzh",
        "reason": "The retrieval case broadens end-to-end latency beyond inference engines, implying that a router optimizing user-visible latency may need evidence from downstream browser, proxy, parsing, and retry stages."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-09-19T15:30:04.000Z",
  "modified_at": "2026-09-19T15:30:04.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/routing-llm-inference-in-production-from-engine-signals-to-policy-qianru-0ug6kw6",
    "json": "https://feed7.dev/p/routing-llm-inference-in-production-from-engine-signals-to-policy-qianru-0ug6kw6.json",
    "markdown": "https://feed7.dev/p/routing-llm-inference-in-production-from-engine-signals-to-policy-qianru-0ug6kw6.md"
  }
}