The State of Model Routing — NVIDIA, Cognition, OpenRouter
Model routing can cut agent costs only when delegation, escalation, and context transfer are designed together; duplicating context across models can erase the savings.
The panel describes routing work between small models and frontier models, with compact handoffs instead of copying full traces. It notes that provider KV caches commonly have a **5-minute lifetime**, though that is an operational policy rather than a physical limit.
Give most context to one model, let workers return file references or compact findings, and keep a frontier model available for escalation. Measure total task cost: larger models may use tools and tokens more efficiently even when their per-token price is higher.
The panel describes routing work between small models and frontier models, with compact handoffs instead of copying full traces. It notes that provider KV caches commonly have a **5-minute lifetime**, though that is an operational policy rather than a physical limit. Give most context to one model, let workers return file references or compact findings, and keep a frontier model available for escalation. Measure total task cost: larger models may use tools and tokens more efficiently even when their per-token price is higher. Routing adds a controller to an already nondeterministic system. Small models may not recognize when they are outside their competence, repeated file reads can multiply cost, and the panel characterizes current orchestration techniques as early.
This makes routing a context-management and total-task economics problem, not simply a per-token price choice. It reinforces planner–worker separation with compact handoffs and frontier escalation, while narrowing confidence in routing gains: controller nondeterminism, repeated reads, short-lived provider caches, and weak self-recognition by small models can erase savings.