AI EngineerVideoSource Linked

Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning

Long-running agents strain context, reward design, and GPU scheduling at once. Compaction and value models can help, but each introduces another optimization trade-off.

AI Engineer · Jul 31, 2026

Open Source Open Markdown Open JSON

Source Summary

Long-horizon RL faces gradient variance that grows with trajectory length, sparse terminal rewards, credit assignment, and variable episode lengths. A **1 million-token context window** is still small relative to the horizons the speakers target.

Practical Implication

Plan agent state beyond the prompt: use compaction, file-based scratch space, trajectory search, or archives as the task permits. For training, critics can provide earlier signals, while pipeline RL trades fresher policy data for better GPU use.

Agent-Ready Context

Long-horizon RL faces gradient variance that grows with trajectory length, sparse terminal rewards, credit assignment, and variable episode lengths. A **1 million-token context window** is still small relative to the horizons the speakers target.

Plan agent state beyond the prompt: use compaction, file-based scratch space, trajectory search, or archives as the task permits. For training, critics can provide earlier signals, while pipeline RL trades fresher policy data for better GPU use.

The team reports pipeline training tolerating about **eight off-policy steps**, but week-long inference can exceed that window. Bootstrapping with a value model keeps hardware occupied at the cost of bias, and its platform’s **350 environments** do not remove that trade-off.

Connected Context · Feed7 Judgment

This establishes long-horizon agents as a coupled state, credit-assignment, and systems problem rather than a context-window scaling problem. External memory can extend operation, but it does not remove sparse rewards or stale-policy training data. The reported off-policy tolerance and environment count narrow confidence in pipeline RL for week-long work, while value bootstrapping exchanges idle hardware for added bias.

TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated ReasoningTurnSight targets the credit-assignment bottleneck with turn-level hindsight, but its evidence does not establish that this supervision remains effective across the much longer, less replayable horizons described here.Data and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke LabsBespoke’s evidence-gated approach to RL is reinforced by these horizon-specific costs: curated SFT should precede infrastructure-heavy RL whose gains may be compromised by stale data and biased value estimates.Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and ExecutionE3 offers a runtime way to avoid unnecessary horizon growth by starting with the minimum viable path and expanding only after failed verification; it reduces exposure to the scaling problem rather than solving long-horizon learning.AI tools for Forward Deployed Engineering — Vasuman Moza, Varick AgentsVarick’s staged automation and explicit human authority provide an organizational boundary for workflows whose duration and exceptions exceed what current long-horizon training can reliably support.

Context Map

agentcodingdata#harness-engineering#agent-reliability#context-engineering

Uncertainty

The team reports pipeline training tolerating about **eight off-policy steps**, but week-long inference can exceed that window. Bootstrapping with a value model keeps hardware occupied at the cost of bias, and its platform’s **350 environments** do not remove that trade-off.