Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning
Long-running agents strain context, reward design, and GPU scheduling at once. Compaction and value models can help, but each introduces another optimization trade-off.
Long-horizon RL faces gradient variance that grows with trajectory length, sparse terminal rewards, credit assignment, and variable episode lengths. A **1 million-token context window** is still small relative to the horizons the speakers target.
Plan agent state beyond the prompt: use compaction, file-based scratch space, trajectory search, or archives as the task permits. For training, critics can provide earlier signals, while pipeline RL trades fresher policy data for better GPU use.
Long-horizon RL faces gradient variance that grows with trajectory length, sparse terminal rewards, credit assignment, and variable episode lengths. A **1 million-token context window** is still small relative to the horizons the speakers target. Plan agent state beyond the prompt: use compaction, file-based scratch space, trajectory search, or archives as the task permits. For training, critics can provide earlier signals, while pipeline RL trades fresher policy data for better GPU use. The team reports pipeline training tolerating about **eight off-policy steps**, but week-long inference can exceed that window. Bootstrapping with a value model keeps hardware occupied at the cost of bias, and its platform’s **350 environments** do not remove that trade-off.
This establishes long-horizon agency as a state-management and training-systems problem that a larger prompt alone does not solve. It deepens prior harness guidance by requiring durable state outside context and exposes a deployment gap: pipeline RL may tolerate limited policy staleness, while week-long trajectories can exceed it, leaving a trade-off among hardware use, fresh data, critic bias, and sparse rewards.