Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning
Long-running agents strain context, reward design, and GPU scheduling at once. Compaction and value models can help, but each introduces another optimization trade-off.
Long-horizon RL faces gradient variance that grows with trajectory length, sparse terminal rewards, credit assignment, and variable episode lengths. A **1 million-token context window** is still small relative to the horizons the speakers target.
Plan agent state beyond the prompt: use compaction, file-based scratch space, trajectory search, or archives as the task permits. For training, critics can provide earlier signals, while pipeline RL trades fresher policy data for better GPU use.
Long-horizon RL faces gradient variance that grows with trajectory length, sparse terminal rewards, credit assignment, and variable episode lengths. A **1 million-token context window** is still small relative to the horizons the speakers target. Plan agent state beyond the prompt: use compaction, file-based scratch space, trajectory search, or archives as the task permits. For training, critics can provide earlier signals, while pipeline RL trades fresher policy data for better GPU use. The team reports pipeline training tolerating about **eight off-policy steps**, but week-long inference can exceed that window. Bootstrapping with a value model keeps hardware occupied at the cost of bias, and its platform’s **350 environments** do not remove that trade-off.
This establishes long-horizon agency as both an external-state and training-systems problem: even very large contexts cannot hold the targeted work, while longer trajectories worsen sparse rewards, credit assignment, and policy freshness. Compaction, files, archives, critics, and pipelining are therefore explicit trade-offs rather than complete solutions, and the reported off-policy tolerance does not extend cleanly to week-long runs.