AI EngineerVideoSource Linked

Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software

Human task duration is a weak proxy for agent difficulty when agents use different shortcuts and tools. Long-horizon evals need environment complexity, queryable trajectories, and judges that inspect final state.

AI Engineer · Aug 1, 2026

Open Source Open Markdown Open JSON

Source Summary

The talk treats long horizon as a changing scalar, not a stable task category. Human-time estimates vary with expertise and methodology, while agents may automate tedious work through tools. Its finance dataset reports **15 average human hours** across a **50-task sample**.

Practical Implication

Evaluate the coordination burden, ambiguity, trajectory, and resulting environment state alongside elapsed human time. For open-ended work, give a judge agent read-only access to the same logs and artifacts, and make long trajectories queryable instead of placing the entire trace in one prompt.

Agent-Ready Context

The talk treats long horizon as a changing scalar, not a stable task category. Human-time estimates vary with expertise and methodology, while agents may automate tedious work through tools. Its finance dataset reports **15 average human hours** across a **50-task sample**.

Evaluate the coordination burden, ambiguity, trajectory, and resulting environment state alongside elapsed human time. For open-ended work, give a judge agent read-only access to the same logs and artifacts, and make long trajectories queryable instead of placing the entire trace in one prompt.

Judge models add flexibility where deterministic checks are brittle, but they also introduce another fallible agent. Broader environments create more valid paths and make standardized grading harder; the talk's benchmark criticism is concentrated mainly on finance tasks.

Connected Context · Feed7 Judgment

This replaces task duration as the main proxy for long-horizon difficulty with environment and trajectory complexity. Relative to the candidates, it strengthens the case for inspecting artifacts and final state, while adding an implementation requirement that long traces remain queryable. It also narrows enthusiasm for flexible model judges because broader valid solution spaces make those judges both more necessary and more fallible.

DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, DatacurveDeepSWE provides a coding-specific case where long-horizon behavior and traces matter, while Theta explains why task duration alone cannot establish that those repository tasks are representative or difficult.Verifiable Environments for AI in Biology — Kenny Workman, LatchBioBiology independently reinforces the central grading problem: as workflows admit multiple valid paths, brittle end-state checks weaken and human-valid solutions must test the evaluator.Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge BiasThe mechanistic evidence of LLM-judge bias sharpens Theta's caveat that adding a judge agent introduces another fallible component, suggesting prompt design alone may not control that risk.Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon LabsVending-Bench adds a complementary environment requirement: repeatable simulations should be paired with real-world tests because agent behavior may drift or change when evaluation is detected.

Context Map

benchmarkcodingdata#agent-evals#benchmark-integrity#agent-reliability

Uncertainty

Judge models add flexibility where deterministic checks are brittle, but they also introduce another fallible agent. Broader environments create more valid paths and make standardized grading harder; the talk's benchmark criticism is concentrated mainly on finance tasks.