AI EngineerVideoSource Linked

Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software

Human task duration is a weak proxy for agent difficulty when agents use different shortcuts and tools. Long-horizon evals need environment complexity, queryable trajectories, and judges that inspect final state.

AI Engineer · Aug 1, 2026

Open Source Open Markdown Open JSON

Source Summary

The talk treats long horizon as a changing scalar, not a stable task category. Human-time estimates vary with expertise and methodology, while agents may automate tedious work through tools. Its finance dataset reports **15 average human hours** across a **50-task sample**.

Practical Implication

Evaluate the coordination burden, ambiguity, trajectory, and resulting environment state alongside elapsed human time. For open-ended work, give a judge agent read-only access to the same logs and artifacts, and make long trajectories queryable instead of placing the entire trace in one prompt.

Agent-Ready Context

The talk treats long horizon as a changing scalar, not a stable task category. Human-time estimates vary with expertise and methodology, while agents may automate tedious work through tools. Its finance dataset reports **15 average human hours** across a **50-task sample**.

Evaluate the coordination burden, ambiguity, trajectory, and resulting environment state alongside elapsed human time. For open-ended work, give a judge agent read-only access to the same logs and artifacts, and make long trajectories queryable instead of placing the entire trace in one prompt.

Judge models add flexibility where deterministic checks are brittle, but they also introduce another fallible agent. Broader environments create more valid paths and make standardized grading harder; the talk's benchmark criticism is concentrated mainly on finance tasks.

Connected Context · Feed7 Judgment

This reframes long-horizon evaluation from a duration label into a multidimensional environment-design problem. It reinforces trace-aware assessment and human review for work with many valid paths, while narrowing confidence in judge agents: queryable artifacts improve their evidence, but do not remove their fallibility or solve standardized grading across broad environments.

DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, DatacurveDeepSWE’s use of behavioral traces reinforces the argument that elapsed time or final scores alone cannot characterize sustained agent work.Verifiable Environments for AI in Biology — Kenny Workman, LatchBioLatchBio independently shows the grading consequence of environments with multiple valid paths: brittle checks can reject sound work, making informed review necessary.Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge BiasIts evidence of LLM-judge bias strengthens the talk’s warning that adding a judge agent introduces another source of evaluation error.Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon LabsVending-Bench complements the environment-state framing by showing that long-running behavior can drift or change between repeatable simulations and real-world settings.

Context Map

benchmarkcodingdata#agent-evals#benchmark-integrity#agent-reliability

Uncertainty

Judge models add flexibility where deterministic checks are brittle, but they also introduce another fallible agent. Broader environments create more valid paths and make standardized grading harder; the talk's benchmark criticism is concentrated mainly on finance tasks.