# Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software

Source: [AI Engineer](https://www.youtube.com/watch?v=2aS7aKoXn64)  
Feed7 permalink: https://feed7.dev/p/rethinking-environments-for-long-horizon-work-rayan-garg-theta-software-11r7wbx  
Published: 2026-08-01T00:00:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

Human task duration is a weak proxy for agent difficulty when agents use different shortcuts and tools. Long-horizon evals need environment complexity, queryable trajectories, and judges that inspect final state.

## Source Summary

The talk treats long horizon as a changing scalar, not a stable task category. Human-time estimates vary with expertise and methodology, while agents may automate tedious work through tools. Its finance dataset reports **15 average human hours** across a **50-task sample**.

## Practical Implication

Evaluate the coordination burden, ambiguity, trajectory, and resulting environment state alongside elapsed human time. For open-ended work, give a judge agent read-only access to the same logs and artifacts, and make long trajectories queryable instead of placing the entire trace in one prompt.

## Agent-Ready Context

The talk treats long horizon as a changing scalar, not a stable task category. Human-time estimates vary with expertise and methodology, while agents may automate tedious work through tools. Its finance dataset reports **15 average human hours** across a **50-task sample**.

Evaluate the coordination burden, ambiguity, trajectory, and resulting environment state alongside elapsed human time. For open-ended work, give a judge agent read-only access to the same logs and artifacts, and make long trajectories queryable instead of placing the entire trace in one prompt.

Judge models add flexibility where deterministic checks are brittle, but they also introduce another fallible agent. Broader environments create more valid paths and make standardized grading harder; the talk's benchmark criticism is concentrated mainly on finance tasks.

## Connected Context

Feed7 judgment across 318 accumulated Signals:

This reframes long-horizon evaluation from a duration label into a multidimensional environment-design problem. It reinforces trace-aware assessment and human review for work with many valid paths, while narrowing confidence in judge agents: queryable artifacts improve their evidence, but do not remove their fallibility or solve standardized grading across broad environments.

- [DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve](https://feed7.dev/p/deepswe-a-contamination-resistant-coding-benchmark-james-shi-datacurve-08p61c0) — DeepSWE’s use of behavioral traces reinforces the argument that elapsed time or final scores alone cannot characterize sustained agent work.
- [Verifiable Environments for AI in Biology — Kenny Workman, LatchBio](https://feed7.dev/p/verifiable-environments-for-ai-in-biology-kenny-workman-latchbio-1vs6y66) — LatchBio independently shows the grading consequence of environments with multiple valid paths: brittle checks can reject sound work, making informed review necessary.
- [Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias](https://feed7.dev/p/2607-11871v1-17vejh0) — Its evidence of LLM-judge bias strengthens the talk’s warning that adding a judge agent introduces another source of evaluation error.
- [Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs](https://feed7.dev/p/vending-bench-long-horizon-agent-evals-lukas-petersson-andon-labs-0fu78nz) — Vending-Bench complements the environment-state framing by showing that long-running behavior can drift or change between repeatable simulations and real-world settings.

## Context Map

- Layer: benchmark
- Domains: coding, data
- Topics: agent-evals, benchmark-integrity, agent-reliability

## Uncertainty

- Judge models add flexibility where deterministic checks are brittle, but they also introduce another fallible agent. Broader environments create more valid paths and make standardized grading harder; the talk's benchmark criticism is concentrated mainly on finance tasks.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.