Quantifying infrastructure noise in agentic coding evals
Anthropic reruns Terminal-Bench 2.0 under six resource configs and finds a 6-point score swing from container limits alone — treat sub-3-point leaderboard gaps as noise until the eval setup is documented.
**The gist** Anthropic ran **Terminal-Bench 2.0** under six container resource configurations with the model, harness, and tasks held constant. Scores rose monotonically with headroom: infrastructure error rates fell from **5.8% to 0.5%**, and the gap between the tightest and loosest setups reached **6 percentage points (p < 0.01)**. A SWE-bench cross-check on 227 problems showed a smaller **1.54-point** effect.
**Why it matters** Memory limits are an unreported eval variable — strict enforcement triggers spurious out-of-memory kills, while generous limits let agents install heavier tooling, a different strategy entirely. When reading agent leaderboards, **differences under 3 points deserve skepticism** until resource configs are documented; the authors suggest about **3x headroom** as a sane default.
**The gist** Anthropic ran **Terminal-Bench 2.0** under six container resource configurations with the model, harness, and tasks held constant. Scores rose monotonically with headroom: infrastructure error rates fell from **5.8% to 0.5%**, and the gap between the tightest and loosest setups reached **6 percentage points (p < 0.01)**. A SWE-bench cross-check on 227 problems showed a smaller **1.54-point** effect. **Why it matters** Memory limits are an unreported eval variable — strict enforcement triggers spurious out-of-memory kills, while generous limits let agents install heavier tooling, a different strategy entirely. When reading agent leaderboards, **differences under 3 points deserve skepticism** until resource configs are documented; the authors suggest about **3x headroom** as a sane default. **Watch out** Time limits and **API latency variance** were observed but not rigorously quantified, and findings were only replicated across **Anthropic models** — the confidence intervals here already span 1–2 points on their own.