Open Brain
arXivPaperNeeds Review

Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

Terminal-Bench 2.1 fixes issues found in roughly 30% of its tasks, shifting accuracy and leaderboard positions. The audit also shows that time limits can change apparent harness performance.

arXiv · Nov 5, 2026
Open Source Open MarkdownOpen JSON
Source Summary

An audit of **Terminal-Bench 2.0** found issues in roughly **30%** of tasks. The repaired release, Terminal-Bench 2.1, raises accuracy by over 10 percentage points and moves some leaderboard positions by up to three ranks; overall rankings remain highly correlated (Spearman rho 0.958).

Practical Implication

When comparing coding agents, check which benchmark version and resource limits were used. The paper finds that extending time limits can close a 20-point harness gap on GPT 5.4 and shrink a 30-point gap on GPT 5.4 mini to 5 points.

Agent-Ready Context
An audit of **Terminal-Bench 2.0** found issues in roughly **30%** of tasks. The repaired release, Terminal-Bench 2.1, raises accuracy by over 10 percentage points and moves some leaderboard positions by up to three ranks; overall rankings remain highly correlated (Spearman rho 0.958).

When comparing coding agents, check which benchmark version and resource limits were used. The paper finds that extending time limits can close a 20-point harness gap on GPT 5.4 and shrink a 30-point gap on GPT 5.4 mini to 5 points.

The authors propose continuous validation using community reports, agent traces, automated checks, and human-gated repairs. These results concern Terminal-Bench; they do not establish how often other benchmarks need repair.
Connected Context · Feed7 Judgment

This makes benchmark maintenance a measurable part of coding-agent evaluation: repairing Terminal-Bench changes scores substantially and some ranks, while overall ordering remains highly correlated. It also narrows comparisons to a specific benchmark version and resource budget, and proposes ongoing, human-gated validation; it does not show that other benchmarks have similar defect rates.

When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AIBoth caution that leaderboard results depend on benchmark integrity; this paper adds measured effects from repairing tasks and changing time limits, while the candidate focuses on contamination, verifier quality, and inspectability.Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval BelferThe vLLM case shows evaluation infrastructure can silently corrupt outputs; this paper adds a separate source of unreliable results—defective benchmark tasks—and proposes continuous validation for them.SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering AgentsSWE-Gate finds that passing benchmark checks can miss review constraints; this paper addresses reliability earlier in the evaluation chain by auditing and repairing benchmark tasks themselves.
Context Map
benchmarkcoding#agent-evals#benchmark-integrity#agent-reliability
Uncertainty
The authors propose continuous validation using community reports, agent traces, automated checks, and human-gated repairs. These results concern Terminal-Bench; they do not establish how often other benchmarks need repair.