Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
Terminal-Bench 2.1 fixes issues found in roughly 30% of its tasks, shifting accuracy and leaderboard positions. The audit also shows that time limits can change apparent harness performance.
An audit of **Terminal-Bench 2.0** found issues in roughly **30%** of tasks. The repaired release, Terminal-Bench 2.1, raises accuracy by over 10 percentage points and moves some leaderboard positions by up to three ranks; overall rankings remain highly correlated (Spearman rho 0.958).
When comparing coding agents, check which benchmark version and resource limits were used. The paper finds that extending time limits can close a 20-point harness gap on GPT 5.4 and shrink a 30-point gap on GPT 5.4 mini to 5 points.
An audit of **Terminal-Bench 2.0** found issues in roughly **30%** of tasks. The repaired release, Terminal-Bench 2.1, raises accuracy by over 10 percentage points and moves some leaderboard positions by up to three ranks; overall rankings remain highly correlated (Spearman rho 0.958). When comparing coding agents, check which benchmark version and resource limits were used. The paper finds that extending time limits can close a 20-point harness gap on GPT 5.4 and shrink a 30-point gap on GPT 5.4 mini to 5 points. The authors propose continuous validation using community reports, agent traces, automated checks, and human-gated repairs. These results concern Terminal-Bench; they do not establish how often other benchmarks need repair.
This makes benchmark maintenance a measurable part of coding-agent evaluation: repairing Terminal-Bench changes scores substantially and some ranks, while overall ordering remains highly correlated. It also narrows comparisons to a specific benchmark version and resource budget, and proposes ongoing, human-gated validation; it does not show that other benchmarks have similar defect rates.