# Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

Source: [arXiv](https://arxiv.org/abs/2611.08417v1)  
Feed7 permalink: https://feed7.dev/p/2611-08417v1-1dwvi6x  
Published: 2026-11-05T18:18:39.000Z  
Trust: Needs Review (needs_review)

## Why Included

Terminal-Bench 2.1 fixes issues found in roughly 30% of its tasks, shifting accuracy and leaderboard positions. The audit also shows that time limits can change apparent harness performance.

## Source Summary

An audit of **Terminal-Bench 2.0** found issues in roughly **30%** of tasks. The repaired release, Terminal-Bench 2.1, raises accuracy by over 10 percentage points and moves some leaderboard positions by up to three ranks; overall rankings remain highly correlated (Spearman rho 0.958).

## Practical Implication

When comparing coding agents, check which benchmark version and resource limits were used. The paper finds that extending time limits can close a 20-point harness gap on GPT 5.4 and shrink a 30-point gap on GPT 5.4 mini to 5 points.

## Agent-Ready Context

An audit of **Terminal-Bench 2.0** found issues in roughly **30%** of tasks. The repaired release, Terminal-Bench 2.1, raises accuracy by over 10 percentage points and moves some leaderboard positions by up to three ranks; overall rankings remain highly correlated (Spearman rho 0.958).

When comparing coding agents, check which benchmark version and resource limits were used. The paper finds that extending time limits can close a 20-point harness gap on GPT 5.4 and shrink a 30-point gap on GPT 5.4 mini to 5 points.

The authors propose continuous validation using community reports, agent traces, automated checks, and human-gated repairs. These results concern Terminal-Bench; they do not establish how often other benchmarks need repair.

## Connected Context

Feed7 judgment across 1020 accumulated Signals:

This makes benchmark maintenance a measurable part of coding-agent evaluation: repairing Terminal-Bench changes scores substantially and some ranks, while overall ordering remains highly correlated. It also narrows comparisons to a specific benchmark version and resource budget, and proposes ongoing, human-gated validation; it does not show that other benchmarks have similar defect rates.

- [When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI](https://feed7.dev/p/when-will-the-benchmaxxing-plague-end-nick-heiner-surge-ai-178gqcg) — Both caution that leaderboard results depend on benchmark integrity; this paper adds measured effects from repairing tasks and changing time limits, while the candidate focuses on contamination, verifier quality, and inspectability.
- [Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer](https://feed7.dev/p/two-bugs-that-hid-in-plain-sight-a-vllm-debugging-detective-story-asaf-g-1s5yaqd) — The vLLM case shows evaluation infrastructure can silently corrupt outputs; this paper adds a separate source of unreliable results—defective benchmark tasks—and proposes continuous validation for them.
- [SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents](https://feed7.dev/p/2609-04167v1-1vvofgw) — SWE-Gate finds that passing benchmark checks can miss review constraints; this paper addresses reliability earlier in the evaluation chain by auditing and repairing benchmark tasks themselves.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, benchmark-integrity, agent-reliability

## Uncertainty

- The authors propose continuous validation using community reports, agent traces, automated checks, and human-gated repairs. These results concern Terminal-Bench; they do not establish how often other benchmarks need repair.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.
