{
  "schema_version": "1.1",
  "id": "archive:https://arxiv.org/abs/2611.08417v1",
  "slug": "2611-08417v1-1dwvi6x",
  "url": "https://feed7.dev/p/2611-08417v1-1dwvi6x",
  "title": "Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks",
  "why_included": "Terminal-Bench 2.1 fixes issues found in roughly 30% of its tasks, shifting accuracy and leaderboard positions. The audit also shows that time limits can change apparent harness performance.",
  "summary": "An audit of **Terminal-Bench 2.0** found issues in roughly **30%** of tasks. The repaired release, Terminal-Bench 2.1, raises accuracy by over 10 percentage points and moves some leaderboard positions by up to three ranks; overall rankings remain highly correlated (Spearman rho 0.958).",
  "practical_implication": "When comparing coding agents, check which benchmark version and resource limits were used. The paper finds that extending time limits can close a 20-point harness gap on GPT 5.4 and shrink a 30-point gap on GPT 5.4 mini to 5 points.",
  "agent_context": "An audit of **Terminal-Bench 2.0** found issues in roughly **30%** of tasks. The repaired release, Terminal-Bench 2.1, raises accuracy by over 10 percentage points and moves some leaderboard positions by up to three ranks; overall rankings remain highly correlated (Spearman rho 0.958).\n\nWhen comparing coding agents, check which benchmark version and resource limits were used. The paper finds that extending time limits can close a 20-point harness gap on GPT 5.4 and shrink a 30-point gap on GPT 5.4 mini to 5 points.\n\nThe authors propose continuous validation using community reports, agent traces, automated checks, and human-gated repairs. These results concern Terminal-Bench; they do not establish how often other benchmarks need repair.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2611.08417v1",
    "published_at": "2026-11-05T18:18:39.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "benchmark",
  "domains": [
    "coding"
  ],
  "topics": [
    "agent-evals",
    "benchmark-integrity",
    "agent-reliability"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "The authors propose continuous validation using community reports, agent traces, automated checks, and human-gated repairs. These results concern Terminal-Bench; they do not establish how often other benchmarks need repair."
  ],
  "connected_context": {
    "meaning": "This makes benchmark maintenance a measurable part of coding-agent evaluation: repairing Terminal-Bench changes scores substantially and some ranks, while overall ordering remains highly correlated. It also narrows comparisons to a specific benchmark version and resource budget, and proposes ongoing, human-gated validation; it does not show that other benchmarks have similar defect rates.",
    "corpus_size": 1020,
    "generated_at": "2026-11-06T18:41:28.753Z",
    "connections": [
      {
        "title": "When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=-npY6XjM8CQ",
        "feed7_url": "https://feed7.dev/p/when-will-the-benchmaxxing-plague-end-nick-heiner-surge-ai-178gqcg",
        "reason": "Both caution that leaderboard results depend on benchmark integrity; this paper adds measured effects from repairing tasks and changing time limits, while the candidate focuses on contamination, verifier quality, and inspectability."
      },
      {
        "title": "Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=btxG75rNJC4",
        "feed7_url": "https://feed7.dev/p/two-bugs-that-hid-in-plain-sight-a-vllm-debugging-detective-story-asaf-g-1s5yaqd",
        "reason": "The vLLM case shows evaluation infrastructure can silently corrupt outputs; this paper adds a separate source of unreliable results—defective benchmark tasks—and proposes continuous validation for them."
      },
      {
        "title": "SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.04167v1",
        "feed7_url": "https://feed7.dev/p/2609-04167v1-1vvofgw",
        "reason": "SWE-Gate finds that passing benchmark checks can miss review constraints; this paper addresses reliability earlier in the evaluation chain by auditing and repairing benchmark tasks themselves."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-11-05T18:18:39.000Z",
  "modified_at": "2026-11-05T18:18:39.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2611-08417v1-1dwvi6x",
    "json": "https://feed7.dev/p/2611-08417v1-1dwvi6x.json",
    "markdown": "https://feed7.dev/p/2611-08417v1-1dwvi6x.md"
  }
}