{
  "schema_version": "1.0",
  "id": "archive:https://www.anthropic.com/engineering/infrastructure-noise",
  "slug": "infrastructure-noise-1jyyyw1",
  "url": "https://feed7.dev/p/infrastructure-noise-1jyyyw1",
  "title": "Quantifying infrastructure noise in agentic coding evals",
  "why_included": "Anthropic reruns Terminal-Bench 2.0 under six resource configs and finds a 6-point score swing from container limits alone — treat sub-3-point leaderboard gaps as noise until the eval setup is documented.",
  "summary": "**The gist** Anthropic ran **Terminal-Bench 2.0** under six container resource configurations with the model, harness, and tasks held constant. Scores rose monotonically with headroom: infrastructure error rates fell from **5.8% to 0.5%**, and the gap between the tightest and loosest setups reached **6 percentage points (p < 0.01)**. A SWE-bench cross-check on 227 problems showed a smaller **1.54-point** effect.",
  "practical_implication": "**Why it matters** Memory limits are an unreported eval variable — strict enforcement triggers spurious out-of-memory kills, while generous limits let agents install heavier tooling, a different strategy entirely. When reading agent leaderboards, **differences under 3 points deserve skepticism** until resource configs are documented; the authors suggest about **3x headroom** as a sane default.",
  "agent_context": "**The gist** Anthropic ran **Terminal-Bench 2.0** under six container resource configurations with the model, harness, and tasks held constant. Scores rose monotonically with headroom: infrastructure error rates fell from **5.8% to 0.5%**, and the gap between the tightest and loosest setups reached **6 percentage points (p < 0.01)**. A SWE-bench cross-check on 227 problems showed a smaller **1.54-point** effect.\n\n**Why it matters** Memory limits are an unreported eval variable — strict enforcement triggers spurious out-of-memory kills, while generous limits let agents install heavier tooling, a different strategy entirely. When reading agent leaderboards, **differences under 3 points deserve skepticism** until resource configs are documented; the authors suggest about **3x headroom** as a sane default.\n\n**Watch out** Time limits and **API latency variance** were observed but not rigorously quantified, and findings were only replicated across **Anthropic models** — the confidence intervals here already span 1–2 points on their own.",
  "source": {
    "name": "Anthropic",
    "url": "https://www.anthropic.com/engineering/infrastructure-noise",
    "published_at": null
  },
  "source_class": "blog_post",
  "content_type": "Engineering Post",
  "layer": "benchmark",
  "domains": [
    "coding"
  ],
  "topics": [
    "agent-evals",
    "benchmark-integrity",
    "sandboxing"
  ],
  "verification": {
    "status": "official_source",
    "label": "Official Source",
    "method": "source_feed",
    "verified_at": null
  },
  "uncertainty": [
    "Time limits and **API latency variance** were observed but not rigorously quantified, and findings were only replicated across **Anthropic models** — the confidence intervals here already span 1–2 points on their own."
  ],
  "lifecycle": "Current",
  "published_at": null,
  "modified_at": null,
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/infrastructure-noise-1jyyyw1",
    "json": "https://feed7.dev/p/infrastructure-noise-1jyyyw1.json",
    "markdown": "https://feed7.dev/p/infrastructure-noise-1jyyyw1.md"
  }
}