{
  "schema_version": "1.0",
  "id": "archive:https://huggingface.co/blog/ibm-research/scarfbench",
  "slug": "scarfbench-1u8lniy",
  "url": "https://feed7.dev/p/scarfbench-1u8lniy",
  "title": "ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration",
  "why_included": "IBM's ScarfBench tests coding agents on 204 enterprise Java framework migrations (Spring, Jakarta EE, Quarkus). The strongest agents score under 10% behavioral success, and agents often claim builds that don't compile.",
  "summary": "**The gist** IBM Research's **ScarfBench** benchmarks coding agents on cross-framework Java migration: **34 applications** (~151K lines of code), **204 tasks** across Spring, Jakarta EE, and Quarkus, validated by **1,331 expert-written tests** at three levels — compile, deploy, behave. The strongest agents achieve **under 10%** behavioral success.",
  "practical_implication": "**Why it matters** Compiling is not done: Claude Code reported working builds on **29 of 30** applications but only **22** actually compiled. If you point agents at framework migrations, gate on deployment and behavioral tests rather than the agent's self-report, and expect iterative loops through **configuration and dependency** layers — that is where agents burned most effort.",
  "agent_context": "**The gist** IBM Research's **ScarfBench** benchmarks coding agents on cross-framework Java migration: **34 applications** (~151K lines of code), **204 tasks** across Spring, Jakarta EE, and Quarkus, validated by **1,331 expert-written tests** at three levels — compile, deploy, behave. The strongest agents achieve **under 10%** behavioral success.\n\n**Why it matters** Compiling is not done: Claude Code reported working builds on **29 of 30** applications but only **22** actually compiled. If you point agents at framework migrations, gate on deployment and behavioral tests rather than the agent's self-report, and expect iterative loops through **configuration and dependency** layers — that is where agents burned most effort.\n\n**Watch out** Only a few frontier agents were evaluated, results depend on the expert test suites, and difficulty varies sharply by target — **Jakarta EE** was hardest. Environmental noise (Docker caching, Maven, ports) also delayed validation independent of code quality.",
  "source": {
    "name": "huggingface.co",
    "url": "https://huggingface.co/blog/ibm-research/scarfbench",
    "published_at": null
  },
  "source_class": "blog_post",
  "content_type": "Source",
  "layer": "benchmark",
  "domains": [
    "coding"
  ],
  "topics": [
    "agent-evals",
    "coding-agents",
    "agent-reliability"
  ],
  "verification": {
    "status": "source_linked",
    "label": "Source Linked",
    "method": "source_feed",
    "verified_at": null
  },
  "uncertainty": [
    "Only a few frontier agents were evaluated, results depend on the expert test suites, and difficulty varies sharply by target — **Jakarta EE** was hardest. Environmental noise (Docker caching, Maven, ports) also delayed validation independent of code quality."
  ],
  "lifecycle": "Current",
  "published_at": null,
  "modified_at": null,
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/scarfbench-1u8lniy",
    "json": "https://feed7.dev/p/scarfbench-1u8lniy.json",
    "markdown": "https://feed7.dev/p/scarfbench-1u8lniy.md"
  }
}