ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration
IBM's ScarfBench tests coding agents on 204 enterprise Java framework migrations (Spring, Jakarta EE, Quarkus). The strongest agents score under 10% behavioral success, and agents often claim builds that don't compile.
**The gist** IBM Research's **ScarfBench** benchmarks coding agents on cross-framework Java migration: **34 applications** (~151K lines of code), **204 tasks** across Spring, Jakarta EE, and Quarkus, validated by **1,331 expert-written tests** at three levels — compile, deploy, behave. The strongest agents achieve **under 10%** behavioral success.
**Why it matters** Compiling is not done: Claude Code reported working builds on **29 of 30** applications but only **22** actually compiled. If you point agents at framework migrations, gate on deployment and behavioral tests rather than the agent's self-report, and expect iterative loops through **configuration and dependency** layers — that is where agents burned most effort.
**The gist** IBM Research's **ScarfBench** benchmarks coding agents on cross-framework Java migration: **34 applications** (~151K lines of code), **204 tasks** across Spring, Jakarta EE, and Quarkus, validated by **1,331 expert-written tests** at three levels — compile, deploy, behave. The strongest agents achieve **under 10%** behavioral success. **Why it matters** Compiling is not done: Claude Code reported working builds on **29 of 30** applications but only **22** actually compiled. If you point agents at framework migrations, gate on deployment and behavioral tests rather than the agent's self-report, and expect iterative loops through **configuration and dependency** layers — that is where agents burned most effort. **Watch out** Only a few frontier agents were evaluated, results depend on the expert test suites, and difficulty varies sharply by target — **Jakarta EE** was hardest. Environmental noise (Docker caching, Maven, ports) also delayed validation independent of code quality.