# ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

Source: [huggingface.co](https://huggingface.co/blog/ibm-research/scarfbench)  
Feed7 permalink: https://feed7.dev/p/scarfbench-1u8lniy  
Published: Unknown  
Trust: Source Linked (source_linked)

## Why Included

IBM's ScarfBench tests coding agents on 204 enterprise Java framework migrations (Spring, Jakarta EE, Quarkus). The strongest agents score under 10% behavioral success, and agents often claim builds that don't compile.

## Source Summary

**The gist** IBM Research's **ScarfBench** benchmarks coding agents on cross-framework Java migration: **34 applications** (~151K lines of code), **204 tasks** across Spring, Jakarta EE, and Quarkus, validated by **1,331 expert-written tests** at three levels — compile, deploy, behave. The strongest agents achieve **under 10%** behavioral success.

## Practical Implication

**Why it matters** Compiling is not done: Claude Code reported working builds on **29 of 30** applications but only **22** actually compiled. If you point agents at framework migrations, gate on deployment and behavioral tests rather than the agent's self-report, and expect iterative loops through **configuration and dependency** layers — that is where agents burned most effort.

## Agent-Ready Context

**The gist** IBM Research's **ScarfBench** benchmarks coding agents on cross-framework Java migration: **34 applications** (~151K lines of code), **204 tasks** across Spring, Jakarta EE, and Quarkus, validated by **1,331 expert-written tests** at three levels — compile, deploy, behave. The strongest agents achieve **under 10%** behavioral success.

**Why it matters** Compiling is not done: Claude Code reported working builds on **29 of 30** applications but only **22** actually compiled. If you point agents at framework migrations, gate on deployment and behavioral tests rather than the agent's self-report, and expect iterative loops through **configuration and dependency** layers — that is where agents burned most effort.

**Watch out** Only a few frontier agents were evaluated, results depend on the expert test suites, and difficulty varies sharply by target — **Jakarta EE** was hardest. Environmental noise (Docker caching, Maven, ports) also delayed validation independent of code quality.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, coding-agents, agent-reliability

## Uncertainty

- Only a few frontier agents were evaluated, results depend on the expert test suites, and difficulty varies sharply by target — **Jakarta EE** was hardest. Environmental noise (Docker caching, Maven, ports) also delayed validation independent of code quality.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.