# Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs

Source: [AI Engineer](https://www.youtube.com/watch?v=7vn4WpqNpck)  
Feed7 permalink: https://feed7.dev/p/benchmarking-coding-agents-on-new-vs-legacy-codebases-denys-linkov-wised-0vhw2s2  
Published: 2026-08-08T19:00:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

A production refactor shows why coding-agent evaluations need acceptance criteria and end-to-end verification: fast output can still be incomplete scaffolding.

## Source Summary

An early O3-assisted task took **3 hours** and produced 10 major mistakes; newer Sonnet 4.6 solved it after one extra iteration and Opus 4.8 nearly one-shot it. A broader GPT-5.5 attempt finished in **10m 22s** but mostly wrote scaffolding.

## Practical Implication

Benchmark agents against explicit requirements, runnable end-to-end tests, deployment constraints, and hidden assumptions—not elapsed time or lines changed. Wisedocs also found a monorepo simpler for verification and sandbox setup across its former **10+ repositories**.

## Agent-Ready Context

An early O3-assisted task took **3 hours** and produced 10 major mistakes; newer Sonnet 4.6 solved it after one extra iteration and Opus 4.8 nearly one-shot it. A broader GPT-5.5 attempt finished in **10m 22s** but mostly wrote scaffolding.

Benchmark agents against explicit requirements, runnable end-to-end tests, deployment constraints, and hidden assumptions—not elapsed time or lines changed. Wisedocs also found a monorepo simpler for verification and sandbox setup across its former **10+ repositories**.

These are task-specific observations from one refactor, not controlled cross-model results. Human review remained part of the project, and only **15 of 17 requirements** were met during the migration.

## Connected Context

Feed7 judgment across 400 accumulated Signals:

This cautions that faster completion and plausible scaffolding are weak evidence of coding-agent success, especially in legacy migrations. It reinforces candidate benchmarks that prioritize behavioral correctness and reproducible verification, while adding deployment constraints and hidden assumptions to the acceptance surface. Its monorepo observation is useful for harness setup but remains task-specific.

- [ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration](https://feed7.dev/p/scarfbench-1u8lniy) — ScarfBench strongly reinforces the gap between apparent progress and behavioral success in legacy migrations, including agents claiming success on code that does not compile.
- [Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute](https://feed7.dev/p/everything-is-a-rollout-alex-shaw-ryan-marten-terminal-bench-harbor-laud-0iz4rgx) — Harbor supplies the reproducible sandbox, outcome verification, and trajectory inspection loop needed to turn this talk’s requirement-based evaluation advice into repeatable practice.
- [ReviewDebt: a practical framework for scoring every pull request — Sachin Gupta, Ebay](https://feed7.dev/p/reviewdebt-a-practical-framework-for-scoring-every-pull-request-sachin-g-0iyjtyk) — ReviewDebt adds a complementary operational measure: even requirement-passing output can become harmful if its verification burden grows faster than human review capacity.
- [DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve](https://feed7.dev/p/deepswe-a-contamination-resistant-coding-benchmark-james-shi-datacurve-08p61c0) — DeepSWE reinforces evaluation on original long-horizon repository tasks and behavioral traces, while this signal adds legacy-code, deployment, and hidden-assumption concerns that its task mix may not fully cover.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, agent-reliability, coding-agents

## Uncertainty

- These are task-specific observations from one refactor, not controlled cross-model results. Human review remained part of the project, and only **15 of 17 requirements** were met during the migration.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.