Back
AI EngineerVideoSource LinkedbenchmarkNew
Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs
Judge coding agents with explicit requirements and end-to-end tests, since fast output may be incomplete scaffolding.
AI EngineerAI EngineerAI EngineerAug 8, 20262 min
Source Summary
A production refactor shows why coding-agent evaluations need acceptance criteria and end-to-end verification: fast output can still be incomplete scaffolding.
Practical Implication
Benchmark agents against explicit requirements, runnable end-to-end tests, deployment constraints, and hidden assumptions—not elapsed time or lines changed. Wisedocs also found a monorepo simpler for verification and sandbox setup across its former 10+ repositories.
Agent-Ready Context
An early O3-assisted task took **3 hours** and produced 10 major mistakes; newer Sonnet 4.6 solved it after one extra iteration and Opus 4.8 nearly one-shot it. A broader GPT-5.5 attempt finished in **10m 22s** but mostly wrote scaffolding. Benchmark agents against explicit requirements, runnable end-to-end tests, deployment constraints, and hidden assumptions—not elapsed time or lines changed. Wisedocs also found a monorepo simpler for verification and sandbox setup across its former **10+ repositories**. These are task-specific observations from one refactor, not controlled cross-model results. Human review remained part of the project, and only **15 of 17 requirements** were met during the migration.
Context Map
Rate This Item
Personal Note
No note yet. Notes are included in exported bundles.
Related — Every Edge Explained
No approved edges yet for this post.