AI EngineerVideoSource Linked
DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve
DeepSWE uses original long-horizon tasks to reduce contamination and expose coding-agent behaviors hidden by saturated PR-mined suites. Its current task mix still underrepresents some everyday work.
AI Engineer · Jul 26, 2026
Source Summary
DeepSWE contains **113 original tasks** spanning **91 repositories** and five languages, with a median of one task per repository. Its short prompts still yield solutions averaging five times the lines of code of SWE-Bench Pro.
Practical Implication
Use it when comparing coding agents on sustained repository work, and inspect behavioral traces alongside scores. **DeepSWE v1.1** separates verifier and agent runtimes and removes extra Git references to limit reward hacking.
Agent-Ready Context
DeepSWE contains **113 original tasks** spanning **91 repositories** and five languages, with a median of one task per repository. Its short prompts still yield solutions averaging five times the lines of code of SWE-Bench Pro. Use it when comparing coding agents on sustained repository work, and inspect behavioral traces alongside scores. **DeepSWE v1.1** separates verifier and agent runtimes and removes extra Git references to limit reward hacking. The suite currently underrepresents bug localization and refactoring. Its authors also want broader repository coverage and hybrid verification, so it is not yet a complete proxy for routine engineering work.
Context Map
benchmarkcoding#agent-evals#benchmark-integrity#agent-reliabilityUncertainty
The suite currently underrepresents bug localization and refactoring. Its authors also want broader repository coverage and hybrid verification, so it is not yet a complete proxy for routine engineering work.