DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve
DeepSWE uses original long-horizon tasks to reduce contamination and expose coding-agent behaviors hidden by saturated PR-mined suites. Its current task mix still underrepresents some everyday work.
DeepSWE contains **113 original tasks** spanning **91 repositories** and five languages, with a median of one task per repository. Its short prompts still yield solutions averaging five times the lines of code of SWE-Bench Pro.
Use it when comparing coding agents on sustained repository work, and inspect behavioral traces alongside scores. **DeepSWE v1.1** separates verifier and agent runtimes and removes extra Git references to limit reward hacking.
DeepSWE contains **113 original tasks** spanning **91 repositories** and five languages, with a median of one task per repository. Its short prompts still yield solutions averaging five times the lines of code of SWE-Bench Pro. Use it when comparing coding agents on sustained repository work, and inspect behavioral traces alongside scores. **DeepSWE v1.1** separates verifier and agent runtimes and removes extra Git references to limit reward hacking. The suite currently underrepresents bug localization and refactoring. Its authors also want broader repository coverage and hybrid verification, so it is not yet a complete proxy for routine engineering work.
DeepSWE adds a harder, contamination-resistant measure of sustained repository work and makes behavioral traces part of score interpretation. It improves benchmark integrity through runtime separation and fewer exploitable Git references, but its limited coverage of bug localization and refactoring means it cannot stand alone as a proxy for everyday coding-agent performance.