Back
arXivPaperNeeds ReviewagentNew
ExecCritic: Learn to Test, Test to Improve for Coding Agents
Separate test writing from repair, qualify repository-native tests, and freeze them before the repair agent sees execution feedback.
arXivSep 8, 20262 min
Source Summary
ExecCritic separates test writing from repair and freezes qualified tests before code changes. On SWE-bench Verified, weak generated tests hurt resolution, while separately trained roles reached 72.6%.
Practical Implication
Test quality determined whether feedback helped on SWE-bench Verified: the no-test baseline resolved 61.2%, base-agent tests lowered that to 57.3%, and GPT-5.6-sol tests raised it to 65.3%. Builders should isolate test generation from patching and reject tests that fail qualification.
Agent-Ready Context
ExecCritic assigns testing and repair to separate Qwen-3.5-35B-A3B agents. A fail-closed harness qualifies and freezes repository-native tests, preventing the Repair agent from changing the target after seeing execution feedback. Test quality determined whether feedback helped on SWE-bench Verified: the no-test baseline resolved **61.2%**, base-agent tests lowered that to **57.3%**, and GPT-5.6-sol tests raised it to **65.3%**. Builders should isolate test generation from patching and reject tests that fail qualification. Separately post-training both Qwen roles produced a **72.6%** resolved rate, 11.4 points above the original no-test baseline without stronger-model or Oracle feedback at evaluation. These are benchmark results; repository-specific costs and generalization are not covered here.
Context Map
Uncertainty
Automatically selected from source material; feed7 has not independently tested the claim.
Rate This Item
Personal Note
No note yet. Notes are included in exported bundles.
Related — Every Edge Explained
No approved edges yet for this post.