ExecCritic: Learn to Test, Test to Improve for Coding Agents
ExecCritic separates test writing from repair and freezes qualified tests before code changes. On SWE-bench Verified, weak generated tests hurt resolution, while separately trained roles reached 72.6%.
ExecCritic assigns testing and repair to separate Qwen-3.5-35B-A3B agents. A fail-closed harness qualifies and freezes repository-native tests, preventing the Repair agent from changing the target after seeing execution feedback.
Test quality determined whether feedback helped on SWE-bench Verified: the no-test baseline resolved **61.2%**, base-agent tests lowered that to **57.3%**, and GPT-5.6-sol tests raised it to **65.3%**. Builders should isolate test generation from patching and reject tests that fail qualification.
ExecCritic assigns testing and repair to separate Qwen-3.5-35B-A3B agents. A fail-closed harness qualifies and freezes repository-native tests, preventing the Repair agent from changing the target after seeing execution feedback. Test quality determined whether feedback helped on SWE-bench Verified: the no-test baseline resolved **61.2%**, base-agent tests lowered that to **57.3%**, and GPT-5.6-sol tests raised it to **65.3%**. Builders should isolate test generation from patching and reject tests that fail qualification. Separately post-training both Qwen roles produced a **72.6%** resolved rate, 11.4 points above the original no-test baseline without stronger-model or Oracle feedback at evaluation. These are benchmark results; repository-specific costs and generalization are not covered here.
This makes test provenance and qualification first-class controls rather than assuming any execution feedback helps. The benchmark reversal shows weak generated tests can be worse than no tests, while isolated roles, frozen qualified tests, and separate post-training turn testing from an informal agent habit into a governed repair interface.