# ExecCritic: Learn to Test, Test to Improve for Coding Agents

Source: [arXiv](https://arxiv.org/abs/2609.09133v1)  
Feed7 permalink: https://feed7.dev/p/2609-09133v1-0zfcxb4  
Published: 2026-09-08T17:53:37.000Z  
Trust: Needs Review (needs_review)

## Why Included

ExecCritic separates test writing from repair and freezes qualified tests before code changes. On SWE-bench Verified, weak generated tests hurt resolution, while separately trained roles reached 72.6%.

## Source Summary

ExecCritic assigns testing and repair to separate Qwen-3.5-35B-A3B agents. A fail-closed harness qualifies and freezes repository-native tests, preventing the Repair agent from changing the target after seeing execution feedback.

## Practical Implication

Test quality determined whether feedback helped on SWE-bench Verified: the no-test baseline resolved **61.2%**, base-agent tests lowered that to **57.3%**, and GPT-5.6-sol tests raised it to **65.3%**. Builders should isolate test generation from patching and reject tests that fail qualification.

## Agent-Ready Context

ExecCritic assigns testing and repair to separate Qwen-3.5-35B-A3B agents. A fail-closed harness qualifies and freezes repository-native tests, preventing the Repair agent from changing the target after seeing execution feedback.

Test quality determined whether feedback helped on SWE-bench Verified: the no-test baseline resolved **61.2%**, base-agent tests lowered that to **57.3%**, and GPT-5.6-sol tests raised it to **65.3%**. Builders should isolate test generation from patching and reject tests that fail qualification.

Separately post-training both Qwen roles produced a **72.6%** resolved rate, 11.4 points above the original no-test baseline without stronger-model or Oracle feedback at evaluation. These are benchmark results; repository-specific costs and generalization are not covered here.

## Connected Context

Feed7 judgment across 713 accumulated Signals:

This makes test provenance and qualification first-class controls rather than assuming any execution feedback helps. The benchmark reversal shows weak generated tests can be worse than no tests, while isolated roles, frozen qualified tests, and separate post-training turn testing from an informal agent habit into a governed repair interface.

- [MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation](https://feed7.dev/p/2607-22471v1-0kivszm) — Both prevent unreliable generated tests from steering code changes; ExecCritic adds strict role isolation and freezes repository-native tests before repair feedback is exposed.
- [How to Kill the Code Review — Ankit Jain, Aviator](https://feed7.dev/p/how-to-kill-the-code-review-ankit-jain-aviator-0rku6kj) — Its qualified, immutable tests provide the kind of verification evidence reviewers can inspect, while the weak-test result supports retaining human judgment over generated checks.
- [How to Get Your Org to Adopt Coding Agents (Without Shipping Garbage) — Eyal Blum, Figma](https://feed7.dev/p/how-to-get-your-org-to-adopt-coding-agents-without-shipping-garbage-eyal-04y3rj2) — It supplies a concrete safeguard for the adoption concern that verification quality, not generation alone, determines whether agent-written code is trustworthy.
- [Loop Engineering from First Principles — Kyle Mistele, HumanLayer](https://feed7.dev/p/loop-engineering-from-first-principles-kyle-mistele-humanlayer-1nuq7gf) — Separating test creation from bounded repair gives the incremental coding loop a fail-closed feedback boundary before a patch proceeds to review.

## Context Map

- Layer: agent
- Domains: coding
- Topics: coding-agents, harness-engineering, agent-reliability

## Uncertainty

- Separately post-training both Qwen roles produced a **72.6%** resolved rate, 11.4 points above the original no-test baseline without stronger-model or Oracle feedback at evaluation. These are benchmark results; repository-specific costs and generalization are not covered here.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.