{
  "schema_version": "1.1",
  "id": "auto-79c903650b",
  "slug": "execcritic-learn-to-test-test-to-improve-for-coding-agen-79c903650b",
  "url": "https://feed7.dev/p/execcritic-learn-to-test-test-to-improve-for-coding-agen-79c903650b",
  "title": "ExecCritic: Learn to Test, Test to Improve for Coding Agents",
  "why_included": "Separate test writing from repair, qualify repository-native tests, and freeze them before the repair agent sees execution feedback.",
  "summary": "ExecCritic separates test writing from repair and freezes qualified tests before code changes. On SWE-bench Verified, weak generated tests hurt resolution, while separately trained roles reached 72.6%.",
  "practical_implication": "Test quality determined whether feedback helped on SWE-bench Verified: the no-test baseline resolved 61.2%, base-agent tests lowered that to 57.3%, and GPT-5.6-sol tests raised it to 65.3%. Builders should isolate test generation from patching and reject tests that fail qualification.",
  "agent_context": "ExecCritic assigns testing and repair to separate Qwen-3.5-35B-A3B agents. A fail-closed harness qualifies and freezes repository-native tests, preventing the Repair agent from changing the target after seeing execution feedback.\n\nTest quality determined whether feedback helped on SWE-bench Verified: the no-test baseline resolved **61.2%**, base-agent tests lowered that to **57.3%**, and GPT-5.6-sol tests raised it to **65.3%**. Builders should isolate test generation from patching and reject tests that fail qualification.\n\nSeparately post-training both Qwen roles produced a **72.6%** resolved rate, 11.4 points above the original no-test baseline without stronger-model or Oracle feedback at evaluation. These are benchmark results; repository-specific costs and generalization are not covered here.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2609.09133v1",
    "published_at": "2026-09-08T00:00:00.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "agent",
  "domains": [
    "coding"
  ],
  "topics": [
    "coding-agents",
    "harness-engineering",
    "agent-reliability"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "Automatically selected from source material; feed7 has not independently tested the claim."
  ],
  "connected_context": null,
  "lifecycle": "New",
  "published_at": "2026-09-08T00:00:00.000Z",
  "modified_at": "2026-09-08T00:00:00.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/execcritic-learn-to-test-test-to-improve-for-coding-agen-79c903650b",
    "json": "https://feed7.dev/p/execcritic-learn-to-test-test-to-improve-for-coding-agen-79c903650b.json",
    "markdown": "https://feed7.dev/p/execcritic-learn-to-test-test-to-improve-for-coding-agen-79c903650b.md"
  }
}