{
  "schema_version": "1.1",
  "id": "s13:https://arxiv.org/abs/2609.26749v1",
  "slug": "2609-26749v1-1fj6jga",
  "url": "https://feed7.dev/p/2609-26749v1-1fj6jga",
  "title": "Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen",
  "why_included": "Compile rate can reward broken security patches and mostly reflect harness artifacts. Vulnerability-repair agents need change-aware screening followed by execution-grounded evaluation.",
  "summary": "Across **203 vulnerable functions**, about **64% of compile failures** were not attributed to the model. A compiler-standard flag changed compile rates by 1.8–2.7× on identical patches, and compiler feedback increased compiling outputs while introducing non-repairs.",
  "practical_implication": "Do not use compilation as the main reward or acceptance gate for vulnerability-fixing agents. Screen whether a patch made a relevant edit, then require execution-grounded security checks; diff_F1 can cheaply reject no-ops before deeper evaluation.",
  "agent_context": "Across **203 vulnerable functions**, about **64% of compile failures** were not attributed to the model. A compiler-standard flag changed compile rates by 1.8–2.7× on identical patches, and compiler feedback increased compiling outputs while introducing non-repairs.\n\nDo not use compilation as the main reward or acceptance gate for vulnerability-fixing agents. Screen whether a patch made a relevant edit, then require execution-grounded security checks; diff_F1 can cheaply reject no-ops before deeper evaluation.\n\nWhole-function CodeBLEU also failed because an unchanged vulnerable function beat every model. **diff_F1 gives zero credit to a no-op**, but misses some deletion-based gaming and is explicitly not a repair-quality metric.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2609.26749v1",
    "published_at": "2026-09-22T17:32:05.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "benchmark",
  "domains": [
    "coding",
    "security"
  ],
  "topics": [
    "agent-evals",
    "benchmark-integrity",
    "agent-reliability"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "Whole-function CodeBLEU also failed because an unchanged vulnerable function beat every model. **diff_F1 gives zero credit to a no-op**, but misses some deletion-based gaming and is explicitly not a repair-quality metric."
  ],
  "connected_context": {
    "meaning": "This identifies compilation and whole-function similarity as actively misleading rewards for vulnerability repair, not merely incomplete metrics. It separates harness-caused failures from model failures and proposes diff_F1 only as a cheap no-op screen before execution-grounded security evaluation, explicitly limiting it from being treated as a quality score.",
    "corpus_size": 856,
    "generated_at": "2026-09-23T09:06:24.727Z",
    "connections": [
      {
        "title": "When LLM Decompilers Recompile More and Preserve Less",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.05370v1",
        "feed7_url": "https://feed7.dev/p/2609-05370v1-03wuqdv",
        "reason": "The decompiler findings independently show that compilability can rise while semantic and security-relevant behavior worsens, reinforcing the need for execution-based acceptance."
      },
      {
        "title": "SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.04167v1",
        "feed7_url": "https://feed7.dev/p/2609-04167v1-1vvofgw",
        "reason": "SWE-Gate shows that functional tests miss review-derived constraints; this study makes the analogous security case that compilation must be separated from actual repair correctness."
      },
      {
        "title": "Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=ZFxh7sqbUZo",
        "feed7_url": "https://feed7.dev/p/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-1ok0f7q",
        "reason": "Its call for deterministic exploit oracles supplies the kind of execution-grounded security evidence needed after diff_F1 rejects obvious no-ops."
      },
      {
        "title": "Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.15939v1",
        "feed7_url": "https://feed7.dev/p/2609-15939v1-0w15mi5",
        "reason": "VLoc Bench evaluates finding the vulnerable location, while this work evaluates whether the subsequent edit is relevant and securely effective, forming distinct gates in a repair pipeline."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-09-22T17:32:05.000Z",
  "modified_at": "2026-09-22T17:32:05.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2609-26749v1-1fj6jga",
    "json": "https://feed7.dev/p/2609-26749v1-1fj6jga.json",
    "markdown": "https://feed7.dev/p/2609-26749v1-1fj6jga.md"
  }
}