# Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

Source: [arXiv](https://arxiv.org/abs/2609.26749v1)  
Feed7 permalink: https://feed7.dev/p/2609-26749v1-1fj6jga  
Published: 2026-09-22T17:32:05.000Z  
Trust: Needs Review (needs_review)

## Why Included

Compile rate can reward broken security patches and mostly reflect harness artifacts. Vulnerability-repair agents need change-aware screening followed by execution-grounded evaluation.

## Source Summary

Across **203 vulnerable functions**, about **64% of compile failures** were not attributed to the model. A compiler-standard flag changed compile rates by 1.8–2.7× on identical patches, and compiler feedback increased compiling outputs while introducing non-repairs.

## Practical Implication

Do not use compilation as the main reward or acceptance gate for vulnerability-fixing agents. Screen whether a patch made a relevant edit, then require execution-grounded security checks; diff_F1 can cheaply reject no-ops before deeper evaluation.

## Agent-Ready Context

Across **203 vulnerable functions**, about **64% of compile failures** were not attributed to the model. A compiler-standard flag changed compile rates by 1.8–2.7× on identical patches, and compiler feedback increased compiling outputs while introducing non-repairs.

Do not use compilation as the main reward or acceptance gate for vulnerability-fixing agents. Screen whether a patch made a relevant edit, then require execution-grounded security checks; diff_F1 can cheaply reject no-ops before deeper evaluation.

Whole-function CodeBLEU also failed because an unchanged vulnerable function beat every model. **diff_F1 gives zero credit to a no-op**, but misses some deletion-based gaming and is explicitly not a repair-quality metric.

## Connected Context

Feed7 judgment across 856 accumulated Signals:

This identifies compilation and whole-function similarity as actively misleading rewards for vulnerability repair, not merely incomplete metrics. It separates harness-caused failures from model failures and proposes diff_F1 only as a cheap no-op screen before execution-grounded security evaluation, explicitly limiting it from being treated as a quality score.

- [When LLM Decompilers Recompile More and Preserve Less](https://feed7.dev/p/2609-05370v1-03wuqdv) — The decompiler findings independently show that compilability can rise while semantic and security-relevant behavior worsens, reinforcing the need for execution-based acceptance.
- [SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents](https://feed7.dev/p/2609-04167v1-1vvofgw) — SWE-Gate shows that functional tests miss review-derived constraints; this study makes the analogous security case that compilation must be separated from actual repair correctness.
- [Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd](https://feed7.dev/p/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-1ok0f7q) — Its call for deterministic exploit oracles supplies the kind of execution-grounded security evidence needed after diff_F1 rejects obvious no-ops.
- [Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale](https://feed7.dev/p/2609-15939v1-0w15mi5) — VLoc Bench evaluates finding the vulnerable location, while this work evaluates whether the subsequent edit is relevant and securely effective, forming distinct gates in a repair pipeline.

## Context Map

- Layer: benchmark
- Domains: coding, security
- Topics: agent-evals, benchmark-integrity, agent-reliability

## Uncertainty

- Whole-function CodeBLEU also failed because an unchanged vulnerable function beat every model. **diff_F1 gives zero credit to a no-op**, but misses some deletion-based gaming and is explicitly not a repair-quality metric.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.