Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
Compile rate can reward broken security patches and mostly reflect harness artifacts. Vulnerability-repair agents need change-aware screening followed by execution-grounded evaluation.
Across **203 vulnerable functions**, about **64% of compile failures** were not attributed to the model. A compiler-standard flag changed compile rates by 1.8–2.7× on identical patches, and compiler feedback increased compiling outputs while introducing non-repairs.
Do not use compilation as the main reward or acceptance gate for vulnerability-fixing agents. Screen whether a patch made a relevant edit, then require execution-grounded security checks; diff_F1 can cheaply reject no-ops before deeper evaluation.
Across **203 vulnerable functions**, about **64% of compile failures** were not attributed to the model. A compiler-standard flag changed compile rates by 1.8–2.7× on identical patches, and compiler feedback increased compiling outputs while introducing non-repairs. Do not use compilation as the main reward or acceptance gate for vulnerability-fixing agents. Screen whether a patch made a relevant edit, then require execution-grounded security checks; diff_F1 can cheaply reject no-ops before deeper evaluation. Whole-function CodeBLEU also failed because an unchanged vulnerable function beat every model. **diff_F1 gives zero credit to a no-op**, but misses some deletion-based gaming and is explicitly not a repair-quality metric.
This identifies compilation and whole-function similarity as actively misleading rewards for vulnerability repair, not merely incomplete metrics. It separates harness-caused failures from model failures and proposes diff_F1 only as a cheap no-op screen before execution-grounded security evaluation, explicitly limiting it from being treated as a quality score.