Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale
VLoc Bench isolates whether security agents can find vulnerable files across real repositories. Its low scores argue for testing localization and post-fix restraint separately from patch generation.
VLoc Bench contains **500 vulnerabilities from 290 repositories**, spanning six package ecosystems and 147 CWE categories. It evaluates 27 language models and four static-analysis tools through one agent interface with read-only repository access.
Security-agent evals should test two distinct behaviors: locating affected files before a fix and declining unsupported findings afterward. The strongest system reached only **0.229 File F1**, so patching ability should not be treated as evidence of reliable repository search.
VLoc Bench contains **500 vulnerabilities from 290 repositories**, spanning six package ecosystems and 147 CWE categories. It evaluates 27 language models and four static-analysis tools through one agent interface with read-only repository access. Security-agent evals should test two distinct behaviors: locating affected files before a fix and declining unsupported findings afterward. The strongest system reached only **0.229 File F1**, so patching ability should not be treated as evidence of reliable repository search. On **38.4% of tasks**, no evaluated model found a correct location. Better localization also did not ensure restraint on patched snapshots, where agents could still name unsupported files.
This carves vulnerability localization and post-patch restraint out as capabilities that repair benchmarks and static-analysis results do not establish. It strengthens the candidates’ broader case for multi-gate repository evaluation: success requires finding the right files on vulnerable code and withholding unsupported locations once fixed. The low scores make localization a measured bottleneck rather than an assumed precursor to patching.