# Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale

Source: [arXiv](https://arxiv.org/abs/2609.15939v1)  
Feed7 permalink: https://feed7.dev/p/2609-15939v1-0w15mi5  
Published: 2026-09-14T17:44:51.000Z  
Trust: Needs Review (needs_review)

## Why Included

VLoc Bench isolates whether security agents can find vulnerable files across real repositories. Its low scores argue for testing localization and post-fix restraint separately from patch generation.

## Source Summary

VLoc Bench contains **500 vulnerabilities from 290 repositories**, spanning six package ecosystems and 147 CWE categories. It evaluates 27 language models and four static-analysis tools through one agent interface with read-only repository access.

## Practical Implication

Security-agent evals should test two distinct behaviors: locating affected files before a fix and declining unsupported findings afterward. The strongest system reached only **0.229 File F1**, so patching ability should not be treated as evidence of reliable repository search.

## Agent-Ready Context

VLoc Bench contains **500 vulnerabilities from 290 repositories**, spanning six package ecosystems and 147 CWE categories. It evaluates 27 language models and four static-analysis tools through one agent interface with read-only repository access.

Security-agent evals should test two distinct behaviors: locating affected files before a fix and declining unsupported findings afterward. The strongest system reached only **0.229 File F1**, so patching ability should not be treated as evidence of reliable repository search.

On **38.4% of tasks**, no evaluated model found a correct location. Better localization also did not ensure restraint on patched snapshots, where agents could still name unsupported files.

## Connected Context

Feed7 judgment across 778 accumulated Signals:

This carves vulnerability localization and post-patch restraint out as capabilities that repair benchmarks and static-analysis results do not establish. It strengthens the candidates’ broader case for multi-gate repository evaluation: success requires finding the right files on vulnerable code and withholding unsupported locations once fixed. The low scores make localization a measured bottleneck rather than an assumed precursor to patching.

- [SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents](https://feed7.dev/p/2609-04167v1-1vvofgw) — Together they split coding-agent success into independent gates: SWE-Gate separates functional repair from constraint compliance, while VLoc Bench separates patching assumptions from accurate localization and post-patch restraint.
- [When LLM Decompilers Recompile More and Preserve Less](https://feed7.dev/p/2609-05370v1-03wuqdv) — Both demonstrate that a downstream success signal can hide a security-relevant failure: recompilation can conceal semantic drift, and a plausible finding can conceal incorrect repository localization.
- [Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd](https://feed7.dev/p/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-1ok0f7q) — The exploit-oracle guidance verifies whether a vulnerability is real, while VLoc Bench adds a prerequisite capability test for locating its affected files and declining findings after remediation.
- [DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve](https://feed7.dev/p/deepswe-a-contamination-resistant-coding-benchmark-james-shi-datacurve-08p61c0) — VLoc Bench directly fills a coverage gap identified for DeepSWE by supplying repository-scale bug localization tasks across many repositories and vulnerability categories.

## Context Map

- Layer: benchmark
- Domains: coding, security
- Topics: agent-evals, benchmark-integrity, agent-reliability

## Uncertainty

- On **38.4% of tasks**, no evaluated model found a correct location. Better localization also did not ensure restraint on patched snapshots, where agents could still name unsupported files.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.