{
  "schema_version": "1.1",
  "id": "s13:https://arxiv.org/abs/2609.15939v1",
  "slug": "2609-15939v1-0w15mi5",
  "url": "https://feed7.dev/p/2609-15939v1-0w15mi5",
  "title": "Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale",
  "why_included": "VLoc Bench isolates whether security agents can find vulnerable files across real repositories. Its low scores argue for testing localization and post-fix restraint separately from patch generation.",
  "summary": "VLoc Bench contains **500 vulnerabilities from 290 repositories**, spanning six package ecosystems and 147 CWE categories. It evaluates 27 language models and four static-analysis tools through one agent interface with read-only repository access.",
  "practical_implication": "Security-agent evals should test two distinct behaviors: locating affected files before a fix and declining unsupported findings afterward. The strongest system reached only **0.229 File F1**, so patching ability should not be treated as evidence of reliable repository search.",
  "agent_context": "VLoc Bench contains **500 vulnerabilities from 290 repositories**, spanning six package ecosystems and 147 CWE categories. It evaluates 27 language models and four static-analysis tools through one agent interface with read-only repository access.\n\nSecurity-agent evals should test two distinct behaviors: locating affected files before a fix and declining unsupported findings afterward. The strongest system reached only **0.229 File F1**, so patching ability should not be treated as evidence of reliable repository search.\n\nOn **38.4% of tasks**, no evaluated model found a correct location. Better localization also did not ensure restraint on patched snapshots, where agents could still name unsupported files.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2609.15939v1",
    "published_at": "2026-09-14T17:44:51.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "benchmark",
  "domains": [
    "coding",
    "security"
  ],
  "topics": [
    "agent-evals",
    "benchmark-integrity",
    "agent-reliability"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "On **38.4% of tasks**, no evaluated model found a correct location. Better localization also did not ensure restraint on patched snapshots, where agents could still name unsupported files."
  ],
  "connected_context": {
    "meaning": "This carves vulnerability localization and post-patch restraint out as capabilities that repair benchmarks and static-analysis results do not establish. It strengthens the candidates’ broader case for multi-gate repository evaluation: success requires finding the right files on vulnerable code and withholding unsupported locations once fixed. The low scores make localization a measured bottleneck rather than an assumed precursor to patching.",
    "corpus_size": 778,
    "generated_at": "2026-09-15T10:06:50.635Z",
    "connections": [
      {
        "title": "SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.04167v1",
        "feed7_url": "https://feed7.dev/p/2609-04167v1-1vvofgw",
        "reason": "Together they split coding-agent success into independent gates: SWE-Gate separates functional repair from constraint compliance, while VLoc Bench separates patching assumptions from accurate localization and post-patch restraint."
      },
      {
        "title": "When LLM Decompilers Recompile More and Preserve Less",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.05370v1",
        "feed7_url": "https://feed7.dev/p/2609-05370v1-03wuqdv",
        "reason": "Both demonstrate that a downstream success signal can hide a security-relevant failure: recompilation can conceal semantic drift, and a plausible finding can conceal incorrect repository localization."
      },
      {
        "title": "Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=ZFxh7sqbUZo",
        "feed7_url": "https://feed7.dev/p/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-1ok0f7q",
        "reason": "The exploit-oracle guidance verifies whether a vulnerability is real, while VLoc Bench adds a prerequisite capability test for locating its affected files and declining findings after remediation."
      },
      {
        "title": "DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=Yk87oUPVaxU",
        "feed7_url": "https://feed7.dev/p/deepswe-a-contamination-resistant-coding-benchmark-james-shi-datacurve-08p61c0",
        "reason": "VLoc Bench directly fills a coverage gap identified for DeepSWE by supplying repository-scale bug localization tasks across many repositories and vulnerability categories."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-09-14T17:44:51.000Z",
  "modified_at": "2026-09-14T17:44:51.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2609-15939v1-0w15mi5",
    "json": "https://feed7.dev/p/2609-15939v1-0w15mi5.json",
    "markdown": "https://feed7.dev/p/2609-15939v1-0w15mi5.md"
  }
}