# SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research

Source: [arXiv](https://arxiv.org/abs/2611.26601v1)  
Feed7 permalink: https://feed7.dev/p/2611-26601v1-07flpj4  
Published: 2026-11-19T18:54:19.000Z  
Trust: Needs Review (needs_review)

## Why Included

A review finds that agents can discover, validate, and patch real software vulnerabilities autonomously, while open-weight models reached comparable autonomy for discovery and validation. Reported vulnerability disclosures rose sharply, but time to fixes and attacks has not noticeably changed.

## Source Summary

The paper rates vulnerability research claims by autonomy, correctness, availability, and evidence strength. It finds that agents handled discovery, validation, and patching without human intervention before restricted frontier models arrived; open-weight models later matched discovery and validation. In **2026**, disclosed vulnerabilities **more than doubled**, with more rated critical or high.

## Practical Implication

If you run coding agents, treat security work as a capability that needs explicit evaluation and review. The paper’s framework offers a way to distinguish demonstrated capability from claims that are hard to compare.

## Agent-Ready Context

The paper rates vulnerability research claims by autonomy, correctness, availability, and evidence strength. It finds that agents handled discovery, validation, and patching without human intervention before restricted frontier models arrived; open-weight models later matched discovery and validation. In **2026**, disclosed vulnerabilities **more than doubled**, with more rated critical or high.

If you run coding agents, treat security work as a capability that needs explicit evaluation and review. The paper’s framework offers a way to distinguish demonstrated capability from claims that are hard to compare.

The reported increase in disclosures has not yet brought a noticeable shift in time from report to fix or disclosure to attack. The material does not detail the paper’s recommended mitigations.

## Connected Context

Feed7 judgment across 1135 accumulated Signals:

This SoK broadens vulnerability-agent evaluation from individual task scores to comparable evidence about autonomy, correctness, availability, and evidence strength. It reports capability across discovery, validation, and patching, while the disclosure increase does not yet show faster fixes or attacks. That makes demonstrated capability and operational impact separate claims.

- [Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen](https://feed7.dev/p/2609-26749v1-1fj6jga) — The SoK offers a framework for judging capability claims; this repair study shows why one common signal, compile rate, can misstate security-patch quality.
- [Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale](https://feed7.dev/p/2609-15939v1-0w15mi5) — The localization benchmark isolates finding vulnerable files and restraint after fixes, narrowing which part of the broader discovery-to-patching capability a benchmark actually establishes.
- [Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents](https://feed7.dev/p/2611-14493v1-1ghdgem) — Both support evaluating distinct failure modes separately: the SoK organizes evidence quality, while this study finds functional checks and red-teaming surface non-overlapping failures.

## Context Map

- Layer: benchmark
- Domains: security, coding
- Topics: benchmark-integrity, agent-reliability

## Uncertainty

- The reported increase in disclosures has not yet brought a noticeable shift in time from report to fix or disclosure to attack. The material does not detail the paper’s recommended mitigations.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.
