Open Brain
arXivPaperNeeds Review

SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research

A review finds that agents can discover, validate, and patch real software vulnerabilities autonomously, while open-weight models reached comparable autonomy for discovery and validation. Reported vulnerability disclosures rose sharply, but time to fixes and attacks has not noticeably changed.

arXiv · Nov 19, 2026
Open Source Open MarkdownOpen JSON
Source Summary

The paper rates vulnerability research claims by autonomy, correctness, availability, and evidence strength. It finds that agents handled discovery, validation, and patching without human intervention before restricted frontier models arrived; open-weight models later matched discovery and validation. In **2026**, disclosed vulnerabilities **more than doubled**, with more rated critical or high.

Practical Implication

If you run coding agents, treat security work as a capability that needs explicit evaluation and review. The paper’s framework offers a way to distinguish demonstrated capability from claims that are hard to compare.

Agent-Ready Context
The paper rates vulnerability research claims by autonomy, correctness, availability, and evidence strength. It finds that agents handled discovery, validation, and patching without human intervention before restricted frontier models arrived; open-weight models later matched discovery and validation. In **2026**, disclosed vulnerabilities **more than doubled**, with more rated critical or high.

If you run coding agents, treat security work as a capability that needs explicit evaluation and review. The paper’s framework offers a way to distinguish demonstrated capability from claims that are hard to compare.

The reported increase in disclosures has not yet brought a noticeable shift in time from report to fix or disclosure to attack. The material does not detail the paper’s recommended mitigations.
Connected Context · Feed7 Judgment

This SoK broadens vulnerability-agent evaluation from individual task scores to comparable evidence about autonomy, correctness, availability, and evidence strength. It reports capability across discovery, validation, and patching, while the disclosure increase does not yet show faster fixes or attacks. That makes demonstrated capability and operational impact separate claims.

Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware ScreenThe SoK offers a framework for judging capability claims; this repair study shows why one common signal, compile rate, can misstate security-patch quality.Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository ScaleThe localization benchmark isolates finding vulnerable files and restraint after fixes, narrowing which part of the broader discovery-to-patching capability a benchmark actually establishes.Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics AgentsBoth support evaluating distinct failure modes separately: the SoK organizes evidence quality, while this study finds functional checks and red-teaming surface non-overlapping failures.
Context Map
benchmarksecuritycoding#benchmark-integrity#agent-reliability
Uncertainty
The reported increase in disclosures has not yet brought a noticeable shift in time from report to fix or disclosure to attack. The material does not detail the paper’s recommended mitigations.