SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research
A review finds that agents can discover, validate, and patch real software vulnerabilities autonomously, while open-weight models reached comparable autonomy for discovery and validation. Reported vulnerability disclosures rose sharply, but time to fixes and attacks has not noticeably changed.
The paper rates vulnerability research claims by autonomy, correctness, availability, and evidence strength. It finds that agents handled discovery, validation, and patching without human intervention before restricted frontier models arrived; open-weight models later matched discovery and validation. In **2026**, disclosed vulnerabilities **more than doubled**, with more rated critical or high.
If you run coding agents, treat security work as a capability that needs explicit evaluation and review. The paper’s framework offers a way to distinguish demonstrated capability from claims that are hard to compare.
The paper rates vulnerability research claims by autonomy, correctness, availability, and evidence strength. It finds that agents handled discovery, validation, and patching without human intervention before restricted frontier models arrived; open-weight models later matched discovery and validation. In **2026**, disclosed vulnerabilities **more than doubled**, with more rated critical or high. If you run coding agents, treat security work as a capability that needs explicit evaluation and review. The paper’s framework offers a way to distinguish demonstrated capability from claims that are hard to compare. The reported increase in disclosures has not yet brought a noticeable shift in time from report to fix or disclosure to attack. The material does not detail the paper’s recommended mitigations.
This SoK broadens vulnerability-agent evaluation from individual task scores to comparable evidence about autonomy, correctness, availability, and evidence strength. It reports capability across discovery, validation, and patching, while the disclosure increase does not yet show faster fixes or attacks. That makes demonstrated capability and operational impact separate claims.