Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd
Cybersecurity agents need deterministic exploit oracles and tasks that reward finding distinct vulnerabilities, not self-reported wins or one crash. A capability ladder makes partial progress measurable.
Brumley describes reproducible vulnerable containers exposed through MCP, with setup, sandboxed file tools, and deterministic grading. He argues that crash-only checks miss exploit quality and duplicate findings; **50%** of DARPA Cyber Grand Challenge tasks reportedly contained unintended vulnerabilities, while AIxCC produced **18 unintended bugs**.
For security-agent evals, grade concrete effects rather than model claims, deduplicate proofs by the vulnerabilities they trigger, and measure precision and recall across an audit set. A **16-capability ladder** can show where an agent stalls between triggering a flaw and building an out-of-sandbox exploit.
Brumley describes reproducible vulnerable containers exposed through MCP, with setup, sandboxed file tools, and deterministic grading. He argues that crash-only checks miss exploit quality and duplicate findings; **50%** of DARPA Cyber Grand Challenge tasks reportedly contained unintended vulnerabilities, while AIxCC produced **18 unintended bugs**. For security-agent evals, grade concrete effects rather than model claims, deduplicate proofs by the vulnerabilities they trigger, and measure precision and recall across an audit set. A **16-capability ladder** can show where an agent stalls between triggering a flaw and building an out-of-sandbox exploit. Even deterministic oracles encode a chosen definition of progress, and newly discovered bugs can change the ground truth. Publishing transcripts is also unresolved when a benchmark agent produces weaponized, previously private exploits.
This specializes benchmark-integrity guidance for security agents: verify concrete exploit effects, prevent duplicate findings from inflating scores, and use a capability ladder to expose partial progress. It strengthens final-state grading but also limits confidence in deterministic oracles, because their definitions and ground truth can change, while useful trace publication may create security risk.