Teaching AI to Find Real Vulnerabilities — Prof. David Brumley, Bugcrowd
Cybersecurity agents need deterministic exploit oracles and tasks that reward finding distinct vulnerabilities, not self-reported wins or one crash. A capability ladder makes partial progress measurable.
Brumley describes reproducible vulnerable containers exposed through MCP, with setup, sandboxed file tools, and deterministic grading. He argues that crash-only checks miss exploit quality and duplicate findings; **50%** of DARPA Cyber Grand Challenge tasks reportedly contained unintended vulnerabilities, while AIxCC produced **18 unintended bugs**.
For security-agent evals, grade concrete effects rather than model claims, deduplicate proofs by the vulnerabilities they trigger, and measure precision and recall across an audit set. A **16-capability ladder** can show where an agent stalls between triggering a flaw and building an out-of-sandbox exploit.
Brumley describes reproducible vulnerable containers exposed through MCP, with setup, sandboxed file tools, and deterministic grading. He argues that crash-only checks miss exploit quality and duplicate findings; **50%** of DARPA Cyber Grand Challenge tasks reportedly contained unintended vulnerabilities, while AIxCC produced **18 unintended bugs**. For security-agent evals, grade concrete effects rather than model claims, deduplicate proofs by the vulnerabilities they trigger, and measure precision and recall across an audit set. A **16-capability ladder** can show where an agent stalls between triggering a flaw and building an out-of-sandbox exploit. Even deterministic oracles encode a chosen definition of progress, and newly discovered bugs can change the ground truth. Publishing transcripts is also unresolved when a benchmark agent produces weaponized, previously private exploits.
This specializes general agent-eval guidance for adversarial security work: success must be an externally verified exploit effect, findings must be deduplicated by the flaw reached, and capability should be measured beyond crashes. It also shows why benchmark ground truth is unusually unstable here—agents can discover unintended bugs—and why trace publication can create a disclosure risk.