Training Frontier Models to Out-Think Hackers — Uri Rolls, Arithmetic & Thom Wolf, Hugging Face

The benchmark builds live, chained application environments around **real zero-days** found by human researchers. Agents start with limited access, receive **no internet or source code**, and must infer the system’s state while pursuing an exploit.

For security-agent evals, test black-box reasoning across authentication, permissions, and service boundaries. Record partial progress with **deterministic graders** so model or harness changes can be compared even when full exploitation is rare.

The presented benchmark remained extremely difficult, with **one solve at k=1**. Its access-control scenarios demonstrate a demanding evaluation method, but not yet the broader claim that specialized open models can replace existing defensive systems.