Simulated Control Evals
Simulated tool responses predicted deployment-specific control safety more closely than a separate executed setting, offering a cheaper way to test agent safeguards before deployment.
On LinuxArena, simulated control evaluations achieved **3.6 percentage-point RMSE** when predicting safety across five monitors at a 5% audit budget. BashArena, a separate executed control setting, reached **10.1 points**.
Builders testing high-risk agents should model deployment-specific tools and tasks instead of relying only on a generic control environment. Simulation can also surface attack strategies that improve monitors on held-out executed attacks.
On LinuxArena, simulated control evaluations achieved **3.6 percentage-point RMSE** when predicting safety across five monitors at a 5% audit budget. BashArena, a separate executed control setting, reached **10.1 points**. Builders testing high-risk agents should model deployment-specific tools and tasks instead of relying only on a generic control environment. Simulation can also surface attack strategies that improve monitors on held-out executed attacks. SCEs explained **78–85% of between-environment variance** and used fewer than 5% as many task-pair draws as the deployment set. The evidence comes from LinuxArena, so transfer to other tools and workflows remains open.
This makes simulated control evaluation a cheaper, deployment-specific forecasting layer rather than a substitute for executed safety tests. It extends prior calls for small eval loops and adversarial testing by showing that modeled tools and tasks can explain much of the safety variation and generate attacks that improve monitors, while the Linux-only evidence leaves transfer to other agent environments unresolved.