EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence
EpiSEM-Bench tests whether tool-using agents notice missing safety evidence, seek it out, and still complete permitted work instead of blindly acting or refusing.
EpiSEM-Bench contains **300 matched unsafe/safe task pairs** across **six domains**. It suppresses critical facts and audits whether agents recover the evidence, block prohibited actions, and complete permissible ones.
Treat missing tool output as an explicit harness state. Agents should request safety-critical facts before acting, while evals should test both unsafe execution and over-refusal on paired tasks.
EpiSEM-Bench contains **300 matched unsafe/safe task pairs** across **six domains**. It suppresses critical facts and audits whether agents recover the evidence, block prohibited actions, and complete permissible ones. Treat missing tool output as an explicit harness state. Agents should request safety-critical facts before acting, while evals should test both unsafe execution and over-refusal on paired tasks. Results varied across **ten models**. Recovery access reduced unsafe execution for every model, but omission resistance did not reliably imply active evidence gathering or appropriate action.
EpiSEM-Bench adds missing evidence as a first-class safety state and separates three behaviors that simpler refusal tests can conflate: seeking critical facts, blocking unsafe action, and completing safe action. Recovery access consistently reduced unsafe execution, but the results show that passive resistance to omissions does not establish active evidence gathering or well-calibrated action.