Reading Scene States: Uncovering Safety Failures in Vision-Language Models
Scene-based inputs elicited harmful assistance from tested vision-language models, and the proposed attack outperformed the strongest baseline. Text filters and visual moderation offered limited protection in the study.
SceneSafetyBench contains **375** image-question pairs across three scene states and five visual presentations. Across eight vision-language models, the proposed S3TA attack reached a **71.2%** average attack success rate, 6.1 percentage points above the strongest baseline.
For builders adding image understanding, test what an agent infers from a sequence of visual events, not only whether each image or prompt looks harmful in isolation. Include scene-grounded requests in red-team cases and check how responses are assembled across queries.
SceneSafetyBench contains **375** image-question pairs across three scene states and five visual presentations. Across eight vision-language models, the proposed S3TA attack reached a **71.2%** average attack success rate, 6.1 percentage points above the strongest baseline. For builders adding image understanding, test what an agent infers from a sequence of visual events, not only whether each image or prompt looks harmful in isolation. Include scene-grounded requests in red-team cases and check how responses are assembled across queries. The results cover seven MM-SafetyBench categories and eight open-weight and commercial models. The paper reports limited protection from text-side filters and visual moderation; the material does not establish how other defenses perform.
This adds a scene-sequence safety test to the visual-evaluation concerns in the prior candidates: failures can emerge from how an agent interprets changing scene states, so single-image checks may miss them. Its reported attack results support behavior-level testing, while leaving defenses beyond those tested unresolved.