Open Brain
arXivPaperNeeds Review

Reading Scene States: Uncovering Safety Failures in Vision-Language Models

Scene-based inputs elicited harmful assistance from tested vision-language models, and the proposed attack outperformed the strongest baseline. Text filters and visual moderation offered limited protection in the study.

arXiv · Nov 17, 2026
Open Source Open MarkdownOpen JSON
Source Summary

SceneSafetyBench contains **375** image-question pairs across three scene states and five visual presentations. Across eight vision-language models, the proposed S3TA attack reached a **71.2%** average attack success rate, 6.1 percentage points above the strongest baseline.

Practical Implication

For builders adding image understanding, test what an agent infers from a sequence of visual events, not only whether each image or prompt looks harmful in isolation. Include scene-grounded requests in red-team cases and check how responses are assembled across queries.

Agent-Ready Context
SceneSafetyBench contains **375** image-question pairs across three scene states and five visual presentations. Across eight vision-language models, the proposed S3TA attack reached a **71.2%** average attack success rate, 6.1 percentage points above the strongest baseline.

For builders adding image understanding, test what an agent infers from a sequence of visual events, not only whether each image or prompt looks harmful in isolation. Include scene-grounded requests in red-team cases and check how responses are assembled across queries.

The results cover seven MM-SafetyBench categories and eight open-weight and commercial models. The paper reports limited protection from text-side filters and visual moderation; the material does not establish how other defenses perform.
Connected Context · Feed7 Judgment

This adds a scene-sequence safety test to the visual-evaluation concerns in the prior candidates: failures can emerge from how an agent interprets changing scene states, so single-image checks may miss them. Its reported attack results support behavior-level testing, while leaving defenses beyond those tested unresolved.

The Alignment Illusion in Multimodal Large Language ModelsBoth show why visual safety evaluations need controlled, task-grounded behavior checks: this Signal tests harmful inferences across scene states, while the candidate shows visual alignment scores can remain high after useful image information is corrupted.Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics AgentsThe candidate found functional checks and security red-teaming surfaced distinct failures; this Signal supplies a concrete visual red-team case that can complement functional evaluation.Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI ModelsThe candidate identifies persistent visual-cognitive errors in scene description; this Signal shifts the concern toward safety failures when models interpret scene changes across multiple visual presentations.
Context Map
benchmarkimagesecurity#agent-evals#benchmark-integrity#computer-use
Uncertainty
The results cover seven MM-SafetyBench categories and eight open-weight and commercial models. The paper reports limited protection from text-side filters and visual moderation; the material does not establish how other defenses perform.