# Reading Scene States: Uncovering Safety Failures in Vision-Language Models

Source: [arXiv](https://arxiv.org/abs/2611.23206v1)  
Feed7 permalink: https://feed7.dev/p/2611-23206v1-1kzrrws  
Published: 2026-11-17T18:48:02.000Z  
Trust: Needs Review (needs_review)

## Why Included

Scene-based inputs elicited harmful assistance from tested vision-language models, and the proposed attack outperformed the strongest baseline. Text filters and visual moderation offered limited protection in the study.

## Source Summary

SceneSafetyBench contains **375** image-question pairs across three scene states and five visual presentations. Across eight vision-language models, the proposed S3TA attack reached a **71.2%** average attack success rate, 6.1 percentage points above the strongest baseline.

## Practical Implication

For builders adding image understanding, test what an agent infers from a sequence of visual events, not only whether each image or prompt looks harmful in isolation. Include scene-grounded requests in red-team cases and check how responses are assembled across queries.

## Agent-Ready Context

SceneSafetyBench contains **375** image-question pairs across three scene states and five visual presentations. Across eight vision-language models, the proposed S3TA attack reached a **71.2%** average attack success rate, 6.1 percentage points above the strongest baseline.

For builders adding image understanding, test what an agent infers from a sequence of visual events, not only whether each image or prompt looks harmful in isolation. Include scene-grounded requests in red-team cases and check how responses are assembled across queries.

The results cover seven MM-SafetyBench categories and eight open-weight and commercial models. The paper reports limited protection from text-side filters and visual moderation; the material does not establish how other defenses perform.

## Connected Context

Feed7 judgment across 1116 accumulated Signals:

This adds a scene-sequence safety test to the visual-evaluation concerns in the prior candidates: failures can emerge from how an agent interprets changing scene states, so single-image checks may miss them. Its reported attack results support behavior-level testing, while leaving defenses beyond those tested unresolved.

- [The Alignment Illusion in Multimodal Large Language Models](https://feed7.dev/p/2609-30210v1-198in2y) — Both show why visual safety evaluations need controlled, task-grounded behavior checks: this Signal tests harmful inferences across scene states, while the candidate shows visual alignment scores can remain high after useful image information is corrupted.
- [Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents](https://feed7.dev/p/2611-14493v1-1ghdgem) — The candidate found functional checks and security red-teaming surfaced distinct failures; this Signal supplies a concrete visual red-team case that can complement functional evaluation.
- [Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models](https://feed7.dev/p/2607-09654v1-0b5dedg) — The candidate identifies persistent visual-cognitive errors in scene description; this Signal shifts the concern toward safety failures when models interpret scene changes across multiple visual presentations.

## Context Map

- Layer: benchmark
- Domains: image, security
- Topics: agent-evals, benchmark-integrity, computer-use

## Uncertainty

- The results cover seven MM-SafetyBench categories and eight open-weight and commercial models. The paper reports limited protection from text-side filters and visual moderation; the material does not establish how other defenses perform.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.
