# SABRE: Scalable and Automated Benchmarking of VLMs under Stress

Source: [arXiv](https://arxiv.org/abs/2608.07435v1)  
Feed7 permalink: https://feed7.dev/p/2608-07435v1-0h6gzdk  
Published: 2026-08-07T17:21:04.000Z  
Trust: Needs Review (needs_review)

## Why Included

SABRE turns a Markdown test design into generated VLM stress tests, then filters and repairs candidates. It offers a repeatable pattern for refreshing evals as models improve.

## Source Summary

SABRE converts a Markdown task design into specifications, images, and question-answer pairs, then applies model filtering and human review. SABRE-Prior includes **600 images** and **1,000 questions** testing whether models follow visual evidence over learned expectations.

## Practical Implication

Teams evaluating vision agents can encode test intent and schema first, generate candidates, discard easy cases, then reserve human effort for validity checks, annotation fixes, and localized image repair.

## Agent-Ready Context

SABRE converts a Markdown task design into specifications, images, and question-answer pairs, then applies model filtering and human review. SABRE-Prior includes **600 images** and **1,000 questions** testing whether models follow visual evidence over learned expectations.

Teams evaluating vision agents can encode test intent and schema first, generate candidates, discard easy cases, then reserve human effort for validity checks, annotation fixes, and localized image repair.

Across **six VLMs**, macro-average accuracy ranged from **17.8% to 31.3%**. A real-image control was comparably difficult for the filtering model, so low scores cannot be attributed only to counterfactual generated imagery.

## Connected Context

Feed7 judgment across 409 accumulated Signals:

SABRE supplies a scalable construction pipeline for adversarial visual-evidence tests, combining specification-first generation, model-based difficulty filtering, and targeted human repair. Relative to the candidates, it makes expectation-versus-evidence conflict directly testable and reports broad failure across six VLMs, while its real-image control narrows the explanation for low scores beyond synthetic-image artifacts.

- [Evidence-Backed Video Question Answering](https://feed7.dev/p/2607-11862v1-18as4nc) — E-VQA’s pixel-level evidence requirement offers a complementary way to diagnose whether SABRE failures reflect weak visual grounding rather than merely incorrect final answers.
- [Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models](https://feed7.dev/p/2607-09654v1-0b5dedg) — The decade study’s spatial-attention differences provide a candidate failure signal for the visual-cognitive errors that SABRE provokes when learned expectations conflict with image evidence.
- [Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber](https://feed7.dev/p/building-closed-loop-evals-for-a-multimodal-agent-at-scale-soumya-gupta-1cqjbe2) — Uber’s golden-set gates, iterative QA, and production feedback reinforce SABRE’s implementation pattern of automated filtering followed by concentrated human validation, while applying it to a deployed image-editing agent.

## Context Map

- Layer: benchmark
- Domains: image
- Topics: agent-evals, benchmark-integrity, generative-media

## Uncertainty

- Across **six VLMs**, macro-average accuracy ranged from **17.8% to 31.3%**. A real-image control was comparably difficult for the filtering model, so low scores cannot be attributed only to counterfactual generated imagery.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.