# QuoteBench: How Matched Scores Can Hide Command-Path Failures

Source: [arXiv](https://arxiv.org/abs/2608.13547v1)  
Feed7 permalink: https://feed7.dev/p/2608-13547v1-130h6xd  
Published: 2026-08-13T17:57:20.000Z  
Trust: Needs Review (needs_review)

## Why Included

QuoteBench shows that shell-command scores can conceal failures introduced by serialization and reparsing. Agent evals should identify the execution path, not attribute every result to the model.

## Source Summary

QuoteBench tests **56 one-shot tasks** from 14 incident-derived families across eight configurations. Adding one unescaped parser cut replayed-command success by **55.4–73.2 percentage points**.

## Practical Implication

When the boundary was disclosed, six configurations recovered **30.4–60.7 points**. Builders should test generated commands through the exact production transport and validate final state, especially where wrappers interpolate or reparse shell text.

## Agent-Ready Context

QuoteBench tests **56 one-shot tasks** from 14 incident-derived families across eight configurations. Adding one unescaped parser cut replayed-command success by **55.4–73.2 percentage points**.

When the boundary was disclosed, six configurations recovered **30.4–60.7 points**. Builders should test generated commands through the exact production transport and validate final state, especially where wrappers interpolate or reparse shell text.

Adaptation was uneven: two configurations recovered nothing or declined slightly. GPT-5.6-sol’s **−3.6-point matched gap** concealed −64.3 points of transport damage plus 60.7 points of compensation, so aggregate scores can misstate both model and harness quality.

## Connected Context

Feed7 judgment across 462 accumulated Signals:

This provides quantified evidence that command transport is part of the evaluated system: a model can appear unchanged in aggregate while suffering severe parser-induced failures and partially compensating after disclosure. It strengthens calls for cross-harness testing and final-state validation, and further narrows leaderboard interpretation because matched scores can hide offsetting model adaptation and harness damage.

- [State of Data — Sean Cai, Independent / State of Data](https://feed7.dev/p/state-of-data-sean-cai-independent-state-of-data-0v9fy69) — QuoteBench quantifies the earlier claim that scores shift with scaffolding and shows why evaluation should inspect tool trajectories and resulting state rather than only aggregate task success.
- [The Bitter Lesson of Tool Calling](https://feed7.dev/p/2608-06370v1-1ar4rfv) — Both establish interface representation as a consequential harness variable; QuoteBench adds that reparsing and escaping along the production command path can dominate apparent model performance.
- [Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd](https://feed7.dev/p/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-1ok0f7q) — Bugcrowd’s deterministic exploit oracles reinforce QuoteBench’s requirement to validate concrete final effects instead of trusting generated commands or self-reported completion.
- [Quantifying infrastructure noise in agentic coding evals](https://feed7.dev/p/infrastructure-noise-1jyyyw1) — Anthropic shows resource configuration can move scores, while QuoteBench demonstrates a different hidden systems effect: transport damage can be masked by model compensation even when the aggregate gap is small.

## Context Map

- Layer: benchmark
- Domains: coding, security
- Topics: agent-evals, benchmark-integrity, harness-engineering

## Uncertainty

- Adaptation was uneven: two configurations recovered nothing or declined slightly. GPT-5.6-sol’s **−3.6-point matched gap** concealed −64.3 points of transport damage plus 60.7 points of compensation, so aggregate scores can misstate both model and harness quality.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.