# The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

Source: [arXiv](https://arxiv.org/abs/2607.12963v1)  
Feed7 permalink: https://feed7.dev/p/2607-12963v1-1oc0qmr  
Published: 2026-07-14T17:01:12.000Z  
Trust: Needs Review (needs_review)

## Why Included

Stable aggregate accuracy can hide individual answers flipping when irrelevant context is added. Agent evaluations should compare outputs per task and probe realistic context noise, not only average scores.

## Source Summary

Across multiple models and datasets, adding irrelevant context caused little aggregate accuracy change but flipped predictions on a subset of examples. Even **random-character pseudo-words** could improve some answers while degrading others.

## Practical Implication

Evaluate coding agents at the example level: rerun the same task with irrelevant files, longer context, or harmless textual noise, then track answer flips and regressions separately from average pass rates.

## Agent-Ready Context

Across multiple models and datasets, adding irrelevant context caused little aggregate accuracy change but flipped predictions on a subset of examples. Even **random-character pseudo-words** could improve some answers while degrading others.

Evaluate coding agents at the example level: rerun the same task with irrelevant files, longer context, or harmless textual noise, then track answer flips and regressions separately from average pass rates.

The affected examples were **largely model-specific**, and instability varied with context type, length, test-time compute, and model development stage. The supplied abstract gives no effect sizes, so it does not establish how frequent the tail risk is in coding workloads.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: context-engineering, agent-evals, agent-reliability

## Uncertainty

- The affected examples were **largely model-specific**, and instability varied with context type, length, test-time compute, and model development stage. The supplied abstract gives no effect sizes, so it does not establish how frequent the tail risk is in coding workloads.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.