arXivPaperNeeds Review
The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context
Stable aggregate accuracy can hide individual answers flipping when irrelevant context is added. Agent evaluations should compare outputs per task and probe realistic context noise, not only average scores.
arXiv · Jul 14, 2026
Source Summary
Across multiple models and datasets, adding irrelevant context caused little aggregate accuracy change but flipped predictions on a subset of examples. Even **random-character pseudo-words** could improve some answers while degrading others.
Practical Implication
Evaluate coding agents at the example level: rerun the same task with irrelevant files, longer context, or harmless textual noise, then track answer flips and regressions separately from average pass rates.
Agent-Ready Context
Across multiple models and datasets, adding irrelevant context caused little aggregate accuracy change but flipped predictions on a subset of examples. Even **random-character pseudo-words** could improve some answers while degrading others. Evaluate coding agents at the example level: rerun the same task with irrelevant files, longer context, or harmless textual noise, then track answer flips and regressions separately from average pass rates. The affected examples were **largely model-specific**, and instability varied with context type, length, test-time compute, and model development stage. The supplied abstract gives no effect sizes, so it does not establish how frequent the tail risk is in coding workloads.
Context Map
benchmarkcoding#context-engineering#agent-evals#agent-reliabilityUncertainty
The affected examples were **largely model-specific**, and instability varied with context type, length, test-time compute, and model development stage. The supplied abstract gives no effect sizes, so it does not establish how frequent the tail risk is in coding workloads.