Open Brain
arXivPaperNeeds Review

Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

Repeated model answers can be correlated, so a self-consistency test may certify one prompt’s result rather than average model reliability. The paper finds that testing across prompts can restore population-level validity.

arXiv · Nov 19, 2026
Open Source Open MarkdownOpen JSON
Source Summary

The paper shows that correlated reasoning rollouts change what an anytime-valid self-consistency test certifies: a realized belief for a sampled prompt, rather than average model behavior. On **AIME**, the single-prompt rule crossed its threshold on **24%** of prompts at a nominal **5%** error level; the prompt-level procedure did not cross.

Practical Implication

When evaluating coding agents, repeated samples from one task may not provide independent evidence about overall reliability. Allocate evaluation runs across prompts when the goal is a claim about population performance.

Agent-Ready Context
The paper shows that correlated reasoning rollouts change what an anytime-valid self-consistency test certifies: a realized belief for a sampled prompt, rather than average model behavior. On **AIME**, the single-prompt rule crossed its threshold on **24%** of prompts at a nominal **5%** error level; the prompt-level procedure did not cross.

When evaluating coding agents, repeated samples from one task may not provide independent evidence about overall reliability. Allocate evaluation runs across prompts when the goal is a claim about population performance.

The study’s examples are **GSM8K** and AIME, and its results depend on the modeled correlation structure. The prompt-level method’s proposed allocation assumes a pure rollout budget.
Connected Context · Feed7 Judgment

This result narrows what repeated self-consistency samples can certify: under correlated rollouts, a threshold crossing supports belief about the sampled prompt, not average model reliability. Its AIME example shows the single-prompt rule can cross far more often than its nominal error level suggests, while the prompt-level procedure does not cross. Population claims therefore require runs distributed across prompts.

On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement LearningBoth show reliability varies by prompt under finite sampling; this paper concerns what evaluation can certify, while the TTRL study uses prompt sensitivity to make training updates more conservative.Not All Reasoning Generalizes: A Case Study on Learning Generalizable Reasoning Strategies in RLVRThe correlated-rollout result cautions against broad reliability claims from repeated samples on one task; the RLVR study separately shows familiar-task success may not generalize under task shifts.Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon ReasoningBoth motivate evaluation across task variation: this paper distinguishes prompt-level from population evidence, while Skill²-Bench tests reliability across dependent skill transitions.
Context Map
benchmarkresearch#agent-evals#agent-reliability#reasoning
Uncertainty
The study’s examples are **GSM8K** and AIME, and its results depend on the modeled correlation structure. The prompt-level method’s proposed allocation assumes a pure rollout budget.