Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning
Repeated model answers can be correlated, so a self-consistency test may certify one prompt’s result rather than average model reliability. The paper finds that testing across prompts can restore population-level validity.
The paper shows that correlated reasoning rollouts change what an anytime-valid self-consistency test certifies: a realized belief for a sampled prompt, rather than average model behavior. On **AIME**, the single-prompt rule crossed its threshold on **24%** of prompts at a nominal **5%** error level; the prompt-level procedure did not cross.
When evaluating coding agents, repeated samples from one task may not provide independent evidence about overall reliability. Allocate evaluation runs across prompts when the goal is a claim about population performance.
The paper shows that correlated reasoning rollouts change what an anytime-valid self-consistency test certifies: a realized belief for a sampled prompt, rather than average model behavior. On **AIME**, the single-prompt rule crossed its threshold on **24%** of prompts at a nominal **5%** error level; the prompt-level procedure did not cross. When evaluating coding agents, repeated samples from one task may not provide independent evidence about overall reliability. Allocate evaluation runs across prompts when the goal is a claim about population performance. The study’s examples are **GSM8K** and AIME, and its results depend on the modeled correlation structure. The prompt-level method’s proposed allocation assumes a pure rollout budget.
This result narrows what repeated self-consistency samples can certify: under correlated rollouts, a threshold crossing supports belief about the sampled prompt, not average model reliability. Its AIME example shows the single-prompt rule can cross far more often than its nominal error level suggests, while the prompt-level procedure does not cross. Population claims therefore require runs distributed across prompts.