{
  "schema_version": "1.1",
  "id": "archive:https://arxiv.org/abs/2611.26592v1",
  "slug": "2611-26592v1-18gptmp",
  "url": "https://feed7.dev/p/2611-26592v1-18gptmp",
  "title": "Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning",
  "why_included": "Repeated model answers can be correlated, so a self-consistency test may certify one prompt’s result rather than average model reliability. The paper finds that testing across prompts can restore population-level validity.",
  "summary": "The paper shows that correlated reasoning rollouts change what an anytime-valid self-consistency test certifies: a realized belief for a sampled prompt, rather than average model behavior. On **AIME**, the single-prompt rule crossed its threshold on **24%** of prompts at a nominal **5%** error level; the prompt-level procedure did not cross.",
  "practical_implication": "When evaluating coding agents, repeated samples from one task may not provide independent evidence about overall reliability. Allocate evaluation runs across prompts when the goal is a claim about population performance.",
  "agent_context": "The paper shows that correlated reasoning rollouts change what an anytime-valid self-consistency test certifies: a realized belief for a sampled prompt, rather than average model behavior. On **AIME**, the single-prompt rule crossed its threshold on **24%** of prompts at a nominal **5%** error level; the prompt-level procedure did not cross.\n\nWhen evaluating coding agents, repeated samples from one task may not provide independent evidence about overall reliability. Allocate evaluation runs across prompts when the goal is a claim about population performance.\n\nThe study’s examples are **GSM8K** and AIME, and its results depend on the modeled correlation structure. The prompt-level method’s proposed allocation assumes a pure rollout budget.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2611.26592v1",
    "published_at": "2026-11-19T18:47:51.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "benchmark",
  "domains": [
    "research"
  ],
  "topics": [
    "agent-evals",
    "agent-reliability",
    "reasoning"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "The study’s examples are **GSM8K** and AIME, and its results depend on the modeled correlation structure. The prompt-level method’s proposed allocation assumes a pure rollout budget."
  ],
  "connected_context": {
    "meaning": "This result narrows what repeated self-consistency samples can certify: under correlated rollouts, a threshold crossing supports belief about the sampled prompt, not average model reliability. Its AIME example shows the single-prompt rule can cross far more often than its nominal error level suggests, while the prompt-level procedure does not cross. Population claims therefore require runs distributed across prompts.",
    "corpus_size": 1135,
    "generated_at": "2026-11-21T11:06:57.237Z",
    "connections": [
      {
        "title": "On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2611.09872v1",
        "feed7_url": "https://feed7.dev/p/2611-09872v1-0hxr9dl",
        "reason": "Both show reliability varies by prompt under finite sampling; this paper concerns what evaluation can certify, while the TTRL study uses prompt sensitivity to make training updates more conservative."
      },
      {
        "title": "Not All Reasoning Generalizes: A Case Study on Learning Generalizable Reasoning Strategies in RLVR",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2611.03627v1",
        "feed7_url": "https://feed7.dev/p/2611-03627v1-0h0gkar",
        "reason": "The correlated-rollout result cautions against broad reliability claims from repeated samples on one task; the RLVR study separately shows familiar-task success may not generalize under task shifts."
      },
      {
        "title": "Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2608.05139v1",
        "feed7_url": "https://feed7.dev/p/2608-05139v1-0ji039d",
        "reason": "Both motivate evaluation across task variation: this paper distinguishes prompt-level from population evidence, while Skill²-Bench tests reliability across dependent skill transitions."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-11-19T18:47:51.000Z",
  "modified_at": "2026-11-19T18:47:51.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2611-26592v1-18gptmp",
    "json": "https://feed7.dev/p/2611-26592v1-18gptmp.json",
    "markdown": "https://feed7.dev/p/2611-26592v1-18gptmp.md"
  }
}