Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
This paper separates three test-time scaling regimes and argues that model evals must report the full inference protocol, compute accounting, uncertainty, and replay artifacts.
The paper divides test-time scaling into **three regimes**: extending one trajectory, reducing completed candidates, and searching partial prefixes. It treats the model plus inference procedure as the evaluated system and assembles **over 2 billion reasoning traces**.
When comparing reasoning models or agent policies, record the exact inference algorithm, compute budget, aggregation or verifier behavior, and uncertainty. Keep end-to-end performance separate from diagnostics on a pre-generated candidate bank.
The paper divides test-time scaling into **three regimes**: extending one trajectory, reducing completed candidates, and searching partial prefixes. It treats the model plus inference procedure as the evaluated system and assembles **over 2 billion reasoning traces**. When comparing reasoning models or agent policies, record the exact inference algorithm, compute budget, aggregation or verifier behavior, and uncertainty. Keep end-to-end performance separate from diagnostics on a pre-generated candidate bank. A single budget number can hide statistically different procedures and failure modes. Exact replay and distributional reproducibility also require different artifacts, so nominally matched evaluations may still not be directly comparable.
This narrows evaluation claims from “more test-time compute” to a fully specified model-and-inference system. It explains why repeated sampling, voting, prefix search, and single-trajectory extension cannot be treated as interchangeable, and adds an artifact-level reproducibility requirement beyond the contamination, grader, and distribution-validity concerns in the prior candidates.