arXivPaperNeeds Review

Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

Readable chain-of-thought is not a reliable map of which steps caused a correct answer. Treat step-level LLM critiques as partial signals, especially on already-correct traces.

arXiv · Sep 3, 2026

Open Source Open Markdown Open JSON

Source Summary

The study defines a reasoning step’s importance as its **advantage**: the change in expected reward when that step is included, estimated through Monte Carlo rollouts. Capable LLM judges beat a prevalence baseline at finding high-advantage steps but remained below the **noise ceiling**.

Practical Implication

Builders using reasoning traces to debug agents or train process rewards should validate textual critiques against behavioral interventions. A plausible explanation of a step is not evidence that the step materially changed the final result.

Agent-Ready Context

The study defines a reasoning step’s importance as its **advantage**: the change in expected reward when that step is included, estimated through Monte Carlo rollouts. Capable LLM judges beat a prevalence baseline at finding high-advantage steps but remained below the **noise ceiling**.

Builders using reasoning traces to debug agents or train process rewards should validate textual critiques against behavioral interventions. A plausible explanation of a step is not evidence that the step materially changed the final result.

Fine-tuning a step-level critic improved judgments for **incorrect responses**, but performance on **correct responses** stayed far from the ceiling. The material provides no numeric effect sizes, so it supports a directional warning rather than a threshold for accepting or rejecting critics.

Connected Context · Feed7 Judgment

This separates readable reasoning from causally useful reasoning. It confirms that LLM critiques can identify some consequential steps, but narrows their use in debugging and process supervision because judged importance remains below intervention-based estimates, especially on correct answers. For trace training, plausible explanations should therefore be validated by behavioral ablation or rollout evidence before being treated as reliable labels.

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared EndpointsBefore textual step critiques can gate training or debugging, the candidate shows their judge must also pass repeatability tests; causal validity and measurement stability are distinct prerequisites.Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge BiasThe candidate’s evidence of hidden-state judge bias offers a complementary failure mode to this Signal’s gap between plausible critique and behaviorally measured importance.SWE-Prime: Fewer Trajectories, Better PerformanceSWE-Prime makes trace filtering an effective training choice, while this Signal warns that selecting supposedly valuable segments requires intervention-validated importance rather than legible critique alone.

Context Map

benchmarkresearch#reasoning#agent-evals#benchmark-integrity

Uncertainty

Fine-tuning a step-level critic improved judgments for **incorrect responses**, but performance on **correct responses** stayed far from the ceiling. The material provides no numeric effect sizes, so it supports a directional warning rather than a threshold for accepting or rejecting critics.