# Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

Source: [arXiv](https://arxiv.org/abs/2609.04194v1)  
Feed7 permalink: https://feed7.dev/p/2609-04194v1-0milybu  
Published: 2026-09-03T17:59:08.000Z  
Trust: Needs Review (needs_review)

## Why Included

Readable chain-of-thought is not a reliable map of which steps caused a correct answer. Treat step-level LLM critiques as partial signals, especially on already-correct traces.

## Source Summary

The study defines a reasoning step’s importance as its **advantage**: the change in expected reward when that step is included, estimated through Monte Carlo rollouts. Capable LLM judges beat a prevalence baseline at finding high-advantage steps but remained below the **noise ceiling**.

## Practical Implication

Builders using reasoning traces to debug agents or train process rewards should validate textual critiques against behavioral interventions. A plausible explanation of a step is not evidence that the step materially changed the final result.

## Agent-Ready Context

The study defines a reasoning step’s importance as its **advantage**: the change in expected reward when that step is included, estimated through Monte Carlo rollouts. Capable LLM judges beat a prevalence baseline at finding high-advantage steps but remained below the **noise ceiling**.

Builders using reasoning traces to debug agents or train process rewards should validate textual critiques against behavioral interventions. A plausible explanation of a step is not evidence that the step materially changed the final result.

Fine-tuning a step-level critic improved judgments for **incorrect responses**, but performance on **correct responses** stayed far from the ceiling. The material provides no numeric effect sizes, so it supports a directional warning rather than a threshold for accepting or rejecting critics.

## Connected Context

Feed7 judgment across 691 accumulated Signals:

This separates readable reasoning from causally useful reasoning. It confirms that LLM critiques can identify some consequential steps, but narrows their use in debugging and process supervision because judged importance remains below intervention-based estimates, especially on correct answers. For trace training, plausible explanations should therefore be validated by behavioral ablation or rollout evidence before being treated as reliable labels.

- [Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints](https://feed7.dev/p/2609-04198v1-1mype86) — Before textual step critiques can gate training or debugging, the candidate shows their judge must also pass repeatability tests; causal validity and measurement stability are distinct prerequisites.
- [Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias](https://feed7.dev/p/2607-11871v1-17vejh0) — The candidate’s evidence of hidden-state judge bias offers a complementary failure mode to this Signal’s gap between plausible critique and behaviorally measured importance.
- [SWE-Prime: Fewer Trajectories, Better Performance](https://feed7.dev/p/2608-27449v1-0lkylqd) — SWE-Prime makes trace filtering an effective training choice, while this Signal warns that selecting supposedly valuable segments requires intervention-validated importance rather than legible critique alone.

## Context Map

- Layer: benchmark
- Domains: research
- Topics: reasoning, agent-evals, benchmark-integrity

## Uncertainty

- Fine-tuning a step-level critic improved judgments for **incorrect responses**, but performance on **correct responses** stayed far from the ceiling. The material provides no numeric effect sizes, so it supports a directional warning rather than a threshold for accepting or rejecting critics.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.