arXivPaperNeeds Review

Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

Readable chain-of-thought is not a reliable map of which steps caused a correct answer. Treat step-level LLM critiques as partial signals, especially on already-correct traces.

arXiv · Sep 3, 2026

Open Source Open Markdown Open JSON

Source Summary

The study defines a reasoning step’s importance as its **advantage**: the change in expected reward when that step is included, estimated through Monte Carlo rollouts. Capable LLM judges beat a prevalence baseline at finding high-advantage steps but remained below the **noise ceiling**.

Practical Implication

Builders using reasoning traces to debug agents or train process rewards should validate textual critiques against behavioral interventions. A plausible explanation of a step is not evidence that the step materially changed the final result.

Agent-Ready Context

The study defines a reasoning step’s importance as its **advantage**: the change in expected reward when that step is included, estimated through Monte Carlo rollouts. Capable LLM judges beat a prevalence baseline at finding high-advantage steps but remained below the **noise ceiling**.

Builders using reasoning traces to debug agents or train process rewards should validate textual critiques against behavioral interventions. A plausible explanation of a step is not evidence that the step materially changed the final result.

Fine-tuning a step-level critic improved judgments for **incorrect responses**, but performance on **correct responses** stayed far from the ceiling. The material provides no numeric effect sizes, so it supports a directional warning rather than a threshold for accepting or rejecting critics.

Connected Context · Feed7 Judgment

This separates readable explanations of reasoning from evidence that a step caused success. Against prior judge-integrity concerns, it adds an intervention-based target: compare textual critiques with the reward change produced by including the step. Fine-tuning helps identify important steps in incorrect answers but leaves correct-answer analysis well below the attainable ceiling, limiting process critics as debugging or training authorities.

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared EndpointsThe observer study asks whether judge outputs are repeatable; this study asks whether step judgments track behavioral importance. A process critic used as a gate must satisfy both reliability and validity.Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge BiasHidden-state evidence of judge bias challenges impartiality, while this Signal shows that even plausible step critiques can diverge from the steps that actually affect reward.SWE-Prime: Fewer Trajectories, Better PerformanceSWE-Prime relies on selecting valuable trajectory segments for supervision; this Signal supplies a caution that legible process-quality judgments should be checked against behavioral interventions before determining which segments receive loss.Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and ReproducibilityFull inference-protocol reporting enables replay of reasoning systems; this Signal adds that replayable traces alone do not establish which recorded steps were causally important.

Context Map

benchmarkresearch#reasoning#agent-evals#benchmark-integrity

Uncertainty

Fine-tuning a step-level critic improved judgments for **incorrect responses**, but performance on **correct responses** stayed far from the ceiling. The material provides no numeric effect sizes, so it supports a directional warning rather than a threshold for accepting or rejecting critics.