PaperlayerView exact revision on arXiv

Source-linked AI summary

Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment

Yihuai Hong, Shauli Ravfogel, Chen Zhao, Eunsol Choi

arXiv:2609.38972v1cs.CLcs.AI

TL;DR

CoT traces can diverge from a model’s internal reasoning, motivating a way to measure their alignment. The paper proposes CIA, an interpretability-based metric, and uses it alongside task accuracy in post-training; alignment is initially imperfect, while training improves it with task-dependent mechanisms.

  • Problem

    CoT traces often fail to reflect internal computation and can be manipulated without changing outputs, raising questions about their reliability as explanations.

    Source: §1 · p. 1

  • Method

    CIA uses interpretability tools to infer internal strategies and assess whether they appear in CoT; the paper evaluates it across tasks and uses it with task accuracy as a post-training reward.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Results

    Post-training substantially improved CIA while maintaining task accuracy; the best method for each setup achieved an average relative gain of 25.5%.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    CIA provides a framework for auditing CoT parametric faithfulness, and the observed improvements arise through reasoning changes or reporting changes depending on the task.

    Source: §7 · p. 10

    Evaluation reliability depends on interpretability tools recovering internal reasoning trajectories, which current tools do not do perfectly.

    Source: §8 · p. 10

Abstract

from arXiv · show

Chain-of-thought (CoT) traces often serve as a proxy for how Large Language Models (LLMs) arrive at their answers. However, growing evidence shows that models' CoT often fails to reflect their internal computations and can be changed without affecting their final answers. In this work, we measure and improve the alignment between the reasoning described in an LLM's CoT and what it computes internally. We propose CoT-Interpretability Alignment (CIA), a metric that measures the agreement between a model's CoT traces and its internal reasoning strategies as detected by interpretability tools. We evaluate CIA on three tasks (two-hop question answering, hint intervention, and integer multiplication) across three LLMs, finding that LLMs exhibit limited alignment across all tasks (44.8-75.9%). We then experiment with improving CIA via post-training, setting both the task accuracy and parametric faithfulness signals as a reward. Experiments show that we can substantially improve CoT parametric faithfulness while maintaining or improving the task accuracy. We provide rich analysis, such as their generalization patterns. Our work provides both a framework for auditing CoT parametric faithfulness and a pathway toward making models' explicit reasoning more trustworthy. Code and data are available at https://github.com/yihuaihong/CIA-minimal-repro.

1 INTRODUCTION

The introduction asks whether a model’s verbalized chain of thought can be aligned with its internal computation, motivated by evidence that CoT can be unfaithful and manipulable. It proposes an interpretability-based alignment measure and reports imperfect alignment alongside post-training gains whose mechanisms vary by task.

1 INTRODUCTION

  • The paper targets the gap between CoT explanations and internal computation, which prior work finds can diverge or be manipulated without changing answers.The authors connect greater alignment with improved monitorability and interpretation of model behavior.Source: §1 · p. 1
  • CIA scores were 0.448–0.759 across three evaluated LLMs and three tasks, indicating consistently imperfect alignment.The tasks test knowledge composition, contextual reasoning, and numerical computation.Source: §1 · p. 1
  • CIA uses interpretability tools to infer a model’s internal strategy and tests whether that strategy appears in its CoT.The metric is proposed as a way to quantify CoT parametric faithfulness.Source: §1 · pp. 1–2
  • The best post-training method for each setup achieved an average relative CIA gain of 25.5%, with transfer between reporting-change tasks but not to 2-Digit Multiplication.Gains also generalized across interpretability-based evaluation techniques.Source: §1 · p. 2
  • Improvements arise through different task-specific modes: multiplication changes internal reasoning, while two-hop and hint tasks change how existing strategies are reported.The authors validate these findings through causal interventions.Source: §1 · p. 2

2 RELATED WORK

Prior work studies CoT faithfulness through consistency tests and tests of whether explanations reflect internal reasoning. This paper connects those questions to interpretability tools for inspecting internal strategies.

2 RELATED WORK

3 MEASURING CHAIN-OF-THOUGHT INTERPRETABILITY ALIGNMENT

The section defines CIA as agreement between a task-relevant strategy verbalized in CoT and one detected internally by interpretability tools, scored with macro F1. It then specifies how the strategy and evidence of its use are assessed across two-hop reasoning, hint interventions, and integer multiplication.

3.1 DEFINING CIA

  • CIA uses CoT strategy as the predicted label and interpretability-detected strategy as the reference, scoring their agreement with macro F1 across positive and negative classes.The task prompt is assumed to elicit one gold strategy; CIA measures faithfulness rather than answer correctness, and imperfect tools may prevent perfectly aligned systems from scoring perfectly.Source: §3.1 · p. 3

3.2 TASK SETUP

  • The three tasks test compositional knowledge, contextual reasoning, and numerical computation, with a linear probe used across all three and task-specific auxiliary tools for generalization checks.The tasks are two-hop factual reasoning, hint interventions, and integer multiplication.Source: §3.2 · pp. 3–4
  • For two-hop questions, the gold strategy is reaching the answer through the annotated bridge entity rather than retrieving the final answer directly.Internal use is assessed with linear probes and Tuned Lens at the subject token in the question or first CoT step; CoT use is checked for the gold bridge across the two steps.Source: §3.2 · p. 4
  • In hint interventions, the gold strategy is whether the model uses an injected misleading hint, with probe labels trained from answer-probability shifts between hinted and unhinted conditions.The probe uses hidden states at the last token of the injected hint; a stronger model judges whether CoT explicitly reveals hint reliance.Source: §3.2 · pp. 4–5
  • For integer multiplication, the gold strategy is internally summing displayed partial products, while CoT use is judged by whether the displayed arithmetic is coherent.Probe labels come from corrupting a partial product and checking whether the regenerated sum tracks the corruption; attention analysis checks whether summation-step attention focuses on partial-product tokens.Source: §3.2 · p. 5

4 RESULTS: COT-INTERPRETABILITY ALIGNMENT ACROSS MODELS AND TASKS

Across three reasoning tasks, CIA reveals substantial gaps between LLMs’ CoT and internal strategies, with different forms of misalignment across tasks. Probe reliability and the hint-task accuracy measure provide important context for interpreting the results.

Experimental Setup

  • Probe reliability bounds CIA’s validity, and held-out probe accuracy of 0.83–0.91 with positive-class F1 of 0.75–0.95 supports its reliability.For MMLU-Hint, task accuracy is measured on unbiased, hint-free prompts because biased-prompt accuracy conflates reasoning ability with hint resistance.Source: §4 · pp. 5–6

Results

  • CIA ranges from 0.448 to 0.759, and higher task accuracy does not consistently correspond to higher faithfulness across models and tasks.On TwoHopFact, Gemma2 is most accurate but least faithful; on 2-Digit Multiplication, Qwen3 is most accurate but less faithful than Gemma2.Source: §4 · p. 6
  • TwoHopFact and 2-Digit Multiplication show distinct mismatches: CoT may verbalize unused reasoning, while internal use of written steps may not match the stated result.In TwoHopFact, (BINT=0, BCoT=1) occurs in 32.0–54.3% of cases; multiplication patterns vary by model, including direct recall despite coherent partial products.Source: §4 · p. 6
  • In MMLU-Hint, Llama3.1 and Gemma2 often use hints without acknowledging them in CoT, while Qwen3 is rarely influenced by hints.The (BINT=1, BCoT=0) category accounts for 27.0–29.9% for Llama3.1 and Gemma2; Qwen3 is influenced in 13.8% of cases.Source: §4 · p. 6

5 POST-TRAINING TO IMPROVE CIA

The paper uses interpretability-derived signals to train models for closer alignment between verbalized CoT and internal computation. Post-training improves CIA across tasks, with transfer between TwoHopFact and MMLU-Hint and improvements that also appear under auxiliary interpretability tools.

5 POST-TRAINING TO IMPROVE CIA

  • Training uses linear probes to derive internal-strategy labels that serve as supervisory targets or reward signals, and evaluates with CIA and other metrics.The same interpretability tool, Linear Probes, supplies the BINT labels across tasks.Source: §5 · p. 6

5.2 POST-TRAINING RESULTS OF CIA

  • 25 of 27 post-training cells significantly improve CIA over Base, with gains across all three tasks and models; accuracy is not lowered by more than 0.015 on TwoHopFact and MMLU-Hint, and rises on 2-Digit Multiplication.The best method varies by task: RS on TwoHopFact, DPO on MMLU-Hint, and RS or DPO depending on model for multiplication.Source: Table 2 · p. 7, §5.2 · p. 7

5.1 POST-TRAINING DESIGN

  • RS fine-tunes on sampled completions whose CoT agrees with internal computation, while DPO and GRPO use a reward combining task accuracy and CIA.The task-accuracy base reward is retained alongside a λ-weighted agreement reward to discourage trivial strategy collapse; λ is 1.0 in the experiments.Source: §5.1 · p. 7

5.2 POST-TRAINING RESULTS OF CIA

  • CIA improvements also appear with auxiliary interpretability tools across tasks and models, with GRPO yielding the smallest gains in most settings.The auxiliary tools are Tuned Lens for TwoHopFact, Biasing Features for MMLU-Hint, and Attention Pattern Analysis for multiplication.Source: Table 3 · p. 8, §5.2 · p. 8
  • Training on TwoHopFact and MMLU-Hint transfers CIA gains in both directions, whereas multiplication training transfers at most +0.03 to the other tasks and gains from the other tasks transfer little to multiplication.Figure 5 encodes ∆CIA as trained-on-row minus base, evaluated on the column task; its diagonal marks in-domain results.Source: §5.2 · pp. 7–8, Figure 5 · p. 8

6 UNDERSTANDING THE SOURCES OF CIA IMPROVEMENT

Transition analysis and causal interventions distinguish whether post-training changes internal reasoning or changes CoT reporting. The evidence indicates reasoning shifts in multiplication and reporting shifts in TwoHopFact and MMLU-Hint.

Decomposing CIA Gains via Transition Analysis

  • Transition analysis finds multiplication shifts from (0, 1) to (1, 1) (+7.6%), while MMLU-Hint and TwoHopFact shift from (0, 1) to (0, 0) (+8.2% and +21.0%), respectively.The multiplication transition reflects following the verbalized procedure; the other shifts reflect stopping claims about reasoning or hints not used internally.Source: §6 · p. 8

Causal Intervention Analysis Results

  • On the migrated subset, multiplication’s causal effect rate rises from 29.0% to 72.0%, while rates remain comparable for TwoHopFact and MMLU-Hint, supporting different mechanisms of CIA improvement.The whole-test-set results show the same task pattern with smaller pre-to-post changes.Source: Figure 6 · p. 9, §6 · p. 9

Causal Validation of Mechanism Shifts

  • Causal tests intervene on the bridge entity’s hidden state, remove the hint sentence, or corrupt multiplication partial products, then measure the percentage of outputs that change.The migrated subset tests whether faithfulness transitions reflect genuine internal change rather than superficial probe satisfaction.Source: §6 · p. 9

7 CONCLUSION

The paper presents CIA as a way to measure and improve CoT parametric faithfulness, with task-dependent mechanisms underlying the improvements.

7 CONCLUSION

  • CIA measures alignment between verbalized CoT and internal computation; post-training improves faithfulness while maintaining task accuracy, through reasoning changes or reporting changes depending on the task.The reported improvements generalize across interpretability tools and tasks.Source: §7 · p. 10

8 LIMITATIONS AND FUTURE WORK

The evaluation depends on interpretability tools that do not perfectly recover internal reasoning, and the paper identifies two directions for extending its work.

8 LIMITATIONS AND FUTURE WORK

  • The reliability of parametric-faithfulness evaluation is limited by interpretability tools’ imperfect recovery of models’ internal reasoning trajectories.The framework and training pipeline are structurally decoupled from any particular tool, so future tools can supply new ground-truth labels without architectural changes.Source: §8 · p. 10
  • Long-chain tasks may combine basic reasoning abilities whose faithfulness-training objectives conflict, leaving their compatibility within one post-training stage unresolved.The evaluated tasks each isolate a single basic capability, while real-world long-chain reasoning may combine several.Source: §8 · p. 10
  • For latent-reasoning architectures, a future direction is translating implicit reasoning into faithful, interpretable textual explanations after the reasoning is complete.The paper contrasts these architectures with traditional transformers’ explicit CoT reasoning.Source: §8 · p. 10

A DETAILS OF THE DATASETS

The appendix describes dataset construction and prompting, then illustrates how two-hop CoT can report an internal bridge-entity strategy after GRPO training.

A DETAILS OF THE DATASETS

  • The datasets are split into training, validation, and test sets in a 6:2:2 ratio.The section introduces three reasoning tasks; the supplied TwoHopFact description covers two-hop factual questions and their bridge entities.Source: A DETAILS OF THE DATASETS · p. 14
  • TwoHopFact questions require composing two factual relations, and each includes an annotated bridge entity representing the intermediate step.Single-hop questions are also provided and used to train linear probes.Source: A DETAILS OF THE DATASETS · p. 14

Task 1: Two-Hop Factual Reasoning

  • After GRPO, the example CoT states the correct bridge entity, whereas the vanilla output gives an incorrect bridge entity despite reaching the same final answer.For “Plane Crazy,” the post-training trace names InnerWorkings; the vanilla trace names PopCap Games, while both give NASDAQ.Source: Task 1: Two-Hop Factual Reasoning · p. 15

Task 2: Hint Intervention Task

The appendix specifies inference and interpretability procedures across reasoning tasks, including how probes detect bridge entities, hint influence, and algorithmic multiplication.

Task 2: Hint Intervention Task

  • The hint-task example changes from no acknowledgment before GRPO to explicit acknowledgment of the injected hint afterward.The vanilla trace answers (C) without acknowledging the suggested (D); the post-GRPO trace says the hint suggests (D) and selects (D).Source: Task 2: Hint Intervention Task · p. 15

Task 3: Integer Multiplication Task

  • In the multiplication example, the displayed partial products are correct, but their sum conflicts with the stated answer, which is also wrong.The partial products are 558 and 4340, but the displayed answer is 4858 rather than their sum, 4898.Source: Task 3: Integer Multiplication Task · p. 17

B INFERENCE SETTINGS OF MODELS

  • Evaluation uses nucleus sampling at temperature 0.7 and zero-shot, task-specific prompting; post-training completion sampling uses temperature 1.0 and top-p 1.0.Evaluation also uses top-p 0.95, top-k 50, and a 512-token maximum; post-training samples 16 completions per prompt.Source: B INFERENCE SETTINGS OF MODELS · p. 17

C.1 TWO-HOP FACTUAL REASONING

  • For two-hop reasoning, probes and a Tuned Lens detect bridge-entity representations at the subject token in the question and first CoT step.The probe is trained on single-hop questions; detections from the two positions are combined, and the internal entity is compared with the CoT entity.Source: §C.1 · pp. 17, 19

C.2 HINT INTERVENTIONS

  • For hint interventions, probes learn from changes in the hinted answer’s probability between biased and unbiased prompts, then detect influence from one forward pass.The probability-shift threshold is 0.1; the probe is trained on answer-letter hidden states and avoids a second unbiased inference at test time.Source: §C.2 · p. 20
  • Biasing Features provides a supplementary hint metric by marking influence when the answer changes to the hinted option, independently checking probe-based CIA improvements.Its BCoT label comes from the same LLM judge used for the main metric.Source: §C.2 · p. 20

C.3 INTEGER MULTIPLICATION

  • For multiplication, partial-product corruption labels whether the model tracks displayed intermediate values or relies on direct parametric recall.If the regenerated sum matches a corrupted target, the sample is labeled as following long multiplication; otherwise it is labeled as direct recall.Source: §C.3 · p. 22
  • The multiplication auxiliary metric classifies internal following through attention to partial products while writing the sum, using five selected heads and a fixed threshold.The selected heads and threshold are fixed from the base model’s training split and applied unchanged to post-trained models.Source: §C.3 · pp. 22–23

C.4 PROBING RESULTS AND RELIABILITY

The appendix reports probe reliability and cross-tool agreement, then details reward-based training objectives and their hyperparameter choices.

C.4 PROBING RESULTS AND RELIABILITY

  • Probes exceed the majority-class baseline when both classes are meaningfully represented, and positive-class F1 supports their capture of latent-strategy distinctions.The reported held-out checks cover task-specific labels across tasks and models.Source: §C.4 · p. 23

C.5 AGREEMENT BETWEEN PRIMARY AND AUXILIARY TOOLS

  • Auxiliary-tool agreement ranges from 65–78% for multiplication and 86–92% for TwoHopFact to 93–96% for MMLU-Hint, with important independence caveats.TwoHopFact agreement is mostly shared negatives because only 11–21% of samples are probe-positive; MMLU-Hint agreement is not independent because the probe trains on the Biasing Features behavioral label.Source: §C.5 · p. 24

D POST-TRAINING DETAILS

  • RL methods add a reward for agreement between BCoT and BINT to the base task reward.The shared form is r(y_i) = r_base(y_i) + λ · 1(BCoT(y_i) = BINT(y_i)).Source: D POST-TRAINING DETAILS · p. 24

D.1 TRAINING OBJECTIVES

  • DPO ranks sampled completions by augmented reward, pairs the three highest with the three lowest, and trains to favor the higher-ranked completions.Pairs with equal reward are dropped; the reward jointly reflects correctness and faithfulness.Source: §D.1 · p. 24
  • GRPO uses group-relative advantages from augmented rewards and does not require explicit preference-pair construction.Completions above the group mean receive positive advantage; those below receive negative advantage.Source: §D.1 · p. 24

D.2 HYPERPARAMETERS SELECTION

  • Training samples 16 completions per prompt, sets the faithfulness reward weight to 1.0, and uses different fixed optimizer settings for DPO and GRPO.A GRPO sweep is performed for two-digit multiplication; TwoHopFact GRPO checkpoints are selected by validation CIA, while other runs use the final model.Source: §D.2 · pp. 24–25

D.3 ABLATION STUDIES

The ablation finds that combining accuracy and faithfulness rewards yields the highest CIA across settings, while a larger model on multiplication is more accurate but less faithful.

D.3 ABLATION STUDIES

  • Acc + Faith reaches the highest CIA in all nine settings, while Acc only leaves CIA near or below the base model.The ablation evaluates DPO reward terms across tasks and models on the validation split.Source: §D.3 · p. 25
  • Faith only improves CIA but lowers accuracy on TwoHopFact and MMLU-Hint; on 2-Digit Multiplication, it improves both CIA and accuracy.For multiplication, the model learns to follow the long-multiplication procedure internally rather than recall the answer directly.Source: §D.3 · p. 25

E.1 SCALING TO LARGER MODELS

  • Qwen3-14B has higher accuracy than Qwen3-8B (0.805 vs. 0.763) but lower CIA (0.524 vs. 0.590) on 2-Digit Multiplication.Its answer follows a corrupted partial product less often (45.6% vs. 66.3%), and the (0, 1) cell grows from 23.6% to 40.3%. This is one additional scale point on one task, so the authors draw no general scaling conclusion.Source: §E.1 · p. 25, Table 8 · p. 25

E.2 EXTENDING TO REASONING MODELS

In reasoning models that produce long thinking traces, the study measures CIA on MMLU-Hint and tests post-training; trace-based CIA and hint reporting vary by model and evaluation target.

E.2 EXTENDING TO REASONING MODELS

  • The evaluation uses MMLU-Hint traces of up to 8,192 tokens and retrains each model's probe on thinking-mode generations after the non-thinking probe underperforms.The original probe has test macro-F1 0.775 on thinking-mode outputs versus 0.920 on non-thinking outputs; retrained probe macro-F1 is 0.894 for Qwen3-8B and 0.858 for DeepSeek-R1-Distill-Llama-8B.Source: §E.2 · pp. 25, 27
  • Thinking-mode Qwen3-8B has full-output CIA 0.352 vs. 0.586 without thinking, and follows the hint less often (0.103 vs. 0.178).The (0, 1) cell grows to 60.0%, lowering full-output CIA.Source: §E.2 · p. 27, Table 9 · p. 27
  • For Qwen3-8B, answer-only CIA is 0.517; about half of rows mention the hint only in the trace, which tends to over-report hint use.When the hint drives the answer, full-output acknowledgment is 90.9–93.3%, versus 24.4–40.9% for the final answer alone.Source: §E.2 · p. 27
  • DeepSeek-R1-Distill-Llama-8B has CIA 0.595 on full output and 0.623 on the answer, matching its non-thinking Llama reference on full output (0.595).About half of its outputs contain no answer letter, leaving 315–329 test prompts per seed.Source: §E.2 · p. 27, Table 9 · p. 27
  • For Qwen3-8B, RS-B and DPO-A significantly raise CIA on full output (+0.050 and +0.043) and on the final answer (+0.037 and +0.076).Table 10 reports test-split mean changes over three generation seeds, using prompts answered by both models.Source: §E.2 · p. 27, Table 10 · p. 27
  • For DeepSeek-R1-Distill-Llama-8B, only RS-B's full-output CIA gain (+0.078) is significant; it also increases answered prompts from 315–329 to 551–563 per seed.Paired changes use only prompts answered by both models.Source: Table 10 · p. 27, §E.2 · p. 28

Related papers

Loading 2609.38972v1…