Source-linked AI summary
Measuring Faithfulness in Chain-of-Thought Reasoning
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, Ethan Perez
TL;DR
The paper investigates whether LLMs’ stated chain-of-thought reasoning faithfully reflects the process behind their answers. It tests this by intervening on CoT and measuring answer changes across tasks and model sizes. Results show task-dependent faithfulness, evidence against test-time compute or phrasing-encoded information as the main drivers of CoT improvement, and lower faithfulness in larger models.
Problem
It is unclear whether LLM-generated CoT faithfully represents the reasoning process behind the model’s prediction, despite claims that CoT improves interpretability.
Method
The paper perturbs CoT through interventions such as truncation, mistakes, and paraphrasing, then evaluates how model answers change across tasks and model sizes.
Results
Faithfulness varies greatly across tasks; CoT improvements do not appear to result from extra test-time computation or particular phrasing, and larger models are often less faithful.
Takeaways & Limitations
CoT can be faithful under selected conditions, making model size and task useful levers for obtaining more faithful reasoning.
Takeaways & Limitations
The study lacks an independent way to observe the model’s real internal reasoning, and its tested hypotheses may not be exhaustive.
Abstract
from arXiv · showhide
Large language models (LLMs) perform better when they produce step-by-step, "Chain-of-Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT (e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT's performance boost does not seem to come from CoT's added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study. Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen.
1. Introduction
The paper asks whether chain-of-thought reasoning faithfully represents an LLM’s reasoning and proposes intervention-based tests to investigate this question. Results vary across tasks, while evidence argues against extra test-time computation and particular CoT phrasing as the main sources of performance gains.
- The paper examines whether stated CoT reasoning accurately represents the reasoning behind a model’s prediction.
- The proposed tests intervene on CoT by truncating it, adding mistakes, paraphrasing it, or replacing it with filler tokens.
- CoT use varies substantially across tasks, with models sometimes largely ignoring the reasoning and sometimes relying heavily on it.
- Replacing CoT with uninformative filler text produces no accuracy gain, suggesting extra test-time computation alone is not responsible for the improvement.
- Similar performance after paraphrasing CoT indicates that its particular phrasing is not a driver of performance.
- Smaller models often generate more faithful reasoning than larger, more capable models, and faithfulness can depend on the task.
- CoT is not always faithful, but conditions may exist where it is more faithful, supporting future work on faithful reasoning and detecting untrustworthy reasoning.
2. Measuring Chain of Thought Faithfulness
The paper measures CoT faithfulness by perturbing reasoning samples and observing changes in model behavior across multiple-choice tasks. In the standard CoT condition, seven of eight tasks improve, while HellaSwag is the exception.
- 2. Measuring Chain of Thought Faithfulness: The experiments perturb CoT and observe the model’s behavior to test hypotheses about unfaithful reasoning.
- 2.1. Methods: Most experiments use a 175B-parameter pretrained decoder-only transformer fine-tuned as a helpful dialog assistant with RLHF.
- 2.1. Methods: Eight multiple-choice tasks are selected because they are expected to benefit from explicit reasoning.
- 2.1. Methods: The task set includes science, algebra, text completion, logical reasoning, multitask understanding, and factual-question benchmarks.
- 2.1. Methods: For each problem, the researchers sample 100 reasoning paths with nucleus sampling, append a final-answer prompt, and analyze answer-choice probabilities.
- 2.2. Chain of Thought Statistics: Collected reasoning samples average 4 sentences, with 89% containing between three and six sentences.
- 2.2. Chain of Thought Statistics: Seven of eight tasks improve with CoT; AQuA improves the most, while HellaSwag alone shows degraded performance.
2.3. Early Answering: Does Truncating the Chain of Thought Change the Predicted Answer?
The paper tests whether CoT is post-hoc by truncating reasoning and measuring whether the predicted answer changes. Faithfulness varies widely across tasks and shows little correlation with CoT’s performance gain.
- Method: Truncating CoT midway tests whether later reasoning remains causally relevant to the model’s predicted answer.Each collected reasoning sample is truncated after successive sentences, and the model answers using each partial chain.
- Metric: AOC summarizes matching behavior across CoT lengths, weighted by the fraction of samples at each length.Higher AOC indicates less post-hoc reasoning and therefore greater faithfulness.
- Results: Less than 10% of final answers change for ARC (Easy), ARC (Challenge), and OpenbookQA, compared with more than 60% for AQuA.AQuA also rarely matches its final answer before the last two reasoning steps, suggesting little post-hoc reasoning.
- Interpretation: Post-hoc reasoning shows little correlation with CoT’s performance improvement across tasks.LogiQA gains negligibly from CoT despite ranking near the top in AOC, while HellaSwag loses 4.69% accuracy despite showing less post-hoc reasoning than several improving tasks.
- Results: Post-hoc reasoning varies considerably across tasks, with AQuA and LogiQA showing the most faithful reasoning by some margin.The two experiments broadly agree on task rankings, although HellaSwag and MMLU switch order.
2.5. Filler Tokens: Do Uninformative Chain of Thought Tokens Also Improve Performance?
The paper tests whether CoT’s performance boost comes from extra test-time computation rather than stated reasoning. Replacing CoT with uninformative filler tokens produces no accuracy increase, suggesting that extra computation alone is insufficient.
- Method: The filler-token experiment replaces CoT with sequences of uninformative “ ...” tokens of varying lengths.Lengths range from zero to the longest sampled CoT for each question, in five-token increments.
- Results: Adding filler tokens produces no accuracy increase, so extra test-time computation alone does not explain CoT’s performance boost.For TruthfulQA and OpenBookQA, accuracy slightly decreases with longer filler contexts, possibly because such sequences are outside the training distribution.
- Interpretation: The filler-token results suggest that models do not use extra context alone for helpful but unstated reasoning.The experiment targets the hypothesis that longer context windows provide performance-improving computation absent from the stated CoT.
- Paraphrasing: Paraphrased CoT closely matches original-CoT accuracy across almost all tasks and filler-token lengths.This indicates that particular phrasing is unlikely to encode the information responsible for CoT’s accuracy gains.
3. Does Model Size Affect CoT Faithfulness?
Faithfulness varies with both model size and task difficulty. Smaller models are often more faithful, while easier addition tasks show more post-hoc reasoning.
- The answer-change metric captures how much the model relies on CoT when predicting answers.It is highly predictive of results from early-answering and mistake-insertion experiments.
- For 7/8 tasks, the 13B model changes its answer after CoT more often than the 175B model.For six of eight tasks, the 13B model provides the most faithful reasoning.
- Smaller models may be less capable of using CoT, and models below 13B obtain little to no accuracy gain from CoT on these tasks.The results suggest faithful CoT may arise at an intermediate capability level rather than from the most capable model.
- The synthetic addition evaluation directly controls task difficulty using problems with 2, 4, 8, or 16 operands.Answers are free responses rather than multiple-choice selections.
- On synthetic addition tasks, post-hoc reasoning increases with model size and with easier tasks at the same model size.Tasks vary by operand count and operand length, while the smallest models are excluded because they do not consistently produce integer answers.
4. Related Work
Related work has measured, elicited, or constructed language-model reasoning in several ways. This paper instead tests faithfulness by perturbing CoT in non-adversarial settings across realistic tasks.
- Prior work uses Shapley analysis to identify CoT tokens that are especially important for the final answer.
- Earlier studies examined CoT under biasing few-shot examples, whereas this work studies non-adversarial settings across a wider range of realistic conditions.
- Prior work proposed explanations for CoT’s performance boost involving equation evaluation, additional test-time compute, and access to pretrained information.
- Some methods aim to increase faithfulness by generating programs and executing them to produce final answers, preventing post-hoc generation.
- Other approaches elicit reasoning through subquestions, thought trees, executable plans, or model debate.
5. Limitations
The investigation cannot directly observe models’ real internal reasoning, and its tested hypotheses may not exhaust possible explanations. Its findings are also limited to RLHF-finetuned models.
- The study lacks an independent way to understand the model’s real internal reasoning process, so it cannot directly establish CoT faithfulness.
- The tested hypotheses may be incomplete, and without ground-truth faithfulness it is unclear how to weight the experiments’ evidence.Additional experiments and combined measurement techniques are needed for a fuller assessment.
- The analysis covers RLHF-finetuned models, whose reasoning faithfulness may differ from pretrained LLMs.The paper identifies alternative training schemes as a direction for future work.
6. Conclusion
This work evaluates CoT faithfulness by perturbing reasoning samples across tasks and model sizes. It finds task-dependent post-hoc reasoning, evidence against test-time compute or phrasing-encoded information as drivers of CoT improvement, and inverse scaling of faithfulness in many cases.
- The experiments show large task-to-task variation in post-hoc reasoning and provide evidence against increased test-time compute or phrasing-encoded information driving CoT improvement.
- Reasoning faithfulness often worsens as models become more capable, suggesting smaller models may be preferable when faithful reasoning matters.
- The proposed metrics may support future efforts to increase CoT faithfulness and develop reasoning that is trustworthy and verifiable.
A. Adding Mistakes & Paraphrasing Prompts
The paper provides prompts for the adding-mistakes and paraphrasing experiments, integrating their generated outputs into the original chain of thought.
- The adding-mistakes experiment uses prompts presented in Table 6 and §2.4.The mistakes are generated using a pretrained base model without RLHF fine-tuning.
- The paraphrasing experiment uses prompts presented in Table 7 and §2.6.
- Generated mistakes and paraphrases are inserted into the original chain of thought for evaluation.
B. Additional Paraphrasing Results
Paraphrasing progressively more of the reasoning usually preserves agreement with the original answer, providing little support for encoded reasoning as an important accuracy source.
- Most tasks show a smooth upward curve in answer agreement as progressively larger portions of reasoning are paraphrased.
- AQuA and LogiQA are slight exceptions for six-step samples, with flatter agreement during earlier paraphrasing steps.
- The results indicate that encoded reasoning is not an important source of accuracy improvement in these experiments.
C. CoT Accuracy Gain Across Model Size
Accuracy with and without CoT is compared across the tasks and models used in §3.1, including how CoT’s improvement varies with model size. For four of eight tasks, the model size with greatest accuracy improvement also has greatest faithfulness.
- Figures 10 and 11 compare accuracy with and without CoT and the resulting accuracy improvement across tasks and models.
- For four of eight tasks, the model size with greatest accuracy improvement is also the model size with greatest faithfulness.The tasks are ARC (Easy), ARC (Challenge), HellaSwag, and AQuA.
- The comparison uses the tasks and models from §3.1.