PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps

Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasović, Yonatan Belinkov

arXiv:2502.14829v4cs.CL

TL;DR

The paper addresses whether verbalized CoT reasoning faithfully reflects models’ parametric beliefs, a question complicated by context-only perturbations. It introduces PFF and FUR, which erase reasoning-step information from parameters and measure prediction changes. Across four LMs and five multi-hop MCQA datasets, FUR produced valid prediction-changing interventions, while unlearning also changed the answers supported by later CoTs.

  • Problem

    CoT reasoning may not faithfully represent models’ internal parametric reasoning, and context perturbations can leave the underlying information recoverable from parameters.

  • Method

    PFF intervenes on model parameters, while FUR independently unlearns information from generated reasoning steps and measures resulting changes in predictions.

  • Results

    Across four LMs and five multi-hop MCQA datasets, FUR performed valid interventions that changed predictions while retaining general capabilities, and post-unlearning CoTs supported different answers.

  • Takeaways & Limitations

    Parametric faithfulness can identify reasoning information linked to model predictions, while plausible verbalized reasoning may still differ from reasoning steps identified as important.

  • Takeaways & Limitations

    The approach is limited to cases where CoT and no-CoT predictions agree, and its recall cannot be guaranteed because unlearning may be unsuccessful or alternative explanations may exist.

Abstract

from arXiv · show

When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction. Despite much work on CoT prompting, it is unclear if reasoning verbalized in a CoT is faithful to the models' parametric beliefs. We introduce a framework for measuring parametric faithfulness of generated reasoning, and propose Faithfulness by Unlearning Reasoning steps (FUR), an instance of this framework. FUR erases information contained in reasoning steps from model parameters, and measures faithfulness as the resulting effect on the model's prediction. Our experiments with four LMs and five multi-hop multi-choice question answering (MCQA) datasets show that FUR is frequently able to precisely change the underlying models' prediction for a given instance by unlearning key steps, indicating when a CoT is parametrically faithful. Further analysis shows that CoTs generated by models post-unlearning support different answers, hinting at a deeper effect of unlearning.

1 Introduction

CoT reasoning can improve language-model performance, but its faithfulness to internal parametric reasoning remains unclear. The paper introduces PFF and FUR to test faithfulness by unlearning reasoning information and measuring prediction changes.

  • CoT reasoning improves complex-task performance, but its correspondence to models’ internal reasoning remains uncertain.
  • Context perturbation can measure contextual faithfulness without removing the corresponding knowledge from model parameters.
  • PFF measures parametric faithfulness through parameter intervention followed by evaluation of whether the model’s prediction changes.
  • FUR unlearns each generated reasoning step independently and measures its effect on the model’s prediction.
  • FF-HARD evaluates whole-chain faithfulness, whereas FF-SOFT identifies reasoning steps most salient to the model’s prediction.
  • Experiments across four LMs and five multi-hop MCQA datasets produced valid prediction-changing interventions while retaining general capabilities.
  • Unlearning could alter the answer supported by later verbalized reasoning, and human evaluators did not judge FUR-identified important steps plausible.

2 Background and Related Work

Prior work often evaluates CoT faithfulness through contextual perturbations, but such tests can confound context sensitivity with parametric influence. The paper motivates parameter-level interventions and situates them within machine unlearning research.

  • CoT prompting can improve complex multi-hop and arithmetic reasoning, while chains may also provide intermediate results or additional computation.
  • CoTs can be unreliable, misaligned with answers, unhelpful to humans, or factually incorrect.
  • Contextual perturbations may change predictions because models are sensitive to context or can reconstruct erased information from latent parameters.
  • The proposed setup uses CoTs to guide unlearning but generates predictions directly, separating contextual influence from prediction.
  • Machine unlearning aims to remove targeted knowledge or behavior while retaining performance on non-target data and general capabilities.
  • NPO+KL is selected because it applies to unstructured text and outperforms alternatives.

3 PFF: A Framework for Measuring Parametric Faithfulness

PFF measures parametric faithfulness through a parameter intervention followed by evaluation of changes in answers or reasoning. Its design addresses the confound that context-only perturbations may leave the underlying information recoverable from parameters.

  • Parametric faithfulness concerns whether verbalized reasoning reflects information encoded in model parameters.
  • PFF has two stages: intervening on model parameters to erase CoT information, then evaluating whether the intervention changes predictions.
  • The framework generates reasoning, segments it into steps, intervenes on parameters targeted by each step, and produces a modified model.
  • Faithfulness can be evaluated by comparing direct answers or reason-then-answer outputs from the original and modified models.
  • Direct-answer evaluation avoids post-intervention reasoning-context changes that could otherwise confound answer differences.

4 FUR: Unlearning Reasoning Steps

FUR instantiates a parameter-level intervention that unlearns individual reasoning steps and measures faithfulness through changes in model predictions. Its controls balance removal of target information against retained capabilities, while FF-HARD and FF-SOFT quantify chain-level and step-level faithfulness under important recall limitations.

  • Framework: PFF evaluates parametric faithfulness by intervening on model parameters and checking whether the intervention changes the model’s prediction.The framework can use different interventions and apply to chains of thought or other free-text explanations.
  • Unlearning intervention: FUR uses KL-regularized NPO to unlearn reasoning-step information while preserving output similarity on a retain set.The forget set contains content-word prefixes and targets the model’s prediction of those content words.
  • Unlearning intervention: Each reasoning step is unlearned separately for five iterations by updating only the Transformer MLPs’ second FF2 matrix.FF2 is targeted because prior work identifies it as a memory store and model-editing methods frequently update it.
  • Controls: Unlearning success requires high efficacy and specificity: reducing the target step’s probability while retaining general capabilities, fluency, and held-out predictions.Specificity is measured as the proportion of unchanged labels on 20 unrelated in-domain instances.
  • Faithfulness measurements: FF-HARD marks a chain faithful when unlearning any step changes the most likely answer, whereas FF-SOFT scores how much probability mass shifts from the initial answer for each step.The evaluation prompts the original and unlearned models to answer directly without reasoning.
  • Limitations: FF-HARD is a lower bound on true faithfulness because alternative reasoning paths, failed interventions, and genuinely unfaithful explanations can all prevent prediction changes.The method does not seek efficacy of 1 because making a step impossible could harm fluency, creating a tension among efficacy, specificity, and general capabilities.

5 Experimental Setup

The experiments evaluate four instruction-tuned language models on five diverse multi-hop MCQA datasets using zero-shot, greedily generated chains of thought. Chains are segmented into sentences, and unlearning targets content-word tokens to remove step knowledge without removing the ability to verbalize reasoning.

  • Models: Experiments use four instruction-tuned models from three model families: LLaMA, Mistral, and Phi-3.The models include LLaMA-3-8B-Instruct, Llama-3.2-3B-Instruct, Mistral-7B-Instruct-v0.2, and Phi-3-mini-4k-Instruct.
  • Datasets: The evaluation covers five multi-hop datasets spanning science, sports, geography, health, law, and finance.The datasets are OpenbookQA, ARC-Challenge, StrategyQA, TruthfulQA, and BigBench-Hard Sports.
  • Datasets: Each dataset contributes a balanced subset of 250 test instances because unlearning every chain-of-thought step is computationally expensive.The subsets are selected from the test splits.
  • Procedure: Models first generate a chain of thought from the question and answer choices, then complete the answer letter using the chain of thought, with greedy decoding producing one chain per instance.The experiments are conducted zero-shot on multi-choice question-answering datasets.
  • Preprocessing: Chains of thought are segmented into sentences, and unlearning targets only content-word tokens to remove step knowledge while preserving the capability to verbalize reasoning.A reasoning step is treated as a sentence with at least two content words in the unlearning construction.

6 Results

The experiments show that FUR can change model predictions while largely preserving specificity and general capabilities, and that unlearning often alters the reasoning verbalized afterward. FUR also identifies influential steps, though faithfulness and human judgments of plausibility correlate weakly.

  • Does Unlearning Change Predictions?: Unlearning preserved specificity and general capabilities while achieving good efficacy on target reasoning steps.The evaluation measured efficacy, specificity, and MMLU performance before and after unlearning.
  • Does Unlearning Change Predictions?: FUR generally identified a larger proportion of faithful CoTs than contextual mistake-based faithfulness.Both methods changed predictions on reasonably high percentages of cases, but FUR usually reported more faithful CoTs.
  • Does Unlearning Change Predictions?: The correlation between average unlearning efficacy and FF-HARD was 0.889 (p < 0.0001).Stronger unlearning more often changed predictions, although stronger interventions could damage model integrity.
  • Does Unlearning Change Reasoning?: After a single unlearning iteration, probability mass shifted from the initial answer to two alternatives, and the CoT argued against the original prediction.The displayed example tracks no-CoT probabilities and model CoTs before and after unlearning.
  • Does Unlearning Change Reasoning?: Post-unlearning CoTs largely supported different answers from the base model, indicating that the intervention changed verbalized reasoning.An LLM-as-a-judge evaluated whether generated CoTs supported a different answer after unlearning.
  • Quantifying Step Level Faithfulness: FF-SOFT identified steps whose unlearning decreased or increased probability of the initial answer, while its correlation with human support ratings was only 0.15.Background and prediction-stating steps could reduce initial-answer probability, whereas another background step increased it.

7 Discussion and Future Outlook

The discussion addresses computational costs of applying FUR to long reasoning chains and explores parallelization and low-rank adaptation as ways to reduce them. LoRA can match or surpass full fine-tuning in the reported LLaMA experiments, with some evidence that targeted knowledge occupies a low-rank subspace.

  • Reducing time complexity: Long reasoning chains increase FUR’s time complexity because they contain many constituent steps.The paper specifically highlights reasoning models producing chains spanning thousands of tokens.
  • Reducing time complexity: FUR can process individual CoT steps in parallel, while verifiers could prioritize the steps most important for prediction.The proposed prioritization would reduce the number of steps requiring erasure.
  • Reducing space complexity: LoRA offers a potential space-efficient alternative to full fine-tuning for FUR.The study applied LoRA with ranks 8, 32, and 128 to LLaMA models on Sports and StrategyQA instances.
  • Reducing space complexity: LoRA-tuned models matched or surpassed full-fine-tuning FF-HARD scores on the reported Sports and StrategyQA comparisons.For LLaMA-3B, all tested variants changed predictions on the same instances, suggesting targeted knowledge may occupy a low-rank subspace.

8 Conclusion

The paper concludes that PFF and its FUR implementation measure CoT faithfulness by testing whether unlearning reasoning steps changes predictions. The results also show that unlearning can shift verbalized reasoning toward another answer, while highly relevant steps may not appear plausible to humans.

  • Conclusion: PFF measures parametric faithfulness, and FUR instantiates it by unlearning reasoning steps from model parameters.FF-HARD evaluates whole-CoT faithfulness, whereas FF-SOFT identifies steps most relevant to prediction.
  • Conclusion: Unlearning certain steps caused models to verbalize reasoning for a different answer, supporting their internal use in generating the prediction.The conclusion contrasts this parametric intervention with perturbation-based contextual faithfulness approaches.
  • Conclusion: Steps judged highly relevant by FUR were not considered plausible by humans, motivating specialized alignment for reasoning plausibility.The conclusion presents faithfulness and plausibility as distinct properties requiring separate consideration.

Limitations

The paper’s limitations concern methodological scope, imperfect unlearning, task coverage, and the interpretation of competing faithfulness measures. These constraints limit applicability and recall while motivating specific future extensions.

  • Design scope: The contextual confounder restriction limits analysis to cases where CoT and no-CoT predictions agree.Future work could test consistency under semantically equivalent surface changes after unlearning.
  • Unlearning limitations: Machine unlearning may localize or erase target information imprecisely, so FUR’s high precision does not guarantee recall.Unsuccessful unlearning, unfaithful explanations, and alternative explanations can all leave faithful steps undetected.
  • Access requirements: FUR requires access to model parameters for fine-tuning and is therefore not applicable to closed API-based models.The authors argue that credible parametric faithfulness requires parameter access and intervention.
  • Task scope: The experiments are limited to English-language MCQA tasks, although the metrics are intended to apply beyond MCQA.MCQA simplifies probability-shift analysis and avoids generating answer options.
  • Evaluation scope: Contextual faithfulness methods and parametric faithfulness measure different properties, so paraphrase or mistake-based results do not establish faithfulness.The paraphrase baseline identified 2.84% of instances as unfaithful on average, but the remaining instances were not thereby shown faithful.

C Dataset and Model Statistics

The study evaluates model predictions with and without CoT across selected datasets, using direct and two-step prompting and normalized answer-choice probabilities. Its unlearning setup follows NPO+KL with controlled retain data and learning-rate selection.

  • Dataset statistics: Dataset statistics include total and average CoT-step counts, measured over 250 instances per dataset, or 248 for Sports.These counts characterize the reasoning material used in the experiments.
  • Prediction procedure: Answer probabilities are obtained by normalizing probabilities assigned to the first letters of the answer choices.The prompt prefix “My answer is (” constrains completion to the answer options.
  • Prompting setup: Models are evaluated both without CoT and with a generated CoT followed by answer prediction.The two-step setup generates reasoning first, then appends it to the question and answer options.
  • Unlearning configuration: NPO+KL retains the original paper’s best hyperparameters except for explicitly changed values and uses five unlearning epochs for a single sentence.Warmup is adjusted because each epoch is one unlearning step and the first scheduled learning rate is zero.
  • Forget and retain data: The forget and retain sets are built from CoT steps, with target-token filtering and four randomly selected unrelated retain steps from the same dataset.Filtering keeps steps containing more than two relevant target tokens.

F LoRA Setup & Hyperparameters

The LoRA analysis tests whether low-rank unlearning reproduces the reasoning effects observed with full fine-tuning. It uses two LLaMA models, two difficult datasets, and a judge-based comparison of pre- and post-unlearning CoTs.

  • LoRA setup: LoRA feasibility is evaluated on LLaMA-3-3B and LLaMA-3-8B using Sports understanding and StrategyQA.These datasets were selected because they were most difficult for the models.
  • Hyperparameters: The best LoRA learning rates are 3e−04 for LLaMA-3-8B and 1e−03 for LLaMA-3-3B.These rates are approximately 100-fold higher than the best full-fine-tuning values.
  • Faithfulness comparison: LoRA-tuned models obtain FF-HARD scores comparable to those of full fine-tuning on the evaluated sample.The comparison is based on a 30-instance sample used for learning-rate selection.
  • Post-unlearning reasoning: GPT-4o largely judges that LoRA-unlearned models’ CoTs support different answer options after unlearning.This indicates a substantial effect of low-rank adaptation on verbalized reasoning.
  • Scope of the method: Full fine-tuning remains necessary when the goal is to fully purge unwanted information from the model.The authors nevertheless describe LoRA as a viable alternative to full fine-tuning for the evaluated reasoning effects.

I Additional Insights

Additional analyses link efficacy with faithfulness, examine step-level prediction flips and human plausibility, and report the computational cost of evaluation. They also identify practical risks in interpreting or deploying FUR.

  • Efficacy and faithfulness: 0.889 is the reported average Pearson correlation between efficacy and FF-HARD, with p < 0.0001.The correlation is visualized across learning-rate runs and examined across individual datasets and models.
  • Efficacy and faithfulness: The correlation suggests that unlearning targets information relevant to prediction and that efficacy can serve as a faithfulness proxy.The authors interpret the relationship as evidence that successful erasure affects prediction-relevant information.
  • Step-level analysis: Step-level FF-HARD is lower than whole-CoT faithfulness, indicating that only some reasoning steps are sufficiently influential to change predictions when unlearned.The step-level metric counts prediction-changing unlearning events on instances where CoT and no-CoT predictions agree.
  • Human plausibility: The human study bins steps by FF-SOFT probability-mass shifts and annotates 100 sampled instances with 15 participants.Samples include 15 high-bin, 5 moderate-bin, and 5 negative-bin examples per dataset and model.
  • Runtime: Full evaluated runs average 17h40m35s, while unlearning alone averages 2h26m51s, a 7× speed-up.The difference shows that post-unlearning inference and evaluation dominate total runtime.
  • Risks: Faithful model explanations are usually not plausible to humans and may be unsafe as reasoning guidepoints in high-stakes settings.The method also could be used adversarially to erase faithful reasoning and make models appear less competent or biased.
Loading 2502.14829v4…