Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning

1 Introduction

• The study addresses whether models reliably use inference chains, a gap left by work focused on task performance or CoT faithfulness without examining their sequential relationship (Huang and Chang, 2023; Lanham et al., 2023).Standard CoT explanations need not guarantee that the final answer follows the reasoning chain (Lyu et al., 2023).Source: §1 · p. 1
• The authors use causal mediation analysis to interpret how reasoning chains contribute to final answers, intervening on model inputs and reasoning chains.They describe the reasoning chain as a mediator between input and output.Source: §1 · p. 2
• Across twelve LLMs, causal effects of reasoning traces varied substantially by task and model, and LLMs generally did not reliably use intermediate reasoning steps to answer.GPT-4 changed its answer only 30% of the time when conditioned on perturbed counterfactual reasoning chains.Source: §1 · p. 2
• FRODO trains small LMs in two modules: one generates correct reasoning chains, and the other reasons over those chains to reach the correct answer.The inference module uses implicit feedback from LLM-generated correct and counterfactual chains; the reasoning module uses a counterfactual and causal preference-ranking objective.Source: §1 · p. 2
• FRODO improves robustness by 4.5% and out-of-distribution performance by 2.6% over supervised fine-tuning.Across four reasoning tasks, it also achieves an absolute accuracy improvement of 2%–3% over standard supervised fine-tuning or CoT distillation methods.Source: §1 · p. 2

2 Reasoning Chain as a Mediator

• The causal graph distinguishes the original problem and chain (X0, R0) from an intervened problem and its chain (X1, R1), then compares outputs under these pairings.This setup represents the chain as a mediator in the relationship between input and output.Source: Figure 2 · p. 3, §2 · p. 3
• The analysis models the reasoning problem as input X, reasoning steps R, and output Y, with R mediating the relationship between X and Y.For the original problem X0, R0 is its reasoning chain; Y00 and Y01 are outputs for X0 paired with R0 or the chain R1 from intervened problem X1.Source: §2 · pp. 2–3
• Direct effect compares outcomes under X0 and X1 while holding the original reasoning chain R0 fixed; indirect effect compares Y00 and Y01 while keeping the input at X0.A high direct effect indicates greater emphasis on the problem input, while a high indirect effect indicates greater emphasis on reasoning steps.Source: §2 · p. 3
• The intervention procedure uses GPT-4 to generate alternative problem inputs, manually filters invalid or low-quality examples, and generates reasoning chains for each problem.The reasoning chains are elicited with task descriptions and a step-by-step prompt.Source: §2 · p. 3
• Vanilla LMs below 20B parameters in zero-shot settings were systematically unfaithful, while instruction-tuned or CoT-pretrained models had better indirect effects; some tasks showed inverse scaling.For certain tasks, indirect effect worsened as models became more capable, suggesting smaller models might be more effective in faithful reasoning.Source: §2 · pp. 3–4

3 FRODO

• FRODO targets models under 10B parameters with separate inference and reasoning modules for generating rationales and reasoning over them.The framework aims to improve the synergy between a reasoning chain and the final answer.Source: §3 · p. 4

3.1 Inference Module

• Without gold rationales, the inference module uses GPT-3 silver rationales and DPO to prefer correct chains over counterfactual or irrelevant ones.Preference pairs are generated for each problem; DPO uses supervised fine-tuning followed by preference learning.Source: §3.1 · p. 4
• The inference module's implicit-reward preference loss increases the likelihood of preferred reasoning chains and decreases that of counterfactual chains.During inference, the reasoning module uses the steps generated by the inference model.Source: §3.1 · p. 4

3.2 Reasoning Module

• The reasoner combines correct-answer language-model loss, counterfactual loss, and a preference-ranking loss based on indirect effect.The counterfactual objective is intended to help avoid shortcuts by requiring different answers for the same question under different reasoning chains.Source: §3.2 · p. 5
• Margin ranking separates question–correct-reasoning–correct-answer examples from examples using counterfactual reasoning and the correct answer.The objective maximizes the margin between these positive and negative examples.Source: §3.2 · p. 5

Datasets

• Causal mediation analysis uses StrategyQA, GSM8K, and Causal Understanding, while FRODO is evaluated on StrategyQA, Quarel, OpenBookQA, and QASC.The experiments use GPT-3-generated silver rationales rather than human-written rationales.Source: §4 · p. 5

Evaluation Metrics

• Causal-effect evaluation reports average indirect and direct effects, while rationale faithfulness is assessed with LAS using a fine-tuned T5-large simulator.For robustness, the experiments use controlled indirect effect; natural indirect effect is used for models over 100B parameters and controlled indirect effect for models under 20B.Source: §4 · pp. 5–6

Causal Effects

• Table 2 reports Natural Indirect Effect, Natural Direct Effect, and CoT accuracy percentages.The caption identifies the measures but does not provide their values.Source: Table 2 · p. 6

Implementation Details

• GPT-4 generates intervened reasoning problems and reasoning chains for the causal mediation analysis.Prompts and hyperparameters are reported in the appendix.Source: §4 · p. 6

Baselines

• The comparison includes SFT + CoT, Rainier, Crystal, Mario, and SCOTT as baselines.The passage describes different training approaches for each baseline, including PPO, QUARK, and contrastive decoding.Source: §4 · p. 6

Natural Direct and Indirect effects

• Across tasks, GPT-4 has higher natural indirect effects than ChatGPT on StrategyQA and causal understanding, whereas ChatGPT has a higher effect on GSM8K.On GSM8K, GPT-4 may reject intervened incorrect steps and continue solving, lowering its indirect-effect score.Source: §5 · p. 6

Controlled Direct and Indirect effects

• Vanilla LMs under 20B are systematically unfaithful in zero-shot controlled tests, while larger size, instruction tuning, and in-context learning improve indirect effects.RLHF-trained models show stronger direct than indirect effects; the authors also report inverse scaling on some tasks and weak effects on causal understanding.Source: §5 · pp. 6–7, Table 3 · p. 7

Comparing FRODO with Baselines

• FRODO improves average accuracy by +4.1 points over SFT + CoT and +3 points over MARIO across four reasoning tasks.The comparison uses small-sized LMs and reports accuracy; the authors attribute gains to both FRODO modules.Source: Table 4 · p. 7, §5 · p. 7

Ablation

• Removing DPO, counterfactual examples, or margin-ranking loss lowers FRODO performance by 1.9%, 3.1%, and 2.8%, respectively.The authors interpret these drops as evidence that implicit feedback and counterfactual examples contribute to performance.Source: §5 · p. 8

6.1 Quantitative Analysis

• FRODO improves robustness by +7.7 pp. on StrategyQA for T5-3B and by +2 pp. over SFT + CoT on QuaRel.The robustness measure is the controlled indirect effect under counterfactual reasoning chains; counterfactual loss contributes most in the ablation.Source: §6.1 · p. 8
• Training on OBQA and QASC and testing on unseen StrategyQA shows that FRODO improves out-of-distribution generalizability over SFT+CoT.The passage reports the direction of the result but no numeric gain.Source: §6.1 · p. 8

6.2 Qualitative Analysis

• For GSM8K, the indirect effect drops as reasoning-step count rises for both LLaMa-3 and ChatGPT, indicating an inverse relationship between chain length and faithfulness.Table 6 reports indirect effect by number of reasoning steps.Source: Table 6 · p. 8, §6.2 · p. 8
• In manually reviewed CoTs, DPO yields higher relevance than SFT on QuaRel and OBQA, but its errors include invalid and unnecessary reasoning steps.DPO CoTs are 77% and 59% relevant versus SFT's 74% and 54%, respectively; DPO errors are 40% invalid and 56% unnecessary.Source: §6.2 · p. 8

Measuring Faithfulness CoT

• Prior faithfulness work tests explanation alignment, adversarial unfaithfulness, ignored CoT errors, or input alignment; this work uses causal mediation to relate CoT to answers.Examples include Turpin et al. (2023), Lanham et al. (2023), and Parcalabescu and Frank (2024).Source: §7 · pp. 8–9

Self-Rationalization and CoT Distillation

• Unlike CoT distillation methods, this work uses implicit feedback to improve the correctness of distilled reasoning chains.The cited distillation work includes Fu et al. (2023), Li et al. (2022), Shridhar et al. (2023), and Li et al. (2023).Source: §7 · p. 9

Casual Mediation Analysis in NLP

• The paper applies causal mediation to LLM reasoning faithfulness, measuring direct input effects and indirect reasoning-chain effects on outputs.The passage identifies Pearl (2001) as the causal mediation basis and distinguishes this use from prior NLP applications.Source: §7 · p. 9

• Causal mediation analysis across twelve LLMs finds large task- and model-level variation in how strongly CoT affects predictions, with intermediate reasoning steps not reliably used.FRODO is reported to outperform strong baselines on four reasoning tasks, including out-of-distribution settings.Source: §8 · p. 9

9 Limitations

• The causal-analysis metric does not evaluate a model’s real internal reasoning process.The authors note that without fully understanding the training data and model internals, it is difficult to determine whether a chain of thought is faithful to that process.Source: §9 · p. 10
• FRODO requires additional LLM-generated counterfactual data and more training time because DPO training is a two-step process.The authors contrast this cost with standard CoT distillation.Source: §9 · p. 10
• Experiments used three random seeds, and FRODO depends on LLM-generated rationales, warranting extra care in critical production settings.The authors cite the scale of their experiments and hyperparameter searches as context for using three seeds.Source: §9 · p. 10

A.1 FRODO - Inference Module (DPO)

• DPO models preferences between reasoning chains with an implicit reward function parameterized by the log-likelihood ratio of πθ and πsft.The preference data is modeled under Bradley–Terry theory, and DPO uses a binary classification loss.Source: §A.1 · p. 16

A.2 Additional Experimental Results

• +3.75% average accuracy: FRODO outperforms SFT on GSM8K.Table 8 reports performance on math reasoning problems.Source: §A.2 · p. 16, Table 8 · p. 16
• FRODO generalizes better than SFT when trained on StrategyQA and evaluated on the Causal Understanding dataset, which has no training data.Table 9 reports accuracy for source-to-target evaluation.Source: §A.2 · p. 16, Table 9 · p. 16
• GPT-4 generated correct CoTs for 94% of StrategyQA examples and 65% of Causal Understanding examples.The authors connect this difference to GPT-4’s lower performance on Causal Understanding.Source: §A.2 · p. 16, Table 10 · p. 16
• Unlike Selection-Inference, FRODO does not assume that each question includes context containing all information needed to solve it, so it applies to open-domain QA.FRODO’s inference module uses counterfactual, language-model, and margin-ranking losses, while Selection-Inference uses supervised language-modeling loss.Source: §A.2 · p. 16
• FRODO clearly outperforms previous baselines on EntailmentBank.Table 11 reports performance on this dataset, but the supplied passage gives no numerical values.Source: §A.2 · p. 16, Table 11 · p. 17
• In zero-shot settings, LLaMA-2 and LLaMA-3 70B models did not generate coherent, meaningful reasoning steps, limiting the reported natural indirect effect analysis.The authors refer to Table 12 for natural indirect effect results and note prior work discussing emergent abilities above 100B parameters.Source: §A.2 · p. 17, Table 12 · p. 17

A.3 Dataset and Implementation Details

• Tasks use multiple-choice questions and require generating a rationale followed by a predicted answer; experiments use the original data splits.The listed answer formats include yes/no for StrategyQA, a/b for Quarel, a/b/c/d for OpenBookQA, and a/b/-/h for QASC.Source: §A.3 · p. 17