Source-linked AI summary
SCOTT: Self-Consistent Chain-of-Thought Distillation
Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, Xiang Ren
TL;DR
SCOTT addresses the lack of faithful, self-consistent rationales in chain-of-thought models by distilling a large teacher into a smaller student. Contrastive decoding produces rationale supervision tied to gold answers, while counterfactual reasoning trains the student to respect its rationales. The resulting model is more faithful than baselines while preserving comparable task performance.
Problem
Chain-of-thought rationales are not guaranteed to be consistent with predictions or to faithfully justify decisions, limiting their usefulness for interpreting model behavior.
Method
SCOTT uses contrastive decoding to obtain consistent teacher rationales and counterfactual reasoning to train a small student that answers according to its generated rationale.
Results
SCOTT produces more faithful rationales and comparable end-task performance to baselines.
Takeaways & Limitations
A student that respects its rationales more can be improved more effectively by refining those rationales.
Takeaways & Limitations
Compared with standard distillation, SCOTT requires extra computation for contrastive decoding and additional counterfactual training data.
Abstract
from arXiv · showhide
Large language models (LMs) beyond a certain scale, demonstrate the emergent capability of generating free-text rationales for their predictions via chain-of-thought (CoT) prompting. While CoT can yield dramatically improved performance, such gains are only observed for sufficiently large LMs. Even more concerning, there is little guarantee that the generated rationales are consistent with LM's predictions or faithfully justify the decisions. In this work, we propose a faithful knowledge distillation method to learn a small, self-consistent CoT model from a teacher model that is orders of magnitude larger. To form better supervision, we elicit rationales supporting the gold answers from a large LM (teacher) by contrastive decoding, which encourages the teacher to generate tokens that become more plausible only when the answer is considered. To ensure faithful distillation, we use the teacher-generated rationales to learn a student LM with a counterfactual reasoning objective, which prevents the student from ignoring the rationales to make inconsistent predictions. Experiments show that, while yielding comparable end-task performance, our method can generate CoT rationales that are more faithful than baselines do. Further analysis suggests that such a model respects the rationales more when making decisions; thus, we can improve its performance more by refining its rationales.
1 Introduction
SCOTT addresses unfaithful chain-of-thought rationales by distilling a large teacher into a small student that generates rationales consistent with its predictions. It uses contrastive decoding for teacher supervision and counterfactual reasoning to make the student respect rationales, achieving comparable performance with greater faithfulness.
- 1 Introduction: SCOTT distills a small self-consistent CoT model from a large teacher to generate rationales consistent with its own predictions.The method targets rationales that faithfully justify answers rather than merely producing plausible text.
- 1 Introduction: Contrastive decoding encourages the teacher to generate tokens that become more plausible when the gold answer is considered, producing more on-topic rationales.The approach is designed to address teacher hallucination and improve rationale–answer consistency.
- 1 Introduction: Counterfactual reasoning trains the student to predict differently when rationales support different answers, preventing shortcuts from question-answer correlations.This links rationale generation to answer prediction rather than treating them as independent processes.
- 1 Introduction: Across the evaluated tasks, the method improves faithfulness while maintaining performance comparable to baseline methods.The experiments also report that larger students can be more inconsistent, but SCOTT robustly remedies this across student sizes.
- 1 Introduction: A more faithful student can be improved more effectively by correcting its rationales, demonstrating the method’s utility for model refinement.The analysis connects rationale faithfulness with the impact of rationale refinement on student behavior.
2 Chain-of-Thought Distillation
The distillation framework automatically obtains rationale annotations from a prompted teacher and trains a smaller student to self-rationalize. The student generates a rationale before answering, using teacher-produced training sequences as supervision.
- 2 Chain-of-Thought Distillation: The overall goal is to obtain teacher rationales that justify gold answers and train a student that answers according to its generated rationale.The task setting is open-domain question answering with an accompanying free-text rationale.
- 2 Chain-of-Thought Distillation: The framework prompts a frozen large LM with demonstrations containing questions, gold answers, and human rationales to generate rationale annotations automatically.Teacher-generated annotations replace human annotation for each new question-answer pair.
- 2 Chain-of-Thought Distillation: The student follows a self-rationalization paradigm by generating a rationale first and then predicting the answer conditioned on that rationale.This differs from post-rationalization, where the rationale is generated after the answer.
- 2 Chain-of-Thought Distillation: Student training concatenates rationale and answer tokens into one output sequence and fine-tunes a text-to-text LM with language-modeling loss.This baseline objective is referred to as factual reasoning loss.
3 Distilling a Self-Consistent Student
SCOTT addresses two faithfulness problems in knowledge distillation: teacher rationales may not support the answer, and students may ignore rationales when predicting. It uses contrastive decoding for grounded teacher rationales and counterfactual training to make student predictions depend on them.
- 3.1 A Consistent Teacher: Contrastive Decoding: The teacher’s rationale-generation and answer-prediction processes can be inconsistent because hallucinated text may not support the answer.This inconsistency can be inherited by the student during distillation.
- 3.1 A Consistent Teacher: Contrastive Decoding: Contrastive decoding prefers rationale tokens whose plausibility increases when the gold answer is considered, producing more answer-grounded teacher rationales.The method compares gold-answer conditioning with empty or incorrect-answer conditioning and aggregates the resulting plausibility growth with language-model scores.
- 3.1 A Consistent Teacher: Contrastive Decoding: Empty-answer or incorrect-answer perturbations provide alternative contrastive references for penalizing generally plausible or insufficiently answer-distinctive rationale tokens.Empty strings target hallucination without the gold answer, while incorrect answers encourage rationales that distinguish gold from wrong answers.
- 3.2 A Faithful Student: Counterfactual Reasoning: Counterfactual reasoning trains the student to answer differently when a rationale is generated for a different answer, discouraging question-answer shortcuts.The teacher is prompted with a randomly selected wrong answer to generate a counterfactual rationale, and the student is trained on the resulting answer.
- 3.2 A Faithful Student: Counterfactual Reasoning: The factual and counterfactual objectives are marked with separate keywords, and their losses are combined during student training.The implementation uses [Factual] and [Counterfactual] indicators and sums the corresponding objectives.
4 Experiments
Across four knowledge-intensive QA datasets, contrastive decoding produced more consistent teacher rationales, and counterfactual training further improved student faithfulness while preserving comparable task performance.
- 4.5 Main Results: Contrastive decoding produced more consistent teacher rationales than human annotation and greedy decoding across four datasets, with wrong-answer contrast generally outperforming empty-string contrast.LAS measured how well rationales supported gold-answer prediction.
- 4.5 Main Results: Students trained on contrastively decoded rationales generally achieved higher faithfulness than baseline students, supporting the link between teacher consistency and student faithfulness.The comparison used LAS on the experimented datasets.
- 4.5 Main Results: Adding counterfactual reasoning loss further increased student faithfulness compared with factual training alone.The objective addresses students treating rationale generation and answer prediction as independent processes.
- 4.5 Main Results: Knowledge-distillation methods achieved comparable accuracy while improving faithfulness over CoT baselines, whereas human-rationale students were slightly more accurate but often ignored their rationales.The reported accuracy advantage for human annotation was associated with less consistent rationales and weaker rationale dependence.
- 4.6 Ablation on the student model size: Larger student models achieved higher performance but lower faithfulness, while the proposed models remained more faithful than baselines across model sizes.This reveals a capacity trade-off between task performance and reliance on rationales.
- 4.7 Controlling the behavior of the Student: Students trained with contrastively decoded rationales were more sensitive to rationale perturbations, and counterfactual training further increased this sensitivity.Rationale sensitivity was evaluated on CSQA and CREAK through performance changes after perturbation.
- 4.7 Controlling the behavior of the Student: Refining rationales yielded larger performance gains for contrastive-decoding students, especially with counterfactual training, than for students trained on human annotations.The refinement analysis used oracle rationales on CSQA and CREAK.
5 Related Works
Prior work uses human or prompted-LM rationales to support language-model predictions, but these approaches face cost, quality, scale, and faithfulness limitations.
- Human-annotated rationales are expensive and have been reported to be poor quality, motivating automatically generated supervision.
- Prompted self-rationalization relies on extremely large LMs and substantial computation or API costs, while generated rationales can contradict their context.
- Existing distillation methods train students to predict answers, extract answer sentences, decompose questions, or jointly generate answers and rationales.
- Joint rationale-and-answer training can regularize the student without allowing rationales to affect its predictions during inference.
6 Conclusion
The paper presents a faithful knowledge-distillation framework that combines contrastive decoding for consistent teacher rationales with counterfactual reasoning for faithful students. Experiments show improved rationale faithfulness relative to baselines while preserving much performance accuracy, and rationale refinement can help debug the model.
- Contrastive decoding and counterfactual reasoning jointly produce a more faithful student than baselines while preserving much performance accuracy.
- Changing the rationales has a larger impact on the student’s behavior, enabling more successful model debugging through rationale refinement.
- The framework learns a small, self-consistent CoT model from a large teacher model.
Limitations
Compared with standard knowledge distillation, the method requires additional computation for contrastive decoding and additional training data for counterfactual reasoning. The work prioritizes rationale faithfulness rather than performance improvement.
- The method requires extra computation during training-data preparation and student training compared with standard knowledge distillation.Contrastive decoding performs one additional teacher forward pass per generated token, and counterfactual reasoning adds training data.
- The work focuses on improving rationale faithfulness rather than performance, complementing prior rationale-based work focused on performance.
Ethics Statement
Using large LMs for rationale annotation may expose social biases that the student could inherit. Faithful rationales can make student predictions more accountable by clarifying whether unintended biases influence decisions.
- Large-LM rationale annotation may expose social bias encoded in those models, which the student model may further inherit.
- Faithful rationales make student predictions accountable and help users assess whether decisions rely on unintended bias.
A.1 Example generations from teachers
Table 2 presents examples in which contrastive decoding with empty or wrong answers produced higher-rated rationales than greedy decoding in human evaluation.
- Contrastive decoding with empty or wrong answers produced higher-rated rationales than greedy decoding in human evaluation.