Source-linked AI summary
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, Yejin Choi
TL;DR
Small language models had not been shown to benefit reliably from chain-of-thought prompting, so the paper introduces SCoTD to distill teacher-generated rationalizations into smaller students. Across commonsense benchmarks, SCoTD improves supervised and few-shot performance, especially in difficult settings, while many teacher samples per instance are particularly important.
Problem
Chain-of-thought prompting had been shown beneficial mainly for sufficiently large models, leaving whether smaller models could be taught this reasoning capacity unresolved.
Method
SCoTD samples chain-of-thought rationales and labels from a large teacher on task instances, then trains a smaller student on the resulting corpus.
Results
SCoTD significantly improves students across three commonsense QA tasks in supervised and few-shot settings, including contrast sets and fully held-out tasks.
Takeaways & Limitations
Sampling multiple and diverse chain-of-thoughts for each input, with self-consistency when teacher chains are noisy, is recommended.
Takeaways & Limitations
The study considers only English tasks, relies on closed-source GPT-3, and focuses on OPT students; rationalizations may also create automation-bias risks.
Abstract
from arXiv · showhide
Chain-of-thought prompting (e.g., "Let's think step-by-step") primes large language models to verbalize rationalization for their predictions. While chain-of-thought can lead to dramatic performance gains, benefits appear to emerge only for sufficiently large models (beyond 50B parameters). We show that orders-of-magnitude smaller models (125M -- 1.3B parameters) can still benefit from chain-of-thought prompting. To achieve this, we introduce Symbolic Chain-of-Thought Distillation (SCoTD), a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model. Experiments across several commonsense benchmarks show that: 1) SCoTD enhances the performance of the student model in both supervised and few-shot settings, and especially for challenge sets; 2) sampling many reasoning chains per instance from the teacher is paramount; and 3) after distillation, student chain-of-thoughts are judged by humans as comparable to the teacher, despite orders of magnitude fewer parameters. We test several hypotheses regarding what properties of chain-of-thought samples are important, e.g., diversity vs. teacher likelihood vs. open-endedness. We release our corpus of chain-of-thought samples and code.
1 Introduction
Chain-of-thought prompting had been shown beneficial mainly for sufficiently large models, motivating SCoTD to teach smaller students through teacher-generated rationalizations. The method improves student performance across settings and tasks, with sampling volume especially important.
- SCoTD fine-tunes 125M–1.3B-parameter students on chain-of-thought demonstrations sampled from a larger teacher model.The demonstrations may be diverse and filtered.
- Chain-of-thought prompting had been shown beneficial mainly for models exceeding 60B parameters.
- SCoTD significantly improves smaller-model performance on three commonsense question-answering tasks in supervised and few-shot settings.Gains are especially pronounced on contrast sets and fully held-out tasks.
- Sampling many rationales per example, such as 30, is key to the distillation process.The paper contrasts this with prior practice using one rationale per example and tests filtering by probability, diversity, and open-endedness.
- Random downsampling performs well among filters, suggesting that rationale volume itself contributes substantially to performance.
- The authors release the sampled chain-of-thought corpus and code.
2 Symbolic Chain-of-Thought Distillation
SCoTD constructs teacher-generated chain-of-thought and label data from unlabeled task instances, then trains and evaluates a smaller student on those demonstrations. It supports both single-path and self-consistency decoding, with supervised filtering when gold labels are available.
- SCoTD assumes an unlabeled training set and a large teacher capable of few-shot chain-of-thought generation.
- Researchers create few-shot teacher prompts from examples with gold labels and manually authored rationales.The prompt set contains tuples of input, label, and rationale.
- For each training instance, the teacher generates N chain-of-thoughts with predicted labels, forming the distillation corpus.
- Incorrect teacher labels can be discarded in supervised settings before the student is fine-tuned with the standard language-modeling loss.
- At evaluation, the student either selects the most likely rationale-label sequence or marginalizes reasoning paths through self-consistency and majority voting.
3 Experiments
Across three commonsense QA tasks, SCoTD improves smaller students in supervised and few-shot settings, with gains strengthened by multiple teacher rationales and extending to difficult or reduced-data settings.
- Default SCoTD results: SCoTD significantly improves student performance in supervised and few-shot settings, and chain-of-thought training usually outperforms label-only distillation.The student learns to generate chain-of-thoughts through distillation despite initially failing to do so through prompting.
- Human evaluations: 59% and 61% of human comparisons favored SCoTD over the undistilled student on random and correctness-controlled samples, respectively, with p<.001 in both settings.The advantage also holds at p < .05 for each QA dataset individually.
- Human evaluations: SCoTD-trained students produce chain-of-thoughts judged comparable to those from the much larger teacher, although teacher task accuracy remains higher in most cases.Against the teacher, SCoTD wins 47% and 51% of human comparisons in the two evaluation settings, with p > .01.
- Data scaling: SCoTD trained with 40% of labelled data can surpass a direct supervised label-only model trained with 100%, while improving accuracy across training-data scales.This result is reported for CSQA and supports the value of teacher-provided explanations relative to additional labels.
- Student model size scaling: Across student sizes from 125M to 1.3B parameters, SCoTD outperforms standard supervised fine-tuning, and multiple rationales remain effective.The model-size comparison is conducted on CSQA.
4 What Factors are Important for Distillation?
The section tests whether data volume, diversity, teacher likelihood, or input open-endedness matters most when distilling chain-of-thoughts. Results indicate that random sampling is a strong baseline, diversity may help, but retaining many rationales is most important.
- Motivation: More sampled rationales improve student performance, motivating comparisons among data volume, diversity, teacher likelihood, and open-endedness.The study samples 30 rationales per instance by default and asks whether volume or specific rationale properties drive gains.
- Filtering setup: All filtered corpora contain the same number of training rationales, enabling comparisons among hypotheses about which properties matter.The filtering setup uses an average budget of 5 chain-of-thoughts per instance.
- Alternative hypotheses: The teacher-likelihood and open-endedness filters test whether high-probability rationales or broader rationale variation are especially valuable.Open-endedness is estimated from unique bigrams across 30 rationales, with different numbers retained by instance quintile.
- Results: Random subsampling is a strong baseline, while diversity shows some evidence of importance across the three evaluated tasks.The diversity filter selects samples intended to cover five embedding clusters.
- Results: None of the subsampled models approaches the model trained on the full 30x-per-instance corpus, indicating that rationale volume is a key contributor.The random-downsampling null hypothesis also performs well, supporting a role for sheer rationale volume.
5 Related Work
The paper builds on work treating explanations as learning signals, chain-of-thought as intermediate reasoning, and large language models as symbolic data generators. It distinguishes SCoTD through broader sampling, difficult transfer settings, and corpus analysis.
- Chain-of-thought prompting: Chain-of-thought extends earlier intermediate-structure approaches and has been studied through methods emphasizing diverse reasoning and improved rationales.The paper situates CoT prompting as more generally applicable than earlier algorithmic or structured reasoning approaches.
- Learning with explanations: Prior explanation-learning work treats explanations as inputs, targets, priors, or intermediate structures for machine-learning models.The cited literature includes methods that generate or use explanations during training and inference.
- Knowledge distillation: Symbolic knowledge distillation uses large language models as training-data generators rather than distilling soft representations such as logits.SCoTD continues this line by training smaller models on teacher-generated rationales and labels.
- Contemporaneous work: Compared with contemporaneous work, this paper emphasizes many sampled chains, transfer to challenge sets and unseen tasks, and analysis of important teacher-corpus factors.The comparison identifies three extensions claimed by the paper.
6 Conclusion
The conclusion presents SCoTD as a way for smaller language models to use chain-of-thought-style reasoning across tasks, model sizes, supervision levels, and difficult settings. It recommends multiple diverse chains and self-consistency while identifying directions for broader evaluation.
- Conclusion: SCoTD enables smaller language models to use chain-of-thought-style reasoning across downstream tasks and varied experimental settings.The settings include different student sizes, supervision levels, challenge sets, and unseen tasks.
- Recommendations: The paper recommends sampling multiple diverse chains for each input and using self-consistency when teacher chains are noisy.These are presented as concrete recommendations from the study.
- Future work: Future work includes extending SCoTD to generation tasks, scaling source tasks for broader generalization, and testing additional chain-of-thought factors.These directions correspond to the paper’s stated avenues for future work.
Limitations
The study’s scope is limited to English data, GPT-3 teacher outputs, and OPT students, and explanation-based systems may encourage overconfidence despite incorrect predictions.
- Study scope: The study considers only English-language chain-of-thoughts and tasks.This limits the evaluated language and task scope.
- Study scope: The teacher relies on closed-source GPT-3 with an unknown training set that may itself include some explanations.The provenance of the teacher’s training data is therefore not fully known.
- Study scope: The experiments focus on a single student-model type, OPT.The paper identifies this as a limitation of the study.
- Interpretability risk: Coherent rationalizations may not reflect the process producing a prediction, and displaying them can create automation-bias risks.The authors observed promising chains followed by incorrect final predictions and urge caution when showing chains to users.
A Crowdworking details
The study used a pairwise-judgment interface to collect human assessments of chain-of-thought quality, with compensation and worker-selection procedures. It also documented IRB exemption and protected annotator privacy.
- Annotators below a $15/hr rate were awarded post-hoc bonuses, and workers were selected from IP addresses in six countries.The listed countries were the US, Canada, New Zealand, Australia, and the United Kingdom.
- The crowdworking study was treated as IRB-exempt, and worker IDs were withheld so annotations could not be traced to individuals.The authors grounded the exemption assessment in 45 CFR 46 and noted that the statement was not legal advice.
- Pairwise human judgments of chain-of-thought quality were collected through a dedicated crowdworking interface.The interface is shown in Figure 8 and was used for judgments described in §3.1.1.