PaperlayerView exact revision on arXiv

Source-linked AI summary

Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning

Xinghao Chen, Zhijing Sun, Wenjin Guo, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, Xiaoyu Shen

arXiv:2502.18001v3cs.CL

TL;DR

CoT prompting supports LLM reasoning but increases computational demands, motivating distillation into SLMs. This study examines teacher choice, granularity, and format, finding that their effects vary by student and that stronger teachers do not always yield better students.

  • Problem

    CoT prompting increases computational demands, while the most effective teacher, granularity, and format for training reasoning-capable SLMs remain an open question.

    Source: §1 · pp. 1–2

  • Method

    The paper studies CoT distillation by varying teacher, granularity, and format in supervision used to train students to generate reasoning steps and answers.

    Source: §3 · p. 3

  • Results

    Stronger SLMs benefit from finer-grained CoTs while weaker ones favor simpler supervision; format affects SLMs less than LLMs, and better teachers do not always yield better students.

    Source: Conclusion · pp. 5, 8, §8 · p. 8

  • Takeaways & Limitations

    Granularity and teacher choice should be tailored to student capacity and task: LLM-generated CoTs help mathematical reasoning, while human annotations excel in commonsense reasoning.

    Source: Conclusion · pp. 5, 8, §8 · p. 9

    Safety refusals led the authors to use provided answers for some fine-tuning examples, potentially constraining reasoning-chain diversity and quality.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought (CoT) prompting. However, CoT prompting greatly increases computational demands, which has prompted growing interest in distilling CoT capabilities into Small Language Models (SLMs). This study systematically examines the factors influencing CoT distillation, including the choice of granularity, format and teacher model. Through experiments involving four teacher models and seven student models across seven mathematical and commonsense reasoning datasets, we uncover three key findings: (1) Unlike LLMs, SLMs exhibit a non-monotonic relationship with granularity, with stronger models benefiting from finer-grained reasoning and weaker models performing better with simpler CoT supervision; (2) CoT format significantly impacts LLMs but has minimal effect on SLMs, likely due to their reliance on supervised fine-tuning rather than pretraining preferences; (3) Stronger teacher models do NOT always produce better student models, as diversity and complexity in CoT supervision can outweigh accuracy alone. These findings emphasize the need to tailor CoT strategies to specific student model, offering actionable insights for optimizing CoT distillation in SLMs. The code and datasets are available at https://github.com/EIT-NLP/Distilling-CoT-Reasoning.

1 Introduction

CoT can improve LLM reasoning but increases computational demands, motivating its distillation into SLMs. The paper examines how teacher choice, reasoning granularity, and format shape student learning, and reports that their effects vary across models.

1 Introduction

  • CoT prompting improves LLM reasoning but increases computational demands, motivating efforts to distill CoT capabilities into SLMs.The introduction frames the central question as identifying effective CoT supervision for robust student reasoning.Source: §1 · pp. 1–2
  • The study varies teacher, granularity, and format across four mathematical and three commonsense reasoning datasets, using four teachers and seven student models.The generation approach aims to maintain teaching-style consistency while controlling granularity.Source: §1 · p. 2
  • Stronger students benefit from finer-grained reasoning, while weaker students can be overwhelmed by excessive explanations and prefer simpler CoT annotations.The introduction contrasts this non-monotonic SLM pattern with LLMs’ monotonic benefit from detailed steps.Source: §1 · p. 2
  • CoT format affects LLMs more than SLMs, which adapt more readily to diverse formats during fine-tuning.The introduction suggests LLMs’ pretraining preferences for certain structures may explain their greater sensitivity.Source: §1 · p. 2
  • Better teachers do not always produce better students, and near-perfect human-annotated CoTs often underperform LLM-generated CoTs.This challenges the assumption that teacher quality alone determines student performance.Source: §1 · p. 2

2 Related Work

Prior work explores longer reasoning and tailored formats for LLMs, and teacher-generated CoTs for transferring reasoning to SLMs. The paper identifies a remaining need for systematic study of how granularity, format, and teaching strategies interact in CoT distillation.

2 Related Work

  • CoT optimization for LLMs includes longer reasoning paths and tailored formats, but these approaches can carry substantial computational costs.Prior work also reports that overly concise steps can reduce effectiveness on complex problems.Source: §2 · p. 2
  • Knowledge distillation transfers teacher-generated CoT steps to SLMs, which struggle with complex reasoning because of limited capacity.Prior studies use a teacher–student approach to guide SLMs on reasoning-intensive tasks.Source: §2 · p. 2
  • A systematic exploration of balancing CoT granularity, format, and teaching strategies remains lacking.The paper presents this as a gap relevant to optimizing distillation for efficient reasoning in SLMs.Source: §2 · p. 3

3 Problem Formulation

The paper formulates CoT distillation as training a student to generate intermediate reasoning for each input and its correct answer. The supervision depends on the teacher, reasoning granularity, and presentation format.

3 Problem Formulation

  • CoT distillation trains a student to generate intermediate reasoning steps for each input so it can produce the correct answer.The formulation treats each example as an input–answer pair in a reasoning dataset.Source: §3 · p. 3
  • The teacher supplies a reasoning chain for each input and may be an LLM or a human with differing styles and expertise.Teacher choice is one of the three factors governing the student’s CoT supervision.Source: §3 · p. 3
  • Granularity is the level of detail in a CoT: high granularity gives step-by-step reasoning, while low granularity skips steps for a more abstract summary.The chain is represented as a sequence of reasoning steps, with more steps and more tokens per step indicating higher granularity.Source: §3 · p. 3
  • Format is the structure used to present reasoning, including natural language, formal logic, or symbolic representation.The student is fine-tuned on teacher-generated CoT annotations concatenated with the ground-truth answer, using a loss that measures discrepancy from ground truth.Source: §3 · p. 3

4 Experimental setup

The experiments generate CoT supervision from several teachers under varying granularity and format configurations, then evaluate student models on mathematical and commonsense reasoning datasets. The setup uses accuracy and examines student performance across granularity levels.

4.1 Generation of CoT Annotation

  • The study uses GPT-4o, Gemini-1.5-Flash, LLaMA 3 70B, and human-annotated CoTs as teacher supervision.A representative training problem serves as a one-shot prompt; teachers generate multiple granularity levels or each format individually.Source: §4.1 · p. 3

4.2 Tasks and Datasets

  • Mathematical evaluation covers SVAMP, GSM8K, AQuA-RAT, and MATH, while commonsense evaluation covers CSQA, OBQA, and StrategyQA.The mathematical datasets vary in difficulty and answer type; the commonsense datasets use multiple-choice or binary formats.Source: §4.2 · p. 3
  • The experiments use accuracy, calculated as the ratio of correctly predicted instances to the total number of instances.Answers are extracted from generated responses using predefined templates and regular expressions.Source: §4.2 · pp. 3–4
  • Most student models reach peak accuracy at intermediate granularity levels.Figure 2 presents student performance across different granularity levels.Source: Figure 2 · p. 4

5 Effects of Granularity

Student accuracy does not improve monotonically with CoT granularity: stronger students benefit from finer reasoning, while weaker students often peak at intermediate levels. The findings support matching granularity to student capacity rather than following teacher preferences uniformly.

5 Effects of Granularity

  • Most student models peak at intermediate CoT granularity; stronger Gemma and LLaMA models gain at higher levels, while BLOOM struggles on challenging datasets.The table covers Gemma 2B, LLaMA 3.2 3B, and BLOOM 3B across six granularity levels.Source: §5 · p. 4, Table 1 · p. 5
  • Granularity, not sequence length alone, is associated with reasoning gains; adding filler may degrade performance.The length-control experiment pads lower-granularity samples with non-informative content to match higher-granularity sequence lengths.Source: §5 · pp. 4–5
  • Length-matching Level 1 chains to Level 5 with filler did not reproduce the gains from actual higher-granularity reasoning.The comparison tests Gemma 2B on GSM8k and AQuA-RAT; filler may add noise or distract the model.Source: §5 · pp. 4–5, Table 2 · p. 5
  • As student capacity increases, performance aligns more closely with the teacher’s granularity preferences, indicating stronger transfer of reasoning structures.Weaker students show lower correlation and less ability to adapt to the teacher’s preferred granularity.Source: §5 · p. 5

Conclusion

  • The conclusion recommends tailoring granularity to student ability to improve CoT distillation efficiency and effectiveness.Stronger students benefit from higher granularity, whereas weaker students peak at intermediate levels and struggle with complex tasks.Source: §5 · p. 4, Conclusion · p. 5

6 Effects of Format

The study compares alternative reasoning formats with original CoT while holding granularity constant. Original CoT often performs better overall, although some alternatives help on particular tasks or for stronger students.

6 Effects of Format

  • Original CoT often outperforms more complex or modified formats for SLMs, while alternatives generally add cognitive load and bring little improvement.Table 3 compares student performance across formats by dataset, marking each dataset’s best performance and changes relative to original CoT.Source: Table 3 · p. 6, §6 · p. 6
  • Figure 3 plots GPT-4o teacher performance against student accuracy across datasets and granularity levels, with each point marking a dataset.Source: Figure 3 · p. 6
  • The format comparison uses GPT-4o to modify original CoT while keeping granularity unchanged, testing three alternative structures.The study compares original CoT with least-to-most, Rephrase and Respond, and Symbolic CoT.Source: §6 · p. 6
  • Rephrase and Respond can benefit commonsense tasks, while least-to-most sometimes helps mathematical reasoning.These gains are task-specific rather than consistent across settings.Source: §6 · p. 6
  • LLaMA 3.2 3B can improve under alternative formats, but those improvements do not generalize across datasets.The reported benefit is tied to specific tasks.Source: §6 · p. 7
  • Format benefits are often task-dependent and come with additional tokens; the paper judges format changes alone less effective than focusing on granularity.The authors suggest weaker models may struggle to learn and use diverse formats, in part because their pretraining corpora contain fewer examples of them.Source: §6 · p. 7

Conclusion

  • For general-purpose SLM training, original teacher CoT often remains most effective, though alternative formats can help in some tasks.Source: Conclusion · p. 7

7 Effects of Teacher Model

Teacher accuracy alone does not determine student performance: stronger students benefit more from stronger teachers, while effective CoT sources differ between mathematical and commonsense tasks. CoT distillation also yields larger gains for stronger students.

Is Higher Teacher Accuracy Always Better?

  • Higher teacher accuracy does not always correspond to higher student accuracy, although stronger students tend to benefit more from stronger teachers.For simpler tasks, a less advanced teacher can produce results comparable to more computationally expensive teachers.Source: Figure 4 · p. 7, Is Higher Teacher Accuracy Always… · p. 7

Human vs LLM: Task-Specific Effectiveness

  • LLM-generated CoTs improve student accuracy on mathematical tasks, whereas human-annotated CoTs improve performance on commonsense tasks such as StrategyQA.The authors associate LLMs’ structured chains with mathematics and human annotations’ contextual understanding with ambiguous or open-ended questions.Source: Human vs LLM: Task-Specific… · pp. 7–8, Figure 5 · p. 8
  • CoT distillation yields greater performance gains for stronger students, whose capacity helps them use detailed reasoning on challenging tasks.The authors relate weaker students’ narrower Zone of Proximal Development to difficulty absorbing complex CoT reasoning.Source: Figure 6 · p. 8, Human vs LLM: Task-Specific… · p. 8

Conclusion

  • Teacher choice should be task-specific: stronger students benefit more from advanced teachers, but a better teacher does not always produce a better student.The conclusion contrasts LLM-generated CoTs for mathematical reasoning with human annotations for commonsense reasoning.Source: Conclusion · p. 8

8 Conclusion

The study examines teacher selection, CoT granularity, and format in SLM distillation. Its findings point to tailoring supervision to student capacity and task.

8 Conclusion

  • Finer-grained CoT benefits stronger SLMs, while weaker models perform better with simpler annotations.The conclusion also reports that CoT format affects SLMs more subtly than LLMs.Source: §8 · p. 8
  • Better teacher models do not always yield better students, and human-annotated CoTs underperform on mathematical tasks but can surpass LLM-generated CoTs on some commonsense tasks.The authors relate distillation effectiveness to students’ ability to absorb reasoning complexity within their Zone of Proximal Development.Source: §8 · p. 8
  • CoT distillation is more effective for stronger SLMs and complex tasks, motivating tailored granularity and teacher selection strategies.This conclusion is scoped to optimizing reasoning performance in resource-constrained settings.Source: §8 · p. 9

Limitations

The study notes limitations in generating CoT data and in its focus on analyzing existing approaches.

Limitations

  • Safety refusals led the authors to fine-tune on provided answers instead of generated CoTs in some cases, potentially constraining reasoning-chain diversity and quality.The authors state that this may have affected distillation outcomes.Source: Limitations · p. 9
  • Some teachers could not reverse-engineer plausible CoTs from given answers for certain tasks, resulting in incomplete or suboptimal reasoning chains.Source: Limitations · p. 9
  • The study analyzes existing approaches rather than novel KD techniques, so alternative training strategies may affect distillation outcomes.The authors identify this as a scope boundary and call for further exploration of distillation techniques.Source: Limitations · p. 9

Ethics Statement

The supplementary material describes the datasets, training and CoT-generation workflows, prompts, examples, results, and a padding procedure. It also states the study’s ethics position.

Ethics Statement

  • The authors state that the datasets are publicly released, labeled through human interaction in English, contain no personal information, and use artifacts consistently with their intended purpose.They report that user privacy is protected and conclude that the research meets ACL ethics.Source: Ethics Statement · p. 9

A Overview of Training and Test Datasets

B Training setup

C.1 Workflow

  • For granularity data, teachers receive a question, its ground-truth answer, and a 1-shot example, then generate outputs at six detail levels for alignment with dataset order.The workflow first creates the 1-shot example; the included prompts aim to maintain consistency across levels.Source: §C.1 · pp. 14–15

C.2 Prompts

  • The prompt appendix includes a synthetic 1-shot granularity example, a six-level prompt, and a British Literature question used as a granularity case study.Tables 7–9 are identified as same-problem granularity comparisons generated by GPT-4o, LLaMA 3 70B, and Gemini-1.5-Flash.Source: Figure 8 · p. 16, §C.2 · p. 16, §C.3 · p. 17

D.1 Workflow

  • The format workflow uses 1-shot examples and asks ChatGPT to generate Least-to-most, RaR, and SymbolicCoT variants while preserving the original logic.The appendix also includes prompts for those formats and a same-problem format case study.Source: §D.1 · p. 20, Figure 10 · p. 21, Figure 11 · p. 21

E Whole Results of Granularity Experiments

  • Table 10 reports model performance and standard deviations across six granularity levels, with relative improvement between the highest and lowest levels indicated.Its caption defines the Only Answer condition as student models fine-tuned to predict answers without CoT.Source: Table 10 · p. 23

F Padding Procedure for Matched-Length CoT Variants

  • The padding procedure repeats and truncates lower-granularity sequences to match higher-granularity sequence lengths, aiming to isolate granularity from sequence length.The described algorithm takes level-1 and level-5 CoT outputs as input.Source: F Padding Procedure for Matched-Length… · p. 24

G Analysis of Symbolic CoT Performance in CoT distillation

The case study shows BLOOM 3B producing an incorrect earnings answer despite correctly calculating the remaining eggs. The authors discuss task fit, implementation, and pretraining as possible contributors to Symbolic CoT’s poor performance, while Figure 13 summarizes student accuracy across teachers.

G Analysis of Symbolic CoT Performance in CoT distillation

  • BLOOM 3B predicts $26 instead of $18 after calculating 9 remaining eggs but later substituting 13 in the earnings calculation.The paper identifies the error as numerical inconsistency across reasoning steps, not a structural flaw in the symbolic expressions.Source: G Analysis of Symbolic CoT Performance… · p. 25
  • Symbolic CoT’s advantages may be less evident on the paper’s mathematical and commonsense datasets than on its primary target, logical reasoning tasks.The authors note that the deeper-reasoning advantages of symbolic reasoning do not manifest as clearly in their dataset setting.Source: G Analysis of Symbolic CoT Performance… · p. 25
  • The authors also cite missing multistage mechanisms and special tokens, plus limited symbolic-format pretraining, as possible reasons for weaker performance.They suggest weaker SLMs may struggle to acquire symbolic reasoning from only a small number of training samples.Source: G Analysis of Symbolic CoT Performance… · p. 25

H Student Performance across Different Teacher Models

  • Figure 13 reports each student model’s average accuracy after training on CoT from different teacher models.Source: Figure 13 · p. 26

Related papers

Loading 2502.18001v3…