Source-linked AI summary
Small Models Struggle to Learn from Strong Reasoners
Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, Radha Poovendran
TL;DR
Small models do not consistently benefit from long CoT reasoning or large-teacher distillation, creating a learnability gap for reasoning transfer. The paper proposes Mix Distillation, which combines reasoning traces of different complexity or teacher scale, and reports improved performance over training on either data alone. The study focuses on math reasoning and identifies broader skill and pre-training-data analyses as limitations.
Problem
Small models with ≤3B parameters do not consistently benefit from long CoT reasoning or distillation from larger models, limiting effective reasoning transfer.
Method
Mix Distillation combines long and short CoT examples or reasoning traces from larger and smaller teachers to balance complexity.
Results
Mix Distillation consistently improves small-model reasoning performance over standard distillation and training on either long or short data alone.
Takeaways & Limitations
Effective knowledge transfer to small models requires adapting reasoning complexity rather than relying only on direct strong-model distillation.
Takeaways & Limitations
The study primarily evaluates math reasoning and does not assess instruction following, code generation, multimodal understanding, or fine-grained pre-training-data effects.
Abstract
from arXiv · showhide
Large language models (LLMs) excel in complex reasoning tasks, and distilling their reasoning capabilities into smaller models has shown promise. However, we uncover an interesting phenomenon, which we term the Small Model Learnability Gap: small models ($\leq$3B parameters) do not consistently benefit from long chain-of-thought (CoT) reasoning or distillation from larger models. Instead, they perform better when fine-tuned on shorter, simpler reasoning chains that better align with their intrinsic learning capacity. To address this, we propose Mix Distillation, a simple yet effective strategy that balances reasoning complexity by combining long and short CoT examples or reasoning from both larger and smaller models. Our experiments demonstrate that Mix Distillation significantly improves small model reasoning performance compared to training on either data alone. These findings highlight the limitations of direct strong model distillation and underscore the importance of adapting reasoning complexity for effective reasoning capability transfer.
1 Introduction
Small models do not consistently learn from long reasoning traces or large teachers, but Mix Distillation improves transfer by balancing reasoning complexity.
- Motivation: High computational costs hinder deploying large language models on resource-constrained devices, motivating smaller models with similar capabilities at lower cost.
- Small Model Learnability Gap: Small models with ≤3B parameters do not consistently benefit from long CoT reasoning or distillation from large teacher models.They perform better with shorter CoT reasoning or smaller teachers that better match their learning capacity.
- Mix Distillation: Mix Distillation balances reasoning complexity through Mix-Long and Mix-Large configurations.Mix-Long combines long and short CoT examples, while Mix-Large combines responses from larger and smaller models.
- Experimental Results: More than 8 points on MATH and AMC are gained by Qwen2.5-3B-Instruct with Mix-Long versus direct training on long CoT data.
- Experimental Results: More than 7 points on MATH, AIME and AMC are gained by Qwen2.5-3B-Instruct with Mix-Large versus training on large teacher CoT data.
- Implications: The findings highlight a limitation of direct strong-model distillation and the importance of adapting reasoning complexity for knowledge transfer.
2 Preliminaries
The paper formalizes input, output, and chain-of-thought notation, then frames supervised fine-tuning as likelihood-based training on reasoning datasets.
- Notation: An input sequence x=(x_1,...,x_n) represents a prompt, while y=(y_1,...,y_m) represents its corresponding output sequence.
- Notation: CoT(y) denotes the subset of generated-output tokens that encodes a chain-of-thought reasoning trace or explanation.
- CoT Types: Short CoT means concise reasoning paths, whereas long CoT denotes extended reasoning with more complex reflective thoughts.
- Teacher CoT: Large teacher CoT and small teacher CoT refer to reasoning steps produced by larger and smaller teacher models, respectively.
- Supervised Fine-Tuning: Supervised fine-tuning uses an instruction dataset whose outputs may contain short, long, strong-model, or weak-model CoT sequences.
- Supervised Fine-Tuning: SFT updates model parameters by minimizing negative log-likelihood over the instruction dataset.
3 Small Model Learnability Gap
The Small Model Learnability Gap shows that reasoning-data complexity and teacher scale interact with student-model capacity: smaller students often favor short CoT and small-teacher traces, while larger students benefit from longer or larger-teacher reasoning.
- Experiment Setup: The study fine-tunes ten Qwen and Llama student models with different CoT and teacher-generated data, evaluating them across five math benchmarks.The benchmarks are MATH, GSM8K, AIME 2024, AMC 2023, and OlympiadBench; average performance is used unless otherwise specified.
- Long CoT Gap: Long CoT benefits larger students but harms smaller ones relative to short CoT.Qwen2.5-32B-Instruct improves about 15 points on average, whereas Qwen2.5-1.5B-Instruct performs over 10 points lower on MATH and AMC with long CoT.
- Large Teacher CoT Gap: Large-teacher CoT similarly favors larger students, while smaller students can learn better from small-teacher CoT.Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct improve over 5 points on average, but Qwen2.5-0.5B-Instruct degrades by more than 10 points on AMC with the large teacher.
- Domain Knowledge Helps: Math expertise is associated with a smaller learnability gap and greater benefit from large-teacher CoT in small models.Qwen2.5-Math-1.5B-Instruct has a significantly smaller long-CoT gap than Qwen2.5-1.5B-Instruct, and its improvement with large-teacher CoT exceeds the general model’s.
- Distribution Mismatch: Distribution mismatch between student and teacher models is identified as one contributing factor to the Small Model Learnability Gap.Base models generally exhibit a more significant gap than Instruct models, making long-CoT or large-teacher learning more challenging for small base models.
- Speaking Styles Shift: Long CoT and large-teacher CoT primarily shift student token distributions toward speaking-style elements.The analysis identifies expressive and stylistic tokens such as “wait”, “But”, and “Let” among the shifted tokens.
4 Mix Distillation: Bridge Small Model Learnability Gap
Mix Distillation bridges the small model learnability gap by combining reasoning traces of different complexity or teacher scale. For Qwen2.5-3B-Instruct, these mixtures outperform single-source training and produce more balanced reasoning behavior.
- Method: Mix-Long combines long and short CoT examples, while Mix-Large combines large-teacher and small-teacher responses.Both configurations blend easier-to-learn data with more challenging data for small models.
- Results: Mix Distillation improves small-model reasoning compared with training on a single dataset.Both Mix-Long and Mix-Large surpass baselines in most evaluation metrics.
- Results: More than 8 points separate Mix-Long from direct long-CoT training for Qwen2.5-3B-Instruct on MATH and AMC.The comparison is against direct training on long CoT data.
- Results: More than 7 points separate Mix-Large from large-teacher-CoT training for Qwen2.5-3B-Instruct on MATH, AIME, and AMC.This result compares Mix-Large with training on large teacher CoT data.
- Results: A mix weight of 0.2 yields the highest average performance across five benchmarks for both Mix-Long and Mix-Large.The student model in this analysis is Qwen2.5-3B-Instruct.
- Analysis: Mix-Long responses retain branching characteristics of long CoT while using fewer tokens and avoiding overly elaborate thinking.Long-CoT fine-tuning led to repeated thoughts and failure to stop, whereas short-CoT fine-tuning produced incorrect answers in the case study.
- Analysis: Average response lengths are 3384.7 tokens for long CoT, 575.7 for short CoT, and 1248.9 for Mix-Long.The mixed data therefore produces an intermediate CoT length.
5 Related Work
Related work covers short and long chain-of-thought prompting, automated reasoning-data construction, and knowledge distillation for transferring capabilities to smaller models.
- Chain-of-Thought Reasoning: Early CoT research emphasized succinct reasoning paths, while recent work increasingly studies extended and detailed reasoning chains.The related work frames short and long CoT as two developments in reasoning research.
- Reasoning Data: Human-crafted reasoning datasets improve LLM reasoning but require time and labor, motivating automated generation and web-based extraction.Recent approaches generate instructions or responses directly from LLMs or extract data from the web.
- Knowledge Distillation: Knowledge distillation transfers knowledge from large teacher models to smaller student models, with LLM research increasingly using token-level rather than logit-level distillation.The passage contrasts token-level distillation with traditional logit-level approaches.
6 Conclusion and Future Work
The paper concludes that small models benefit from reasoning data matched to their capacity, and introduces Mix Distillation to combine complementary reasoning distributions. It also identifies several directions for refining the approach and understanding the Small Model Learnability Gap.
- Conclusion: Mix-Long combines long and short CoT data, while Mix-Large integrates large- and small-model teacher CoT.Both configurations are presented as ways to bridge mismatches between reasoning complexity and small-model capacity.
- Conclusion: Both Mix-Long and Mix-Large outperform baselines across most evaluation metrics, implying advantages over training on a single data distribution.
- Conclusion: Short CoT and small-model CoT may better suit small students than long CoT and large-model responses.
- Future Work: Future work will refine mixing algorithms, generate teacher data better suited to small students, and study the gap theoretically and through model interpolability.
Limitations
The study’s limitations concern its narrow evaluation scope and incomplete analysis of how pre-training data composition affects the Small Model Learnability Gap.
- Scope: The study focuses on math reasoning and does not evaluate instruction following, code generation, or multimodal understanding.
- Scope: The study does not investigate how fine-grained variations in pre-training data composition affect the Small Model Learnability Gap.
- Future Analysis: Analyzing different pre-training data sources and proportions could provide further insight into data selection strategies for mitigating the gap.
Ethical Statement
The paper states that its focus is small-model reasoning through distillation and reports no introduced or endorsed harmful applications or ethical concerns.
- Scope: The paper focuses on evaluating and enhancing small-language-model reasoning capabilities through distillation techniques.
- Resources: The datasets and benchmarks used in the experiments are publicly available.
- Ethics: The paper does not introduce or endorse applications that could cause harm or be misused, and it reports no ethical concerns.
A.3 Evaluation Setup
The evaluation measures fine-tuned student models on standard math benchmarks under zero-shot greedy decoding, with answer correctness determined by exact matching and, when needed, a judge model. Results indicate that small and large students favor different CoT lengths.
- Evaluation Benchmarks: The evaluation covers MATH, GSM8K, AMC 2023, AIME 2024, and English math problems from OlympiadBench.
- Evaluation Procedure: Models are evaluated zero-shot with greedy decoding and a maximum generation length of 16k tokens.
- Evaluation Procedure: Answer correctness uses exact matching first, followed by Qwen-32B-Instruct judging when the extracted answer is incorrect.
- Results: Small students tend to benefit more from short CoT, whereas large students gain greater advantages from long CoT.
B.2 Large Teacher CoT Gap: Additional Results
Additional experiments show that small students align better with short CoT and small-teacher reasoning, while larger students benefit more from complex reasoning and large teachers. Perplexity, similarity, hyperparameter, prompting, and token-shift analyses support distribution mismatch as a key factor.
- Distribution alignment: Small students assign higher perplexity to large-teacher or long-CoT sequences, while small-teacher CoT yields lower perplexity.The lower perplexity indicates better alignment between the student distribution and training-data distribution.
- Distribution alignment: The perplexity gap between long and short CoT, and between large- and small-teacher CoT, shrinks as student size increases.This pattern indicates that larger students more easily adapt to complex reasoning distributions.
- Text similarity: Student responses are consistently more similar to small-teacher CoT than large-teacher CoT under both lexical and semantic metrics.The reported confidence intervals are tight and non-overlapping.
- Robustness checks: Short CoT consistently outperforms long CoT for Qwen2.5-1.5B-Instruct across tested epoch and learning-rate settings.The experiments varied epochs from 2 to 5 and learning rates from 5×10^-6 to 1×10^-4.
- Prompting analysis: Revised prompting does not make the large teacher outperform the small teacher for small students.The prompt explicitly requests simpler, more concise reasoning for student comprehension, but the reported teacher gap remains.
- Token-shift analysis: Most shifted tokens after long-CoT or large-teacher fine-tuning are analyzed through token rank changes and are predominantly expressive or stylistic.The analysis uses Qwen2.5-3B-Instruct as the student model and highlights tokens such as “But” and “Let”.
D Examples of Various CoT Data
This section presents examples of long, short, large-teacher, and small-teacher CoT data alongside analyses of shifted tokens, perplexity, similarity, and hyperparameter sensitivity. The examples include algebraic manipulation, divisor calculations, and arithmetic or geometry-style reasoning.
- CoT data examples: The section distinguishes long CoT, short CoT, large-teacher CoT, and small-teacher CoT through representative examples.The examples include separate long-CoT, short-CoT, large-teacher, and small-teacher labels.
- Token-shift illustrations: The section also includes figures identifying most shifted tokens after fine-tuning on long-CoT data and large-teacher CoT.The figures label the most shifted tokens in red and accompany the stated rank-shift calculation process.
- Small-teacher CoT: A conversational example introduces a geometry-style problem by restating the ordered-pair equation before solving for the maximum value of y.The example begins with x^2 + y^2 = 14x + 48y and frames the task as understanding the equation.
- Small-teacher CoT: Small-teacher examples show stepwise algebraic simplification, including clearing fractions, rearranging terms, and factoring a quadratic.The cited passages present these operations as consecutive solution steps.