Source-linked AI summary
Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation
Yijun Tian, Yikun Han, Xiusi Chen, Wei Wang, Nitesh V. Chawla
TL;DR
Existing LLM distillation methods can lack knowledge diversity and rich reasoning supervision, despite the need for smaller, cheaper models. TinyLLM trains a small student from multiple teachers’ rationales and answers using contextual examples and teacher-forcing Chain-of-Thought. Across six datasets and two reasoning tasks, it reports substantial gains over full fine-tuning, teacher models, and prior distillation methods at much smaller model sizes.
Problem
Existing distillation methods often rely on a single teacher and ground-truth labels, limiting knowledge diversity and omitting the reasoning processes behind answers.
Source: §1 · p. 1Method
TinyLLM uses multiple large teachers, in-context examples, teacher-forcing Chain-of-Thought, and multi-task instruction tuning to train a small student on answers and rationales.
Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 4Results
Takeaways & Limitations
TinyLLM transfers broader reasoning knowledge to substantially smaller models, whose sizes are reported as 1.1% to 26.0% of the teachers.
Source: §5 · p. 8Future work must address conflicting rationales from broader teacher sets and improve in-context example selection through semantic similarity.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Transferring the reasoning capability from stronger large language models (LLMs) to smaller ones has been quite appealing, as smaller LLMs are more flexible to deploy with less expense. Among the existing solutions, knowledge distillation stands out due to its outstanding efficiency and generalization. However, existing methods suffer from several drawbacks, including limited knowledge diversity and the lack of rich contextual information. To solve the problems and facilitate the learning of compact language models, we propose TinyLLM, a new knowledge distillation paradigm to learn a small student LLM from multiple large teacher LLMs. In particular, we encourage the student LLM to not only generate the correct answers but also understand the rationales behind these answers. Given that different LLMs possess diverse reasoning skills, we guide the student model to assimilate knowledge from various teacher LLMs. We further introduce an in-context example generator and a teacher-forcing Chain-of-Thought strategy to ensure that the rationales are accurate and grounded in contextually appropriate scenarios. Extensive experiments on six datasets across two reasoning tasks demonstrate the superiority of our method. Results show that TinyLLM can outperform large teacher LLMs significantly, despite a considerably smaller model size. The source code is available at: https://github.com/YikunHan42/TinyLLM.
1 Introduction
TinyLLM addresses the limited diversity and shallow contextual supervision of existing distillation methods by training a small student from multiple teachers’ rationales and answers. Across six datasets and two reasoning tasks, it substantially outperforms full fine-tuning, teacher models, and prior distillation methods while using much smaller models.
Motivation
- Existing distillation methods commonly rely on one teacher and ground-truth labels, limiting knowledge diversity and omitting the reasoning processes behind answers.These limitations can restrict the student to one perspective and fail to capture nuanced teacher decision-making.Source: §1 · p. 1
Approach
- TinyLLM distills knowledge from multiple large teachers while supervising the student to generate both correct answers and supporting rationales.The approach is designed to provide broader skills and knowledge than single-teacher distillation.Source: §1 · p. 2
- An in-context example generator and teacher-forcing Chain-of-Thought strategy help produce rationales grounded in task-appropriate contexts.The examples provide demonstrations, while teacher forcing incorporates the correct answer to support accurate rationale generation.Source: §1 · p. 2
Results
- +15.69% is the largest reported improvement over full fine-tuning, while TinyLLM also improves up to +23.40% over teacher models and up to +11.79% over state-of-the-art distillation methods.These results are reported across six datasets covering commonsense and biomedical reasoning tasks.Source: §1 · p. 2
- TinyLLM achieves these gains with model sizes reported as only 1.1% to 26.0% of the teacher models.The paper presents this as a substantially smaller deployment footprint alongside the reported performance improvements.Source: §1 · p. 2
2 Related Work
Prior work establishes the capabilities and deployment costs of LLMs, then develops chain-of-thought rationales and knowledge distillation to transfer reasoning to smaller models. TinyLLM builds on these directions by combining rationale supervision from multiple teachers with a student that can reason without teacher models at test time.
Large Language Models
- LLMs support sophisticated question answering and text interpretation but remain costly to adapt and deploy because of their large parameter counts and computational demands.Related approaches such as prompt tuning seek to reduce intensive training and computational costs.Source: §2 · p. 2
Chain of Thought
- Generated rationales can improve few-shot or zero-shot performance and train smaller task-specific models, but human rationales are expensive and teacher-model inference can remain a deployment burden.PINTO, for example, still relies on an LLM to generate rationales at test time.Source: §2 · p. 2
Knowledge Distillation
- Knowledge distillation transfers larger teachers’ behavior to smaller students, while Chain-of-Thought samples expose reasoning processes beyond final answers.White-box methods use internal teacher signals, whereas black-box settings rely on generated outputs or rationales.Source: §2 · p. 2
- Single-teacher rationale distillation risks bias and limited reasoning diversity, motivating multi-teacher integration as a way to capture broader reasoning styles.TinyLLM’s pipeline obtains rationales from multiple large LLMs before training the student.Source: §2 · p. 2, Figure 1 · p. 3
3 Method
TinyLLM first obtains contextually grounded rationales from multiple teachers, then trains a small student to predict answers and reproduce teacher reasoning. Its objective combines ground-truth answer supervision with weighted rationale-generation objectives for the different teachers.
3.1 Preliminary
- TinyLLM frames multiple-choice question answering as selecting the ground-truth answer from a set of candidate options.Knowledge distillation then trains a smaller student to learn from a larger teacher’s outputs.Source: §3.1 · p. 3
3.2 Obtaining Rationales from Teachers
- The rationale-acquisition stage uses in-context examples, teacher forcing, and multiple teachers to generate higher-quality supervision.The correct answer is included with the question and options, while examples provide task-specific context.Source: §3.2 · pp. 3–4
- Teacher rationales are generated for each question from its question, options, correct answer, and in-context examples.Each teacher contributes a rationale based on the same task instance and contextual demonstrations.Source: §3.2 · pp. 3–4
3.3 Learning a Small Student
- The student uses teacher rationales as training supervision rather than requiring pre-generated rationales at inference time.This trains the student to generate explanations itself and removes the need for teacher-generated rationales on the test set.Source: §3.3 · p. 4
- Multi-task instruction tuning assigns distinct prefixes to answer and teacher-specific rationale tasks, allowing one student to learn labels and multiple reasoning targets.The framework integrates rationales from multiple teachers while preserving task distinctions through unique prefixes.Source: §3.3 · p. 4
- The overall loss combines ground-truth answer learning with weighted objectives for the rationales generated by the M teachers.L_A trains answer prediction, while each L_Tm trains the student to mimic teacher T_m’s rationale; α_m weights each teacher objective.Source: §3.3 · p. 4
4 Experiments
TinyLLM is evaluated across six datasets and two reasoning tasks against fine-tuning, distillation, and teacher-model baselines, alongside efficiency, ablation, sensitivity, and case-study analyses. It consistently improves performance while using smaller models and less training data, with results indicating complementary contributions from its components and task-dependent rationale weighting.
4.1 Experimental Setup
- The experiments use Open-BookQA, ARC, PIQA, Riddle-Sense, PubMedQA, and BioASQ, with FLAN-T5 students ranging from 80M to 780M parameters and FLAN-T5 and LLaMA 2-chat teachers.Baselines include inference-only evaluation, LoRA, full fine-tuning, PINTO, standard KD, and Distill-step-by-step.Source: §4.1 · p. 4, §4 · p. 4
4.2 Performance Comparison
- TinyLLM improves over full fine-tuning and Distill-step-by-step across student sizes, with gains of +15.69%, +11.55%, and +5.07% over full fine-tuning and +10.00%, +10.32%, and +11.79% over Distill-step-by-step for 80M, 250M, and 780M students, respectively.The evaluation covers six datasets spanning commonsense and biomedical reasoning, with accuracy as the metric.Source: §4.2 · p. 4, Table 1 · p. 5
- TinyLLM also surpasses larger teachers: the 780M student improves by +14.56% over the 3B teacher and +23.40% over the 7B teacher, while the 250M model uses only 8.3% and 3.6% of their parameters.The 250M student exceeds the 3B and 7B teachers by +0.82% and +8.60%, respectively.Source: §4.2 · p. 5
4.3 Efficiency Analysis of Training Set Size in Knowledge Transfer
- TinyLLM remains superior with reduced training data, including matching or exceeding full-data Distill-step-by-step performance on PQA using only 12.5% of the training set.Across 80M and 250M students, it outperforms Distill-step-by-step across tested training-data ratios and can surpass full fine-tuning with 12.5% data on specified ARC and PQA settings.Source: Figure 2 · p. 6, §4.3 · p. 6
4.4 Ablation Study
- The complete TinyLLM model outperforms every ablation, while similar gaps among ablations indicate that in-context examples, multiple-teacher rationales, and teacher forcing contribute complementarily.The ablation variants remove in-context examples, individual teachers, teacher diversity, or teacher forcing.Source: §4.4 · pp. 6–7, Table 2 · p. 6
4.5 Parameter Sensitivity
- Optimal rationale weights vary by task: biomedical PQA favors smaller α values, whereas concise commonsense ARC tasks require larger α values to exploit teacher guidance.Increasing α generally helps until excessive rationale emphasis degrades prediction accuracy; α_T5 is more sensitive on commonsense tasks, while sensitivities are more balanced on biomedical tasks.Source: §4.5 · p. 7, Figure 3 · p. 7
4.6 Case Study
- Case studies show TinyLLM can combine conflicting teacher rationales, filtering misleading T5 reasoning when LLaMA supplies the accurate rationale on ARC and PIQA examples.Other methods fail in the described examples, while TinyLLM predicts the correct answer in both cases.Source: §4.6 · p. 8, Figure 4 · p. 8
5 Conclusion and Future Work
TinyLLM combines multi-teacher distillation, contextually grounded rationale generation, and teacher-forcing Chain-of-Thought to improve smaller student models. Across six datasets and two reasoning tasks, it achieves substantial gains while using only 1.1%–26.0% of teacher model size, but future work must address conflicting rationales and more targeted example selection.
Conclusion
- TinyLLM improves performance by up to +15.69% over full fine-tuning, +23.40% over teacher models, and +11.79% over state-of-the-art distillation methods.These gains are reported across six datasets spanning two reasoning tasks.Source: §5 · p. 8
- TinyLLM transfers reasoning capabilities from multiple large teachers to a student model through rationale guidance, an in-context example generator, and teacher-forcing Chain-of-Thought.The design aims to provide contextually appropriate and credible rationales while broadening the knowledge transferred to the student.Source: §5 · p. 8
- The student models use only 1.1% to 26.0% of the teachers’ model sizes while achieving the reported performance improvements.Source: §5 · p. 8
Future Work
- Future work must manage conflicting rationales caused by teacher hallucinations, differing reasoning styles, or different domains of expertise.Source: §5 · p. 8
- Future work also considers open-source embeddings to select semantically similar in-context examples and test whether more targeted context improves student comprehension and performance.Source: §5 · p. 8