PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models Are Reasoning Teachers

Namgyu Ho, Laura Schmid, Se-Young Yun

arXiv:2212.10071v2cs.CLcs.AIcs.LG

TL;DR

Prompt-based CoT reasoning depends on extremely large models that are costly to deploy, creating a need for complex reasoning in smaller models. Fine-tune-CoT uses large teachers to generate rationales for student fine-tuning, and diverse reasoning further improves student performance across tasks. The approach supports reasoning in small, accessible models, while teacher-generated data can transmit bias and toxicity.

  • Problem

    Prompt-based CoT methods rely on extremely large models that are prohibitive to deploy at scale, motivating complex reasoning in smaller models.

  • Method

    Fine-tune-CoT prompts very large teacher models with zero-shot CoT, then fine-tunes smaller students on the generated reasoning samples, optionally using diverse rationales.

  • Results

    Fine-tune-CoT elicits substantial reasoning in small models across 12 tasks, with models as small as 0.3B outperforming larger students and the 175B teacher on some tasks.

  • Takeaways & Limitations

    The task-agnostic approach makes complex reasoning feasible for real-world deployment using smaller, more accessible models.

  • Takeaways & Limitations

    Student models trained on teacher-generated reasoning samples may inherit biases and toxicity from the teacher model.

Abstract

from arXiv · show

Recent works have shown that chain-of-thought (CoT) prompting can elicit language models to solve complex reasoning tasks, step-by-step. However, prompt-based CoT methods are dependent on very large models such as GPT-3 175B which are prohibitive to deploy at scale. In this paper, we use these large models as reasoning teachers to enable complex reasoning in smaller models and reduce model size requirements by several orders of magnitude. We propose Fine-tune-CoT, a method that generates reasoning samples from very large teacher models to fine-tune smaller models. We evaluate our method on a wide range of public models and complex tasks. We find that Fine-tune-CoT enables substantial reasoning capability in small models, far outperforming prompt-based baselines and even the teacher model in many tasks. Additionally, we extend our method by leveraging the teacher model's ability to generate multiple distinct rationales for each original sample. Enriching the fine-tuning data with such diverse reasoning results in a substantial performance boost across datasets, even for very small models. We conduct ablations and sample studies to understand the emergence of reasoning capabilities of student models. Our code implementation and data are available at https://github.com/itsnamgyu/reasoning-teacher.

1 Introduction

Prompt-based chain-of-thought reasoning helps with complex tasks but depends on prohibitively large models. Fine-tune-CoT transfers teacher-generated rationales to smaller students, with diverse reasoning improving performance without human annotation.

  • Motivation: Prompt-based CoT methods require extremely large models, making them costly and difficult to deploy at scale.The paper targets models spanning hundreds of billions of parameters and their overwhelming computational and inference costs.
  • Approach: Fine-tune-CoT uses very large language models as reasoning teachers and fine-tunes smaller students on generated rationales.The approach applies zero-shot CoT prompting to teachers, then transfers the resulting reasoning samples to students.
  • Approach: Diverse reasoning generates multiple distinct solutions for each training example to augment student-model training data.The method uses stochastic sampling because complex tasks can admit multiple reasoning paths.
  • Results: Performance scales consistently with diverse reasoning, dataset size, teacher performance, and student model size.The analyses examine these axes and study how reasoning capabilities emerge in student models.

2 Related Work

Prior work established prompting and explicit reasoning as routes to downstream performance, but CoT generally requires very large models. Fine-tune-CoT instead transfers model-generated rationales to smaller models across accessible settings.

  • Prior paradigms: Pre-train-and-fine-tune approaches improve downstream performance, while prompting conditions models on exemplars or task descriptions.Smaller models often require additional engineering to perform similarly under prompting.
  • Chain-of-thought reasoning: Few-shot and zero-shot CoT prompt models to generate intermediate reasoning steps, enabling strong performance across tasks.Zero-shot-CoT does not require fine-tuning or task-specific conditioning.
  • Research gap: Prior work found that optimal CoT performance requires extremely large models, motivating rationale-based fine-tuning for smaller models.The paper generates those rationales with a very large model rather than relying on small-model prompting alone.
  • Distinction from prior work: Unlike related single-task or proprietary-model efforts, this work evaluates rationale transfer across many datasets with accessible open-source models.The paper contrasts its broad analysis with prior results limited to one dataset or inaccessible teachers.
  • Knowledge distillation: Fine-tune-CoT is related to knowledge distillation, but its teacher specifically transfers intermediate reasoning rather than only model behavior.The broader distillation literature targets smaller models that preserve accuracy and generalization while reducing size and latency.

3 Chain-of-Thought Fine-Tuning

Fine-tune-CoT generates, filters, and packages teacher rationales before training a small student. Its diverse-reasoning extension samples multiple explanations per example, trading development cost against student inference cost and quality.

  • Method overview: Fine-tune-CoT is a task-agnostic method that uses teacher-generated CoT samples to fine-tune small student language models.Zero-shot-CoT avoids requiring reasoning examples or long inference contexts.
  • Step 1: Reasoning generation: A teacher generates a rationale and final prediction for each question using zero-shot CoT prompting.The resulting sequence contains the question, a step-by-step instruction, the rationale, and the predicted answer.
  • Step 2: Curation: Generated samples are retained when the teacher’s final prediction matches the ground-truth answer, then reformatted as prompt-completion pairs.The curated sample contains the question, rationale, and answer with token-efficient delimiters.
  • Caveat: Answer-based filtering does not guarantee that the generated rationale itself is correct, especially for multi-choice questions.The paper identifies this as an important limitation and analyzes it separately.
  • Step 3: Fine-tune: The student is fine-tuned on assembled reasoning samples with the autoregressive next-token prediction objective.This is the same training objective used during pre-training.
  • Diverse reasoning: Diverse reasoning samples multiple teacher explanations per training example using temperature sampling before the same curation and fine-tuning steps.The degree of reasoning diversity creates a tradeoff between development cost and student inference cost or quality.

4 Experiments

Experiments across 12 complex-reasoning datasets and multiple model families show that Fine-tune-CoT transfers reasoning to much smaller students, with diverse rationales and scaling improving performance.

  • Main results: Fine-tune-CoT enables notable reasoning in small models, outperforming prompt-based baselines and exceeding fine-tuning or Few-shot-CoT on more than half the tasks.On MultiArith, the 6.7B student reaches 33% accuracy versus 5% for Zero-shot-CoT, 10% for Few-shot-CoT, and 15% for fine-tuning.
  • Main results: Fine-tune-CoT students with 1.3B or 6.7B parameters outperform the 175B teacher on Shuffled Objects and Coin Flip, reducing required parameters by approximately 25–100x.Even the 0.3B student consistently exceeds the 6.7B model under Zero-shot-CoT.
  • Diverse reasoning: Diverse reasoning improves Fine-tune-CoT across student sizes and can surpass Few-shot-CoT, vanilla fine-tuning, and larger non-diverse models.For 6.7B students, the reported boosts are around 26% on MultiArith and 17% on SVAMP; diversity also exceeds the teacher in Date Understanding and Last Letter.
  • Dataset size: The 6.7B student scales with dataset size across tasks, whereas vanilla fine-tuning does not always scale and can worsen with more data on Date Understanding.Fine-tune-CoT also gains from diverse reasoning, which is unavailable to standard fine-tuning.
  • Teacher performance: Student performance scales with teacher performance, especially on less complex tasks where student and teacher accuracy closely match.Teacher successes and failures are correlated with those of students, unlike the non-guaranteed scaling pattern noted for knowledge distillation.
  • Student scale: Fine-tune-CoT performance consistently scales with student model size, unlike Few-shot-CoT and vanilla fine-tuning on Date Understanding.Across T5, Flan-T5, and GPT-2, the same broad trends appear; diverse reasoning often adds significant gains.

5 Discussion

Fine-tune-CoT can produce interpretable reasoning in small models and remains applicable to complex tasks without task-specific engineering. The discussion highlights both evidence about how reasoning emerges and practical limits involving viability and tradeoffs.

  • Viability of Fine-tune-CoT: Zero-shot-CoT-generated rationales provide task-agnostic training data that can be applied to complex tasks without hand-crafted reasoning or task engineering.The setup can use publicly available teacher APIs and is suited to low-resource scenarios.
  • Viability of Fine-tune-CoT: Performance on some difficult datasets remains unsuitable for real-world use, including 30.33% on SVAMP.The authors identify diverse reasoning, dataset size, teacher performance, and student size as scalable improvement axes.
  • Tradeoffs of Fine-tune-CoT: Diverse reasoning improves student performance at the cost of additional teacher inference, creating a development-cost versus inference-cost/quality tradeoff.The added inference can instead be used to improve students or reduce the need for larger student models.
  • Tradeoffs of Fine-tune-CoT: Diverse reasoning has costs linear in the number of generated rationales and original samples, though preliminary analysis finds it can remain cost-effective versus hand-annotating data.The reported cost-performance Pareto front always incorporates diverse reasoning.
  • Emergence of CoT reasoning: More potent supervision gradually reduces reasoning errors, with differences in semantic and calculation errors across Zero-, Few-shot-CoT, and Fine-tune-CoT.Sample studies suggest explicit reasoning supervision can support the emergence of reasoning abilities in small models.
  • Distillation of emergent abilities: Fine-tune-CoT may distill chain-of-thought reasoning to much smaller models through fine-tuning, under certain domains.The paper frames this as evidence that an ability associated with very large models can be transferred to smaller students.

6 Conclusion

The paper presents Fine-tune-CoT, using large language models as reasoning teachers to transfer broad reasoning capabilities to students as small as 0.3B parameters. Diverse reasoning strengthens these teaching effects, and experiments show that the approach can make complex reasoning feasible for real-world deployment.

  • Conclusion: Fine-tune-CoT transfers reasoning capabilities previously found in >100B models to student models as small as 0.3B.The method uses large language models as reasoning teachers.
  • Conclusion: Diverse reasoning exploits the learning setup to maximize teaching effects and vastly improve performance.It is introduced as a novel extension of Fine-tune-CoT.
  • Conclusion: Fine-tune-CoT elicits significant reasoning performance in small models, demonstrating distillation of chain-of-thought reasoning described as an emergent ability of scale.The experiments use publicly available models and zero-shot prompting in a task-agnostic approach.

7 Limitations

The limitations discussion identifies inference-cost tradeoffs, unresolved performance gaps, and opportunities to improve Fine-tune-CoT with stronger models or prompting methods. It also connects the approach to knowledge-distillation tradeoffs between quantity and quality of reasoning data.

  • Exploring a wider array of models: The method is currently not state-of-the-art, although stronger or more efficient teachers and students may improve it.The authors specifically identify difficult datasets such as GSM8K as potential beneficiaries of stronger teachers.
  • Tradeoffs of Fine-tune-CoT: Diverse reasoning requires additional teacher inference, trading development cost against inference cost and student quality.This tradeoff should be considered when evaluating similar distillation methods.
  • Better CoT inference methods: Eight-shot chain-of-thought prompting can cost approximately 8 times more than zero-shot prompting under token-based pricing.The authors therefore trade teacher inference budget between few-shot prompting and diverse zero-shot reasoning.
  • Connection with knowledge distillation: Fine-tune-CoT shares characteristics with knowledge distillation, including benefits from training on some incorrect reasoning samples.The authors describe a quantity-quality tradeoff in student generalization between fewer curated rationales and more varied reasoning data.

8 Ethics Statement

The ethics discussion focuses on risks from transferring teacher-model bias and toxicity, while noting that sample curation may also help filter harmful outputs. It warns that small models could scale malicious reasoning capabilities.

  • Bias and toxicity: Teacher-model biases and toxic characteristics may be passed to students trained on teacher-generated reasoning samples.Teacher selection is therefore an important ethical consideration.
  • Bias and toxicity: Fine-tune-CoT's sample-based setup could support filtering biased or toxic samples before student fine-tuning.The paper mentions neural-network verifiers and prompt optimization as possible approaches.
  • Malicious use: Small models could let bad actors deploy complex reasoning capabilities at scale for malicious purposes.The authors call for safeguarding by major providers and stricter precautions around distillation schemes.

B Datasets

The study uses 12 reasoning datasets and distinct training and testing samples, with model and API choices documented for reproducibility.

  • The study documents the datasets and models used, including their original licenses and model descriptions.
  • The experiments cover 12 datasets used to measure reasoning performance.
  • Train-test split: Fine-tuning requires distinct training and testing sets; otherwise, the authors use a 70:30 samplewise random split.AQUA uses 10,000 randomly sampled training instances because its original training set is disproportionately large.
  • InstructGPT is the default teacher because of its superior zero-shot reasoning performance relative to same-size GPT-3.The default implementation uses text-davinci-002, while small InstructGPT models were unavailable through the OpenAI API.
  • Total API expenditure was $1,981, with most usage costs devoted to teacher-model inference.

D.1 Error analysis

Error analysis finds that difficulty, arithmetic, semantic sensitivity, and benchmark subjectivity constrain student reasoning, while diverse reasoning improves performance on some arithmetic tasks.

  • Difficult datasets: GSM8K and AQUA are too difficult for small students, with the teacher below 50% accuracy on both.On these datasets, correct answers are often apparently obtained by chance; Fine-tune-CoT nevertheless performs best among student methods.
  • Arithmetic mistakes: Small students frequently make arithmetic errors that compound across intermediate reasoning steps.These errors are prominent in SingleEq, AddSub, SVAMP, and GSM8K, where correct multi-step reasoning may not compensate for incorrect calculations.
  • Problematic benchmarks: CommonsenseQA often permits logical reasoning paths even when the marked answer is false because the exact answer can be subjective.
  • Aligned failures: Successes and failures tend to align across the teacher and student methods, with lower accuracy on instances where Fine-tune-CoT fails.The authors hypothesize that content bias may contribute to uneven performance across tasks and model sizes.
  • Semantic issues: Models are sensitive to question formulation, especially redundant information and hidden variables in SVAMP and MultiArith.These semantic issues are identified as major contributors to uneven vanilla Fine-tune-CoT performance.
  • Semantic issues: Diverse reasoning produces smoother reasoning and vastly improves accuracy on MultiArith and SVAMP.It helps the model understand questions better than Few-shot-CoT or vanilla Fine-tune-CoT.
  • Text-based datasets: Vanilla Fine-tune-CoT performs well on mainly text-based, non-calculational datasets such as Coin Flip, which reaches 98.7% accuracy.Clear task patterns provide cues for solving these datasets, where classic fine-tuning performs significantly worse.

E.1 Rationale filtering

The study examines rationale filtering, reasoning length, data splitting, and acquisition costs, finding tradeoffs between sample quality, quantity, and sequence length.

  • Rationale filtering: 28% of teacher samples with correct answers contain incorrect rationales, exceeding the 17.12% random-guess rate.
  • Rationale filtering: Answer-based filtering outperforms human rationale filtering by 5-11% when starting from the same initial samples.
  • Rationale filtering: When post-filtering sample counts are matched, golden samples outperform correct samples by 5-8%.The results indicate a tradeoff between reasoning-sample quality and quantity, alongside the quantity increase from diverse reasoning.
  • Reasoning length: Longer rationale limits improve performance significantly on AQUA and commonsense reasoning tasks.The appropriate reasoning length varies by dataset, so sequence-length parameters require careful selection.
  • Reasoning length: Long students generate rationale-length distributions similar to the teacher, whereas short students typically remain below approximately 128 tokens.
  • Data splitting: Naive samplewise splitting can leak shared templates across training and testing, reducing evaluation to pattern matching.Template-wise splitting is used to assess whether students learn reasoning rather than reproduce templates.
  • Rationale filtering: Answer-based filtering can outperform stricter human filtering, but matched-size comparisons favor golden samples.
  • Data acquisition: Diverse reasoning can improve cost-effectiveness, although its cost grows with dataset size and requires cost-performance tradeoffs.

F Data Annotation vs Diverse Reasoning

The analysis compares annotation and diverse-reasoning costs, finding that diverse reasoning can improve acquisition cost-effectiveness while introducing dataset-size-dependent expense.

  • Diverse reasoning can enhance data-acquisition cost-effectiveness compared with using D = 1 reasoning sample.
  • Diverse-reasoning costs correlate with dataset size, requiring explicit cost-performance tradeoffs.The analysis estimates annotation at 30 annotations per hour and $20 per hour, or $0.67 per question-answer sample.

G Experiments on Open Source Models

Experiments on open-source student models compare Fine-tune-CoT with prompt-based and vanilla fine-tuning baselines across complex reasoning tasks. Diverse reasoning generally improves performance, while scaling effects vary across model families and some task-specific limitations remain.

  • Baseline comparisons: Fine-tune-CoT enables notable reasoning in small models, whereas Few-shot-CoT performs inconsequentially across most student models.The comparison includes standard prompting, Few-shot-CoT, vanilla fine-tuning, and Fine-tune-CoT.
  • Baseline comparisons: 44.00%→89.33% on Tracking Shuffled Objects for Flan-T5 and 11.67%→19.44% on MultiArith for GPT-2 show gains from diverse reasoning over vanilla fine-tuning.The reported comparisons are specifically for Fine-tune-CoT enhanced with diverse reasoning versus vanilla fine-tuning.
  • Diverse reasoning: Performance scales with diverse reasoning in all student models except T5-Small, and diverse reasoning outperforms standard fine-tuning in all reported cases.Figure 9 examines MultiArith and SVAMP under varying degrees of diverse reasoning.
  • Student model scale: Fine-tune-CoT performance scales with GPT-3 model size but does not show the same pattern in other open-source models.The authors attribute this difference to potentially under-tuned fixed hyperparameters used for the open-source models.
  • Evaluation setup: Fine-tune-CoT is evaluated on open-source student models across 12 tasks, with accuracy compared against baseline methods.Table 9 reports maximum diverse-reasoning degrees of D = 64 for MultiArith and SVAMP and D = 8 for other datasets.
Loading 2212.10071v2…