PaperlayerView exact revision on arXiv

Source-linked AI summary

MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation

Yangsong Lan, Renkai Hu, HongKai Zheng, Bo Zhang, Renzhi Wang, Hongliang Dai, Piji Li

arXiv:2608.29623v1cs.CLcs.AI

TL;DR

Small models often gain less from direct Long CoT distillation than from concise rationales. MI-Distillation builds an interpolated Instruct-Reasoning spectrum and uses SeqLSS to select student-learnable trajectories, improving performance over strong baselines.

  • Problem

    Direct Long CoT supervision often fails to benefit small students, motivating reasoning data matched to student capacity and distribution.

  • Method

    MI-Distillation interpolates instruction- and reasoning-oriented models to create a controllable data spectrum, while SeqLSS selects informative, student-aligned trajectories.

  • Results

    1.12 and 1.40 points: MI-Distillation outperforms the strongest baseline on Qwen2.5-3B-Instruct and Llama3.2-3B-Instruct, respectively.

  • Takeaways & Limitations

    Effective CoT distillation requires balancing rationale informativeness with student learnability rather than assuming longer reasoning is better.

  • Takeaways & Limitations

    Experiments mainly cover mathematical reasoning benchmarks, leaving applicability to open-ended, code, multilingual, and weakly verifiable tasks unresolved.

Abstract

from arXiv · show

Recent advances in large reasoning models (LRMs) have shown strong performance on complex problems through long chain-of-thought (Long CoT) reasoning. However, distilling such trajectories into smaller student models remains challenging: direct Long CoT supervision often provides limited gains and can be less effective than concise Short CoT rationales. In this work, we investigate this phenomenon from a gradient-centric perspective. Our analysis shows that Long CoT induces larger gradient magnitudes and more concentrated update directions than Short CoT, with this effect becoming more pronounced as student model capacity increases. These findings suggest that effective Long CoT distillation requires balancing the reasoning information density of reasoning trajectories with their distributional alignment to the student model. Motivated by this insight, we propose \textbf{M}odel \textbf{I}nterporlation \textbf{Distillation} (\textbf{MI-Distillation}), a framework that constructs a continuous Instruct-Reasoning data spectrum through model interpolation. To select suitable trajectories from this spectrum, we further introduce \textbf{Seq}uential \textbf{L}earnable \textbf{S}urprisal \textbf{S}core (\textbf{SeqLSS}), which favors reasoning paths that are both informative and learnable for the student. Extensive experiments on reasoning benchmarks show that MI-Distillation consistently improves small model CoT distillation over strong Long CoT baselines.

1 Introduction

The introduction identifies a scale-dependent challenge in distilling Long CoT into small models and motivates MI-Distillation, which constructs a continuous Instruct-Reasoning spectrum and selects student-aligned trajectories. Gradient analysis explains why Long CoT becomes more suitable as student capacity increases, while experiments show consistent improvements over strong CoT distillation baselines.

  • Scale-dependent distillation: Short CoT supervises smaller Qwen2.5 students better, whereas Long CoT marginally overtakes it at 7B on MATH-500.The comparison covers 1.5B, 3B, and 7B students.
  • Motivation: Transferring LRM reasoning to smaller models is necessary because the computational cost of LRMs limits deployment in resource-constrained scenarios.Long CoT trajectories have emerged as supervision sources for supervised fine-tuning and cold-start reinforcement learning.
  • Gradient analysis: Long CoT induces larger, more concentrated gradients than Short CoT, with the difference increasing as student model scale grows.This suggests Long CoT is effective when students have sufficient capacity to convert verbose trajectories into coherent, low-dimensional parameter updates.
  • Contributions: MI-Distillation constructs a fine-grained Instruct-Reasoning data spectrum through model interpolation and introduces SeqLSS to select student-aligned reasoning trajectories.Interpolating instruction-oriented and reasoning-oriented models produces trajectories with varying length, correctness, and reasoning depth, controlled by coefficient λ.
  • Contributions: MI-Distillation consistently improves small-model reasoning and surpasses strong CoT distillation baselines on challenging reasoning benchmarks.

2 Related Work

Related work establishes CoT as a mechanism for improving LLM reasoning and knowledge distillation as a way to transfer capabilities to smaller students. It also shows that Long CoT can be difficult for small models, motivating more adaptive supervision than fixed mixtures or curricula.

  • CoT improves language-model reasoning, while early distillation commonly uses concise Short CoT rationales and RLVR enables Long CoT trajectories (Wang et al.; Jaech et al., 2024; Shao et al., 2024; Yu et al., 2025).
  • Directly distilling Long CoT into small students does not consistently improve performance, which often favors shorter, more concise rationales (Li et al., 2025; Kim et al., 2025; Jiang et al., 2025).Long trajectories can increase imitation difficulty when teacher rationales mismatch the student’s learning capacity and intrinsic distribution.
  • Existing remedies mix Short and Long CoT data or use curricula that gradually increase supervision complexity, but depend largely on manually designed mixtures or schedules (Jiang et al., 2025; Li et al., 2025).
  • MI-Distillation addresses limited control over suitable reasoning traces by analyzing scale-dependent Short-versus-Long CoT effects and constructing a continuous model-interpolated reasoning-data spectrum.The spectrum enables more adaptive CoT-trajectory selection for small-model distillation.

3 Gradient Signatures of Short and Long CoT Distillation

Gradient analysis shows that Long CoT produces stronger and more concentrated optimization signals than Short CoT, especially for larger students. This motivates balancing trajectory information density with the student’s distributional and optimization capacity.

  • Gradient signatures: Gradients of Query, Key, Value, and Output attention projections are characterized with SVD using nuclear norm and effective rank, following spectral analysis from Li et al. (2024a) and Carlini et al. (2024).For gradient matrix G, singular values define the normalized spectral distribution used in the analysis.
  • Gradient signatures: Nuclear norm measures total gradient spectral energy, whereas effective rank captures spectral dimensionality; lower effective rank indicates updates concentrated in fewer principal directions.The analysis uses Shannon entropy over normalized singular values to quantify effective rank and characterize structurally concentrated optimization trajectories.
  • Experimental setup: The study compares Short and Long CoT distilled on MATH-500 using Qwen2.5 students at 1.5B, 3B, and 7B scales.Short CoT trajectories come from Qwen2.5-32B-Instruct (Yang et al., 2024), while Long CoT trajectories come from QwQ-32B (Team, 2025).
  • Gradient comparison: Long CoT consistently induces larger nuclear norms than Short CoT across student scales, with the gap widening especially for 3B and 7B models.This indicates stronger optimization signals and richer gradient information, while also imposing a more demanding learning target that larger students can absorb more effectively.

4 Synthesizing Instruct-Reasoning Spectrum via Model Interpolation

MI-Distillation constructs a continuous Instruct-Reasoning teacher spectrum by interpolating reasoning-oriented and instruction-oriented models. Varying the interpolation coefficient smoothly balances reasoning depth and compactness, and this spectrum transfers to distilled students.

  • Method: Interpolation between reasoning-oriented and instruction-oriented teachers creates a continuous spectrum that balances reasoning compactness and depth.Unlike discrete Short/Long CoT mixtures, varying λ provides finer control over the reasoning signal.
  • Method: The interpolation can be interpreted as task arithmetic over reasoning and instruction task vectors, independently of the reference model.For any base model, the task vectors are defined as τ_Thi = Θ_Thi − Θ_Base and τ_Ins = Θ_Ins − Θ_Base.
  • Results and Analysis: Across 14B and 32B teacher families, changing the interpolation coefficient produces a trade-off between reasoning correctness and depth, with CoT length changing smoothly.The interpolated teachers therefore form an approximate spectrum of reasoning behaviors.
  • Results and Analysis: Distilling trajectories from 32B interpolated teachers transfers the spectrum to Qwen and Llama students, whose reasoning length also varies smoothly with the interpolation coefficient.The evaluation uses Qwen2.5-3B-Instruct and Llama3.2-3B-Instruct students; endpoint teachers within the Qwen2.5-32B family help control for pretraining and world-knowledge differences.

5 MI-Distillation: SeqLSS-Guided CoT Selection from Interpolated Teachers

MI-Distillation builds a continuous instruct–reasoning spectrum by interpolating instruction- and reasoning-oriented teachers, then uses SeqLSS to select correct rationales that balance informativeness with student learnability. The selected question–rationale pairs are distilled into a small student model.

  • Interpolated Teacher Spectrum: MI-Distillation interpolates instruction- and reasoning-oriented teachers at coefficients Λ = {0.2, 0.4, 0.6, 0.8, 1.0}, producing trajectories with different reasoning depth, compactness, and instruction-oriented behavior.This spectrum avoids committing to a single teacher or fixed Short/Long CoT style.
  • Accuracy Selection: The framework first filters candidate rationales by correctness, retaining only trajectories whose final predictions match the ground-truth answers.This prevents erroneous chains of thought from providing misleading supervision.
  • SeqLSS Selection: SeqLSS selects rationales that combine informative reasoning signals with alignment to the student’s predictive distribution.It penalizes tokens outside the student’s high-probability region and aggregates token-level scores with surprisal-based normalization to reduce instability from extreme tokens.
  • SeqLSS Selection: For each problem, MI-Distillation chooses the correct candidate rationale with the highest SeqLSS, estimating the proportion of informative reasoning within the student’s learnable region.The selected question–rationale pairs are then used to distill a small student model.

6 Experiments

Across two 3B student models and diverse reasoning benchmarks, MI-Distillation achieves the strongest overall performance and improves over fixed-coefficient interpolation through adaptive trajectory selection. Its learnability-aware selection also produces more stable optimization, supporting better alignment between reasoning complexity and student capacity.

  • Experiment Details: Experiments use Qwen2.5-3B-Instruct and Llama3.2-3B-Instruct students, with QwQ-32B as the Long CoT teacher and Qwen2.5-32B-Instruct as the other teacher.Students are fine-tuned for 3 epochs with global batch size 32 and peak learning rate 1 × 10−5, then evaluated using Pass@1 across reasoning benchmarks of varying difficulty.
  • Main Results: MI-Distillation achieves the best average performance on Qwen2.5-3B-Instruct and Llama3.2-3B-Instruct, outperforming the strongest baseline by 1.12 and 1.40 points, respectively.It also gains on challenging benchmarks including AMC23, MATH-500, and OlympiadBench, while Long CoT often underperforms Short CoT.
  • Comparison with Fixed Interpolation Coefficients: MI-Distillation outperforms the strongest fixed-λ baseline by 0.19, 2.70, and 1.09 points on GSM8K, MATH-500, and AMC23, respectively.No single coefficient dominates: λ = 0.8 is strongest on GSM8K, while λ = 0.4 and λ = 0.6 perform better on MATH-500 and AMC23.
  • Training Dynamics under Different Learnability Penalties: MI-Distillation shows substantially lower and more stable training loss than Long CoT and Mix Long across both student backbones.Increasing the learnability penalty coefficient α generally smooths optimization and lowers loss, especially for Llama-3.2-3B-Instruct.

7 Conclusion

The conclusion frames Long CoT distillation for small students through gradient behavior, showing that effective reasoning transfer depends on balancing rationale informativeness with student learnability.

  • The work investigates Long CoT distillation for small student models from a gradient-centric perspective.
  • Long CoT supervision produces larger and more concentrated gradient updates than Short CoT, increasingly so as model capacity grows.
  • These findings motivate balancing rationale informativeness with student learnability and lead to the proposed MI-Distillation framework.

Limitations … C.1 Hardware Infrastructure

The paper identifies scope, scalability, and faithfulness limitations while formally defining the distillation objective, gradient analyses, experimental setup, and hardware infrastructure. Together, these sections frame MI-Distillation as student-aware reasoning supervision evaluated primarily on mathematical reasoning benchmarks.

  • Limitations: MI-Distillation is mainly evaluated on mathematical reasoning benchmarks, leaving open-ended reasoning, code generation, multilingual tasks, and weak-verification domains for future work.SeqLSS also requires scoring multiple candidate rationales before training, creating scalability costs for larger corpora and broader teacher spectra.
  • Limitations: SeqLSS estimates learnability from token-level likelihoods, which may miss logical faithfulness, conciseness, robustness, and step-level correctness.Future extensions could combine learnability-aware selection with process-level verification, step-wise feedback, or stronger rationale-quality assessment.
  • A Detailed Problem Setup: The setup distills teacher-generated reasoning trajectories containing intermediate steps and final predictions into compact students as process-level supervision.More detailed rationales can provide richer information but also increase learning difficulty, motivating student-aware trajectory selection.
  • B.1 Supervised Fine-Tuning Objective: Student fine-tuning minimizes normalized token-level negative log-likelihood over target sequences containing the CoT rationale followed by the final answer.The student models are parameterized by θ and trained on inputs x with target sequences y.
  • B.2 Layer-wise Gradient Extraction: The gradient analysis extracts query, key, value, and output projection gradients layer by layer from the student’s attention modules.Each projection gradient is represented as a matrix GX,i for spectral analysis.
  • B.3 SVD-based Gradient Metrics: SVD-based metrics summarize gradient spectra through nuclear norm for update magnitude and effective rank for spectral concentration.Together, these metrics compare Short CoT and Long CoT supervision in update strength and dimensional structure.
  • C Experimental Setup Details: Experiments instantiate 32B and 14B Instruct–Reasoning teacher spectra and distill them into Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct students.The corpus samples 7,500 MATH training problems and generates candidate trajectories with temperature T = 0.6 and nucleus parameter p = 0.95.
  • C.1 Hardware Infrastructure: The infrastructure uses separate servers with 8 NVIDIA L20 GPUs and 8 NVIDIA RTX A5000 GPUs, and implements models and pipelines in PyTorch.Training uses LLaMAFactory with DeepSpeed ZeRO-3 Offloading for memory efficiency and throughput.

C.2 Inference and Evaluation … D.2 Comparison between Direct Interpolation and TIES-Merging

Across controlled benchmark evaluations, MI-Distillation’s interpolated teachers form a generally progressive Instruct–Reasoning spectrum, while direct interpolation remains simple, competitive, and behaviorally effective relative to TIES-Merging. Results use repeated pass@1 evaluation with matched baseline construction and teacher-generation settings.

  • C.2 Inference and Evaluation: Models are evaluated with the teacher-generation hyperparameters, using vLLM, temperature T = 0.6, top-p = 0.95, and an 8,192-token maximum sequence length.The same generation configuration is applied during inference to maintain consistency with teacher trajectory generation.
  • C.2 Inference and Evaluation: Evaluation uses pass@1 accuracy, with 16 independent runs for AIME 2024 and AMC 2023 and 4 runs for GSM8K, MATH-500, GPQA-Diamond, Minerva, and OlympiadBench.Reported results are means across runs with corresponding standard deviations, improving reliability across different test-set sizes.
  • C.3 Evaluation Datasets: The study evaluates mathematical reasoning across standard benchmarks, including GSM8K, MATH-500, GPQA-Diamond, Minerva, OlympiadBench, AIME 2024, and AMC 2023.These benchmarks span larger-scale and limited-test-set settings, which motivates the differing numbers of independent evaluation runs.
  • C.3 Evaluation Datasets: Baseline comparisons control the question set, answer filtering, prompt formatting, student architecture, and optimization hyperparameters, so baselines differ from MI-Distillation primarily in trajectory-selection strategy.Mix Long (Li et al., 2025) uses a fixed 4:1 Short-/Long-CoT mixture, Mix Large (Li et al., 2025) adds teacher-scale diversity, and Curriculum Learning uses a two-stage reasoning-format schedule.
  • D.1 Evaluation of Interpolated Teachers on MATH-500: As the interpolation coefficient increases, interpolated teachers generally become more accurate and produce longer reasoning trajectories, transitioning progressively from concise instruction following toward elaborate reasoning.The spectrum is clearest as problem difficulty increases, though it is not perfectly monotonic across every difficulty level because of endpoint differences in formatting, chat templates, and RoPE scaling.
  • D.2 Comparison between Direct Interpolation and TIES-Merging: Direct interpolation constructs the Instruct–Reasoning spectrum by combining endpoint models, while TIES-Merging is tested under matched ratios with QwQ-32B and Qwen2.5-32B-Instruct on five reasoning benchmarks.The comparison directly tests whether MI-Distillation’s findings depend on the specific model-merging method.
  • D.2 Comparison between Direct Interpolation and TIES-Merging: Both direct interpolation and TIES-Merging preserve the interpolated teachers’ overall spectrum behavior, although TIES-Merging is better at λ ∈ {0.6, 0.8} and worse at λ ∈ {0.2, 0.4}.This supports direct interpolation as a simple and competitive merging strategy rather than making MI-Distillation dependent on naive linear interpolation.

D.3 Full Student Distillation Results with Fixed Interpolation Coefficients

Fixed-coefficient distillation shows that intermediate points on the Instruct-Reasoning teacher spectrum generally outperform pure instruct and pure reasoning endpoints, with the best coefficient varying by student family.

  • D.3 Full Student Distillation Results with Fixed Interpolation Coefficients: The experiment evaluates how fixed points on the Instruct-Reasoning teacher spectrum affect downstream student performance.λ = 1.0 denotes the pure instruct teacher, while λ = 0.0 denotes the pure reasoning teacher.
  • D.3 Full Student Distillation Results with Fixed Interpolation Coefficients: Intermediate interpolation coefficients generally outperform both endpoints across Qwen2.5-3B-Instruct and Llama3.2-3B-Instruct.The best average performance occurs at λ = 0.8 for Qwen2.5-3B-Instruct and λ = 0.6 for Llama3.2-3B-Instruct.

D.4 Embedding-space analysis of the interpolated CoT spectrum.

Embedding-space analysis shows that model interpolation produces a continuous transition between Long-CoT and Short-CoT supervision. As λ changes, trajectories shift progressively, with neighboring coefficients remaining closer than endpoints.

  • D.4 Embedding-space analysis of the interpolated CoT spectrum.: Figure 5 shows a clear, progressive shift in CoT embeddings as λ moves from the reasoning-oriented to instruction-oriented endpoint.This pattern appears for both Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct.
  • D.4 Embedding-space analysis of the interpolated CoT spectrum.: For both models, neighboring interpolation coefficients occupy adjacent PCA regions, whereas the endpoints are more clearly separated.
  • D.4 Embedding-space analysis of the interpolated CoT spectrum.: Pairwise cosine-distance matrices further support the spectrum structure, with trajectories from nearby coefficients showing smaller distances.

D.5 Ablation on the Learnability Penalty Coefficient · E Examples of Model Outputs with Different Interpolation Weights

SeqLSS favors α = 4 as the most reliable balance between informative trajectories and student learnability across two 3B students. Model-output examples solve the same bacteria problem with varied reasoning formats, consistently reaching 640.

  • D.5 Ablation on the Learnability Penalty Coefficient: α = 4 achieves the best average performance across Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct, balancing informative CoT selection with student alignment.Table 9 evaluates SeqLSS under different α values on both 3B students.
  • D.5 Ablation on the Learnability Penalty Coefficient: Different α values produce non-monotonic trade-offs across the interpolation spectrum rather than a uniformly improving trend.Smaller coefficients remain competitive on some individual benchmarks, so the preferred regularization strength depends partly on task and model.
  • D.5 Ablation on the Learnability Penalty Coefficient: α = 4 provides the most reliable overall behavior across the two student families and is therefore used as the default in the main experiments.The default reflects aggregate reliability rather than dominance on every individual benchmark.
  • E Examples of Model Outputs with Different Interpolation Weights: The examples use the same bacteria-growth question, requiring the population after 15 minutes when 20 bacteria double every 3 minutes.The supplied outputs repeatedly identify 15 minutes and five doubling periods.
  • E Examples of Model Outputs with Different Interpolation Weights: One output reasons conversationally by dividing 15 by 3, counting five doublings, and multiplying 20 by 32.Its intermediate sequence is 20, 40, 80, 160, 320, 640.
  • E Examples of Model Outputs with Different Interpolation Weights: Other outputs use exponential-growth formulas or explicit numbered steps, but they also compute 20 × 2^5 = 640.The formula-based and stepwise variants both report the same final population.
  • E Examples of Model Outputs with Different Interpolation Weights: Across the shown output variants, the final answer is 640 bacteria at 3:15 p.m.The examples differ in presentation, including conversational checking, formula substitution, and enumerated calculations.
Loading 2608.29623v1…