Source-linked AI summary
Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning
Shaotian Yan, Kaiyuan Liu, Chen Shen, Bing Wang, Sinan Fan, Jun Zhang, Yue Wu, Zheng Wang, Jieping Ye
TL;DR
The paper addresses limitations of SFT-based sequence-level distillation, including incomplete teacher-distribution coverage, student–teacher misalignment, and exposure bias. It proposes a pipeline combining temperature-scheduled learning, divergence-aware sampling, and mixed-policy distillation, yielding strong comparable-scale benchmark performance with 448K samples.
Problem
Existing sequence-level distillation methods inadequately cover the teacher’s sequence-level distribution, misalign it with student learning capacity, and expose students to teacher-forced inputs unlike autoregressive inference.
Method
The pipeline combines temperature-scheduled learning, divergence-aware sampling, and mixed-policy distillation with explicit teacher–student interaction.
Results
DASD-4B-Thinking achieves state-of-the-art performance among comparable-scale models across mathematics, code generation, and scientific reasoning, using only 448K training samples.
Takeaways & Limitations
The framework transfers complex reasoning capabilities from a large teacher to a lightweight 4B-parameter student and supports distillation across heterogeneous model families.
Takeaways & Limitations
Teacher-forced training can cause exposure bias, distribution shift, error accumulation, and student outputs that diverge from the training distribution.
Abstract
from arXiv · showhide
In this report, we introduce DASD-4B-Thinking, a lightweight yet highly capable, fully open-source reasoning model. It achieves SOTA performance among open-source models of comparable scale across challenging benchmarks in mathematics, scientific reasoning, and code generation -- even outperforming several larger models. We begin by critically reexamining a widely adopted distillation paradigm in the community: SFT on teacher-generated responses, also known as sequence-level distillation. Although a series of recent works following this scheme have demonstrated remarkable efficiency and strong empirical performance, they are primarily grounded in the SFT perspective. Consequently, these approaches focus predominantly on designing heuristic rules for SFT data filtering, while largely overlooking the core principle of distillation itself -- enabling the student model to learn the teacher's full output distribution so as to inherit its generalization capability. Specifically, we identify three critical limitations in current practice: i) Inadequate representation of the teacher's sequence-level distribution; ii) Misalignment between the teacher's output distribution and the student's learning capacity; and iii) Exposure bias arising from teacher-forced training versus autoregressive inference. In summary, these shortcomings reflect a systemic absence of explicit teacher-student interaction throughout the distillation process, leaving the essence of distillation underexploited. To address these issues, we propose several methodological innovations that collectively form an enhanced sequence-level distillation training pipeline. Remarkably, DASD-4B-Thinking obtains competitive results using only 448K training samples -- an order of magnitude fewer than those employed by most existing open-source efforts. To support community research, we publicly release our models and the training dataset.
1 Introduction
The report reexamines sequence-level distillation based on SFT over teacher responses, identifying gaps in distribution coverage, student alignment, and training–inference consistency. It introduces a teacher–student-interactive pipeline whose components support DASD-4B-Thinking’s strong multi-domain results with limited data.
- Motivation: SFT on teacher-generated responses is efficient and architecture-flexible, but existing work emphasizes heuristic data filtering over learning the teacher’s full output distribution.The approach does not require token-level logits or arbitrary constraints on teacher and student architectures.
- Limitations: Randomly sampled, quality-filtered responses may provide poor coverage of the teacher’s sequence-level distribution and overrepresent low-probability or noisy sequences.These issues can make learning particularly difficult for smaller or less capable students.
- Limitations: SFT can produce misleading gradients because it increases ground-truth token likelihood without matching the teacher’s full predictive distribution.For example, it can further increase probabilities for tokens that the teacher assigns low probability but the student initially assigns high probability.
- Limitations: Teacher-forced training creates exposure bias because inference relies on the student’s autoregressive predictions, causing distribution shift, error accumulation, and possible response divergence.Observed divergences include response length and unexpected states that result in incorrect answers.
- Method: The proposed pipeline combines temperature-scheduled learning, divergence-aware sampling, and mixed-policy distillation to improve distribution coverage, student learning, and robustness to self-generated errors.It uses low-temperature cross-domain sampling before larger, more diverse higher-temperature training and adds student-generated prefixes with teacher continuations.
- Results: 88.5 on AIME24, 83.3 on AIME25, 69.3 on LiveCodeBench v5, and 68.4 on GPQA-Diamond are reported using 448K training samples.The model achieves state-of-the-art performance among comparable-scale models across mathematics, code generation, and scientific reasoning, including outperforming some 32B-scale models.
2 Preliminaries
Sequence-level distillation trains a student to match the teacher’s distribution over complete responses, but exact computation is intractable and practical sampling approximations recover standard SFT. Existing methods often underrepresent that distribution, motivating greater teacher–student interaction.
- Sequence-level distillation minimizes divergence between the teacher’s and student’s distributions over complete responses rather than next-token distributions.The response space Y contains all responses the teacher can generate for prompt x.
- Unlike logit distillation, sequence-level distillation aligns entire output sequences while pursuing the same goal of transferring the teacher’s generalization-relevant distributional information.
- Exact sequence-level divergence is intractable because the response space grows exponentially, so practical methods approximate it with sampled responses.
- Replacing the teacher distribution with a point mass at one sampled response reduces the objective to standard SFT loss on teacher-generated outputs.
- Current methods prioritize filtering teacher outputs while neglecting teacher–student interaction, causing inadequate coverage of the teacher’s sequence-level distribution among their identified limitations.
3 Temperature-scheduled Learning
Temperature changes trade off learnability against coverage of the teacher’s response modes: higher temperatures broaden the sampled distribution but make training harder. Temperature-scheduled learning exploits both regimes, and experiments report stronger benchmark gains than temperature-only baselines in several settings.
- Empirical motivation: T=0.6 produces a sharper, narrower distribution of high-likelihood responses, whereas T=1.0 covers a broader probability range with greater diversity and rarer modes.Figure 3 characterizes response likelihood using the geometric mean of token-level probabilities and also compares SFT loss curves.
- Empirical motivation: +1.4 on AIME24 and +4.2 on AIME25 are achieved by T=1.0 samples versus T=0.6 samples, despite slower convergence and harder optimization.
- Empirical motivation: Doubling the dataset to 100K samples yields no AIME24 gain and only +2.8 on AIME25 relative to the 50K T=1.0 setup, indicating a student-capacity bottleneck.
- Method: Temperature-scheduled learning starts with concentrated, easier-to-learn low-temperature modes before switching to higher-temperature samples that capture rarer modes and richer information.
- Cross-model validation: +4.1 on AIME24 and +1.8 on AIME25 result when Qwen3-Next-80B-A3B-Thinking training continues from T=0.6 samples with additional T=1.0 samples.This scheduled setup also surpasses T=1.0-only training in that experiment.
4 Divergence-aware Sampling
Divergence-aware Sampling decomposes sequence-level responses into sentence types using teacher, student, and distilled-model probabilities, then identifies distributions that better support student learning. Experiments report consistent advantages over random sampling across settings, teachers, and domains.
- Distribution decomposition: The framework computes sentence-level probabilities under teacher, pre-distillation student, and distilled models to decompose sequence-level responses into four distribution types.Responses are segmented into sentences, and each sentence probability is computed as the geometric mean of its per-token probabilities.
- Distribution decomposition: Student and Teacher Sentences capture cases where large teacher–student probability discrepancies persist in the distilled model’s output.A Teacher Sentence is primarily originated from the teacher, while a Student Sentence is primarily originated from the student.
- Distribution decomposition: Boosted Sentences are present in both teacher and student models before distillation but receive amplified probabilities after training on distilled data.These sentences have similar teacher and student probabilities but substantially different distilled-model probabilities.
- Learning analysis: The analysis evaluates each sentence type by correlating its position-wise probability with test-set answer correctness.Sentence-position analysis helps reduce confounding from differing numbers of sentences in correct and incorrect solutions.
- Empirical findings: DAS consistently outperforms random sampling under identical budgets and retains advantages across teacher models and domains.The method can also generalize data curated for Qwen3-4B-Instruct-2507 to Qwen3-30B-A3B-Instruct-2507 without resampling.
5 Mixed-policy Distillation
Mixed-policy distillation addresses exposure bias by combining student-generated on-policy contexts with teacher-completed guidance. The reported ablation finds that a small mixed-policy dataset improves performance, while masking student-generated portions performs worse.
- Motivation: Exposure bias arises because training uses teacher-forced prefixes whereas inference relies on the student’s autoregressive predictions.This mismatch can produce distribution shift and error accumulation over time.
- Method: Mixed-policy distillation regenerates responses with the student, identifies substantial deviations from teacher outputs, and has the teacher continue truncated student solutions.Teacher continuation provides targeted guidance on student errors.
- Ablation: 7.7K mixed-policy samples enhanced performance in the reported ablation, whereas masking student-generated portions tended to worsen performance.The ablation trains for one epoch and adds 20K off-policy samples to balance the training mixture.
6 Overall Training Recipe
The overall recipe combines question collection, candidate sampling, quality filtering, and multi-stage training for DASD-4B-Thinking. It uses diverse reasoning domains, temperature variation, divergence-aware selection, and mixed-policy refinement.
- Pipeline: The training pipeline consists of question collection, candidate response sampling, response filtering, and multi-stage training on curated data.These stages form the concrete implementation of DASD-4B-Thinking.
- Question collection: Questions span mathematical reasoning, code generation, scientific reasoning, and instruction following.The paper draws these questions from several public datasets.
- Model pair: Qwen3-4B-Instruct-2507 and gpt-oss-120b serve as the primary student–teacher pair.The pair is selected to highlight compatibility across model families and architectures.
- Sampling: Candidate responses are sampled at low and high temperatures and prioritized with divergence-aware sampling.The goal is to broaden coverage of teacher behavior while supporting student learning and preserving correct gradient directions.
- Filtering: Filtering removes overlong, function-calling, structurally incomplete, and repetitive responses.The filters target context limits, the long-CoT focus, required reasoning and final-answer structure, and repetitive generations.
- Data construction: 105K low-temperature responses and 330K high-temperature responses are collected before filtering and training.The resulting domain distribution is presented in Figure 8.
- Training: Temperature-scheduled learning is followed by mixed-policy distillation in the two-stage training pipeline.The implementation applies low-temperature data before high-temperature data and then uses student-generated responses for targeted refinement.
7 Experimental Evaluation
DASD-4B-Thinking achieves strong performance across mathematics, coding, and scientific reasoning benchmarks, often exceeding comparable and larger models. Ablations indicate complementary benefits from staged temperature scheduling and mixed-policy distillation.
- DASD-4B-Thinking achieves state-of-the-art performance for its scale across mathematical, coding, and scientific reasoning benchmarks, including models substantially larger than 4B parameters.Figure 9 evaluates performance against model size on AIME25 and LCB v5, with top-left positions representing higher performance at smaller scale.
- Mathematical Reasoning (AIME24, AIME25): 88.5 on AIME24 and 83.3 on AIME25 establish top-tier mathematical reasoning performance among the listed models.The report states that these scores achieve state-of-the-art performance among all listed models and compare favorably with larger-scale systems.
- 448K training samples support results that exceed larger open-data and open-weights baselines, including models trained on 2.9M and 30M examples.Against AM-thinking-v1, DASD-4B-Thinking scores 83.3 versus 74.4 on AIME25 and 88.5 versus 85.3 on AIME24 using roughly six times fewer examples.
- Coding (LiveCodeBench): 69.3 on LiveCodeBench v5 and 67.5 on v6 surpass several 7B–32B baselines, including Qwen3-32B on v6.The reported comparisons include 69.3 versus 60.5 for DeepSeek-R1-0528-Qwen3-8B on v5 and 67.5 versus 65.7 for Qwen3-32B on v6.
- Scientific QA (GPQA-D): 68.4 on GPQA-D matches Qwen3-32B and narrows the gap to substantially larger scientific-reasoning models.The report highlights GPQA-D as challenging for compact models because it relies heavily on parametric knowledge.
- Low-temperature training raises AIME25 from 47.4% to 74.0% and LCB v6 from 35.1% to 50.6%, while high-temperature training adds further gains.High-temperature training contributes +11.8% on LCB v5, +16.6% on LCB v6, and +9.0% on AIME25 after the low-temperature stage.
- Mixed-policy distillation improves all reported benchmarks on top of an already strong model, supporting its role in addressing exposure bias with minimal training overhead.The reported gains range from +0.3% on AIME25 and LCB v6 to +0.9% on LCB v5.
8 Conclusion and Future Work
The report concludes that DASD-4B-Thinking combines three distillation innovations to achieve strong reasoning performance with 448K samples. Future work targets distribution-aware reweighting, more efficient mixed-policy distillation, and complementary agentic capabilities.
- DASD-4B-Thinking addresses sequence-level distillation limitations through temperature-scheduled learning, divergence-aware sampling, and mixed-policy distillation.
- Using 448K samples, DASD-4B-Thinking achieves state-of-the-art performance across most reasoning benchmarks and surpasses several larger models.
- Future work will explore teacher-probability-based distribution-aware reweighting during SFT to improve distillation effectiveness and data efficiency.
- The authors also plan to refine mixed-policy distillation for greater training efficiency and stability and integrate complementary agentic capabilities.