Source-linked AI summary
How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data
Zixian Huang, Kaichen Yang, Xu Huang, Feiyang Hao, Qiming Ge, Bowen Li, He Du, Kai Chen, Qipeng Guo
TL;DR
Teacher-generated SFT data can fail for reasoning models when stylistic and distributional differences conflict with the student’s training distribution. TESSY alternates teacher and student generation to preserve teacher reasoning while matching student style, and it improves Qwen3-8B code-generation performance where teacher-only SFT causes substantial drops.
Problem
SFT with stronger-model synthetic data can degrade reasoning-model performance when teacher data differs stylistically and distributionally from the student’s training data.
Method
TESSY alternately uses teacher and student models to generate capability-related and stylistic tokens, with generate-then-rollback control over their responsibilities.
Results
TESSY improved Qwen3-8B by 11.25% on LiveCodeBench-Pro and 6.68% on OJBench, while teacher-only SFT caused drops of 3.25% and 10.02%, respectively.
Takeaways & Limitations
Involving the student in synthesis preserves teacher reasoning information while aligning the synthesized data distribution with the student’s inherent style.
Takeaways & Limitations
Teacher-Only slightly outperforms TESSY when generation length is unrestricted, indicating a higher upper bound for its generation capability.
Abstract
from arXiv · showhide
A widely adopted strategy for model enhancement is to use synthetic data generated by a stronger model for supervised fine-tuning (SFT). However, for emerging reasoning models like Qwen3-8B, this approach often fails to improve reasoning capabilities and can even lead to a substantial drop in performance. In this work, we identify substantial stylistic divergence between teacher generated data and the distribution of student as a major factor impacting SFT. To bridge this gap, we propose a Teacher-Student Cooperation Data Synthesis framework (TESSY), which interleaves teacher and student models to alternately generate style and non-style tokens. Consequently, TESSY produces synthetic sequences that inherit the advanced reasoning capabilities of the teacher while maintaining stylistic consistency with the distribution of the student. In experiments on code generation using GPT-OSS-120B as the teacher, fine-tuning Qwen3-8B on teacher-generated data leads to performance drops of 3.25% on LiveCodeBench-Pro and 10.02% on OJBench, whereas TESSY achieves improvements of 11.25% and 6.68%.
1. Introduction
Reasoning-model SFT can be harmed by distributional and stylistic mismatches between teacher-generated data and the student. TESSY addresses this by assigning reasoning content to the teacher and stylistic content to the student, improving code-generation performance where teacher-only SFT degrades it.
- Reasoning models generate explicit thinking content for fine-grained reasoning alongside final-answer content, enabling breakthroughs on reasoning-intensive tasks such as code generation.
- Large models commonly generate SFT data for smaller models, but this strategy has been studied less extensively for reasoning models than for Base or Instruct models.
- Higher-quality synthetic data can trigger catastrophic forgetting when its undisclosed distribution conflicts with the proprietary data used to fine-tune open-source reasoning models.
- Teacher-only data can impose unnecessary stylistic adaptations on the student because GPT-OSS-120B and Qwen3-8B differ in their reasoning-response styles.
- TESSY alternates teacher and student generation so the teacher supplies solution-relevant reasoning while the student supplies stylistic text, using generate-then-rollback control.
- 3.25% and 10.02% drops under teacher-only SFT on LiveCodeBench-Pro and OJBench became 11.25% and 6.68% improvements with TESSY.
2. Approach
TESSY addresses stylistic mismatch in teacher-generated SFT data by separating capability and style tokens, then alternately generating and boundary-truncating spans with teacher and student models.
- SFT aligns the student’s output distribution with the training-data distribution, which can be synthesized from a stronger teacher model.
- Capability tokens encode task-solving content, whereas style tokens express task-irrelevant tone or discourse.Examples of capability tokens include code or numerical tokens; style tokens include connective words such as “wait” and “but”.
- For reasoning models, teacher–student stylistic discrepancies make direct style-loss optimization difficult and can adversely affect capability learning.The issue is especially pronounced when teacher and student models come from different sources.
- TESSY constructs data by sampling capability tokens from the teacher while encouraging style tokens to follow the student distribution.The framework alternates teacher and student generation to preserve reasoning capability and student-consistent style.
- Alternating Generation: TESSY begins with a student-generated span, alternates generation roles across spans, and uses prior spans as context for each subsequent span.The student starts because model outputs often begin with stylistic phrases such as “Okay, let’s see,” while spans s_i and t_i are generated by the student and teacher, respectively.
- Generation Rollback: Boundary predictors truncate fixed-length generated spans so student spans contain style tokens and teacher spans contain capability tokens.The teacher predictor locates the last capability token, while the student predictor locates the last style token.
- Generation Rollback: Algorithm 1 initializes generation with the student, generates spans up to k tokens, truncates them, and switches models when truncation indicates a token-type change.The truncated spans are appended iteratively until the final answer is generated by the student model.
3. Experiments
The experiments evaluate TESSY across data-construction settings, datasets, and model configurations. TESSY consistently improves Qwen3-8B on the reported code-generation benchmarks while teacher-only synthesis can substantially degrade performance.
- Experiments use GPT-OSS-120B as teacher and Qwen3-8B as student, with additional evaluations involving DS-R1, Qwen3-235B, and Qwen3-30B-A3B.
- TESSY increases Qwen3-8B performance by 7.78%, 5.85%, 11.34%, and 6.68% on the three LiveCodeBench datasets and OJBench, respectively.
- Teacher-Only causes a performance drop of up to 10.02% on OJBench even after training on 80K samples for 9 epochs.
- TESSY consistently improves the student on in-domain datasets despite using up to 77.65% teacher-generated tokens.
- TESSY improves Qwen3-8B by 3.75% on AIME2024 and 0.93% on AIME2025 while preserving comparable performance on GPQA-Diamond and OlympiadBench.
4. Analysis
Analysis tests TESSY across student models, teacher models, generation quality, and model versions. The results connect performance to student-consistent data distributions and show that TESSY can retain teacher-level quality while shortening responses.
- TESSY improves Qwen3-30B-A3B by 6.37% on LiveCodeBench-Pro and 10.13% on OJBench.
- Training Qwen3-8B on Qwen3-30B-A3B data reduces OJBench performance by 1.94% relative to training on Qwen3-8B data.
- TESSY outperforms Teacher-Only by 1.07% with Qwen3-235B, and by 3.45% and 16.79% with DS-R1 and GPT-OSS as teachers, respectively.
- TESSY outperforms GPT-OSS-120B by 10.99% at a 40K-token limit, while GPT-OSS remains 6.04% behind at 48K tokens.
- TESSY may guide the teacher toward earlier reasoning termination, potentially because the student produces shorter thought traces.
- Reasoning-model fine-tuning remains ahead of base-model fine-tuning, while TESSY still beats Teacher-Only by 0.43% on OJBench and 7.09% on LiveCodeBench-Pro for Qwen3-8B-Base.
5. Related Work
Related work frames TESSY against reasoning-model training, catastrophic forgetting, and teacher–student collaboration. These lines of work motivate preserving model-specific styles while transferring capabilities through practical offline synthesis.
- Reasoning models use extended thinking traces to decompose complex tasks, but further refining them with SFT remains underexplored because original training recipes are largely undisclosed.
- Reasoning models exhibit distinct styles across organizations and scales, and some stylistic tokens may functionally facilitate complex reasoning.
- Catastrophic forgetting concerns preserving knowledge acquired during earlier training stages, with parameter-efficient fine-tuning offering mitigation while limiting acquisition capacity.
- Teacher–student methods address capacity and style differences, but teacher retraining can be prohibitively expensive and self-distillation may introduce shortcut behaviors in reasoning models.
- TESSY differs from online alternating-generation approaches by synthesizing offline SFT data for knowledge transfer across models and tasks.
6. Conclusion
The conclusion presents TESSY as alternating teacher–student synthesis that preserves teacher reasoning while aligning data with student style. It also identifies efficiency and broader-task generalization as future work.
- TESSY alternates teacher and student models to preserve essential teacher reasoning information while aligning synthesized data with the student’s inherent style.
- Future work will refine identification of style and capability boundaries and extend the cooperative paradigm to broader tasks and heterogeneous model settings.
- TESSY synthesis speed remains below its theoretical limit because of engineering constraints, motivating implementation optimization and acceleration techniques.
Impact Statement
The paper presents TESSY as a framework for addressing distribution mismatch during reasoning-model training and reports implications for research and industrial practice.
- TESSY alleviates training-data distribution mismatch through a collaborative reasoning-based data synthesis framework.
- The study provides empirical evidence about differences between emerging reasoning models and traditional base models.
- The framework offers a potentially generalizable approach for enhancing reasoning models and supporting domain-specific development.
- The authors aim for the study’s insights to guide future research and help avoid costly missteps.
Limitations
TESSY performs better than Teacher-Only at a fixed generation limit, but Teacher-Only retains a higher observed capability ceiling when generation length is unrestricted.
- Under the same maximum generation length of 40K tokens, TESSY achieves higher code generation quality than Teacher-Only.
- Further extending TESSY’s capability upper bound requires mitigating quality degradation introduced by synthetic data.
- The paper identifies distribution mismatch as a primary factor affecting reasoning-model training while maintaining that data quality remains important.
A.1. Evaluation Beyond the Qwen3 Family
Evaluation with DeepSeek-R1-Distill-Llama-8B as the student shows that TESSY improves reasoning capability, while Teacher-Only and self-distillation produce different outcomes across student models.
- TESSY consistently improves the reasoning capability of DeepSeek-R1-Distill-Llama-8B, whereas self-distillation causes severe catastrophic forgetting.
- The evaluation uses GPT-OSS-120B as teacher and DeepSeek-R1-Distill-Llama-8B as student.
- Teacher-Only performs significantly worse than TESSY but still improves DeepSeek-R1-Distill-Llama-8B’s code reasoning on several benchmarks.
- The authors suggest Qwen3-8B’s more extensive post-training may make distributionally mismatched SFT data more damaging.
A.2. Evaluation on Math Reasoning Tasks
Math-reasoning evaluations extend TESSY beyond code-focused and Qwen3-family settings, finding gains over Teacher-Only on AIME2025 while self-distillation degrades performance.
- A.2. Evaluation on Math Reasoning Tasks: Qwen3-family models are already sufficiently trained on the evaluated math datasets, with Qwen3-8B outperforming GPT-OSS-120B on the training set.
- A.2. Evaluation on Math Reasoning Tasks: The supplementary comparisons include Teacher-Restyle and TESSY versus Teacher-Only under LoRA and full-parameter training.
- A.2. Evaluation on Math Reasoning Tasks: On AIME2025, TESSY outperforms Teacher-Only by up to 4.28%.
- A.2. Evaluation on Math Reasoning Tasks: Self-Distillation leads to severe degradation in reasoning performance on the evaluated math task.
- A.2. Evaluation on Math Reasoning Tasks: The study samples 12K questions from DeepMath for data synthesis and applies rejection sampling to retain samples with correct answers.
A.3. Evaluation on Science Reasoning Task
Across additional evaluations and analyses, TESSY generally preserves or improves Qwen3-8B performance while reducing style and distribution conflicts relative to Teacher-Only synthesis.
- 2.72% improvement on GPQA-Diamond contrasts with a 5.11% degradation from Teacher-Only training.
- 5.28% and 2.55% gains from system-prompt isolation still left Teacher-Only performance 4.74% and 0.70% below baseline on LiveCodeBench-Pro and OJBench.
- 12.33% and 13.58% drops occurred when the teacher, rather than the student, generated final answers in TESSY.
- TESSY remained effective with only 500 predictor-training samples, while 100K samples better supported synthetic-data generation.
- At epoch 8, OJBench reached 26.08%, while LiveCodeBench-Pro reached 36.69% at epoch 9.
- TESSY reduced average token counts versus Teacher-Only by 7,594, 8,938, and 2,816 for three teacher models.
- TESSY increased output similarity between Qwen3-8B and each teacher, reaching 0.971, 0.960, and 0.898.
B.4. Training Loss Curve
TESSY-generated SFT data yields lower training loss than Teacher-Only data for both reasoning and base model settings, consistent with reduced style-related loss.
- TESSY data consistently produced lower loss than Teacher-Only data for Qwen3-8B-Base and Qwen3-8B.
- The paper attributes the loss improvement to mitigating spurious loss from style tokens that do not contribute to knowledge learning.
- Reasoning-model training produced lower loss than base-model training, while TESSY remained below Teacher-Only for base models.