Source-linked AI summary
Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment
Yuming Yang, Mingyoung Lai, Wanxu Zhao, Xiaoran Fan, Zhiheng Xi, Mingqi Wu, Chiyue Huang, Jun Zhao, Haijun Lv, Jian Tong, Yunhua Zhou, Yicheng Zou, Qipeng Guo, Tao Gui, Qi Zhang, Xuanjing Huang
TL;DR
Reasoning distillation lacks a reliable way to identify trajectories that are both informative and aligned with a student, since stronger teachers and high-likelihood data do not consistently produce better students. The paper proposes Rank-Surprisal Ratio (RSR), which combines token rank with surprisal, and finds that it correlates strongly with post-training performance while improving trajectory and teacher selection.
Problem
Stronger teachers do not necessarily yield better students, while likelihood-based suitability metrics favor familiar trajectories that may provide insufficiently informative learning signals.
Method
RSR assesses trajectory suitability by dividing average token-wise rank by average negative log-likelihood to jointly capture alignment and informativeness.
Results
Across five students and trajectories from eleven teachers, RSR achieved an average Spearman correlation of 0.86 with post-training performance and consistently outperformed existing metrics in trajectory and teacher selection.
Takeaways & Limitations
Informative alignment is a promising direction for reasoning distillation, with RSR providing a practical criterion for selecting suitable trajectories and teachers.
Takeaways & Limitations
Selection gains are constrained by the diversity and quality of available trajectories or teachers, and systematic evaluation beyond mathematical reasoning remains incomplete.
Abstract
from arXiv · showhide
Long chain-of-thought (CoT) trajectories provide rich supervision signals for distilling reasoning from teacher to student LLMs. However, both prior work and our experiments show that trajectories from stronger teachers do not necessarily yield better students, highlighting the importance of data-student suitability in distillation. Existing methods assess suitability primarily through student likelihood, favoring trajectories that align closely with the student model's current behavior but overlooking more informative ones. Addressing this, we propose Rank-Surprisal Ratio (RSR), a simple metric that captures both alignment and informativeness to assess the suitability of a reasoning trajectory. RSR is motivated by the observation that effective trajectories typically balance learning signal strength and behavioral alignment by combining low absolute probability with relatively high-ranked tokens under the student model. Concretely, RSR is defined as the ratio of a trajectory's average token-wise rank to its average negative log-likelihood, and is straightforward to compute and interpret. Across five student models and reasoning trajectories from 11 diverse teachers, RSR strongly correlates with post-training reasoning performance (average Spearman 0.86), consistently outperforming existing metrics. We further demonstrate its practical utility in both trajectory selection and teacher selection.
1. Introduction
The paper argues that teacher trajectory quality depends on student-specific suitability, not teacher strength alone, and introduces RSR to balance informativeness with alignment. RSR measures this balance through token rank and surprisal and supports trajectory and teacher selection.
- Long CoT trajectories provide supervision for distilling reasoning into student models and for cold-starting reinforcement learning.
- Stronger teachers do not necessarily produce better students, because trajectory effectiveness varies substantially across student models.Existing likelihood-based methods favor familiar, high-likelihood trajectories that may provide limited new learning signals.
- RSR jointly captures trajectory informativeness and alignment by favoring low absolute probability with relatively high-ranked tokens under the student model.
- RSR is the ratio of average token-wise rank to average negative log-likelihood, is computable in one forward pass, and lower values indicate better informative alignment.
- Across five students and trajectories from eleven teachers, RSR achieved an average Spearman correlation of 0.86 with post-training performance and outperformed alternative metrics.
- RSR serves as an effective criterion for both trajectory selection and teacher selection, outperforming existing methods in these data-engineering scenarios.
2. The Need for Student-Specific Data
The study evaluates reasoning distillation across 55 teacher-student pairings and finds that student outcomes vary substantially with the teacher trajectory. These results show that teacher strength and surface attributes do not reliably determine student improvement.
- Experimental Settings: The study evaluates five student models with eleven reasoning teachers, creating 55 teacher-student pairings for supervised fine-tuning experiments.
- Experimental Settings: Each teacher generates long CoT responses for 5,000 math problems, after which each student is fine-tuned on the corresponding teacher dataset.Three independent generation runs are conducted for each teacher, with results averaged across runs.
- Experimental Settings: The teachers span 4B to 671B parameters across DeepSeek, GPT-OSS, Qwen, LLaMA-Nemotron, and Phi model families.
- Experimental Settings: Student reasoning is evaluated with average Acc@4 across AIME’25, AIME’24, AMC’23, and MATH500.
- Results: Teacher capability does not reliably predict student improvement: 671B and 235B teachers often underperform smaller teachers such as QwQ-32B.
- Results: Data-student suitability is student-specific, with strong teachers often failing weaker students and weaker teachers failing stronger students; distant model families can also yield inferior results.
3. Measuring Data-Student Suitability
Effective reasoning trajectories must balance informativeness with alignment to the student model. The section develops this view through surprisal, rank, simulation, and the Rank-Surprisal Ratio (RSR).
- 3.1. Surprisal and Rank: Surprisal measures absolute unfamiliarity from token probability, whereas rank measures relative familiarity among vocabulary alternatives.A token can have low absolute probability while still ranking among the student’s top candidates.
- 3.2. Limitations of Probability-Based Metrics: Probability-based suitability metrics favor familiar trajectories, but lower surprisal does not necessarily produce better post-training reasoning performance.Trajectories from Nemotron-Super and Magistral-Small receive lower surprisal yet fail to improve reasoning performance, while very high-surprisal trajectories also perform poorly.
- 3.3. Insight and Simulation: Effective trajectories should deviate from the student’s dominant generations while remaining compatible with its broader prediction patterns.This balance corresponds to low absolute probability together with relatively high-ranked tokens.
- 3.3. Insight and Simulation: Simulation results show that preferred X_B trajectories have higher surprisal than X_A and X_D while retaining relatively low rank values.X_B represents trajectories that depart from the dominant mode but align with minor patterns within the student model.
- 3.4. Proposed Metric: Rank-Surprisal Ratio: RSR combines token rank and surprisal to identify trajectories that jointly provide informative learning signals and behavioral alignment.Its interpretation is a rank-to-surprisal relationship: lower rank indicates stronger relative familiarity, while higher surprisal indicates greater deviation from dominant patterns.
- 3.4. Proposed Metric: Rank-Surprisal Ratio: The trajectory-level RSR is interpreted as the ratio of average token rank to average token surprisal, with lower values indicating better informative alignment.Directly averaging token-level ratios can be unstable, motivating the trajectory-level formulation.
4. Correlation Analysis
The correlation analysis evaluates whether RSR measures data-student suitability by comparing metric scores with post-training reasoning performance across students, teachers, and alternative metrics. RSR shows the strongest overall association, while ablations support the roles of rank clipping and surprisal-weighted averaging.
- Evaluation setup: Table 4 reports absolute Spearman correlations between suitability metrics and post-training reasoning accuracy for different student models.Student-agnostic metrics are computed independently of the specific student model.
- Evaluation setup: The study aggregates trajectory-level scores across datasets from eleven teachers and correlates them with each student’s post-training reasoning performance.Spearman correlation is the primary measure, while Pearson correlation shows similar trends.
- Main correlation results: 0.86 average Spearman correlation: RSR consistently outperforms alternative suitability metrics across all student models.Surprisal-based and rank-based metrics alone reach correlations of at most 0.59.
- Ablation study: Removing rank clipping or surprisal-weighted averaging substantially degrades dataset-level RSR correlation.The ablation tests whether these two design components are necessary for the metric’s correlation strength.
- Robustness: Using 200 trajectories per teacher produces comparable correlations to the full sample, and r_max=500 gives similar results.These findings indicate robustness to reduced data and reasonable hyperparameter variation.
5. Practical Applications
The paper applies RSR to selecting trajectories and teachers before training, including settings designed to reflect limited data or generation resources. RSR-selected data achieves the best or near-oracle post-training performance across the reported applications.
- Trajectory selection: The trajectory-selection task chooses one trajectory from 33 candidates generated by 11 teachers, supporting data-efficient training when resources are limited.Each teacher contributes three candidates per problem or prompt.
- Trajectory selection: RSR-selected trajectory datasets consistently achieve the best post-training reasoning performance across student models.In four students, RSR even surpasses the best performance obtained from any single teacher’s dataset.
- Teacher selection: Teacher selection uses 200 sampled trajectories per candidate teacher to estimate dataset-level suitability in a low-resource setting.The candidate pool contains six diverse teachers selected to make the task non-trivial.
- Teacher selection: RSR-selected teachers achieve average post-training results close to oracle teachers and outperform other teacher-selection methods.Both the best and second-best RSR-selected teachers perform strongly.
6. Related Work
Related work studies knowledge distillation and the construction or filtering of reasoning trajectories for supervised fine-tuning. This paper differs by explicitly quantifying the trade-off between informativeness and alignment for student-specific data selection.
- Knowledge distillation: Knowledge distillation transfers knowledge from large models to smaller ones and is widely used to train LLMs.Prior approaches include on-policy distillation, self-distillation, reinforcement learning integration, and other alignment or filtering methods.
- Positioning: RSR explicitly quantifies informativeness and alignment to identify effective teacher data for a given student.This frames the contribution as a student-specific suitability measure rather than another implicit alignment procedure.
- SFT with reasoning trajectories: Reasoning-trajectory research constructs or filters CoT data because high-quality trajectories can improve student reasoning through supervised fine-tuning.Recent work also recognizes that suitable reasoning data may vary across students.
7. Broader Applicability of RSR
The paper evaluates RSR only on reasoning tasks while noting that the metric is not inherently specific to CoT trajectories. It identifies general-text evaluation and cross-problem subset selection as broader directions.
- Scope boundary: The experiments focus solely on reasoning tasks, although RSR is not specifically designed for reasoning trajectories.The paper presents evaluation beyond reasoning settings as future work.
- Subset selection: RSR may also support subset selection, where samples are filtered across heterogeneous reasoning problems rather than compared within one problem.This setting is relevant when only one trajectory is available per problem and improved data efficiency is desired.
8. Conclusion
The paper studies data-student suitability in reasoning distillation and proposes RSR to identify suitable trajectories by jointly capturing informativeness and alignment. Across diverse teacher-student pairs, RSR correlates with post-training performance and supports trajectory and teacher selection.
- RSR identifies reasoning trajectories for a given student by combining informativeness with alignment to the student’s behavior.
- RSR favors trajectories with low absolute probability but relatively high-ranked tokens under the student model.
- Experiments across diverse teacher-student pairs show that RSR strongly correlates with post-training performance and consistently outperforms existing metrics.
- RSR is effective for both reasoning-trajectory selection and teacher selection.
Limitations
The paper identifies limitations concerning candidate-pool quality, theoretical grounding, and experimental scope. It also focuses its controlled studies on mathematical reasoning and fixed datasets because broader extensions require substantial computation.
- Selection gains are constrained by the diversity and quality of available candidate trajectories or teacher models.When no available teacher trajectories suit a student, selection alone may provide limited gains.
- RSR lacks a suitable theoretical framework that formally characterizes the metric.The paper leaves deeper theoretical grounding for future investigation.
- The experiments focus primarily on mathematical reasoning and do not systematically evaluate newly generated trajectories or qualitatively different domains such as code and commonsense reasoning.The authors attribute this scope to resource constraints and the cost of regeneration and retraining.
- Teacher trajectory datasets use a fixed 5,000-problem mathematical dataset to control problem-composition effects.The problems are drawn from MATH, AIME/AMC, and Olympiads, with difficulty filtering and balanced sampling.
- Each teacher contributes three independently generated datasets, using a maximum generation budget of 31,000 tokens per trajectory.The resulting collection contains 11 × 3 = 33 datasets.
- The study uses five open-source Qwen and LLaMA base models as students rather than chat models.This choice provides a clean starting point, greater potential training effects, and avoids stylistic mismatch and overlapping supervision.
A.7. Details of Correlation Analysis
The correlation analysis evaluates whether data-suitability metrics track post-training performance while accounting for aggregation choices, computational cost, and baseline implementations. RSR requires a single forward pass and is substantially cheaper than methods involving gradients or generation.
- Spearman correlation is used to measure monotonic consistency between suitability metrics and post-training performance.Pearson results are additionally reported, while averaged correlations are computed across student models before taking absolute values.
- Dataset-level metrics average the three trajectory datasets generated by each teacher because post-training performance likewise averages three generation runs.The metric varies only marginally across datasets generated by the same teacher.
- Dataset-level RSR uses surprisal-weighted averaging to reduce instability from trajectories with very small average surprisal.The weighting also emphasizes trajectories with larger average surprisal.
- The analysis compares surprisal-weighted dataset-level RSR with the simple average of trajectory-level RSR in an additional ablation.
- RSR computes token surprisals and clipped ranks in one forward pass, with worst-case trajectory complexity O(TV).T denotes response-token count and V vocabulary size.
- Computing RSR for 5,000 trajectories with a 32,768-token context on a 7B model typically takes under one hour on one H200 GPU.This cost is significantly lower than SFT and lower in total GPU-hours than the compared metrics on the fixed benchmark dataset.
D.1. Statistical Analysis of RSR-Selected Trajectories
The statistical analysis examines what RSR-selected trajectories look like and how token-level rank, surprisal, and RSR differ across teachers. It also identifies numerical instability at extremely low surprisal and reports strong Pearson as well as Spearman correlations.
- RSR-selected trajectories have moderate token length and verified accuracy rather than being selected solely for either property.The comparison uses aggregate mean, maximum, and minimum statistics from source teacher datasets.
- The case study compares token-level rank, surprisal, and clipped RSR across GPT-OSS-20B, Nemotron-Super-49B-v1.5, and QwQ-32B trajectories.The measurements are taken using the Qwen-2.5-7B student, with rank clipping at r_max = 100.
- GPT-OSS-20B trajectories often have excessively high token ranks, indicating unfamiliarity and misalignment with the student’s behavior.The analysis associates this pattern with fast-paced reasoning flow and uncommon phrasing.
- Extremely low surprisal can inflate token-level RSR values above 100, creating numerical instability.Surprisal-weighted trajectory-level averaging is used to stabilize the metric and improve effectiveness.
- RSR shows strong Pearson correlation with post-training performance in addition to the reported Spearman correlation results.
E.2. Additional Ablation Study
Additional ablations show that RSR’s performance depends on combining rank with surprisal, using student-specific estimation, and applying it to trajectory selection across settings.
- Metric ablations: Student-specific RSR estimation matters: using a fixed Qwen-3-14B model instead of the target student reduces correlation.The ablation attributes this drop to the importance of student-specific suitability estimation.
- Metric ablations: Simple averaging of trajectory-level RSR remains a robust dataset-level estimator, while surprisal-weighted aggregation improves reliability at both levels.The simple-average variant causes only slight correlation degradation.
- Metric ablations: Rank alone is insufficient: clipped average rank retains a notable gap versus RSR, while entropy, subtraction, multiplication, and rank-entropy alternatives align less well with performance.These alternatives tend to emphasize high-likelihood trajectories without adequately capturing informativeness.
- Metric ablations: RSR remains robust across exponent choices, although the default power-1 formulation is retained for simplicity.Tuning exponents may yield higher correlation, but the authors use equal rank and surprisal powers of 1.
- Trajectory selection: RSR-selected 5k trajectories are comparable to or better than the 55k No-Selection baseline on most students, with stronger gains for Qwen-3-14B and Qwen-3-4B.LLaMA-3.1-8B is an exception, possibly requiring larger-scale reasoning data.
- Trajectory selection: RSR selection remains effective with correctness filtering, fewer candidates per teacher, limited teacher pools, and out-of-domain GPQA-Diamond evaluation.The GPQA-Diamond results are less stable than mathematical results, but RSR-selected datasets achieve the best overall performance.
- Teacher selection: Selected teacher distributions vary by student, and selected datasets consistently have lower RSR values than teacher trajectory datasets.This supports student-tailored trajectory and teacher selection.
F. Complete Results Tables
The appendix provides complete post-training evaluation and metric-assessment tables, alongside the evaluation prompt and scoring criteria used for LLM-judged quality assessment.
- Complete results tables: Tables 25–29 provide full metric-assessment results across the five student models.The tables cover Qwen-3-14B, LLaMA-3.1-8B, Qwen-2.5-7B, Qwen-3-4B, and Qwen-2.5-3B.
- Complete results tables: Tables 20–24 contain complete trajectory-selection evaluation results for Qwen-3-14B, LLaMA-3.1-8B, Qwen-2.5-7B, Qwen-3-4B, and Qwen-2.5-3B.Each table reports mathematical evaluation results, including Math Avg. over AIME’24, AIME’25, AMC’23, and MATH500.
- Scope and materials: The paper states that its artifacts are publicly available, while the training datasets contain only mathematics problems and solutions without offensive content or personal information.These statements describe artifact availability and dataset composition rather than experimental outcomes.