Order-Only On-Policy Distillation

1 INTRODUCTION

• Teacher probabilities mix token preferences with confidence, which a smaller student may struggle to reproduce; OPD is also temperature-sensitive and can fall below the base model during training.The best OPD temperature differs across teacher–student pairs.Source: §1 · p. 2
• The paper distinguishes token choice, token ranking, and token probabilities, arguing that ranking carries the useful distillation knowledge while probability values constrain the student.Sequence KD imitates the sampled token, while distribution-matching methods reproduce probabilities.Source: §1 · p. 2
• O3PD trains on student-generated prefixes by ordering the teacher’s top-K tokens with adjacent ranking constraints, without fitting their probability values.Because positive temperature scaling preserves order, O3PD needs no teacher-temperature tuning.Source: §1 · pp. 2–3
• Across six Qwen3 pairs, O3PD beats OPD on the three-benchmark average in every pair by up to 2.9 points, improves AIME25 by up to 5.6 points, and exceeds SFT and GRPO.The benchmarks are HMMT25, AIME24, and AIME25.Source: §1 · pp. 2–3
• O3PD matches the teacher’s ranking more faithfully than OPD despite matching its probabilities less, and lets the student move further from initialization, especially at the smallest scale.O3PD continues improving through training while OPD peaks early and declines.Source: §1 · p. 2

2 RELATED WORK

• Knowledge distillation can be viewed along two axes: where supervision is applied and what teacher information is transferred.Earlier methods often use fixed sequences, creating a train–inference mismatch because students later condition on their own generations.Source: §2 · p. 3, §2.1 · p. 3

2.1 LLM DISTILLATION: FROM TEACHER STATES TO STUDENT STATES

• On-policy methods query teachers on student-generated states, but prior approaches generally retain probability distributions as per-state targets; O3PD changes that target to a ranking.Some methods alter states, divergences, trajectory construction, or data schedules instead.Source: §2.1 · p. 3

2.2 DISTILLATION BEYOND PROBABILITY MATCHING

• Score-based relaxations such as standardization and DIST still depend on continuous scores, while ordinal targets preserve preferences under strictly increasing transformations.Prior ranking methods may retain KL or confidence weighting, or use fixed-label or fixed-pool settings; O3PD ranks changing candidate sets at student-generated states.Source: §2.2 · p. 3
• Figure 2 presents adjacent logistic loss as O3PD’s default, while other order-only objectives can replace it.The paper contrasts this order-only target with prior methods that retain probability or confidence information.Source: §2.2 · p. 3, Figure 2 · p. 4

3.1 FROM SEQUENCE AND DISTRIBUTION TARGETS TO ORDINAL SUPERVISION

• Distribution matching ties candidate order to probability strength, whereas O3PD uses the teacher’s top-K candidates and ordering without fitting probability magnitudes.This retains on-policy supervision on contexts encountered by the current student and makes the objective invariant to positive temperature scaling and other strictly increasing score transformations.Source: §3.1 · pp. 4–5

3.2 LEARNING THROUGH ADJACENT COMPARISONS

• O3PD trains on adjacent teacher-ranked candidates using the student’s pairwise preference, with the logit difference determining that preference.The K−1 adjacent inequalities recover the full order by transitivity, using fewer comparisons than the K(K−1)/2 all-pairs relations.Source: §3.2 · p. 5

3.3 ORDER-ONLY AS A DISTILLATION PRINCIPLE

• Order-only supervision is a principle: objectives can vary in which comparisons they use or impose the ranking pairwise versus as a whole.Logistic uses all pairs, Top-Weighted Logistic emphasizes higher-ranked candidates, and ListMLE imposes the order over the full list.Source: §3.3 · pp. 5–6, Table 2 · p. 5

4.1 EXPERIMENTAL SETUP

• The setup pairs each Qwen3 student with every larger teacher, freezes the teacher, and trains on OpenThoughts mathematical reasoning data.Evaluation uses HMMT25, AIME24, and AIME25; Avg@12 averages final-answer accuracy over 12 responses per problem.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• For 8B→1.7B, O3PD reaches 40.7 at the final step while OPD peaks at 38.1 and then falls below the 37.7 base student.On AIME25, O3PD rises from 35.6 to 43.1 while OPD falls back to its starting point.Source: §4.2 · p. 6, Figure 3 · p. 8
• O3PD raises Avg@12 over OPD in all six teacher–student configurations: +0.6 for the 8B student, +2.3/+1.9 for 4B, +2.6/+2.9 for 1.7B, and +0.3 for 14B→1.7B.It also exceeds the base model, SFT, and GRPO in average score in every configuration.Source: §4.2 · p. 6, Table 3 · p. 7
• Best-checkpoint selection does not explain the gains: the end-of-training gap exceeds the 2.6-point gap between the methods’ best checkpoints.OPD peaks early and declines, whereas O3PD continues improving in the 8B→1.7B training curve.Source: §4.1 · p. 6, §4.2 · p. 7, Figure 3 · p. 8
• O3PD with a smaller teacher beats OPD with a 14B teacher: 40.9 vs. 39.1 for the 1.7B student, and 63.2 vs. 60.9 for the 4B student.The reported comparisons show stronger distillation with a cheaper teacher in these configurations.Source: §4.2 · p. 7
• OPD’s best temperature varies by pair, and temperature choice alone shifts HMMT25 accuracy by up to 6.4 points; O3PD’s top-K order is unchanged for T > 0.For 8B→4B, OPD scores 44.7 at T = 1 versus 38.3 at T = 8.Source: §4.2 · p. 7

4.3 ABLATION STUDY

• All tested ranking losses beat OPD by 1.3 to 2.6 points at K = 8; adjacent logistic and ListMLE perform within 0.1 points of each other.Adjacent Logistic supervises 7 of 28 pairs; transitivity makes those adjacent relations sufficient to fix the full ranking.Source: §4.3 · p. 7, Table 4 · p. 8
• All tested candidate-list sizes beat OPD by 1.6 to 2.6 points, with K = 8 best and K = 4 and K = 16 not far behind.The ablation tests candidate-set size on the 8B→1.7B pair.Source: §4.3 · p. 8, Table 4 · p. 8

5.1 LEARNING THE TEACHER’S ORDER, NOT ITS PROBABILITIES

• Agreement is measured on student-generated responses using probability divergence, ordering, candidate overlap, top-token agreement, and student entropy.JSD@Top-8 compares renormalized distributions restricted to the models’ top-8 tokens.Source: §5.1 · p. 8
• O3PD achieves higher Kendall’s τ for most of training and ends above OPD, keeps top-8 overlap higher at nearly every evaluation, and has comparable top-1 agreement.Its JSD is consistently higher, while student entropy stays above OPD throughout training.Source: §5.1 · pp. 8–9, Figure 5 · p. 9

5.2 RESHAPING THE STUDENT

• For the 1.7B student, O3PD checkpoints move several times farther from initialization than OPD; at 4B the distances are comparable, and at 8B OPD moves farther.Across student sizes, O3PD checkpoints remain farther from the teacher’s probabilities than OPD checkpoints.Source: §5.2 · p. 9
• A smaller student may be unable to reproduce a much larger teacher’s probability values, while ordinal supervision lets it restructure its distribution if it preserves the teacher’s candidate ordering.The smallest student makes the most use of this freedom.Source: §5.2 · p. 9

• O3PD trains students to reproduce the teacher’s top-K token ranking without matching probability values, making it invariant to teacher temperature and requiring no temperature tuning.Source: §6 · p. 9
• O3PD outperforms OPD on the three-benchmark average in all six Qwen3 teacher–student configurations and also exceeds SFT and GRPO.On the 8B→1.7B pair, O3PD keeps improving where OPD degrades.Source: §6 · p. 9
• O3PD reproduces the teacher’s ranking and candidate set more faithfully than OPD despite departing further from its probability values; other ranking objectives such as ListMLE also perform well.Source: §6 · p. 9

• The work does not involve human subjects, user data, or personally identifiable information.Source: ETHICS STATEMENT · p. 10
• Experiments use publicly released Qwen3 models, OpenThoughts mathematical-reasoning data, and HMMT25, AIME24, and AIME25 problems, under their respective licenses.Source: ETHICS STATEMENT · p. 10
• Evaluation is restricted to mathematical reasoning and does not assess ordinal distillation’s effects on safety alignment or other open-ended-generation behaviors.Source: ETHICS STATEMENT · p. 10

• O3PD’s objective is specified in Section 3, and Algorithm 1 provides pseudocode for one training iteration.Source: REPRODUCIBILITY STATEMENT · p. 10
• Appendix A lists training and evaluation hyperparameters, including LoRA configuration, rollout length, candidate-set size, decoding parameters, and random seeds.Figure 6 gives the prompt template used for training rollouts and evaluation; the OPD baseline shares the setup.Source: REPRODUCIBILITY STATEMENT · p. 10
• The models, training data, and evaluation benchmarks are publicly available, and the protocol specifies Avg@12 over three seeds and answer extraction from \boxed{}.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 HYPERPARAMETERS

• Students are fine-tuned with LoRA for 100 steps at effective batch size 32; rollouts are limited to 1,024 tokens, and both methods use the teacher’s top-8 candidates at each prefix.Checkpoints are evaluated every 20 steps; Avg@12 uses four samples per problem under each of three seeds.Source: §A.1 · p. 13, Table 5 · p. 14

A.2 PROMPT TEMPLATE

• The shared prompt asks models to reason step by step and put the final answer in \boxed{}, which is extracted for scoring.Figure 6 identifies the prompt template and notes that {problem} is replaced with the problem statement.Source: §A.2 · p. 13, Figure 6 · p. 13

A.3 TRAINING PROCEDURE

• At every prefix of a student-sampled response, the teacher supplies its top-K tokens in descending probability order; the student is updated using averaged pairwise logistic terms over positions and adjacent pairs.Teacher probabilities select and order candidates but do not appear in the loss.Source: §A.3 · p. 14

A.4 INFRASTRUCTURE

• Training uses Transformers and evaluation uses vLLM on NVIDIA H100 GPUs.Source: §A.4 · p. 15

B TEACHER TEMPERATURE SENSITIVITY OF OPD

• O3PD’s ranking targets are identical for every T > 0, avoiding OPD’s temperature-selection problem.Source: B TEACHER TEMPERATURE SENSITIVITY OF OPD · p. 15
• 14B→0.6B falls from 11.94 at T = 1 to 1.11 at T = 8, while OPD’s best temperature varies by pair and sensitivity increases for smaller students.The sweep measures HMMT25 Avg@12 across six configurations plus a Qwen3-0.6B student.Source: Figure 7 · p. 15, B TEACHER TEMPERATURE SENSITIVITY OF OPD · p. 15