Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training

• Across nine Qwen3 student–teacher pairs spanning 2× to 53× parameter ratios, OPD depended on compatibility, not teacher scale alone, and lower reverse KL did not guarantee higher accuracy.Reinforceable fraction was a complementary diagnostic of compatibility where scalar divergence was insufficient.Source: §1 · p. 2
• The main controlled study used a Qwen3-4B-Base student and Qwen3-14B teacher across six math and one science benchmark, holding data, optimization, and evaluation fixed within comparisons.Source: §1 · p. 2
• Brief SFT warm-up improved student distillability, whereas an RLVR-strengthened student could regress under distillation from the same teacher.The student’s capability and distillability can move in opposite directions.Source: §1 · pp. 2–3
• RLVR adaptation of the teacher improved downstream OPD as teacher accuracy rose, with diminishing returns once teacher performance plateaued.Source: §1 · pp. 2–3
• At comparable starting accuracy, OPD provided a more effective initialization than SFT for subsequent RLVR, and aggregate policy entropy alone did not explain the advantage.Source: §1 · p. 2
• 43.8% versus 29.2%: teacher RLVR adaptation plus brief student SFT warm-up raised average final OPD accuracy by 50% at the same OPD step budget, with additional preparatory training.The resulting recipe is to adapt the teacher, briefly warm up the student, and then distill.Source: §1 · pp. 2–3

Post-Training Objectives

• SFT uses dense token-level supervision on fixed labeled trajectories, RLVR uses sparse scalar rewards on student rollouts, and OPD gives dense teacher supervision on student-generated trajectories.The default OPD objective matches student and teacher with token-level reverse KL.Source: §2 · p. 3

Measuring Transfer

• Transfer is measured by distillation effectiveness, the reinforceable fraction of student tokens receiving a positive teacher signal at initialization, and normalized reverse KL tracking on-policy divergence during training.Distillation effectiveness is the fraction of the student–teacher accuracy gap recovered; the reinforceable fraction captures how often the teacher reinforces the student’s behavior.Source: §2 · p. 3

From Individual Stages to Composition

• Prior work studies OPD dynamics and adjacent-stage effects, but how changing one post-training stage alters the next stage’s learning conditions remains less understood.The paper addresses this gap by characterizing OPD under a controlled protocol (§3), then studying how SFT, RLVR, and OPD reshape student–teacher compatibility when composed (§4).Source: §2 · p. 3

Observation 3.1

• For Qwen3-4B-Base, switching the teacher from 8B to 32B lowers accuracy from 32.70% to 29.25% and DE from 65.1% to 43.3%.The preferred teacher varies by student: Qwen3-8B performs best for the 4B and 1.7B students, while Qwen3-14B leads for the 0.6B student.Source: Figure 2 · p. 4, §3 · p. 4

Observation 3.2

• Under forward KL, DE falls as teacher size rises at every top-k support, while base students gain 18.54–19.14 points from OPD versus 2.48–2.66 for instruct students.Post-trained teachers improve peak accuracy in four of five matched-size comparisons, and Qwen3-8B gives the highest accuracy for the 4B student on both DAPO-Math and OpenThoughts-Math.Source: §3 · pp. 4–5, Figure 3 · p. 4

Observation 3.3

• Before OPD, initial ρ+_0 decreases with teacher size for every student, and maximum ρ+_0 identifies the step-200 best teacher for two of three students.Larger teachers leave more tokens without positive distillation signal; for both 0.6B and 1.7B students, the 32B teacher has the smallest initial rKL but the lowest step-200 accuracy, showing that scalar divergence alone does not identify productive pairings.Source: §3 · p. 5, Figure 4 · p. 5
• Effective OPD depends on the learning signal induced by the current student–teacher pairing, not teacher capability or scalar policy distance alone.The study characterizes nine pairs of Qwen3 students and post-trained teachers spanning 2×–53× parameter ratios.Source: §3 · pp. 4–5

4.1 STUDENT WARM-UP BEFORE DISTILLATION

• A short SFT warm-up raised downstream OPD performance from 33.6% to 35.4%, while an RLVR-warmed student fell from 38.5% to 28.4% during OPD.The RLVR-warmed student began within 3.6 points of the teacher, showing that higher starting capability did not ensure better distillability.Source: Figure 5 · p. 6, §4.1 · p. 6
• Downstream OPD gains rose from 30.9% after 25 SFT steps to 35.1–35.4% after 75–150 steps, then saturated near the first validation-loss plateau around step 100.Initial entropy fell from 0.2559 at K = 100 to 0.1507 at K = 250; the plateau is a candidate signal for checking diminishing warm-up returns.Source: Figure 5 · p. 6, §4.1 · p. 6

4.2 TEACHER ADAPTATION BEFORE DISTILLATION

• Distillation from RLVR-adapted teachers raised student accuracy from 18.9% to 38.5–43.7%, versus 33.6% from the off-the-shelf teacher; STaR-adapted teachers yielded 33.4–34.9%.The STaR control did not meaningfully improve teacher capability, and downstream student accuracy was correspondingly unchanged.Source: Figure 6 · p. 7, §4.2 · p. 7
• Student OPD peaks tracked teacher capability gains: a 56.2% teacher yielded a 42.2% student peak, while later 56.7–57.0% teacher checkpoints yielded 40.7–43.7%.Teacher adaptation raised accuracy from 42.1% to 51.0% at K = 50 and 56.2% at K = 100; downstream returns flattened when teacher accuracy plateaued.Source: Figure 6 · p. 7, §4.2 · p. 7

4.3 STAGE ORDERING FOR RLVR INITIALIZATION

• After 260 RLVR steps, every OPD-initialized run reached 38.5–43.1% versus 33.4–36.8% for every SFT-initialized run, and OPD runs were still improving.The initial accuracy was similar, with SFT slightly ahead on average; the trajectories diverged over the observed horizon.Source: Figure 7 · p. 8, §4.3 · p. 8
• OPD-initialized and SFT-initialized checkpoints had substantially overlapping initial entropy but clearly separated RLVR trajectories, so scalar entropy alone did not explain the advantage.Some SFT checkpoints began with higher entropy than OPD checkpoints but still finished lower.Source: Figure 7 · p. 8, §4.3 · pp. 8–9
• RLVR from the step-50 OPD peak reached 43.11%, versus 38.5–40.8% from later OPD checkpoints that had regressed.RLVR recovered only part of the lost performance, favoring a handoff near OPD’s task-performance peak rather than continued distillation toward teacher convergence.Source: Figure 7 · p. 8, §4.3 · pp. 8–9

5 A SIMPLE COMPOSITION RECIPE

• Teacher adaptation supplied most of the improvement, while adding student warm-up contributed a further 1.6 points; the interventions’ gains were not additive.The recipe adapted Qwen3-14B with RLVR to convergence (K = 200), warmed Qwen3-4B-Base with SFT on adapted-teacher trajectories (K = 75), then distilled from that teacher.Source: §5 · p. 9
• 43.8% average accuracy from teacher adaptation plus student warm-up exceeded vanilla OPD’s 29.2% at the common 200-step endpoint.This is a 14.6-percentage-point, or 50% relative, improvement with additional preparatory training.Source: Figure 8 · p. 9, §5 · p. 9
• The composition principle is to adapt the teacher with RLVR, warm the student with SFT, distill with OPD, then distill to the cold-start student.The section presents this as a simple recipe, not a new algorithm.Source: §5 · p. 9
• Whether the recipe holds across model scales, domains, and longer training horizons remains outside the study’s scope.The paper describes its contribution as a first controlled map of how the post-training stages interact.Source: §5 · p. 9

6 CONCLUSION

• 43.8% versus 29.2%: teacher adaptation plus student warm-up raises final OPD accuracy at the same distillation-step budget.The experiments investigate how SFT, RLVR, and OPD interact under a shared protocol.Source: §6 · p. 9
• Brief SFT improves distillability, an RLVR-strengthened student can regress under distillation, and teacher RLVR improves transfer until teacher accuracy plateaus.The reported findings concern controlled interventions and the learning state inherited by each stage.Source: §6 · p. 9
• At similar starting accuracy, OPD supports stronger downstream RLVR scaling than SFT within the studied budgets.The main Qwen3-4B-Base comparison also showed overlapping aggregate entropy.Source: §6 · p. 9
• Experiments mainly use Qwen3-4B and Qwen3-14B; transferability at matched teacher accuracy and the mechanisms behind teacher-RLVR gains remain open.Future work could test larger MoE models and agentic learning, and study adaptive stage transitions and refreshed teachers, motivated by the recursive OPD case study.Source: §6 · p. 9

AI USE STATEMENT

• Generative AI tools helped edit text, check number consistency, suggest related work, and write or debug plotting scripts; authors reviewed and verified assistance.The authors state that AI tools did not design experiments, implement training methods, generate or clean data, or produce or interpret results, and that they take responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• Because distillation may transfer teacher errors or biases, improved benchmark accuracy alone is not evidence of safe deployment.Applications beyond the studied settings require appropriate safety and reliability evaluation.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The appendices document objectives, diagnostics, model and dataset details, training and evaluation settings, analyses, and procedures across the experiments.The authors report three rollout seeds for main Qwen3-4B-Base scaling experiments, one run per teacher for smaller-student scaling, and plans to release code, data-preparation scripts, configurations, available logs, evaluation outputs, and reproduction scripts.Source: REPRODUCIBILITY STATEMENT · p. 10

A BACKGROUND AND RELATED WORK

• Appendix A defines the three post-training objectives and student–teacher transfer measures, summarizes established OPD findings, and identifies the gap studied.Definitions are placed alongside the prior work they draw on.Source: A BACKGROUND AND RELATED WORK · p. 16

A.1 POST-TRAINING OBJECTIVES

• SFT uses fixed trajectories, RLVR gives sparse outcome rewards on student rollouts, and OPD gives dense teacher supervision on student-generated trajectories.In the sampled-token OPD estimator, the teacher advantage reinforces teacher-preferred sampled tokens when positive and suppresses them when negative, without specifying a replacement.Source: §A.1 · p. 16

A.2 MEASURING TRANSFER

• Transfer is measured by the fraction of the teacher–student accuracy gap recovered, the initialization-time fraction of student tokens the teacher would reinforce, and on-policy reverse KL during training.Gap recovery is defined when the teacher outperforms the student; negative values indicate regression.Source: §A.2 · p. 17

A.3 WHAT IS KNOWN, AND WHAT IS NOT

• The paper’s central gap is understanding how composing post-training stages changes the learning conditions faced by the next stage.It first characterizes productive OPD dynamics, then studies how SFT, RLVR, and OPD reshape student–teacher pairing, and finally examines whether compatibility can be maintained as the student evolves.Source: §A.3 · p. 18

B EXPERIMENTAL DETAILS AND ADDITIONAL ANALYSES ON OPD DYNAMICS

• The appendix provides configurations, learning curves, and additional analyses of how teacher scale interacts with student state, objective, and training duration.It also examines reinforceable fraction and conditions under which an early handoff can preserve task performance.Source: B EXPERIMENTAL DETAILS AND ADDITIONAL… · p. 18

B.1 EXPERIMENTAL SETUP AND TRAINING CONFIGURATIONS

• The main OPD grid pairs Qwen3-0.6B, 1.7B, and 4B-Base students with 8B, 14B, and 32B post-trained teachers, using nine runs on DAPO-Math-17k-Processed2.The 4B experiments use three seeds; smaller-student comparisons use one run per teacher, and the core schedule is one epoch with a 279-step optimizer horizon.Source: §B.1 · p. 18

B.2 METRIC DEFINITIONS AND AGGREGATION

• The primary aggregate equally weights seven benchmarks, and mean@4 averages correctness across sampled responses rather than using majority voting or counting any-correct questions as solved.Accuracy summaries aggregate benchmarks within runs and then runs; reported peak summaries take each run’s maximum before averaging.Source: Table 3 · p. 19, §B.2 · p. 20

B.3 FULL STUDENT–TEACHER SCALING RESULTS

• At step 260, the 8B teacher gives the highest accuracy for all three students; for Qwen3-4B-Base, its 31.49% exceeds the 14B and 32B outcomes by 2.30 and 2.92 points.The corresponding DE values show that increased teacher scale does not consistently translate into greater recovery of teacher capability.Source: Table 5 · p. 21, §B.3 · p. 21

B.4 ROBUSTNESS ACROSS DISTILLATION OBJECTIVES

• Across forward-KL top-k settings, Qwen3-32B yields the lowest accuracy and DE; Qwen3-14B leads at k = 16 and 32, while Qwen3-8B leads at k = 64.The comparison uses Qwen3-4B-Base at a common budget of 180 steps.Source: Table 6 · p. 21, §B.4 · p. 21
• At k = 16, Qwen3-14B distillation reaches student accuracy of 34.48% at step 180 versus 33.66% with Qwen3-8B, despite lower DE.The larger capability gap for Qwen3-14B increases DE's normalization denominator, so higher student accuracy can correspond to a smaller transferred fraction of teacher capability.Source: §B.4 · pp. 21–22

B.5 MODEL STATES, TRAINING DATA, AND EVALUATION DOMAINS

• For Qwen2.5-1.5B, OPD gains were 18.54–19.14 points for base students versus 2.48–2.66 for instruct students; post-trained Qwen3-4B fell below its 39.58% baseline with every teacher.At matched sizes, post-trained teachers produced higher peak student accuracy in four of five pairs; Qwen3-1.7B-Base with a 14B teacher was the exception.Source: Table 7 · p. 22, §B.5 · p. 22, Table 8 · p. 22
• Qwen3-8B was the top teacher on both datasets at 200 steps and all six math benchmarks, while Qwen3-14B led GPQA-Diamond and the ten-benchmark average.The corresponding aggregates were 33.22% for Qwen3-8B on math-six and 28.14% for Qwen3-14B across ten benchmarks; at step 240 on OpenThoughts-Math, Qwen3-32B overtook Qwen3-14B.Source: Table 9 · p. 23, §B.5 · p. 23, Table 10 · p. 23

B.6 REINFORCEABLE FRACTION AND TEACHER SELECTION; B.7 TEACHER-DEPENDENT TRAINING DYNAMICS

• At step 260, the largest reinforceable fraction selected the best-performing teacher for all three students, but its advantage over teacher size alone was not established.The rule incurred 0.71 points of regret for the 0.6B student at step 200 and zero for the other two; lowest initial reverse KL did not reliably identify the best teacher. During training, reverse KL and positive-supervision fractions followed distinct trajectories, so diagnostics should be assessed alongside accuracy.Source: Table 12 · p. 24, §B.6 · p. 24, §B.7 · p. 24

B.8 OPD CAN LOSE TASK PERFORMANCE DESPITE CONTINUED TEACHER MATCHING.; B.9 EARLY STOPPING AND TRAINING-BUDGET ACCOUNTING

• 33.13% mean accuracy at step 40 was only 0.10 points below individual-run peaks, versus 30.39% at step 200 and 29.75% at step 260.The step-40 state used 80% fewer updates than step 200 and 84.6% fewer than step 260; these retrospective comparisons do not validate a stopping rule for new pairs.Source: §B.9 · p. 25, Table 13 · p. 26

C STUDENT WARM-UP BEFORE DISTILLATION; C.1 SFT WARM-UP

• SFT warm-up used full-parameter training on correct Qwen3-14B-generated trajectories, with checkpoints at 25, 75, 150, 250, and 348 steps.Each checkpoint independently initialized OPD against the same fixed Qwen3-14B teacher, isolating the effect of SFT duration.Source: C STUDENT WARM-UP BEFORE DISTILLATION · p. 26, §C.1 · p. 26

C.2 RLVR WARM-UP

• The RLVR control trained Qwen3-4B-Base with GRPO and binary correctness rewards, sampling eight responses per prompt before distillation from the fixed Qwen3-14B teacher.The downstream OPD stage used DAPO-Math-17K, and its step counter restarted at the handoff.Source: §C.2 · p. 28

C.3 DOWNSTREAM OPD SETUP

• Across the SFT checkpoint sweep, only student initialization changed; all runs used the same reverse-KL OPD recipe and seven-benchmark mean@4 evaluation.The fixed teacher was Qwen3-14B, training data were DAPO-Math-17K, and evaluations used four responses per problem.Source: §C.3 · p. 28

C.4 SENSITIVITY TO SFT WARM-UP DURATION

• 35.39% is the OPD peak after 150 SFT steps, versus 30.87% after 25 steps; longer warm-ups did not raise the peak further.The 150-step run finished at 35.30%, only 0.09 points below its peak, while the 25-step run had a larger subsequent drop; most of the gain appeared by 75–150 steps.Source: Figure 12 · p. 30, §C.4 · p. 30

C.5 INITIALIZATION DIAGNOSTICS

• Student entropy fell from 0.2940 at K = 25 to 0.1507 at K = 250, but the strongest downstream OPD result occurred at K = 150.Initial reverse KL likewise reached its minimum, 0.1028, at K = 150; the first validation-loss plateau and OPD saturation place the useful warm-up region around K = 75–150, as a practical signal rather than a universal stopping rule.Source: §C.5 · p. 30, Table 18 · p. 30

D TEACHER ADAPTATION BEFORE DISTILLATION

• Appendix D documents teacher preparation and downstream OPD settings for §4.2, along with additional results and discussion.Source: D TEACHER ADAPTATION BEFORE DISTILLATION · p. 30

D.1 EXPERIMENTAL SETUP

• The teacher comparison uses Qwen3-14B models adapted by SFT or RLVR, then distills fresh Qwen3-4B-Base students under identical downstream settings.This holds student initialization, OPD objective, learning rate, batch size, sampling, and evaluation constant.Source: §D.1 · p. 31

D.2 TEACHER ADAPTATION WITH SFT

• SFT teacher adaptation uses approximately 16K prompts paired with verified-correct self-generated solutions, controlling adaptation-data scale against RLVR.The teacher is fully fine-tuned for three epochs, and selected checkpoints are evaluated as downstream OPD teachers.Source: §D.2 · p. 31

D.3 RLVR TEACHER ADAPTATION

• RLVR teacher adaptation uses the same approximately 16K OpenThoughts3-Math prompts and GRPO with a binary verifiable-correctness reward.Five checkpoints from the training trajectory each distill a freshly initialized Qwen3-4B-Base student under the same OPD configuration.Source: §D.3 · p. 31

D.4 DOWNSTREAM OPD CONFIGURATION AND EVALUATION

• Downstream OPD uses sampled-token reverse-KL policy gradients on student-generated responses, with task rewards and auxiliary KL regularization disabled.Accuracy is the unweighted mean of mean@4 scores on seven benchmarks, with four responses sampled per evaluation question.Source: §D.4 · p. 32

D.5 TEACHER SELECTION AT FIXED MODEL SIZE

• 29.77% accuracy at OPD step 200 came from the K = 200 teacher, versus 28.00% for the minimum-reverse-KL K = 100 teacher.The 1.77-percentage-point selection regret shows that initial reverse KL alone does not identify the best teacher; ρ+ sign information helps distinguish useful checkpoints at fixed model size.Source: §D.5 · pp. 32–33, Table 22 · p. 32

D.6 ADDITIONAL RESULTS

• With SFT-adapted teachers, OPD runs stay closely aligned through about step 140, then several decline, consistent with little added capability for distillation.More heavily adapted teachers converge to nearly the same accuracy as earlier checkpoints, with only small differences.Source: §D.6 · p. 33, Figure 13 · p. 34
• RLVR teacher checkpoints generally raise OPD accuracy through about step 100; K = 200 performs strongest later, while gains saturate near the teacher’s capability plateau.Less-adapted K = 50 and K = 100 runs become less stable after about step 100; the passage recommends selecting checkpoints using downstream OPD validation rather than automatically using the final checkpoint.Source: §D.6 · p. 33, Figure 13 · p. 34

E DOWNSTREAM RLVR INITIALIZATION

• The appendix compares SFT and OPD as student preparations for the same downstream RLVR procedure, with separate initialization and RLVR configurations and added scale and model-family results.Source: E DOWNSTREAM RLVR INITIALIZATION · p. 33

E.1 EXPERIMENTAL SETTING

• The comparison pairs Qwen3-Base students with Qwen3-14B and a Qwen2.5-1.5B-Base student with Qwen2.5-14B-Instruct, using fixed-teacher SFT or student-sampled OPD before GRPO.Both preparations use OpenThoughts3-Math; downstream settings are held fixed within each student comparison, and the initialization launchers specify batch size 64 and learning rate 10−6.Source: §E.1 · p. 34

E.2 SFT INITIALIZATION

• SFT uses one verified teacher solution per solvable OpenThoughts3-Math prompt, while OPD receives the same unique prompts in the same order, except SFT withholds 256 validation examples.The teacher used to construct solutions matches the family’s later OPD teacher: Qwen3-14B or Qwen2.5-14B-Instruct.Source: §E.2 · p. 34

E.3 OPD INITIALIZATION

• OPD uses student-generated trajectories scored token-by-token by a fixed teacher, applying sampled-token reverse-KL policy gradients to the same ordered prompts as SFT.Students sample at temperature 1.0, and figures compare multiple OPD hand-off checkpoints using the same step sets for each student.Source: §E.3 · p. 34

E.4 COMMON DOWNSTREAM RLVR STAGE

• Both initialization arms enter separate GRPO runs over the full processed DAPO-Math-17K dataset, with held-out core-7 mean@4 accuracy reported across seven reasoning benchmarks.Each prompt is seen once; evaluation samples four responses per question, using temperature 0.6 and top-p 0.95 for Qwen3, and temperature 0.7 for Qwen2.5.Source: §E.4 · pp. 34, 36

E.5 GENERALIZATION ACROSS STUDENT SCALE, MODEL FAMILY, AND HANDOFF CHECKPOINT

• Within each student family, OPD and SFT use the same teacher, RLVR data and recipe, and matched handoff-checkpoint grid.Figure 14 reports held-out core-7 mean@4 accuracy and uses corresponding 14B teachers; its shading shows the minimum–maximum range across handoff checkpoints, not random seeds.Source: §E.5 · p. 36, Figure 14 · p. 37
• The endpoint advantage is reproducible across handoff checkpoints, but the smaller students’ higher starting accuracy does not isolate downstream RLVR gains from matched initial performance.For Qwen3-4B, OPD and SFT start from overlapping accuracy ranges, with SFT slightly stronger on average before RLVR; checkpoint sweeps are not a substitute for variation across random seeds.Source: §E.5 · p. 36
• Across Qwen3-1.7B, Qwen3-4B, and Qwen2.5-1.5B, OPD initialization finished above SFT at every matched handoff checkpoint.Final accuracy ranges for OPD versus SFT were 26.66–27.53% versus 23.13–25.70% for Qwen3-1.7B, 38.53–43.11% versus 33.36–36.80% for Qwen3-4B, and 20.12–21.54% versus 18.69–19.61% for Qwen2.5-1.5B.Source: §E.5 · p. 36, Figure 14 · p. 37

F CASE STUDY: CAN POST-TRAINING COMPOSITION BE RECURSIVE?

• The case study compared four SFT–OPD rounds using progressive, reverse, or fixed RLVR-adapted Qwen3-14B teacher checkpoints, with a Qwen3-1.7B-Base student carrying its learned state across rounds.Each round used 200 SFT steps followed by short OPD; recursive schedules had additional training and were not compute-matched to single-stage OPD and plain RLVR references.Source: F CASE STUDY: CAN POST-TRAINING… · p. 37, Figure 15 · p. 37
• 21.4% four-benchmark accuracy for progressive round 4 exceeded fixed round 4’s 16.6%, although the best recursive result remained 0.9 percentage points below single-stage OPD on the full seven-benchmark evaluation.The progressive schedule had the highest observed endpoint among recursive runs; reverse and fixed schedules did not sustain improvement across rounds.Source: Figure 15 · p. 37, F CASE STUDY: CAN POST-TRAINING… · p. 38
• Under the fixed teacher schedule, four-benchmark accuracy fell from 19.5% to 16.6% as the reinforceable fraction declined from 0.48 to 0.44.Reverse KL decreased from 0.42 to 0.38, with little further change after round 2; repeated exposure to the final teacher did not sustain improvement in this run.Source: F CASE STUDY: CAN POST-TRAINING… · p. 37, Table 26
• Progressive and fixed round 4 both had ρ+ = 0.44, yet their four-benchmark accuracies were 21.4% and 16.6%, respectively.The reinforceable fraction alone therefore does not explain the differences across schedules.Source: F CASE STUDY: CAN POST-TRAINING… · p. 38
• Recursive runs reached the 8,192-token response limit and truncated OPD trajectories, while single-stage OPD did not; one run per schedule limits interpretation of the ordering.The case study does not establish that recursive composition improves over a single OPD stage and motivates testing schedules under matched budgets and response-length conditions.Source: F CASE STUDY: CAN POST-TRAINING… · p. 38

G.1 WHY COMPOSITION MATTERS

• Post-training stages are order-dependent because SFT uses fixed trajectories, whereas RLVR and OPD use states induced by the current policy, changing the data distribution and supervision encountered next.The paper schematically expresses this non-commutativity as UB(UA(π)) ≠ UA(UB(π)).Source: §G.1 · p. 38
• Experiments show that RLVR can strengthen a student while making later OPD destructive, RLVR teacher adaptation changes what OPD transfers, and accuracy-matched SFT and OPD checkpoints respond differently to later RLVR.The SFT and OPD checkpoints differed even when their aggregate entropies overlapped, motivating evaluation of each stage by both added capability and the learning interface it leaves.Source: §G.1 · p. 38

G.2 LIMITATIONS

• The composition study centers on Qwen3-4B students and Qwen3-14B teachers, so it demonstrates controlled stage interaction rather than a universally optimal ordering across models or domains.OPD diagnostics span more student and teacher scales, architectures, objectives, and datasets than the main composition study.Source: §G.2 · p. 38
• RLVR is a more effective teacher-preparation method than the STaR-style control under this protocol, but the comparison does not show greater transferability at matched teacher accuracy.Source: §G.2 · p. 38
• The diagnostics distinguish productive from destructive student–teacher states better than scalar reverse KL, but do not fully explain downstream learnability or the OPD-to-RLVR advantage.ρ+ 0 separates the states where scalar reverse KL does not, and entropy alone does not explain the OPD-to-RLVR advantage.Source: §G.2 · p. 38
• Across controlled interventions, a stage’s effect depends on the state created by the preceding stage; locally improving one stage can worsen conditions for the next.Source: §G.2 · p. 38, §G.3 · p. 39

G.3 UNSUCCESSFUL ATTEMPTS

• RLVR can produce a stronger pre-OPD student than brief SFT yet leave it below its starting accuracy after OPD, so pre-OPD capability alone is a poor warm-up criterion.Source: §G.3 · p. 39
• Additional SFT eventually stops improving downstream OPD, while continued OPD can lower task accuracy despite further improvement in teacher–student KL.More optimization of an intermediate objective is not necessarily better optimization of the complete pipeline.Source: §G.3 · p. 39
• Repeatedly reusing the final teacher did not keep improving the student, whereas progressive teacher refresh improved it across tested rounds; limited replication and trajectory truncation make stale supervision an interpretation, not proof of an optimal algorithm.Source: §G.3 · p. 39