PaperlayerView exact revision on arXiv

Source-linked AI summary

Train4Merge: RL or SFT---Which Yields Better Teachers for OPD-Based Model Merging?

Jingyuan Huang, Zuming Huang, Yucheng Shi, Zhongzhi Li, Xiaoming Zhai, Wei Chu, Ninghao Liu

arXiv:2609.32303v3cs.AIcs.LG

TL;DR

The paper addresses limited systematic evidence on whether SFT or RL produces better OPD teachers. It compares matched teachers from a shared initialization in single-teacher and multi-teacher OPD, finding stronger student outcomes with RL teachers across the tested domains.

  • Problem

    OPD research has focused mainly on improving capability transfer, while the teacher-training algorithm has received less systematic comparison.

    Source: §1 · p. 2

  • Method

    The study compares SFT and RL teachers trained from a shared initialization at similar task performance in single-teacher OPD across three domains and in MOPD.

    Source: §3 · p. 4, §3.1 · p. 4

  • Results

    RL teachers yield stronger students and higher teacher gain recovery across three domains; in MOPD, mean score is 60.04% versus 58.97% and recovery is 138.27% versus 84.51%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A teacher’s own score is an incomplete guide to OPD value, and the findings support prioritizing RL for constructing domain teachers from a shared base model.

    Source: §1 · p. 2

    The hypothesis and comparison are scoped to teachers with shared initialization and comparable task performance.

    Source: §3 · p. 4

Abstract

from arXiv · show

Domain experts trained from a shared checkpoint can transfer their specialized capabilities to a single student through on-policy distillation (OPD). Existing research primarily focuses on improving this merging process, while the algorithms used to train the experts have received limited systematic comparison. We investigate which training algorithm produces teachers better suited to OPD through controlled single-teacher comparisons of supervised fine-tuning (SFT) and reinforcement learning (RL) across Agentic, Reasoning, and Perception. We further examine whether this advantage extends to multi-teacher OPD (MOPD). Teachers and students share the same Qwen3.5-9B initialization, and the two teacher types are compared at similar task performance. Our single-teacher experiments show that RL teachers yield stronger students and higher recovery of teacher performance gains across all three domains. In MOPD, the RL teacher group also yields a student with a higher three-domain mean score and higher overall teacher gain recovery than the SFT teacher group. Our further analysis shows that RL teachers undergo smaller parameter displacements from the shared initialization than SFT teachers. Teacher--student Top-K overlap also increases faster early in OPD or remains consistently higher with RL teachers than with SFT teachers. These findings support the hypothesis that RL teachers' smaller departures from the student's starting point facilitate learning through OPD, resulting in stronger students.

1 INTRODUCTION

The paper asks whether SFT or RL produces more effective OPD teachers when training begins from the same checkpoint, addressing limited systematic comparison of teacher-training algorithms. Controlled single-teacher comparisons and MOPD experiments favor RL teachers, while parameter displacement and prediction alignment support a hypothesis about their advantage.

1 INTRODUCTION

  • Prior OPD research has focused mainly on the transfer algorithm, leaving the teacher-training algorithm less systematically compared despite comparable-performing teachers potentially differing in transfer effectiveness.Existing teacher pipelines use SFT, RL, or combinations of both.Source: §1 · p. 2
  • The study compares SFT and RL teachers with comparable task performance in single-teacher OPD across Agentic, Reasoning, and Perception, then tests whether the findings extend to MOPD.Single-teacher OPD is used first because cross-domain teacher interactions in MOPD could blur the comparison.Source: §1 · p. 2
  • Across three domains, RL teachers yield stronger single-teacher students and higher gain recovery; in MOPD, mean score is 60.04% versus 58.97% and recovery is 138.27% versus 84.51%.The single-teacher advantage also extends to the multi-teacher setting.Source: §1 · p. 2
  • RL teachers show smaller parameter displacements across all three domains, while Top-K overlap rises faster early in Agentic and Perception or stays higher in Reasoning.These patterns, alongside stronger RL-guided students, support the hypothesis that smaller departures from the shared initialization aid prediction alignment through OPD.Source: §1 · p. 2
  • A teacher’s own score is an incomplete guide to OPD value; the findings support prioritizing RL when constructing domain teachers from a shared base model.The contribution is evidence for an RL teacher advantage in controlled single-teacher comparisons and MOPD, with parameter displacement and Top-K dynamics as mechanistic evidence.Source: §1 · p. 2

2 RELATED WORK

Prior work has improved how OPD transfers teacher capabilities, and recent pipelines use varied algorithms to construct experts. This paper addresses the less-studied upstream choice through controlled SFT–RL comparisons.

Capability integration through model merging and OPD

  • OPD research has developed methods for allocating, gating, weighting, or enriching teacher guidance, while the upstream choice of teacher-training algorithm has received less systematic attention.Examples include error-aware supervision allocation and interaction-depth curricula [Yan et al., 2026], and correctness-based filtering or reweighting.Source: §2 · p. 3

Expert construction for subsequent OPD

  • Existing expert-construction pipelines use RL, SFT, or combinations, but do not establish which algorithm yields more effective OPD teachers under comparable conditions.Examples include Xiaomi’s RL-trained experts, SFT-based MAGA and UI-MOPD [Yan et al., 2026; Lian et al., 2026], and mixed-algorithm recipes [DeepSeek-AI et al., 2026; Team et al., 2026].Source: §2 · p. 3

Teacher compatibility and controlled comparison

  • Prior compatibility and parameter-dynamics studies motivate examining teacher training; this work compares SFT and RL from the same initial checkpoint with comparable teachers and students initialized from that checkpoint.The comparison spans Agentic, Reasoning, and Perception and is extended to MOPD.Source: §2 · p. 3

3 METHOD

The method tests whether teacher parameter distance from a shared initialization relates to OPD transfer under comparable teacher performance. It compares SFT and RL in single-teacher OPD and domain-routed MOPD, using task-specific demonstrations or rewards.

3 METHOD

  • The study hypothesizes that, with shared initialization and comparable teacher performance, RL teachers’ smaller parameter displacements may make them easier to learn from through OPD.Controlled single-teacher OPD and MOPD experiments test this hypothesis.Source: §3 · pp. 3–4

3.1 TEACHER CONSTRUCTION AND DISTILLATION

  • Teachers are trained from the shared Qwen3.5-9B initialization using SFT on demonstrations or RL with GRPO rewards, then checkpoints with comparable domain performance are selected.Matching teacher performance reduces its influence when comparing the teacher-training algorithms.Source: §3.1 · p. 4
  • For single-teacher OPD, separate students start from the shared initialization, generate responses, and receive supervision from a fixed teacher on student-generated prefixes.Students continue generating rollouts during training; the teacher-training algorithm is the comparison variable, with inputs and OPD configuration held constant within each domain.Source: §3.1 · p. 4
  • In MOPD, three teachers from one algorithm supervise a shared student, with each example routed to its domain teacher and the students initialized from the shared checkpoint.The student parameters are shared across domains, while the supervising teacher depends on the input domain.Source: §3.1 · p. 4

3.2 TRAINING DATA

  • Within each domain, teacher training and single-teacher OPD use the same task pool, with SFT using demonstrations, RL using task rewards, and OPD using teacher supervision.The Agentic data use demonstration actions or rewarded model-generated actions; Reasoning uses Geo3K problems with chain-of-thought targets or answer-label rewards; Perception uses GUI grounding demonstrations or location rewards.Source: §3.2 · pp. 4–5

4 EXPERIMENTAL SETUP

The experiments compare SFT and RL teachers selected for comparable task performance, with all students initialized from the same Qwen3.5-9B checkpoint. Evaluation covers three domains and measures task performance, teacher-gain recovery, parameter displacement, and teacher–student prediction alignment.

4 EXPERIMENTAL SETUP

  • SFT and RL teachers with comparable task performance are reused in single-teacher OPD and MOPD, with all students initialized from the same Qwen3.5-9B checkpoint.Table 1 summarizes initial student and teacher performance under the domain-specific criteria.Source: §4 · p. 5, Table 1 · p. 5

4.1 EVALUATION BENCHMARKS AND METRICS

  • Agentic uses MobileWorld GUI-only success over 351 tasks; single-teacher Reasoning uses four benchmarks, MOPD uses Geo3K, and Perception averages accuracy across six GUI-grounding benchmarks.Teacher checkpoints are selected by MobileWorld mean success, Geo3K accuracy, or the unweighted six-benchmark Perception mean, respectively.Source: §4.1 · p. 5
  • Distillation is assessed by task performance and teacher-gain recovery, the fraction of teacher gain over the initial model recovered by the student.Recovery is defined only when the teacher beats the initial model; values above 100% mean the student surpasses its teacher. MOPD checkpoints use the unweighted three-domain mean.Source: §4.1 · p. 6

4.2 DIAGNOSTIC METRICS

  • Diagnostic metrics track parameter displacement from shared initialization and teacher–student prediction alignment during OPD.Displacement is measured with L2 norms over the same 760 non-MTP parameter tensors; Top-K overlap compares the highest-probability next-token sets at K = 20, with larger overlap indicating closer agreement.Source: §4.2 · p. 6

5 EXPERIMENTAL RESULTS

The experiments compare SFT and RL teachers in single-teacher OPD across three domains, then test whether the advantage persists in MOPD. RL teachers yield stronger single-teacher students and better overall MOPD results.

5 EXPERIMENTAL RESULTS

  • The study first compares single-teacher OPD across Agentic, Reasoning, and Perception, then extends the comparison to MOPD.Figure 2 shows single-teacher learning curves; gain recovery is reported at each group’s best checkpoint, and per-benchmark results are in Appendix C.Source: §5 · p. 6

5.1 SINGLE-TEACHER OPD

  • RL-guided students outperform SFT-guided students in all three single-teacher domains, with higher teacher gain recovery (Figure 2).In Agentic, the teachers score 22.79% and 23.36%, while their best students score 19.94% and 24.22%, respectively; teacher gain recovery is 44.44% versus 115.00%.Source: §5.1 · p. 7, Figure 2 · p. 7

5.2 MULTI-TEACHER OPD

  • 60.04% versus 58.97% three-domain mean score and 138.27% versus 84.51% overall teacher gain recovery favor the RL teacher group in MOPD (Table 3).The RL-guided student performs better in Agentic and Perception, while the SFT-guided student scores higher in Reasoning; the RL group has a lower three-domain teacher mean.Source: §5.2 · pp. 7–8, Table 3 · p. 7
  • Figure 3 tracks three-domain mean accuracy and overall teacher gain recovery during MOPD training.Solid lines encode mean accuracy on the left axis; dashed lines encode teacher gain recovery on the right axis.Source: §5.2 · p. 7, Figure 3 · p. 8

6 MECHANISTIC ANALYSIS OF THE RL TEACHER ADVANTAGE

The analysis examines whether smaller teacher displacement from shared initialization relates to better OPD outcomes, and whether teacher–student prediction alignment follows similar patterns. RL teachers have smaller displacements, while overlap comparisons and matched Agentic training paths support the proposed relationship.

6 MECHANISTIC ANALYSIS OF THE RL TEACHER ADVANTAGE

  • The analysis hypothesizes that smaller OPD parameter changes favor learning from teachers nearer the shared initialization, enabling students to align more readily with teacher predictions.It examines this hypothesis in parameter space and then output space.Source: §6 · p. 8

6.1 TEACHER DISPLACEMENT AND OPD PERFORMANCE

  • Across Agentic, Reasoning, and Perception, RL teacher displacement is 0.23, 0.20, and 0.37 times the SFT teacher displacement, respectively.In Perception, the displacement difference is smallest and the best-student accuracy gap is 0.86 percentage points, consistent with the hypothesis that displacement may affect transfer.Source: §6.1 · p. 8, Figure 4 · p. 9
  • The three Agentic teacher paths begin from shared initialization, use teachers selected for similar task performance, and train students initialized from the original Qwen3.5-9B checkpoint.The paths are SFT alone, SFT followed by RL, and RL alone.Source: §6.1 · p. 8
  • For three Agentic teacher paths with success rates of 22.79%, 23.36%, and 23.36%, displacements are 2.271, 1.765, and 0.528, while student gain recovery is 44.44%, 65.00%, and 115.00%.Across these paths, larger teacher displacements are associated with lower recovery of teacher gains.Source: §6.1 · p. 8
  • The SFT→RL and direct-RL teachers both score 23.36%, but the direct-RL teacher has smaller displacement and produces a student with 2.85 percentage points higher success rate.The comparison supports the hypothesis that teachers closer to the shared initialization are easier for students to learn from through OPD.Source: §6.1 · p. 9

6.2 TEACHER–STUDENT PREDICTION ALIGNMENT

  • RL teachers show larger early Top-20 overlap increases than SFT teachers in Agentic and Perception; in Reasoning, RL overlap is higher from initialization through every measured checkpoint.Overlap is measured on student rollouts across all three domains (Figure 5).Source: §6.2 · pp. 9–10, Figure 5 · p. 9
  • In Agentic, direct RL and SFT→RL produce total overlap gains of 20.07 and 16.77 percentage points, versus 16.31 for SFT, and final overlaps of 90.17% and 87.73%, versus 87.55%.Direct RL has the smallest displacement, largest overlap gain, and strongest student; SFT→RL falls between direct RL and SFT on all three measures.Source: §6.2 · p. 10

7 CONCLUSION

Across Agentic, Reasoning, and Perception, RL teachers yield stronger students and higher teacher-gain recovery than SFT teachers under shared initialization and comparable teacher performance. Smaller RL-teacher displacements and stronger prediction alignment support the authors’ hypothesis that such teachers are easier for students to follow through OPD.

7 CONCLUSION

  • RL teachers yield stronger students and higher teacher-gain recovery than SFT teachers across Agentic, Reasoning, and Perception under shared initialization and comparable teacher performance.The difference is most pronounced in Agentic.Source: §7 · p. 10
  • RL teachers remain closer to the shared initialization, while their students show faster early prediction alignment or consistently higher output overlap.The authors present these observations as supporting their hypothesis that teachers closer to initialization are easier for students to follow through OPD.Source: §7 · p. 10
  • The findings support prioritizing RL when constructing domain teachers for subsequent OPD.Source: §7 · p. 10

A ADDITIONAL DETAILS FOR THE MECHANISTIC ANALYSIS

Across all three domains, RL teachers have smaller parameter displacements than SFT teachers, while early student displacements are similar between teacher groups. The experiments also use domain-specific training pools, with both student groups sharing each domain’s inputs.

A ADDITIONAL DETAILS FOR THE MECHANISTIC ANALYSIS

B.1 TRAINING DATA

  • Training pools comprise 25,904 Agentic step examples, 2,078 Reasoning problems, and 6,990 Perception examples, with the same input pool used for both student groups within each domain.OPD uses the original prompts and images without reference answers or additional teacher inputs.Source: §B.1 · p. 15

B.2 TEACHER TRAINING CONFIGURATIONS

Teachers share a Qwen3.5-9B starting point and training data, but SFT uses demonstration responses while RL uses GRPO with task rewards. Domain-specific targets, rewards, and sampling settings define the Agentic, Reasoning, and Perception teacher configurations.

B.2 TEACHER TRAINING CONFIGURATIONS

  • All teachers start from Qwen3.5-9B; SFT minimizes token-level cross-entropy on demonstrations, while RL uses GRPO with sequence-level task rewards and no entropy or length terms.No teacher uses LoRA, and all keep the visual encoder and merger frozen during training.Source: §B.2 · p. 15, Table 5 · p. 16
  • For Agentic, SFT targets combine demonstration rationales, action descriptions, and tool calls; RL rewards parsed calls that match the reference action type and arguments.Other parsable calls receive −0.5, and an unparseable call receives −1.Source: §B.2 · p. 15
  • Reasoning SFT uses released chain-of-thought demonstrations, while Reasoning RL rewards matching the Geo3K answer; Perception SFT targets box-center clicks, while its RL reward checks whether the predicted coordinate lies inside the box.Three Reasoning demonstrations longer than 16,384 tokens are excluded, leaving 2,075.Source: §B.2 · pp. 15–16

B.3 STUDENT TRAINING CONFIGURATIONS

Both student groups use shared OPD settings within each domain, with domain-specific objectives and parameter freezing. Checkpoints are selected for comparable teacher task performance and evaluated using domain-specific benchmarks and procedures.

B.3 STUDENT TRAINING CONFIGURATIONS

  • The shared OPD configurations yield optimizer batches of 1,024, 32, and 112 sequences for Agentic, Reasoning, and Perception, respectively.Table 6 summarizes the shared student OPD settings.Source: §B.3 · p. 16, Table 6 · p. 16
  • Agentic uses sampled-token distillation, while Reasoning and Perception use full-vocabulary reverse KL; both teacher groups share each domain’s objective.Reasoning freezes the visual encoder and merger, whereas Agentic and Perception freeze no parameters.Source: §B.3 · p. 16

B.4 CHECKPOINT SELECTION AND EVALUATION

  • Teacher checkpoints are selected for comparable MobileWorld mean success rates, Geo3K accuracies, and Perception six-benchmark unweighted mean accuracies.Student checkpoints use the same evaluation procedures.Source: §B.4 · p. 16
  • MobileWorld evaluation uses greedy decoding for up to 50 interaction steps, while Reasoning evaluates four test sets, including 601 Geo3K problems and three vision-math benchmarks.Reasoning generates one answer per problem with thinking disabled and uses benchmark answer-scoring procedures.Source: §B.4 · p. 17
  • Perception evaluation covers ScreenSpot-v2, ScreenSpot-Pro, UI-Vision, MMBench-GUI L2, OSWorld-G, and OSWorld-G-Refine; cross-benchmark means weight benchmarks equally.MMBench-GUI L2 averages Basic and Advanced accuracies equally.Source: §B.4 · p. 17

C RESULTS ACROSS CHECKPOINTS

Tables 7–10 report benchmark results for OPD and MOPD student checkpoints. The further-directions discussion proposes testing how teacher-training choices interact with domain mixing, routing, and more challenging agentic benchmarks.

C RESULTS ACROSS CHECKPOINTS

D FURTHER DIRECTIONS

  • Future work could test whether teacher-training choice interacts with domain mixing and teacher routing.Source: D FURTHER DIRECTIONS · p. 19
  • MobileWorld results suggest teacher-training choice may be particularly important for agentic tasks.Source: D FURTHER DIRECTIONS · p. 19
  • Further directions include testing the effect on more challenging agentic benchmarks, including long-horizon settings, and across training distributions.The passage also points to data-synthesis methods that improve post-training data diversity as a way to support this study.Source: D FURTHER DIRECTIONS · p. 19

Related papers

Loading 2609.32303v3…