PaperlayerView exact revision on arXiv

Source-linked AI summary

Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation

Tianlei Chen, Jiao Ou, Ziyuan Liu, Ruiming Tang, Jian Liang, Han Li

arXiv:2605.27115v2cs.AI

TL;DR

Domain specialization can weaken general capabilities, and recovery is studied under incomplete prompt coverage using proxy general prompts. CaMOPD combines alternating recovery and preservation updates with gap-based sample selection; across role-play dialogue and medical reasoning QA, it improves general recovery over baselines while maintaining domain capability.

  • Problem

    Domain specialization can weaken general capabilities, and recovery is studied using proxy general prompts under incomplete prompt coverage.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    CaMOPD alternates dedicated general-recovery and domain-preservation updates and selects samples with larger token-level teacher-student log-probability gaps.

    Source: §1 · p. 2

  • Results

    Across role-play dialogue and medical reasoning QA, CaMOPD improves general recovery over baselines while maintaining domain capability.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Gradient and selection analyses suggest CaMOPD reduces mixed-update counteraction and produces more coherent correction signals.

    Source: §5 · p. 9

    Experiments use 4B and 8B models and a fixed 32K rollout budget, leaving larger scales and the effects of varying rollout lengths unexamined.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Domain specialization can improve LLM behavior, but often weakens the general capabilities inherited from the original model. Recent Multi-Teacher On-Policy Distillation (MOPD) pipelines recover model capabilities by supervising student-generated trajectories with teacher feedback, but typically assume teacher-aligned prompt coverage, requiring prompts to match the teachers' training distributions. This assumption is difficult to satisfy when the general teacher is an open-source model whose post-training data are unknown. Instead of attempting to reconstruct this hidden distribution, we study general capability recovery with readily available proxy general prompts. We identify two failure modes of vanilla MOPD in this incomplete-coverage situation: recovery-preservation counteraction from mixing conflicting recovery and preservation gradients, and weak-signal flattening from uniformly averaging samples with unequal correction demand. We propose \textbf{Counteraction-Aware Multi-Teacher On-Policy Distillation} (\textbf{CaMOPD}), which addresses these issues with decoupled alternating training and gap-based sample selection. CaMOPD allocates dedicated updates to general recovery, periodically performs domain-preservation updates, and selects samples with larger averaged token-level teacher-student log-probability gaps to concentrate correction signals. Across role-play dialogue and medical reasoning QA scenarios, CaMOPD outperforms all baselines in general capability recovery while maintaining domain-specific behavior. Gradient coherence analyses further support the intended effect of CaMOPD in producing more coherent correction signals.

1 Introduction

Domain specialization can weaken general capabilities, while MOPD often assumes access to prompts aligned with teachers’ training data. CaMOPD addresses two MOPD failure modes under incomplete prompt coverage and improves general recovery while preserving domain behavior.

1 Introduction

  • MOPD’s teacher-aligned prompt-coverage assumption is difficult to meet when an open-source teacher’s post-training data are unavailable.The paper uses public general-domain prompts as proxies rather than reconstructing the hidden distribution.Source: §1 · pp. 1–2
  • Under incomplete coverage, vanilla MOPD faces recovery-preservation counteraction between gradient streams and weak-signal flattening among prompts from one teacher.The first occurs when conflicting gradients are averaged; the second when small, dispersed gradients flatten larger correction signals.Source: §1 · p. 2, Figure 1 · p. 2
  • Across role-play dialogue and medical reasoning QA, CaMOPD outperforms all baselines in general-capability recovery while maintaining domain-specific behavior.Evaluations across two stylistically distinct vertical domains validate the method.Source: §1 · p. 2
  • CaMOPD combines decoupled alternating training with gap-based sample selection to address gradient counteraction and concentrate correction on higher-demand prompts.Its default schedule uses three general-recovery steps followed by one domain-preservation step.Source: §1 · p. 2

2 Related work

Prior work studies unreliable or imbalanced on-policy feedback, multi-teacher distillation, and ways to mitigate capability forgetting. CaMOPD targets incomplete-coverage prompts while separating recovery and preservation updates.

2.1 On-Policy Distillation

  • On-policy distillation feedback can be unreliable or uneven across trajectories and tokens, prompting work on signal imbalance, teacher-student compatibility, and low-coherence feedback.Other approaches calibrate, relax, select, or reweight local token- or sample-level supervision; CaMOPD instead targets cross-sample gradient coherence.Source: §2.1 · pp. 2–3

2.2 Multi-Teacher On-Policy Distillation

  • Multi-teacher OPD supplies token-level guidance from multiple teachers, and recent pipelines use it for broad capability integration or vertical-domain and staged post-training.Unlike pipelines assuming fully aligned teacher data, CaMOPD targets incomplete coverage with proxy prompts.Source: §2.2 · p. 3

2.3 Mitigating Catastrophic Forgetting

  • Forgetting-mitigation approaches commonly mix general-domain or pretraining data with domain data, or replay prior-task and synthesized examples.CaMOPD shares their capability-preservation objective but organizes recovery and preservation through separate alternating updates.Source: §2.3 · p. 3

3 Method

Under incomplete prompt coverage, vanilla MOPD can counteract recovery and preservation updates and dilute useful correction signals. CaMOPD separates the updates and selects samples by teacher–student gaps.

3.1 Preliminaries

  • The specialized model initializes the student and acts as domain teacher, while the original open-source model provides general-teacher feedback on proxy prompts.The setup targets recovery with proxy general prompts when the teacher's original post-training data are unknown.Source: §3.1 · p. 3

3.2 Failure Modes of Vanilla MOPD

  • Vanilla MOPD can counteract recovery because it mixes general and domain gradients that generally point in conflicting directions during each update.Predominantly negative cross-domain gradient dot products reflect this conflict.Source: §3.2 · p. 4, Figure 2 · p. 4
  • Full-batch averaging can flatten useful signals when low-demand samples dilute high-demand examples and weak-signal examples have poorly aligned gradients.Dispersed directions weaken the coherent gradients most useful for training.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Counteraction-Aware MOPD (CaMOPD)

  • CaMOPD assigns general recovery and domain preservation to separate updates, reducing direct mixed-update counteraction.Its periodic schedule uses multiple recovery steps followed by one preservation step.Source: §3.3 · pp. 4–5
  • CaMOPD ranks samples by branch-specific token-average gaps and selects the smallest prefix covering a target share of cumulative gap scores.General recovery scores absolute gaps; domain preservation scores positive gaps to avoid overpenalizing already plausible domain behaviors.Source: §3.3 · p. 5

4 Experiments

Across role-play dialogue and medical reasoning QA, the experiments test whether CaMOPD recovers general capability while preserving specialized behavior. They compare matched distillation baselines, analyze selection and training dynamics, and examine hyperparameters and ablations.

4.1 Experimental Setup

  • Experiments use Qwen3-4B-Instruct-2507 for role-play and Qwen3-8B for medical QA, with 10K Nemotron proxy prompts for each general-recovery branch.General evaluation spans knowledge and reasoning, coding, instruction following, preference-based generation, and live evaluation; domain evaluations cover role-play and medical QA.Source: §4.1 · pp. 5–6

4.3 Main Results

  • CaMOPD raised the strongest-baseline general averages from 45.26 to 49.07 in Role-Play and from 43.35 to 45.84 in Medical QA, while MOPD methods remained close to or above domain teachers.It achieved best or tied-best results on seven of nine general benchmarks in each specialization setting.Source: Table 1 · p. 6, Table 2 · p. 6, §4.3 · p. 7

4.2 Baselines

  • Baselines include Vanilla MOPD, two OPD methods, and off-policy KD, with matched training settings for the reproduced OPD comparisons.Relaxed OPD clips token rewards and samples by entropy, while SelecTKD proposes and verifies tokens before weighting or masking distillation loss.Source: §4.2 · p. 6

4.4 Training Dynamics and Selection Analysis

  • At the same 80% gap-score mass target, the general branch retains broad coverage in both tasks, while domain selection is sparse for role-play but broader for medical QA.The task-specific difference suggests domain correction demand varies across specialized tasks.Source: Figure 3 · p. 7, §4.4 · pp. 7–8

4.5 Hyperparameter Analysis

  • The 3-1 schedule gives the best recovery-preservation trade-off; recovery scaling of 2 and a gap-score mass target of 0.8 give the strongest aggregate general recovery.The 5-1 schedule achieves similar general recovery, but domain performance starts to decline.Source: Figure 4 · p. 8, §4.5 · p. 8

4.4 Training Dynamics and Selection Analysis

  • Selected-subset gradient coherence gain is predominantly positive in both settings, especially in the role-play domain branch, and smaller but consistent in medical QA.Together with retained-fraction analysis, the results suggest gap-based selection reduces weak-signal flattening and produces more coherent update directions.Source: §4.4 · p. 8

4.6 Ablation Study

  • Across both settings, general and domain averages decrease at each successive matched ablation as selection, scaling, and then decoupled updates are removed.The ablation chain changes one component at a time; the final variant uses mixed updates with a 3:1 ratio of general to domain samples.Source: Table 3 · p. 8, §4.6 · p. 8

5 Conclusion

The conclusion studies general-capability recovery after specialization when only proxy general prompts are available. It proposes CaMOPD to address two Vanilla MOPD failure modes and reports improved recovery while maintaining domain capability.

5 Conclusion

  • The study identifies recovery-preservation counteraction and weak-signal flattening as two failure modes of Vanilla MOPD under incomplete coverage.The setting provides only proxy general prompts for the general teacher.Source: §5 · p. 8
  • CaMOPD combines alternating recovery/preservation updates with gap-based sample selection to address recovery-preservation counteraction and weak-signal flattening.The conclusion studies this approach under incomplete prompt coverage, where only proxy general prompts are available for the general teacher.Source: §5 · pp. 8–9
  • Experiments in role-play dialogue and medical reasoning QA show improved general recovery over baselines while maintaining domain capability.Gradient and selection analyses further suggest reduced mixed-update counteraction and more coherent correction signals.Source: §5 · p. 9

Limitations

The experiments use 4B and 8B models and a fixed 32K response-length budget. Larger scales and the effect of alternative rollout budgets remain unstudied.

Limitations

  • Experiments use 4B models for role-play and 8B models for medical reasoning QA; larger model scales are not evaluated.Extending CaMOPD to larger specialized models is identified as future work.Source: Limitations · p. 9
  • On-policy rollouts use a fixed 32K response-length budget, so the experiments do not test how different budgets affect the recovery-preservation trade-off.The budget was chosen to avoid artificially limiting long-reasoning models' output capacity during distillation.Source: Limitations · p. 9
  • Shorter and adaptive rollout budgets are left for future study.The paper specifically notes that their effects on the recovery-preservation trade-off were not studied.Source: Limitations · p. 9

Ethical Considerations

The appendix describes the model and prompt sources, data roles, and training streams used for role-play and medical reasoning QA. It also limits the medical setting to research and educational benchmarks rather than clinical advice.

Ethical Considerations

  • Medical models and evaluations are used only for research on medical QA and educational reasoning benchmarks, not professional medical advice, diagnosis, or treatment.Clinical or patient-facing deployment would require stronger validation, expert oversight, safety guardrails, and regulatory compliance; training uses public datasets and no private patient records or personally identifiable medical information.Source: Ethical Considerations · p. 9

A.1 Training Sources

  • Role-play starts from Qwen3-4B-Instruct-2507, with a CoSER-fine-tuned teacher; medical QA uses Qwen3-8B and II-Medical-8B, and Nemotron supplies sampled general prompts.CoSER contains multi-turn, multi-character conversations built from 771 novels; Nemotron is sampled proportionally across its general-domain categories.Source: §A.1 · p. 12

A.2 CoSER Split for Specialization and Domain Preservation

  • The role-play preservation set reserves 10K longest-turn CoSER conversations, using all but each final turn as context; the remaining 300K conversations train the role-play teacher.Reserved conversations are excluded from role-play SFT, and their long contexts test persona, dialogue-state, and storyline consistency.Source: §A.2 · p. 12

A.3 Nemotron Sampling for General Recovery

  • Because the open-source teacher's training data are unknown, the study samples 10K prompts proportionally from Nemotron's 25.66M examples as proxy general prompts.This creates the incomplete-coverage recovery setting studied in the experiments.Source: §A.3 · p. 12

A.4 Medical Reasoning QA Model and Domain-Preservation Prompts

  • Medical preservation is prompt-only: the student generates on-policy continuations and the domain teacher supplies token-level feedback, without gold answers during training.The 10K prompts draw from MedMCQA, MedReason, Medical-R1-Distill-Data, and m23k-tokenized.Source: §A.4 · pp. 12–13

A.5 Data Examples

  • Figures 5–7 depict the Nemotron general-recovery prompt structure, CoSER role-play conversation structure, and medical domain-preservation template.The figures show the field structure and training usage of the three streams.Source: Figure 5 · p. 13, Figure 6 · p. 13, Figure 7 · p. 13

A.6 Resulting Data Roles

  • Table 4 separates specialization data, domain-preservation prompts, and proxy general-recovery prompts by their roles in the training pipeline.The general branch uses a compact proportional Nemotron sample, while domain-preservation prompts provide gradients for the active vertical domain.Source: §A.6 · p. 13

A.7 Training Protocol

The training protocol specifies software, default CaMOPD settings, and the evaluation and gradient-analysis procedures. The default schedule uses three recovery steps for each preservation step.

A.7 Training Protocol

  • SFT uses ms-swift, while the MOPD training pipeline is implemented with verl.Source: §A.7 · p. 13
  • Runs train for 120 steps with prompt batches of 512 and a 2 × 10^-6 learning rate; default CaMOPD uses a 3-1 recovery-to-preservation schedule.The default gap-score mass targets are ρg = ρd = 0.8, with recovery scale rg = 2 and domain scale rd = 1; each run uses 32 H100 GPUs for about seven hours.Source: §A.7 · p. 13
  • For analyses in Figures 1 and 3, gradients are computed over all model parameters at each analyzed step and aggregated across parameter shards.The aggregation accounts for tensor and pipeline parallelism.Source: §A.7 · p. 13
  • Table 4 summarizes dataset roles, while Table 5 specifies general-benchmark scoring settings, including generation temperature, sample count, and pass@k.LLM-based verification handles objective answer extraction and equivalence checking; GPT-4.1 judges preference-based generation benchmarks.Source: Table 5 · p. 14, Table 4 · p. 14

B Evaluation Protocols

The protocols combine benchmark-specific scoring with distinct procedures for role-play preservation and medical reasoning QA.

B Evaluation Protocols

  • General-capability benchmarks use exact-answer verification, official rule-based or execution checkers, or an LLM judge for preference-based generation.Arena-Hard v2 uses GPT-4.1 with its automatic judging pipeline.Source: B Evaluation Protocols · pp. 14–15
  • For GPQA-Diamond and HMMT25, Qwen2.5-14B-Instruct extracts answers and checks equivalence; its decisions matched official extraction on all 100 validation outputs.The validation subset contained 50 outputs from each benchmark.Source: B Evaluation Protocols · pp. 14–15

B.1 Role-Play Evaluation Protocols

  • Role-play preservation uses CoSER’s GCA protocol, with Qwen2.5-72B-Instruct as environment model, next-speaker predictor, and judge; the four GCA dimensions are averaged.The models had the same ranking under the official GPT-4o-based configuration, but cost led the study to use the Qwen configuration for reported evaluations.Source: §B.1 · p. 15

B.2 Medical Reasoning QA Evaluation Protocols

  • Medical reasoning QA preservation is evaluated on MedXpertQA Text and MedQA-USMLE using extracted final-answer option letters and exact-match accuracy.The protocols use the full 2,450-example MedXpertQA Text test split and a 1,273-example MedQA-USMLE test split.Source: §B.2 · p. 15

C Hyperparameter Analysis

The hyperparameter analysis compares recovery frequency and gap-signal settings, selecting a configuration that balances general recovery with role-play preservation.

Effect of Recovery-to-Preservation Schedule

  • The 3-1 recovery-to-preservation schedule gives the best trade-off; 5-1 has similar general recovery but lower role-play average, while 1-1 is more conservative for recovery and also lowers domain average.Table 7 reports general and role-play capability across the compared schedules and general-recovery hyperparameters.Source: C Hyperparameter Analysis · p. 16, Table 7 · p. 16

Effect of Recovery Scaling and Gap-score Mass Target

  • With a fixed 3-1 schedule, r_g = 2 and ρ_g = 0.8 gives the strongest aggregate general recovery.A larger gap-score mass target covers a broader set of informative samples, while recovery scaling amplifies selected high-demand samples’ gradient signal.Source: C Hyperparameter Analysis · p. 16

Trade-off Across General and Domain Performance

  • The main experiments use a 3-1 schedule, r_g = 2, and ρ_g = 0.8, which balances general recovery and role-play preservation across the evaluated benchmark families.The analysis compares schedule and hyperparameter variants while keeping teachers, data, and other optimization settings unchanged.Source: C Hyperparameter Analysis · pp. 15–16

D Detailed Ablation Results

The detailed ablation tables report per-metric results for matched variants in the role-play and medical reasoning QA settings.

Matched ablation variants

Role-Play dialogue instantiation

  • Table 8 presents detailed ablation results for the role-play dialogue instantiation across four matched variants.Higher is better for all metrics, and bold indicates the best result among the compared variants.Source: Table 8 · p. 17

Medical Reasoning QA instantiation

  • Table 9 presents detailed ablation results for the medical reasoning QA instantiation across four matched variants.Higher is better for all metrics, and bold indicates the best result among the compared variants.Source: Table 9 · p. 17
Loading 2605.27115v2…