PaperlayerView exact revision on arXiv

Source-linked AI summary

Preserving General Capabilities during Domain Specialization with Uncertainty-Calibrated MOPD

Ziyuan Liu, Jiao Ou, Jian Liang, Ruiming Tang, Cheng Luo

arXiv:2608.26735v1cs.CL

TL;DR

Vertical specialization can degrade general capabilities, while standard MOPD struggles to discover strong positive learning signals and to determine whether advantage-based updates are reliable. The paper proposes uncertainty-calibrated MOPD, combining exploratory trajectory selection with CLL-based token validation, and reports stronger general-capability recovery across role-playing and medical specialization while maintaining vertical performance.

  • Problem

    Standard MOPD rarely exposes tokens with large positive teacher–student advantages, and advantage sign alone does not establish whether an update direction is reliable.

  • Method

    Uncertainty-calibrated MOPD uses dual-temperature sampling and positive-advantage-density filtering for signal discovery, followed by CLL direction-consistency gating for token updates.

  • Results

    Across role-playing and medical specialization, uncertainty-calibrated MOPD achieves stronger general-capability recovery than standard MOPD while maintaining vertical-domain performance.

  • Takeaways & Limitations

    Trajectory filtering enriches positive learning signals before unified CLL validation, supporting the paper’s uncertainty-aware view of MOPD signal discovery and validation.

  • Takeaways & Limitations

    The theoretical properties justify local filter behavior but do not provide a global convergence guarantee for neural policy optimization.

Abstract

from arXiv · show

Specializing large language models to vertical domains improves domain-specific behavior but often degrades general capabilities such as reasoning, coding, instruction following, and creative writing. We study this domain--general trade-off in Multi-Teacher On-Policy Distillation (MOPD), where a specialized student is supervised on its own sampled trajectories by domain and general teachers. Standard MOPD faces two limitations: ordinary on-policy sampling rarely exposes tokens with large positive teacher--student advantages, while the advantage sign alone does not establish whether the resulting update direction is reliable. We propose uncertainty-calibrated MOPD to address these limitations. Dual-temperature sampling broadens the candidate trajectory pool, and positive-advantage-density filtering selects trajectories with stronger positive learning signals. Centered log-likelihood (CLL) filtering then computes an entropy-calibrated teacher-endorsement score and probabilistically retains token updates according to direction--endorsement consistency. Experiments on role-playing and medical-domain specialization show that our method improves the general-capability average over standard MOPD by $4.73\%$ and $10.84\%$, respectively, while maintaining vertical-domain performance. Ablations and diagnostic analyses further confirm that the gains do not merely result from a larger rollout budget and that the proposed trajectory- and token-level mechanisms address their intended failure modes.

1 INTRODUCTION

Vertical specialization can damage general capabilities, motivating MOPD as a way to combine domain and general expertise without reconstructing unavailable general-model data. Uncertainty-calibrated MOPD addresses standard MOPD’s weak signal discovery and unreliable advantage directions through trajectory exploration, filtering, and teacher-endorsement validation.

  • Vertical specialization improves domain behavior but can degrade reasoning, coding, instruction following, and open-ended communication needed by domain tasks.
  • Mixing vertical and general data adds corpus-construction and heuristic mixture-tuning costs, while too much general data may weaken specialization.
  • MOPD combines a domain-expert teacher and a general-expert teacher to integrate both capabilities into one specialized student.
  • Standard MOPD rarely samples large positive advantages, and advantage sign alone can incorrectly suppress teacher-endorsed tokens.
  • Uncertainty-calibrated MOPD uses dual-temperature sampling and positive-advantage-density filtering to discover stronger signals, then CLL filtering validates token update directions.
  • The method analyzes MOPD signal utilization, combines signal discovery with direction-consistency validation, and improves general-capability recovery while preserving or improving vertical-domain performance.

2 RELATED WORK

Related work situates the paper within on-policy distillation, multi-teacher capability integration, and the broad problem of catastrophic forgetting during vertical specialization.

  • On-policy distillation trains on student-sampled trajectories to reduce exposure mismatch, while recent methods vary in objective geometry and signal usage.
  • Multi-teacher distillation aggregates complementary teachers or tasks, and MOPD-style systems use domain-selected teacher policies to recover or consolidate skills.
  • Vertical specialization can improve targeted behavior while degrading broad abilities, a pattern related to catastrophic forgetting in continual learning.

3 METHOD

The method combines on-policy distillation with trajectory-level signal discovery and token-level validation. It broadens exploration, retains candidates with strong prompt-relative positive signal, and uses entropy-adaptive teacher endorsement to filter update directions.

  • Problem Setup: MOPD routes each prompt to a capability-matched frozen teacher and distills teacher feedback on student-sampled tokens.The teacher is selected by domain label rather than mixing teacher distributions at the token level.
  • Learning Opportunity and Update Reliability: Advantage sign proposes reinforcement or suppression but does not establish whether that direction is reliable under teacher endorsement.A positive advantage may have weak teacher endorsement, while a negative advantage may still correspond to a teacher-plausible token.
  • Dual-Temperature Sampling: Dual-temperature sampling broadens student-generated candidates by combining an ordinary anchor with higher-temperature exploration responses.The anchor uses Ta = 1.0, while exploration uses Te > Ta with top-p truncation.
  • Positive-Density Trajectory Filtering: Positive-density filtering retains exploration trajectories whose average positive teacher–student gap is at least as strong as the prompt-matched anchor.The anchor is always retained, and the filter rejects exploration that adds diversity without a commensurate distillation opportunity.
  • CLL Direction-Consistency Filtering: Centered log-likelihood (CLL) compares teacher token probability with an entropy-induced typical scale and continuously favors direction–endorsement-consistent updates.Tokens at or above the typical scale receive maximal endorsement; lower-probability tokens receive smoothly weaker endorsement, with retention tendencies depending on update direction.

4 EXPERIMENTS

Experiments in role-playing and medical specialization show that uncertainty-calibrated MOPD improves general-capability recovery while preserving strong vertical-domain performance. Ablations and diagnostics attribute the gains to trajectory exploration and filtering plus token-level endorsement validation, rather than rollout budget alone.

  • Experimental setup: The evaluation compares uncertainty-calibrated MOPD with Base, SFT, vanilla MOPD, SelecTKD, and ReOPOLD across role-playing, medical, and general-capability measures.General capability is summarized by Gen. Avg., while vertical performance is summarized by Vertical Avg.
  • Main results: 53.10 general average and 45.00 role-playing average are achieved by uncertainty-calibrated MOPD, the strongest post-specialization trade-off in role-playing.Compared with vanilla MOPD, the method gains +2.40 points on the general average and +3.78 points on the role-playing average.
  • Main results: 54.38 general average is reached in medical specialization, improving over vanilla MOPD by +5.32 points while retaining a 60.65 medical average.The medical average remains close to the best recovery baseline, reported as 60.95.
  • Ablations: CLL masking improves both general and role-playing averages over vanilla MOPD, while dual-temperature sampling and positive-density filtering further improve the trade-off.Positive-density filtering selects trajectories with denser useful positive signal before token-level CLL refinement.
  • Rollout-budget control: Under eight sampled responses per prompt, uncertainty-calibrated MOPD exceeds rollout-budget-matched vanilla MOPD on both general and role-playing averages.This supports gains from selecting and filtering trajectory- and token-level signals rather than from a larger sampling budget alone.
  • Trajectory diagnostics: 0.5026 to 0.5939 positive-token fraction and 0.0705 to 0.1375 mean positive advantage distinguish retained from rejected trajectories.Figure 4 and Table 5 show that retained trajectories contain more positive-advantage tokens with higher mean positive advantage.

5 CONCLUSION

The paper frames capability recovery as selecting token-level learning signals in MOPD according to both positive advantage and update reliability. Uncertainty-calibrated MOPD combines broader sampling, trajectory filtering, and direction-consistency filtering, improving general-capability recovery across role-playing and medical specialization while preserving domain performance.

  • Positive-advantage magnitude identifies valuable learning opportunities, while advantage–endorsement consistency indicates whether update directions are reliable.
  • Dual-temperature sampling broadens candidates, positive-density filtering selects stronger trajectories, and CLL retains token updates with consistent direction and endorsement.
  • Uncertainty-calibrated MOPD improves general-capability recovery across role-playing and medical settings while maintaining vertical-domain performance.In role-playing, vertical-domain performance also improves over recovery baselines; in medical adaptation, medical ability remains close to the best recovery baseline.
  • Ablations indicate that the gains are not merely due to a larger rollout budget, and diagnostics confirm that trajectory filtering enriches positive learning signals before CLL validation.

A.1 MODELS AND SYSTEMS

The experiments use same-scale Qwen3 students and teachers, with teachers differing by specialization rather than model size. Student optimization and teacher or rollout inference use BF16 systems with specified parallelism across four eight-GPU nodes.

  • Role-playing uses a CoSER-fine-tuned Qwen3-4B-Instruct-2507 student and domain teacher, with the original checkpoint as general teacher.
  • Medical adaptation uses II-Medical-7B-Preview for the student and domain teacher, with Qwen3-8B as general teacher.
  • Both settings use homogeneous, same-scale MOPD, and only student parameters are updated.
  • Student optimization, rollout generation, and teacher inference use BF16 without quantization across four nodes with eight NVIDIA H800 GPUs per node.Pipeline parallelism is 2 for role-playing and 4 for medical adaptation; rollout inference uses tensor parallelism of 1.

A.2 DATA CONSTRUCTION AND ROUTING

Recovery corpora combine vertical-domain and general prompts, shuffle them globally, and route examples deterministically by source label. The role-playing and medical mixtures are approximately balanced while using prompt-only on-policy distillation inputs.

  • The role-playing corpus contains 10,000 CoSER prompts and 10,000 general prompts, while the medical corpus contains 9,926 medical and 10,000 general prompts.The mixtures are exactly 1:1 for role-playing and approximately 1:1 for medical adaptation.
  • Role-playing prompts come from the 10,000 longest reserved CoSER conversations, with final continuations generated on-policy and excluded from role-playing SFT.
  • General prompts are sampled from chat, code, math, and STEM subsets of the Nemotron Post-Training Dataset v1 without reference responses in the distillation loss.
  • The medical prompt pool combines four public sources and retains 9,926 valid prompts after preprocessing, without gold answers or explanations during MOPD training.
  • Training prompts are verified not to overlap evaluation instances, and deterministic prompt-level routing preserves source labels through rollout generation.

A.3 TRAINING CONFIGURATION

The training configuration distinguishes prompt batches from replicated responses and uses anchor and exploration rollouts. Both settings process 1,024 candidate responses per outer update before positive-density filtering.

  • Both settings process 1,024 candidate responses per outer update before positive-density filtering.
  • Role-playing samples eight responses per prompt, whereas medical adaptation samples four, with one anchor and the remainder used for exploration.
  • Table 8 reports training and rollout configuration, with batch sizes defined as global prompt counts before response replication.

B.1 CLL SAMPLE MASKING VERSUS WEIGHTING

The CLL sample-masking variant stochastically retains nonzero-advantage token updates, while a weighting variant keeps all tokens and scales updates by the same probability. Both improve vanilla MOPD on the general average, but their strengths differ across role-playing and GPQA, and positive-density filtering outperforms trajectory-likelihood filtering.

  • CLL sample masking stochastically keeps or drops every nonzero-advantage token update using the unified direction-consistency probability.
  • Both CLL variants improve vanilla MOPD on the general average, but sample masking yields a stronger role-playing average and substantially better GPQA than weighting.Weighting gives a slightly higher general average and slightly better scores on some individual benchmarks in the early CLL-only comparison.
  • Positive-density filtering outperforms trajectory-likelihood filtering under both strict and non-strict keep rules.The result supports using teacher-student improvement-opportunity density rather than generic teacher likelihood as the useful trajectory-level signal.

B.3 EXPLORATION-TEMPERATURE SENSITIVITY

The exploration-temperature ablation finds that moderate exploration is most effective for positive-density trajectory filtering. T = 1.5 is selected as the default because it gives the strongest role-playing average, whereas higher temperatures are less stable overall.

  • T = 1.5 gives the strongest role-playing average among the retained exploration settings and is used as the default configuration.
  • Higher exploration temperatures occasionally improve individual benchmarks such as LiveCodeBench v5 and LiveBench but are less stable overall.
  • The ablation frames exploration as a trade-off between exposing additional positive learning opportunities and maintaining stability.
Loading 2608.26735v1…