Preserving General Capabilities during Domain Specialization with Uncertainty-Calibrated MOPD

• Vertical specialization improves domain behavior but can degrade reasoning, coding, instruction following, and open-ended communication needed by domain tasks.Source: §1 · p. 1
• Mixing vertical and general data adds corpus-construction and heuristic mixture-tuning costs, while too much general data may weaken specialization.Source: §1 · p. 2
• MOPD combines a domain-expert teacher and a general-expert teacher to integrate both capabilities into one specialized student.Source: §1 · p. 2
• Standard MOPD rarely samples large positive advantages, and advantage sign alone can incorrectly suppress teacher-endorsed tokens.Source: §1 · p. 2, Figure 1 · p. 3
• Uncertainty-calibrated MOPD uses dual-temperature sampling and positive-advantage-density filtering to discover stronger signals, then CLL filtering validates token update directions.Source: §1 · pp. 2–3
• The method analyzes MOPD signal utilization, combines signal discovery with direction-consistency validation, and improves general-capability recovery while preserving or improving vertical-domain performance.Source: §1 · p. 3

• On-policy distillation trains on student-sampled trajectories to reduce exposure mismatch, while recent methods vary in objective geometry and signal usage.Source: §2 · pp. 3–4
• Multi-teacher distillation aggregates complementary teachers or tasks, and MOPD-style systems use domain-selected teacher policies to recover or consolidate skills.Source: §2 · p. 4
• Vertical specialization can improve targeted behavior while degrading broad abilities, a pattern related to catastrophic forgetting in continual learning.Source: §2 · p. 4

Problem Setup

• MOPD routes each prompt to a capability-matched frozen teacher and distills teacher feedback on student-sampled tokens.The teacher is selected by domain label rather than mixing teacher distributions at the token level.Source: §3.1 · p. 4

Learning Opportunity and Update Reliability

• Advantage sign proposes reinforcement or suppression but does not establish whether that direction is reliable under teacher endorsement.A positive advantage may have weak teacher endorsement, while a negative advantage may still correspond to a teacher-plausible token.Source: §3.1 · p. 5, §3.2 · p. 5

Dual-Temperature Sampling

• Dual-temperature sampling broadens student-generated candidates by combining an ordinary anchor with higher-temperature exploration responses.The anchor uses Ta = 1.0, while exploration uses Te > Ta with top-p truncation.Source: §3.3 · p. 6

Positive-Density Trajectory Filtering

• Positive-density filtering retains exploration trajectories whose average positive teacher–student gap is at least as strong as the prompt-matched anchor.The anchor is always retained, and the filter rejects exploration that adds diversity without a commensurate distillation opportunity.Source: §3.4 · p. 6, §3.6 · p. 8

CLL Direction-Consistency Filtering

• Centered log-likelihood (CLL) compares teacher token probability with an entropy-induced typical scale and continuously favors direction–endorsement-consistent updates.Tokens at or above the typical scale receive maximal endorsement; lower-probability tokens receive smoothly weaker endorsement, with retention tendencies depending on update direction.Source: §3.6 · p. 8

Experimental setup

• The evaluation compares uncertainty-calibrated MOPD with Base, SFT, vanilla MOPD, SelecTKD, and ReOPOLD across role-playing, medical, and general-capability measures.General capability is summarized by Gen. Avg., while vertical performance is summarized by Vertical Avg.Source: §4.1 · pp. 8–9, Table 2 · p. 10

Main results

• 53.10 general average and 45.00 role-playing average are achieved by uncertainty-calibrated MOPD, the strongest post-specialization trade-off in role-playing.Compared with vanilla MOPD, the method gains +2.40 points on the general average and +3.78 points on the role-playing average.Source: §4.2 · p. 9
• 54.38 general average is reached in medical specialization, improving over vanilla MOPD by +5.32 points while retaining a 60.65 medical average.The medical average remains close to the best recovery baseline, reported as 60.95.Source: §4.2 · p. 9

Ablations

• CLL masking improves both general and role-playing averages over vanilla MOPD, while dual-temperature sampling and positive-density filtering further improve the trade-off.Positive-density filtering selects trajectories with denser useful positive signal before token-level CLL refinement.Source: §4.3 · p. 10, §4.4 · p. 11

Rollout-budget control

• Under eight sampled responses per prompt, uncertainty-calibrated MOPD exceeds rollout-budget-matched vanilla MOPD on both general and role-playing averages.This supports gains from selecting and filtering trajectory- and token-level signals rather than from a larger sampling budget alone.Source: §4.3 · p. 10, Table 4 · p. 11

Trajectory diagnostics

• 0.5026 to 0.5939 positive-token fraction and 0.0705 to 0.1375 mean positive advantage distinguish retained from rejected trajectories.Figure 4 and Table 5 show that retained trajectories contain more positive-advantage tokens with higher mean positive advantage.Source: Figure 4 · p. 10, §4.4 · p. 11, Table 5 · p. 11

• Positive-advantage magnitude identifies valuable learning opportunities, while advantage–endorsement consistency indicates whether update directions are reliable.Source: §5 · p. 12
• Dual-temperature sampling broadens candidates, positive-density filtering selects stronger trajectories, and CLL retains token updates with consistent direction and endorsement.Source: §5 · p. 12
• Uncertainty-calibrated MOPD improves general-capability recovery across role-playing and medical settings while maintaining vertical-domain performance.In role-playing, vertical-domain performance also improves over recovery baselines; in medical adaptation, medical ability remains close to the best recovery baseline.Source: §5 · p. 12
• Ablations indicate that the gains are not merely due to a larger rollout budget, and diagnostics confirm that trajectory filtering enriches positive learning signals before CLL validation.Source: §5 · p. 12

• Role-playing uses a CoSER-fine-tuned Qwen3-4B-Instruct-2507 student and domain teacher, with the original checkpoint as general teacher.Source: §A.1 · p. 17
• Medical adaptation uses II-Medical-7B-Preview for the student and domain teacher, with Qwen3-8B as general teacher.Source: §A.1 · p. 17
• Both settings use homogeneous, same-scale MOPD, and only student parameters are updated.Source: §A.1 · p. 17
• Student optimization, rollout generation, and teacher inference use BF16 without quantization across four nodes with eight NVIDIA H800 GPUs per node.Pipeline parallelism is 2 for role-playing and 4 for medical adaptation; rollout inference uses tensor parallelism of 1.Source: §A.1 · p. 17

• The role-playing corpus contains 10,000 CoSER prompts and 10,000 general prompts, while the medical corpus contains 9,926 medical and 10,000 general prompts.The mixtures are exactly 1:1 for role-playing and approximately 1:1 for medical adaptation.Source: §A.2 · p. 17
• Role-playing prompts come from the 10,000 longest reserved CoSER conversations, with final continuations generated on-policy and excluded from role-playing SFT.Source: §A.2 · p. 17
• General prompts are sampled from chat, code, math, and STEM subsets of the Nemotron Post-Training Dataset v1 without reference responses in the distillation loss.Source: §A.2 · p. 18
• The medical prompt pool combines four public sources and retains 9,926 valid prompts after preprocessing, without gold answers or explanations during MOPD training.Source: §A.2 · p. 18
• Training prompts are verified not to overlap evaluation instances, and deterministic prompt-level routing preserves source labels through rollout generation.Source: §A.2 · p. 18

• Both settings process 1,024 candidate responses per outer update before positive-density filtering.Source: §A.3 · p. 18
• Role-playing samples eight responses per prompt, whereas medical adaptation samples four, with one anchor and the remainder used for exploration.Source: §A.3 · p. 18
• Table 8 reports training and rollout configuration, with batch sizes defined as global prompt counts before response replication.Source: Table 8 · p. 18

• CLL sample masking stochastically keeps or drops every nonzero-advantage token update using the unified direction-consistency probability.Source: §B.1 · p. 18
• Both CLL variants improve vanilla MOPD on the general average, but sample masking yields a stronger role-playing average and substantially better GPQA than weighting.Weighting gives a slightly higher general average and slightly better scores on some individual benchmarks in the early CLL-only comparison.Source: §B.1 · p. 19, Table 9 · p. 19
• Positive-density filtering outperforms trajectory-likelihood filtering under both strict and non-strict keep rules.The result supports using teacher-student improvement-opportunity density rather than generic teacher likelihood as the useful trajectory-level signal.Source: §B.2 · p. 19

• T = 1.5 gives the strongest role-playing average among the retained exploration settings and is used as the default configuration.Source: §B.3 · p. 19, Table 11 · p. 20
• Higher exploration temperatures occasionally improve individual benchmarks such as LiveCodeBench v5 and LiveBench but are less stable overall.Source: §B.3 · p. 19
• The ablation frames exploration as a trade-off between exposing additional positive learning opportunities and maintaining stability.Source: §B.3 · p. 19