PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs

Xixiang He, Xingming Li, Baiqi Wu, Qiyao Sun, Xuanyu Ji, Ao Cheng, Qingyong Hu

arXiv:2609.02548v1cs.LGcs.AI

TL;DR

Multi-domain LLM post-training needs to combine specialized teachers into one deployable model, but domain-matched teachers are not always correct on individual samples. MT-SDPO verifies teacher answers per sample, combines eligible feedback with student self-anchors through a privileged EMA self-teacher, and reports stronger balance, including a 14.79-point worst-domain gain for Qwen3-8B. The method’s scope is limited by its reliance on exact-answer verification and samples without any correct teacher.

  • Problem

    Domain-matched routing assumes the teacher for a prompt’s domain is correct on every sample, despite teacher reliability varying across samples.

  • Method

    MT-SDPO uses self-anchors, answer-verified per-sample teacher eligibility, and privileged distillation through an EMA self-teacher into one student.

  • Results

    Across five students from three families, MT-SDPO improves balance; for Qwen3-8B, worst-domain accuracy rises by 14.79 points and the domain gap falls from 20.96 to 5.30.

  • Takeaways & Limitations

    Teacher supervision should be allocated by verified sample-level reliability rather than domain membership.

  • Takeaways & Limitations

    Exact-answer eligibility is limited to settings with a reliable task-appropriate verifier, and one third of samples admit no correct teacher.

Abstract

from arXiv · show

Modern large language models (LLMs) rely on reinforcement learning to build strong capabilities in individual domains, but integrating those capabilities into a single deployable model remains challenging. By routing each sample to the teacher whose domain matches it, existing approaches let a domain label decide which teacher provides supervision. However, domain expertise holds only on average: the matched teacher is not always correct on a given sample, while a teacher from another domain sometimes is. The reliable teacher therefore has to be identified per sample, not per domain. In this paper, we introduce Multi-Teacher Self-Distillation Policy Optimization (MT-SDPO), an on-policy distillation method that unifies several frozen teachers into one student model. MT-SDPO consists of three components: (1) self-anchors, where a rollout is supervised by a correct rollout from its own group; (2) answer-verified eligibility, where a teacher may supervise a sample only if its own answer passes a verifier; and (3) privileged distillation, which merges the anchor and all verified feedback into one context that an exponential moving average self-teacher reads and the student does not, thereby keeping one policy at deployment. Across five students from three model families, MT-SDPO lifts the weakest domain of Qwen3-8B by 14.79 points and narrows its domain gap by 74.7%, a better balance than serving one matched teacher per domain. Verified reliability, not domain membership, should decide who teaches. Code is available at https://github.com/hexixiang/MT-SDPO.

1 Introduction

Multi-domain post-training must consolidate specialized teachers into one deployable policy, but domain-matched supervision can be wrong on individual samples. MT-SDPO instead verifies teacher answers per sample and distills eligible feedback with self-anchoring into one student.

  • Motivation: Domain-specific RL produces separate teachers, while deployment requires one policy that handles requests across domains.Each domain uses its own data and rollout protocol, creating a consolidation problem for multi-domain post-training.
  • Motivation: 52.9% of samples have a correct matched teacher, compared with 65.7% having at least one correct teacher under per-sample selection.The matched teacher is not reliably correct for every sample, and cross-domain teachers can sometimes provide the correct answer.
  • Method: MT-SDPO makes any teacher with a verified-correct answer eligible for that sample, while excluding unverified sources.The student’s own correct rollout also acts as a verified self-anchor, and frozen teachers are consulted where the student fails.
  • Results: 14.79 points in worst-domain accuracy and a domain-gap reduction from 20.96 to 5.30 are reported for Qwen3-8B.The evaluation uses five student models from three families across chemistry, materials science, and physics; Qwen3-8B also gains 4.64 Macro accuracy points.
  • Contributions: Sample-level diagnosis, MT-SDPO, and same-initialization ablations provide the paper’s main contributions and controlled evidence.The diagnosis separates matched coverage, cross-domain rescue, and failure across all teachers.
  • Method: MT-SDPO combines self-anchors, answer-verified teacher eligibility, and privileged self-distillation into one deployable student policy.An EMA self-teacher reads the anchor and eligible feedback, while the student retains the single policy used at deployment.

2 Related Work

Prior work consolidates multi-domain capabilities, studies on-policy and self-distillation, and combines multiple teachers, but MT-SDPO allocates supervision by verified correctness rather than domain labels or learned weights.

  • Capability Consolidation for Multi-Domain LLMs: Prior capability-consolidation routes include multi-domain RL, off-policy teacher-completion finetuning, weight-space merging, and mixed-domain training.MOPD is the closest comparison: it routes student rollouts to frozen teachers matched by prompt domain.
  • On-Policy and Self-Distillation: On-policy distillation trains on student-generated rollouts, while self- and context-distillation use signals produced or exposed through additional model context.These approaches differ from classical fixed-corpus, off-policy distillation.
  • Multi-Teacher Distillation and Supervision Reliability: Multi-teacher distillation includes output mixing, per-sample routing, and adaptive teacher weighting, alongside verifiable signals for correctness.The related-work framing connects teacher combination with reliability-based supervision.
  • Multi-Teacher Distillation and Supervision Reliability: MT-SDPO selects teachers by verified correctness per sample, allows cross-domain teachers to help when correct, and combines all verified teachers with a self-anchor.This distinguishes the method from domain-label routing and learned teacher weights.

3 Preliminaries

SDPO trains a student on its own on-policy rollouts while an EMA self-teacher reads privileged context. MT-SDPO extends this setup to frozen multi-domain teachers and verifier-based feedback allocation.

  • Self-Distillation Policy Optimization: SDPO samples an on-policy rollout y from the student policy πθ for each prompt x.The rollout is the sequence on which the student is trained.
  • Self-Distillation Policy Optimization: The self-teacher is a slowly updated copy of the student that observes privileged context c, unlike the student distribution used for the rollout.The context changes the self-teacher distribution without changing the rollout being trained.
  • Multi-Teacher Setting: Each training instance includes a prompt, a verifiable reference answer, and a domain label, with one student learning from K frozen domain teachers.Teachers can provide private answers or critiques of a given rollout.
  • Multi-Teacher Setting: A programmatic verifier extracts an answer from a completion and compares it with the reference answer to determine eligibility.The reference answer is used by the verifier and sanitizer, not inserted into student or teacher prompts.

4 Method

MT-SDPO consolidates frozen domain teachers into one deployable policy by selecting supervision per sample through verified answers, then distilling privileged feedback alongside self-anchors.

  • Answer-Verified Teacher Eligibility: MT-SDPO consolidates K frozen domain teachers into one deployable policy using a sample-level eligibility rule based on verified correctness.The student’s own correct rollout also serves as a supervision source.
  • Rollout-Specific Self-Anchors: A rollout group supplies self-anchors when a correct peer exists, allowing incorrect rollouts to reuse a correct solution while excluding the rollout itself.The earliest correct peer is selected and truncated to the anchor token budget; self-exclusion prevents sequence leakage.
  • Answer-Verified Teacher Eligibility: Teachers are queried only for verifier-marked incorrect rollouts, and any teacher whose private answer is verified correct enters the sample-specific eligible set regardless of domain.The nominal-domain teacher can be excluded, while a teacher from another domain can become eligible.
  • Answer-Verified Teacher Eligibility: Eligible teachers provide independently generated, sanitized feedback without majority voting or domain-based weighting.Sanitization rejects label-revealing messages and requires a question-linked concept anchor, but its lexical gates do not certify semantic validity.
  • Privileged Distillation and Student Update: The EMA self-teacher reads a privileged context combining the self-anchor and surviving teacher feedback, while the student observes neither and is the only updated model.The context falls back to self-anchor-only or teacher-only cases; when both sources are absent, the rollout contributes no distillation loss.
  • Privileged Distillation and Student Update: Training uses a masked, importance-weighted token-level divergence between the student’s question-only distribution and the EMA self-teacher’s context-conditioned distribution.The EMA teacher uses a shared rollout prefix for token alignment, and symmetric Jensen–Shannon divergence is used for distillation.

5 Experiments

Experiments diagnose per-sample teacher reliability, evaluate MT-SDPO across five students and controlled baselines, and ablate verification, reassignment, and aggregation. MT-SDPO substantially improves Qwen3-8B balance, while gains depend on remaining weakest-domain headroom.

  • Experimental Setup: The study asks whether domain identifies the correct teacher, whether MT-SDPO integrates capabilities into one policy, and how its supervision components contribute.Experiments use SciKnowEval L3 chemistry, materials science, and physics questions with five students from three model families and controlled online methods.
  • Teacher-Pool Diagnosis: 65.69% per-sample teacher coverage exceeds the matched teacher’s 52.94%, with 27.09% of matched misses rescued and the largest rescue in physics at 18.74 points.About half of samples have two or three correct teachers, while 34.31% have none, motivating self-anchors alongside verified feedback.
  • Main Capability-Integration Results: 14.79 points in Worst-domain accuracy and 4.64 Macro points over Multi-Domain Post-Training reduce Qwen3-8B’s domain gap from 20.96 to 5.30.Chemistry, the weakest domain, rises 18.59 points, while materials and physics together give back under five.
  • Main Capability-Integration Results: MT-SDPO beats the three-model matched-teacher reference by 3.79 Macro and 11.15 Worst-domain points with one model at inference.Domain Routing reaches the same 5.30 gap but remains 2.90 and 2.05 points lower on the two accuracy metrics.
  • Main Capability-Integration Results: Domain-gap reductions occur at all three Qwen3 scales, but Macro improves only at 8B; at smaller scales, flattening mainly reflects declines in stronger domains.The gap falls by 49.6%, 41.5%, and 74.7% at 1.7B, 4B, and 8B respectively, while weakest-domain gains at 8B and OLMo carry Macro upward.
  • Ablation Studies: Verification, cross-domain reassignment, and aggregation each matter: unverified feedback loses 6.56 Macro and 8.94 Worst-domain points, while full aggregation restores balance.Removing aggregation widens the gap from 5.14 to 7.99, whereas full aggregation brings it to 5.30; unverified teacher answers are incorrect in 50.21% of pairs.

6 Conclusion

The conclusion finds that allocating teachers per sample outperforms domain-based allocation, while identifying limits from unavailable correct teachers, limited headroom, and exact-answer verification.

  • MT-SDPO allocates a teacher pool per sample rather than per domain.
  • Sample-level verification recovers a quarter of the samples missed by domain routing.
  • Cross-domain reassignment improves accuracy at the cost of balance, whereas aggregating every eligible teacher recovers both.
  • A third of samples have no correct teacher, limiting available verified supervision.
  • Balanced initializations leave little headroom, and exact-answer eligibility requires a reliable task-appropriate alternative beyond exact-answer settings.

A Post-Training and Initialization

The study constructs three domain teachers and one shared Multi-Domain Post-Training policy for each of five student backbones using family-matched base models.

  • Five student backbones receive three domain teachers and one Multi-Domain Post-Training policy from family-matched base models.
  • Domain teachers use separate domain datasets, while the Multi-Domain policy uses their balanced union.
  • Post-training, controlled online optimization, and evaluation follow shared recipes across the student backbones.

B Online Optimization and Teacher Feedback

The online procedure caches verified teacher eligibility, gathers rollout-specific self and teacher feedback, applies masked importance-weighted distillation, and updates an EMA self-teacher.

  • Each controlled method processes 32 prompts per update, draws eight on-policy responses per prompt, and runs for 100 updates on eight H100 80GB GPUs.
  • The implemented objective uses truncated-support Jensen–Shannon divergence, detached clipped importance ratios, and per-micro-batch normalization.
  • Algorithm 1 combines a student, EMA self-teacher, and frozen teachers, then returns the student while discarding the EMA copy and all teachers.
  • Before online training, each frozen teacher answers every training question once and a verifier records the sample-level eligible set.
  • External teacher feedback is requested only when the student rollout is incorrect, and retained messages must pass concept-anchor and answer-leakage checks.

C Evaluation Protocol

Evaluation uses a shared frozen protocol with 79 held-out questions per domain and 16 sampled responses per question, reporting avg@16 domain metrics.

  • Each domain contains 79 held-out questions, and each question receives 16 independently sampled responses.
  • The evaluation reports avg@16 results under settings shared across all student backbones.
  • When the primary parser fails to establish correctness, a fixed Qwen3-8B base model provides a deterministic binary-judge fallback.
  • The fallback cannot overturn an explicitly extracted non-gold answer, and judge parse failures count as incorrect.

D.1 Cross-Scale Training Dynamics

Figure 6 compares held-out trajectories across Qwen3 scales, using step 100 as the common checkpoint because training is non-monotonic. On Qwen3-8B, MT-SDPO finishes above initialization and SDPO on both Macro and Worst-domain accuracy, while feedback exposure does not directly establish effectiveness.

  • D.1 Cross-Scale Training Dynamics: Qwen3-8B MT-SDPO finishes above its initialization and SDPO on both Macro and Worst-domain accuracy.
  • D.1 Cross-Scale Training Dynamics: At step 100, Qwen3-1.7B and Qwen3-4B mainly recover weakest-domain capability rather than improving Macro accuracy over initialization.
  • D.2 Additional Feedback Diagnostics: MT-SDPO uses retained teacher feedback on 5.64% of Qwen3-8B rollouts on average, below Domain Routing at 23.07% and w/o Verification at 27.32%.Average teacher-only prompt coverage is 1.12% for MT-SDPO, compared with 9.16% for Domain Routing and 11.66% for w/o Verification.
  • D.2 Additional Feedback Diagnostics: Feedback volume alone does not explain integration quality, and the teacher-only metric does not test whether retained feedback is semantically correct or repairs the student’s answer.The teacher-only metric uses all prompts as its denominator and is not the conditional probability of receiving feedback given a missing self-anchor.
  • D.3 Verifiable-Reward RL Reference: GRPO results are separate from the controlled supervision-allocation comparison because GRPO changes the objective and uses no teacher pool.GRPO uses a policy gradient on a scalar reward rather than the token-level distillation objective, so its descriptive results do not establish a general ordering between the objectives.

E Prompt Templates

The appendix specifies shared and family-specific prompts for data construction, student rollout, self-teacher reprompting, frozen-teacher diagnosis, and evaluation. These templates constrain answer formatting, feedback context, diagnostic scope, and verifier behavior across the pipeline.

  • E Prompt Templates: Qwen3, Llama, and OLMo use family-specific data-construction and validation pipelines, while rollout, self-teacher, teacher-feedback, and evaluation instructions are shared at the natural-language level.
  • E Prompt Templates: Figure 6 evaluates Qwen3-1.7B, Qwen3-4B, and Qwen3-8B with Macro and Worst-domain accuracy, using step 100 for Domain Routing on Qwen3-8B.
  • E.1 SFT Data Construction: SFT data construction independently samples two candidates without including the reference answer, then uses a gold-conditioned fallback only when neither candidate yields a parseable correct label.
  • E.1 SFT Data Construction: The data-construction prompt requests a concise derivation while keeping the private correctness constraint internal and requiring a final answer field containing one choice letter.
  • E.1 SFT Data Construction: The fallback and independent-generator prompts both use rigorous domain-expert instructions and require the selected choice letter as the final answer field.
  • E.2 Student SFT and Initial Rollout: SFT uses a domain-specific system message, whereas online rollout, manifest construction, and evaluation use a generic scientific-reasoning system message with a shared multiple-choice user prompt.
  • E.3 Online Self-Teacher Reprompts: The self-teacher reprompt combines the original prompt with available successful-solution and environment-feedback blocks, inserting an empty string when a source is unavailable.
Loading 2609.02548v1…