PaperlayerView exact revision on arXiv

Source-linked AI summary

Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution

Yue He, Jiahui Hu, Shaman Tang, Hua Zhou, Qiman Wu, Menglin Li, Yizhao Wang, Yuheng Zhang, Da Zhu, Zhou Wang, Anhua Liu, Hanlin Chen, Mingyuan Wang, Rui Xin, Yuhua Wei, Shuang Su, Siqi Liu, Kai Yang, Weichao Huang, Hang Zhang, Honggou Yang, Xianming Liu

arXiv:2610.39876v1cs.LGcs.AI

TL;DR

Capability-specific post-training can produce strong specialists, but consolidating them into one dynamic-MoE model raises calibration and execution challenges. The paper combines calibrated multi-teacher on-policy distillation with an efficient execution path, reporting strong specialist integration, broad benchmark gains, and faster profiled consolidation.

  • Problem

    Capability-specific pipelines can develop strong specialists, but separate deployment can increase storage and serving overhead, and consolidation must handle teacher-loss scale variation and dynamic-MoE execution challenges.

    Source: §1 · p. 2

  • Method

    C-MOPD distills frozen domain specialists on student-generated trajectories with teacher-local Top-K matching, teacher-wise loss normalization, and clipping, alongside bounded-width dispatch, final-route replay, and asynchronous rollout–actor overlap.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    The final pipeline improves over SFT on all four reasoning-mode mathematics and code benchmarks and ten of twelve broader metrics, with a 0.9988 normalized four-domain integration score.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On PPU hardware, the combined execution configuration reduces profiled average consolidation step time from 398.7 to 139.3 seconds, an observed 2.86× speedup over serial execution.

    Source: §4 · p. 10

    Execution-checkpoint code scores occur at different training steps and therefore provide a descriptive comparison of execution-checkpoint quality.

    Source: §3.5 · p. 10

Abstract

from arXiv · show

We study post-training for dynamically routed mixture-of-experts (MoE) models, focusing on integrating independently optimized specialists while keeping consolidation efficient. Our approach combines calibrated multi-teacher on-policy distillation (C-MOPD), which matches each selected teacher on a local Top-$K$ support with teacher-wise loss normalization and clipping, with an execution path that uses post-capacity bounded-width dispatch, final-route replay, and asynchronous rollout--training overlap. On Turing-20B-A2B, the selected model achieves a four-domain normalized integration score of 0.9988 and retains at least 90.16\% of the measured specialist gain in every domain. At matched $K=32$, the full calibration objective scores 0.9962, compared with 0.9724 without calibration and 0.9834 without clipping. At a matched 8K response cap, the model exceeds both shared SFT and Qwen3-8B on all four reasoning-enabled mathematics and code benchmarks (unweighted mean: 85.92, versus 80.33 for Qwen3-8B and 74.21 for SFT), and improves over SFT on ten of twelve broader evaluation metrics. In a PPU-based profile, the combined execution configuration reduces consolidation step time from 398.7 to 139.3 seconds, an observed 2.86$\times$ speedup over serial execution. These results demonstrate strong specialist integration alongside more efficient consolidation, while reflecting the complete training and execution framework rather than any single component.

1 Introduction

Capability-specific specialists can broaden task coverage but increase storage and serving overhead when deployed separately, while consolidation must address teacher-scale variation and dynamic-MoE execution challenges. The paper combines calibrated multi-teacher distillation with an efficient execution path, reporting strong integration and faster profiled consolidation.

1 Introduction

  • Separate specialist deployment can increase storage and serving overhead; integration also faces teacher-loss scale variation and dynamic-MoE dispatch and route-consistency challenges.Existing alternatives include mixed-domain or sequential RL, fixed-trajectory off-policy distillation, and parameter-space merging.Source: §1 · p. 2
  • C-MOPD distills frozen specialists on student-generated prefixes using teacher-local Top-K reverse-KL matching, teacher-wise loss normalization, and token-level clipping.Task routing selects one teacher per sample, while explicit teacher weights are retained.Source: §1 · p. 2
  • 0.9988 normalized four-domain integration score; the student retains at least 90.16% of specialist improvement over SFT in every domain.The retained-gain measure compares each domain’s consolidated student with its corresponding specialist and shared SFT baseline.Source: §1 · p. 2
  • The complete pipeline improves over SFT on all four reasoning-mode mathematics and code benchmarks and ten of twelve broader evaluation metrics.Source: §1 · p. 2
  • 2.86× speedup over serial execution; the combined configuration reduces profiled consolidation step time from 398.7 to 139.3 seconds.The configuration uses post-capacity dispatch width capped at 32, asynchronous rollout–actor overlap, one-step policy lag, and final-route replay.Source: §1 · p. 2
  • At matched K = 32, the full calibration objective scores 0.9962, versus 0.9724 without normalization and clipping and 0.9834 without clipping.Source: §1 · p. 2

2 Method

The method trains domain specialists from a shared SFT checkpoint, then consolidates them into one student through calibrated multi-teacher on-policy distillation. Its dynamic-MoE execution path retains base-model routing while bounding dispatch width, replaying final assignments, and overlapping rollout with actor updates.

2.1 Framework Overview / 2.2 Shared Initialization and Specialist Construction

  • The framework proceeds from shared SFT through four domain-specific RL specialists to C-MOPD consolidation of the frozen teachers into one student.The specialists cover mathematics, code, instruction following, and agentic tool use.Source: §2.1 · p. 3, §2.2 · p. 3

2.3.1 Task-Conditioned On-Policy Supervision

  • For each prompt, a task router selects one frozen domain teacher, which evaluates the student’s generated prefixes during on-policy distillation.With asynchronous overlap, trajectories come from the preceding student policy version, introducing a one-step policy lag.Source: §2.3.1 · p. 3

2.3.2 Teacher-Local Reverse KL

  • The reverse-KL objective independently renormalizes student and selected-teacher distributions over the teacher’s local Top-K support, matching relative probabilities within that support.It does not directly constrain the student’s probability mass outside the support.Source: §2.3.2 · p. 4

2.3.3 Teacher-Wise Loss-Scale Calibration

  • Teacher-wise loss-scale normalization and token-level clipping address differences in distillation-loss scale across teachers.The selected model uses cmax = 5; scaling without mean centering preserves non-negativity, and over-threshold tokens contribute zero direct gradient through the clipped term.Source: §2.3.3 · p. 4

2.4 Efficient Execution for Dynamic MoE

  • The execution workflow combines capacity-constrained routing, bounded dispatch width, final-route replay, and asynchronous rollout–actor overlap.The rollout representation reserves 256 expert slots per token, while the profiled 99th-percentile active count is approximately 19 and 0.21% of routes exceed 32 experts.Source: §2.4 · p. 4, §2.4.2 · p. 5, Figure 2 · p. 5

2.4.1 Capacity-Constrained Quantile Routing

  • Base-model Quantile Routing uses expert-specific score thresholds to select a token-adaptive number of experts under a controlled average budget, followed by per-expert capacity filtering.The paper retains this routing rule throughout post-training.Source: §2.4.1 · p. 5

2.4.2 Post-Capacity Dispatch-Width Bounding

  • RW32 caps post-capacity dispatch width at 32, preserving assignments on routes of at most 32 experts and retaining the highest-scoring 32 on wider routes.This reduces the width-dependent route representation to one eighth of its default size.Source: §2.4.2 · p. 5, Figure 2 · p. 5

2.4.3 Final-Route Replay and Asynchronous Overlap

  • Final-route replay reuses the post-capacity, width-bounded assignment map for actor computation, while AsyncOverlap runs rollout generation concurrently with actor updates.The actor recomputes routing weights under current parameters; rollout trajectories use the preceding student policy version.Source: Figure 2 · p. 5, §2.4.3 · p. 6

3 Experiments

Experiments evaluate normalized recovery of specialist gains, calibration variants, broader benchmark performance, and consolidation efficiency. The selected model retains at least 90.16% of specialist improvement in every domain, while the combined execution configuration achieves an observed 2.86× speedup over serial execution.

3.1 Experimental Setup

  • The selected Turing-20B-A2B checkpoint retains its architecture and dynamic Quantile Routing, with approximately 2B active parameters per token.The evaluation uses a common OpenCompass-based pipeline, with prompt mode, decoding settings, and response limits matched within comparisons unless stated otherwise.Source: §3.1 · p. 6

3.2 Multi-Domain Capability Integration

  • 0.9988 is the selected model’s macro integration score, with every domain recovery ratio at or above 0.9016.Code and Agent domain means exceed their specialist references; Math and IF retain 94.07% and 90.16% of specialist improvements over SFT.Source: Table 2 · p. 7, §3.2 · p. 7

3.3.1 Loss-Scale Normalization and Clipping

  • 0.9962 is the full calibration objective’s macro integration score at K = 32, versus 0.9724 without normalization and clipping and 0.9834 without clipping.The full objective scores higher in Code, IF, and Agent, but lower in Math, than both removal variants.Source: Table 4 · p. 7, §3.3.1 · p. 7

3.3.2 Support Size and Clipping Threshold

  • 0.9988 is the highest observed macro score at cmax = 5, achieved by K = 16; K = 32 scores 0.9962.At K = 32, cmax = 5 also achieves the highest macro score among tested thresholds, versus 0.9181 at cmax = 2 and 0.8829 at cmax = 10.Source: §3.3.2 · p. 7, Table 5 · p. 8

3.4 Broader Capability Evaluation

  • 10 of 12 reported metrics improve over SFT; GPQA Diamond decreases by 5.05 points, from 40.40 to 35.35.The largest gains are IFEval (+20.33 points), BFCL v4 (+13.15), and MATH (+8.24).Source: Table 6 · p. 8, §3.4 · p. 8
  • 85.92 is C-MOPD’s unweighted mean on reasoning-enabled mathematics and code benchmarks at an 8K response cap, versus 80.33 for Qwen3-8B and 74.21 for SFT.C-MOPD improves over SFT on all four benchmarks in this comparison.Source: Table 7 · p. 8, §3.4 · p. 8

3.4.1 Output-Length Budgets

  • At an 8K cap, C-MOPD has fewer output-cap hits than Qwen3-8B on all four tasks; across 2,382 responses, 0.92% hit the cap, versus 8.27% for Qwen3-8B.Increasing Qwen3-8B’s cap to 32K raises its unweighted mean accuracy to 84.64, while its MATH-500 score of 95.00 exceeds C-MOPD’s 82.40 at 8K.Source: Table 8 · p. 9, §3.4.1 · p. 9

3.5 Dynamic-MoE Consolidation Efficiency

  • 398.7 to 139.3 seconds is the step-time reduction for the combined configuration, an observed 2.86× speedup over serial execution.The profile compares serial Consolidate, one-step AsyncOverlap, and AsyncOverlapRW32 over a common profiling interval.Source: Table 9 · p. 9, §3.5 · p. 9

4 Conclusion

The framework consolidates four independently trained domain specialists into one dynamic-MoE student, with strong integration results and a faster profiled execution workflow.

4 Conclusion

  • 0.9988 normalized four-domain integration score; the selected model retains at least 90.16% of each specialist’s measured improvement over shared SFT in every domain.The complete pipeline also improves over SFT on all four reasoning-enabled mathematics and code benchmarks and ten of twelve broader evaluation metrics.Source: §4 · p. 10
  • At fixed support size K = 32, the full calibration objective achieves a higher macro integration score than variants with calibration components removed, while the selected model scores 0.9988 normalized four-domain integration.Source: §4 · p. 10
  • 2.86× observed speedup: on PPU hardware, the combined execution workflow reduces profiled average consolidation step time from 398.7 to 139.3 seconds.The workflow combines asynchronous rollout–training overlap, bounded-width dispatch after capacity filtering, and final-route replay.Source: §4 · p. 10

5 Contributors

The report credits contributors across research, data, infrastructure, and evaluation, and identifies the advisors and project lead.

5 Contributors

  • The work brought together algorithm research, data and reward engineering, distributed training infrastructure, model evaluation, and report preparation.The project combined data construction, supervised fine-tuning, domain-specific reinforcement learning, calibrated distillation, route replay, and systems optimization.Source: §5 · p. 10
  • Hang Zhang, Honggou Yang, and Xianming Liu are listed as advisors; Qiman Wu is the project lead.Source: §5 · p. 10
  • The listed contributors are Yue He, Jiahui Hu, Shaman Tang, Hua Zhou, Menglin Li, Yizhao Wang, Yuheng Zhang, Da Zhu, Zhou Wang, Anhua Liu, Hanlin Chen, Mingyuan Wang, Rui Xin, Yuhua Wei, Pengfei Diao, Shuang Su, Minghao Li, Haojie Yang, Siqi Liu, Kai Yang, and Weichao Huang.Yue He and Jiahui Hu are marked as core contributors.Source: §5 · p. 10

A Training Recipe Details

The appendix summarizes post-training stages and data budgets, including how pool sizes and batch sizes are reported.

A Training Recipe Details

A.1 Supervised Data and Response Modes

The appendix describes the SFT mixture and response modes, mixed-outcome prompt filtering, multiple-choice answer-format randomization, and C-MOPD sampling and loss settings.

A.1 Supervised Data and Response Modes

  • The shared SFT mixture contains approximately 5.27M examples, with 53% reasoning-enabled and 47% direct-response examples overall, and 85% reasoning-enabled examples in mathematics and code distillation.For selected prompts, ten candidate responses are generated; incorrect responses are retained for teacher correction, with reasoning traces preserved in reasoning-enabled targets and empty think tags used for direct-response targets.Source: §A.1 · p. 12

A.2 Capability-Boundary Filtering

  • The capability-boundary filter retains prompts with at least two verified successes and two failures among at least four sampled responses.A task verifier marks each response correct or incorrect; responses are sampled at temperature 1.0.Source: §A.2 · p. 12

A.3 Multiple-Choice Answer-Format Randomization

  • Multiple-choice answer-format randomization uses 29 English and 29 Chinese formats, rewarding only responses that follow the selected format and identify the correct option.The method samples a template, format, and candidate permutation, with a deterministic parser rejecting responses that violate the format.Source: §A.3 · p. 13

A.4 C-MOPD Domain Sampling and Loss Configuration

  • The 51,500-prompt consolidation pool allocates 15,000 Math, 10,000 Code, 16,500 IF, and 10,000 Agent prompts.These are consolidation-pool counts, not specialist RL pool sizes; sampling probabilities and teacher-loss multipliers remain fixed for 400 steps, including 20 warmup steps, with selected C-MOPD settings K = 16, standard-deviation floor 0.1, and cmax = 5.Source: §A.4 · p. 13

B Fixed Teacher Weights and Specialist Diagnostics

Teacher-wise loss multipliers remain fixed during consolidation, while specialist reward, prompt-count, response-length, and KL curves provide diagnostic context. Loss-scale calibration rescales token losses but does not normalize the number of contributing tokens.

B Fixed Teacher Weights and Specialist Diagnostics

  • The teacher-loss multipliers are fixed at (1.0, 1.0, 0.3, 1.0) for Math, Code, IF, and Agent throughout training, rather than derived from Figure 4 statistics.The weights are fixed hyperparameters.Source: B Fixed Teacher Weights and Specialist… · p. 13
  • Figure 4 tracks normalized verifier reward, effective-prompt counts, generated response length, and KL against each specialist’s RL reference policy.Light lines show per-step values; dark lines show TensorBoard EMA curves with s = 0.9.Source: Figure 4 · p. 13
  • Effective-prompt counts depend on per-step prompt budgets, which differ between Agent and the other domains; response length counts tokens generated during specialist RL.Reward and effective-prompt counts reflect each domain’s verifier and sampling setup.Source: B Fixed Teacher Weights and Specialist… · p. 13
  • The scale statistic rescales token losses and cmax caps normalized values, but scale calibration does not normalize the number of contributing tokens.Under token_mean, a domain’s contribution also depends on its valid student-response token count.Source: B Fixed Teacher Weights and Specialist… · p. 14

C Evaluation Protocol

The evaluation protocol specifies benchmark splits, prompting settings, scoring metrics, and response caps. It separately reports base-model results because those checkpoints use different prompts and response formats.

C Evaluation Protocol

  • Few-shot and CoT settings describe benchmark prompting, while Direct-response and Think labels specify the model’s response mode.Table 12 summarizes benchmark splits, prompting settings, and scoring metrics.Source: C Evaluation Protocol · p. 14, Table 12 · p. 14
  • The direct-response mathematics and code comparisons use caps of 512 tokens for GSM8K, 2,048 for MATH, and 8,192 for HumanEval and MBPP.The reasoning-enabled comparison uses an 8K cap; Qwen3 at 32K is a larger-budget reference.Source: C Evaluation Protocol · p. 14

C.1 Base-Model Reference

  • Base-model results use a different prompting and response format, and Table 13 reports GPQA Main rather than the GPQA Diamond subset used in Table 6.Table 13 provides the base-model reference results under the base-model prompting setup.Source: §C.1 · p. 14, Table 13 · p. 14

D Full Raw C-MOPD Results

The raw-results tables provide benchmark scores for reference checkpoints, calibration variants, and support-size and clipping-threshold sweeps. The selected C-MOPD model uses K = 16 and cmax = 5, while the calibration variants use K = 32.

D Full Raw C-MOPD Results

  • Table 14 reports raw benchmark scores for reference checkpoints, sampled-token MOPD, three K = 32 calibration variants, and the selected C-MOPD model at K = 16 and cmax = 5.Each teacher reference is the corresponding domain specialist; unmarked rows use direct-response evaluation and Think-marked rows use reasoning-enabled evaluation.Source: D Full Raw C-MOPD Results · pp. 14–15, Table 14 · p. 15
  • Table 15 reports support-size and clipping-threshold sweeps, with domain scores and the macro integration score calculated from raw results using Equation 10.The table caption specifies raw benchmark scores and marks the highest score within each row and sweep block.Source: Table 15 · p. 15, D Full Raw C-MOPD Results · p. 15

E Execution Configuration Quality

The PPU execution configurations are evaluated using timing and code benchmark scores. Their code-score differences are not attributable to execution configuration alone because the evaluated model states come from different training steps.

E Execution Configuration Quality

  • Table 16 reports timing and code benchmark scores for the PPU execution configurations as a run-level view of their quality–efficiency trade-off.The table caption identifies timing and code benchmark scores for these configurations.Source: E Execution Configuration Quality · p. 16, Table 16 · p. 16
  • Code-score differences should not be attributed to execution configuration alone because the compared model states come from different training steps.Common-interval system averages are reported separately in Table 9.Source: E Execution Configuration Quality · p. 16

Related papers

Loading 2610.39876v1…