PaperlayerView exact revision on arXiv

Source-linked AI summary

Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs

Yuanyi Wang, Yanggan Gu, Su Lu, Guanghao Zhu, Pengkai Wang, Yifan Yang, Congkai Xie, Zhaoyi Yan, Jianmin Wu, Hongxia Yang

arXiv:2609.32821v1cs.AIcs.LG

TL;DR

Routing drift after MoE merging changes expert assignments, but it remains unclear whether those changes indicate task-relevant routing failure. The paper uses controlled routing interventions and task-level comparisons to assess recoverable loss; its results show that source disagreement alone is insufficient evidence for repair.

  • Problem

    After MoE merging, changed expert assignments may appear to signal router failure, but whether routing drift reliably indicates task-relevant harm remains unclear.

    Source: §1 · p. 1

  • Method

    The paper crosses source and merged router inputs and parameters, then tests recoverable task loss under specified routing interventions with non-routing parameters fixed.

    Source: §1 · p. 2

  • Results

    Across evaluated merged models, source-route restoration does not establish reliable task benefits, and SRR provides no reliable evidence that source-likelihood advantages identify beneficial corrections.

    Source: §1 · p. 2, §6.2 · p. 9

  • Takeaways & Limitations

    Routing drift alone is insufficient evidence of routing failure; repair claims require task-level effects for the specific intervention.

    Source: §1 · p. 2, §6.2 · p. 9

    The source-advantage and parent-relative weights (ω_t and ρ_tj) are not labels of routing failure.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

Model merging efficiently combines specialized large language models (LLMs) without joint retraining, but can substantially alter expert routing in Mixture-of-Experts (MoE) models. Such \emph{routing drift} is often interpreted as routing failure, raising a fundamental question that remains unclear: \emph{does routing drift after MoE merging actually indicate routing failure, and what evidence should justify repair?} We investigate these questions across DeepSeekMoE, OLMoE, and Qwen3-MoE proposing a routing analysis toolkit for controlled counterfactual interventions and token-level analysis. By crossing source and merged router inputs and parameters, we attribute most expert reassignments to input shifts rather than parameter changes at the same layer. However, source-relative routing differences poorly predict next-token likelihood gains from source-route restoration, and different expert selections can produce directionally similar mixture outputs. We therefore operationalize routing failure as \textit{task loss recoverable under a specified routing intervention, with non-routing parameters fixed.} These tests detect recoverable loss under deliberate router corruption, whereas source-route restoration does not establish reliable task benefits in the evaluated merged models. Motivated by these, we propose \emph{Selective Router Repair (SRR)} as a case study, and find that source-specialist token-likelihood advantages do not reliably identify beneficial local corrections. Together, these findings show that \textbf{routing drift alone is insufficient evidence of routing failure}: source-informed corrections must be judged by their task-level intervention effects. The analysis toolkit and SRR code are released.

1 INTRODUCTION

The introduction asks whether post-merge routing drift signals task-relevant failure, rather than treating changed expert assignments as failure by themselves. It presents controlled routing analysis and intervention-based tests to distinguish route changes from recoverable task harm.

Motivation

  • MoE merging can change token-to-expert assignments, but routing drift alone does not establish that routing caused task-relevant degradation.The paper defines routing failure by degradation in task behavior attributable to routing, rather than structural routing change.Source: §1 · p. 1

Analysis toolkit

  • The toolkit crosses source and merged router inputs and parameters, attributing most reassignments to input shifts while finding that source-route gains are poorly predicted.It evaluates DeepSeekMoE, OLMoE, and Qwen3-MoE; local attribution does not exclude upstream routing effects.Source: §1 · p. 2

Mixture outputs

  • Source and native merged routers can produce similar mixtures, so different expert selections need not imply substantially different mixture outputs.The comparison holds merged states and expert parameters fixed and finds observed mixtures more cosine-similar than matched random-route controls.Source: §1 · p. 2

Task-grounded diagnosis

  • Routing failure is operationalized as task loss recoverable under a specified routing intervention while non-routing parameters remain fixed.The tests compare policies on identical items and allow downstream representations to respond.Source: §1 · p. 2

Selective Router Repair

  • Selective Router Repair tests source-informed local corrections, but source-likelihood advantages do not reliably identify beneficial corrections or establish average task gains.Candidate construction and task recovery are distinct claims.Source: §1 · p. 2

2 RELATED WORK

Prior model-merging work primarily studies how dense-model parameters should be combined; this paper instead examines routing drift in already merged MoE models and what evidence should justify repair.

Model Merging

Routing Drift and Repair

  • Most related routing realignment treats source-to-merged mismatch as breakdown, whereas this work distinguishes a repair objective from evidence of failure.HARC realigns the merged router [Huang et al., 2026].Source: §2 · p. 3
  • The paper tests routing effects with local input-versus-parameter interventions and measures recoverable task loss under specified policies.It also tests whether source-likelihood advantages support useful local corrections rather than treating them as failure labels.Source: §2 · p. 3

3 ROUTING DRIFT IS NOT ROUTING FAILURE

The section separates the origins of routing drift, its predictive value for source-route restoration, and whether different routes preserve similar mixture outputs.

3.1 MOST POST-MERGE ROUTE CHANGES ARE REPRESENTATION-INDUCED

  • Across six settings, 26.5–54.3% of token–layer expert sets change; 77.7–96.9% of changed routes are representation-induced, while router-parameter-induced changes account for at most 1.8%.Figure 2 compares expert-set distances under component replacements and attributes changed routes; these local findings do not establish harm.Source: Figure 2 · p. 3, §3.1 · p. 4

3.2 ROUTING DRIFT POORLY PREDICTS SOURCE-ROUTE GAINS

  • JS divergence predicts source-route next-token NLL gains near chance (AUROC 0.47–0.52), and task-margin intervals show no reliable improvement from source-route restoration.Figure 3 plots JS divergence against NLL gain and task-margin changes; tested comparisons neither establish optimal merged routing nor exclude recoverable loss under other policies.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 DIFFERENT ROUTES CAN PRESERVE SIMILAR MIXTURE OUTPUTS

  • Source-route and native merged-route mixtures have mean cosine 0.896–0.976, exceeding matched random controls by 0.112–0.147 across all six settings.Mixture cosine exceeds the entering-versus-leaving expert-pair maximum in all 727 sampled events; this does not guarantee unchanged magnitudes or downstream behavior.Source: §3.3 · pp. 4–5, Figure 4 · p. 5

4 TASK-GROUNDED DIAGNOSIS OF ROUTING FAILURE

The section defines routing failure relative to a specified alternative policy and tests whether task loss is recoverable under controlled and natural routing interventions.

4.1 INTERVENTION-RELATIVE RECOVERABLE LOSS

  • Recoverable loss compares a baseline routing policy with a specified alternative on identical task items while non-routing parameters remain fixed.The alternative policy and intervention scope are part of the estimand, and downstream representations may respond.Source: §4.1 · p. 5
  • For zero–one loss, positive recovery means an accuracy gain; an interval containing zero establishes neither harmlessness nor optimality.Recovering task loss does not by itself identify router-parameter changes as the original cause of degradation.Source: §4.1 · p. 5

4.2 CONTROLLED RECOVERY AND NATURAL ROUTING ALTERNATIVES

  • Clean-route replay after imposed OLMoE router-logit corruption recovers accuracy loss, but does not reconstruct the clean forward pass or demonstrate an SRR-learned repair.Figure 5 presents corrupted-versus-clean-route replay accuracy and paired gains across parent–permutation settings.Source: Figure 5 · p. 5, §4.2 · p. 6
  • Natural source-derived and LC-calibrated route replays show no strictly positive accuracy or correct-choice-margin interval in displayed settings.LC replay changes correctness on one item per parent, yielding −0.39 pp for Average and +0.39 pp for TA.Source: §4.2 · p. 6

5 SELECTIVE ROUTER REPAIR

SRR separates source-based correction proposals from evidence that corrections recover task behavior. It selects expert pairs, fits router updates, and evaluates local utility and executed effects.

5.1 SELECTING EXPERT PAIRS

  • Pair selection ranks active experts by profile separation, sign consistency, and activity, with fixed activity and consistency thresholds and no benchmark scores.Source: §5.1 · p. 6

5.2 FITTING ROUTER CORRECTIONS

  • The source-parent residual is clipped by the source-base difference; fits target logit residuals, updating selected rows in the final five sparse layers rather than optimizing task loss.Source: §5.2 · pp. 6–7

5.3 LOCAL SUPERVISION AND EXECUTED CORRECTIONS

  • 83.4% of supported events show source/fitted sign agreement (95% CI [80.0, 86.6]%), but neither pooled U nor D establishes improved local utility.Source: Figure 6 · p. 7, §5.3 · p. 7
  • At the final updated layer, input-response RMS is approximately 0.38–0.86 times direct-correction RMS across settings, so joint execution need not reproduce fixed-input correction.Source: Figure 6 · p. 7, §5.3 · p. 7

6 EXPERIMENTS

Across the evaluated settings, SRR does not show consistent average improvement over its merged parent, and beating HARC alone does not establish recovery. Task responses vary, and the matched diagnostic finds no evidence that source-likelihood selection adds useful local information.

6.1 EXPERIMENTAL SETUP

  • The evaluation covers three MoEs, four merging methods, and eight benchmarks; SRR updates selected router parameters in the final five sparse layers while other parameters remain fixed.Scores are averaged over five runs; ∆Avg. is relative to the corresponding unrepaired parent, equally averaged across 12 settings.Source: §6.1 · pp. 8–9

6.2 MAIN RESULTS

  • Five of eight OLMoE/Qwen3-MoE settings have positive SRR average-score changes, spanning −0.056 to +0.139 pp; WUDI-Merge is negative on both backbones.Table 2 compares each method with its own merged parent.Source: Table 2 · p. 8, §6.2 · p. 9
  • Each method has the higher point estimate in four of eight active SRR settings, but OLMoE–WUDI SRR and HARC both remain below Parent.SRR is −0.021 pp versus HARC at −0.075 pp on OLMoE–WUDI.Source: §6.2 · p. 9
  • Task effects vary: OLMoE–Average SRR gains 1.74 pp on GSM8K but loses 0.63 pp on WinoGrande, without establishing task-level significance.The cross-backbone comparison changes architecture and parameter count together, so it does not isolate model size.Source: §6.2 · p. 9
  • Across 440 matched pairs, neither source-direction contrast establishes higher utility on positive-support events.The matching test evaluates incremental local selection information, not equivalence or the cause of benchmark changes.Source: §6.2 · p. 9, Table 3 · p. 9
  • Fitted source-derived updates and outperforming another repair method do not substitute for demonstrating improvement over the unrepaired parent under the specific intervention and task.These fitted updates differ from the source-route replay tested earlier.Source: §6.2 · p. 9

7 CONCLUSION

The paper concludes that routing drift alone does not establish routing failure, and that repair requires task-level evidence for the specific intervention. The included appendix details crossed interventions, representation visualizations, and sampling and uncertainty procedures.

7 CONCLUSION

  • Routing drift alone is insufficient evidence of routing failure; the appendix’s analysis measurements use event populations distinct from the SRR task evaluation.The appendix reports structural, local-likelihood, and mixture-output comparisons.Source: §7 · p. 10, A ROUTING ANALYSIS: PROTOCOLS AND… · p. 14

AI USE STATEMENT

  • The authors report using generative AI to polish the manuscript and figures, help test experiments, and check code and infrastructure, with author review and responsibility.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The anonymized code, toolkit, and configurations are reported as available; training settings, benchmarks, configuration, and validation are specified elsewhere in the paper.Source: REPRODUCIBILITY STATEMENT · p. 10

A ROUTING ANALYSIS: PROTOCOLS AND COMPLETE RESULTS

A.1 CROSSED INTERVENTIONS

  • Crossed states isolate whether replacing router inputs or parameters changes the source expert set, but establish local sufficiency rather than the origin of upstream representation changes.The first state subscript denotes the router and the second its input.Source: §A.1 · p. 14

A.2 ALIGNED REPRESENTATIONS UNDER MERGING AND SRR

  • Joint t-SNE views show router inputs above layer outputs, with domain colors and shapes and arrows marking token displacements associated with merging and SRR.The output view is routed-mixture output for OLMoE and MoE-block output for DeepSeekMoE and Qwen3-MoE.Source: §A.2 · p. 14
  • The aligned projections show heterogeneous, token-dependent changes across the three model examples rather than a single uniform SRR displacement.These projected arrow lengths do not quantify original-space changes or task recovery.Source: §A.2 · p. 14

A.3 SAMPLING AND UNCERTAINTY

  • Sampling frames balance prompts across mathematics, instruction following, science, and code, while hash selection avoids using intervention outcomes.Attribution resamples prompts with dependent token–layer records; task comparisons instead resample matched items.Source: §A.3 · p. 16

A.4 ROUTING ORIGIN ACROSS ARCHITECTURES AND DOMAINS

Across the appendix’s routing analyses, representation shifts dominate local route attribution, while structural predictors and source-route replay do not establish reliable gains. The appendix also bounds candidate comparisons, examines mixture-output similarity, and describes SRR fitting.

A.4 ROUTING ORIGIN ACROSS ARCHITECTURES AND DOMAINS

  • Representation-induced changes dominate throughout OLMoE and most DeepSeekMoE layers; across domains they range from 75.1% to 90.2%, versus 0.1% to 2.6% for router-parameter-induced changes.Across 16 OLMoE layers, hidden-state cosine distance and Top-k disagreement are positively associated (Pearson r = 0.730, p = 0.0013; Spearman ρ = 0.653, p = 0.0061).Source: §A.4 · p. 16, §A.5 · p. 17

A.6 STRUCTURAL PREDICTORS AND MULTIPLICITY

  • None of the 16 corrected primary tests linking changed-set predictors to source-policy or source-hidden-state gains survives Holm correction.The 18 corrected secondary discoveries concern all-event set distance versus set-only gain, where unchanged sets mechanically create a no-op.Source: §A.6 · p. 17, Table A3 · p. 19

A.7 LOCAL AND TASK-LEVEL SOURCE-ROUTE REPLAY

  • All four token-local source-route replay intervals are negative, and all 12 task margin and accuracy intervals include zero.The local test replaces one changed route while holding the merged input and expert parameters fixed; task replay uses source routing across question–choice tokens and sparse layers.Source: §A.7 · pp. 17–18, Table A4 · p. 20, Table A5 · p. 21

A.8 QWEN3-MOE MEASUREMENTS

  • For Qwen3, all changed-set predictor correlation intervals include zero and all AUROC intervals include 0.5; its 64-item ARC-Challenge replay loses one correct answer and gains none for each parent.The Qwen3 replay sample includes no mathematics or computer-science evaluation.Source: §A.8 · pp. 18–19, Table A6 · p. 21, Table A8 · p. 22

A.9 LOCAL REFERENCE QUALITY

  • The local reference comparison uses a bounded set of 7–12 candidate routes per event, and reports limited native optimality without a reliable source advantage.It is not an exhaustive route search.Source: §A.9 · p. 19, Table A9 · p. 22

A.10 MIXTURE OUTPUTS AND MATCHED CONTROLS

  • Across 727 changed-route events, mixture-minus-expert cosine gaps are all positive, ranging from 0.078 to 1.417.Cosine measures direction, not output magnitude or task recovery; DeepSeekMoE’s fixed shared-expert branch is excluded.Source: §A.10 · p. 19, Table A10 · p. 22

A.12 VARIATION OUTSIDE MERGING

  • Source–base set distances exceed source–merged distances, while specialist routers also disagree on the same source state, showing routing variation outside merging.Source: §A.12 · p. 21, Table A13 · p. 22

B SRR ALGORITHM AND FITTING OBJECTIVE

  • SRR changes selected rows of the merged parent’s router while leaving experts and other parameters fixed; its fit matches selected router-logit gaps, not task loss.Successful fitting alone does not imply downstream recovery.Source: B SRR ALGORITHM AND FITTING OBJECTIVE · p. 23

C TASK-GROUNDED INTERVENTION PROTOCOLS

The protocols pair routing interventions on identical items while holding non-routing parameters fixed, and distinguish task recovery under deliberate corruption from natural source-route replay. The appendix also specifies task scoring and SRR’s deterministic fitting and writeback procedures.

C.2 FULL-LAYER AND GRADED CORRUPTION CONTROLS

  • Both original OLMoE logit-permutation controls improve accuracy from 22.65625% to 50.78125%, recovering 72 net answers per comparison.The injected-corruption comparisons correct 94 and 90 answers and break 22 and 18, respectively, for Average and TA.Source: Table C14 · p. 24, §C.2 · p. 24

C.1 TASK SCORING AND PREDICTION TRANSITIONS

  • Task scoring selects the choice with the highest score from answer-token NLL, while paired accuracy change counts wrong-to-correct minus correct-to-wrong transitions.Margins are continuous score readouts distinct from accuracy, and intervals use 10,000 paired item-bootstrap draws.Source: §C.1 · p. 24

C.3 NATURAL LC REPLAY AND NUMERICAL VALIDITY

  • Natural LC schedule replay changes accuracy by −0.390625 pp for Average and +0.390625 pp for TA; both intervals include zero.Each change represents one changed-correctness item per parent in opposite directions; these natural and injected cohorts are distinct.Source: §C.3 · p. 24

C.2 FULL-LAYER AND GRADED CORRUPTION CONTROLS

  • All eight four- and sixteen-layer corruption recoveries pass Holm correction, while none of the one-layer tests does; replay accuracy remains near the clean reference.Replay ranges from 50.39% to 51.56% around the 50.78% clean reference, so it does not establish exact reconstruction.Source: §C.2 · p. 24, Table C15 · p. 25

D.1 DETERMINISTIC EXPERT-PAIR SELECTION

D.2 NUMERICAL SOLVE AND WRITEBACK

  • SRR fits weighted ridge corrections from unchanged parent traces, requires every selected pair to converge, then jointly writes updates and checks changed-layer scope after BF16 conversion.The covariance-based logit-change budget is diagnostic rather than an enforced projection; sparse row updates do not guarantee unchanged probabilities or expert choices elsewhere.Source: §D.2 · p. 26

D.3 HELD-OUT WEIGHTING ABLATION

The weighting ablation compares held-out residual-fitting MSE, not task performance. A constructed example shows why proxy fit need not imply task recovery, and the independent local diagnostic evaluates matched routing-direction perturbations without converting NLL into accuracy.

D.3 HELD-OUT WEIGHTING ABLATION

  • All-uniform fitting had the lowest held-out MSE in all four settings: DS/Average .01624, DS/TA .04973, OL/Average .05471, and OL/TA .09864.The ablation reports point estimates rather than task scores or paired uncertainty.Source: §D.3 · pp. 26–27, Table D17 · p. 27

D.4 A COUNTEREXAMPLE TO TASK IMPROVEMENT FROM PROXY FITTING

  • A hypothetical Top-1 example reduces fitting error to one quarter yet switches routing to a worse merged expert and raises NLL by log 8.The construction illustrates that proxy fitting alone gives no general task-recovery guarantee when source and merged expert functions differ; it is not an experiment.Source: §D.4 · p. 27

E.1 INDEPENDENT SAMPLE AND PAIRED CONTRASTS

  • Positive source advantage occurs in 471 of 1,024 independent local events, whose proposed and opposite logit perturbations are compared against native routing.The local NLL results are not converted into task accuracy.Source: §E.1 · p. 27

E.2 MATCHED SUPPORT AND DIRECTION PRESERVATION

Matched local contrasts do not establish that source-supported or fitted directions improve likelihood, despite frequent sign agreement. The execution analysis separately measures direct and input-response pair-gap changes and cautions against additive or causal interpretations.

E.2 MATCHED SUPPORT AND DIRECTION PRESERVATION

  • Source and fitted directions agree in sign on 83.4% of supported events, but agreement does not establish local benefit; only two of four settings passed the source-direction sign screen.The contingent original scale-up was therefore not run.Source: §E.2 · pp. 27, 29
  • For 440 matched pairs, source-direction U is −0.000341 nats/token (95% interval [−0.003323, 0.002631]) and D is −0.000997 (95% interval [−0.005653, 0.003868]).These matched contrasts do not establish enrichment.Source: §E.2 · p. 27, Table E18 · p. 28

F.1 ALIGNED TRAJECTORIES AND NUMERICAL CHECKS

  • The maximum normalized algebra residual is 6.9 × 10^-18, while the largest native-forward versus FP64 logit discrepancy is 0.0621; algebraic agreement validates the accounting identity, not task-effect causation.Inputs agree at the first updated layer.Source: §F.1 · p. 29

F.2 COMPLETE LAYER-WISE MEASUREMENTS

  • Direct, response, and executed RMS values are not additive, and their cross-setting association with task effects is descriptive rather than event-wise or causal attribution.The response/direct RMS ratios are ratios of point estimates, not additive attribution shares.Source: §F.2 · p. 29

G SUPPLEMENTARY TASK EVALUATION

The supplementary evaluation uses matched items and paired item bootstraps, with intervals conditional on fixed checkpoints. Its selection test retains Parent in every family-setting decision, while the reported pair-gap tables are conditional diagnostics rather than task-score results.

G SUPPLEMENTARY TASK EVALUATION

  • Each Parent/SRR pair is evaluated five times on 35,326 matched items, and the aggregate equally weights the eight task percentages.Paired item bootstraps resample item identities within each task after averaging paired score differences over the five runs.Source: G SUPPLEMENTARY TASK EVALUATION · p. 29, Table G23 · p. 31
  • SRR–Parent intervals condition on fixed checkpoints and do not capture variation across repair seeds, calibration samples, or training runs; they do not apply to HARC comparisons.The table captions describe conditional pair-logit-gap RMS diagnostics, not additive terms.Source: G SUPPLEMENTARY TASK EVALUATION · p. 30, Table F19 · p. 30, Table F20 · p. 30
  • All 20 family-setting selection decisions retained Parent, yielding zero confirmation differences by checkpoint identity; fixed, nonselected source-scale-one updates yielded −0.016 pp (95% paired CI [−0.179, +0.146]).Rejection under this rule does not establish that every candidate is ineffective.Source: G SUPPLEMENTARY TASK EVALUATION · p. 30

Related papers

Loading 2609.32821v1…