Source-linked AI summary
Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation
Cedric Caruzzo, Donggeun Yoo, Tae Soo Kim
TL;DR
Same-weight MoE self-distillation can route identical tokens through different experts, but the downstream influence of that divergence is unclear. This paper decomposes routing and content effects, measures residual exposure, and finds routing influence bounded, less direction-specific, and not inferable from router movement alone.
Problem
Gate divergence in same-weight MoE self-distillation does not establish how much a routing mismatch reaches the residual stream or model output.
Method
The paper decomposes routed-output differences into a fixed-content routing term and a dense-like content term, then measures exposure and applies behavioral patches.
Results
Routing influence is bounded and mostly magnitude-generic, whereas the dense-like content term is more direction-specific; the narrow block-level range is not universal.
Takeaways & Limitations
Router movement alone is not evidence of behavioral influence; measure residual exposure and use behavioral intervention when the decision matters.
Takeaways & Limitations
The evidence uses an opportunistic, family-correlated checkpoint sample rather than a factorial architecture sweep, and behavioral patches measure next-token effects rather than task-level generation quality.
Abstract
from arXiv · showhide
Two Mixture-of-Experts (MoE) forward passes can share every weight yet route the same token through different experts. This creates a possible blind spot in same-weight self-distillation, where a demonstration-conditioned teacher supervises a query-only student. We study this mismatch in its single-step form, with frozen weights rather than as a proxy for a full training trajectory. An exact blockwise decomposition separates a routing term, which changes gates at fixed content, from a dense-like content term. Across seven open-weight checkpoints and two domains, the routing term spans only $1.6\times$ as a fraction of block output, while its residual-stream exposure spans $3.2\times$. Exposure is ordered by the routed block's share of the residual. Scaling the always-on backbone in two confirmatory models moves exposure monotonically; common-mode controls support a mass-and-coherence mechanism rather than denominator dilution alone. Preregistered PubMedQA patches on three models show that the full routing term moves outputs by less than half the natural context effect and is largely reproduced by matched-norm noise, whereas the content term is strongly direction-specific. Scale and merged-expert probes show that the narrow block-level range is not universal, although exposure remains small at the tested boundaries. Router movement alone is therefore not evidence of behavioral influence: measure exposure first, and use a behavioral intervention when the decision matters.
1 Introduction
This study shows that same-weight MoE routing divergence can persist from a conditioning-induced mismatch into the residual stream without reliably predicting downstream behavioral influence. It separates routing mismatch from content shift and evaluates exposure and causal output effects rather than treating router movement itself as influence.
- Motivation: Same-weight teacher and student passes can route the same token through different experts, but gate divergence alone does not quantify the resulting perturbation.The complication arises because identical parameters can execute different sparse computations under different contexts.
- Method: An exact A/B decomposition isolates routing mismatch at fixed content from dense-like content shift at fixed gates, then traces each term into residual exposure and output behavior.The comparison holds weights fixed in the single-step regime before any update.
- Main findings: Across seven checkpoints and two domains, the routing term is relatively stable as a share of block output, while downstream exposure varies with the routed block’s residual share.The most fine-grained router is among the least exposed, inverting the prediction that larger router movement necessarily creates greater downstream risk.
- Mechanism: Backbone rescaling on two confirmatory checkpoints supports a mass-and-coherence account in which always-on mass contains routing mismatch.The intervention links exposure ordering to backbone share rather than router movement alone.
- Behavioral validation: Three-model causal patches show bounded, mostly magnitude-generic routing effects, whereas dense-like content effects are direction-specific; scale and merged-expert probes delimit the observed range.These results motivate the narrow claim that routing can matter, but router movement is not itself a measure of downstream influence.
2 Preliminaries and the decomposition
This section defines an exact decomposition for comparing teacher and student forward passes with shared weights but different conditioning-induced routing. It separates gate-only routing mismatch from dense-like content change and defines residual exposure, conventions, and scope for measuring the mismatch.
- Setup: Teacher and student passes share weights but differ in conditioning, with the teacher receiving a demonstration and the student receiving none.The instrument measures conditioning-induced routing mismatch between two forward passes of the same model.
- Decomposition: The exact decomposition splits routed-output change into A, different gates applied to the same hidden state, and B, shifted content under teacher routing.A isolates gate movement with hidden state and expert weights fixed; B is the dense-like remainder that a dense model would also incur.
- Metrics: Routing factor measures A’s share of block output, while exposure measures its share of the residual stream read by the rest of the network.Exposure satisfies rA/res = rA/blk · rblk/res, so a smaller routed-block backbone share dilutes a fixed routing perturbation.
- Residual convention: Exposure is measured against both pre-block and post-block residuals because screening and intervention use different residual targets.The screen uses the pre-block residual; the intervention uses the post-block residual, res + y, and tables report which convention applies.
- Gate convention: Applied gate conventions affect A’s raw scale, whereas normalization by block output makes the routing factor less convention-sensitive.The exact bound is ∥A∥≤Fmax∥∆g∥1, and for renormalized gates it becomes ∥A∥≤2ΣgFmaxTV.
3 Measurement methodology
The methodology evaluates routing divergence across seven open-weight MoE checkpoints using paired PubMedQA and GSM8K probes, with bootstrap intervals quantifying uncertainty. It also measures gate-overlap and total-variation controls, while treating the observed routing-factor range as checkpoint-specific rather than universal.
- Models: Seven open-weight MoE checkpoints vary expert count, top-k, shared-expert presence, and gate conventions.The checkpoints range from 8/2 for Mixtral to 128/8 for Qwen3; several models include an always-on shared expert.
- Probes: PubMedQA and GSM8K each use n = 200 paired probes differing only by a worked demonstration.The paired design uses medical and mathematical-reasoning domains, with the demonstration isolating the routing response.
- Uncertainty: 0.003 and 0.0012 are the maximum per-model half-widths for the routing factor and exposure confidence intervals, respectively.These intervals come from 2000 bootstrap resamples over the n = 200 probe prompts.
- Routing metrics and controls: Top-k overlap uses Jaccard, while gate-distribution change uses total variation on renormalized gates.Confound and format controls are summarized in Appendix C.
- Primary measurements: 0.085–0.139 is the observed routing-factor range across the seven primary checkpoints and both domains shown in Figure 2.The figure reports 95% bootstrap confidence intervals over 200 prompts and states that the shaded range describes these checkpoints, not a universal confidence band.
4 Results I: backbone share orders exposure
Across seven checkpoints and two domains, backbone share orders routing exposure more strongly than the routing factor itself. Scale probes show the narrow block-level range is empirical rather than universal, while exposure remains small at tested boundaries.
- Backbone share orders exposure: Exposure rA/res spans 0.019 to 0.061, a 3.2× range, while routing factor rA/blk spans 0.085 to 0.139, a 1.6× range.Directly aggregated backbone share spans 0.174 to 0.515.
- Backbone share orders exposure: Spearman association between ∥y∥/∥res∥ and exposure is 0.96 on PubMedQA and 1.00 on GSM8K across seven checkpoints.Qwen3-30B pairs routing factor 0.133 on PubMedQA with exposure 0.019, whereas OLMoE pairs 0.124 with 0.056.
- Scale boundary: At 235B, Qwen3 routing factor rises to 0.188/0.200 on PubMedQA/GSM8K while exposure remains 0.030.DeepSeek-V2 at 236B remains at or inside the range at 0.114/0.139, with exposure 0.012/0.019.
5 Results II: the causal intervention
Graded backbone rescaling causally increases routing exposure through reduced coherence, with common-mode controls and replication supporting a mass-and-coherence mechanism. Behavioral patches show that the routing term has a real but bounded, largely generic effect, while the content term is more direction-specific.
- Causal backbone intervention: Backbone removal raises post-block exposure monotonically, while a frozen-numerator null stays nearly flat and routing coherence declines.Exposure rises from 0.024 to 0.040 on DeepSeek-V2-Lite and from 0.040 to 0.077 on Gravity; ||A|| rises to 1.57× and 1.69× baseline.
- Causal backbone intervention: At matched input drift, shared-content replacement causes less routing decoherence than matched-norm off-manifold perturbation, which causes less than mass removal.The three-way ordering is preserved in 96% of bootstrap samples.
- Replication and scope: On GSM8K, n = 100 replication reproduces exposure increases against a nearly flat null and preserves the same control ordering on both checkpoints.Exposure rises by 72% on DeepSeek-V2-Lite and 106% on Gravity.
- Behavioral intervention: In preregistered patches, injecting student-anchored A flips 3.1–4.0% of greedy tokens, while matched-norm noise reproduces 58–74% of that rate.The A effect is 40–45% of the natural context reference across OLMoE, DeepSeek-V2-Lite, and Qwen3-30B.
- Behavioral intervention: At matched norm, content-term B has a 5.6–80× premium over random direction versus 1.54–2.05× for A, so exposure is a screening coordinate, not a behavioral threshold.The natural context effect is carried more strongly by B’s direction; the causal claim is specific to the student-anchored A.
6 Related work
Prior work studies sparse-MoE routing, router stability, shared computation, and counterfactual route effects. This paper distinguishes realized conditioning-induced route differences from optimized alternatives and isolates same-weight self-distillation as a single-step setting.
- Sparse MoE routing: Sparse MoE research routes tokens to subsets of experts and studies load balance, router stability, router drift, and consistency-based suppression.These lines of work include continual tuning, replay, and compatible expert selection.
- Shared computation: Shared-expert and sparse-upcycling architectures use common computation or dense checkpoints to stabilize behavior, but do not measure residual exposure to routing perturbations.The paper’s backbone intervention supplies that measurement and separates routing from content effects.
- Route-effect measurement: Measurement studies interpret experts, separate routing inputs from ignored content, and evaluate counterfactual route changes, whereas this paper measures realized conditioning-induced route differences.An effect available somewhere in route space need not result from a particular training example.
- Self-distillation scope: Same-weight self-distillation uses additional conditioning to create the teacher–student two-pass construction measured here.Separate-checkpoint distillation adds a weight-induced mismatch, while self-teacher reinforcement learning adds a trajectory; both are outside this single-step scope.
7 Discussion and limitations
Routing motion, routing influence, and behavioral influence are distinct: exposure should be measured directly, and behavioral interventions are needed when decisions matter. The evidence supports diagnostic use rather than universal thresholds, while checkpoint, scaling, and patch coverage limit generalization.
- From motion to influence: Routing motion, block-output perturbation, and residual-stream exposure are distinct quantities, so router overlap alone does not establish behavioral influence.The decomposition measures perturbation at fixed content, while exposure measures how much reaches the residual stream.
- What the observed range means: The two scale probes and one merged-expert boundary rule out a universal routing-factor range without localizing a scaling law.Exposure remained within or below the primary observed range in both scale probes.
- A boundary that fails in the informative direction: 0.014 on PubMedQA and 0.011 on GSM8K are Beyonder-4x7B-v3’s routing factors, while exposure is 0.005–0.006.This merged-expert checkpoint fell below the primary routing-factor range, contrary to the predicted upward break.
- Use as a diagnostic, not a threshold: A checkpoint audit should verify decomposition reconstruction, measure exposure, and use a behavioral intervention when downstream decisions matter.Exposure within the observed range indicates residual containment in this regime, not proof that behavior is unchanged.
- Limits of the evidence: Evidence is limited by seven family-correlated checkpoints, two confirmatory rescaling models, one merged boundary, and output patches covering three checkpoints.Exposure did not order the patched behavioral effects, and all-layer patches can compound local terms differently from natural effects.
8 Conclusion
Same weights do not guarantee identical MoE computation because conditioning can change routing, but route changes alone do not establish downstream influence. The study traces this mismatch from gates through the routed block into the residual stream and finds exposure governed by backbone share, with bounded, mostly magnitude-generic output effects.
- Conditioning can change an MoE route despite identical weights, but route change is not equivalent to downstream influence.
- The decomposition follows routing mismatch from the gates through the routed block and into the residual stream.
- Backbone share orders exposure, while causal rescaling indicates that always-on mass contains mismatch chiefly by preserving routing coherence.
- Output patches reveal a bounded routing effect that is mostly generic in magnitude.
Reproducibility statement
The analysis uses fixed seeds, stochastic controls, and frozen captures for deterministic reduction, while fresh model acquisition is treated as scientific replication rather than guaranteed bitwise replay.
- Reproducibility statement: Code and analysis-ready captures underlying the reported figures and tables are publicly available, with public model release names listed in Appendix B.The repository is https://github.com/CedricCaruzzo/routing-divergence-moe.
A Derivation and anchor choice · B Uncertainty and boundary checkpoints
The derivation uses a non-unique routing/content split anchored on the student hidden state, while alternate-anchor checks show the reported norm-based quantities are robust. Prompt-level bootstrap intervals quantify sampling uncertainty, and boundary checkpoints extend the analysis beyond the primary screen.
- A Derivation and anchor choice: The routing/content split is not unique because terms can be added and subtracted without changing the decomposition.
- A Derivation and anchor choice: The analysis uses the hS anchor, evaluating gate differences against the student-produced hidden state.The two anchors agree exactly when fj(hT ) = fj(hS).
- A Derivation and anchor choice: 0.96 to 0.99 was the median per-layer ∥A′∥/∥A∥ across seven models, indicating close agreement between alternate-anchor and reported magnitudes.The alternate-anchor magnitude agreed with the reported one to within a few percent per layer.
- A Derivation and anchor choice: 4.6% was the maximum per-model movement in the routing factor under A′, with ranges 0.084–0.127 versus 0.085–0.133 and the same ordering.These results support anchor robustness for the reported norm-based quantities.
- B Uncertainty and boundary checkpoints: 95% bootstrap CIs resampled n=200 probe prompts over 2000 resamples to quantify prompt-sampling uncertainty for each checkpoint.The intervals concern each checkpoint rather than variation across checkpoints or uncertainty over a population.
- B Uncertainty and boundary checkpoints: Boundary checkpoints comprised descriptive family-pair scale probes and a merged checkpoint testing transfer outside jointly trained expert families.Exposure used the pre-block screen convention.
- B Uncertainty and boundary checkpoints: Qwen3-235B-A22, DeepSeek-V2-Chat, and Beyonder-4x7B-v3 were used for the boundary probes.
- B Uncertainty and boundary checkpoints: 2.5 and 97.5 percentiles defined the bootstrap intervals, which pinned each checkpoint tightly relative to the observed two-domain range.The full-data reduction reproduced the committed headline exactly.
C Robustness and exclusions · D Intervention details and stopped extension · E Wedge consistency
Robustness checks support the preregistered estimator while defining clear limits from reconstruction validity, format sensitivity, held-out-domain results, and the stopped Moonlight extension. Confirmatory interventions reproduce across domains, and wedge calculations remain numerically consistent under their stated aggregation conventions.
- C Robustness and exclusions: 20–38% inflation from tokenwise-ratio averaging widened the seven-checkpoint ranges but preserved their central ordering, motivating the preregistered per-layer-first reduction.The widened ranges were 0.089–0.150 for the routing factor and 0.020–0.073 for exposure.
- C Robustness and exclusions: All seven primary checkpoints passed identity closure; one quantized checkpoint with reconstruction error 1.06 was excluded from results using A or B.The exclusion defines the instrument’s operating envelope rather than providing evidence for the containment claim.
- C Robustness and exclusions: Removing the chat template shifted the routing factor by 0.007 on Mixtral and 0.017 on Qwen3-30B, while the held-out MBPP screen left six of seven checkpoints inside the primary range.OLMoE was marginally below the primary floor at 0.082 versus 0.085, so no further inference was drawn from that third-domain screen.
- D Intervention details and stopped extension: The PubMedQA dose trajectory and matched-drift controls used post-block exposure, keeping the intervention target distinct from the screen’s pre-block convention.Tables 4 and 5 contain the two confirmatory checkpoints’ dose and control results.
- D Intervention details and stopped extension: The same n = 100 GSM8K protocol reproduced a monotone response against a nearly flat null and the high-drift control ordering on both confirmatory checkpoints.The GSM8K table reports content replacement, matched noise, and mass removal in the highest drift bin.
- D Intervention details and stopped extension: Moonlight-16B-A3B was excluded from the confirmatory mechanism claim after its screened routing factor fell below range and exposure turned over at full removal on both domains.The preregistered all-dose monotonicity gate and parking rule fired; only the stopped trajectory is reported.
- E Wedge consistency: 10−16 agreement between recomputed and stored per-layer-first rA/res values validates the wedge identity across all models and both domains.Figure 3 uses a different aggregate reduction for backbone share, so its product with the plotted routing factor need not reproduce the headline value.