PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

Huiyuan Tian, Bonan Xu, Shijian Li

arXiv:2607.28308v1cs.LG

TL;DR

Route quality alone does not reveal why several experts help. This paper separates geometric overlap, route coherence, candidate quality, contextual interaction, and functional value, finding that actual context narrows geometric advantage even as later experts can remain useful.

  • Problem

    Route quality alone does not reveal why several experts help.

    Source: §7 · p. 11

  • Method

    The paper separates geometric overlap, route coherence, candidate quality, contextual interaction, and functional value using ESSI, a prefix-controlled factorial, and functional interventions.

    Source: §7 · p. 11

  • Results

    In all 39 factorial cells, the interaction is negative and every 95% interval lies below zero; across six MoEs, routes remain token-coherent despite overlap, and later experts can remain useful.

    Source: §4.2 · p. 9, §7 · p. 11

  • Takeaways & Limitations

    Multi-expert benefit can arise from distinct computations within a shared token-relevant neighborhood without disjoint linear coverage.

    Source: §7 · p. 11

    Conclusions concern a linear, rank-128 router-input metric, factorial analyses of three architectures, and one small-scale matched-compute setting.

    Source: §6 · p. 11

Abstract

from arXiv · show

Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions. Existing evidence often conflates route coherence, candidate quality, and candidate-by-context interaction. We distinguish these quantities using an Expert Subspace Separation Index (ESSI), matched-route residuals, and a prefix-controlled $2\times2$ factorial; frozen-route interventions and a controlled Top-$k$ study assess functional value. Three paired contrasts organize the findings. First, across six MoE architectures, expert subspaces overlap substantially, yet actual routes explain token representations better than matched alternatives. Second, across the 39 factorial cells in OLMoE, Mixtral, and DeepSeek, the selected candidate explains more of the residual representation than the strongest unselected rival in every cell, yet the actual prefix narrows this advantage throughout: all interactions are negative, and every 95% confidence interval lies below zero. Third, this geometric narrowing does not imply functional redundancy: adding later experts improves next-token prediction in 24 of 39 frozen-route comparisons, while the other 15 estimates are inconclusive; a controlled training study also favors Top-2 over Top-1 in all three seeds. We call this joint pattern coherent overlap: routing selects token-relevant experts from a shared geometric neighborhood, while useful multi-expert computation persists without disjoint linear coverage. Separating these quantities clarifies why geometric similarity alone cannot determine redundancy or pruning value.

1. Introduction

This introduction argues that multi-expert routing cannot be explained by geometric separation alone: it distinguishes route coherence, candidate quality, and candidate-by-context interaction, then reports a pattern of coherent overlap.

1. Introduction

  • Route coherence, candidate quality, and positive geometric complementarity are distinct: strong candidates can form coherent routes even when preceding experts have covered directions they might add.Identifying candidate-by-context interaction requires crossing candidates and contexts while holding each fixed in turn.Source: §1 · pp. 2–3
  • ESSI calibrates between-expert separation against local within-expert dispersion, while a prefix-controlled 2 × 2 factorial crosses selected versus rival candidates with actual versus matched prefixes.The factorial’s difference-in-differences tests whether a selected candidate’s advantage is larger in its actual co-selection context.Source: §1 · p. 3
  • Across six architectures, expert subspaces overlap while actual routes fit tokens better than matched alternatives; in 39 factorial cells, actual prefixes narrow selected candidates’ residual-fit advantage.Selected candidates explain more of the residual than their strongest unselected rivals, but their advantage is consistently narrower in the actual context.Source: §1 · p. 3
  • Later experts reduce next-token NLL in 24 of 39 frozen-route additions, with remaining estimates inconclusive; controlled Top-1/Top-2 training favors Top-2 in all three seeds.These functional findings show that geometric overlap alone does not establish redundancy or pruning value.Source: §1 · p. 3

2. Method

The method separates global expert-subspace geometry from prefix-conditioned candidate novelty, then uses a 2×2 factorial to distinguish candidate quality, context effects, and their interaction. It evaluates these measures across six MoE models and reports the geometry survey, factorial analyses, and frozen-route interventions.

2.1. Measuring Geometric Fit

  • Across 18 model–layer cells, ESSI ranged 0.776–1.060 (median 0.969), while alternative/actual residual ratios exceeded one in every cell (1.012–1.901; median 1.283).ESSI near one indicates inter-expert separation comparable to local within-expert variation; ratios above one indicate actual routes fit tokens better than matched alternatives.Source: Figure 2 · p. 4, §2.1 · pp. 4–5
  • ESSI compares global rank-128 expert subspaces with local tangent variation, while a separate uncentered rank-128 basis measures candidate novelty after a route prefix.The global basis is fitted from train-split router inputs; local tangent bases are fitted around sampled expert anchors from nearby assigned representations.Source: §2.1 · pp. 4–5

2.2. The Factorial Design

  • At each route position, the 2×2 factorial crosses the selected candidate or strongest outside-route rival with the actual preceding experts or a legal alternative context.Its contrasts measure candidate advantage, context effects, and the candidate-by-context interaction; a negative interaction means the actual co-selection context narrows the selected candidate’s advantage.Source: §2.2 · pp. 5–6

2.3. Models and Protocol

  • The study spans six MoEs and 18 geometry cells; factorial analyses cover 24 OLMoE/Mixtral and 15 DeepSeek cells, with frozen-route NLL interventions across all 39.Factorial cells use up to 2,048 held-out tokens per layer, load-matched alternative contexts, and 95% intervals from 1,000 paired source-context bootstrap resamples.Source: §2.3 · p. 6

3. Geometric Overlap and Route Coherence

This section tests global expert-subspace separation and whether actual routes fit tokens better than matched alternatives as distinct claims.

3.1. Expert Subspaces Overlap

  • ESSI ranged from 0.776 to 1.060, with median 0.969, across six models and 18 analyzed layers, indicating substantial expert-subspace overlap.Median overlap with a shared core rose from roughly 0.27–0.44 at dimension 64 to 0.77–0.98 at dimension 512; mean offsets and other forms of specialization are outside this metric’s scope.Source: §3.1 · p. 6

3.2. Routes Remain Coherent

  • The alternative-to-actual residual ratio exceeded one in all 18 cells (1.012–1.901; median 1.283), showing actual routes fit tokens better than matched alternatives.The comparisons use load-matched alternatives of the same size; the pattern indicates structured overlap rather than interchangeability.Source: §3.2 · p. 7

4. Strong Candidates, Negative Interaction

Across the factorial tests, selected candidates remained stronger than eligible rivals, yet actual prefixes consistently reduced their relative linear novelty. Sensitivity analyses preserved this negative interaction pattern.

4.2. Factorial Results

  • Across 39 cells, selected experts explained more residual than eligible rivals, but actual prefixes narrowed this advantage; every interaction and 95% confidence interval was negative.The interaction was negative in all 39 cells, with every 95% interval below zero.Source: Figure 4 · p. 8, §4.2 · pp. 8–9

4.1. Why Naive Comparisons Reverse

  • When candidate and context changed together, later selected experts underperformed controls; with the prefix fixed, selected experts won all 39 comparisons.Changing both candidate and context confounds candidate quality with residual opportunity, producing the apparent reversal.Source: §4.1 · p. 8

4.2. Factorial Results

  • Every Ts and Tr estimate was negative, showing that the actual prefix left less residual opportunity for either candidate; selected-candidate novelty also ranked near the bottom.Equal-length prefixes isolate context effects; actual-prefix novelty percentiles were 0.006–0.154 for OLMoE/Mixtral and 0.007–0.079 for DeepSeek.Source: §4.2 · p. 9

4.3. Sensitivity Analyses

  • Across sensitivity variants, the actual prefix consistently reduced selected-candidate relative linear novelty, supporting coherent overlap in a shared token-conditioned geometric neighborhood.Raw-gain and nearest-match intervals remained negative in all 24 OLMoE/Mixtral cells; strict-caliper point estimates were negative in all 24, with 20 intervals excluding zero.Source: §4.3 · p. 9

5. Functional Value Within Overlapping Geometry

Later experts can improve prediction despite low residual linear novelty, and their value is distributed across the route; a controlled Top-2 configuration also outperforms Top-1. Thus, overlapping input geometry does not by itself establish functional redundancy, because experts apply nonlinear transformations.

5. Functional Value Within Overlapping Geometry

  • Negative geometric interaction describes residual linear coverage, not computation value: similar input subspaces can support different features or output directions because experts apply parameterized nonlinear transformations.The probes are complementary: additions measure marginal value in a pretrained route, replacement tests concentration in the leader, and matched training permits adaptation.Source: §5 · pp. 9–10
  • 24 of 39 frozen-route additions reduce next-token NLL; gains are strongest early and decay with router rank, from 0.0906 [0.0836, 0.0975] for the first OLMoE layer-16 addition to 0.0030 [0.0021, 0.0039] for the last.The other 15 intervals include zero. OLMoE is positive in 17 of 21 cases, Mixtral at all three layers, and DeepSeek has four positive additions at early ranks.Source: Figure 5 · p. 10, §5 · p. 10
  • Top-2 has lower validation loss than Top-1 in all three seeds, with ∆= 0.1016 ± 0.0025 under the controlled-compute comparison.The six-layer study matches active intermediate capacity using Top-1 with 4 experts of width 1,024 and Top-2 with 8 experts of width 512; total parameters and active compute differ by less than 0.04%. Expert count and width change jointly, so the result applies to this configuration and regime.Source: Figure 5 · p. 10, §5 · p. 10
  • In seven of nine configurations, replacing the leader causes more NLL damage than replacing the aggregate later set, but OLMoE layers 8 and 16 reverse this pattern.At OLMoE layer 16, replacing the seven later experts causes 0.0848 [0.0724, 0.0971] more aggregate NLL damage than replacing the leader.Source: §5 · p. 10, Figure 5 · p. 10

6. Potential Impact on MoE Analysis and Design

The framework separates routing coherence, fixed-prefix candidate quality, and context-specific geometric synergy, while emphasizing that geometric overlap alone cannot establish functional necessity.

6. Potential Impact on MoE Analysis and Design

  • A better route establishes coherence, a better candidate under a fixed prefix establishes quality, and only their interaction establishes context-specific geometric synergy; functional necessity requires an output-level counterfactual.Source: §6 · p. 11
  • The negative interaction reflects geometric saturation: the actual prefix can remove directions available to both candidates, reducing the stronger candidate’s marginal novelty despite its fixed-prefix advantage.Changing both context and candidate mixes candidate quality with residual opportunity, explaining the reversal in Figure 3.Source: §6 · p. 11
  • Overlap applies to rank-128 linear coverage of router inputs before expert transformation; it can coexist with distinct nonlinear features, output directions, or logit effects.Claims of overlap should specify representation, rank, metric, and intervention; semantic routing and expert collaboration may reveal specialization this subspace metric does not resolve (Olson et al., 2025).Source: §6 · p. 11
  • For compression, similarity can screen candidates, but pruning, merging, or skipping should be tested under retained routes because removal changes other experts’ context (Lee et al., 2025; Chen et al., 2025).Adaptive-k routing could instead estimate another expert’s output gain subject to load and compute constraints (Zeng et al., 2024).Source: §6 · p. 11
  • The conclusions are limited to a linear, rank-128 router-input metric, factorial analyses of three architectures, and one small-scale matched-compute setting.Appendix S7 provides additional scope and interpretation details.Source: §6 · p. 11

7. Conclusion

The paper identifies coherent overlap: routes select token-relevant experts from overlapping subspaces, and actual context narrows candidate advantage without eliminating the predictive value of later experts.

7. Conclusion

  • ESSI, a prefix-controlled 2 × 2 factorial, and functional interventions separate expert overlap, route coherence, candidate quality, contextual interaction, and predictive value.Route quality alone does not explain why multiple experts help.Source: §7 · p. 11
  • Across all 39 factorial cells, selected candidates explain more residual representation than rivals, yet negative interactions show actual context narrows this geometric advantage.Across six MoEs, eligible experts’ centered directional subspaces overlap under the study’s metric while routes remain token-coherent.Source: §7 · p. 11
  • Multi-expert benefits can arise from distinct computations within a shared token-relevant neighborhood, without requiring disjoint linear coverage.Frozen-route interventions and a controlled training study show that later experts can remain useful.Source: §7 · p. 11

Appendix S1. Scope, Notation, and Experimental Assets

This appendix distinguishes factorial rivals from Figure 3’s load-near controls and defines the conventions used to report intervals and layer indices. It also specifies the datasets used for pretrained analyses and matched-compute training.

Appendix S1. Scope, Notation, and Experimental Assets

  • “Rival” means the highest-scoring eligible unselected expert in the factorial; Figure 3 uses distinct load-near control candidates.Functional intervals overlapping zero are inconclusive.Source: Appendix S1. Scope, Notation, and… · p. 18
  • Unless a table says otherwise, θ̂ [l, u] is a point estimate with a 95% confidence interval; L is the analyzed layer index, not model depth.The three-seed matched-training summary uses mean ± standard deviation.Source: Appendix S1. Scope, Notation, and… · p. 18
  • Pretrained analyses use an 8,192-record corpus, while matched-compute training uses a separate 8,192-record corpus with an ARC science component.The first corpus has 2,048 records each from C4 (Raffel et al., 2020), CodeSearchNet-Python, ccdv/arxiv-summarization (Cohan et al., 2018), and UltraChat 200k (Ding et al., 2023). The second uses the same general, code, and dialogue sources plus 1,024 ARC-Easy and 1,024 ARC-Challenge training examples (Clark et al., 2018); raw third-party records are not redistributed.Source: Appendix S1. Scope, Notation, and… · p. 18

S2. ESSI and Route-Coherence Protocol · S2.1. Centered global and local tangent subspaces

The ESSI protocol estimates centered global and local tangent subspaces for eligible experts using routing-weighted samples and rank-128 PCA. It measures local subspace distance and normalizes by a routing-load-weighted mean.

S2.1. Centered global and local tangent subspaces

  • Each eligible expert’s global basis is a rank-128 centered, routing-weighted PCA basis; experts require at least 2,048 selected tokens.Assignment weights are native Top-k router weights renormalized to sum to one within each token.Source: S2.1. Centered global and local… · p. 18
  • At each anchor, the protocol centers the 256 nearest candidate representations and fits a rank-128 local PCA tangent basis; distance is normalized projection/chordal distance.Anchors and candidate pools are sampled without replacement in proportion to each expert’s routing weights; the distance is the standard Grassmannian metric (Edelman et al., 1998).Source: S2.1. Centered global and local… · p. 18
  • OLMoE, Mixtral, and DeepSeek use 2,048 anchors and up to 8,192 candidate tokens, while Qwen3, Gemma4, and Qwen3.6 use 512 anchors and up to 4,096 candidates.Sampling uses base seed zero plus layer index; the ESSI denominator is the routing-load-weighted mean expert-level local tangent distance, with floor ϵ = 10^-12.Source: S2.1. Centered global and local… · p. 18

S2.2. Route coherence

Route coherence compares normalized residual energy for the actual routed set against an equal-cardinality alternative, with shared experts held fixed outside the comparison. Alternative-set sampling differs across the six evaluated model families.

S2.2. Route coherence

  • Route coherence compares normalized residual energy for the actual routed set with an equal-cardinality alternative; shared experts remain fixed outside the comparison.Residuals use a centered basis refit per run and route-weighted expert means; alternatives reuse the actual route’s weights. Ratios above one mean alternatives leave more router-input energy unexplained.Source: S2.2. Route coherence · pp. 18–19
  • OLMoE, Mixtral, and DeepSeek use one load-matched alternative per token, while Qwen3, Gemma4, and Qwen3.6 use five alternatives per token.The latter runs restrict expert eligibility to 1,024 support tokens, retain tokens whose complete routed set is fitted, and average within source context before cell means.Source: S2.2. Route coherence · p. 18

S2.3. Shared-core overlap … S7.2. Reproducibility

Across the 39 factorial cells, selected-candidate advantage is positive while every candidate-by-context interaction interval is negative. Frozen-route additions improve next-token prediction in 24 of 39 comparisons, and Top-2 achieves lower validation loss in all three training seeds.

S2.3. Shared-core overlap; S3.1. Uncentered bases for the factorial analysis

  • The shared-core diagnostic measures the fraction of each expert basis captured by a common layer-wide core, complementing ESSI’s comparison of between-expert distance with within-expert dispersion.The factorial protocol uses uncentered rank-128 router-input bases and compares candidate and rival at the actual rank-j router weight, so expert identity changes without changing mixture scale.Source: S2.3. Shared-core overlap · p. 19, S3.1. Uncentered bases for the… · p. 19

S3.2. Load-near control diagnostic in Figure 3

  • Under actual context, all 39 same-prefix control intervals are positive, whereas all 39 later-expert intervals are negative when both context and candidate change.In the fixed-context comparison, point estimates range from 0.001382 to 0.038642 across 79,852/79,872 token-cells, motivating separation of candidate quality from contextual opportunity.Source: S3.2. Load-near control diagnostic in… · pp. 20–21, Table S4 · p. 21

S4. Complete Factorial Estimates

  • 39/39 factorial cells have positive selected-candidate advantage and negative candidate-by-context interaction intervals.The equal-cell interaction macro is −0.05548 [−0.05780, −0.05340] for OLMoE and Mixtral combined, and −0.05770 [−0.06165, −0.05405] for DeepSeek.Source: S4. Complete Factorial Estimates · p. 22, Table S5 · p. 23

S5. Sensitivity Analyses and Numerical Validation

  • Sensitivity analyses preserve negative interaction estimates under raw-gain and nearest-residual controls, while strict-caliper results have limited coverage.For OLMoE and Mixtral, raw-gain and nearest-residual intervals are negative in every cell; strict-caliper point estimates are negative in all 24 cells, with 20 intervals below zero and 7.88% token-cell coverage. Independent numerical validation stayed below tolerance with zero rank mismatches or nonfinite comparisons.Source: Table S6 · p. 24, S5. Sensitivity Analyses and Numerical… · p. 26, Table S7 · p. 26

S6.1. Leader-versus-later replacement; S6.2. Adjacent ordered-prefix NLL recovery

S6.2. Adjacent ordered-prefix NLL recovery

  • 24 of 39 frozen-route additions improve next-token prediction, while 15 estimates are inconclusive.OLMoE contributes 17 positive and four inconclusive additions; Mixtral has three positive additions; DeepSeek has four positive and 11 inconclusive additions.Source: Table S9 · p. 27, S6.2. Adjacent ordered-prefix NLL… · p. 27

S6.3. Matched-compute Top-1 versus Top-2

S7.1. Scope of the evidence; S7.2. Reproducibility

  • The factorial evidence covers OLMoE, Mixtral, and DeepSeek, with shared experts fixed in DeepSeek; score-ordered prefixes are analytical because standard Top-k experts execute in parallel.The analyses use seed-zero source-context splits, train-only expert fits, and context-level paired bootstrap intervals; NLL interventions and matched training provide functional evidence distinct from router-input linear geometry.Source: S7.1. Scope of the evidence · p. 28, S7.2. Reproducibility · p. 28

Related papers

Loading 2607.28308v1…