PaperlayerView exact revision on arXiv

Source-linked AI summary

The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise

Xi Wang, Soufiane Hayou, Eric Nalisnick

arXiv:2604.09780v1cs.AI

TL;DR

Expert specialization is commonly observed in MoEs, but its mechanism and interpretability remain poorly understood. The paper analyzes the linear router’s connection to hidden-state geometry and the role of load balancing. It finds that geometry explains routing similarity, while specialization patterns often fail to align with human semantic expectations.

  • Problem

    The mechanisms behind observed expert specialization and the interpretability of its patterns remain poorly understood.

    Source: §1 · p. 1

  • Method

    The paper analyzes how a linear router maps hidden states to expert logits, formalizes a geometry-aware bound, and examines load-balancing effects.

    Source: §4.1 · pp. 3–4

  • Results

    Different models solving the same question show ∼60% expert overlap, comparable to different questions, and specialization patterns also diverge between prompts and rollouts or collapse across unrelated inputs.

    Source: §1 · p. 2, §5.1 · p. 9

  • Takeaways & Limitations

    Understanding specialization is at least as difficult as understanding hidden-state geometry, and distinct hidden states can still activate similar experts.

    Source: §1 · p. 2

    Rigorous verification of the proposed mechanism requires a comprehensive set of MoE pretraining experiments.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, and specialization is therefore an emergent property of the representation space, not of the routing architecture itself. We confirm this at both token and sequence level across five pre-trained models. We additionally prove that load-balancing loss suppresses shared hidden state directions to maintain routing diversity, which might provide a theoretical explanation for specialization collapse under less diverse data, e.g. small batch. Despite this clean mechanistic account, we find that specialization patterns in pre-trained MoEs resist human interpretation: expert overlap between different models answering the same question is no higher than between entirely different questions ($\sim$60\%); prompt-level routing does not predict rollout-level routing; and deeper layers exhibit near-identical expert activation across semantically unrelated inputs, especially in reasoning models. We conclude that, while the efficiency perspective of MoEs is well understood, understanding expert specialization is at least as hard as understanding LLM hidden state geometry, a long-standing open problem in the literature.

1 Introduction

The paper explains expert specialization through hidden-state geometry: a linear router maps similar hidden states to similar experts, while load balancing suppresses shared directions. Yet the resulting patterns often resist human interpretation.

1 Introduction

  • Expert specialization is widely observed across domains, but its mechanisms and the human interpretability of its resulting patterns remain open questions.The paper asks what produces specialization and how interpretable the patterns are.Source: §1 · p. 1
  • Because the router is a linear projection, similar hidden states activate similar experts, making specialization an emergent property of representation geometry rather than routing architecture.The paper formalizes this relation as Prop. 1 and verifies it across five pre-trained MoEs.Source: §1 · p. 1
  • Load-balancing loss suppresses shared directions in correlated hidden states, potentially explaining specialization collapse under small-batch training.The proposed explanation connects reduced data diversity with less diverse routing.Source: §1 · pp. 1–2
  • Different models solving the same math question show ∼60% expert overlap, comparable to different questions, while prompt and deeper-layer routing also defy semantic expectations.The reported patterns include prompts with nearly identical prefilling experts but divergent generation routing, and unrelated inputs with identical deeper-layer prefilling usage.Source: §1 · p. 2
  • Understanding specialization is at least as difficult as understanding hidden-state geometry, and may be harder because distinct hidden states can still activate similar experts.The paper identifies hidden-state geometry as an open problem even in dense models.Source: §1 · p. 2

2 Related work

Prior work studies domain-specific experts, practical uses of specialization, and layer-wise representation changes. This paper focuses instead on why routing does or does not align with domains or semantics.

Identification of domain-specialized experts

  • Prior interpretability studies find a compact, domain-invariant expert alongside peripheral experts with domain sensitivity, while this paper explains such patterns through hidden-state geometry.The cited approaches include LogitLens-based analysis, causal-effect identification, and CommitteeAudit.Source: §2 · p. 2

Applications of expert specialization

  • Specialization supports language-specific expert selection for low-resource performance, while deterministic routing can also expose query information as a privacy side channel.The passage attributes these applications and concerns to separate recent work.Source: §2 · p. 2

Expert specialization insights from pre-trained MoE reports

  • Pre-training reports monitor specialization mainly to prevent expert homogenization and wasted capacity; DeepSeek V3 assesses redundancy through sensitivity to removing top experts.Mistral is noted as an exception, with little specialization likely related to sparse upcycling.Source: §2 · p. 2

Layer-wise behavior of hidden states in dense models

  • Because hidden-state geometry evolves across layers, layer-wise representation changes can produce different specialization patterns in early and late layers.The related dense-model work describes layer differences in inference function, representation richness, and syntactic versus semantic encoding.Source: §2 · p. 3

3 Background: MoE

A token-choice MoE replaces each transformer block’s dense MLP with a sparse expert module. A router selects a small expert subset for each token, keeping per-token computation relatively small while scaling total capacity.

3 Background: MoE

  • A transformer block applies self-attention followed by an MLP, with each sublayer wrapped in layer normalization and a residual connection.The background assumes a pre-norm formulation.Source: §3 · p. 3
  • Token-choice MoEs replace the dense MLP with a sparse mixture, activating only a small subset of parameters for each token while maintaining total model capacity.This formulation covers recent sparse MoE models.Source: §3 · p. 3
  • The router projects each token representation to expert logits and selects the top-k experts with the largest logits.The selected experts receive routing weights, usually derived through softmax or sigmoid-and-normalize.Source: §3 · p. 3
  • MoE designs vary in shared experts, routing noise, sparsity ratio, and the function used to normalize routing weights.These are described as variations within the token-choice formulation.Source: §3 · p. 3

4 Expert “specialization” arises from hidden state similarity

The section explains expert-usage similarity through hidden-state geometry and examines how load balancing and context shape routing. It then shows that these geometric patterns do not make expert specialization straightforward to interpret at finer scales.

4.1 Hidden states, expert usage, and the role of auxiliary loss

  • Across five pre-trained MoEs, similar hidden states imply similar router logits, while dissimilar states can yield similar or dissimilar logits.The bound is data-aware and sharp when data variance is concentrated in the top r directions.Source: §4.1 · pp. 3–4, Figure 1 · p. 4
  • Load-balancing loss pushes routers to suppress shared hidden-state directions, helping preserve diverse routing as hidden states become correlated.The authors suggest this may explain specialization collapse under small-microbatch training; rigorous verification would require comprehensive MoE pretraining experiments.Source: §4.1 · pp. 4–5, Figure 2 · p. 5, §4.2 · p. 6
  • Shuffled and reversed sequences produce lower router confidence than original inputs, consistent with reduced router–data alignment.Confidence is measured by maximum softmax probability.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 Expert activation patterns are context dependent

  • For the same token or sentence, expert usage becomes more context-dependent in deeper layers.The study reports high first-layer overlap for the token across contexts, with overlap decreasing in later layers.Source: §4.2 · pp. 6–7, Figure 4 · p. 7

4.3 Sequence and dataset-level expert specialization

  • Across models, sequence-level hidden-state similarity strongly correlates with expert-usage similarity; cross-dataset pairs also have lower hidden-state similarity than within-dataset pairs.Some models show a many-to-one pattern: dissimilar sequences may activate either similar or different experts.Source: §4.3 · p. 7, Figure 5 · p. 7
  • Different models answering the same question share only ∼60% expert overlap, comparable to same-model answers to different questions and below the same-model, same-question baseline.The gap is consistent across layers.Source: Figure 6 · p. 8
  • Paired queries with nearly identical prompt-phase expert usage can diverge during generation, and the prompt alone does not predict that change.Across the three query pairs, generation-phase similarity either holds or drops sharply.Source: Figure 7 · p. 8
  • Expert specialization is at least as hard to understand as hidden-state geometry, since different hidden states can still activate similar experts.The section also notes that similar hidden states can route differently once generation begins.Source: §4.3 · p. 8, Figure 7 · p. 8

5 Why expert specialization patterns are hard to understand

This section tests whether expert specialization can be read from surface content and finds that patterns vary across models, generation trajectories, and layers. The results make prompt-level or semantic interpretations unreliable.

5.1 Math solutions from different models activate different experts

  • ∼60% expert overlap separates different models solving the same question, comparable to same-model solutions to different questions and below the same-model same-question baseline.The gap holds across all layers and three evaluated MoEs.Source: §5.1 · p. 9

5.2 Rollout content dominates expert usage

  • All three matched query pairs show nearly identical prefill usage, but generation either preserves similarity or sharply reduces it, unpredictably from the prompt alone.Prompt-phase expert usage is therefore not a reliable proxy for full-generation usage; assessing specialization requires complete trajectories.Source: §5.2 · pp. 9–10

5.3 Router collapse for prefilling stage

  • Unrelated HLE and Wimbledon sequences activate identical experts in deeper gpt-oss-20b layers during prefilling, while their profiles diverge during generation.Pruning experts in the last two layers changes prompt-only NLL by less than 10%, but those experts become consequential with on- or off-policy completions. The authors hypothesize that reasoning outputs and convergent initial-token representations drive prefilling collapse; duplicating inputs can also force convergence where hidden-state similarity alone does not.Source: Figure 9 · p. 9, Figure 8 · p. 9, §5.3 · p. 11

6 Conclusion

Expert specialization signals diverse feature learning and is necessary for parameter efficiency. Without an explicit objective, hidden-state and router-logit spaces should not be expected to align with human standards.

6 Conclusion

  • Specialization is a necessary condition for parameter efficiency: without distinct routing, experts would become duplicates and waste parameters.It also signals feature learning that produces diverse hidden states for different inputs.Source: §6 · p. 11
  • Without an explicit objective, hidden-state and router-logit spaces should not be expected to have a metric system aligned with human standards.Source: §6 · p. 11
  • Specialization confirms that the model is performing feature learning that produces diverse hidden states for different inputs.Source: §6 · p. 11

A Summarization of analysis metric

The analysis compares hidden-state similarity and expert usage at token and sequence levels using cosine, Hamming, and overlap measures. It also tracks hidden-state directional energy and how strongly the router preserves those directions.

A Summarization of analysis metric

  • Directional energy measures feature energy captured by singular directions, while retained energy measures how strongly the router preserves each direction.Low retained energy for dominant shared directions suggests router suppression that may help maintain balanced expert usage.Source: A Summarization of analysis metric · p. 14
  • Inter-sequence average cosine similarity summarizes pairwise hidden-state or router-logit similarity across the lower triangular kernel matrix.Source: A Summarization of analysis metric · p. 14
  • Router Hamming similarity averages pairwise token-level expert-usage similarity, measuring expert usage more directly than Eq. (10).Source: A Summarization of analysis metric · p. 14
  • Sequence-level expert-frequency similarity compares normalized expert-visit frequency vectors using cosine similarity.Top-P overlap instead compares the Jaccard similarity of the smallest frequency-ranked expert sets whose cumulative mass exceeds P.Source: A Summarization of analysis metric · p. 15
  • Pooled hidden-state cosine similarity compares sequence representations formed by normalizing token states and pooling across tokens.Source: A Summarization of analysis metric · p. 15

B Expert properties arise from the embeddings

A data-aware Lipschitz bound projects hidden states onto their principal subspace before bounding router-logit distances. When variance concentrates in a few directions, this bound is sharper for correlated hidden states.

B Expert properties arise from the embeddings

B.1 Shared directions become routing-insignificant under load balancing.

Under a simplified load-balancing objective and a small-logit condition, the analysis shows that a shared hidden-state direction becomes routing-insignificant. In practice, the effect is expected to be partial because load balancing is only an auxiliary part of pretraining.

B.1 Shared directions become routing-insignificant under load balancing.

  • Under the small-logit condition, minimizing the simplified load-balancing loss forces the shared direction to become routing-insignificant.Source: §B.1 · pp. 17–18
  • A shared direction that shifts every expert logit equally creates no relative expert preference, so routing is invariant along it.Load balancing therefore suppresses the common component and routes using residual variations in the correlated-data model.Source: §B.1 · p. 18
  • The analysis isolates load-balancing training, so real pretraining should downscale correlated directions rather than make them fully insignificant.The router is trained alongside other weights, and load balancing is an auxiliary loss.Source: §B.1 · p. 18

C.1 GPT-OSS-20b

This section presents two GPT-OSS-20b prompts with partial output excerpts: a Wimbledon-history summary and a step-by-step HLE question response.

C.1 GPT-OSS-20b

  • The Wimbledon prompt asks GPT-OSS-20b to summarize a paragraph about Wimbledon history, and the excerpt ends with “Very long history.”Source: §C.1 · p. 19
  • The HLE prompt requests a step-by-step answer, and the excerpt begins by framing the task as interpreting the problem.Source: §C.1 · p. 19

C.2 Duplication

The duplication experiments link increased hidden-state similarity with increased expert-usage similarity, but this does not mean different inputs produce the same outputs. They also report distinct routing behavior across reasoning models and expert ranks.

C.2 Duplication

  • Duplication increased expert-usage similarity, but different inputs did not collapse to the same output; random decoding instead lost randomness.Source: §C.2 · p. 19
  • For the same sequence and layer, AdamW-trained embeddings had higher RMS distance and greater norm variation, while Muon-trained embeddings had more concentrated norms and smaller pairwise RMS distances.Source: Figure 11 · p. 21
  • On GPT-OSS, rollout hidden-state similarity fell below the question-only line; on another dense reasoning model, similarity increased with depth without clear separation in deeper layers.Source: §C.2 · p. 22, Figure 14 · p. 22
  • At K = 1200 of 2048 directions, more than 40% of tokens changed their selected expert for the sixth-ranked expert when tail directions were removed.Source: Figure 24 · p. 28
  • On Ernie-4.5, duplicating sequences increased token and sequence hidden-state similarity alongside expert-usage similarity.Source: Figure 26 · p. 30

Related papers

Loading 2604.09780v1…