The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level

Jeremy Herbst; Stefan Wermter; Jae Hee Lee

Source-linked AI summary

The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level

Jeremy Herbst, Stefan Wermter, Jae Hee Lee

arXiv:2604.02178v2Apr 2, 2026cs.CLcs.AIcs.LG

TL;DR

The paper asks whether sparse routing makes MoE models more interpretable than dense FFNs, and investigates this with probing and expert-level analysis. It finds lower polysemanticity in MoE neurons and identifies experts as fine-grained task specialists, supporting scalable expert-level interpretation.

Problem
MoE sparsity improves computational efficiency, but it remains unclear whether it also makes these models easier to interpret than dense FFNs.
Method
The study compares MoE experts with dense FFNs using k-sparse probing, then automatically labels and scores hundreds of experts at the expert level.
Results
MoE expert neurons exhibit lower polysemanticity than dense FFN neurons, with the gap widening under sparser routing; experts specialize in fine-grained computational tasks rather than broad domains or simple token processing.
Takeaways & Limitations
MoE experts can serve as scalable, functional units for interpreting model computation and mapping modular computational pipelines.
Takeaways & Limitations
The study excludes the largest current MoE models and does not claim that experts are entirely monosemantic; labels may also hide contextual interactions and should be validated with counterfactual tests.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) architectures have become the dominant choice for scaling Large Language Models (LLMs), activating only a subset of parameters per token. While MoE architectures are primarily adopted for computational efficiency, it remains an open question whether their sparsity makes them inherently easier to interpret than dense feed-forward networks (FFNs). We compare MoE experts and dense FFNs using $k$-sparse probing and find that expert neurons are consistently less polysemantic, with the gap widening as routing becomes sparser. This suggests that sparsity pressures both individual neurons and entire experts toward monosemanticity. Leveraging this finding, we zoom out from the neuron to the expert level as a more effective unit of analysis. We validate this approach by automatically interpreting hundreds of experts. This analysis allows us to resolve the debate on specialization: experts are neither broad domain specialists (e.g., biology) nor simple token-level processors. Instead, they function as fine-grained task experts, specializing in linguistic operations or semantic tasks (e.g., closing brackets in $\LaTeX{}$). Our findings suggest that MoEs are inherently interpretable at the expert level, providing a clearer path toward large-scale model interpretability. Code is available at: https://github.com/jerryy33/MoE_analysis.

1. Introduction

The paper asks whether sparse routing makes MoE models more interpretable than dense FFNs, and investigates this across models using probing and expert-level analysis. It finds lower polysemanticity in MoE experts and characterizes them as fine-grained task specialists.

1. Introduction: MoE sparsity is motivated by efficient scaling because only a fraction of parameters is activated for each token.The paper frames interpretability as an additional potential benefit of this architectural choice.
1. Introduction: MoE experts’ neurons are less polysemantic than dense FFN neurons, with the gap widening as routing becomes sparser.The authors attribute this pattern more strongly to architectural sparsity than to total parameter count.
1. Introduction: The study treats the expert, rather than the individual neuron, as a scalable unit for interpreting MoE model components.Automatic labeling and scoring are used to interpret hundreds of experts, with causal attribution supporting the generated labels.
1. Introduction: Experts are neither broad domain specialists nor simple token processors; they often perform fine-grained linguistic or semantic tasks.For example, an expert may operate on closing brackets in LATEX rather than representing the entire LATEX domain.

2. Related Work

Prior work has studied both post-hoc interpretability methods and the possibility that MoE sparsity produces cleaner representations. The paper positions its contribution between competing accounts of expert specialization.

Interpretability in Dense Transformers: Dense-model interpretability commonly uses sparse autoencoders and transcoders to disentangle polysemantic neuron activations, but these methods are computationally expensive.The related-work discussion describes large datasets and substantial computation as requirements.
MoE as a Path to Interpretability: Earlier toy-model results suggest that increasing routing sparsity can decrease superposition, but whether this extends to large-scale LLMs remains open.The paper tests this unresolved question in production-scale models.
Architectures Aimed at Interpretability: Existing research has proposed MoE-style architectures to promote modularity and monosemanticity, complementing this paper’s empirical analysis of existing models.
The Debate on Expert Specialization: The expert-specialization debate contrasts broad domain specialization with token-level or syntactic specialization.This paper instead argues that experts function as fine-grained task experts.

3. Preliminaries

The preliminaries define MoE routing, component activations, sparse probing, attribution, and automatic labeling. These tools support comparisons of expert and dense-FFN representations and causal interpretation of expert outputs.

Transformers and the Residual Stream: Expert or dense-FFN outputs are added to the residual stream, enabling attribution methods to decompose a model’s representation into component contributions.The final logits are obtained by normalization and projection through the unembedding matrix.
Mixture-Of-Experts (MoE): An MoE layer routes each token to a selected subset of N experts, weighting their outputs according to router scores.Routing sparsity is defined as N_A/N, with smaller values indicating sparser routing.
Probing: The paper measures component polysemanticity with k-sparse probes that predict binary concepts from only k activation dimensions.Varying k tests how widely a concept is distributed across neurons; k=1 indicates a monosemantic neuron when prediction is accurate.
Logit-Space Projections and Attribution: Direct Logit Attribution estimates how a component update affects a target-token logit by projecting it through the unembedding matrix.A first-order linearization around the final residual state makes these contributions approximately additive.
Automatic Interpretability: Automatic interpretability uses an LLM explainer to propose expert labels and a separate scorer to evaluate those labels on held-out examples.

4. Quantifying the Monosemanticity of MoE Experts

k-sparse probing shows that MoE experts represent concepts more monosemantically than dense FFNs, with the advantage increasing as routing becomes sparser. This supports analyzing experts as interpretable modules rather than relying only on individual neurons.

4.2. Experts Approach Monosemanticity: MoE experts often achieve near-optimal F1 with a single neuron, while dense models show their largest performance gap between k = 1 and larger probes.The comparison evaluates best-layer probes across matched models and concepts.
4.3. Consistency Across Concept Categories: MoE models outperform dense models across Part-of-Speech, LATEX, code, and natural-language concepts, with nearly every comparison above the equality line.This indicates the pattern is consistent across the four evaluated concept categories.
4.3. Consistency Across Concept Categories: OLMoE-1B-7B, with 1B active parameters, significantly outperforms dense OLMo-7B despite the dense model using 7× more active parameters per token.The within-family comparison addresses whether reduced polysemanticity merely reflects greater total model capacity.
4.4. The Impact of Routing Sparsity: Models with lower routing ratios NA/N exhibit cleaner representations, with Qwen3-30B-A3B outperforming the denser Mixtral-8x7B at approximately 0.06 versus 0.25.The reported trend links greater routing sparsity with increasingly monosemantic internal units.
4.5. From Neurons to Experts: Because sparse routing filters expert inputs and expert neurons are less polysemantic, the entire expert can serve as an interpretable module.This motivates shifting analysis from individual neurons to experts without additional sparse autoencoders.

5. Automatically Interpreting MoE Experts

The paper introduces a scalable pipeline that automatically labels MoE experts from causally active examples and validates those labels with separate scoring and attribution tests. Across multiple models, experts receive consistently high interpretability scores, especially under sparser routing, and matched prompts elicit expert-specific causal behavior.

5.1. Automatic Labeling: The pipeline selects sequences by maximizing the router-weighted output norm gi(x) ∥Ei(x)∥2, targeting examples where an expert makes a large residual-stream contribution.This measure distinguishes actual computation from merely being selected by the router or having active internal neurons.
5.1. Automatic Labeling: An explainer LLM uses 20 top-activating sequences and top-3 promoted tokens to generate a one-sentence expert hypothesis, which a separate scorer evaluates with F1 on positive and negative examples.The scorer receives 10 positive and 10 negative examples, while both models are Gemini 3 Flash Preview.
5.2. Expert Interpretability: Most experts achieve F1 scores above 0.8 across analyzed layers, with Qwen3-30B-A3B frequently exceeding 0.9.The evaluation covers 8 layers of OLMoE-1B-7B, 3 layers of ERNIE-4.5-21B-A3B, and 3 layers of Qwen3-30B-A3B.
5.2. Expert Interpretability: Sparser routing is associated with more reliable expert labels: Qwen3-30B-A3B at NA/N ≈0.06 scores highest, while denser ERNIE-4.5-21B-A3B is lower and more variable.The paper links this pattern to cleaner expert representations under sparse routing.
5.3. Causal Attribution: Trigger-target tests show causal specificity: matched prompts usually place the labeled expert among the Top-1 or Top-8 contributors, whereas control prompts almost never route it.In 80% of cases, the expert was not routed to control prompts but was consistently routed and influential for matched prompts.

6. Expert Specialization

Experts are best understood as fine-grained task specialists rather than broad domain experts or simple token processors. Quantitative specialization analysis supports this interpretation, especially for deep-layer experts.

6. Expert Specialization: Experts function as fine-grained task specialists performing precise, often domain-restricted computational operations.Their roles include morphological, syntactic, semantic, and operational behaviors.
6.1. A Taxonomy of Expert Roles: Morphological experts combine subword fragments, such as suffixes or stems, to support rare-word continuations.OLMoE-L1-E57 activates on amine in glutamine and promotes continuations for rare chemical terms.
6.1. A Taxonomy of Expert Roles: Syntactic experts promote likely continuations after coordination markers, such as completing “boots and” with alike or like.ERNIE-L15-E0 exemplifies this coordination-specific behavior.
6.1. A Taxonomy of Expert Roles: Semantic experts reinforce domain-specific continuations, while operational experts enforce local formatting constraints such as closing LaTeX delimiters.Examples include legal and patent continuations and OLMoE-L15-E17 promoting closing delimiters inside LaTeX blocks.
6.2. Experts in the Output Embedding Space: Specialization scores compare expert behavior with layer base rates using Routing Specialization for inputs and Functional Specialization for outputs.The method uses native k-means clusters of the output embedding matrix, Jensen-Shannon divergence, and a random-expert baseline.
6.2. Experts in the Output Embedding Space: At high granularity, k = 5000 substantially exceeds broad-category k = 10 specialization, supporting deep-layer experts as granular task specialists rather than broad domain experts.Routing specialization is bimodal, while functional specialization shifts most dramatically in late layers.

7. Discussion

The discussion argues that increasing routing sparsity may improve interpretability and that experts can be viewed as modular computational sub-circuits. It also qualifies the findings by noting model coverage and residual superposition limitations.

The Scaling of Sparsity: Increasing routing sparsity may make models more interpretable by moving them toward monosemantic experts and reducing the interpretability cost of dense architectures.The authors frame this as a possibility if the observed relationship holds at the limit.
Experts as Sub-Circuits: MoE experts can be viewed as discrete sub-routines in modular computational toolboxes rather than components of monolithic thematic encyclopedias.This interpretation aligns with a circuits view of interacting functional units.
Limitations: The study excludes the largest current MoE models because of compute constraints and does not claim that experts are entirely monosemantic.The authors expect similar findings for models with comparable sparse routing, while acknowledging residual and between-expert superposition.

8. Conclusion

The paper concludes that sparse routing gives MoE models an interpretability advantage over dense transformers. Expert-level analysis scales to identifying task-specific functional operations.

8. Conclusion: MoE neurons show lower polysemanticity than dense-FFN neurons, and expert-level analysis identifies hundreds of specialized task experts performing functional operations.The conclusion presents sparse routing as closely associated with this relative monosemanticity.

Impact Statement

The paper argues that expert-level interpretability may reduce the cost of auditing and enable targeted interventions. It also warns that labels can be misused or overtrusted without counterfactual validation.

Impact Statement: If the findings generalize, expert-level analysis may reduce auditing and debugging costs and support targeted interventions on small sets of experts.The paper contrasts this with neuron-level analyses.
Impact Statement: Expert labels should remain tentative because prompt dependence, dataset artifacts, expert interactions, and dual-use risks can undermine safe interpretation.The authors recommend ablations, counterfactual tests, and out-of-distribution checks rather than treating interpretability as a safety guarantee.

A. Further Probing Results

The appendix reports additional probing comparisons between dense and MoE models and summarizes expert-count analyses across concepts and layers.

The probing comparisons match dense and MoE models where possible and report 95% confidence intervals, with dense models shown in red and MoE models in blue.
Figure 9 counts experts whose F1 probe score is within 95% of the best expert for each concept and layer, stacking concept counts by layer.
The appendix lists model routing configurations and architectural details needed to interpret parameter-matching choices.

C. Probing Datasets

The probing evaluation covers reproducible datasets and concepts spanning linguistic, LaTeX, code, and natural-language categories, with regex-based extraction for several categories.

The appendix documents concept definitions and regexes to clarify how concepts are extracted and support reproducibility.
The probing datasets include Part-of-Speech, LaTeX, code, and natural-language text concepts, with the code category designed specifically for fine-grained concepts.
Token positions for LaTeX, code, and text concepts are extracted with regular expressions, while Part-of-Speech concepts are available at the word level.
OpenAI’s implementation differs from the other models by including clamping and a residual connection.

D. Automatic Interpretability Data Selection

The automatic interpretability data-selection appendix describes collecting expert activations and illustrates several low-scoring cases where generated hypotheses overfit the examples.

The pipeline samples one random 32-token sequence per document until 2 × 10^6 tokens, then selects top expert examples for explanation and scoring.
The appendix analyzes five failure cases with unusually low final F1 scores, using text examples from the explainer model’s data.
The OLMoE-L1-E2 hypothesis received F1 0.38 because it overfit prefix-following activations to brands, technical terms, and proper nouns.
The OLMoE-L11-E5 hypothesis received F1 0.46, with frequency bias producing perfect precision but recall of 0.3.

E.3. OLMoE-L15-E10

The appendix details low-scoring automatic interpretations for several experts, showing failures from semantic, domain, and syntactic overfitting despite sometimes precise hypotheses.

E.3. OLMoE-L15-E10: The OLMoE-L15-E10 interpretation scored F1 0.18 because semantic overfitting preserved precision 1.0 but reduced recall to 0.1.
E.4. Qwen3-L24-E76: The Qwen3-L24-E76 interpretation overfit rhetorical examples to philosophical, legal, or logical domains, yielding recall of 0.2.
E.4. Qwen3-L24-E76: The Qwen3-L24-E76 expert actually fired across explanatory and causal reasoning contexts spanning technical, everyday, business, and moderation text.
E.5. ERNIE-L15-E54: The ERNIE-L15-E54 interpretation incorrectly required a demonstrative-pronoun trigger, producing precision 1.0 but recall 0.1.
The appendix provides additional test-case examples and identifies highlighted trigger and target words in the DLA experiment.

G. Cluster Examples

The paper presents example clusters from k-means analysis to illustrate how expert-related token groups organize into syntactic or semantic patterns.

The example clusters are derived from the k-means clustering analysis in Section 6.2.
The clusters of OLMoE-1B-7B output embeddings mostly form syntactic or semantic groups of related tokens.
Three expert case studies illustrate notable specialization patterns identified through this analysis.

H.1. LATEX Bracket Closer

The case studies show experts specializing in distinct operational, domain-specific, and subword-level behaviors. These examples are paired with specialization scores and a divergence-based analysis relative to each layer’s base rate.

LATEX Bracket Closer: A perfect F1 score of 1.0 supports the label that OLMoE-L15-E17 closes LaTeX mathematical environments by predicting closing braces and formatting markers.Its highest activations occur on symbolic tokens, while promoted tokens include structural continuations such as closing braces and formatting markers.
Specialization Scores: For the LaTeX bracket closer, routing specialization remains near zero while functional specialization rises sharply at high k because the expert promotes a small set of syntactically similar tokens.The routed tokens span many domains, whereas the promoted brackets form a loose syntactic group.
RPG Specialist: OLMoE-L14-E59 receives a 0.82 F1 score for predicting mechanics, stats, and character classes in tabletop and video game RPGs.The expert promotes related terminology, classes, system terms, and combat statistics in RPG contexts.
Named-Entity Subword Detector: OLMoE-L9-E60 receives a 0.88 F1 score for recognizing proper names and locations from non-Western cultures, especially Asian and African contexts.Its activations target subword fragments in names and transliterated text, consistent with detecting recurring orthographic patterns.
Specialization Scores: Specialization is measured as Jensen-Shannon divergence between an expert’s cluster distribution and the aggregate distribution for its layer.The score ranges from 0 for indistinguishability from the layer average to 1 for an entirely disjoint distribution; a random-expert baseline is subtracted to reduce sampling-variance effects.

K. Automatic Interpretability Labels

Automatically generated labels cover many experts across the analyzed models, spanning syntactic operations, subword patterns, semantic categories, and domain-linked behaviors. The labels repeatedly describe fine-grained token and continuation functions rather than only broad topics.

The generated labels include syntactic delimiters, punctuation, operators, formatting markers, and structural text boundaries across code, mathematics, markup, and metadata.
Many labels identify subword fragments, suffixes, morphemes, and internal segments within technical terms, proper nouns, scientific names, URLs, and identifiers.
Other experts specialize in recognizable domains or content types such as technical metadata, scientific and medical language, proper names, religious terminology, legal text, and culinary instructions.
The labels include predictive continuation roles, such as completing proper nouns, idioms, multi-word phrases, relative-clause verbs, and domain-specific terminology.
The labels also capture semantic and linguistic functions, including logical relations, negation, explanatory structures, noun-phrase modifiers, and relational or directional expressions.