PaperlayerView exact revision on arXiv

Source-linked AI summary

The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models

Yan Wang, Yitao Xu, Nanhan Shen, Jinyan Su, Jimin Huang, Zining Zhu

arXiv:2601.03425v2cs.LGcs.AI

TL;DR

MoE models are commonly expected to specialize by domain, but whether experts form stable coalitions across contexts remains unclear. COMMITTEEAUDIT analyzes routing at the group level and finds high overlap and concentration across three models, with committee members anchoring reasoning and syntax while peripheral experts handle domain knowledge.

  • Problem

    Individual-expert analyses leave unclear whether experts form stable, domain-invariant coalitions across contexts.

    Source: §1 · p. 2

  • Method

    COMMITTEEAUDIT analyzes domain-conditioned routing profiles, inter-domain divergence, and Standing Committees in pre-trained MoE models.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    All three models show high expert overlap and routing-mass concentration, and committee members anchor reasoning and syntax while peripheral experts handle domain knowledge.

    Source: §4.1.1 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    The observed centralized structure suggests a tension with load-balancing objectives that encourage uniform expert use.

    Source: §5 · p. 9

    Evaluation centers primarily on domain-level MMLU analyses and may not capture conversational, multi-step reasoning, coding, or tool-augmented scenarios.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Mixture of Experts models are widely assumed to achieve domain specialization through sparse routing. In this work, we question this assumption by introducing COMMITTEEAUDIT, a post hoc framework that analyzes routing behavior at the level of expert groups rather than individual experts. Across three representative models and the MMLU benchmark, we uncover a domain-invariant Standing Committee. This is a compact coalition of routed experts that consistently captures the majority of routing mass across domains, layers, and routing budgets, even when architectures already include shared experts. Qualitative analysis further shows that Standing Committees anchor reasoning structure and syntax, while peripheral experts handle domain-specific knowledge. These findings reveal a strong structural bias toward centralized computation, suggesting that specialization in Mixture of Experts models is far less pervasive than commonly believed. This inherent bias also indicates that current training objectives, such as load-balancing losses that enforce uniform expert utilization, may be working against the model's natural optimization path, thereby limiting training efficiency and performance.

1 Introduction

MoE routing is often expected to divide computation among domain-specialized experts, but the paper reports cross-domain sharing even in architectures with shared experts. It introduces COMMITTEEAUDIT to study expert coalitions and their roles in that organization.

1 Introduction

  • MoE’s divide-and-conquer intuition assigns distinct experts to domains, yet substantial cross-domain sharing appears among routed experts even in architectures with shared experts.The paper argues that high-frequency, domain-agnostic language patterns motivate shared experts, but that architectural separation does not fully suppress centralization.Source: §1 · pp. 1–2
  • Prior individual-expert analyses leave relational stability across contexts and the possibility of stable, domain-invariant coalitions unresolved.The study frames existence, depth-wise dynamics, and functional roles as its central questions.Source: §1 · p. 2
  • A domain-invariant Standing Committee emerges regardless of shared-expert architecture, revealing structural bias that challenges fairness-oriented load balancing.The paper’s framing contrasts this committee with domain-specific specialization.Source: §1 · pp. 2–3
  • COMMITTEEAUDIT audits group-level organization with Pareto-optimal ranking and stability diagnostics across three models, including standard and shared-expert architectures.The models are OLMoE, Qwen3-30B-A3B, and DeepSeek-V2-Lite.Source: §1 · pp. 2–3
  • Committee members anchor logical and syntactic structures, while peripheral experts manage domain-specific knowledge.Source: §1 · p. 3

2 Related Works

Prior work has studied expert specialization, representations, and activity, while leaving stable group-level organization across tasks less clear. This work shifts the focus to domain-invariant expert committees.

2 Related Works

  • Early multilingual studies reported language-preferential experts, while analyses of general-purpose models find polysemy, weak alignment with semantic domains, and context-dependent specialization.Source: §2 · p. 3
  • Related work also examines shared latent representations and architectural interventions intended to make expert roles or routing more interpretable.Source: §2 · p. 3
  • Most prior studies treat experts individually, leaving stable co-activated groups across tasks unclear; this work identifies domain-invariant standing committees.Source: §2 · p. 3

3 COMMITTEEAUDIT

COMMITTEEAUDIT aggregates routing behavior by domain and uses expert contribution profiles to analyze task specificity and cross-domain expert organization. Its framework identifies candidate committees through cross-task agreement and rank stability, then quantifies expert sharing and contribution concentration.

3.1 Preliminaries

  • ECI measures an expert’s expected routing weight for a domain task, preserving router-preference magnitude rather than merely counting activations.Source: §3.1 · p. 3

3.2 Framework Description

  • The framework records full routing vectors, assesses task specificity with a silhouette score, and constructs committees from expert ranks and their cross-domain stability.Only tasks with sufficiently high silhouette scores proceed to committee construction.Source: §3.2 · p. 4
  • The Pareto-optimal Standing Committee favors experts with high average rank and low cross-domain rank variability.Experts must also appear among domain-level Top-k lists frequently enough to meet the cross-domain agreement threshold; this study uses γ > 0.8.Source: §3.2 · pp. 4–5

3.3.1 Dataset

  • MMLU’s 57 subjects are grouped into nine semantically coherent domains so the routing analysis reflects domain-level patterns rather than subject-specific idiosyncrasies.Source: §3.3.1 · p. 5

3.3.2 Model

  • The evaluation analyzes three inference-only MoE models; DeepSeek-V2-Lite has two always-active shared experts, while Qwen3-30B-A3B and OLMoE-1B-7B use routed experts only.Source: §3.3.2 · p. 5

3.4.1 Jaccard Similarity (Cross-Domain Expert Sharing)

  • Jaccard similarity between domain-level Top-k expert sets measures expert reuse; values near 0 indicate domain-specific routing, while larger values indicate more sharing.Source: §3.4.1 · p. 5

3.4.2 Gini Coefficient (Expert Concentration)

  • The per-layer Gini coefficient summarizes inequality in experts’ task-averaged ECI, with values near 0 indicating uniform contribution and high values indicating concentration in few experts.Source: §3.4.2 · p. 5

4 Experiment

Across the experiments, routed experts form a compact, persistent core that captures disproportionate routing mass across domains and layers. Qualitative analysis and masking interventions indicate that this core supports reasoning and syntax, while peripheral experts handle specialized knowledge.

4.1.1 Jaccard–Gini Analysis of Expert Sharing and Concentration

  • OLMoE achieved a mean Jaccard overlap of 0.8735 and minimum of 0.7963, while Qwen3 averaged 0.8670; all models had Gini coefficients above 0.88.Qwen3’s minimum overlap was 0.5300, and its overall Gini was 0.9465.Source: §4.1.1 · p. 6, Table 4 · p. 6

4.1.2 Analyzing the Stability and Contribution of Standing Committees

  • Committees of 2–5 experts captured up to 70.5% of routing mass, with size stable as capacity increased from E = 64 to E = 128.Representative committee members had mean ranks of about 3.1–3.8 and rank variance σ2 ≤ 3.44; the middle-layer OLMoE committee had variance 0.49.Source: §4.1.2 · pp. 6–7, Table 5 · p. 6, Figure 3 · p. 7

4.2.1 Robustness of Standing Committees

  • Across all three models, top-k expert sets had domain-overlap often ≥0.85, while OLMoE committee influence density stayed above 1 across layers and routing budgets.Increasing k added relatively low-density peripheral experts; committee size remained typically around 1–4 experts and changed only mildly.Source: §4.2.1 · pp. 7–8, Figure 4 · p. 7, Figure 5 · p. 8

4.2.2 Top-k Sensitivity Sweep

  • Against the k=8 reference committee, retention was 0.39 at k=6, 0.17 at k=4, and below 0.3 at k=12 and k=16.The committee can partially reorganize when the routing budget is substantially more constrained or more aggressive.Source: §4.2.2 · p. 8, Figure 6 · p. 8

4.3 Interaction and Behavior: Functional Interpretation

  • Across OLMoE and Qwen3, committee experts repeatedly handled logical framing and structural tokens, while domain-specific terminology was distributed across experts by context.The qualitative analysis supports a core-periphery pattern: committee members anchor reasoning and syntax, and peripheral experts are recruited for specialized knowledge.Source: §4.3 · pp. 8–9, Figure 7 · p. 9

4.4 Masking-Based Evidence of Functional Importance

  • Masking committee experts on DeepSeek-V2-Lite reduced the correct rate from 0.39 to 0.12, 0.09, and 0.03 across representative shallow, middle, and deep layers.No Answer responses rose from 0.03 to 0.36–0.38.Source: Table 6 · p. 9, §4.4 · p. 9

5 Conclusion

The conclusion presents COMMITTEEAUDIT as evidence that MoE models rely on a domain-invariant Standing Committee, and connects this structure to a tension with uniform-utilization objectives.

5 Conclusion

  • COMMITTEEAUDIT finds a domain-invariant committee that anchors reasoning and syntax while peripheral experts handle domain knowledge, suggesting a tension with load-balancing objectives that encourage uniform expert use.The authors propose function-aware routing and architectures that explicitly support a core-periphery organization.Source: §5 · p. 9

Limitations

The authors identify limits in architectural coverage, causal controls, evaluation scope, and analysis of training dynamics.

Limitations

  • The analysis covers a small set of representative MoE architectures and settings, not hybrid, hierarchical, or dynamically adaptive routing designs.Whether the same organizational patterns persist in broader systems remains open.Source: Limitations · p. 10
  • Masking interventions cover representative layers but do not constitute a fully controlled causal analysis, including comparisons with random or frequency-matched non-committee masking across models and tasks.This limits causal interpretation of the intervention results.Source: Limitations · p. 10
  • Evaluation centers on domain-level MMLU analyses, with supplementary C4 and intervention evidence, and may not capture conversational, multi-step reasoning, coding, or tool-augmented behavior.Source: Limitations · p. 10
  • COMMITTEEAUDIT analyzes routing statistics rather than training dynamics, leaving when and how committees emerge during optimization for future study.Source: Limitations · p. 10

Potential Risks

The authors flag risks of misreading routing structure as a deployment guarantee and of exposing persistent expert coalitions to adversarial targeting.

Potential Risks

  • Centralized computation could be misread as evidence of safety or robustness, while identified coalitions could enable adversarial targeting of critical routing pathways.The authors say these findings should be used as analytical tools rather than deployment guarantees.Source: Potential Risks · p. 10

Ethical Considerations and Usage Disclaimer

The notice limits the work's intended use and clarifies that COMMITTEEAUDIT is a structural analysis, not a certification for high-stakes properties. It also discloses the limited writing and editing role of AI tools.

Ethical Considerations and Usage Disclaimer

  • The work is intended for academic and educational purposes, not as guidance for production deployment.Source: Ethical Considerations and Usage… · p. 10
  • The framework does not certify fairness, safety, robustness, or regulatory compliance.High-stakes applications should involve domain experts, risk assessment, and human oversight.Source: Ethical Considerations and Usage… · p. 10
  • AI-assisted tools supported writing and editing, not research ideas, experimental results, model outputs, or claims.The authors state that they designed, verified, and interpreted the technical content, analyses, and conclusions.Source: Ethical Considerations and Usage… · p. 10

Licenses and Terms of Use

The study uses publicly available datasets and models under their respective licenses. Its inference-only MMLU analyses use specified software, hardware, model scales, and computational budget.

Licenses and Terms of Use

  • The datasets and evaluated models are publicly available and used under their respective licenses, including MMLU's public research license.Source: Licenses and Terms of Use · p. 10

A Computational Budget and Infrastructure

  • Experiments use Python, PyTorch, and HuggingFace Transformers in inference-only mode to evaluate each model on the full MMLU benchmark.Source: A Computational Budget and… · p. 12
  • The analyses use two NVIDIA A100 (80 GB) GPUs, with routing analyses costing about 40 GPU-hours across models of approximately 7B, 16B, and 30B parameters.Source: A Computational Budget and… · p. 12

B C4 Dataset

The paper describes a language-grouped C4 subset, clarifies the formal committee-selection rule, and reports compact Standing Committees across MMLU models and layers. The MMLU results show early centralization, stronger middle- and deep-layer concentration, and similar committee behavior across architectures.

B C4 Dataset

  • The C4 evaluation subset groups 1,000 samples from 10 languages into four language-oriented domains, with 100 samples per language and a moderately imbalanced domain distribution.The group counts are 400 European, 300 Asian, 100 Middle Eastern, and 200 Other samples; sequence lengths vary across groups, and each group includes samples at the 512-token truncation limit.Source: Table 7 · p. 12, B C4 Dataset · p. 12

C Illustrative Expert-Level Statistics for Committee Selection

  • Committee membership requires cross-domain agreement γ > 0.8 and Pareto-optimal rank statistics; domain coverage alone is insufficient.The auxiliary stability score is for interpretive presentation only, not formal selection.Source: C Illustrative Expert-Level Statistics… · p. 13

D.1 MMLU Dataset Results

  • Across OLMoE, DeepSeek-V2-Lite, and Qwen, compact Standing Committees emerge early, consolidate in middle layers, and persist into the deepest layers.Committee composition changes only gradually across layers.Source: §D.1 · p. 13
  • Committees typically contain 1–4 experts in OLMoE and DeepSeek-V2-Lite and occasionally up to five in Qwen, while middle-layer contribution often exceeds 50–65% in DeepSeek-V2-Lite and Qwen.Source: §D.1 · p. 13
  • Shallow committees already capture 20–40% of routing mass, while deep-layer committees in DeepSeek-V2-Lite and Qwen control 50–70%.The authors associate shallow centralization with token normalization and shallow syntactic and lexical patterns, and describe deep-layer decision-making as funneled through a narrow coalition.Source: §D.1 · p. 13

D.2 C4 Dataset Results

On free-form C4 inputs, DeepSeek-V2-Lite retains compact committees with substantial routing coverage. Lorenz-curve analyses further show concentrated expert contributions across layers in OLMoE, DeepSeek-V2-Lite, and Qwen3.

D.2 C4 Dataset Results

  • Committee-member to non-member average contribution ratios range from 13.48× to 45.22×, with the strongest concentration in deeper layers.Source: §D.2 · p. 15
  • C4 committees typically contain 1–4 experts and capture 12.43%–66.32% of routing mass across reported layers, including 66.32% at Layer 0 and 62.81% at Layer 24.The C4 analysis uses representative shallow, middle, and deep layers and finds the pattern persists beyond MMLU's multiple-choice format.Source: §D.2 · p. 14, Table 12 · p. 15

E.1 Lorenz Curve for OLMoE Model

  • OLMoE's Gini coefficients are around 0.88–0.90 across layers; typically 12%–20% of experts are used, and concentration remains stable from shallow to deep layers.Source: §E.1 · p. 15

E.2 Lorenz Curve for DeepSeek-V2-Lite Model

  • DeepSeek-V2-Lite's Gini coefficients are around 0.91–0.92 across layers, with fewer than 15% of experts accounting for nearly all effective contributions in several layers.The contribution pattern remains steeply skewed even when some layers activate roughly 20% of the pool and others as little as 9%.Source: §E.2 · p. 16

E.3 Lorenz Curve for Qwen3 Model

  • Qwen3's Gini coefficients are around 0.94 across layers, and fewer than 10% of experts account for nearly all effective contribution in several layers.The active-expert proportion fluctuates between 6% and 12% without changing the concentration pattern; the authors report stronger centralization with increased model capacity.Source: §E.3 · p. 16

E.4 Cross-Model Synthesis.

Across three architectures, routing mass is concentrated in a small, persistent expert core across layers and changes in the proportion of experts used. Masking that core in DeepSeek-V2-Lite degrades MMLU answer outcomes, especially in deeper layers.

E.4 Cross-Model Synthesis.

  • Across OLMoE, DeepSeek-V2-Lite, and Qwen-30B-A3B, Lorenz curves show extreme contribution concentration, with Gini coefficients typically above 0.88 and often exceeding 0.94.Only a small fraction of experts accounts for most effective routing mass; the remaining experts contribute negligibly.Source: §E.4 · p. 16, Figure 8 · p. 17, Figure 9 · p. 18
  • Early, middle, and late layers show nearly identical inequality profiles, with compact expert subsets repeatedly receiving computation regardless of layer position or domain.The observed concentration does not broaden in later layers as representations become more abstract.Source: §E.4 · p. 16
  • Increasing the proportion of nominally used experts does not materially change concentration; the same small core dominates while additional experts remain low-impact auxiliaries.Source: §E.4 · p. 16
  • The cross-model findings suggest concentration is not merely an artifact of scale, architecture, or routing hyperparameters, but a robust tendency toward a Standing Committee structure.The persistent core monopolizes computation while most experts operate peripherally.Source: §E.4 · p. 17

F Full Masking-Based Intervention Results

  • Masking Standing Committee routing weights on DeepSeek-V2-Lite sharply reduced correct answers and increased No Answer responses, with correct responses nearly collapsing in deeper layers.Table 13 gives the full results across evaluated layers.Source: F Full Masking-Based Intervention… · p. 17, Table 13 · p. 17
Loading 2601.03425v2…