Source-linked AI summary
Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering
Yuxin Chen, Zhengzhou Cai, Xiangtian Ji, Weixiang Zhao, An Zhang, Xiang Wang, Tat-Seng Chua
TL;DR
Multilingual MoE models have strong capabilities, but their internal routing mechanisms and cross-language differences remain insufficiently understood. This paper systematically analyzes routing, expert specialization, and layerwise roles, then proposes inference-time steering through middle-layer shared experts. The analysis finds structured language- and depth-dependent routing, and steering yields consistent multilingual improvements, especially for linguistically related languages.
Problem
Multilingual MoE models show performance disparities across languages, while mechanistic understanding of their routing and expert specialization remains limited compared with dense models.
Method
The paper analyzes routing behavior, expert specialization, and layerwise functional roles, then applies routing-guided inference-time steering toward dominant-language shared experts in middle layers.
Results
Routing aligns with language families and network depth, early and late layers support language-specific processing, middle layers act as language-agnostic capacity hubs, and steering consistently improves multilingual performance.
Takeaways & Limitations
Aligning target languages with dominant languages from the same family through middle-layer experts supports cross-lingual capacity transfer without changing early and late routing.
Takeaways & Limitations
The study uses one representative large-scale MoE model and restricts expert-level steering to inference-time interventions.
Abstract
from arXiv · showhide
Mixture-of-Experts (MoE) architectures have shown strong multilingual capabilities, yet the internal mechanisms underlying performance gains and cross-language differences remain insufficiently understood. In this work, we conduct a systematic analysis of MoE models, examining routing behavior and expert specialization across languages and network depth. Our analysis reveals that multilingual processing in MoE models is highly structured: routing aligns with linguistic families, expert utilization follows a clear layerwise pattern, and high-resource languages rely on shared experts while low-resource languages depend more on language-exclusive experts despite weaker performance. Layerwise interventions further show that early and late MoE layers support language-specific processing, whereas middle layers serve as language-agnostic capacity hubs. Building on these insights, we propose a routing-guided steering method that adaptively guides routing behavior in middle layers toward shared experts associated with dominant languages at inference time, leading to consistent multilingual performance improvements, particularly for linguistically related language pairs. Our code is available at https://github.com/conctsai/Multilingualism-in-Mixture-of-Experts-LLMs.
1 Introduction
MoE models offer strong multilingual capabilities, but performance disparities and limited mechanistic understanding motivate analysis of routing and expert specialization. This work finds structured language-, layer-, and resource-dependent routing and uses those findings to steer inference-time routing for improved multilingual performance.
- MoE models scale parameters through sparse expert activation while maintaining manageable compute costs.
- Persistent performance disparities across language resource levels motivate understanding how multilingual capacity is utilized inside MoE models.
- Prior mechanistic analyses largely study dense models, leaving multilingual routing and expert specialization in MoE architectures insufficiently understood.
- Languages in the same linguistic family tend to share routing distributions, whereas distant languages use more distinct expert subsets.
- Middle layers favor shared experts, while early and late layers show stronger language-specific routing and greater use of language-exclusive experts.
- Routing-guided steering toward dominant-language shared experts in middle layers consistently improves multilingual performance, especially for linguistically related languages.
2 Preliminary
MoE layers combine expert networks with a router that dynamically selects a sparse subset of experts for each token. Selected experts are weighted by routing probabilities to produce the layer output.
- Each MoE layer contains E expert networks and a gating network that routes input tokens to selected experts.
- For token x, the router projects its layer representation through a learnable matrix to obtain expert routing logits.
- Top-K routing selects the K experts with highest logits and computes their expert probabilities with a softmax.
- The MoE layer output is a linearly weighted sum of activations from the selected experts.
3 Analysis Setup
The study analyzes Qwen3-30B-A3B using the BELEBELE multilingual understanding benchmark and ten languages spanning diverse families and resource levels.
- The analysis uses Qwen3-30B-A3B as a representative advanced MoE language model.
- BELEBELE provides the multilingual natural language understanding benchmark with broad language coverage and minimal domain variation.
- The study examines ten languages spanning diverse linguistic families and resource levels, including Arabic, Bengali, German, English, Spanish, French, Japanese, Korean, Swahili, and Chinese.
- English and Chinese are treated as dominant languages; Swahili and Bengali represent low-resource languages, while six others are categorized as high-resource.
4 Analysis of Routing Behavior
The analysis measures how frequently experts are selected for each language and compares language routing distributions across layers. Routing similarity follows linguistic relationships and peaks in middle layers, revealing structured depth-wise specialization.
- Routing Metrics: Routing behavior is characterized by expert selection frequencies for tokens from each language at each layer.
- Routing Metrics: Routing distributions summarize how expert selections are allocated across experts for each language and layer.
- Routing Metrics: Cross-language routing similarity compares language routing distributions, with larger values indicating more similar routing behavior.
- Cross-Language Findings: Languages from the same linguistic family or with similar writing systems show higher routing similarity than linguistically distant languages.
- Layerwise Findings: Cross-language routing similarity is highest in middle layers and decreases toward both early and late layers.
- Layerwise Findings: Higher similarity indicates greater overlap in expert utilization, whereas lower similarity reflects increasingly language-specific routing.
5 Analysis of Expert Specialization
Expert specialization varies systematically across languages and network depth: high-resource languages rely more on shared experts, while language-exclusive experts concentrate in early and late layers and diminish in middle layers.
- 5.1 Metrics: Routing analysis characterizes token distribution across experts, while expert analysis measures how individual experts associate with different languages.Language-related experts are identified from routing frequencies and classified by their cross-language usage.
- 5.1 Metrics: Language-related experts are selected as the Top-K experts with the highest routing frequencies for each language and layer.The selected set contains experts most frequently chosen for that language at that layer.
- 5.1 Metrics: Experts are language-exclusive when one language dominates their routing frequency; otherwise, they are treated as language-shared.A threshold controls how dominant a language must be for exclusivity, while shared experts lack single-language dominance.
- 5.2 Findings: High-resource languages rely on few language-exclusive experts, whereas low-resource languages rely more heavily on them despite weaker performance.Dominant languages are associated with the largest number of language-exclusive experts, while high-resource languages often remain competitive through shared experts.
- 5.2 Findings: Language-exclusive expert usage follows a U-shaped depth pattern: it is higher in early layers, decreases in middle layers, and rises again toward late layers.This pattern corresponds to language-specific specialization at the network edges and language-agnostic processing in middle layers.
6 Layerwise Intervention Study
Layerwise masking reveals distinct functional roles across MoE depth: early exclusive experts support multilingual understanding, late exclusive experts control output-language consistency, and middle layers are comparatively language-agnostic.
- 6.1 Intervention Method: The intervention masks language-exclusive experts at selected depths while leaving other experts unchanged, using a large negative logit to prevent their selection.The study applies this intervention separately to early, middle, and late layers.
- 6.1 Intervention Method: The evaluation measures reasoning and understanding with MGSM and XQuAD, and language consistency with MGSM and FLORES-200.FLORES-200 explicitly conditions generation on a specified target language.
- 6.2 Intervention Results: Masking target-language-exclusive experts in early or late layers causes accuracy drops, while masking experts exclusive to other languages has little effect on target-language performance.These results support a language-specific association for exclusive experts and a language-agnostic role for middle-layer shared experts.
- 6.2 Intervention Results: Early-layer masking causes query-understanding failures and substantial XQuAD degradation, showing that early exclusive experts support multilingual understanding.The model often misinterprets problem conditions, producing erroneous reasoning trajectories.
- 6.2 Intervention Results: Late-layer masking can preserve or mildly improve task accuracy for some languages but severely disrupts language consistency and causes language mixing.The degradation persists even when the task remains solvable, separating output-language control from reasoning competence.
7 Routing Steering for Multilingual Enhancement
The paper steers inference-time routing in middle layers toward shared experts associated with dominant languages. This improves multilingual performance, especially when the steering source and target language are linguistically related.
- Routing-guided method: Routing-guided steering adaptively biases middle-layer routers toward shared experts associated with dominant languages.Early and late layer routing remains unchanged to preserve language-specific processing.
- Steering results: 1.9% average improvement is achieved using English as the steering source across high- and low-resource target languages.Chinese steering produces a smaller but consistent 0.7% average gain.
- Steering results: 0.7% average gain is achieved using Chinese as the steering source.The improvement is described as more modest but consistent than English-based steering.
- Steering results: Steering effectiveness depends strongly on linguistic proximity between source and target languages.English benefits Latin-script languages such as French, Spanish, and Swahili, whereas Chinese steering benefits Japanese.
- Steering results: Applying steering to early or late layers consistently degrades performance by disrupting language-specific understanding and generation.The method therefore targets middle layers while leaving early and late layers unchanged.
8 Conclusion
The paper concludes that multilingual MoE behavior arises from structured routing, expert specialization, and layerwise functional roles. These findings motivate middle-layer steering toward dominant-language experts to improve multilingual performance.
- Conclusion: Routing patterns align with language-family structure, while expert utilization differs systematically across languages with different resource levels.The conclusion frames multilingual capability as emerging from structured expert specialization and sharing.
- Conclusion: Middle-layer routing toward dominant-language experts produces consistent multilingual performance improvements.The proposed approach aligns target languages with dominant languages within the same language family through middle-layer experts.
Limitations
The study’s scope is limited by its evaluation setting, inference-only interventions, and supplementary routing-entropy analysis. Its entropy measure does not distinguish shared from language-exclusive experts, so the main analysis relies on specialization and routing similarity.
- Study scope: The experiments use a representative large-scale MoE model, leaving generality and scaling to substantially larger systems for future study.The authors plan to extend the analysis as larger-scale MoE models become accessible.
- Intervention scope: The steering investigation is restricted to inference-time interventions rather than training-time or post-training optimization.The authors identify training-time expert shaping as a complementary direction for future work.
- Evaluation scope: The evaluation uses selected language sets and benchmarks, with the exact evaluated languages varying across benchmarks.The study analyzes languages from BELEBELE and evaluates steering on PolyMath using dominant-language sources.
- Routing-entropy caveat: Routing entropy indicates how broadly routing mass is distributed but cannot distinguish language-exclusive from shared experts.The authors therefore treat entropy as supplementary and prioritize expert specialization and cross-language routing similarity.
- Routing-entropy findings: Dominant, high-resource, and low-resource languages occupy distinct routing-entropy regimes.Dominant languages have the highest entropy, high-resource languages moderately high entropy, and low-resource languages more concentrated expert usage.
- Routing-entropy findings: Routing entropy is relatively stable in early and middle layers but decreases toward the network top.The decline is consistent with increasingly concentrated expert routing in later layers.
C.2 Expert Specialization
Expert specialization is strongly structured across both languages and depth: related languages share routing pathways, while early and late layers concentrate language-specific experts and middle layers minimize specialization. Low-resource languages use more exclusive experts, indicating weaker integration into shared capacity.
- Layer-wise Distribution: Almost all languages exhibit a U-shaped specialization pattern, with exclusive experts concentrated in early and late layers and minimal in middle layers.This fine-grained pattern matches the group-level findings.
- Layer-wise Routing Divergence: Related language pairs activate similar expert pathways, whereas distant language pairs show substantially higher routing divergence in early and late layers.Randomly selected pairs follow the same layerwise comparison framework.
- Total Exclusive Experts: Low-resource languages possess the highest number of language-exclusive experts, while high-resource languages more effectively leverage shared capacity.The comparison aggregates exclusive experts across all layers by language resource level.
- Layer-wise Intervention: Masking early or late exclusive experts disrupts language understanding or consistency, whereas middle-layer interventions have minimal performance impact.The intervention results distinguish functional roles across network depth.
D.1 Hyperparameter Study
The steering coefficient controls the strength of routing-logit perturbations, and its effectiveness depends on applying steering at the appropriate layers and tuning its magnitude. Performance improves in a narrow intermediate range but degrades when steering is too strong or applied to early or late layers.
- Coefficient Definition: The steering coefficient λ controls the magnitude of the perturbation added to MoE routing logits and biases expert selection toward dominant-language experts.It determines the strength of inference-time routing steering.
- Layer Placement: Steering at early or late layers consistently degrades performance by disrupting language-specific capabilities needed for input understanding or output generation.The ablation motivates restricting steering to middle layers.
- Coefficient Sensitivity: For λ < 0.01, steering produces only marginal influence and stochastic fluctuations around baseline performance.The perturbation is insufficient to overcome the original router’s confidence.
- Coefficient Sensitivity: For 0.01 < λ ≤0.022, cumulative accuracy increases stably and monotonically, peaking at approximately λ ≈0.022.This range balances transferable routing guidance with preservation of necessary model functions.
- Coefficient Sensitivity: When λ > 0.026, performance degrades precipitously as excessive steering diverges from pretrained routing priors.Over-correction disrupts the balance of expert specialization established during pre-training.
E.2 Robustness of Language Capability
Language capability is unevenly robust across resource levels: low-resource languages depend heavily on fragile exclusive experts, whereas high-resource and dominant languages benefit more from shared computation. This robustness gap helps explain persistent multilingual performance disparities.
- Resource-Level Robustness: Low-resource languages show the most severe degradation when language-exclusive experts are masked, especially in early and late layers.In several cases, accuracy collapses almost entirely, indicating dependence on a small specialized expert set.
- Resource-Level Robustness: High-resource languages are more robust because stronger integration into the shared expert space enables compensation through shared computation paths.Middle-layer interventions have almost no negative effect for these languages.
- Dominant-Language Robustness: Dominant languages, especially English, retain limited performance degradation and largely preserve language consistency when their exclusive experts are masked.Their robustness aligns with strong connectivity to shared experts across layers.
- Implications: The resulting robustness gap offers a mechanistic explanation for persistent multilingual performance disparities.The paper attributes this gap to differing reliance on fragile exclusive pathways versus shared experts.