PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilingual Routing in Mixture-of-Experts

Lucas Bandarkar, Chenyuan Yang, Mohsen Fayyaz, Junlin Hu, Nanyun Peng

arXiv:2510.04694v2cs.CLcs.AIcs.LG

TL;DR

MoE LLMs offer scalable computation, but their sparse routing in multilingual settings remains poorly understood. This paper analyzes multilingual expert routing and tests inference-time steering toward English task experts, finding consistent multilingual gains under specific middle-layer conditions.

  • Problem

    Sparse routing in MoE LLMs is poorly understood for multilingual data, although multilingual performance gaps remain beyond a select group of languages and prior interpretability work focused on dense models.

  • Method

    The paper analyzes routing across parallel multilingual datasets and uses inference-time interventions to steer MoE routers toward or away from specialized experts.

  • Results

    Consistently across three models and two evaluation tasks, steering middle-layer routers toward English task experts improves multilingual performance by 1-2 accuracy points.

  • Takeaways & Limitations

    The findings identify middle-layer language-universal experts and improved expert sharing as important to multilingual generalization in MoE LLMs.

  • Takeaways & Limitations

    Intervening in heavily trained MoE routers is difficult: conditions outside the effective middle-layer range can degrade performance, and many intervention conditions do not work.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) architectures have become the key to scaling modern LLMs, yet little is understood about how their sparse routing dynamics respond to multilingual data. In this work, we analyze expert routing patterns using parallel multilingual datasets and present highly interpretable layer-wise phenomena. We find that MoE models route tokens in language-specific ways in the early and late decoder layers but exhibit significant cross-lingual routing alignment in middle layers, mirroring parameter-sharing trends observed in dense LLMs. In particular, we reveal a clear, strong correlation between a model's performance in a given language and how similarly its tokens are routed to English in these layers. Extending beyond correlation, we explore inference-time interventions that induce higher cross-lingual routing alignment. We introduce a method that steers the router by promoting middle-layer task experts frequently activated in English, and it successfully increases multilingual performance. These 1-2% gains are remarkably consistent across two evaluation tasks, three models, and 15+ languages, especially given that these simple interventions override routers of extensively trained, state-of-the-art LLMs. In comparison, interventions outside of the middle layers or targeting multilingual-specialized experts only yield performance degradation. Altogether, we present numerous findings that explain how MoEs process non-English text and demonstrate that generalization is limited by the model's ability to leverage language-universal experts in all languages.

1 INTRODUCTION

The paper examines how multilingual behavior emerges in sparse MoE LLMs, where prior multilingual interpretability work focused mainly on dense models. It analyzes routing patterns and tests whether steering expert activation can improve multilingual performance.

  • 1 INTRODUCTION: MoE routing is studied against a backdrop of persistent performance gaps beyond a small set of languages and limited understanding of sparse multilingual representations.Earlier work identified shared feature spaces in dense models’ middle layers, but their relevance to MoE architectures remained unexplored.
  • 1 INTRODUCTION: The study investigates multilingual routing in four MoE LLMs and finds language-agnostic intermediate-layer parameters alongside language-specific behavior elsewhere.The analysis covers QWEN3-30B-A3B, PHI-3.5-MOE, GPT-OSS-20B, and OLMOE.
  • 1 INTRODUCTION: Language performance strongly correlates with cross-lingual routing alignment to English, especially in intermediate layers.The paper presents this relationship as a central finding of its routing analysis.
  • 1 INTRODUCTION: Inference-time steering of routers toward English task experts improves multilingual performance on MGSM and medicine-domain GLOBAL-MMLU.The interventions are evaluated across QWEN3, PHI-3.5-MOE, and GPT-OSS and produce statistically significant improvements on both tasks.
  • 1 INTRODUCTION: The intervention results support improved expert sharing as a route to generalizing complex capabilities across languages.The paper presents these findings as motivation for methods that promote cross-lingual expert sharing during training.

2 RELATED WORK ON MULTILINGUAL LLMS

Prior multilingual research pursued shared representations and cross-lingual alignment, while newer work connects middle-layer alignment and English-pivot strategies to multilingual transfer. MoE research has also explored multilingual translation and scaling, but the supplied figure illustrates routing alignment across languages and layers.

  • 2 RELATED WORK ON MULTILINGUAL LLMS: Prior multilingual models used cross-lingual embedding alignment to unify feature spaces and facilitate multilingual generalization.This line of work was associated particularly with smaller encoder-decoder models before decoder-only LLM scaling.
  • 2 RELATED WORK ON MULTILINGUAL LLMS: Figure 2 plots routing divergence from English across model layers, coloring each language by BELEBELE accuracy to show their relationship.The visualization is based on Qwen3-30B-A3B and extends to more languages than the main discussion labels.
  • 2 RELATED WORK ON MULTILINGUAL LLMS: Recent studies associate stronger middle-layer alignment and language-shared representations with improved multilingual performance.The cited work includes explicit steering toward shared representations, away from language-specific ones, and prompting English as a pivot.
  • 2 RELATED WORK ON MULTILINGUAL LLMS: MoE research has applied modularity to massively multilingual machine translation and used MoE upcycling to scale multilinguality in LLM final layers.The supplied related work distinguishes these directions from the paper’s routing analysis.

3 MIXTURE-OF-EXPERTS PRELIMINARIES

MoE LLMs replace each transformer layer’s MLP with multiple experts and use a router to select a sparse subset for each token. The selected expert outputs are then aggregated, typically by weighted summation.

  • 3 MIXTURE-OF-EXPERTS PRELIMINARIES: MoE layers replace a standard MLP with E expert MLPs and route each token to only the top-K experts.A router, also called a gating network, computes logits to determine the selected experts.
  • 3 MIXTURE-OF-EXPERTS PRELIMINARIES: The selected experts’ hidden states are aggregated, typically through a weighted sum.

4 INTERPRETABILITY ANALYSIS

The analysis measures how multilingual inputs are routed across MoE layers and finds language-specific routing at the boundaries but stronger cross-lingual alignment in intermediate layers. Routing alignment with English is strongly associated with language performance, while routing entropy and within-sequence consistency reveal additional layer-wise differences.

  • Method: The analysis averages token-level routing weights into sequence-level expert distributions and compares paired languages using entropy-normalized Jensen-Shannon divergence.FLORES parallel data provides the multilingual routing comparison, while BELEBELE evaluates understanding on the same passages.
  • Routing divergence: Routing divergence from English follows a U-shaped layer profile, with substantially higher divergence in the first and last layers than in intermediate layers.This pattern is common across the evaluated models, though OLMOE and PHI-3.5-MOE show notable deviations.
  • Routing divergence: Intermediate-layer routing alignment with English strongly correlates with BELEBELE language performance across models and languages.For OLMOE, the correlation coefficient r ranges from -0.95 to -0.80 across middle layers, while GPT-OSS shows weaker correlations from -0.40 to -0.60.
  • Routing divergence: Language similarity also predicts routing similarity, but language-family relationships explain only a small part of the broader routing-alignment trends.Related language pairs have lower routing divergence than unrelated pairs, while comparisons centered on other high-resource languages yield similar patterns with slight attenuation.
  • Routing entropy and consistency: Routing entropy decreases across layers for every language, with a much steeper decline for non-English languages.The authors suggest that increasingly refined representations or more diverse experts may explain the general decline, while final-layer specialization may contribute to the non-English drop.
  • Routing entropy and consistency: Non-English tokens generally show higher within-sequence routing consistency than English tokens, especially in the final layers.The metric averages Jaccard similarity over randomly sampled token-pair expert activations; PHI-3.5-MOE is shown as an example, with similar patterns reported elsewhere.

5 INTERVENTION METHODOLOGY

The intervention methodology identifies task- and language-specialized experts from activation differences, then steers router logits before softmax in selected layers. It focuses on middle-layer task experts while accounting for expert specialization, intervention type, and router sensitivity.

  • 5.2 EXPERT IDENTIFICATION: Experts are selected when their activation-frequency difference from the baseline exceeds the positive threshold τ.The method uses discrete top-k activation counts and relative activation frequencies to identify experts specialized for a domain or language.
  • 5.2 EXPERT IDENTIFICATION: Task and multilingual experts do not overlap at τ ≥ 0.3 across four models, revealing separate language- and task-specialized expert sets.Multilingual experts occur in all layers but are less prevalent and less strongly specialized in middle layers.
  • 5.3 INTERVENTION METHOD: Router interventions alter logits before softmax, changing both sparse expert selection and aggregation weights.Soft interventions add or subtract a standard-deviation-scaled value, while hard interventions force or prevent selection by setting extreme logits.
  • 5.4 INTERVENTION EXPLORATION: Middle layers are targeted because prior layer-wise analysis links them to strong alignment–performance relationships, reducing the intervention search space.The methodology varies layer, expert type, threshold, direction, and intervention strength, but prioritizes middle-layer regions.
  • 5.4 INTERVENTION EXPLORATION: Random or suboptimal router interventions generally degrade performance, underscoring the sensitivity of heavily trained MoE routers.Even hard interventions force only one or two experts into top-K selection, functioning as targeted nudges rather than complete routing homogenization.

6 INTERVENTION RESULTS

Interventions that promote English task-expert activation in precise middle layers consistently improve multilingual performance, while broader or misplaced interventions can harm it.

  • 1–2 accuracy points from English task-expert steering improve multilingual performance significantly across 3 models and 2 tasks.The gains span diverse languages, are somewhat larger for lower-resource languages, and are less pronounced for medicine than math.
  • Target-layer choice matters more than tunable intervention hyperparameters, with slightly displaced layers causing significant degradation.The successful region is restricted to language-universal middle layers.
  • Activating multilingual experts in other layers can nullify gains, whereas targeting a small number of task-specific middle-layer experts preserves consistent improvements.
  • The intervention strategy is difficult to apply safely because many baselines and conditions fail to improve performance.

7 CONCLUSION AND FUTURE WORK

The paper concludes that MoE multilingual specialization is concentrated in early and late layers, while middle-layer experts support language-universal generalization. Its intervention results suggest that improving cross-lingual expert sharing can strengthen multilingual transfer.

  • Early and late layers contain the strongest multilingual specialization, while middle-layer experts act as language-universal mechanisms for generalization.The strongest non-English specialization appears in the last layers, which show lower entropy and higher token-to-token consistency.
  • Steering routers toward English activation patterns consistently improves multilingual performance and supports a relationship between routing alignment and cross-lingual transfer.
  • The intervention implementation and evaluation details are documented across the main methodology sections and referenced appendices.

A.1 MODEL DETAILS

Appendix A.1 provides a table detailing the MoE language models discussed in the paper.

  • Table A.1 lists details of the MoE LLMs discussed.

A.2 ROUTING DIVERGENCE PLOTS FOR ALL MODELS

The appendix presents layer-wise routing-divergence plots across 12 non-English languages for OLMOE, GPT-OSS, and PHI-3.5-MOE, revealing model-specific patterns.

  • Language-code mappings for the plotted languages are provided separately.
  • The appendix includes mean entropy-normalized JS divergence by layer for 12 non-English languages across OLMOE and GPT-OSS.
  • OLMOE’s divergence plot retains a U-shape for French and Chinese despite its poor multilingual capabilities.
  • PHI-3.5-MOE lacks the typical U-shape because its first layers show very low divergence across languages.A small subset of experts is called for all languages in layer 1, requiring further investigation.

A.3 LANGUAGE CODE INDEX

This section documents language-code conventions and methodological details for comparing expert-routing distributions across layers and languages.

  • FLORES and BELEBELE use FLORES language codes, whereas MGSM and GLOBAL-MMLU use two-letter codes.
  • Routing divergence is normalized to account for differing distribution entropy and expert-set size.The normalization factor uses the number of experts and average entropy.
  • Appendix figures report per-layer entropy for QWEN3, PHI-3.5-MOE, OLMOE, and GPT-OSS.
  • Random perturbation preserves top-k routing when force-activating experts would otherwise create tied maximum scores.
  • Specialized experts are selected using a mean-zero activation-frequency difference and threshold τ.Positive values indicate greater activation in Swahili than English in the example.
Loading 2510.04694v2…