Cross-Lingual Alignment for Decoder-Only Models using MoE Routers

Background

• Encoder-decoder models enabled contrastive learning on translated sentence pairs by pulling together sequence embeddings and separating unrelated pairs.The passage describes this as standard practice for multilingual encoder training.Source: §1 · p. 2

Motivation

• Decoder-only LLMs lack a natural sequence-level representation to align, while tokenization and linguistic differences make matching tokens across translations difficult.The authors note that common hidden-state pooling approaches do not reliably represent whole-sequence meaning.Source: §1 · p. 2

Contribution

• The proposed continual-pretraining loss aligns mean-pooled middle-layer router weights on parallel data; experiments report higher routing agreement, hidden-state similarity, and downstream multilingual performance.The experiments cover four MoE models and seven languages.Source: §1 · p. 2

2.1 CROSS-LINGUAL REPRESENTATIONAL ALIGNMENT IN ENCODER MODELS

• Parallel data has supported cross-lingual contrastive learning in encoder models, where aligned sequence representations can facilitate multilingual generalization.The passage traces this approach from word embeddings and encoder-decoder models to current multilingual embedding models.Source: §2.1 · pp. 2–3

2.2 MULTILINGUAL REPRESENTATIONS IN LLMS

• LLMs use shared representations particularly in middle layers, and prior work explores alignment through activation patching, steering, parallel-data exposure, and hidden-state losses.Stronger alignment may come at a cost to language- and culture-specific information.Source: §2.2 · p. 3

2.3 CENTRAL HYPOTHESES

• The authors hypothesize that cross-lingual alignment improves transfer and that mean-pooled router weights provide meaningful sequence-level targets because they summarize expert use across a sequence.They contrast these distributions with mean-pooled hidden states, whose token features may attenuate or cancel when averaged.Source: §2.3 · p. 3

3.1 MIDDLE LAYERS SELECTION

• Cross-lingual routing-divergence curves identify layers with expert sharing, guiding a fixed middle-layer range without exhaustive subset search.The cited prior work indicates that some layers benefit from alignment while others can be harmed by enforcing it.Source: §3.1 · p. 3

3.2 LOSS FORMULATION

• The routing loss compares token-mean routing distributions across parallel sequences at selected layers and is combined with the language-modeling loss as L = LLM + α · LXLR.Only the target-language sequence receives gradient updates; the English sequence supplies routing distributions for LXLR.Source: §3.2 · pp. 3–4

3.3 PACKING OPTIMIZATIONS

• A shared packed pass processes source and target sequences without padding, preserves sequence boundaries, and stops source-token processing after the last alignment layer.These optimizations eliminate padding and redundant computation, with only small computational overhead relative to baseline single-language CPT.Source: §3.3 · p. 4, Figure 2 · p. 4

4.1 MODELS, LANGUAGES, AND TRAINING DATA

• Four MoE models are evaluated across seven target languages, with smaller and larger models assigned different language sets.The smaller models use Vietnamese, Sinhala, and Hungarian; the larger models use Telugu, Kannada, Thai, and Kyrgyz.Source: §4.1 · p. 4
• Each language has a curated 200k-sample parallel training set, supplemented with OPUS translations when curated examples are scarce.Indic languages account for three of the seven target languages, reflecting greater availability of parallel SFT datasets.Source: §4.1 · p. 5

4.2 EXPERIMENTAL COMPARISONS

• Comparisons include standard CPT, router-only training with other parameters frozen, and a mean-pooled hidden-state alignment variant tested on Marco-Nano in Hungarian and Sinhala.The hidden-state variant uses cosine similarity and the same layers as the router objective.Source: §4.2 · p. 5

4.3 HYPERPARAMETER SELECTION

• For each model-language pair, learning rate and auxiliary-loss strength are tuned while other hyperparameters remain fixed, using language-model validation loss for selection.The auxiliary-loss scale is initialized to make its magnitude approximately match the language-modeling loss.Source: §4.3 · p. 5

4.4 EVALUATION

• Evaluation uses generation, comprehension, knowledge-transfer, local-knowledge, physical-reasoning, and mathematical-reasoning benchmarks, with four to nine tasks available per target language.The suite includes FLORES, Belebele, Global-MMLU, MMLU-ProX, MultiLoKo, INCLUDE, Global-PIQA, MGSM, and PolyMath.Source: §4.4 · p. 5

5.1 IMPACT ON ROUTING AND REPRESENTATIONS

• Across models and languages, routing-loss training produces the largest increase in routing alignment, while baseline continued pre-training makes only minor changes.The comparison uses routing divergence from English; Figure 3 shows selected model–language examples and marks the layers where LXLR is applied.Source: Figure 3 · p. 6, §5.1 · p. 6
• For Qwen3, the auxiliary loss makes hidden-state representations more similar to English across the evaluated languages.Figure 4 plots layer-wise similarity for Telugu, Kyrgyz, Kannada, and Thai.Source: §5.1 · p. 7, Figure 4 · p. 7

5.2 IMPACT ON DOWNSTREAM TASKS

• Router-only training updates fewer than 0.1% of parameters and sometimes matches or exceeds the LM-only baseline, but performance varies substantially and it is not a reliable full-model substitute.The internal alignment metrics are also noisy in this training condition.Source: §5.2 · pp. 7–8
• 13 of 14 model–language pairs improve over baseline, with one tie, a mean gain of 0.9 points, and gains up to 2.1 points.The averages cover available tasks for each model–language pair.Source: §5.2 · p. 7, Table 1 · p. 8
• On Sinhala, LXLH raises baseline 26.1 to 26.3, below LXLR’s 26.8; on Hungarian, LXLH lowers 37.4 to 36.9 while LXLR reaches 38.1.This comparison covers Marco-Nano and two languages, so the authors describe it as targeted support rather than a comprehensive comparison of representation-alignment methods.Source: §5.2 · p. 7, Table 2 · p. 8
• The gains appear to arise primarily from changes to the representations supplied to routers, rather than updates to the routing function alone.Source: §5.2 · p. 8

Experimental Limitations

• The experiments use fully post-trained models; sensitive, inefficient continual pre-training may limit gains, and this setting may trade larger deltas for control over routing-loss effects.The authors chose post-trained models to avoid noisy evaluations with base models.Source: §6 · p. 8

Real-Life Implementation

• The authors expect the method to be more effective during pretraining, where learning across languages could be tied together more natively.They note that routing dynamics are established early in pretraining (Muennighoff et al., 2025).Source: §6 · pp. 8–9

Dependence on Parallel Data

• The method relies entirely on parallel data, which is predominantly sentence-level and narrow in distribution; the authors position MoE alignment as one part of a broader curriculum.Their baseline improved only a small amount after 200k continual-pretraining samples.Source: §6 · p. 9

No Negatives

• The experiments omit negative pairs and do not test their interaction with load balancing, which the authors leave for potentially expensive large-scale pretraining.They argue load balancing already encourages equitable expert use and that small-scale alignment has low collapse risk.Source: §6 · p. 9

Transfer-Dependent Tasks

• The results do not convincingly establish that alignment benefits reasoning, math, and knowledge tasks more than translation and reading comprehension.The proposed task-dependence hypothesis is based on prior work (Zhu et al., 2024; Yoon et al., 2024; Hu et al., 2025).Source: §6 · p. 9

Direct Early-Layer Gradients

• The authors describe routing divergence loss and load-balancing losses as more direct supervision to early-layer parameters that can help mitigate gradient instability.They contrast these auxiliary losses with the gradient bottleneck from the language-model loss (Godey & Artzi, 2026; Lee et al., 2015; Kedia et al., 2024; Li et al., 2025).Source: §6 · p. 9

Cross-Modal Representation Learning

• Future work could test whether MoE alignment facilitates cross-modal alignment and improves reasoning across modalities.Prior work cited by the authors identifies modality-dependent routing discrepancies that can hinder activation of task-relevant reasoning experts (Xu et al., 2026).Source: §6 · p. 9

7 CONCLUSION

• The proposed objective minimizes divergence between mean-pooled routing distributions for translated sequences, and its effects extend to hidden representations and aggregate multilingual evaluation gains.The authors characterize the evidence as support for cross-lingual alignment's role in multilingual transfer, even in a limited-scale post-training setting.Source: §7 · p. 9
• Future work should examine token-level information loss, where alignment supervision works best in the training curriculum, and how to reduce reliance on parallel data.The authors identify pretraining-scale study as a way to investigate shared representations without sacrificing language- or modality-specific structure.Source: §7 · p. 9

CODE AVAILABILITY

• Code for the work is available on GitHub.The paper gives the repository as github.com/lucasbandarkar/xl_moe_alignment.Source: CODE AVAILABILITY · p. 10

• Figure 5 gives an overview of the cross-lingual alignment method alongside the formal description in Section 3.2.Source: Figure 5 · p. 19

B COMPLETE EXPERIMENTAL RESULTS

• Tables 3–6 distinguish target-language LM-only Baseline, + routing, Router-only, and + hidden conditions; AVG averages available benchmarks and LM loss excludes auxiliary losses.Original refers to the released post-trained checkpoint before training; percentage signs are omitted, and FLORES retains its original scale.Source: B COMPLETE EXPERIMENTAL RESULTS · p. 19
• The Qwen3-30B-A3B, GPT-OSS-20B, Granite-4.0-H-Tiny, and Marco-Nano tables report complete results, with auxiliary alignment applied to model-specific layer ranges.The captions specify layers 7–34, 4–17, 10–35, and 7–19, respectively.Source: Table 3 · p. 20, Table 4 · p. 21, Table 5 · p. 22

C VISUALIZATIONS OF MOE ROUTING FOR LAYER SELECTION

• Routing-divergence curves guide layer selection; Figure 6 shows divergence for the four models, with lower values indicating higher alignment and corresponding to higher-resource languages.Qwen3-30B-A3B and GPT-OSS-20B use previously selected layers, while layers for Granite-4-H-Tiny and Marco-Nano were chosen subjectively from the visualizations.Source: C VISUALIZATIONS OF MOE ROUTING FOR… · p. 23, Figure 6 · p. 24

D MODEL MOE DETAILS

• Table 7 summarizes MoE architecture and router-layer details for each model, with active parameter counts shown in parentheses.Source: Table 7 · p. 24

E PARALLEL TRAINING DATA

• Each of seven target-language datasets targets 200,000 training examples and reserves 1,000 for validation, combining sentence-aligned bitext with parallel instruction–response conversations.Instruction examples retain their multi-turn chat structure; empty, identical, and within-source duplicate pairs are removed.Source: E PARALLEL TRAINING DATA · p. 24
• Table 8 records source inclusion rather than mixture ratios; requested source allocations may differ from realized counts because shortfalls are refilled from available sources.Some sources lack enough examples that pass filtering.Source: E PARALLEL TRAINING DATA · p. 24, Table 8 · p. 25
• Examples are tokenized on both sides and discarded if they exceed sequence-length limits, with stricter caps for Sinhala, Kannada, and Telugu.The default is 768 tokens per side; caps are 512 for Sinhala and 640 for Kannada and Telugu.Source: E PARALLEL TRAINING DATA · p. 25

F TRAINING DETAILS

• Each run uses one epoch over 200,000 examples and a warmup–stable–decay schedule: 10,000-example warmup, 150,000-example constant-rate interval, and 40,000-example decay.All remaining training settings are fixed across experiments.Source: F TRAINING DETAILS · p. 25
• Runs use fused PyTorch AdamW without NEFTune, with gradient accumulation adjusted to maintain an effective batch size of 32 examples.Per-device batch size varies with each model’s memory requirements.Source: F TRAINING DETAILS · p. 25
• GPT-OSS-20B and Qwen3-30B-A3B require multi-GPU FSDP, while Granite-4.0-H-Tiny and Marco-Nano fit on one GPU.Single-GPU runs for Granite and Marco-Nano take approximately 15 hours on H100 or A100.Source: F TRAINING DETAILS · p. 25