PaperlayerView exact revision on arXiv

Source-linked AI summary

Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs

Bo Li, Tianyu Dong, Shaolin Zhu, Deyi Xiong

arXiv:2605.24681v1cs.CLcs.AI

TL;DR

LLM-based multilingual MT must improve translation without losing pretrained monolingual capabilities because parallel-data fine-tuning risks parameter interference. Mix-MoE separates LM and MT Experts across monolingual and bilingual post-pretraining, with FFT-enhanced routing; experiments report broad improvements over baselines and better cross-lingual transfer.

  • Problem

    Fine-tuning LLMs on parallel corpora risks parameter interference, motivating methods that improve multilingual translation while preserving pretrained monolingual knowledge.

  • Method

    Mix-MoE uses two-stage MoE post-pretraining with specialized LM Experts for monolingual knowledge, MT Experts for bilingual translation, and FFT-enhanced routing.

  • Results

    Mix-MoE consistently achieves the highest scores across all metrics and language pairs in Table II, including BLEU 31.8 versus 20.8 for Two-Stage Full Fine-tuning on DE-EN.

  • Takeaways & Limitations

    Fine-tuning Mix-MoE on a single language direction generally improves performance on unseen directions, indicating positive transfer to more generalized translation capabilities.

Abstract

from arXiv · show

Large Language Models (LLMs) have shown great promise in multilingual machine translation (MT), even with limited bilingual supervision. However, fine-tuning LLMs with parallel corpora presents major challenges, namely parameter interference. To address these issues, we propose Mix-MoE, a mixed Mixture-of-Experts framework designed to train LLMs for multilingual MT. Our framework operates in two distinct stages: (1) post-pretraining with MoE on monolingual corpora, and (2) post-pretraining with MoE on parallel corpora. Crucially, we divide the MoE layers into two specialized groups: Language Model Experts (LM Experts) and Machine Translation Experts (MT Experts). LM Experts are designed to capture and retain the monolingual knowledge learned by the pre-trained LLM. MT Experts, on the other hand, are specifically trained to acquire and store bilingual translation knowledge. Furthermore, to facilitate effective interaction between these specialized experts and leverage potential underlying structural patterns in text, we introduce a routing mechanism enhanced by Fourier Transform features derived from model representations. The experimental results demonstrate that Mix-MoE excels in multilingual MT, significantly outperforming existing baselines and showing notable progress in mitigating parameter interference.

I. INTRODUCTION

Multilingual MT needs extensive parallel data, while adapting LLMs can cause parameter interference that degrades existing monolingual abilities. Mix-MoE separates language-model and translation expertise across two training stages and reports broad gains over baselines.

  • Traditional multilingual MT depends heavily on expensive, time-consuming parallel corpora, especially for low-resource languages.
  • Post-pretraining can cause parameter interference, shifting LLM parameters toward translation and degrading monolingual understanding and generation.
  • Mix-MoE trains LM Experts on monolingual corpora first, then freezes them while training MT Experts on bilingual parallel corpora.
  • FFT-enhanced routing supplements semantic content with frequency-domain patterns indicative of text structure when selecting experts.
  • Experiments on 14 WMT language directions found almost all directions superior to the baseline and reported significant progress in mitigating parameter interference.
  • The paper reports state-of-the-art results across multiple translation tasks and language pairs.

III. METHODOLOGY

Mix-MoE transforms selected FFN layers of a dense LLM into specialized LM and MT experts while keeping attention layers dense. Its two-stage training isolates monolingual and translation knowledge to reduce interference.

  • Model Overview: The model sparsifies FFN layers while leaving Attention layers dense because FFNs are treated as key-value memories for linguistic and factual knowledge.Attention layers are described as handling contextual dependency and information routing.
  • Model Overview: Mix-MoE replaces every n-th FFN layer with an MoE layer containing separate LM Experts and MT Experts.Each group contains n experts, set to n = 4 in the experiments.
  • Model Overview: Each expert applies router, up-projection, element-wise gating, and down-projection operations to the input hidden state.The equations define gi, ui, hi, and hexperti through these transformations.
  • LM Experts: LM Experts are initialized by slicing and copying weights from the original LLM FFN layers through an upcycling strategy.
  • MT Experts: MT Experts are initialized from pretrained LM Experts and trained on parallel data while LM Expert parameters and their router remain frozen.This lets MT Experts use pretrained language knowledge while adapting to translation.

B. Router Module

Mix-MoE augments hidden states with FFT-derived spectral features before routing tokens to specialized experts. Separate routers use the combined semantic and spectral representation for expert selection.

  • FFT Feature Extraction: The router extracts FFT features from the input hidden states to expose potential frequency-domain cues about language and text structure.The FFT is applied along the last hidden-state dimension and the real part is retained.
  • Routing Network: Separate linear routing networks for LM and MT Expert groups convert the augmented representation into expert logits.
  • Feature Concatenation: The extracted spectral features are concatenated with the original hidden states to form an augmented routing representation.The concatenated input supplies both semantic and spectral information.
  • Routing Network: Softmax converts expert logits into routing probabilities, after which top-k routing selects experts with the highest probabilities.
  • Routing Network: With k = 1, the model selects the single most relevant expert from each group for each token.
  • FFT Feature Extraction: The method treats FFT output as an orthogonal feature transformation intended to distinguish global features from local details for routing.

C. Two-Stage Post Pretraining

Mix-MoE uses sequential expert specialization: monolingual training develops LM Experts, then parallel-data training develops MT Experts while preserving the former. This selective training supports knowledge transfer and aims to mitigate parameter interference.

  • Two-stage strategy: The two-stage strategy targets general language understanding first and specialized translation skills second through distinct expert groups and routing networks.Stage 1 trains LM Experts; Stage 2 trains MT Experts while leveraging the pretrained model’s knowledge.
  • Stage 1: LM Expert Training: Stage 1 updates only LM Experts and their routing network on monolingual corpora using language-model cross-entropy plus LM-group load-balancing loss.The load-balancing weight λLB is set to 0.01.
  • Stage 2: MT Expert Training: Stage 2 activates and trains MT Experts with parallel corpora while freezing LM Experts and their router parameters to preserve monolingual knowledge.The translation objective combines translation cross-entropy with MT-group load-balancing loss; the frozen LM router remains active for utilization balance.
  • Knowledge transfer: Sequentially training specialized expert groups facilitates transfer from pretrained knowledge to monolingual understanding and then bilingual translation.The paper identifies this process as important for mitigating parameter interference while supporting language representation and multilingual translation.

IV. EXPERIMENT

The experiments evaluate Mix-MoE using WMT monolingual and translation data, standard WMT test sets, and three metrics covering lexical overlap, semantic similarity, fluency, and accuracy.

  • Model and datasets: The study uses Llama3.2-1B with WMT17–WMT19 Monolingual News Crawl data for seven target languages during first-stage post-training.The monolingual data undergo deduplication and length filtering.
  • Training setup: Table I reports the training hyperparameters used for the two-stage training strategy.
  • Model and datasets: Evaluation uses corresponding WMT task test sets for joint translation to support comparison with prior work and standard benchmarks.
  • Metrics: Performance is evaluated with BLEU, METEOR, and COMET across accuracy, fluency, semantic similarity, and lexical-overlap dimensions.BLEU measures n-gram overlap, METEOR incorporates stemming and synonym matching, and COMET is a reference-based neural metric.

B. Baselines

The experiment compares Mix-MoE with dense models, parameter-efficient fine-tuning, architecture-modification, and MoE baselines. Baseline training protocols were addressed to support fair comparison with the proposed two-stage pipeline.

  • Fine-tuning baselines: Full Fine-tuning updates all dense-model parameters, whereas LoRA updates linear modules with rank 8.
  • Architecture-modification baseline: LLaMA-Pro inserts 12 trainable transformer blocks at fixed intervals while preserving original blocks during post-pretraining.
  • MoE baselines: The MoE baseline trains all MoE parameters in one post-pretraining stage, while MoE-LPR combines MoE with Language Priors Routing.
  • MoE baselines: LLaMA-MoE constructs MoE modules for attention and MLP components, using eight experts and selecting the top two per token.

D. Main Results

Mix-MoE achieves the highest reported scores across the evaluated WMT metrics and language pairs, including against baselines strengthened with the same two-stage training pipeline. Additional evaluation targets domain generalization on FLORES-200 across the same 14 directions.

  • WMT results: Mix-MoE consistently achieves the highest BLEU, COMET, and METEOR scores across all reported language pairs against strengthened baselines.These baselines receive the same monolingual post-pretraining and parallel-data fine-tuning pipeline.
  • WMT results: 31.8 BLEU in DE-EN exceeds 20.8 for the Two-Stage Full Fine-tuning baseline.
  • WMT results: The reported COMET and METEOR improvements indicate gains in semantic accuracy and fluency alongside lexical overlap.
  • FLORES-200 evaluation: FLORES-200 evaluates Mix-MoE across the same 14 language directions on a wider range of topics than the primarily news-domain WMT test sets.The benchmark is used to assess robustness and domain generalization.

E. Ablation Study

The ablations separate the effects of data, model capacity, expert specialization, and FFT-enhanced routing. Mix-MoE’s specialized experts and frequency-domain routing improve translation performance while addressing parameter interference.

  • Data, Capacity, and Specialization: +1.5 BLEU on DE-EN results from adding monolingual Stage 1 data to dense supervised fine-tuning.The gain is limited by potential parameter interference in dense shared weights.
  • Data, Capacity, and Specialization: MoE SFT significantly outperforms Dense SFT because expert layers increase model capacity even without monolingual data.
  • Data, Capacity, and Specialization: 31.8 vs 28.7 on DE-EN shows Mix-MoE outperforming generic two-stage MoE with the same monolingual and bilingual data.Mix-MoE freezes LM Experts during Stage 2 and trains only MT Experts, whereas the generic MoE updates all experts.
  • FFT-Enhanced Routing: 17.6 BLEU with FFT and 17.2 BLEU with DCT outperform Base at 15.4 and Random Projection at 15.5, supporting meaningful spectral routing features.The comparison reports FFT and DCT as effective transformations with O(d log d) efficiency.
  • FFT-Enhanced Routing: Expert selection is uneven across tasks, with LM Expert 2 selected significantly more often for ZH-EN, indicating language-pair-specific specialization.The authors hypothesize that experts become sensitive to different frequency components, including vocabulary- and syntax-related patterns.

B. The Effect of Injecting MoEs in Different Layers

The study compares where MoE layers are inserted and evaluates Mix-MoE against dense two-stage fine-tuning. Uniform MoE placement gives the best reported BLEU, while generic two-stage fine-tuning can suffer parameter conflict.

  • MoE Layer Placement: Uniformly distributed MoE layers achieve the highest BLEU for both ZH-EN and DE-EN among the tested placement strategies.Bottom-only placement is comparable for ZH-EN, while top-only placement causes a significant performance drop.
  • MoE Layer Placement: Dense-bottom, sparse-top placement performs between uniform placement and exclusive bottom placement, suggesting broader layer coverage remains beneficial.The authors attribute the result to expert specialization operating at multiple model levels.
  • Parameter Interference: PT+SFT often underperforms SFT and the base model on EN-CS, EN-DE, and EN-ZH, despite monolingual exposure in Stage 1.The comparison evaluates base, one-stage bilingual fine-tuning, two-stage dense fine-tuning, and Mix-MoE.
  • Parameter Interference: Mix-MoE consistently and significantly outperforms the listed baselines, especially the dense two-stage PT+SFT method.The authors link this result to mitigation of parameter interference through specialization.

D. The Effect of Expers Number

Increasing the number of experts improves BLEU but raises routing complexity and computational cost. The experiments also identify a moderate load-balancing coefficient as the best trade-off between translation quality and expert utilization.

  • Expert Number: BLEU improves as experts per LM and MT group increase from 2 to 4 to 8 on the evaluated translation tasks.The authors use 4 experts per group as the default to balance capacity, computational limits, and baseline comparability.
  • Expert Number: More experts provide greater specialization capacity but make routing more complex and increase computational cost.
  • Load Balancing: λLB = 0.01 achieves the best balance between translation quality and expert utilization in the reported experiments.Higher λLB values balance expert use more evenly, but BLEU first improves and then declines.
  • Routing Trade-offs: Increasing top-k generally improves MoE performance but requires routing each token to more experts, increasing FLOPs and memory requirements.

G. Cross-Lingual Generalization and Knowledge Transfer

Mix-MoE transfers translation improvements beyond the single language pair used for fine-tuning and scales across larger LLMs. The reported gains extend across most or all evaluated language directions.

  • Cross-Lingual Generalization: Fine-tuning on one language direction generally improves BLEU on other unseen language pairs without further training.The results suggest positive transfer rather than overfitting exclusively to the fine-tuning pair.
  • Knowledge Transfer: Stage 2 MT Expert training provides incremental gains over the Stage 1 monolingual expert-training results on unseen target pairs.
  • Scalability: Mix-MoE outperforms full fine-tuning and the original Llama3.1-8B across all 14 language directions.
  • Scalability: Mix-MoE achieves significant BLEU improvements over full fine-tuning and the base Qwen2.5-14B model across nearly all language pairs.These results are presented as evidence of robustness and scalability on a larger model.
Loading 2605.24681v1…