PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Model and Data for Multilingual Machine Translation with Open Large Language Models

Yuzhe Shang, Pengzhi Gao, Wei Liu, Jian Luan, Jinsong Su

arXiv:2602.11961v3cs.CL

TL;DR

Open LLM multilingual MT performance and the effects of model and data scaling across adaptation stages remain insufficiently understood. The paper studies these questions across 46 languages by adapting Gemma3 through continual pretraining and instruction finetuning, producing MiLMMT-46. MiLMMT-46 consistently outperforms open-source alternatives and remains competitive with proprietary systems, while the study is limited to models below 15 billion parameters.

  • Problem

    The paper addresses limited evidence about open LLM performance on multilingual MT and about model- versus data-scaling effects across continual pretraining and instruction finetuning.

  • Method

    The authors benchmark 46 languages and adapt Gemma3 through multilingual continual pretraining followed by instruction finetuning with a small, high-quality parallel dataset.

  • Results

    MiLMMT-46 consistently outperforms open-source alternatives and remains competitive with proprietary systems across varying language coverage.

  • Takeaways & Limitations

    The results support scalable, transparent, and deployable open multilingual translation systems based on large-scale multilingual pretraining and high-quality instruction finetuning.

  • Takeaways & Limitations

    The study investigates open LLMs with fewer than 15 billion parameters, leaving larger-model translation performance and scaling behavior unexplored.

Abstract

from arXiv · show

Open large language models (LLMs) have demonstrated improving multilingual capabilities in recent years. In this paper, we present a study of open LLMs for multilingual machine translation (MT) across a range of languages, and investigate the effects of model scaling and data scaling when adapting open LLMs to multilingual MT through continual pretraining and instruction finetuning. Based on the Gemma3 model family, we develop MiLMMT-46, which achieves top-tier multilingual translation performance across 46 languages. Extensive experiments show that MiLMMT-46 consistently outperforms recent state-of-the-art (SOTA) models, including Seed-X, HY-MT-1.5, and TranslateGemma, and achieves competitive performance with strong proprietary systems such as Google Translate and Gemini 3 Pro. Models are released at https://huggingface.co/collections/xiaomi-research/milmmt-46. Codes are released at https://github.com/xiaomi-research/gemmax.

1 Introduction

The paper studies how open LLMs can be adapted for multilingual MT and which model- and data-scaling strategies work across languages and training stages. It benchmarks 46 languages and releases MiLMMT-46 as an open alternative to proprietary systems.

  • 1 Introduction: The work addresses limited understanding of open LLM multilingual MT performance and the relative contributions of scaling choices across adaptation stages.
  • 1 Introduction: MiLMMT-46 provides many-to-many translation models of varying sizes based on Gemma3, designed as a strong and deployable open alternative.
  • 1 Introduction: The authors benchmark open-source LLMs for multilingual MT across 46 widely spoken languages and both English-centric and Chinese-centric directions.
  • 1 Introduction: The study examines model scaling versus data scaling across continual pretraining and instruction finetuning for multilingual MT.It continues pretraining Gemma3 on multilingual corpora and then applies instruction finetuning with a small, high-quality parallel dataset.
  • 1 Introduction: MiLMMT-46 consistently outperforms other open-source alternatives and remains competitive with Google Translate and Gemini 3 Pro.

2 Related Work

Prior multilingual MT systems combine continual pretraining, supervised finetuning, and reinforcement learning, while recent work explores data mixing and specialized supervision.

  • 2 Related Work: Recent multilingual LLMs typically improve MT through combinations of continual pretraining, supervised finetuning, and reinforcement learning.
  • 2 Related Work: Cui et al. (2025) study data mixing for continual pretraining and prioritize parallel corpora to better align multilingual representations.
  • 2 Related Work: Seed-X combines linguist-authored chain-of-thought supervision with multi-reward reinforcement learning for semantic and cultural distinctions across directions.

3 Datasets and Baseline Settings

The evaluation covers 46 languages using FLORES+ and WMT24++ and compares open LLMs with commercial, proprietary, and supervised multilingual MT systems.

  • 3 Datasets and Baseline Settings: The WMT24++ evaluation uses English sentences and excludes items marked as low quality for reference-free evaluation.
  • 3 Datasets and Baseline Settings: The study compares Qwen2.5, Qwen3, Gemma2, and Gemma3 series with Google Translate, Gemini 2.5/3 Pro, GPT-5, and NLLB-54.5B.
  • 3 Datasets and Baseline Settings: FLORES+ uses spBLEU and COMET, whereas WMT24++ uses the reference-free metrics XCOMET and COMETKiwi.

4 Benchmarking Open LLMs for Multilingual Machine Translation

The paper evaluates tokenizer efficiency and in-context multilingual translation across 46 languages. Gemma3 is the most balanced open-LLM tokenizer, while Gemma3 models achieve the strongest open-LLM translation performance and surpass NLLB-54.5B.

  • 4.1 Tokenizer Efficiency: Smaller length ratios indicate more efficient non-English tokenization relative to English, and Gemma3 has the most balanced tokenizer among open LLMs.
  • 4.1 Tokenizer Efficiency: Tokenizer efficiency is measured on FLORES+ devtest across languages by comparing tokenized English and non-English sentence lengths.
  • 4.2 In-context Multilingual Translation Performance with Open LLMs: Gemini 3 Pro and GPT-5 surpass Google Translate in most translation directions, showing strong multilingual translation among closed-source LLMs.
  • 4.2 In-context Multilingual Translation Performance with Open LLMs: Gemma3 models perform best among open LLMs and surpass the supervised NLLB-54.5B system in multilingual translation quality.The evaluation reports average performance on FLORES+ using eight randomly selected translation pairs as in-context exemplars per direction.

5 Model and Data Scaling for Multilingual MT with Open LLMs

The study examines model and data scaling for multilingual MT by continually pretraining Gemma3 models and instruction-finetuning them with high-quality parallel data. Increasing pretraining and finetuning data improves translation, while larger models perform better and become more data-efficient, with diminishing returns at higher scales.

  • Instruction Finetuning Data Scaling: The instruction-finetuning dataset contains approximately 264K quality-filtered sentence pairs generated from diverse multilingual sources and selected using XCOMET and COMETKiwi.
  • Continual Pretraining Data Scaling: The continual-pretraining corpus uses PFMS mixing, prioritizing parallel data and supplementing it with monolingual data to reach language-specific token budgets.An additional 0.1n billion monolingual tokens per language preserves long-context modeling capability.
  • Continual Pretraining Data Scaling: Increasing continual-pretraining data scale yields stable translation improvements across model sizes and language directions.
  • Continual Pretraining Data Scaling: Larger models achieve higher absolute performance, but COMET gains from additional continual-pretraining data diminish, especially for larger models.
  • Instruction Finetuning Data Scaling: Instruction finetuning consistently improves translation across model sizes and directions, with the largest gains from small to moderate datasets.
  • Instruction Finetuning Data Scaling: Approximately 100K high-quality sentence pairs suffice for robust performance across all 46 languages for the largest models, with diminishing returns beyond this scale.In the 100K setting, only 154 sentence pairs cover zh ↔xx directions excluding English, indicating strong zero-shot translation capability.

2. All MiLMMT models are trained using the full

MiLMMT models are evaluated against open-source and proprietary multilingual translation systems on overlapping languages using WMT24++ and FLORES+ metrics. MiLMMT-46-12B leads in limited-coverage settings and remains competitive with proprietary systems while outperforming major open-source baselines at broader coverage.

  • Multilingual MT Evaluation: The evaluation reports XCOMET and COMETKiwi for WMT24++, and spBLEU and COMET for FLORES+.
  • Multilingual MT Evaluation: At 31- and 46-language coverage, MiLMMT substantially surpasses Hunyuan-MT and HY-MT1.5 and remains competitive with Google Translate, Gemini 3 Pro, and GPT-5.
  • Multilingual MT Evaluation: MiLMMT clearly outperforms NLLB and TranslateGemma at comparable or larger scales as language coverage expands to 31 and 46 languages.

6 Conclusion

The paper studies model and data scaling for open-LLM multilingual MT across 46 languages and develops MiLMMT-46 from Gemma3. The models outperform open-source baselines, compete with proprietary systems, and show improved data efficiency with larger models.

  • 6 Conclusion: MiLMMT-46 consistently outperforms open-source baselines and remains competitive with proprietary systems across varying language coverage.
  • 6 Conclusion: Experiments on WMT24++ and FLORES+ show that properly adapted open LLMs can achieve strong many-to-many translation across 46 languages.
  • 6 Conclusion: Large-scale multilingual pretraining and high-quality instruction finetuning are highlighted as important for multilingual MT, while larger models show improved data efficiency and cross-lingual generalization.
  • 6 Conclusion: The released models are intended to support scalable, transparent, and deployable multilingual translation systems.

Limitations

The study is restricted by computational resources to multilingual in-context translation evaluation and open LLMs with fewer than 15 billion parameters. Scaling behavior for larger models remains unexplored.

  • Limitations: The study evaluates open LLMs with fewer than 15 billion parameters, leaving the translation performance and scaling behavior of larger models unexplored.

A Appendix

The appendix documents dataset composition, supported languages, tokenization, experimental settings, and evaluation results across baseline and MiLMMT models.

  • Figure 4 compares sentence-pair counts in simplified Chinese-centric and English-centric parallel datasets.
  • Figures 5 and 6 report spBLEU translation performance while varying continual-pretraining data scaling and instruction-finetuning sentence-pair counts.
  • The appendix lists 46 supported languages, their resource classifications, and tokenization efficiency across models.
  • Tables 10–13 provide evaluation results and hyperparameter settings for Gemma models and the pretraining and finetuning experiments.
  • Tables 14–18 summarize continual-pretraining dataset statistics for n values from 0.1 through 3.
  • Tables 19–24 report English-centric, Chinese-centric, and WMT24++ evaluation results for baseline models and MiLMMT models.
Loading 2602.11961v3…