PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study

Menglong Cui, Pengzhi Gao, Wei Liu, Jian Luan, Bin Wang

arXiv:2502.02481v4cs.CL

TL;DR

The paper asks how practical-scale open LLMs perform on multilingual MT and how their training data should combine monolingual and parallel resources. It benchmarks recent models across 28 languages, introduces PFMS continual pretraining, and develops GemmaX2-28-9B. The resulting model achieves translation performance comparable with GPT-4-turbo and Google Translate, while the study is limited to models below ten billion parameters.

  • Problem

    The multilingual translation performance of the latest practical-scale open LLMs and the optimal balance of monolingual and parallel training data remain unclear.

  • Method

    The paper benchmarks recent open LLMs across 28 languages and introduces Parallel-First Monolingual-Second data mixing during continual pretraining, followed by instruction finetuning on high-quality translation pairs.

  • Results

    GemmaX2-28-9B consistently outperforms open-source alternates and achieves competitive translation performance with Google Translate and GPT-4-turbo across 28 languages.

  • Takeaways & Limitations

    GemmaX2-28-9B is presented as an open many-to-many translation model with top-tier quality across 28 languages.

  • Takeaways & Limitations

    Because of limited computational resources, the study evaluates only open LLMs smaller than ten billion parameters, leaving larger models and their optimal data recipes unclear.

Abstract

from arXiv · show

Large language models (LLMs) have shown continuously improving multilingual capabilities, and even small-scale open-source models have demonstrated rapid performance enhancement. In this paper, we systematically explore the abilities of open LLMs with less than ten billion parameters to handle multilingual machine translation (MT) tasks. We conduct comprehensive evaluations on six popular LLMs and find that models like Gemma2-9B exhibit impressive multilingual translation capabilities. We then introduce the Parallel-First Monolingual-Second (PFMS) data mixing strategy in the continual pretraining stage to further enhance the MT performance and present GemmaX2-28, a 9B model achieving top-tier multilingual translation performance across 28 languages. Specifically, GemmaX2-28 consistently outperforms the state-of-the-art (SOTA) models such as TowerInstruct and XALMA and achieves competitive performance with Google Translate and GPT-4-turbo.

1 Introduction

This paper examines practical-scale open LLMs for multilingual MT across 28 languages and investigates how continual-pretraining data mixtures affect translation quality. It introduces PFMS and releases GemmaX2-28-9B, which is competitive with major closed-source systems.

  • Existing evidence is outdated for the latest open LLMs, leaving their multilingual translation performance unclear.
  • Gemma2-9B shows remarkable multilingual translation capabilities but still trails strong closed-source models before further enhancement.
  • The study benchmarks recent open LLMs on multilingual MT across 28 widely spoken languages, including English-centric and Chinese-centric directions.
  • The paper introduces Parallel-First Monolingual-Second (PFMS) data mixing for continual pretraining to improve multilingual MT.
  • GemmaX2-28-9B is a released 9B many-to-many translation model supporting 28 languages and competing with Google Translate and GPT-4-turbo.

2 Related Work

Related work shows growing multilingual LLM and MT efforts, but recent open-model translation performance and the best balance of monolingual and parallel data remain open questions.

  • Prior evaluations found evolving multilingual translation ability, but assessed older open LLMs rather than the latest practical-scale models.
  • The study evaluates the latest open-source LLMs on translation tasks across 28 languages and reports that they still show significant performance disparities.
  • In-context translation methods provide parallel examples to guide target-sentence generation, with some studies using semantically related examples.
  • Continual-pretraining research combines monolingual and parallel data, but the optimal mixture for multilingual MT remains underexplored.

3 Datasets and Baseline Settings

The experiments evaluate multilingual translation across 28 languages using FLORES-200 and WMT-24, comparing recent open LLMs with commercial, instruction-tuned, and large multilingual baselines.

  • The evaluation covers 28 languages and uses FLORES-200 plus WMT-24 test sets to assess multilingual translation while reducing data-leakage concerns.
  • The open-model comparison includes Mistral-7B-v0.3, Qwen2/2.5-7B, LLaMA3/3.1-8B, and Gemma2-9B.
  • Google Translate and GPT-3.5/4 Turbo represent commercial and closed-source systems, with GPT models evaluated using the same in-context strategy as open LLMs.
  • NLLB-54.5B provides a large encoder-decoder multilingual NMT baseline.
  • FLORES-200 performance is measured with spBLEU and COMET, while WMT-24 uses the reference-free metrics XCOMET and COMETKiwi.

4 Benchmarking Open LLMs for Multilingual Machine Translation

This section evaluates tokenizer efficiency and multilingual MT performance for open-source LLMs across FLORES-200 and WMT-24. It finds strong open-LLM translation capability, while noting gaps relative to supervised and closed-source systems.

  • Tokenizer Efficiency: Tokenizer efficiency is evaluated using length ratios between tokenized English sentences and corresponding non-English sentences, with smaller ratios preferred.The evaluation uses FLORES200 devtest sentences and compares open-source tokenizers with NLLB-54.5B.
  • In-context Multilingual Translation Performance: Open LLMs are assessed with five randomly selected translation pairs as in-context exemplars across 28 languages on FLORES-200 and WMT24.The same five exemplars are used for each translation direction.
  • In-context Multilingual Translation Performance: Gemma2-9B performs best among the evaluated open LLMs and outperforms GPT-3.5-turbo on average.The comparison is based on averaged multilingual translation performance reported for WMT-24 and FLORES-200.
  • In-context Multilingual Translation Performance: Open LLMs still lag behind supervised NMT models such as NLLB-54.5B, especially for low-resource languages.Google Translate outperforms GPT-4-turbo on both evaluated benchmarks, while open-LLM results on WMT-24 and FLORES-200 show similar trends.
  • In-context Multilingual Translation Performance: COMET scores increase dramatically from 1 to 5 exemplars and then plateau, except for Mistral-7B-v0.3.The exception is attributed to its relatively weak multilingual capability.

5 GemmaX: Boosting Multilingual Translation with Gemma Models

This section develops GemmaX2-28 by combining multilingual data, systematic data-recipe experiments, and PFMS continual pretraining with supervised finetuning. PFMS generally improves translation, enabling strong performance across 28 languages and competitiveness with leading systems.

  • 5.1 Pretraining Data: Gemma2-9B is continually pretrained on multilingual monolingual and parallel corpora, then finetuned with a small high-quality translation dataset.The data sources include CulturaX, MADLAD-400, OPUS parallel corpora, and human-annotated or sampled translation pairs.
  • 5.4 Exploring the Best Data Recipe for Multilingual Translation with LLMs: The experiments compare monolingual-only, 2:1, 1:1, 1:2, and parallel-only mixtures across low-, mid-, and high-resource language groups.Performance is reported for both en→xx and xx→en directions using COMET and BLEU-based evaluations.
  • 5.4 Exploring the Best Data Recipe for Multilingual Translation with LLMs: Monolingual-only continual pretraining is suboptimal because it improves overall performance while reducing high-resource xx→en translation quality.The degradation is especially apparent for high-resource languages.
  • 5.4 Exploring the Best Data Recipe for Multilingual Translation with LLMs: Adding parallel data improves overall translation, with larger parallel quantities benefiting high-resource directions but not clearly improving low- and mid-resource languages.The reported resource-dependent pattern motivates prioritizing parallel data while retaining monolingual data when needed.
  • 5.5 Main Result: PFMS prioritizes parallel data within 2 billion tokens per language and supplements it with monolingual data, generally outperforming alternative mixtures except for high-resource xx→en.GemmaX2-28-9B uses PFMS continual pretraining followed by high-quality translation finetuning and outperforms current open-source SOTA models.

6 Conclusion

The paper evaluates practical-scale open LLMs for multilingual translation, proposes PFMS to improve their data mixture, and reports translation performance comparable to GPT-4-turbo and Google Translate. GemmaX2-28-9B is presented as the open model with the highest translation quality, while broader language coverage remains future work.

  • 6 Conclusion: The study evaluates latest open LLMs below ten billion parameters and proposes PFMS for multilingual translation.GemmaX2-28-9B is obtained by applying PFMS during continual pretraining on Gemma2-9B.
  • 6 Conclusion: PFMS brings GemmaX2-28-9B’s translation performance to a level comparable with GPT-4-turbo and Google Translate.The conclusion reports this comparison after systematic multilingual evaluation and data-recipe exploration.
  • 6 Conclusion: The paper identifies GemmaX2-28-9B as the open model with the highest translation quality and aims to support more languages in future work.The claim is explicitly limited to the paper’s stated comparison and future-work scope.

Limitations

The study is limited to open LLMs with fewer than ten billion parameters because of computational-resource constraints. Performance and optimal data recipes for larger models remain unclear.

  • Limitations: The study evaluates multilingual translation and data mixing only for open LLMs below ten billion parameters.The authors state that larger-model translation performance and optimal data recipes remain unclear.

A Appendix

The appendix documents the model’s 28-language coverage, evaluation benchmarks, training settings, data statistics, and translation-performance comparisons across models and language directions.

  • Table 3 lists the 28 languages supported by the model and classifies their resource levels using Joshi et al.’s taxonomy.
  • Tables 4 and 10 report WMT-24 translation performance using XCOMET and COMETKiwi.Open LLM results in Table 4 use 5-shot in-context learning.
  • Tables 5 and 11 report English-centric FLORES-200 performance with spBLEU and COMET.
  • Tables 6 and 12 report Chinese-centric FLORES-200 performance with spBLEU and COMET.
  • Figure 5 compares BLEU across data recipes and language-resource groups in both en→xx and xx→en directions.The figure separates low-, mid-, and high-resource languages and uses a right-hand y-axis for directly finetuned Gemma2-9B.
  • Tables 7–9 provide hyperparameter settings for pretraining and finetuning, plus dataset statistics for the PFMS strategy.
Loading 2502.02481v4…