NiuTrans.LMT: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs

1 Introduction

• Symmetric reuse of multi-way SFT data can improve pivot-to-X translation while substantially degrading reverse X-to-pivot translation, producing fluent but less faithful outputs.The authors attribute the degradation to repeated pivot targets that encourage shortcut learning.Source: §1 · p. 2
• Strategic Downsampling retains full pivot-to-X supervision while keeping only a small fraction of reverse-direction instances to reduce many-to-one mappings.The authors report that this stabilizes reverse-direction translation.Source: §1 · p. 2
• Parallel Multilingual Prompting adds an auxiliary parallel sentence to translation instructions to promote cross-lingual transfer during training and optionally at inference.Inference-time use requires an available auxiliary translation.Source: §1 · p. 2
• LMT is a Chinese–English-centric suite spanning four sizes, 60 languages, and 234 translation directions.Its four sizes are 0.6B, 1.7B, 4B, and 8B.Source: §1 · p. 2
• LMT-60-4B is on par with or better than substantially larger baselines on overlapping directions.The comparison includes X-ALMA-13B, Aya-101-13B, and NLLB-54B.Source: §1 · p. 2

2 The Pitfall of Directional Degeneration

• Symmetric multi-way reuse pairs a pivot target with many source languages, encouraging shortcut learning that sacrifices source-semantic faithfulness.The resulting outputs can be fluent but factually unfaithful.Source: §2 · p. 3
• Disjoint reverse-direction data avoids the collapse seen with fully symmetric reuse, implicating data reuse rather than intrinsic X→En/Zh difficulty.Figure 3 contrasts the disjoint-data setting with fully symmetric reuse for Qwen3-4B-Base.Source: §2 · p. 3, Figure 3 · p. 4
• Reverse-direction COMET performance peaks at p ≈5% and declines sharply as reverse-example retention approaches 100%.The authors interpret the trajectory as evidence that excessive target repetition triggers degeneration.Source: §2 · p. 4, Figure 3 · p. 4
• The same asymmetric degradation appears across evaluated model sizes and backbone families, and becomes more pronounced as the number of SFT languages increases.The study varies Qwen3 from 0.6B to 8B, tests Llama-3.1-8B and Gemma-2-9B, and varies the number of languages from 10 to 50.Source: §2 · pp. 3–4

2.1 Mitigation via Strategic Downsampling

• Strategic Downsampling keeps all En/Zh→X data and independently retains each multi-way X→En/Zh instance with probability p, using p=5% by default.The strategy operates within the existing multi-way SFT corpus rather than requiring disjoint external SFT data.Source: §2.1 · p. 4

3 Parallel Multilingual Prompting

• PMP addresses the underexplored use of multi-source signals directly in large-scale MT SFT by adding auxiliary parallel context to prompts.The auxiliary anchor is intended to support more direct cross-lingual transfer.Source: §3 · p. 4
• PMP models the target given both the source sentence and an auxiliary-language translation, providing two parallel views of the same semantic content.The auxiliary sentence acts as a semantic anchor for interpreting the source and guiding target generation.Source: §3 · p. 5
• Auxiliary-language selection considers both linguistic affinity to the supported language and the model’s proficiency in the anchor language.For Chinese↔X, the method uses English as the auxiliary language.Source: §3 · p. 5
• Training mixes PMP and standard-prompt samples, while inference defaults to standard prompting and can activate PMP when an auxiliary sentence is supplied.The auxiliary sentence may come from an external MT system or be self-generated.Source: §3 · p. 5

4 The LMT Framework

• LMT spans 60 languages and 234 translation directions across four sizes: 0.6B, 1.7B, 4B, and 8B.Source: §4 · p. 5

4.1 Adaptation Pipeline

• The three-stage pipeline uses CPT, SFT with SD and PMP, then GRPO-based preference optimization using SFT prompts and COMET-22 rewards.The progression of components is tracked with COMET-22 scores for 4B and 8B models in Table 2.Source: §4.1 · p. 5, Table 2 · p. 6

4.2 CPT Data Curation

• Curation yields approximately 2.1B English-centric and 2.9B Chinese-centric sentence pairs across 117 language pairs, after synthetic-data augmentation and quality filtering.Chinese-centric parallel data were scarce in the initial pool, motivating augmentation with synthetic translations.Source: §4.2 · p. 6

5.1 Setup

• Evaluation uses FLORES-200 Devtest and COMET-22 as the primary metric, with SacreBLEU in the appendix and 4B/8B results in the main text.CPT training uses 90B tokens balanced across monolingual, Chinese-centric bilingual, and English-centric bilingual data; reverse-direction SFT retains 5% of examples.Source: §5.1 · p. 6

5.2 Main Results

• SD recovers X→En/Zh directions by approximately 2–13 COMET points over the SFT baseline, surpassing base scores while maintaining strong En/Zh→X results.Source: §5.2 · p. 7
• CPT adds 1–3 COMET points in high- and medium-resource scenarios and 5–9 in low-resource directions; PMP improves X→En/Zh, while GRPO adds about 0.3–0.8 across resource tiers.Source: §5.2 · p. 7

5.3 Comparison with Existing MMT Systems

• LMT exceeds Aya-101-13B and LLaMAX3-Alpaca by approximately 3–6 COMET points on average, outperforms NLLB-54B and GemmaX2-28-9B, and matches Seed-X-PPO-7B.Table 3 compares systems over overlapping languages and four directions; LMT-60-4B matches or exceeds substantially larger models on those overlapping directions.Source: Table 3 · p. 7, §5.3 · pp. 7–8

5.4 Understanding the Effect of PMP

• At inference, PMP-S performs comparably to or better than PMP-O on X→En/Zh, while clear Zh→X gains appear only with PMP-O.The reported result supports PMP-S as a self-generated enhancement for X→En/Zh, but not for Zh→X.Source: §5.4 · p. 8
• PMP training raises In-Group average COMET-22 from 85.20 to 86.11 and also yields positive zero-shot uplift for Out-of-Group directions.Out-of-Group directions were not explicitly used as PMP anchors during training.Source: §5.4 · p. 8

• Recent work adapts decoder-only LLMs for translation, adding capabilities such as context awareness and reasoning alongside translation quality.Source: §6 · p. 8
• Existing LLM-based MMT systems are often limited in language breadth, remain English-centric, and struggle with low-resource translation.Source: §6 · p. 8
• Although multi-way signals can support cross-lingual alignment, their consistent use in large-scale MMT and the pitfalls of symmetric SFT mixtures remain under-explored.Source: §6 · pp. 8–9

• LMT covers 60 languages across 234 directions and introduces SD to mitigate directional degeneration and PMP to enhance cross-lingual transfer.The models are released as publicly available baselines for future multilingual translation research.Source: §7 · p. 9

Limitations

• The evaluation relies primarily on COMET and academic benchmarks, the Chinese–English-centric design may not suit other language communities, and 60 languages remain a small subset of global linguistic diversity.Expansion is constrained by training cost and by the availability and quality of text and parallel data, especially for underrepresented languages with limited written resources.Source: Limitations · p. 9

A Continued Pre-training Data Curation

• Because high-quality, large-scale, diverse CPT data are challenging to construct, the authors use a systematic, multi-stage curation pipeline to target linguistic breadth and consistent quality.Source: A Continued Pre-training Data Curation · p. 13

A.1 CPT Monolingual Data

• The monolingual CPT corpus uses SlimPajama for English, Skywork for Chinese, and CulturaX, OpenData-Lab, and Wikimedia for other languages.Source: §A.1 · p. 13

A.2 CPT Bilingual Data

• The bilingual corpus is built from OPUS 3 sub-corpora and expanded with direct and English-pivoted pseudo-parallel synthesis.Direct synthesis translates monolingual data into En/Zh→X pairs; pivoted synthesis leverages En↔X and En→Zh models to obtain Zh↔X data.Source: §A.2 · p. 13
• Filtering removes sentence pairs with a maximum length ratio above 3.0 and uses resource-adaptive language identification thresholds before CometKiwi scoring and selection.The resulting corpus contains approximately 2.1B English-centric and 2.9B Chinese-centric sentence pairs across 117 targeted directions.Source: §A.2 · p. 13

• COMETKiwi scores for low-resource language pairs are noticeably more left-skewed than for high-resource pairs, especially for Chinese-centric pairs.Source: §A.3 · p. 13
• The authors attribute the skew potentially to scarce clean bilingual data and to less reliable COMETKiwi scores for underrepresented or non-English pairs.The latter could yield systematically lower scores regardless of true pair quality.Source: §A.3 · p. 13
• The findings point to both limited non-English-centric bilingual data and potential quality-estimation calibration limitations, motivating more robust QE methods for diverse language pairs.Source: §A.3 · p. 13

B CPT Gains Across Languages

• CPT improves performance across nearly all languages and directions, with larger gains for medium- and low-resource languages and substantial gains in En/Zh→X generation.The baseline often scores lower in En/Zh→X than in X→En/Zh, consistent with prior observations (Arivazhagan et al., 2019).Source: B CPT Gains Across Languages · pp. 13–14

C Additional Evaluation

• The additional evaluation reports FLORES results by resource tier and uses WMT24++ to test robustness beyond FLORES.Source: C Additional Evaluation · p. 14

C.1 FLORES Breakdown by Resource Tier

• On the 54-language overlap with Aya-101-13B, LMT-60-4B improves low-resource En→X from 81.68 to 86.92 and low-resource X→Zh from 81.58 to 85.98.On the 59-language overlap with NLLB-54B, it improves low-resource X→Zh from 80.56 to 85.88 and Zh→X from 82.33 to 84.52.Source: §C.1 · p. 14
• LMT is broadly competitive on high-resource pairs and delivers larger, more consistent gains on the long tail rather than relying on a narrow subset of directions.Source: §C.1 · p. 14

C.2 Results on WMT24++

• WMT24++ emphasizes document-level cross-sentence coherence and discourse adequacy; the reported comparisons include models without WMT24++ training and use overlapping supported directions.Source: §C.2 · p. 14, Table 8 · p. 19, Table 9 · p. 19
• The authors say LMT’s sentence-pair-focused SFT and limited document-context training can weaken cross-sentence consistency, and propose targeted document-level data and context-aware SFT.Source: §C.2 · p. 14
• On WMT24++, LMT remains competitive on many directions but trails Hunyuan-MT on several subsets.Source: §C.2 · p. 14, Table 8 · p. 19, Table 9 · p. 19