PaperlayerView exact revision on arXiv

Source-linked AI summary

Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation

Chris Han, Pengzhi Gao, Pei Fu, Jian Luan

arXiv:2608.10812v2cs.CLcs.AI

TL;DR

High-quality parallel data is scarce and unevenly distributed, motivating reference-free multilingual machine-translation post-training. The paper applies language-gated reference-free reinforcement learning and checkpoint interpolation, producing models that consistently improve learned quality metrics and compare favorably with strong open and proprietary systems across 46 languages.

  • Problem

    Scarce and unevenly covered parallel data, especially for low-resource languages and non-English directions, motivates exploiting abundant source-side text for reference-free multilingual MT post-training.

  • Method

    The authors apply GRPO with language-gated rewards averaging two reference-free quality estimators, then interpolate SFT and RL checkpoints across three scales and 46 languages.

  • Results

    Across 46 languages, MiLMMT-46-v1.0 consistently improves learned quality metrics over SFT; WMT24++ XCOMET and COMETKiwi rise by 2.75 and 2.44 points.

  • Takeaways & Limitations

    Checkpoint interpolation retains most neural-quality gains while recovering substantial spBLEU, and the 12B model achieves leading reference-free scores among evaluated open and proprietary systems.

  • Takeaways & Limitations

    On-policy distillation reaches but does not surpass the quality frontier established by reinforcement learning with checkpoint interpolation.

Abstract

from arXiv · show

We study reference-free post-training for multilingual machine translation with open large language models. Starting from the supervised-finetuned MiLMMT-46-v0.1 models, we apply Group Relative Policy Optimization (GRPO) with a reward that averages two reference-free quality estimation models and is gated by language identification. We then linearly interpolate the supervised fine-tuning (SFT) and reinforcement learning (RL) model checkpoints to obtain MiLMMT-46-v1.0. Across 46 languages, the resulting models consistently improve translation quality over their SFT counterparts, outperform strong recent open baselines, including Seed-X, HY-MT2, and TranslateGemma, and achieve leading reference-free scores against evaluated proprietary systems such as Google Translate, Gemini 3 Pro, and GPT-5. We further investigate on-policy distillation (OPD) and find that it reaches, but does not surpass, the quality frontier achieved by RL with checkpoint interpolation. We release the models and code to facilitate future research.

1 Introduction

The paper addresses multilingual machine translation’s parallel-data limitations with reference-free post-training of open LLMs, using reinforcement learning on model-generated translations. Across three model scales and 46 languages, it evaluates RL, checkpoint interpolation, and on-policy distillation, releasing the resulting models and code.

  • Motivation: High-quality parallel data is scarce and unevenly distributed, especially for low-resource languages and non-English-centric directions, while source-side text is abundant.This motivates reference-free post-training methods that use source-side data without aligned target sentences.
  • Method: RL optimizes sequence-level rewards on model-generated translations rather than token-level likelihood against references, with GRPO and reference-free QE models providing the framework.The introduction situates this approach alongside multilingual translation systems such as Tower+ and HY-MT2.
  • Contributions: Across three model scales and 46 languages, reference-free RL consistently improves learned quality metrics over SFT and compares favorably with strong open and proprietary systems.The experiments build on the MiLMMT-46-v0.1 models as the starting point for further post-training.
  • Contributions: On-policy distillation reaches, but does not surpass, the quality frontier achieved by RL with checkpoint interpolation.The paper investigates OPD as an alternative post-training approach.
  • Contributions: The authors release the MiLMMT-46-v1.0 models and code to facilitate future research.The released models result from the studied post-training process.

2 Methodology

The method applies GRPO to SFT models using a reference-free reward that combines quality estimation with language identification, then interpolates SFT and RL checkpoints. Training uses reference-discarded multilingual data and KL-regularized optimization to preserve SFT behavior while incorporating RL gains.

  • SFT–RL checkpoint interpolation: Linear interpolation between θSFT and θRL reduces reward-induced drift while retaining RL gains without additional training.α controls the SFT contribution and 1 −α controls the RL contribution.
  • Training objective: GRPO samples translation groups and optimizes group-normalized advantages under a clipped objective with KL regularization toward the fixed SFT policy.The clipping range is controlled by ϵ, while β controls the KL penalty.
  • Reward design: The reference-free reward combines XCOMET2 and COMETKiwi3 scores, retaining them only when OpenLID-v3 identifies the intended target language.The gate prevents fluent but wrong-language outputs from receiving high quality-estimation rewards, mitigating reward hacking.
  • RL data: 263982 instances spanning 192 translation directions form the RL dataset after references are discarded while sources and translation directions are retained.The method samples G translations per instance and scores them to provide within-group reward variation for GRPO.
  • SFT–RL checkpoint interpolation: The RL checkpoint is named MiLMMT-46-v0.1-RL, while the selected interpolated checkpoint is named MiLMMT-46-v1.0.The starting SFT model is MiLMMT-46-v0.1.

3 Experimental Setup

Experiments cover 46 languages on FLORES+ and WMT24++, comparing against proprietary, open-source, and NLLB baselines. Models use a shared GRPO recipe across 1B, 4B, and 12B scales, followed by SFT–RL checkpoint interpolation and reference-free evaluation with XCOMET and COMETKiwi.

  • Datasets: Experiments span 46 languages and evaluate multilingual translation on FLORES+ and WMT24++, using English-source sentences and excluding low-quality WMT24++ examples.The WMT24++ filtering is applied specifically for reference-free evaluation.
  • Baselines: Baselines include Google Translate, Gemini 2.5/3 Pro, GPT-5, NLLB, and open-source multilingual translation models such as Seed-X, Hunyuan-MT, HY-MT2, and TranslateGemma.Other open-source baselines are Tower-Plus and GemmaX2-28.
  • Training and checkpoint interpolation configurations: The same RL recipe is applied at 1B, 4B, and 12B scales with verl4, vLLM rollouts, G=8 sampled candidates, and KL regularization toward the SFT reference.Final checkpoints are formed by interpolating SFT and RL parameters, with additional interpolation values explored.
  • Evaluation: XCOMET and COMETKiwi provide reference-free evaluation on both benchmarks, while FLORES+ additionally reports reference-based spBLEU and XCOMET scores.Both reference-free evaluators are 10B-parameter models with high correlation with human judgments.
  • Evaluation: Baseline evaluations use languages shared with the 46-language model, and translations are generated with greedy decoding.This evaluation subset applies to each baseline.

4 Experimental Results

Reference-free post-training consistently improves MiLMMT-46 translation quality across model scales and language subsets, outperforming strong open and proprietary systems in reference-free evaluation. Checkpoint interpolation provides a controllable spBLEU–reference-based XCOMET trade-off, with α = 0.5 recovering substantial spBLEU while limiting XCOMET loss.

  • Effects of reference-free post-training: Averaged across model scales, WMT24++ XCOMET and COMETKiwi improve by 2.75 and 2.44 points, respectively, over supervised-finetuned MiLMMT-46-v0.1.On FLORES+, averaged across direction groups and scales, reference-based XCOMET, reference-free XCOMET, and COMETKiwi improve by 1.17, 1.41, and 1.17 points.
  • Full 46-language comparison: MiLMMT-46-12B-v1.0 achieves the highest reference-free XCOMET and COMETKiwi scores on WMT24++ and across all four FLORES+ direction groups.It outperforms evaluated proprietary systems including Google Translate, Gemini 2.5/3 Pro, and GPT-5.
  • Full 46-language comparison: MiLMMT-46-12B-v1.0 surpasses GPT-5 in three of four FLORES+ direction groups and achieves a higher average spBLEU, 35.14 vs. 34.94.Its average spBLEU also exceeds all evaluated external open-source baselines, including NLLB-54.5B and TranslateGemma-27B.
  • Reference-free post-training gains persist across model scales: Reference-free XCOMET improves by 2.87, 2.59, and 2.79 points at the 1B, 4B, and 12B scales, respectively.Training and validation rewards increase early and then stabilize at all three model scales.
  • Comparisons on shared language subsets: Across four shared language subsets, MiLMMT-46-12B-v1.0 ranks first in 15 of 16 reference-based XCOMET comparisons.It also achieves the best reference-free scores in nearly all comparisons against Tower-Plus, Seed-X, GemmaX2, and the HY-MT series.
  • Interpolation as a controllable quality trade-off: At α = 0.5, interpolation recovers 2.79, 3.70, and 4.21 spBLEU points at 1B, 4B, and 12B, while reducing reference-based XCOMET by 0.53, 0.56, and 0.54 points.As α increases, spBLEU rises monotonically while reference-based XCOMET decreases; beyond α = 0.5, spBLEU gains diminish.

5 On-Policy Distillation Analysis

On-policy distillation transfers much of a post-trained 12B teacher’s performance to smaller students without separate reinforcement learning, but does not surpass the quality frontier established by RL with checkpoint interpolation. Across objectives, distillation weights, and initializations, OPD is therefore an alternative transfer method rather than an improvement over RL.

  • Method: OPD trains 1B and 4B students from their own rollouts using a post-trained MiLMMT-46-12B-v1.0 teacher’s signal at student-induced states.The students use the same reference-free RL data described in Section 2.
  • Method: PG-OPD estimates reverse-KL divergence from student-sampled tokens and optimizes its negative as a per-token reward with GRPO.The study adopts PG-OPD as its primary implementation; GKD is reported only as a control experiment in Appendix E.
  • Results: 34.17 versus 33.96 spBLEU and 90.82 versus 90.91 reference-based XCOMET show the 4B OPD student closely matches its baseline, while the 1B student scores 85.22 versus 85.94 XCOMET.These comparisons are against the corresponding MiLMMT-46-v1.0 baselines on FLORES+.
  • Results: At 4B, reducing λ from 1 to 0.001 raises reference-based XCOMET from 90.98 to 91.47 while lowering spBLEU from 33.98 to 30.20.Intermediate λ values follow the same trade-off, and λ = 1 remains close to MiLMMT-46-v1.0 baselines across benchmarks and scales.
  • Overall finding: Across objectives, distillation weights, and student initializations, OPD and RL + OPD remain close to but do not extend the quality frontier established by RL and checkpoint interpolation.The paper presents OPD as a robustness finding and an alternative way to transfer post-training gains to smaller models.

6 Conclusion

The paper studies reference-free post-training for multilingual machine translation with open language models across multiple scales and 46 languages. Language-gated quality-estimation rewards improve neural quality metrics, while SFT–RL checkpoint interpolation recovers much of RL’s spBLEU loss while retaining most neural-metric gains.

  • Conclusion: The study analyzes GRPO, SFT–RL checkpoint interpolation, and on-policy distillation across 1B, 4B, and 12B model scales.The experiments cover multilingual machine translation with open large language models.
  • Conclusion: Experiments on 46 languages using WMT24++ and FLORES+ show that language-gated quality-estimation rewards improve reference-free XCOMET and COMETKiwi, plus reference-based XCOMET.The gains are reported consistently across the evaluated metrics.
  • Conclusion: SFT–RL checkpoint interpolation in MiLMMT-46-v1.0 recovers a substantial portion of RL’s spBLEU loss while retaining most neural quality-metric gains.The interpolation builds on MiLMMT-46-v0.1 and addresses the lower spBLEU of the RL checkpoints.

A Distribution of RL Training Data · B Language Details

The RL data pool contains 31,572 retained source sentences across 192 translation directions, with separate training and validation splits. The models support 46 languages.

  • A Distribution of RL Training Data: 31,572 source sentences remain after effective-data selection.The retained pool is the basis for RL training data.
  • A Distribution of RL Training Data: 30,572 sentences are allocated for training.The remaining sentences form the held-out validation set.
  • A Distribution of RL Training Data: 1,000 sentences are held out for validation.This split is reported alongside the training allocation.
  • A Distribution of RL Training Data: The retained data spans 192 translation directions.Table 3 reports sentence counts for each direction.
  • A Distribution of RL Training Data: Table 3 organizes sentence counts by translation direction.It follows the language-code convention of Shang et al. (2026).
  • B Language Details: The models support 46 languages.Table 4 provides information about these supported languages.

C Training Hyper-parameters · D Detailed Evaluation Results

Table 5 provides the complete GRPO training hyper-parameters, shared across the 1B, 4B, and 12B models. The supplied passage does not describe the detailed evaluation results.

  • C Training Hyper-parameters: Table 5 lists the complete set of GRPO hyper-parameters.
  • C Training Hyper-parameters: The listed hyper-parameters are shared across the 1B, 4B, and 12B models.
  • C Training Hyper-parameters: The passage identifies Table 5 as the source for GRPO training settings.
  • C Training Hyper-parameters: The hyper-parameter configuration applies to the 1B model.
  • C Training Hyper-parameters: The same hyper-parameter configuration applies to the 4B model.
  • C Training Hyper-parameters: The same hyper-parameter configuration applies to the 12B model.

D.1 Reference-Free Results

Tables 6–11 present per-direction reference-free translation results, reported using XCOMET and COMETKiwi.

  • Reference-Free Results: Tables 6–11 report the reference-free results.The results are organized by translation direction.
  • Reference-Free Results: The reported results are per-direction.Each translation direction is evaluated separately.
  • Reference-Free Results: The metrics are XCOMET and COMETKiwi.Both are used to report reference-free translation quality.

D.2 Reference-Based Results

This section reports per-direction reference-based FLORES+ results using spBLEU and XCOMET.

  • Tables 12–15 report per-direction reference-based FLORES+ results with spBLEU and XCOMET.

E Full On-Policy Distillation Results

The full on-policy distillation experiments compare OPD and RL+OPD students distilled from a fixed MiLMMT-46-12B-v1.0 teacher, using either distillation alone or combined with the QE-based task reward. Results are reported as 46-language macro-averages across reference-free and reference-based metrics.

  • OPD Training Details: Pure OPD samples one student trajectory per prompt without the task reward, while RL+OPD uses the same task reward and group size G = 8 as GRPO.The OPD experiments use a learning rate of 1 × 10−6, prompt and PPO mini-batches of 128, three epochs, and maximum prompt and response lengths of 4096 tokens.
  • Full Results: RL+OPD combines the QE-based task reward with a distillation loss weighted by λ, whereas OPD uses only the distillation loss.All students are distilled from the fixed MiLMMT-46-12B-v1.0 teacher.
  • Full Results: Unless noted, students are initialized from v0.1; rows marked init. v1.0 instead start from the interpolated checkpoint.The full results distinguish initialization from the interpolated checkpoint when reporting OPD and RL+OPD outcomes.
  • Full Results: Table 16 reports complete OPD and RL+OPD results as 46-language macro-averages, with WMT24++ reporting reference-free XCOMET / COMETKiwi.Each FLORES+ cell reports spBLEU / reference-based XCOMET / reference-free XCOMET / COMETKiwi.
Loading 2608.10812v2…