PaperlayerView exact revision on arXiv

Source-linked AI summary

MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning

Zhaopeng Feng, Shaosheng Cao, Jiahan Ren, Jiayuan Su, Ruizhe Chen, Yan Zhang, Zhe Xu, Yao Hu, Jian Wu, Zuozhu Liu

arXiv:2504.10160v1cs.CLcs.AIcs.LG

TL;DR

Machine translation lacks the fixed, automatically verifiable outputs that make pure R1-Zero-style reinforcement learning straightforward, leaving its use for MT underexplored. MT-R1-Zero addresses this gap with a rule-metric mixed reward and achieves competitive or leading results across benchmarks, including strong multilingual, low-resource, and out-of-distribution generalization. The study also finds that reward design, model adaptability, and emergent reasoning patterns shape outcomes, while sophisticated iterative self-correction remains limited.

  • Problem

    Machine translation outputs are flexible and difficult to evaluate with explicit rules, leaving pure R1-Zero-style reinforcement learning underexplored for MT.

  • Method

    MT-R1-Zero adapts the R1-Zero-like RL paradigm to MT using rule-metric mixed rewards over format, lexical, semantic, and mixed translation-quality objectives.

  • Results

    MT-R1-Zero achieves competitive or leading performance across in-domain, out-of-distribution, multilingual, and low-resource MT tasks, including a 1.26-point average gain of 3B-Mix over TowerInstruct-7B-v0.2.

  • Takeaways & Limitations

    Reward metric choice, LLM adaptability, training dynamics, and emergent reasoning patterns are central considerations when adapting R1-Zero-like RL to MT.

  • Takeaways & Limitations

    The emergent reasoning observed does not achieve the sophisticated iterative self-correction demonstrated in mathematical reasoning tasks.

Abstract

from arXiv · show

Large-scale reinforcement learning (RL) methods have proven highly effective in enhancing the reasoning abilities of large language models (LLMs), particularly for tasks with verifiable solutions such as mathematics and coding. However, applying this idea to machine translation (MT), where outputs are flexibly formatted and difficult to automatically evaluate with explicit rules, remains underexplored. In this work, we introduce MT-R1-Zero, the first open-source adaptation of the R1-Zero RL framework for MT without supervised fine-tuning or cold-start. We propose a rule-metric mixed reward mechanism to guide LLMs towards improved translation quality via emergent reasoning. On the WMT 24 English-Chinese benchmark, our MT-R1-Zero-3B-Mix achieves competitive performance, surpassing TowerInstruct-7B-v0.2 by an average of 1.26 points. Meanwhile, our MT-R1-Zero-7B-Mix attains a high average score of 62.25 across all metrics, placing it on par with advanced proprietary models such as GPT-4o and Claude-3.5-Sonnet, while the MT-R1-Zero-7B-Sem variant achieves state-of-the-art scores on semantic metrics. Moreover, our work exhibits strong generalization capabilities on out-of-distribution MT tasks, robustly supporting multilingual and low-resource settings. Extensive analysis of model behavior across different initializations and reward metrics offers pioneering insight into the critical role of reward design, LLM adaptability, training dynamics, and emergent reasoning patterns within the R1-Zero paradigm for MT. Our code is available at https://github.com/fzp0424/MT-R1-Zero.

1 Introduction

MT-R1-Zero adapts R1-Zero-like pure reinforcement learning to machine translation, where flexible outputs make explicit rule-based evaluation difficult. Its mixed rewards and experiments show competitive translation performance, broad generalization, and gains driven primarily by RL rather than forced reasoning or verbosity.

  • MT outputs are difficult to evaluate with explicit rules because they are flexibly generated, leaving R1-Zero-like methods underexplored for translation.
  • Existing MT reasoning methods often require structured Chain-of-Thought data, complex search, or multi-stage prompting, while pure RL remains largely unexplored.
  • MT-R1-Zero introduces the first open-source R1-Zero-like MT implementation with lexical, semantic, and mixed reward objectives.
  • 1.26 points: MT-R1-Zero-3B-Mix surpasses TowerInstruct-7B-v0.2 by an average across BLEU, COMETKiwi, and XCOMET on WMT 24 EN-ZH.
  • MT-R1-Zero-7B-Mix surpasses LLaMA-3.1-70B by 1.24 points and Qwen2.5-72B by 0.48 points, while performing on par with GPT-4o and Claude-3.5-Sonnet.
  • The approach demonstrates generalization across in-domain, out-of-distribution, multilingual, and low-resource MT settings, while analyses examine reward choice, reasoning patterns, training dynamics, and model adaptability.
  • Experiments show that pure RL primarily drives MT improvements, with minimal dependence on forced reasoning or output length.

2 Related Work

Prior R1-Zero research shows that pure RL can induce emergent reasoning when correctness and format are verifiable, but machine translation lacks a single rule-checkable correct output. Existing MT reasoning approaches therefore rely on structured data, search, or staged prompting, and pure RL remains insufficiently explored.

  • DeepSeek-R1-Zero showed that formatting and final-answer correctness rewards can induce self-emergent reasoning in mathematical and coding tasks.
  • R1-Zero-like RL has been replicated or extended in open-source mathematical, logic-game, and vision reasoning settings.
  • Machine translation remains an open application because output quality is not rule-based and is difficult to validate automatically.
  • Existing MT reasoning methods often require manually designed or synthetic Chain-of-Thought data, complex MCTS search, or explicit multi-stage prompting.
  • Reported performance from these methods often lags behind state-of-the-art open-source or proprietary models.

3 Method

MT-R1-Zero trains translation models with GRPO using a rule-metric mixed reward that combines format checking with lexical and semantic translation-quality metrics. The method supports multiple reward configurations and computes group-relative advantages to optimize the policy.

  • Reward design: The rule-metric mixed reward combines a format reward with a translation-quality metric reward within GRPO training.Format checking handles output structure, while metric evaluation addresses the absence of a single correct translation.
  • Reward design: The format reward requires reasoning inside <think></think> tags and the final translation inside <translate></translate> tags.Correct formatting receives 1, while incorrect formatting receives -1.
  • Metric rewards: Reward-Lex evaluates lexical overlap with reference translations using BLEU, while Reward-Sem uses COMETKiwi to assess meaning preservation.COMETKiwi requires the source sentence and generated translation, whereas BLEU compares the translation with a human-written reference.
  • Metric rewards: Reward-Mix adds lexical and semantic rewards to capture both lexical fidelity and semantic adequacy.The selected configuration determines whether Smetric uses BLEU, COMETKiwi, or their sum.
  • Reward computation: The final reward skips metric evaluation for invalid formats and otherwise adds the metric reward to the format reward.When the format is correct, the final reward becomes r = 1 + Smetric; incorrect formats receive a fixed penalty.
  • RL optimization: GRPO samples a group of candidate translations and computes each candidate’s advantage from its reward relative to the group mean and standard deviation.The objective uses PPO clipping and a KL-divergence penalty to constrain policy updates relative to the reference policy.

4 Experiments

Experiments evaluate MT-R1-Zero on in-domain EN-ZH and ZH-EN benchmarks, plus OOD EN-JA, DE-EN, and DE-ZH tasks, against proprietary and open-source baselines. The 7B variants achieve strong in-domain semantic and average scores, while Sem and Mix generalize competitively to unseen language pairs.

  • Dataset and Benchmarks: 13,130 EN⇌ZH training examples from WMT 2017–2020 support evaluation on WMT 24 EN-ZH, WMT 23 ZH-EN, and three OOD directions.The OOD tasks are EN-JA, DE-EN document-level, and DE-ZH Flores-200.
  • Baselines: The evaluation compares MT-R1-Zero 3B and 7B variants with Claude-3.5-Sonnet, GPT-4o, Gemini-1.5-Pro, Qwen2.5, LLaMA-3.1, and TowerInstruct baselines.Baselines include both proprietary and advanced open-source models.
  • Training Configuration: Training uses GRPO with 8 rollouts per prompt, learning rate 5e-7, temperature 1.0, maximum response length 1024 tokens, and β=0.The models are Qwen2.5-base 3B and 7B variants trained for one epoch on four NVIDIA H800 GPUs.
  • In-Domain Performance: 62.25 average score for MT-R1-Zero-7B-Mix on EN-ZH surpasses GPT-4o (61.86) and Qwen2.5-72B (61.77).MT-R1-Zero-7B-Sem reaches 72.07 COMETKiwi and 79.37 XCOMET on EN-ZH, the best semantic-level performance reported there.
  • In-Domain Performance: 69.75 COMETKiwi for MT-R1-Zero-3B-Sem on EN-ZH exceeds LLaMA-3.1-70B by approximately 1.7 points and Qwen2.5-72B by more than 0.7 points.On ZH-EN, MT-R1-Zero-7B-Sem scores 71.66 COMETKiwi, comparable to Claude-3.5-Sonnet (71.69) and GPT-4o (71.63).
  • Out-of-Distribution Performance: 82.43 average XCOMET for MT-R1-Zero-7B-Sem is the highest across OOD tasks, while MT-R1-Zero-7B-Mix scores 81.96 and leads DE-ZH with 88.69.Sem leads EN-JA with 71.95 and DE-EN with 87.68; both variants outperform same-size base-model baselines by at least 3.4 points.

5 Key Findings and Insight

The findings show that reward selection shapes translation behavior, while RL training produces evolving reasoning patterns and different base-model adaptability. Across training, models develop changing response lengths, semantic reasoning, and target-language internal thought.

  • Reward Metric Selection: Reward metric choice critically shapes optimization targets and translation style.Reward-Lex favors lexical quality, Reward-Sem favors semantic quality, and Reward-Mix balances both objectives.
  • Reward Metric Selection: Reward-Lex maximizes BLEU, Reward-Sem maximizes COMETKiwi, and Reward-Mix improves both metrics with a COMETKiwi trade-off.XCOMET evaluations consistently support the stronger semantic performance of the Sem and Mix variants.
  • Reward Metric Selection: Different rewards produce distinct translation styles, ranging from literal n-gram-focused outputs to semantically faithful or balanced translations.These stylistic differences reflect the quality dimension emphasized by the reward.
  • Training Dynamics: Response length initially declines rapidly and then gradually increases as training progresses.The decline reflects format mastery and more efficient translation, while the later increase aligns with richer semantic analysis and deeper contextual reasoning.
  • Emergent Reasoning: Diverse reasoning patterns emerge autonomously, including structured decomposition, colloquial processing, and dynamic transitions in internal reasoning language.During out-of-distribution testing, reasoning shifts from English toward the target language without direct supervision on reasoning language.
  • LLM Adaptability: Qwen shows the highest compatibility with format learning and reasoning generation, while LLaMA and Tower adapt more slowly and tend toward format hacking.Qwen models nevertheless sometimes regress to simplistic one-sentence outputs, underscoring instability during exploration.

6 Analysis and Ablation

Ablation and generalization analyses indicate that MT-R1-Zero’s gains primarily come from the RL process, while the framework remains effective across multilingual and low-resource translation settings.

  • KL Penalty: Removing the KL penalty causes response length to fluctuate and trend upward after an initial drop during training.The comparison uses β = 0 without KL and β = 0.01 with KL in MT-R1-Zero-7B-Sem.
  • Thinking Ablation: RL without explicit thinking performs comparably to RL with thinking across in-domain and out-of-distribution tasks, while both outperform SFT.The same pattern appears on the DRT literature-translation benchmark, where RL variants show marginal differences but significant gains over SFT.
  • Thinking Ablation: The principal performance improvements arise from the RL framework itself rather than explicit reasoning steps or verbosity.The <think> tag may facilitate emergent reasoning patterns, but it is not necessary for comparable translation gains.
  • Multilingual Generalization: Multilingual training yields consistent translation-quality improvements across languages with diverse resource levels, including Icelandic and Norwegian.The learning curves support effectiveness in multilingual and low-resource settings.

7 Conclusion

MT-R1-Zero adapts the R1-Zero RL paradigm to machine translation through a rule-metric mixed reward mechanism. The framework improves translation quality, generalizes across out-of-distribution and multilingual settings, and attributes gains primarily to RL rather than reasoning steps or verbosity.

  • Contribution: MT-R1-Zero is the first successful adaptation of the R1-Zero RL framework to MT using format enforcement combined with quality metrics.The method uses a novel rule-metric mixed reward mechanism.
  • Results: The framework achieves leading results across multiple benchmarks, with 3B models competing with larger open-source models and 7B models matching advanced proprietary models.It also demonstrates strong out-of-distribution generalization and multilingual applicability.
  • Insights: Reward choice, LLM adaptability, training dynamics, and emergent reasoning patterns are central factors in adapting R1-Zero to MT.The analysis identifies these factors as practical design considerations for effective adaptation.
  • Implications: The results establish R1-Zero as a viable paradigm for advancing machine translation and suggest broader potential for RL in language-processing tasks.The broader claim is presented as potential beyond translation rather than as a demonstrated result for every language task.

Limitations

MT-R1-Zero’s emergent reasoning is diverse but lacks the sophisticated iterative self-correction observed in mathematical reasoning tasks. The authors identify task-specific reasoning design, cold-start SFT, broader architectural evaluation, and improved rewards as future directions.

  • MT-R1-Zero generates diverse emergent reasoning patterns but does not achieve sophisticated iterative self-correction.The authors contrast this behavior with mathematical reasoning tasks using similar RL or R1-like methods.
  • The gap may reflect fundamental differences between translation and mathematical reasoning tasks or the need for specialized translation-task design.
  • A task-specific cold-start dataset for SFT before RL is proposed as one promising direction, although it would depart from the pure RL paradigm.
  • Future work includes deeper MT-specific reasoning, broader architectural adaptability studies, improved reward mechanisms, and applications in specialized domains.The cited domains include law and healthcare.

A Evaluation Details

Evaluation uses locally deployed open-source models with sampling-based decoding. Generation is constrained by a low temperature, nucleus-sampling threshold, and a 1024-token maximum length.

  • Open-source models were evaluated locally using vLLM or HuggingFace implementations.
  • Sampling decoding used temperature 0.2 and top_p 0.95.
  • The maximum generation length was capped at 1024 tokens.
  • Translation prompts followed Table 5, with specific chat templates applied when needed.

B SFT Training Details

The supplementary materials specify the SFT baseline setup, translation prompt variables, evaluation tables, dataset statistics, and multilingual training/test data organization. They also include figures for multilingual training progression and qualitative reasoning patterns.

  • SFT Training Details: The SFT baseline used the same base-model architecture and parallel dataset as the corresponding RL experiments.The dataset contained 13,130 filtered WMT 2017–2020 examples.
  • SFT Training Details: SFT was performed with LLaMA-Factory on 8 NVIDIA H800 80G GPUs for 2 epochs at a 5e-6 learning rate.
  • Evaluation and Data Materials: The translation prompt defines target language, source language, and source test sentence placeholders.
  • Evaluation and Data Materials: Tables 6 and 7 report out-of-distribution comparisons on EN-JA, DE-EN (Doc), and DE-ZH using COMETKiwi and BLEU.
  • Evaluation and Data Materials: Tables 8 and 9 provide data statistics for EN⇌ZH experiments and multilingual language pairs classified by resource level.
  • Additional Analyses: Figures 11 and 12 show multilingual training progression with reference-free XCOMET and qualitative examples of diverse thinking patterns.
Loading 2504.10160v1…