PaperlayerView exact revision on arXiv

Source-linked AI summary

Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation

Yifeng Liu, Siqi Ouyang, Yatish Hosmane Revanasiddappa, Lei Li

arXiv:2603.13045v1cs.CL

TL;DR

Low-resource translation remains limited because existing post-training methods depend on scarce high-quality parallel or preference data. WALAR continually trains multilingual LLMs with monolingual text using quality estimation, word alignment, and language alignment rewards. Across FLORES-101, it substantially improves translation quality and language consistency across more than 1,400 directions, including unseen languages during training.

  • Problem

    Low-resource translation remains markedly weaker, while existing post-training methods rely on scarce or unavailable high-quality parallel or preference data.

  • Method

    WALAR uses monolingual source text and combines source-based quality estimation, word alignment, and language alignment in reinforcement learning.

  • Results

    WALAR substantially improves translation quality and language consistency across over 1,400 FLORES-101 language directions and generalizes to unseen languages during training.

  • Takeaways & Limitations

    Alignment-enhanced rewards mitigate QE holes and support multilingual translation improvements across low-resource, English-centric, and unseen language directions.

  • Takeaways & Limitations

    Language identification can struggle with code-switching translations, while neural metrics may be susceptible to over-optimization and metric inflation.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable capability in machine translation on high-resource language pairs, yet their performance on low-resource translation still lags behind. Existing post-training methods rely heavily on high-quality parallel data, which are often scarce or unavailable for low-resource languages. In this paper, we introduce WALAR, a reinforcement training method using only monolingual text to elevate LLMs' translation capabilities on massive low-resource languages while retaining their performance on high-resource languages. Our key insight is based on the observation of failure modes (or "holes") in existing source-based multilingual quality estimation (QE) models. Reinforcement learning (RL) using these QE models tends to amplify such holes, resulting in poorer multilingual LLMs. We develop techniques including word alignment and language alignment to mitigate such holes in WALAR's reward for RL training. We continually trained an LLM supporting translation of 101 languages using WALAR. The experiments show that our new model outperforms LLaMAX, one of the strongest open-source multilingual LLMs by a large margin on 1400 language directions on Flores-101 dataset.

1. Introduction

LLM translation remains substantially weaker for low-resource languages, while existing post-training methods depend on scarce high-quality parallel or preference data. WALAR uses monolingual data and alignment-enhanced rewards to mitigate QE failure modes and improve multilingual translation.

  • Low-resource translation remains markedly weaker, while prior post-training methods rely on scarce high-quality parallel or preference data.
  • QE-based rewards can miss over-translation, under-translation, and wrong-language outputs, allowing RL to amplify reward-hacking holes.A reported failure is repeating the source sentence while receiving a perfect QE score.
  • WALAR combines source-based quality estimation with word alignment and language alignment within GRPO to train multilingual LLMs using monolingual source text.Word alignment encourages coverage, while language alignment targets the desired output language.
  • Three trained LLMs outperform the strongest prior same-size LLM across 1,414 FLORES-101 language directions, with gains also extending to unseen directions.

2. Related Work

Prior reinforcement-learning approaches improve translation with reference-based or source-based rewards, but reference-dependent methods require translation data. Multilingual LLM work has expanded supported languages while leaving a persistent high-resource versus low-resource performance gap.

  • Reference-based RL rewards compare generated translations with references, whereas source-based rewards evaluate translations without reference translations.
  • Existing RL translation approaches rely heavily on reference translation data for their reward signals.
  • Multilingual LLMs have improved on high-resource languages, but a significant performance gap with low-resource languages remains.

3. Proposed Method

WALAR continually trains a pretrained LLM with monolingual multilingual text using GRPO and a hybrid reward combining quality estimation, word alignment, and language alignment. The design addresses reward hacking by constraining translation coverage and target-language conformity while retaining MetricX as the base quality signal.

  • Overall Framework: WALAR prompts several translation rollouts from each monolingual sentence, evaluates them with three reward components, and iteratively trains the LLM with GRPO.The components are language alignment, quality estimation, and word alignment.
  • Quality Estimation Reward: MetricX provides the source-based quality-estimation reward, but quality estimation alone can assign high rewards to degenerate hypotheses.The source sentence and generated hypothesis are concatenated and scored as a scalar reward.
  • Word Alignment Reward: Word alignment scores whether target words cover the source without omissions or unsupported additions, using alignment precision and recall combined as an F1 score.The score penalizes over-translation through lower precision and under-translation through lower recall.
  • Language Alignment Reward: Language alignment verifies that the generated translation matches the desired target language, assigning positive reward only when the detected language is as expected.MaskLID filters code-switching segments before GlotLID performs language detection.
  • Overall Reward: The overall reward assigns -25 when language alignment fails; otherwise it combines MetricX with α times the word-alignment score, using α = 20.The masked translation y′ is used for the word-alignment term, and α is analyzed as a scaling hyperparameter.
  • RL Training: GRPO samples G candidate sequences, computes their WALAR rewards and relative advantages, then updates the policy with clipping and a KL-divergence penalty.The clipping threshold and KL penalty are controlled by ϵ and β.

4. Experiments

Experiments evaluate WALAR across FLORES-101 language directions, model families, translation-quality metrics, LLM judging, and language consistency. WALAR improves translation quality and consistency, with especially large gains for low-resource directions.

  • Experimental Setup: Training data uses 22 source languages from WMT News Crawl, selects directions with spBLEU between 1 and 20, and samples 250 instances per direction.NER, length clipping, and data decontamination filter the monolingual training data.
  • Experimental Setup: The study evaluates multilingual encoder-decoder and decoder-only models on FLORES-101 using multiple translation-quality metrics across 1,414 language directions.The evaluation includes NLLB-200-1.3B and several multilingual LLMs, with spBLEU, XCOMET-XL, MetricX, and Gemini 3 Flash.
  • Main Results: WALAR produces significant average improvements across spBLEU, xCOMET*, and MetricX* for Qwen3-8B, Translategemma-4B-it, and LLaMAX3-8B-Alpaca.The before-and-after comparison supports generalization across different model families.
  • Main Results: Within the LLaMAX family, WALAR raises xCOMET* from 54.00 to 60.31 for Swahili-X and from 68.66 to 76.42 for English-X translation.The reported gains cover both low-resource-centric and English-centric translation.
  • Main Results: Under LLM-as-a-Judge evaluation, WALAR-trained LLaMAX3-8B-Alpaca increases the average score from 57.25 to 67.03 across over 1,400 language directions.The WALAR-trained model outperforms its baseline across all evaluated directions, and its average exceeds 66 on the judging rubric.
  • Main Results: WALAR consistently improves language consistency across evaluated directions, with LLaMAX3-8B-Alpaca reaching the highest LCR and Swahili increasing from 83% to nearly 100%.LCR measures whether outputs are identified as being in the desired target language.

5. Analysis

WALAR analysis identifies reward-hacking failure modes in QE-based reinforcement learning and shows that word and language alignment mitigate them. Ablations, human evaluation, and unseen-language tests support WALAR’s translation improvements and cross-lingual generalization.

  • 5.1. Holes in Machine Translation Metrics: QE-only rewards produce self-generated references, non-translation, over-translation, under-translation, and wrong-language translation during training.These failure modes arise when models exploit weaknesses in unreliable reward signals.
  • 5.1. Holes in Machine Translation Metrics: Self-generated references repeat the hypothesis as a pseudo-reference, activating reference-based QE evaluation and yielding a high score.The behavior is attributed to MetricX and xCOMET’s hybrid source-based and reference-based design.
  • 5.1. Holes in Machine Translation Metrics: 92.43% wrong-language output occurs with QE-only reward, while language alignment fixes this issue; WALAR uniquely keeps translation length close to the reference.Word alignment is therefore critical for preventing omission and over-generation.
  • 5.2. Ablation Study: WALAR achieves the best spBLEU and xCOMET* performance, whereas QE-only performs worst and language alignment alone improves xCOMET* but degrades spBLEU through over-translation.SpBLEU-filtered language directions also outperform training on all language directions.
  • 5.3. Effects of Hyperparameters: Increasing α from 0 to 20 raises spBLEU from 12.88 to 19.71 while MetricX* and xCOMET* decline; α = 25 performs worst across all metrics.The experiments select α = 20 and prioritize spBLEU for model selection because neural metrics may be over-optimized.
  • 5.4. Human Evaluation: WALAR is preferred in 42% of Az-Pt cases and 51% of En-Kn cases, with comparable quality in 34% and 39%, respectively.These human evaluations corroborate improved translation quality on the tested language pairs.
  • 5.5. Generalization of WALAR: WALAR improves observed language directions and generalizes strongly to unseen target languages.The reported transfer may reduce the parallel data and language-direction requirements for training multilingual models.

6. Conclusion

The paper presents WALAR, a reinforcement training method that combines quality estimation, word alignment, and language alignment rewards for low-resource multilingual translation. Experiments across FLORES-101 show improved translation quality and language consistency, supported by LLM-as-a-Judge and human evaluation, with generalization to unseen languages.

  • Conclusion: WALAR integrates quality estimation, word alignment, and language alignment into a reinforcement-training reward for multilingual translation.The method targets low-resource languages using the paper’s reward design.
  • Conclusion: Experiments across 100 languages and over 1400 FLORES-101 language directions show substantial improvements in translation quality and language consistency.The paper also reports corroboration from LLM-as-a-Judge and human evaluation.
  • Conclusion: The analysis identifies holes in current machine-translation metrics and reports WALAR generalization to languages unseen during training.

A. Data Curation

The data-curation pipeline uses WMT News Crawl monolingual data, removes contamination, and filters source-language samples by length and named-entity density.

  • Data Curation: Monolingual data come from WMT News Crawl and undergo decontamination and source-language filtering.Filtering includes length-based and named-entity-based stages.
  • Data Decontamination: Decontamination uses an 8-gram search against FLORES-101 devtest data and marks samples contaminated when matching exceeds 70% of target tokens.
  • Length-based Filtering: Length-based filtering retains samples within empirically determined language-specific token-length thresholds.The thresholds are based on FLORES-101 token-length distributions and reported in Table 5.
  • NER-based Filtering: NER-based filtering excludes samples where named entities exceed 60% of total token length for English, Hindi, Arabic, and Turkish.

B. Evaluation Details

Evaluation uses BenchMAX across models and language directions with specified decoding, prompting, and LLM-as-a-Judge procedures. The setup combines greedy decoding for decoder-only models with beam search for NLLB-200-1.3B.

  • Evaluation Setup: BenchMAX is used to evaluate all models and language directions.
  • Decoding: Decoder-only LLMs use greedy decoding, while NLLB-200-1.3B uses beam search with beam size 5 and length penalty 0.6.
  • Prompting: LLaMAX3-8B-Alpaca uses the original-work prompt consistently for training and evaluation.The prompt translates sentences from a specified source language to a specified target language.
  • LLM-as-a-Judge: LLM-as-a-Judge scores translations from 0 to 100 against a human reference using an ESA-like prompt.The judge is instructed to return only a whole-number score.

D. Additional Results on FLORES-101

Table 6 reports the average rank of each multilingual LLM on the FLORES-101 test set, including results for seven representative languages.

  • Table 6 reports each model’s average rank on the FLORES-101 test set.The passage identifies average rank as the reported comparison measure.
  • Results for 7 representative languages are shown in the table.

E. More cases of Holes in Machine Translation Metrics

Additional cases show that quality-estimation holes are varied and can cause reward hacking, while WALAR is illustrated improving translation quality across several language directions.

  • More cases of Holes in Machine Translation Metrics: MetricX failure cases across Figures 7–10 show versatile holes that lead to reward hacking during reinforcement training.The cases involve English–Spanish, English–Polish, English–Chinese, and French–German directions, with source-based MetricX scores and highlighted translation errors.
  • Qualitative Examples of WALAR: Figures 11–14 provide qualitative examples of WALAR improving translation quality relative to baselines across four language directions.The examples cover English–Xhosa, English–Chinese, Chinese–Swahili, and Chinese–French, with reference-based xCOMET scores.
  • Metric Annotations: The examples use source-based MetricX or reference-based xCOMET scores to annotate translation outputs.MetricX is shown for the hole cases, while xCOMET scores are provided for the WALAR improvement cases.
Loading 2603.13045v1…