PaperlayerView exact revision on arXiv

Source-linked AI summary

A Rising Tide Lifts All Boats: MTQE Rewards for Idioms Improve General Translation Quality

Ishika Agarwal, Zhenlin He, Dhruva Patil, Dilek Hakkani-Tür

arXiv:2601.06307v1cs.CL

TL;DR

Non-compositional expressions remain difficult to translate because their meanings are culturally embedded, context dependent, and not recoverable from individual words. The paper uses GRPO-style fine-tuning with MTQE rewards on Chinese and Hindi idioms, finding improvements in idiomatic, general, and cross-lingual translation. These results support MTQE rewards as a way to improve translation while retaining broader language-model abilities, although performance is bounded by MTQE quality and training is data- and computation-intensive.

  • Problem

    Non-compositional expressions create a translation gap because their meanings are not derivable from individual words and can differ from literal translations.

  • Method

    The paper applies GRPO-style fine-tuning to language models, using MTQE models as reward models for idiom translation.

  • Results

    13.67 absolute points improve idiom translation, 8.39 improve non-idiomatic translation, and 5.73 improve cross-lingual translation.

  • Takeaways & Limitations

    MTQE rewards improve semantic relationships in multilingual language models without compromising current language-model abilities and support meaningful generalization to other languages.

  • Takeaways & Limitations

    Performance is bounded by MTQE models, which require substantial parallel data, while reinforcement learning is expensive.

Abstract

from arXiv · show

Non-compositional expressions (e.g., idioms, proverbs, and metaphors) pose significant challenges for neural machine translation systems because their meanings cannot be derived from individual words alone. These expressions encode rich, cultural meaning, and have both figurative and literal meanings, making accurate translation difficult. Because models are fairly good at translating compositional text, we investigate GRPO-style fine-tuning using Machine Translation Quality Estimation (MTQE) models as reward functions to train models to better translate idioms. Using Chinese and Hindi idiom datasets, we find that idiom translation abilities improve by ~14 points, general, non-idiomatic translation implicitly improves by ~8 points, and cross-lingual translation abilities (trained on one language, evaluated on another) improves by ~6 points. Overall, our work quantifies the non-compositional translation gap and offers insights for developing LLMs with stronger cross-cultural and figurative language understanding.

1 Introduction

Multilingual language modeling has grown because multilingual conversation improves accessibility, but alignment problems make multilingual modeling difficult.

  • Multilingual language modeling has grown significantly because it makes LLMs accessible for conversation across languages.
  • Alignment problems make multilingual language modeling difficult.
  • Language-specific knowledge is one category of multilingual alignment problem.

1. Meaning cannot be derived from words

The examples illustrate that some expressions have no direct equivalent across languages, even when their component words can be translated literally.

  • Some expressions have an equivalent translation across languages.
  • Other expressions have no mapping across languages.
  • “The devil is beating his wife” and “Sunny showers” exemplify expressions requiring cross-lingual interpretation.

2. Idioms cannot be literally translated

Idioms challenge translation because their meanings are non-compositional, culturally embedded, context dependent, and often lack semantically equivalent translations. The paper proposes prompting and GRPO fine-tuning with MTQE rewards, reporting gains for idiomatic, general, and cross-lingual translation.

  • 2. Idioms cannot be literally translated: Idiomatic meanings cannot be derived from constituent words and can diverge sharply from literal translations.
  • 2. Idioms cannot be literally translated: Idioms often lack semantically equivalent translations, causing literal decoding to erase figurative intent.
  • 2. Idioms cannot be literally translated: Context dependence gives non-compositional phrases different meanings and contributes to a broader cultural translation gap.
  • 2. Idioms cannot be literally translated: The paper combines a three-step cultural-context prompting pipeline with GRPO-style fine-tuning using MTQE models as reward models.
  • 2. Idioms cannot be literally translated: Chinese and Hindi idiom datasets are evaluated with semantic and n-gram similarity measures on Qwen’s 3B and Llama’s 8B models.
  • 2. Idioms cannot be literally translated: ∼14 points, ∼8 points, and ∼6 points improve idiomatic, general non-idiomatic, and cross-lingual translation quality, respectively.

2 Background

MTQE models estimate machine-translation quality through semantic equivalence, using either source and translation alone or an additional reference. The paper motivates using these models to distill non-compositional translation knowledge into language models.

  • 2 Background: MTQE models estimate the quality of machine-translated text.
  • 2 Background: Reference-free MTQE models score semantic equivalence from a source text and its translation.
  • 2 Background: Reference-based MTQE models score semantic equivalence using source, translation, and a human-annotated reference.
  • 2 Background: The paper posits that MTQE training data teaches models to represent non-compositional language, enabling weak distillation into translation models.
  • 2 Background: Prior work studies idiom detection, generation, and datasets, but translation remains a problem, especially for open-source language models.

3 Improving Non-Compositional Translation

The paper evaluates GRPO fine-tuning with MTQE-based reward settings for Chinese–English and Hindi–English idiom translation, alongside a training-free prompting method. The reward designs encourage semantic equivalence, discourage literal overlap, balance both objectives, or use a reference translation.

  • Data and evaluation: Chinese and Hindi idiom datasets support evaluation of the proposed translation methods, with deterministic train–test splits for direct baseline comparison.The Chinese data contain 1,623 valid pairs split into 1,000 training and 623 test examples; the Hindi data contain 1,000 pairs split into 800 training and 200 test examples.
  • GRPO-QE methods: GRPO fine-tuning uses MTQE models as reward models to improve an LLM’s translation abilities.The reward evaluates generated translations, with reference-free and reference-based MTQE variants explored.
  • GRPO-QE methods: QE-Positive rewards semantic equivalence between the source idiom and the generated translation.The MTQE model receives the source idiom and machine translation, denoted QEpos(idiom, mt).
  • GRPO-QE methods: QE-Negative applies a negative reward for similarity between the idiom’s literal meaning and the generated translation, discouraging literal translations.Both inputs are English in this setting, so MTQE is used as a semantic similarity metric rather than for cross-lingual evaluation.
  • GRPO-QE methods: QE-Constrained combines positive semantic-equivalence reward with negative literal-similarity reward, while QE-DA adds a ground-truth reference translation.The joint reward is QEpos(idiom, mt) − QEneg(literal, mt), and QE-DA guides outputs toward a specific target translation.
  • Training-free method: A training-free three-step prompting pipeline explains figurative meaning, derives literal semantics, and produces a natural idiomatic translation without additional optimization.The method uses explicit reasoning decomposition and prompt engineering to mitigate literalism.

4 Experiments

Experiments compare GRPO-based MTQE-reward fine-tuning with translation models, prompting baselines, and supervised fine-tuning across idiomatic, non-idiomatic, and transfer settings. GRPO improves idiom translation, preserves or improves regular translation, transfers across languages, and remains robust across reward designs.

  • Baselines: Prompting baselines are inconsistent across model families, motivating fine-tuning: LIA beats TrainingFree by 1.65 points on Qwen, whereas TrainingFree beats LIA by 2.18 on Llama.The baseline comparison indicates that prompting performance depends on the underlying model.
  • Idiom Translation: GRPO-based methods outperform base and supervised fine-tuned models on Chinese and Hindi idiom translation.Across the evaluated idiom settings, RL models deliver an absolute improvement of 13.67 points in idiom-translation ability.
  • Non-Idiomatic Performance: 8.39 points is the average improvement of GRPO models over base models on mostly non-idiomatic translation, while translation models remain 7.90 points ahead of the best GRPO model.Performance does not deteriorate when models are trained on idiomatic data; it also improves over SFT by an average of 25.07 points.
  • Effects of Language-Specific Training: 8.62 absolute points is the overall transfer improvement showing that idiom training in one language does not hinder performance in another.Chinese-trained models improve Hindi idiom translation by 8.04(Qwen)/15.73(Llama) points over base models, while Hindi-trained models improve Chinese by 7.73(Qwen)/12.70(Llama).
  • Effect of Reward Model: MTQE reward variants show no noticeable individual effects, and the least expensive QE-Positive reward does not lag far behind the most accurate QE-DA reward.QE-DA requires ground-truth idiom translations, whereas QE-Positive requires only an input idiom.

5 Conclusion

The paper presents GRPO-based fine-tuning with MTQE rewards for non-compositional translation and reports improvements in idiom, non-idiomatic, and cross-lingual translation.

  • 13.67 absolute points: idiom translation abilities increased across languages and architectures over base models.
  • 8.39 absolute points: non-idiomatic translation abilities improved implicitly.
  • 5.73 absolute points: cross-lingual translation abilities improved.
  • MTQE rewards distill multilingual language embedding capabilities into language models.The reported effects include improved semantic relationships, reliable preservation of existing abilities, and meaningful generalization to other languages.

6 Limitations

The method is bounded by MTQE quality and requires substantial data and computation, limiting accessibility despite MTQE language coverage and human-preference alignment.

  • MTQE models impose an upper bound because GRPO-trained translation models can only be as good as their reward models.
  • MTQE models require substantial parallel data for training.
  • 6-12 hours on 4 NVIDIA H100s: training remained computationally inaccessible even with small models and datasets.

A Prompts

Tables 9, 10, and 11 provide the prompts for the TrainingFree method described in Section 3.2.

  • Tables 9, 10, and 11 contain the prompts used by the TrainingFree method.The method is described in Section 3.2.

B Licenses

The paper documents dataset verification practices and the licenses governing its datasets and models.

  • The PETCI, Open Subtitles, NLLB, Command-R, Qwen, and Llama resources are distributed under specified open or model-specific licenses.The passage lists Creative Commons, GNU GPL, Apache 2.0, and Llama licensing terms.
  • Human annotators verified dataset validity using guidelines to avoid overly literal or ambiguous idiomatic data.The authors state that Chinese- or Hindi-speaking authors conducted unbiased verification.

D Usage of AI Assistants

AI assistants were used for limited figure-plotting support and small text revisions, while structured prompts guided idiom explanation and translation steps.

  • AI assistants helped write plotting code after the authors processed the data themselves and specified the desired figure format and color scheme.
  • AI assistants revised small portions of the title, abstract, and figure captions for clarity and brevity.
  • The authors report that no AI assistants were used for paper writing, code writing, results analysis, ideation, or this section.
  • The training-free structured prompting approach uses one prompt to elicit an idiomatic explanation and separate prompts for literal semantics and natural idiomatic translation.
  • The natural idiomatic translation prompt takes an idiom explanation and a literal word-by-word gloss as inputs.
Loading 2601.06307v1…