PaperlayerView exact revision on arXiv

Source-linked AI summary

The Lay of the Land in Idiom-Translation Quality Estimation

Malik Marmonier, Benoît Sagot, Rachel Bawden

arXiv:2610.35261v1cs.CL

TL;DR

Whether QE systems can reliably recognize idiomatic translations, despite LLM progress in translating figurative language, remains unclear. The paper builds a parallel idiom dataset and benchmarks conventional QE metrics and LLM judges. Most systems favor literal translations, while only a few consistently prefer idiomatic ones and one reliably tracks intermediate degrees.

  • Problem

    Whether QE systems have kept pace with LLM progress on figurative translation and can score idiomatic renderings above faulty literal ones remains unclear.

    Source: §1 · p. 1

  • Method

    The authors build a parallel dataset of English idiom-bearing sentences with literal and idiomatic translations across nine target languages, then benchmark 11 conventional QE metrics and 10 GEMBA-style LLM judges.

    Source: §1 · p. 2

  • Results

    Most systems, especially older metrics and smaller models, favor literal translations, while only MetricX-24-Hybrid XL/XXL and Gemma-4-31B consistently prefer idiomatic ones; only Gemma-4-31B reliably ranks intermediate degrees.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    In domains rich in figurative language, higher standard QE scores might reward undesirable literalism.

    Source: §1 · p. 2

    The dataset uses English as its only source language and comparatively well-resourced target languages, so results may not transfer to other source languages or rarer, more culture-specific idioms.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Idioms have long been a challenge for machine translation, and while large language models (LLMs) have proven increasingly adept at handling them, whether quality estimation (QE) systems have kept pace and can reliably score idiomatic renderings of these expressions above faulty literal ones remains unclear. We introduce the first parallel dataset pairing idiom-bearing English sentences with both literal and idiomatic translations across nine target languages, augmented into multi-sentence segments and a graded literality continuum. Using this dataset, we benchmark eleven conventional QE metrics from the COMET and MetricX families and ten LLMs from the Qwen-3, Gemma-4 and OLMo-3 families used as GEMBA-style judges. We find that most systems, especially older metrics and smaller models, display a marked literality bias, rating literal translations of idioms above idiomatic ones. Higher parameter counts and newer training sets appear to reduce this bias to an extent; nevertheless, of all the benchmarked systems, only MetricX-24-Hybrid (XL, XXL) and Gemma-4-31B consistently rate idiomatic translations of idioms above literal ones, and only Gemma-4-31B reliably ranks intermediate degrees of idiomaticity and literality. Our findings caution against deploying standard QE metrics in domains rich in figurative language, where higher scores might be rewarding undesirable literalism.

1 Introduction

Idioms often require paraphrase or cultural equivalence rather than literal translation, making them a longstanding MT challenge. This study tests whether QE systems can distinguish idiomatic translations from faulty literal ones and finds that only a few do so consistently.

1 Introduction

  • Idioms generally require paraphrase or cultural equivalence because their meanings are not usually recoverable through literal translation.This makes idioms a longstanding challenge for machine translation.Source: §1 · p. 1
  • Whether QE techniques have kept pace with LLM improvements in translating figurative language remains unclear.The authors frame idiom translation as a test of QE quality estimation’s ability to assess figurative language.Source: §1 · p. 1
  • QE is reference-free, and its growing roles in translation workflows motivate understanding how dominant systems behave on idioms.The paper notes applications including data filtering and quality-aware decoding.Source: §1 · pp. 1–2
  • Only MetricX-24-Hybrid XL/XXL and Gemma-4-31B consistently prefer idiomatic translations, and only Gemma-4-31B reliably ranks intermediate idiomaticity levels.Most systems, especially older metrics and smaller models, rate literal translations above idiomatic ones; larger models and newer training sets reduce this bias to an extent.Source: §1 · p. 2
  • The study introduces a parallel dataset pairing English idiom-bearing sentences with literal and idiomatic translations across nine target languages, and benchmarks 11 QE metrics and 10 LLM judges.The LLMs are used in a GEMBA-style judging regime.Source: §1 · p. 2

2 Methods

The methods introduce a parallel English-to-nine-language idiom dataset and use it to compare conventional QE systems and LLM judges. The dataset includes single-sentence literal/idiomatic pairs, augmented multi-sentence segments, and graded literality variants.

2.1 Idiom Dataset

  • The dataset pairs English idiom-bearing sentences with literal and fluent idiomatic translations across nine target languages, addressing the absence of corpora systematically pairing both translation types.Its 163 base triplets were manually curated, with target-language idiom equivalents required to differ from literal renderings; the resulting set is not strictly multiparallel.Source: §2.1 · p. 2
  • For each target language, randomly concatenating base triplets of lengths 2–5 produced 300 unique combinations per length, or 1,200 augmented segments.Table 2 illustrates augmented segments at L = 3.Source: §2.1 · p. 2, Table 2 · p. 4
  • The literality gradient uses three-triplet instances with four target variants containing 0, 1, 2, or 3 literal sentences, corresponding to 0%, 33%, 66%, and 100% literality.The variants test sensitivity to graded rather than strictly binary changes in idiomaticity.Source: §2.1 · pp. 2–3

2.2 QE System Query

  • Systems scored literal and idiomatic dataset translations in a GEMBA-style regime, after which the authors compared their scores.For LLM judges, prompts supplied an English source idiom and candidate translation and requested a score from 0 to 100.Source: §2.2 · p. 3, §2.2.2 · p. 4

2.2.1 Conventional QE Systems

  • The conventional benchmark includes 11 COMET- and MetricX-family systems, including the MetricX-24-Hybrid models trained additionally on synthetic data for underrepresented failure modes.The tested families include COMET-Kiwi, XCOMET, MetricX-23-QE, and MetricX-24-Hybrid.Source: §2.2.1 · p. 3

2.2.2 LLMs

  • Ten Qwen-3, Gemma-4, and OLMo-3 models served as LLM judges, with zero-shot DA-style prompts requesting scores from 0 (“no meaning preserved”) to 100 (“perfect meaning and grammar”).Qwen-3 and Gemma-4 were queried with reasoning mode on and off.Source: §2.2.2 · p. 4

3 Results

The benchmark finds that preference for idiomatic translations varies with model family, scale, and training data. Only MetricX-24-Hybrid XL and XXL reliably reward idiomaticity among conventional metrics, while Gemma-4-31B also tracks graded idiomaticity reliably.

3 Results

  • The preference ratio compares mean scores for fully literal versus fully idiomatic translations, with MetricX’s error-based ratio inverted for readability.For the displayed ratios, values above 1.0 indicate literality bias, below 1.0 indicate idiomatic preference, and exactly 1.0 indicates neutral scoring.Source: §3 · pp. 4–5, Table 4 · p. 6
  • Only MetricX-24-Hybrid XL and XXL achieve reliable idiomatic sensitivity among conventional QE systems, with scale and finetuning-data composition both associated with preference shifts.MetricX-24-Hybrid reduces the predecessor’s literality bias despite matched parameter counts; scaling also shifts MetricX-24 and XCOMET toward idiomatic preference.Source: §3 · p. 5
  • Gemma-4-31B robustly prefers idiomatic translations across all target languages and is the only LLM reported to reliably penalize literality.Qwen-3-32B favors idiomatic translations in most, but not all, languages; OLMo-3-32B retains a weaker literality bias than its 7B counterpart and trails comparably sized Qwen and Gemma models.Source: §3 · p. 6
  • Gemma-4-31B tracks graded idiomaticity substantially better than MetricX-24-Hybrid-XXL, with τ above 0.900 for most languages versus moderate correlations for the metric on Arabic and French.MetricX-24-Hybrid-XXL records τ = 0.355 for Arabic and τ = 0.387 for French; intermediate mixtures distinguish systems that appear roughly on par at the binary extremes.Source: §3 · p. 7, Table 5 · p. 7

4 Related Work

Related work traces quality estimation from earlier confidence-estimation research through shared tasks and modern pretrained-model and LLM approaches. Prior studies examine QE bias and non-literal translation, while existing idiom datasets leave a gap for sentence-level paired translations across graded literality.

  • QE developed through WMT shared tasks, with earlier work framed as confidence estimation and later approaches grouped into handcrafted, pretrained-model, and LLM-based systems.Source: §4 · pp. 7–8
  • Zaranis et al. (2025) found gender bias in QE, while Tian et al. (2026) found traditional metrics correlated poorly with human preferences for non-literal translations.Source: §4 · p. 8
  • Earlier idiom datasets include idioms in context with idiomatic translations, dictionary-form idioms, and monolingual paraphrases; G-IdiomAlign adds dictionary-form literal translations.Source: §4 · p. 8
  • Unlike dictionary-form resources, this work aligns idiom-containing sentences with literal and idiomatic translations across a graded continuum.Source: §4 · p. 8

5 Conclusion

The paper introduces a parallel dataset of idiom translations and uses it to assess QE systems. Results show literality bias among smaller models and older metrics, with different strengths for binary judgments and graded distinctions.

  • The dataset pairs literal and idiomatic translations of idioms across nine languages for studying QE performance.Source: §5 · p. 8
  • Smaller models and older QE metrics exhibit marked literality bias, while increasing model scale reliably improves idiomatic sensitivity.Source: §5 · p. 8
  • The largest conventional QE metrics handle binary extremes, while GEMBA-regime LLMs uniquely rank intermediate literality and idiomaticity gradations accurately.Source: §5 · p. 8
  • Almheiri et al. (2026) recently compiled another dataset offering such annotations.Source: §5 · p. 8

Limitations

The study’s evidence is bounded by its English source language, comparatively well-resourced target languages, and dataset construction choices. Its model comparisons also omit proprietary LLMs and rely on limited prompt and checkpoint coverage.

  • English-only sourcing may overestimate performance on rarer, more culture-specific idioms and may not transfer to other source languages.Source: Limitations · p. 9
  • The nine target languages are comparatively well resourced, and triplets lacked formal validation and inter-annotator agreement measures.Source: Limitations · p. 9
  • Topically incoherent concatenated segments may restrict the dataset’s usefulness for downstream tasks beyond evaluation.Source: Limitations · p. 9
  • Only open-weight systems were evaluated, each LLM received one zero-shot DA-style prompt, and scale comparisons used few checkpoints with possible confounding.Source: Limitations · p. 9

Ethics Statement

The authors acknowledge environmental costs from large-scale LLM inference and explain that they dropped a planned synthetic-error analysis. They describe dataset and output releases, participation safeguards, and a caution about possible cultural or ethnic stereotypes in idioms.

  • The authors abandoned a planned synthetic-error sensitivity analysis because its compute cost and environmental impact seemed unjustified by its expected insight.Source: Ethics Statement · p. 9
  • Experiments used a cluster operating on largely decarbonized electricity, and the authors release LLM completions and QE scores for reuse.Source: Ethics Statement · p. 9
  • The dataset was assembled voluntarily without compensation or personal-data collection, and triplets were screened for offensive content despite possible stereotype connotations.Source: Ethics Statement · p. 9
  • The dataset is released under CC BY-SA 4.0, while outputs and scores use CC BY-NC-SA 4.0; its password-protected archive is intended to prevent web-crawl inclusion.Source: Ethics Statement · p. 9
  • The authors recommend human oversight when QE scores inform decisions about translated content or translators’ work, especially in figurative-language-rich domains.Source: Ethics Statement · p. 9

A GEMBA-Style Prompt Template

The GEMBA-style prompt asks a model to evaluate an English-to-target-language translation, with variants that either encourage or prohibit reasoning.

A GEMBA-Style Prompt Template

User Prompt Template (Reasoning Off)

The reasoning-off prompt requests an English-to-target-language translation score and restricts the response to the score inside XML tags.

User Prompt Template (Reasoning Off)

B Bootstrap Significance Test

A stratified bootstrap compares systems’ macro-preference ratios pairwise, with heatmap colors and asterisks communicating the direction and significance of differences.

B Bootstrap Significance Test

Related papers

Loading 2610.35261v1…