PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimating Machine Translation Difficulty

Lorenzo Proietti, Stefano Perrella, Vilém Zouhar, Roberto Navigli, Tom Kocmi

arXiv:2508.10175v2cs.CL

TL;DR

Near-perfect MT results make it difficult to distinguish leading systems, while broader general translation difficulty has not been systematically investigated. This paper formalizes difficulty estimation and evaluates it with a dedicated metric; dedicated models outperform heuristic and LLM-as-a-judge approaches, with Sentinel-src-24 reported as state of the art.

  • Problem

    Near-perfect MT scores make it difficult to distinguish top-performing models, and general translation difficulty has not been systematically investigated.

    Source: §1 · p. 1

  • Method

    The paper defines difficulty as expected translation quality, introduces a dedicated evaluation metric, and evaluates estimators and their use in selecting challenging benchmark samples.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Dedicated models significantly outperform heuristic-based and LLM-as-a-judge approaches, and Sentinel-src-24 is identified as state of the art.

    Source: §5.3 · p. 6, §7 · p. 9

  • Takeaways & Limitations

    Difficulty estimators identify samples where modern MT models underperform, and Sentinel-src-24 remains the top-performing method for creating difficult benchmarks.

    Source: §7 · p. 9

    The influence of target language on translation difficulty is only briefly discussed and not investigated directly.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Machine translation quality has steadily improved over the years, achieving near-perfect translations in recent benchmarks. These high-quality outputs make it difficult to distinguish between state-of-the-art models and to identify areas for future improvement. In this context, automatically identifying texts where machine translation systems struggle holds promise for developing more discriminative evaluations and guiding future research. In this work, we address this gap by formalizing the task of translation difficulty estimation, defining a text's difficulty based on the expected quality of its translations. We introduce a new metric to evaluate difficulty estimators and use it to assess both baselines and novel approaches. Finally, we demonstrate the practical utility of difficulty estimators by using them to construct more challenging benchmarks for machine translation. Our results show that dedicated models outperform both heuristic-based methods and LLM-as-a-judge approaches, with Sentinel-src achieving the best performance. Thus, we release two improved models for difficulty estimation, Sentinel-src-24 and Sentinel-src-25, which can be used to scan large collections of texts and select those most likely to challenge contemporary machine translation systems.

1 Introduction

Near-perfect MT results and the lack of systematic study of general translation difficulty motivate methods for identifying challenging source texts. The paper formalizes difficulty estimation, evaluates estimators, and uses them to build challenging benchmarks.

  • Near-perfect MT scores make easy test sets less useful for distinguishing top-performing models or assessing headroom for improvement.Source: §1 · p. 1
  • The paper defines translation difficulty estimation, introduces the Difficulty Estimation Correlation metric, and tests estimators by selecting challenging samples from large corpora.Source: §1 · p. 1
  • Dedicated difficulty estimators outperform heuristic-based methods and LLM-as-a-judge approaches; Sentinel-src is reported to outperform other methods.Source: §1 · p. 1
  • The authors train and publicly release Sentinel-src-24 and Sentinel-src-25 as improved difficulty estimators.Source: §1 · p. 2

2 Related Work

Prior work has studied human and machine translation difficulty using different proxies, while PreQuEL focused on predicting translation quality before generation. This paper distinguishes difficulty estimation, evaluates it with a dedicated metric across broader test data, and investigates challenging-benchmark construction.

  • Earlier work studied human translation difficulty through text complexity, translation time, eye movements, and proxies such as alignment distributions and decoder perplexity.Source: §2 · p. 2
  • Machine translation difficulty estimates have used linguistic heuristics, model confidence, target-aware quality-estimation features, artificial crowds, and embedding similarity.Source: §2 · p. 2
  • PreQuEL predicted translation quality before generation, but used a quality-estimation evaluation, two language directions, and translations from one MT model.Source: §2 · p. 2
  • The paper defines difficulty estimation as a distinct task with a dedicated metric and evaluates 11 language directions using 11 to 19 translations per segment from MT systems and human translators.Source: §2 · p. 2

3 The Difficulty Estimation Task

Translation difficulty is defined relative to a target language and translator, as the quality of a text’s translation; lower quality means greater difficulty. Difficulty estimators are evaluated by how well they rank texts, using correlations averaged across systems and languages.

  • The task recognizes that translation difficulty can depend on text features, translation direction, and translator or MT-system characteristics.Source: §3 · p. 2
  • Difficulty is relative to a target language and a particular human or automatic translator, rather than an absolute property of a text.Source: §3 · p. 2
  • Difficulty Estimation predicts translation quality from the source text without access to the translations being assessed.Source: §3 · p. 3
  • Difficulty Estimation Correlation averages Kendall’s τb rank correlations between predicted and ground-truth difficulty scores across models and languages.Source: §3 · p. 3
  • DEC uses Group-by-System correlations, comparing assessments across different source texts translated by the same MT system, then averaging across systems.Source: §3 · p. 3
  • Group-by-Item is unsuitable for difficulty estimators because they assign the same score to all translations of a given source.Source: §3 · p. 3

4 Methods for Difficulty Estimation

The paper evaluates heuristic, learned, LLM-as-a-Judge, and crowd-based estimators. These approaches range from source-text features and predictors to scores derived from generated translations; True Crowd serves only as an upper bound.

4.1 Heuristic-based estimators

  • Heuristic estimators use simple features including text length, word rarity, and dependency-tree height as a proxy for syntactic complexity.Source: §4.1 · pp. 3–4

4.2 Learned estimators

  • Learned approaches include PreCOMET models for sample usefulness and difficulty, alongside Sentinel-src, which predicts translation quality from the source alone.Source: §4.2 · p. 4

4.3 LLM-as-a-Judge

  • LLM-as-a-Judge prompts GPT-4o and CommandA to assess the proficiency needed to translate a text, optionally using target-language information.Source: §4.3 · p. 4

4.4 Crowd-based Estimators

  • Artificial Crowd averages reference-less metric quality scores across translations generated by three instruction-tuned LLMs and one encoder-decoder MT model.Source: §4.4 · p. 4
  • True Crowd scores translations from the actual systems under evaluation, making it equivalent to quality estimation and an upper bound rather than a proper difficulty estimator.Source: §4.4 · p. 4

5 Experiments

The experiments evaluate difficulty estimators with DEC on WMT test sets and compare learned, heuristic, LLM-judge, and crowd-based approaches. Sentinel-src-24 leads the evaluated estimators, while human and machine translators show different difficulty-score agreement patterns.

5.1 Experimental Setup

  • DEC is measured on WMT 2024 General MT and Metrics test sets with human quality annotations, using ESA in the reported results.The evaluation also tests Sentinel-src-24, trained on WMT data through 2023, and Sentinel-src-25, which additionally includes the WMT 2024 test set; Sentinel-src-25 is excluded from Table 1.Source: §5.1 · pp. 4–5, §5 · p. 4, §5.2 · p. 5

5.2 Results

  • Sentinel-src-24 achieves the highest DEC overall, outperforming the earlier Sentinel-src and all other evaluated estimators.Learned estimators also outperform heuristic-based ones.Source: §5.2 · p. 5, Table 1 · p. 5, §5.3 · p. 6
  • LLM judges perform poorly overall, even below Text Length heuristics, although target-language information improves both models.For CommandA, target-language information increases correlation by 0.032 points.Source: §5.2 · p. 5
  • True Crowd methods yield the highest correlation as upper bounds, while Artificial Crowd performance is comparable to Sentinel-src-24 but requires translation and quality estimation.Because True Crowd uses ground-truth translations, its result also shows XCOMET outperforming MetricX on this task.Source: §5.2 · p. 5

5.3 Discussion

  • Artificial Crowd benchmark construction assumes evaluation on models other than those used to build it, since its intermediate translations would bias the test set against those models.The downstream benchmark analysis excludes True Crowd estimators because they serve as upper bounds rather than genuine difficulty estimators.Source: §5.3 · p. 6, §5.4 · p. 6

5.4 Comparing Human and Machine Translation Difficulty

  • Human-to-model difficulty-score correlations are 0.109 to 0.137, below the 0.151 to 0.221 range between machine models, suggesting differing perceptions of difficulty.The analysis averages Kendall’s τb across language directions and is restricted to one human translator and four models with annotations for all directions.Source: §5.4 · p. 6, Table 2 · p. 6

6 Creating Difficult Benchmarks

Difficulty estimators can select challenging WMT24 benchmark subsets, with Sentinel-src-24 and Artificial Crowd performing closer to oracles than random selection. The section also examines selection risks and the kinds of texts Sentinel-src-25 judges difficult.

6.2 Results

  • On WMT24, the best model for each language almost always scored 90–100 ESA, and at least one system did so for every source text.Top systems made barely any errors on English-to-Spanish, motivating more difficult benchmarks.Source: §6 · p. 7

6.1 Setup

  • The benchmark task selects the B highest-difficulty texts from WMT24 English and Czech sources, evaluating subsets by average human score and the share of perfect outputs.Source-only estimators score texts directly; Artificial Crowd averages XCOMET translation-quality estimates across MT models and target languages.Source: §6.1 · p. 7

6.2 Results

  • Sentinel-src-24 and Artificial Crowd perform closer to the oracles than text-length heuristics or LLM-as-a-Judge, and surpass oracle-src on the proportion of perfect outputs.Figure 2 reports average score and perfect-output proportion across simulated WMT24 language pairs; lower values are better.Source: Figure 2 · p. 7, §6.2 · p. 8

6.3 Potential Pitfalls of Selecting by Difficulty

  • All estimators favor longer sources, none appear to prioritize source errors, and Sentinel-src-24 and Artificial Crowd select texts with more diverse outputs.The analysis treats negative correlations with estimator scores as positive associations with difficulty because lower scores mean greater difficulty.Source: §6.3 · p. 8
  • Difficulty scores align relatively uniformly with individual MT models, but ONLINE-B has higher-than-average alignment with Sentinel-src-24 and Gemini-1.5-Pro lower alignment with both estimators.The authors leave open whether the discrepancy reflects estimation bias or variance, noting the limited number of translation directions.Source: Figure 3 · p. 8, §6.3 · p. 9

6.4 Qualitative Analysis

  • Sentinel-src-25’s difficult texts often contain complex constructions, incomplete sentences, or direct speech, while its easy texts tend to be short and simple.The qualitative review manually inspected 200 source texts, split evenly between easy and difficult, across ten length-based buckets.Source: §6.4 · p. 9

7 Conclusion

The paper defines translation difficulty estimation and evaluates it with a dedicated metric, finding that task-trained models outperform heuristic and LLM-as-a-judge approaches. Sentinel-src-24 leads both difficulty estimation and difficult-benchmark selection, and the authors release Sentinel-src-24 and Sentinel-src-25.

7 Conclusion

  • The paper defines translation difficulty estimation and introduces Difficulty Estimation Correlation (DEC), a dedicated evaluation measure; task-trained models outperform heuristic methods and LLM-as-a-judge approaches.Sentinel-src-24 is identified as the state of the art, including in the downstream task of creating difficult benchmarks.Source: §7 · p. 9
  • Sentinel-src-24 remains the top-performing method for creating difficult benchmarks, and both Sentinel-src-24 and Sentinel-src-25 are released publicly.Sentinel-src-25 adds data to Sentinel-src-24’s training pipeline; the paper also qualitatively analyzes its predictions.Source: §7 · p. 9

Limitations

The paper proxies text difficulty with translation quality for contemporary MT models rather than defining a model-independent intrinsic property. Its analysis also notes limited investigation of target-language effects and an artificial phenomenon distribution in the human-vetted WMT24 test set.

Limitations

  • Translation difficulty is treated as the quality of translations produced by contemporary MT models, not as an abstract, model-independent property of the source.The authors acknowledge that ideally difficulty would be intrinsic to the source, independent of any particular model.Source: Limitations · p. 10
  • The impact of target language is only briefly discussed, not investigated; the authors report that LLM-as-a-Judge improves when given target-language information.They encourage further research into target-language effects and their incorporation into other estimators.Source: Limitations · p. 10
  • WMT24 sources were human-vetted, making the distribution of investigated phenomena artificial; the authors repeat the analysis on a larger batch of data.Source: Limitations · p. 10

Ethics Statement

The supplied passage states that the authors foresee no ethical issues. The remaining passages describe Kendall’s τb and Sentinel-src-24/25 training, rather than additional ethics claims.

Ethics Statement

A Kendall τb

  • Kendall’s τb compares concordant and discordant pairs while accounting for ties in gold and hypothesis rankings.The paper favors τb over Pearson correlation because it uses pairwise rankings and is more robust to scale differences and outliers.Source: A Kendall τb · p. 15

B Training Sentinel-src-24 and Sentinel-src-25

  • Sentinel-src-24 and Sentinel-src-25 use a two-stage training pipeline: DA training followed by MQM fine-tuning.Sentinel-src-25 adds WMT24 annotations; the training instances retain individual human scores rather than averaging multiple annotations.Source: B Training Sentinel-src-24 and… · p. 15

C Implementation Details

Implementation details specify tools for the difficulty heuristics and output-diversity assessment, describe the artificial-crowd models, and define the benchmark evaluation measures.

C Implementation Details

  • Word frequencies for the rarity heuristic come from the wordfreq Python library, while spaCy pipelines provide dependency trees and tokens for syntactic complexity and text length.The language-specific pipelines are en_core_web_sm for English, ja_core_news_sm for Japanese, and spacy_udpipe for Czech.Source: C Implementation Details · pp. 15–16
  • Output diversity is measured using pairwise multilingual sentence-embedding inner products and chrF scores between translations.Source-text embeddings use sentence-transformers/paraphrasemultilingual-MiniLM-L12-v2.Source: C Implementation Details · p. 16
  • The artificial crowd includes NLLB-moe-54B, Gemma-3-27B-IT, Qwen2.5-72B-IT, and CommandA.The passage describes their model families and sizes, including CommandA at 111B parameters.Source: C Implementation Details · p. 16
  • Evaluation reports average human scores and the proportion of perfect translations for selected subsets of size B.Table 5 statistics identify the number of source texts and available translations per source text in the WMT 2024 General Machine Translation test set.Source: C Implementation Details · p. 16, Table 5 · p. 16

D Complete Results

The complete results section reports difficulty-estimation correlations for MQM-annotated WMT24 data and defines three oracle variants by the information available to them.

D Complete Results

  • Table 6 reports difficulty estimation correlation scores for all considered methods using MQM annotations.The oracle methods use the true human judgments from which difficulty scores are derived.Source: D Complete Results · p. 16, E Oracles · p. 16

E Oracles

  • The three oracles condition their estimates on source text, target language, and translation to different degrees.The fullest oracle uses the specific translation; the source-plus-language oracle averages across models, while the source-only oracle averages across models and target languages.Source: E Oracles · p. 16

F Creating Difficult Benchmarks – Quantitative Results

The paper evaluates difficulty-based selection of the hardest quarter of WMT24 test samples and reports that Sentinel-src-24 generally selects harder subsets than other automatic methods. Related work includes approaches aimed at efficient testing or removing easy examples, rather than specifically constructing difficult test sets.

F Creating Difficult Benchmarks – Quantitative Results

  • The evaluation selects the most difficult 25% of WMT24 source texts and assesses the subsets with human annotations using AvgScore and %Perfect.Lower values on both measures indicate greater difficulty.Source: F Creating Difficult Benchmarks –… · pp. 16–17
  • Text Length consistently outperforms Command A LLM-as-a-Judge, while Command A only marginally improves over random selection in both ESA and MQM results.The passage notes that Command A requires significantly more computational resources.Source: F Creating Difficult Benchmarks –… · p. 17
  • Across News, Social, Literary, and Speech domains, the overall patterns hold, with Sentinel-src-24 and Artificial Crowd strongest in nearly all domain-specific evaluations.The analysis supports the practical utility of difficulty estimation for controlled test-set construction.Source: F Creating Difficult Benchmarks –… · p. 17

G Related work for Benchmark Creation

  • Prior NLG benchmark methods often target efficient testing rather than difficult test-set construction, while prior MT work includes proxy-based difficulty and removal of easy examples.The related-work discussion also notes that Item Response Theory approaches are not applicable to MT, where output quality is represented as a continuous score.Source: G Related work for Benchmark Creation · p. 18

F Creating Difficult Benchmarks – Quantitative Results

  • Sentinel-src-24 has the best %Perfect in ESA Table 9, is surpassed only by Artificial Crowd in AvgScore there, and beats all automatic methods on both measures in MQM Table 10.Artificial Crowd's ESA AvgScore advantage comes from an approach requiring translations from multiple large models and an XXL MT metric for evaluation.Source: F Creating Difficult Benchmarks –… · p. 17, Table 9 · p. 22, Table 10 · p. 22
Loading 2508.10175v2…