PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Hala Almaghout, Christian Federmann, Qin Gao

arXiv:2610.10918v1cs.CL

TL;DR

This paper examines whether LLMs can reliably serve as alternatives to human MT evaluation across annotation schemes, language pairs, and domains. It compares LLM and human agreement on MQM and ESA data, assessing both scores and error spans. LLM–human agreement matches human–human agreement for some tasks, but agreement varies substantially and remains below the tentative-agreement threshold in many settings.

  • Problem

    LLMs are considered a more efficient and cost-effective alternative to human MT evaluation, but their agreement with humans across annotation schemes, language pairs, and domains requires assessment.

    Source: §1 · pp. 1–2

  • Method

    The study compares four LLMs with human annotators on MQM and ESA data, evaluating score and error-span agreement across datasets and settings.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    LLM–human agreement matches human–human agreement for some tasks, but varies considerably across language pairs and domains and often remains below the 0.667 tentative-agreement threshold.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Both human and LLM annotation performance have room for improvement, with humans challenged by fine-grained MQM and low-resource pairs and LLMs by minor errors, wrong language variants, and span annotation.

    Source: §1 · p. 2, §5 · p. 9

    The study’s significance testing assesses whether human–human and LLM–human agreement are equal, but does not establish equivalence between them.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.

1 Introduction

Human MT evaluation is costly and can be unreliable, while LLMs are being evaluated as a possible alternative. This study compares LLM and human agreement across MQM and ESA, finding that performance varies by task and setting and that both annotator groups face distinct challenges.

1 Introduction

  • Human MT evaluation is expensive and time-consuming, and its quality and reliability can vary with evaluator skills, training, motivation, and leniency.Fine-grained approaches such as MQM and ESA can improve quality, interpretability, and consistency, but take more time and cost more to obtain.Source: §1 · p. 1
  • LLM–human agreement is indistinguishable from human–human agreement for some tasks, but both vary substantially across annotation schemes, language pairs, and domains.The study compares agreement for MQM and ESA across a long-context test set of 70 language pairs and WMT23 and WMT25 data.Source: §1 · p. 2
  • Human agreement is particularly challenged by fine-grained MQM annotation and low-resource language pairs.Source: §1 · p. 2
  • LLMs particularly struggle with minor errors, translations in the wrong language or variant, and error-span annotation, especially for rarer error categories.Source: §1 · p. 2
  • The authors identify improving the reliability of both human and LLM evaluation as a need raised by low agreement levels.Source: §1 · p. 2

2 Related Work

Prior work examines automatic MT metrics, AI-assisted human annotation, and the reliability of LLMs as annotators. This paper extends comparisons across both MQM and ESA, scores and spans, and broader language-pair coverage.

2.1 Automatic MT Evaluation Metrics

  • At WMT25, LLM-based metrics led at system level but were outperformed by surface- and embedding-based metrics at segment level; span detection remained difficult for both systems and humans.The best autoraters achieved 9–13% character-level F1, compared with 47% for human annotators; Zhan et al. (2025) also found minor errors more misaligned with human annotations than major errors.Source: §2.1 · p. 2

2.2 AI-Assisted and Collaborative Human Annotation

  • AI-assisted ESA annotation and human revision of automatic MQM annotations have been reported to improve error recall or annotation quality while reducing annotation burden or time.Zouhar et al. (2025) report 1.6 versus 0.5 spans per segment and a 24% budget reduction; Riley et al. (2025) report time reductions of up to 12%.Source: §2.2 · pp. 2–3
  • Other work reports higher agreement and more consistent spans when MQM annotations are performed side by side rather than independently.Source: §2.2 · p. 3

2.3 Reliability of LLMs as Annotators

  • Research on LLM annotation finds reliability depends on the task and annotation aspect, with low human agreement making replacement tests unreliable and span annotation remaining difficult.Prior studies also report inconsistent prompting gains and the need to re-align LLM-generated span indices.Source: §2.3 · p. 3
  • This study differs from earlier work by comparing LLM and human agreement across both MQM and ESA, including scores and spans, with MQM coverage of the full taxonomy on 70 language pairs.It also uses two annotator pools and platforms and reports statistical significance tests.Source: §2.3 · p. 3

3 Experimental Setup

The experimental setup combines three MT evaluation datasets, four LLMs, and agreement analyses for scores, error counts, and error spans. It uses statistical testing to compare LLM–human and human–human agreement.

3.1 Data

  • The study uses WMT23 MQM for English–German and Chinese–English, an internal long-context web MQM set, and WMT25 ESA across 14 language pairs and five domains.The Long-Context Web MQM set includes human translations and outputs from neural and LLM-based MT systems; human annotations used two independent rounds and different annotator pools and platforms.Source: §3.1 · p. 3

3.2 Large Language Models

  • Four LLMs are evaluated using adapted human annotation guidelines and zero-shot prompting; few-shot examples did not improve LLM–human agreement.The models are Gemini 2.5 Flash, GPT-OSS-120B, DeepSeek-V3.1, and Qwen3.5-397B. Segments with an unparseable output from any model are excluded.Source: §3.2 · p. 4
  • Because at least 80% of extracted span indices failed to match their corresponding span text, the study re-aligns spans by searching for the text in the source or target.Source: §3.2 · p. 4

3.3 MQM Scoring

  • MQM scores are calculated from annotated errors using weights of 5 for major errors, 1 for minor errors, and 0.1 for punctuation errors.Source: §3.3 · p. 4

3.4 Annotation Evaluation

  • The study computes F1 for human–human and human–LLM annotation pairs and averages the results within each group.Source: §3.4 · p. 4
  • For score and error-count agreement, the study uses interval-distance Krippendorff’s α and averages LLM–human agreement across leave-one-out annotator groups.The measured scores include segment-level MQM and ESA scores and MQM major and minor error counts.Source: §3.4 · p. 4
  • Span evaluation uses character-overlap F1, span-overlap F1, category/severity-specific span-overlap F1, and F1 for identifying error-free segments.The span metrics compare LLM annotations with gold spans and vary in whether category and severity must match.Source: §3.4 · p. 4

3.5 Significance Testing

  • Agreement comparisons use two-sided paired bootstrap resampling with 10,000 iterations, with significance set at p < 0.05; multiple comparisons use Benjamini–Hochberg FDR correction.Source: §3.5 · pp. 4–5

4 Results

Across MQM and ESA, agreement varies by annotation task, language pair, and domain, with human-human agreement often exceeding LLM-human agreement. Both human and LLM annotation have notable weaknesses, including low agreement on fine-grained spans and minor errors.

4.1.1 Score and Error Count Evaluation

  • On WMT23 MQM, human-human score agreement exceeds LLM-human for en-de and zh-en, except Gemini is on par for zh-en; all annotator combinations remain below tentative agreement.Major-error agreement is closer for Gemini on zh-en, while human-human agreement substantially exceeds LLM-human agreement on minor errors for both language pairs.Source: §4.1.1 · p. 5, Figure 1 · p. 5

4.1.2 Span Annotation Evaluation

  • On WMT23 MQM, human-human span agreement exceeds LLM-human for most metrics, and agreement is especially weak for low-frequency categories and minor errors.Human-human span overlap reaches at most F1 = 0.51; many low-frequency categories have zero F1 for all LLMs, and both human-human and LLM-human agreement are lower for minor than major errors.Source: §4.1.2 · pp. 5–6, Table 1 · p. 6

4.2.1 Score Evaluation

  • On WMT25 ESA scores, agreement falls below α = 0.667 for nearly all language pairs and domains, and varies substantially across domains and language pairs.Agreement is exceptionally low across annotator pairs for low-resource en-mas_KE; en-arz_EG appears to be an exception for systematic reasons, as LLMs score many wrong-variant translations highly while humans score them zero.Source: §4.2.1 · pp. 6–7, Figure 2 · p. 7
  • On WMT25 ESA scores, human-human agreement significantly exceeds LLM-human agreement for most language pairs and domains for most models.Gemini-human agreement is higher than that of other models in most language-pair and domain combinations, but it both exceeds and falls below human-human agreement across domains.Source: §4.2.1 · p. 7, Table 2 · p. 8

4.2.2 Span Annotation Evaluation

  • On WMT25 ESA span metrics, human-human agreement significantly exceeds LLM-human agreement, and both groups agree more on major than minor errors.Both humans and LLMs have low agreement on missing spans; DeepSeek-human agreement on identifying error-free segments is 0.05.Source: §4.2.2 · pp. 7–8, Table 3 · p. 8

4.3 Annotation Quality Validation

  • For en_US-hi_IN and en_US-th_TH, Gemini’s agreement varies substantially across human annotation rounds, and one round may be problematic.Human-human agreement is negative or close to zero in these examples, while Gemini agrees substantially more with one round than the other, suggesting it may help flag rounds for investigation.Source: §4.3 · p. 8, Figure 3 · p. 8

5 Conclusion

The paper evaluates four LLMs as MQM and ESA annotators across long-context and public WMT data. Agreement varies considerably and remains weak in many settings, revealing challenges for both human and LLM annotation.

5 Conclusion

  • The study compares four LLMs with human annotators on MQM and ESA scores and error spans across 70 language pairs and WMT23 and WMT25 data.The evaluation covers different language pairs, domains, and annotation settings.Source: §5 · p. 8
  • LLM-human agreement sometimes exceeds human-human agreement, but it varies by model, language pair, and domain and often remains below α = 0.667.The authors caution that LLM-human agreement superiority is not evidence that LLMs can substitute for humans when human agreement is weak.Source: §5 · pp. 8–9
  • Humans struggle with fine-grained MQM annotation and low-resource pairs, while LLMs struggle with minor errors, wrong language variants, and span annotation, especially rare errors.The paper identifies clearer guidelines, better training, improved annotation processes, and targeted prompting or fine-tuning as possible improvement paths, and proposes collaborative human-LLM annotation.Source: §5 · p. 9

6 Limitations

The paper identifies limits in reliability thresholds, replicability, study coverage, contamination controls, statistical testing, and bias risks.

6 Limitations

  • The 0.667 threshold for tentative agreement may not be realistic for fine-grained MQM, and reliable thresholds across settings remain to be established.The authors call for research across text lengths, domains, and annotation counts, as well as on low-agreement factors in long-context annotation.Source: §6 · p. 9
  • Licensing prevents publication of the internally annotated Long-Context Web MQM sets, limiting replication of the MQM results.Source: §6 · p. 9
  • The study uses four LLMs, one zero-shot prompt, and a simple span-correction heuristic, leaving broader model and prompting evaluations and improved span localization for future work.Source: §6 · p. 9
  • WMT23 MQM data may have appeared in model training and could inflate LLM-human agreement, whereas the WMT25 ESA test set likely postdates the models’ cutoffs.The unreleased Long-Context Web MQM set cannot be in these models’ training data.Source: §6 · p. 9
  • Significance tests compare whether human-human and LLM-human agreement are equal, but do not establish whether the agreements are equivalent.Source: §6 · p. 9
  • LLM evaluation may amplify biases in evaluated models; the authors observed failure to penalize Egyptian Arabic translations when Modern Standard Arabic was expected.They attribute the bias to greater training-data representation of the favored variant.Source: §6 · p. 9

A.1 Long-Context Web MQM Human Annotation

The Long-Context Web MQM set was produced through a managed annotation process with two independent rounds and overlapping segment assignments.

A.1 Long-Context Web MQM Human Annotation

  • An annotation vendor managed recruitment, project oversight, quality control, and task-specific training, using a secure platform without third-party or crowdsourcing ties.Source: §A.1 · p. 12
  • The researchers supplied guidelines and UI mock-ups, communicated with the vendor about annotation issues, and annotators were paid market rates.Source: §A.1 · p. 12
  • Annotators were fluent in source languages and native-level in target languages; linguists filled rare-language-pair gaps when professional graders were unavailable.Source: §A.1 · p. 12
  • Two rounds used different annotators, with each segment annotated by two different annotators across the rounds.Source: §A.1 · p. 12

A.2.1 Score and Error Count Evaluation

On the 70-pair Long-Context Web MQM set, agreement varies by direction, error severity, and language pair; score and count agreement is often below the tentative threshold.

A.2.1 Score and Error Count Evaluation

  • For MQM scores, en-X LLM-human agreement is not significantly below human-human for Gemini and GPT, while DeepSeek and Qwen are significantly behind; in X-en, only GPT is not significantly behind.Source: §A.2.1 · p. 12, Figure 4 · p. 13
  • In X-Y, most LLM-human MQM score averages exceed human-human agreement, which is notably lower there; all agreement levels remain below 0.667.Source: §A.2.1 · p. 12, Figure 4 · p. 13
  • For major error counts, most LLMs match or exceed human-human agreement, whereas minor-count agreement is significantly lower for most LLMs across all directions.Source: §A.2.1 · p. 12, Figure 4 · p. 13
  • Table 4 comparisons show mostly no significant score-agreement difference in en-X and X-en, while Gemini and GPT exceed human-human agreement for most X-Y pairs.The X-Y result is associated with the lower human-human baseline in that direction.Source: Table 4 · p. 12, §A.2.1 · p. 12
  • Across language pairs, agreement varies substantially and falls below α = 0.667 for nearly all pairs; non-English, predominantly low-resource pairs have particularly low agreement.The few human-human exceptions reaching the threshold include Russian-to-English and Simplified and Traditional Chinese-to-English.Source: §A.2.1 · p. 13

A.2.2 Span Annotation Evaluation

Span-annotation agreement is low across categories, and human-human agreement generally exceeds LLM-human agreement on the Long-Context Web MQM set.

A.2.2 Span Annotation Evaluation

  • Across error categories and annotator pairs, F1 agreement reaches at most 0.36 on the Long-Context Web MQM set.Table 5 averages over language pairs and orders categories by frequency.Source: §A.2.2 · p. 13, Table 5 · p. 14
  • Human-human agreement significantly exceeds LLM-human agreement on all span metrics except mistranslation, major errors, and non-translation, where Gemini-human agreement is significantly higher.Source: §A.2.2 · p. 13, Table 5 · p. 14
  • All annotators have notably low agreement on identifying error-free segments, with F1 ≤ 0.18; LLMs also score low on several low-frequency categories.Categories with fewer than five source segments in the human-human baseline are omitted from Table 5.Source: §A.2.2 · p. 13, Table 5 · p. 14
Loading 2610.10918v1…