PaperlayerView exact revision on arXiv

Source-linked AI summary

The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation

Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, Orhan Firat

arXiv:2308.07286v1cs.CLcs.LG

TL;DR

Machine translation metrics often provide scalar scores without the detailed error information available in MQM, limiting interpretability. This paper studies prompting and finetuning of LLM evaluators and introduces AutoMQM, which identifies and categorizes errors to derive scores. AutoMQM improves segment-level performance over score prompting while producing error spans aligned with human annotations.

  • Problem

    Existing MT metrics can correlate well with human judgments but generally provide a single uninterpretable score instead of detailed error feedback.

    Source: §1 · p. 1, §5 · p. 4

  • Method

    The paper evaluates PaLM and PaLM-2 through score prompting, in-context learning, and finetuning, then proposes AutoMQM to identify and classify error spans under MQM.

    Source: §1 · p. 2, §5 · p. 5, §7 · p. 11

  • Results

    AutoMQM improves segment-level performance over score-prediction prompting and produces error-span annotations that align with human annotations.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    LLMs can provide more interpretable MT evaluation by pairing quality assessment with fine-grained error annotations.

    Source: §1 · p. 2, §7 · p. 11

    Meaningful AutoMQM outputs are substantially more challenging in zero-shot settings, where LLMs tend to produce uninformative outputs.

    Source: §5 · p. 5

Abstract

from arXiv · show

Automatic evaluation of machine translation (MT) is a critical tool driving the rapid iterative development of MT systems. While considerable progress has been made on estimating a single scalar quality score, current metrics lack the informativeness of more detailed schemes that annotate individual errors, such as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap by proposing AutoMQM, a prompting technique which leverages the reasoning and in-context learning capabilities of large language models (LLMs) and asks them to identify and categorize errors in translations. We start by evaluating recent LLMs, such as PaLM and PaLM-2, through simple score prediction prompting, and we study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores (with particularly large gains for larger models) while providing interpretability through error spans that align with human annotations.

1 Introduction

MT metrics increasingly need to move beyond single, often uninterpretable quality scores toward detailed error feedback. This paper studies LLM-based evaluation and proposes AutoMQM, which identifies and classifies translation errors while deriving scores through MQM.

1 Introduction

  • Current learned metrics correlate strongly with human judgments but usually output only a single uninterpretable score, limiting diagnostic value for system developers.MQM instead records error spans, categories, and severity, but most automatic metrics discard that information.Source: §1 · p. 1
  • LLM evaluation research had focused mainly on score prediction, with limited use of annotated data and testing concentrated on high-resource language pairs.The paper therefore studies prompting, finetuning, and low-resource evaluation across PaLM and PaLM-2 models.Source: §1 · p. 2
  • AutoMQM prompts LLMs to identify and classify translation error spans under MQM, deriving a quality score while exposing the errors behind it.Its interpretability lets users inspect the errors responsible for a score.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • LLMs are zero-shot state-of-the-art system-level evaluators but show lower segment-level correlation with human judgments than learned metrics.Finetuning with human judgments mitigates low segment-level performance, particularly for smaller LLMs.Source: §1 · p. 2
  • LLM-based evaluation also appears promising for low-resource language pairs, although it lags behind state-of-the-art learned metrics.This study is presented as the first evaluation of LLM-based evaluation methods in that setting.Source: §1 · p. 2
  • AutoMQM enables PaLM-2 models to outperform score-prediction counterparts at the segment level while generating MQM-like annotations.The predicted annotations identify over 50% of words involved in major errors and are comparable to state-of-the-art supervised word-level evaluators.Source: §1 · p. 2

2 Background: MT Evaluation

Machine translation evaluation judges whether a candidate conveys the source meaning and remains fluent, optionally using a reference translation. The field has shifted from single-score judgments toward MQM’s richer error-span annotations.

2 Background: MT Evaluation

  • MT evaluation assesses a candidate translation against the source meaning and fluency, often with a professional reference translation in reference-based evaluation.Without a reference, the task is called reference-less evaluation or quality estimation.Source: §2 · p. 2
  • Single-score evaluation can produce rushed, noisy human judgments and hides the errors underlying a translation’s ranking or score.This makes score prediction problematic both for annotation and for diagnosing metric mistakes.Source: §2 · pp. 2–3
  • MQM asks human evaluators to identify and categorize error spans across dimensions such as fluency and accuracy, then derives a quality score automatically.Despite this richer framework, many automatic metrics using MQM data retain only the final score and discard span information.Source: §2 · p. 3

3 Related Work

Prior work established learned metrics and LLMs as strong MT evaluators, while also exploring error-based and word-level quality estimation. AutoMQM differs by combining LLM prompting with MQM-style structured errors and automatic scoring.

3 Related Work

  • Learned metrics achieve strong human-judgment correlation through transfer learning, while GPT-based evaluators demonstrated promising zero- and few-shot MT evaluation.Kocmi and Federmann (2023) showed that GPT models can reach state-of-the-art system-level evaluation performance.Source: §3 · p. 3
  • Perrella et al. (2022) used MQM-like error prediction for automatic scoring, but their supervised encoder-based MaTASe metric underperformed other top metrics.The paper compares against MaTASe in its experiments.Source: §3 · p. 3
  • Lu et al. (2023) found that prompted error analysis can improve ChatGPT evaluators, but still relied on the LLM to produce the score and evaluated only 40 examples per language pair.AutoMQM instead derives the score automatically from identified errors.Source: §3 · p. 3
  • INSTRUCTSCORE asks LLaMA to identify and categorize translation errors, but explores only a 7B model and relies on knowledge distilled from GPT-4 rather than zero- or few-shot capabilities.The paper positions its PaLM and PaLM-2 study as broader in model scale and prompting setup.Source: §3 · p. 3
  • WMT word-level quality-estimation tasks use MQM-derived error tags, and this paper compares AutoMQM with state-of-the-art word-level quality-estimation models.Some shared-task systems predicted word-level tags that were subsequently converted to spans.Source: §3 · p. 3

4 Using LLMs to Predict Quality Scores

The paper first evaluates LLMs as score predictors through prompting and in-context learning, then tests finetuning on human judgments. Prompt examples are sampled uniformly or by score range, and finetuning uses regression or generative classification.

4 Using LLMs to Predict Quality Scores

4.1 Prompting

  • In-context learning adds a small amount of labeled translation-quality data without parameter updates, but example selection can determine whether performance improves or worsens.The paper compares uniform sampling with stratified sampling across score ranges.Source: §4.1 · p. 4

4.2 Finetuning

  • Finetuning tests whether LLMs can benefit from human judgment data, motivated by the strong performance of learned metrics trained on such data.The experiments use two objectives: continuous-score regression and discrete generative classification.Source: §4.2 · p. 4
  • Regression minimizes a loss between model-generated continuous scores and human scores, whereas generative classification buckets scores and treats evaluation as text-to-text classification.These objectives adapt decoder-only LLMs to score-based MT evaluation in different output spaces.Source: §4.2 · p. 4

5 Using LLMs to Predict Error Spans

This section introduces AUTOMQM, which prompts LLMs to identify and categorize translation errors using MQM-style annotations rather than directly producing a score. The approach uses structured prompting and in-context examples, but meaningful zero-shot outputs remain substantially more challenging.

Prompt design

  • The AUTOMQM prompt distinguishes major translation or grammatical errors from minor imperfections and subjective opinions, with slots for structured error outputs.The prompt can include source and reference information and formats outputs as error spans paired with severity and category.Source: §5 · p. 5

Prompting strategy

  • Chain-of-thought prompting asks LLMs to generate intermediate reasoning steps, which can improve task performance and support structured rationales.Structured rationales have been reported to outperform free-text rationales in related prompting settings.Source: §5 · p. 5

AUTOMQM

  • AUTOMQM instructs LLMs to identify translation errors and categorize them according to the MQM framework, deriving scores algorithmically from error penalties.The scoring procedure roughly assigns −5 to major errors and −1 to minor errors.Source: §5 · p. 5

Few-shot setting

  • Meaningful AUTOMQM outputs are substantially harder to obtain in zero-shot settings than score predictions, motivating the use of few-shot in-context examples.The supplied passage states that zero-shot outputs tend to be uninformative or difficult to parse.Source: §5 · p. 5

6 Experiments

Experiments compare PaLM and PaLM-2 LLM evaluators with prompting, in-context learning, and finetuning across high- and low-resource translation, then assess AutoMQM’s score and span predictions. AutoMQM improves larger-model evaluation and produces error spans aligned with human annotations, while score-prompted LLMs show weaker segment-level performance and sensitivity to examples.

6.1 Experimental Setup

  • The experiments cover WMT high-resource and low-resource language pairs, compare PaLM-family models with learned metrics and GEMBA, and evaluate both scalar scores and AutoMQM spans.AutoMQM is restricted to en→de and zh→en because it requires MQM-annotated in-context examples.Source: §6.1 · pp. 5–7

6.2.1 Score Prediction

  • Zero-shot LLMs generally outperform learned metrics at system level but do not consistently surpass them at segment level; PaLM-2 BISON is competitive with learned baselines, while GEMBA is stronger in some segment-level settings.PaLM-2 models show higher system-level performance than GPT-based GEMBA, whereas GEMBA achieves better segment-level accuracy, particularly without references.Source: §6.2.1 · p. 7, Table 3 · p. 8
  • Finetuning substantially improves segment-level LLM evaluation, with finetuned PaLM-2 BISON reaching state-of-the-art segment-level correlations and comparable system-level accuracy across language pairs.Smaller finetuned models remain competitive with larger finetuned counterparts despite weaker zero-shot performance.Source: §6.2.1 · p. 8, Figure 5 · p. 9
  • In-context examples almost never improve score prediction because example scores strongly bias output distributions, whereas AutoMQM performance rises with up to four examples and is more robust to example-set choice.For AutoMQM, this behavior is similar in reference-based and reference-less evaluation, and well-performing example sets transfer between the two settings.Source: §6.2.1 · pp. 8–9, §6.3.1 · pp. 9–10, Figure 8 · p. 10

6.3 Low Resource Languages

  • Low-resource PaLM-2 score prediction reaches around 90% system-level accuracy but underperforms learned metrics, even when those baselines lack data from the evaluated languages.The low-resource evaluation uses direct-assessment scores rather than MQM ratings.Source: §6.1 · p. 5, §6.3 · p. 9, Table 4 · p. 9

6.3.1 AUTOMQM

  • AutoMQM improves PaLM-2 evaluation most clearly for larger models, with UNICORN outperforming score prediction across meta-evaluations and its reference-less version competitive with the best learned metric at segment level.For smaller BISON, AutoMQM and score prediction perform comparably, suggesting scale matters for zero- and few-shot fine-grained evaluation.Source: §6.3.1 · p. 10, Table 5 · p. 10
  • PaLM-2 AutoMQM identifies most major errors and achieves span correlations comparable to learned word-level evaluators, but over-predicts errors and has lower span precision.UNICORN generally outperforms BISON; predicted spans average about 5 words versus about 2 words in the ground truth.Source: §6.3.1 · p. 11, Table 6 · p. 11

7 Conclusion

The paper systematically evaluates LLMs for MT evaluation and proposes AutoMQM for interpretable, fine-grained assessment. Score prompting reaches state-of-the-art system-level evaluation, while AutoMQM improves performance without finetuning and produces error spans aligned with human annotations.

  • The study systematically investigates score prediction and introduces AutoMQM as an MQM-based prompting technique for interpretable MT evaluation.Source: §7 · p. 11
  • AutoMQM improves LLM evaluation performance without finetuning while providing interpretable error spans aligned with human annotations.Source: §7 · p. 11
  • Finetuning on fine-grained human judgment data such as MQM is identified as a direction for further improving LLM-based MT evaluation.Source: §7 · p. 11

A Multidimensional Quality Metric (MQM)

MQM is a human-evaluation framework that identifies and categorizes translation errors, then derives segment-level scores from weighted error annotations. AutoMQM uses the same weighting scheme for scores derived from its predicted errors.

B Sampling in-context learning examples for AutoMQM

The appendix documents the rejection criteria used when sampling example sets for AutoMQM's in-context learning prompts.

C Additional Results

The appendix provides additional experimental results, including sampling criteria, in-context-example analyses, evaluator score distributions, and MQM reference material.

  • The appendix includes additional experimental results in Figures 12, 13, and 14.Source: C Additional Results · p. 16
  • The appendix includes the MQM hierarchy and error-weighting reference tables.Source: Table 8 · p. 17, Table 9 · p. 17
  • Figure 11 documents the rejection criteria used when sampling AutoMQM example sets.Source: Figure 11 · p. 18
  • Figures 12 and 13 show mean Pearson correlation and interquartile ranges as the number of in-context examples increases, comparing uniform and stratified sampling for WMT22 EN-DE and ZH-EN.Source: Figure 12 · p. 19, Figure 13 · p. 19
  • Figure 14 shows score distributions for reference-based LLM evaluators on the ZH-EN test set using a logarithmic y-axis.Source: Figure 14 · p. 19

Related papers

Loading 2308.07286v1…