PaperlayerView exact revision on arXiv

Source-linked AI summary

Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models

Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, Dacheng Tao

arXiv:2303.13809v4cs.CL

TL;DR

LLM-based MT evaluation can achieve strong system-level performance but standard prompting performs poorly at the segment level and lacks interpretable error information. The paper introduces EAPrompt, combining Chain-of-Thought and Error Analysis to emulate MQM through major/minor error identification and scoring. EAPrompt improves evaluation at both levels, surpasses GEMBA in 8 of 9 scenarios, and remains effective without references.

  • Problem

    Standard LLM prompting achieves strong system-level MT evaluation but performs poorly at the segment level and does not provide interpretable translation-error information.

  • Method

    EAPrompt combines Chain-of-Thought and Error Analysis to emulate MQM by identifying major and minor errors before scoring translations.

  • Results

    EAPrompt improves evaluation at both system and segment levels, with GPT-3.5-Turbo establishing a new state-of-the-art and EAPrompt surpassing GEMBA in 8 out of 9 scenarios.

  • Takeaways & Limitations

    EAPrompt provides explainable and reliable MT evaluations and remains suitable for reference-less quality estimation settings.

  • Takeaways & Limitations

    The study evaluates a limited number of LLMs, and not all existing LLMs necessarily serve as reliable evaluators under EAPrompt.

Abstract

from arXiv · show

Generative large language models (LLMs), e.g., ChatGPT, have demonstrated remarkable proficiency across several NLP tasks, such as machine translation, text summarization. Recent research (Kocmi and Federmann, 2023) has shown that utilizing LLMs for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level but \textit{performs poorly at the segment level}. To further improve the performance of LLMs on MT quality assessment, we investigate several prompting designs, and propose a new prompting method called \textbf{\texttt{Error Analysis Prompting}} (EAPrompt) by combining Chain-of-Thoughts (Wei et al., 2022) and Error Analysis (Lu et al., 2023). This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM, Freitag et al. (2021)) and \textit{produces explainable and reliable MT evaluations at both the system and segment level}. Experimental Results from the WMT22 metrics shared task validate the effectiveness of EAPrompt on various LLMs, with different structures. Further analysis confirms that EAPrompt effectively distinguishes major errors from minor ones, while also sharing a similar distribution of the number of errors with MQM. These findings highlight the potential of EAPrompt as a human-like evaluator prompting technique for MT evaluation.

1 Introduction

LLMs offer promising translation evaluation but standard prompting performs poorly at the segment level and lacks interpretable error information. EAPrompt addresses this gap through error analysis and shows strong system- and segment-level results across WMT22 scenarios.

  • Motivation: Standard LLM prompting can outperform existing MT metrics at the system level but performs poorly at the segment level and lacks interpretable error information.This limits its ability to provide human-like translation evaluation.
  • Approach: EAPrompt combines Chain-of-Thought and Error Analysis to emulate MQM by identifying major and minor translation errors before scoring them.The method separates error identification from translation scoring.
  • Results: GPT-3.5-Turbo prompted with EAPrompt outperforms all other metrics and prompting strategies at the system level.The authors describe this result as establishing a new state-of-the-art.
  • Results: EAPrompt surpasses GEMBA in 8 out of 9 test scenarios across language models and language pairs at the segment level.The comparison spans diverse model and language-pair settings.
  • Results: EAPrompt remains strong in reference-less settings, supporting its suitability for quality estimation tasks.These settings omit human reference translations.
  • Design recommendations: The recommended prompt uses a 2-step separated design with itemized error demonstrations, while regular expressions can reduce inference costs.The cost optimization replaces counting queries with automated counting.
  • Analysis: Further analysis finds that EAPrompt distinguishes major from minor errors and produces an error-count distribution close to MQM.This connects the model’s outputs to the human evaluation framework used for comparison.
  • Scope: EAPrompt is presented as an initial error-analysis approach that may extend to summarization and data-to-text evaluation.The paper frames these extensions as future applications rather than evaluated results.

2 Prompt LLMs with Error Analysis

Translation metrics score hypotheses using source, reference, and translation inputs, while MQM provides human error annotations. EAPrompt combines Chain-of-Thought and Error Analysis in a two-stage process that identifies and counts major and minor errors before converting them into a weighted score.

  • 2.1 Translation Evaluation Metric: Translation evaluation metrics assess machine translation quality using source, reference, and translation inputs, producing a score or rank for each hypothesis.Reference-less metrics omit the reference signal and support quality estimation.
  • 2.1 Translation Evaluation Metric: MQM verifies metric reliability by having human experts annotate hypothesis errors as Major or Minor according to severity.MQM is used as a high-quality human evaluation strategy in WMT.
  • 2.2 Prompt LLMs as Evaluation Metrics: GEMBA directly prompts an LLM to generate a translation-quality score and achieves state-of-the-art performance on GPT models, but its segment-level performance is poorer.This motivates combining direct scoring with error analysis.
  • 2.3 Error Analysis Prompting: Error Analysis enhances automatic scoring by explicitly identifying errors, while Chain-of-Thought prompts models to reason step-by-step instead of answering directly.Both paradigms are used to make evaluation more human-like and improve reasoning performance.
  • 2.3 Error Analysis Prompting: EAPrompt divides evaluation into two stages: identifying major and minor errors, then counting those errors.The design emulates MQM and aims to produce more explainable and reliable automatic evaluations.
  • 2.3 Error Analysis Prompting: The prompt uses one-shot examples, language-pair-specific demonstrations, itemized error responses, and a two-stage evaluation process.These modifications are intended to improve task understanding, error quantification, reliability, and inference efficiency.
  • 2.4 Post-processing of LLM responses: The final translation score is computed from the numbers of major and minor errors weighted by their respective severities.EAPrompt fixes w_minor = 1 and treats w_major as a latent variable because LLMs may apply different error criteria.

3 Experimental Results

Experiments on WMT22 evaluate EAPrompt across three language pairs, 54 MT systems, and 106,758 segments. EAPrompt achieves strong system- and segment-level performance, including state-of-the-art system-level results and consistent gains over GEMBA.

  • 3.1 Experiment Setup: The WMT22 test set covers En-De, En-Ru, and Zh-En across four domains, comprising 106,758 segments from 54 MT systems.Human judgments use MQM, with pairwise accuracy evaluated at the system level and tie-calibrated pairwise accuracy at the segment level.
  • 3.4 Inference Costs for EAPrompt: EAPrompt can require two LLM queries per translation, increasing inference costs relative to simpler prompting strategies.The paper notes that regular expressions may replace the scoring query to reduce costs.
  • 3.5 Experimental Results: GPT-3.5-Turbo with EAPrompt achieves state-of-the-art system-level performance, reaching 91.2% pairwise accuracy.EAPrompt surpasses other metrics and prompting strategies in the reported system-level comparison.
  • 3.5 Experimental Results: EAPrompt outperforms GEMBA in 8 of 9 tested segment-level scenarios across language models and language pairs.The strongest reported segment-level results are 56.7 for GPT-3.5-Turbo on En-De, 53.4 for GPT-3.5-Turbo on En-Ru, and 50.2 for Llama2-70b-Chat on Zh-En.
  • 3.5 Experimental Results: In reference-less settings, EAPrompt retains its advantage over GEMBA across all three LLMs at the system level and in 8 of 9 segment-level scenarios.The improvement is slightly lower than in reference-based settings.
  • 3.6 Ablation Study of Prompt Variants: Itemized error demonstrations and a 2-step separated prompt improve performance over detailed demonstrations and single-step prompting.The recommended variant separates error identification from scoring and uses itemized responses.

4 Analysis

The analysis finds that EAPrompt’s error distributions broadly align with MQM and that its scoring distinguishes major from minor errors. Replacing the second counting query with regular expressions can reduce inference costs with minimal performance change.

  • Error distribution: LLMs’ major-error distributions closely resemble MQM, while most minor-error distributions also remain consistent with MQM.Mixtral-8x7b-Instruct produces slightly more minor errors than the other tested LLMs.
  • Error weighting: When wmajor < 3, all three tested LLMs show substantial performance declines, indicating discrimination between major and minor errors.The analysis varies the major-error weight while fixing wminor = 1.
  • Error weighting: When wmajor ≥5, system- and segment-level accuracies fluctuate minimally, showing limited sensitivity to this scoring variable.
  • Inference cost: Regular-expression matching produces minimal system- and segment-level performance variation compared with the original counting query.The authors suggest this substitution when inference cost is a concern, while noting that patterns may need tailoring to each LLM’s response structure.

5 Related Work

Related work positions LLMs as translation evaluators alongside established MT metrics and prior evaluator prompting approaches. EAPrompt combines error analysis with chain-of-thought prompting to pursue human-like evaluation.

  • Translation evaluation metrics: Modern MT metrics such as COMET, MetricsX-XXL, BLEURT, and UniTE leverage human evaluations and high-quality data.
  • LLMs as evaluators: LLMs are used as evaluators for chatbot performance, self-critique, improvement procedures, summarization, and dialogue generation.
  • LLMs as evaluators: GEMBA introduced zero-shot GPT-based translation evaluation, while EAPrompt combines error analysis and chain-of-thought prompting for human-like evaluation.
  • Subsequent work: Subsequent work includes AutoMQM, GEMBA-MQM, InstructScore, and XCOMET, extending prompting or fine-tuning approaches for translation evaluation.

6 Conclusion

The paper presents EAPrompt as a chain-of-thought and error-analysis strategy for translation evaluation. It reports improved system- and segment-level performance, recommends a two-step variant, and finds alignment with MQM.

  • Conclusion: EAPrompt is a one-shot prompting strategy based on chain-of-thought and error analysis that improves evaluation at both system and segment levels.
  • Conclusion: The selected EAPrompt design uses two prompting steps with itemized error demonstrations.
  • Conclusion: Further analysis confirms EAPrompt’s proficiency in error identification and alignment with the human evaluation framework MQM.

Limitations

The study’s limitations concern possible test-data contamination, constrained exploration of prompt choices, and evaluation on only a limited range of LLMs.

  • Limitations: Residual contamination of WMT22 test data remains possible despite its selection to reduce training-data leakage risk.The authors caution that contamination may affect future comparison use of these datasets.
  • Limitations: Budget constraints limited the number of prompt choices explored, so the findings reflect initial experiments.The impact of additional prompt choices is left for future investigation.
  • Limitations: The study evaluates only a limited number of LLMs, and not all existing LLMs necessarily serve as reliable EAPrompt evaluators.The authors call for broader testing of model suitability.

Ethics Statement

The study states that it follows relevant ethical standards and limits EAPrompt to translation evaluation without generating harmful information.

  • The procedures are described as conforming to the Code of Ethics and applicable ethical standards.
  • EAPrompt evaluates translation quality and does not include statements intended to induce harmful information.
  • The method extracts and processes only numerical scores from model responses, further mitigating potential risks.

A Description of MQM

MQM is a human translation-evaluation framework that categorizes errors by severity and supplies expert annotations used as a reference for MT assessment.

  • MQM is commonly used as the WMT metrics shared tasks’ human-evaluation golden standard.
  • MQM categorizes translation errors into major and minor types to represent their severity.
  • WMT22 MQM annotations for En-De and Zh-En were produced by 11 professional translators, while En-Ru annotations used 4 professional native-language annotators.
  • Each segment receives annotations from 2 or 3 annotators, and its final score averages their scores.
  • Pairwise inter-rater agreement was 0.584 for En-De and 0.412 for Zh-En, exceeding agreement reported for Scalar Quality Metric and Direct Assessment.
  • EAPrompt emulates MQM by identifying major and minor errors and scoring translations according to their severity.

D Counting Errors using Regular Expressions Matching

The error-counting procedure segments EAPrompt responses, uses regular expressions to detect numbered major and minor errors, and records their totals.

  • The procedure first locates “major error” and “minor error” in the response and segments the response accordingly.
  • Regular-expression matching identifies initial error counts using formats such as “1.”, “1)”, and “(1)”.
  • The method records the resulting numbers of major and minor errors.
  • On GPT-4, EAPrompt outperformed GEMBA-DA by 1.05 points on a 600-sample subset.

E.2 Results at the system-level

At the system level, EAPrompt generally outperforms GEMBA across tested LLMs and language pairs, while the experiments also document practical evaluation biases and response variability.

  • Across all LLMs and three language pairs, EAPrompt consistently outperforms GEMBA in most system-level scenarios.
  • Discrepancies between Pearson correlation and pairwise accuracy are typically less than 1.0 point and may reflect Pearson’s sensitivity to low-quality MT systems.
  • Pearson correlation results show EAPrompt surpassing GEMBA by a significant margin across all considered scenarios.
  • Repeated generations can produce different scores, including 98, 95, and 100 for the same translation.
  • Presenting multiple translations together creates order bias, with reversed input order yielding an entirely different ranking sequence.
  • The authors recommend evaluating one translation at a time because decoder attention may favor later inputs.
  • LLMs may sometimes prioritize BLEU instead of producing judgments from their own evaluation capabilities.
Loading 2303.13809v4…