Source-linked AI summary
Large Language Models Are State-of-the-Art Evaluators of Translation Quality
Tom Kocmi, Christian Federmann
TL;DR
The paper addresses whether large language models can provide effective translation-quality assessment. It proposes GEMBA, a zero-shot GPT-based metric operating with or without reference translations, and reports state-of-the-art system-level accuracy on WMT22 data, with effective assessment limited to GPT 3.5 and larger models.
Problem
The paper asks whether large language models can be used for effective quality assessment of translations.
Method
GEMBA uses zero-shot GPT prompting to assess translation quality with or without a human reference, using four scoring and classification prompt variants.
Results
GEMBA achieves state-of-the-art system-level performance on WMT22 MQM data across English–German, English–Russian, and Chinese–English, with capability appearing only for GPT 3.5 and larger models.
Takeaways & Limitations
The results provide an initial indication that generative pretrained large language models can assess translation quality.
Takeaways & Limitations
The state-of-the-art result holds only at system level and is based on human labels for three language pairs, while further analysis is needed before primary use for translation-quality decisions.
Abstract
from arXiv · showhide
We describe GEMBA, a GPT-based metric for assessment of translation quality, which works both with a reference translation and without. In our evaluation, we focus on zero-shot prompting, comparing four prompt variants in two modes, based on the availability of the reference. We investigate nine versions of GPT models, including ChatGPT and GPT-4. We show that our method for translation quality assessment only works with GPT~3.5 and larger models. Comparing to results from WMT22's Metrics shared task, our method achieves state-of-the-art accuracy in both modes when compared to MQM-based human labels. Our results are valid on the system level for all three WMT22 Metrics shared task language pairs, namely English into German, English into Russian, and Chinese into English. This provides a first glimpse into the usefulness of pre-trained, generative large language models for quality assessment of translations. We publicly release all our code and prompt templates used for the experiments described in this work, as well as all corresponding scoring results, to allow for external validation and reproducibility.
1 Introduction
This study explores whether large language models can assess translation quality and introduces GEMBA, a GPT-based metric evaluated through zero-shot prompting with and without reference translations. It reports state-of-the-art system-level performance on WMT22 data, with effective assessment emerging for GPT 3.5 and larger models.
- The study asks whether large language models can effectively assess translation quality.
- GEMBA evaluates each segment translation independently and averages the scores into a final system-level score.
- The evaluation tests zero-shot translation-quality assessment with and without a human reference translation.
- GEMBA achieves state-of-the-art system-level performance against MQM-based human labels across English–German, English–Russian, and Chinese–English WMT22 data.
- Four prompt templates are compared, and the least constrained template achieves the best performance.
- GPT 3.5 and larger models are capable of translation quality assessment across the tested prompt variants.
2 The GEMBA Metric
GEMBA assesses translation quality by prompting GPT models on individual segments and averaging their scores, using reference-based or reference-free prompts. The paper explores four prompt types, two reference modes, and multiple GPT models, finding that GPT 2 is not useful for this task.
- The prompts require source and target language names, source segments, candidate translations, and optionally human reference translations.
- GEMBA evaluates each translation segment independently and averages the resulting scores into a system-level score.
- Four prompt types comprise two scoring tasks, GEMBA-DA and GEMBA-SQM, and two classification tasks, GEMBA-stars and GEMBA-classes.The scoring prompts output values in [0, 100], while stars and classes use five-level outputs.
- The metric supports two modes: reference-based assessment with a human translation and reference-free quality estimation.Reference-free variants are marked with the suffix "[noref]".
- The study evaluates GPT models ranging from GPT 2 to GPT-4, using GPT-4 as the default model for most experiments.The models are ordered by estimated power or release date.
- GPT 2 is found not to be useful for translation quality assessment.
3 Experiments
The experiments benchmark GEMBA against established WMT22 metrics using MQM human judgments across three translation directions. They evaluate system-level pairwise accuracy and segment-level Kendall’s Tau, with the latest tie-adjusted definition used in the experiments.
- The evaluation follows the WMT22 Metrics shared-task methodology and compares GEMBA with COMET, BLEURT, and MetricX XXL.
- The MQM 2022 test set covers English–German, English–Russian, and Chinese–English translation, with 54 system or human outputs and 106k segments.
- Human MQM ratings from professional annotators provide the gold standard across news, social, conversational, and e-commerce domains.
- System-level evaluation uses pairwise accuracy, defined as correctly ranked system pairs divided by all system-pair comparisons.
- Segment-level evaluation uses Kendall’s Tau, with Concordant and Discordant comparisons representing agreement and disagreement with human rankings.
- The experiments use the latest Kendall’s Tau-b definition, which adjusts for ties; ties are especially relevant for metrics producing discrete scores.
4 Results
GEMBA achieves strong system-level translation-quality assessment with and without references, while segment-level performance is more mixed because its discrete scores produce frequent ties.
- System-level evaluation: GEMBA-DA is evaluated in both reference-based and reference-less modes using system-level pairwise accuracy on WMT22 data.The reference-less mode is treated as quality estimation, while the reference-based mode uses a human reference translation.
- Reference-based evaluation: Reference-based GEMBA-GPT4-DA sets a new state of the art and outperforms all other reference-based WMT22 metrics.The comparison uses noisy MQM-based human labels, so perfect automatic accuracy is not expected.
- Quality estimation: Reference-less GEMBA-GPT4-DA[noref] achieves the highest quality-estimation performance and strongly outperforms other reference-less metrics.It also surpasses the other reference-based metrics and is only slightly worse than GEMBA-GPT4-DA.
- Comparison of GPT models: GPT-2 and Ada fail to produce meaningful zero-shot scoring replies, while Babbage and Curie perform close to random guessing.The authors conclude that these smaller models are not powerful enough for the tested prompts.
- Comparison of GPT models: The main performance jump occurs with GPT-3.5 and larger models, which achieve highly competitive results across all tested prompt variants.GPT-4 performs best overall, and successive model generations generally improve performance.
- Segment-level performance: GPT-4 remains slightly behind top-performing metrics at segment level, while GEMBA-Dav3-DA[noref] performs significantly worse than other leading metrics.Segment-level results are reported with Kendall’s Tau for each language pair.
- Segment-level performance: Frequent segment-level ties may reduce Kendall’s Tau because GEMBA returns discrete scores between 0 and 100.The authors conjecture that the discrete scale and tie frequency contribute to lower segment-level correlation.
5 Conclusion
The paper presents GEMBA as a GPT-based translation-quality assessment method and reports state-of-the-art system-level performance across three WMT22 language pairs. The authors propose extending this work with few-shot prompting, fine-tuning, and broader evaluation applications.
- Conclusion: GEMBA achieves state-of-the-art performance across English–German, English–Russian, and Chinese–English system-level evaluations.The comparison is against automated metrics from the WMT22 Metrics shared task.
- Future work: Future work will investigate few-shot prompting, model fine-tuning, MQM error-based evaluation, and post-editing-oriented prompts.The authors state that these directions may improve GEMBA accuracy.
- Future work: GPT-enhanced metrics may support document-level evaluation through larger context windows, an area with limited existing research.The passage presents this as a potential benefit rather than an established result.
Limitations
The evaluation covers only three language pairs, and the authors caution that GEMBA may perform worse for other, especially under-resourced, languages. Its state-of-the-art result is limited to system-level evaluation, and further analysis is needed before relying on it for translation-quality decisions.
- Three language pairs provide the human-label basis for the reported results, limiting how broadly GEMBA’s performance can be generalized.
- Under-resourced language pairs may yield lower metric performance, consistent with previously reported weaknesses in translation quality for such languages.
- State-of-the-art performance holds only at the system level; segment-level scores still have room for improvement.
- Further analysis is needed before GEMBA is used as the main tool for deciding translation quality.
- The study cannot definitively establish that its evaluation data were absent from OpenAI’s training dataset.
A Appendix: Prompt Templates
The appendix provides prompt templates for four GEMBA variants, spanning continuous scores, star ratings, and discrete quality classes, with reference-conditioned inputs where available.
- A Appendix: Prompt Templates: Prompt templates use bold portions only when a human reference translation is available.
- A.1 DA: Direct Assessment: GEMBA-DA outputs a continuous translation-quality score from 0 to 100 relative to a human reference.
- A.1 DA: Direct Assessment: The GEMBA-DA template supplies source text, human reference, and candidate translation before requesting a score.
- A.2 SQM: Scalar Quality Metrics: GEMBA-SQM also outputs scores from 0 to 100, using ordered meaning-preservation and grammar-quality descriptions.
- A.2 SQM: Scalar Quality Metrics: The GEMBA-SQM template presents the source, reference, and translation, then requests a score on the 0-100 scale.
- A.3 Stars: One to Five Stars Ranking: GEMBA-stars ranks translations from one to five stars, with labels ranging from no meaning preserved to perfect meaning and grammar.
- A.3 Stars: One to Five Stars Ranking: The stars template includes source, reference, and translation fields before requesting the star rating.
- A.4 Classes: Quality Class Labels: GEMBA-classes assigns one of five discrete labels, from no meaning preserved to perfect translation, using the source, reference, and translation.
B Appendix: Full Results
The appendix lists full GEMBA results and identifies the evaluation measures used for system-level and segment-level assessment.
- Table 7 reports system-level pairwise accuracy and segment-level Kendall Tau scores.
C Appendix: Ada Answers
The appendix illustrates invalid Ada-model responses across prompts, with SQM prompts closer to expected outputs than corresponding DA prompts; GPT 2 showed similar behavior.
- Ada frequently produced invalid answers across the tested prompts, and GPT 2 exhibited similar behavior.
- SQM prompts produced answers closer to the expected output format than corresponding DA prompts for Ada.