Source-linked AI summary
GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4
Tom Kocmi, Christian Federmann
TL;DR
Translation-quality evaluation needs reference-free error-span detection that can work across languages. GEMBA-MQM uses GPT-4 with fixed three-shot MQM prompting and language-agnostic examples, and preliminary evaluations report leading system-ranking performance. The authors caution that proprietary GPT-4 makes results difficult to verify, reproduce, and compare over time.
Problem
The paper addresses reference-free translation-quality error detection and the need to evaluate translation systems across language pairs without manual prompt preparation.
Method
GEMBA-MQM uses GPT-4 with a fixed three-shot prompting strategy to mark translation-quality error spans under the MQM framework.
Results
GEMBA-MQM outperforms other techniques on the three evaluated MQM23 languages for system ranking and outperforms traditional metrics on MQM22 data.
Takeaways & Limitations
GEMBA-MQM provides a language-agnostic GPT-based metric for translation-quality error marking and system ranking.
Takeaways & Limitations
GEMBA-MQM depends on proprietary GPT-4, whose training data, deployment continuity, and performance stability are unknown; evaluations focus on high-resource languages.
Abstract
from arXiv · showhide
This paper introduces GEMBA-MQM, a GPT-based evaluation metric designed to detect translation quality errors, specifically for the quality estimation setting without the need for human reference translations. Based on the power of large language models (LLM), GEMBA-MQM employs a fixed three-shot prompting technique, querying the GPT-4 model to mark error quality spans. Compared to previous works, our method has language-agnostic prompts, thus avoiding the need for manual prompt preparation for new languages. While preliminary results indicate that GEMBA-MQM achieves state-of-the-art accuracy for system ranking, we advise caution when using it in academic works to demonstrate improvements over other methods due to its dependence on the proprietary, black-box GPT model.
1 Introduction
GEMBA-MQM extends LLM-based translation evaluation from segment scoring and error classification to MQM error-span marking. It uses streamlined, language-agnostic prompting, while related approaches use zero-shot or language-specific few-shot strategies.
- GEMBA-MQM builds on evidence that LLMs can assess machine-translation quality.
- GEMBA-DA used zero-shot segment scoring and surpassed non-LLM metrics on the WMT22 metrics test set.
- EAPrompt prompted LLMs to classify MQM error spans by class, subclass, and severity, then aggregated weighted errors into segment scores.
- GEMBA-MQM differs from EAPrompt through single-step prompting and prompts applicable across languages without manual language-pair preparation.
- Fernandes et al. used PaLM-2 with pooled MQM annotations for few-shot examples, but fine-tuning did not improve top-tier system-level performance.
- The general GEMBA-MQM prompt omits a component that performed subpar internally, whereas GEMBA-locale-MQM includes it and repeats user-assistant sections for each few-shot example.
2 Description
GEMBA-MQM uses GPT-4 few-shot prompting to mark translation-quality error spans under the MQM framework. Three predetermined examples make the prompts usable across language pairs without manual language-specific examples.
- GEMBA-MQM prompts GPT-4 to mark translation-quality error spans using the MQM framework and a template modeled on human-annotation guidelines.
- Three predetermined examples allow GEMBA-MQM to operate across language pairs without creating language-pair-specific MQM few-shot examples.The design prioritizes cross-language evaluation rather than performance gains from domain- or language-specific prompts.
3 Experiments
The experiments evaluate GEMBA-MQM on WMT and internal datasets, emphasizing system-level pairwise accuracy and broader meta-evaluation scenarios. The design also examines blind, unseen data and discusses limitations of Pearson correlation for metric evaluation.
- Datasets and comparisons: The evaluation uses WMT22 Metrics shared-task data, comparing GEMBA-MQM with reference-based, string-based, reference-less, and LLM-based metrics.
- Datasets and comparisons: Experiments include MQM22, the blind MQM23 test set, and internal Microsoft data, with MQM23 designed to prevent participant and LLM overfitting.The internal set covers 15 high-resource languages paired with English.
- Evaluation methods: System-level pairwise accuracy measures the proportion of system pairs ranked correctly relative to human rankings.
- Evaluation methods: The Metrics 2023 meta-evaluation combines four equally weighted scenarios: system-level pairwise accuracy, system-level Pearson correlation, segment-level Accuracy-t, and segment-level Pearson correlation.
- Evaluation methods: The paper questions Pearson correlation because small samples, outliers, and non-linear or discrete MQM scores can affect its evaluation behavior.MQM23 may contain as few as 12 systems, while segment-level error weights include 0.1, 1, 5, and 25.
- Evaluation methods: The authors conclude that more research is needed to determine how to test metrics intended for segment-level evaluation and correlation with human annotation behavior.
4 Results
Across WMT and internal evaluations, GEMBA-MQM generally performs strongly for system ranking, but results vary by test set and prompt design. Removing the locale convention error class addresses a documented GPT misclassification pattern, though behavior differs between internal and external data.
- MQM23 blind test set: GEMBA-MQM outperforms all other techniques on the three MQM23 languages in system ranking, while achieving the third cluster rank in meta-evaluation.These results come from the blind MQM23 test set.
- MQM22 test set: On MQM22, all GEMBA-MQM variants outperform traditional metrics such as COMET or Metric XXL.GEMBA-locale-MQM-Turbo slightly outperforms EAPrompt in quality estimation.
- Internal test set: On the internal test set, GEMBA-MQM-Turbo outperforms almost all metrics, losing only to COMETKIWI-22.The internal evaluation covers all 15 languages paired with English and a Czech–German subset.
- Locale convention: The locale convention prompt assigned errors to correct translations, including Czech sentences using euros, indicating misuse of this error class.The authors assume GPT marked text as nonstandard for the language, but state that deeper conclusions require more investigation.
- Cross-dataset behavior: Internal data showed a 1.7% accuracy gain, but performance degraded by 0.2% across 15 languages and degraded even more on MQM22.The authors note different behavior between internal and external test data.
- Locale convention: Locale convention represented 32% of errors for GEMBA-locale-MQM across fifteen high-resource languages, so the authors removed the class and reassigned or omitted those errors.About half of the original locale errors were reassigned to other classes, while the other half were not marked.
5 Caution with “Black Box” LLMs
The paper cautions that GPT-4’s proprietary, black-box nature complicates academic evaluation because its training data and future availability are uncertain. Results may therefore be tainted or difficult to reproduce over time.
- GPT-4’s undisclosed training data means published test data may have been included in its training, potentially tainting evaluations.
- Because GPT-4 may become unavailable or change, results from the model are relevant only to the specific sampling time.
- The authors advocate greater use of publicly available models to enable comparisons with both black-box and open models.
6 Conclusion
The paper introduces GEMBA-MQM, a GPT-based metric using fixed three-shot prompting to mark translation-quality error spans. Preliminary results report state-of-the-art system ranking and performance beyond established metrics, but the method depends on proprietary GPT models.
- GEMBA-MQM uses GPT-4 with a fixed three-shot prompting strategy to mark translation quality errors.
- Preliminary results show GEMBA-MQM achieves state-of-the-art system ranking, outperforming established metrics such as COMET and BLEURT-20.
- The authors recommend caution when using GEMBA-MQM with proprietary GPT models and plan to evaluate the approach with more open LLMs.
Limitations
The study’s main limitations are reliance on a proprietary and changing GPT-4 model and evaluation concentrated on high-resource languages. These constraints limit conclusions about reproducibility and low-resource-language performance.
- GEMBA-MQM depends on proprietary GPT-4, whose training data and deployed version are unknown, making result comparability uncertain.
- GPT-4 performance fluctuated throughout 2023, further limiting confidence that results remain stable across evaluations.
- Because WMT evaluations primarily cover high-resource languages, the study cannot conclude whether GEMBA-MQM performs well on low-resource languages.
A Three examples Used for Few-shot Prompting
The appendix presents three translation examples used for few-shot prompting across languages. They illustrate MQM error-span annotation, including severity and error categories in translated outputs.
- The examples include English-to-German and English-to-Czech translations annotated with MQM error spans.
- The Chinese-to-English example marks critical, major, and minor errors involving addition, mistranslation, and awkward style.
- Figure 2 contains three examples used for all languages in the few-shot prompting setup.