PaperlayerView exact revision on arXiv

Source-linked AI summary

xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection

Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, André F. T. Martins

arXiv:2310.10482v1cs.CL

TL;DR

Existing learned metrics mainly provide opaque sentence-level scores, while granular LLM-based evaluation targets explicit error spans. xCOMET bridges these approaches with a unified metric that predicts scores and severity-labeled spans, achieving state-of-the-art results across evaluation levels and identifying most localized critical errors and hallucinations.

  • Problem

    Learned metrics provide limited insight into which translation spans are erroneous and their severity, while generative evaluation remains less competitive with dedicated learned metrics.

  • Method

    xCOMET jointly performs sentence-level regression and error span detection using a curriculum over public sentence- and span-level data plus synthetic robustness data.

  • Results

    xCOMET achieves state-of-the-art performance across sentence-level, system-level, and error span prediction evaluation and generally outperforms neural metrics and generative LLM-based evaluation.

  • Takeaways & Limitations

    Predicted error spans provide detailed, MQM-aligned signals that complement sentence-level scores and support interpretation of translation quality.

  • Takeaways & Limitations

    The authors note that whether observed hallucination-detection trends hold for other, including low-resource, language pairs remains to be explored.

Abstract

from arXiv · show

Widely used learned metrics for machine translation evaluation, such as COMET and BLEURT, estimate the quality of a translation hypothesis by providing a single sentence-level score. As such, they offer little insight into translation errors (e.g., what are the errors and what is their severity). On the other hand, generative large language models (LLMs) are amplifying the adoption of more granular strategies to evaluation, attempting to detail and categorize translation errors. In this work, we introduce xCOMET, an open-source learned metric designed to bridge the gap between these approaches. xCOMET integrates both sentence-level evaluation and error span detection capabilities, exhibiting state-of-the-art performance across all types of evaluation (sentence-level, system-level, and error span detection). Moreover, it does so while highlighting and categorizing error spans, thus enriching the quality assessment. We also provide a robustness analysis with stress tests, and show that xCOMET is largely capable of identifying localized critical errors and hallucinations.

1 Introduction

xCOMET bridges sentence-level scoring and fine-grained error analysis by predicting both translation quality and error spans with severity. It reports state-of-the-art performance across sentence-, system-, and span-level evaluation while improving interpretability.

  • Learned metrics such as COMET and BLEURT correlate well with human judgments but usually provide only a single, difficult-to-interpret sentence-level score.
  • xCOMET combines sentence-level evaluation with error span detection to provide a more detailed view of translation errors.
  • Its curriculum uses high-quality public data at sentence and error-span levels, supplemented with synthetic data to improve robustness.
  • xCOMET is a unified metric that supports source-only, reference-only, and source-plus-reference evaluation within one model.
  • xCOMET achieves state-of-the-art performance across sentence-level, system-level, and error span prediction evaluation, generally outperforming neural metrics and generative LLM-based evaluation.
  • Stress tests show that xCOMET identifies the vast majority of localized critical errors and hallucinations.

4. We release two evaluation models: xCOMET-XL, with 3.5B parameters, and xCOMET-XXL, featuring 10.7B parameters.1

The full xCOMET metric suite will be released through the HuggingFace Hub.

  • The full xCOMET metric suite will be released through the HuggingFace Hub.

2 Background

Machine translation quality is assessed through human methods and automatic metrics, each offering different levels of detail and interpretability. Recent work increasingly combines neural scoring with fine-grained error analysis and generative evaluation.

  • Human MT evaluation commonly uses post-edits, direct assessments, and the MQM framework.
  • Post-editing evaluates translation quality by quantifying the edits professional translators make to improve an output.
  • Direct assessment asks annotators to score translation adequacy and fluency from 0 to 100.
  • MQM annotates translation error spans with severity ratings and categorical labels, enabling more detailed error analysis.
  • Evaluation has increasingly favored MQM comparisons over traditional direct-assessment and post-editing methodologies.
  • Lexical metrics rely on overlap statistics, while neural metrics use complex networks to estimate translation quality.
  • Neural metric scores can be difficult to explain, motivating efforts toward interpretable neural metrics and error-focused evaluation.
  • Generative LLM approaches can directly score translations or identify and categorize their errors.

3 Problem Statement

MT evaluation models assess a translation against source and/or reference context at sentence or span granularity. xCOMET unifies these input modes and produces both quality scores and MQM-style error annotations.

  • Neural MT metrics evaluate translation t using a reference r, source s, or both as contextual input.
  • The REF scenario uses the translation and reference, whereas SRC+REF jointly uses source and reference.
  • The SRC scenario uses only the source and is commonly called quality estimation or reference-free evaluation.
  • Sentence-level prediction produces one global score representing how well a translation aligns with its source and/or reference context.
  • Span-level prediction identifies potentially erroneous words or phrases and assigns severity labels such as minor, major, or critical.
  • Its architecture feeds pooled [cls] representations to a quality-score head and translation subwords to a severity-classification layer.
  • xCOMET simultaneously supports SRC, REF, and SRC+REF evaluation while providing sentence-level scores and MQM-aligned error spans.

4 Design and Methodology of xCOMET

xCOMET is a unified multi-task metric that combines sentence-level regression with token-level error detection across source, reference, and combined-input evaluation settings. It derives interpretable error spans and MQM-based scores while training on DA and MQM data through a staged curriculum.

  • Architecture: xCOMET uses a large pretrained encoder with separate sentence-level regression and word-level sequence-tagging heads.The regression head predicts a sentence score, while the tagger assigns severity labels to translation tokens.
  • Architecture: A unified input design supports SRC, REF, and SRC+REF evaluation scenarios within one model.When a reference is available, separate forward passes produce sentence-level and word-level predictions for the evaluation scenarios.
  • Inference: Predicted token tags are grouped into contiguous error spans, whose severity equals the most severe tag within each span.The model averages word-level output distributions across forward passes before constructing spans.
  • Inference: xCOMET combines direct sentence predictions with an inferred MQM score derived from severity-weighted error counts.The inferred MQM score is bounded between 0 and 1, with 1 representing a perfect translation, and is included in the final weighted sum.
  • Corpora: Training uses publicly available DA and MQM annotations spanning about 1 million samples and 36 language pairs, augmented with synthetic critical errors.The MQM corpus includes WMT, IndicMT, and DEMETR annotations; synthetic hallucinations address underrepresented pathological translations.
  • Score scaling: DA scores require min-max scaling because z-normalized DA values are unbounded, whereas MQM-derived sentence scores lie between 0 and 1.The scaling procedure uses practical minimum and maximum z-score values based on unanimously scored translations.
  • Training: The models use a three-phase curriculum that shifts emphasis from DA sentence regression to MQM word supervision and then reunifies both tasks.The final phase increases emphasis on sentence-level prediction using high-quality MQM data.

5 Experimental Setting

The experiments evaluate xCOMET on WMT 2022 News MQM annotations across three language pairs and three evaluation levels. Comparisons include neural metrics and prompted GPT models, using resampling-based significance testing and character-level span F1.

  • Data: The test set covers Chinese→English, English→German, and English→Russian, with segments drawn from 15 translation systems.It contains 7,575 zh-en segments and 4,500 segments for each of en-de and en-ru.
  • Metrics: Evaluation measures sentence-level Pearson ρ and Kendall’s τ, system-level pairwise accuracy, and character-level F1 for error spans.Span F1 accounts for partial matches involving minor and major errors.
  • Statistical testing: Sentence-level results report significance using 200 resampling runs and a significance threshold of p = 0.05.The Perm-Both hypothesis test is used for Pearson and Kendall evaluations.
  • Baselines: Sentence- and system-level benchmarks include COMET-22, BLEURT-20, METRICX, and GPT-4-based GEMBA.METRICX is identified as the best-performing metric from the WMT22 Metrics shared task.
  • Baselines: Error-span comparisons use GPT-3.5 and GPT-4 prompted in the style of AUTOMQM, with five fixed shots across samples.The fixed examples enable direct comparison with state-of-the-art LLM-based error detection.

6 Correlations with Human Judgements

xCOMET achieves strong performance across sentence-level, system-level, and error-span evaluation, while its predicted error spans also support interpretable sentence-level scoring.

  • Overall performance: xCOMET is state-of-the-art for sentence-level and error-span prediction and competitive with generative LLMs for system-level evaluation.The comparison spans all three evaluation settings described in the section.
  • Sentence-level evaluation: xCOMET-XXL sets a new state-of-the-art for en-de and en-ru sentence-level evaluation, while xCOMET-XL remains very competitive.xCOMET-XL outperforms METRICX despite METRICX using a larger model than xCOMET-XXL.
  • Span-derived scoring: MQM scores inferred exclusively from predicted error spans outperform BLEURT-20 and COMET-22, providing high-quality sentence-level scores.Table 4 also shows strong correlations between xCOMET score types and MQM scores derived from error spans.
  • System-level evaluation: xCOMET-XXL remains competitive with GEMBA using GPT4 across all language pairs in system-level evaluation.MQM scores inferred exclusively from predicted error spans also show very competitive system-level accuracy.
  • Error span prediction: xCOMET error-span predictions outperform GPT3.5 and GPT4, with performance close to GPT4 even without a reference.The models achieve this despite having smaller capacity than the generative models.
  • Span-derived scoring: The strong correlation between span-derived MQM scores and regression outputs suggests the shared encoder aligns sentence-level and word-level tasks during multi-task training.The paper notes that no explicit alignment mechanism governs this relationship.

7 Robustness of xCOMET to pathological translations

xCOMET is stress-tested on localized critical errors and hallucinations, showing strong detection of both while exposing some score-sensitivity and benchmark limitations.

  • 7.1 Localized errors: Synthetic perturbations target addition, negation, mask in-filling, named entities, and numbers in WMT 2023 data.SMAUG corrupts synthetic references that previously received no predicted errors, isolating perturbation effects.
  • 7.1 Localized errors: xCOMET predicts most localized errors as major or critical, especially negation errors and number mismatches.The authors connect these gains to training data containing negative translations, including synthetic hallucinations.
  • 7.1 Localized errors: Median sentence-level score decreases are around 20 points for negation, number, and named-entity perturbations.Rare increases occur, but over 90% of those cases have decreases smaller than 1 point.
  • 7.2 Hallucinations: The hallucination analysis uses a German-to-English benchmark with over 3.4k translations spanning omissions, named-entity errors, and several hallucination types.The benchmark is designed to test whether severe hallucinations rank below other severe errors and incorrect translations.
  • 7.2 Hallucinations: xCOMET largely ranks hallucinations below other errors, with fully detached hallucinations exceeding 95 AUROC for xCOMET-XXL.Over 90% of fully detached hallucinations receive xCOMET-XXL scores under 10.
  • 7.2 Hallucinations: Reference-free evaluation also benefits in this benchmark, while overlapping hallucinated text can receive higher scores when a reference is available.The authors hypothesize that reference-based metrics may over-rely on reference overlap.

8 Conclusions

The paper concludes that xCOMET combines sentence-level and fine-grained error-span prediction into a state-of-the-art evaluation suite. Stress tests show broad localized-error detection and appropriate hallucination penalties, while broader hallucination coverage remains future work.

  • 8 Conclusions: xCOMET combines sentence-level prediction with fine-grained error-span prediction for machine translation evaluation.The suite uses predicted spans both as downstream signals and to interpret sentence-level predictions.
  • 8 Conclusions: xCOMET achieves state-of-the-art performance across sentence-level, system-level, and error-span evaluation.The conclusion also reports that the predicted spans provide a lens for understanding and interpreting model predictions.
  • 8 Conclusions: Stress tests find that xCOMET identifies the vast majority of localized errors and appropriately penalizes hallucination severity.
  • 8 Conclusions: The full xCOMET-XL and xCOMET-XXL suite is intended for release through the HuggingFace Hub.
  • 8 Conclusions: The conclusion identifies evaluation on other language pairs, including low-resource directions, as a future need because the HalOmi benchmark was not yet publicly released.

Supplemental Material

The supplemental material documents synthetic hallucination examples and training configurations, including phase-specific loss weighting and class weights for error labels.

  • Supplemental Material: Synthetic hallucination examples are provided for Phases II and III of xCOMET training.
  • Supplemental Material: Phase I, II, and III training configurations are documented through relevant hyperparameters and listings.The configurations use the COMET framework and share most settings across phases.
  • Supplemental Material: The main difference between Phases II and III is the loss λ parameter, tuned with Optuna for phase-specific objectives.The reported values are closer to 1 for Phase II and closer to 0 for Phase III.
  • Supplemental Material: Class weights are optimized with Optuna and fixed throughout Phases II and III, assigning the smallest weight to OK and the largest to MAJ and CRIT.

C.1 Data construction

The supplemental data-construction material describes SMAUG-generated localized errors and hallucinations, their training configurations, and additional examples and perturbation analyses for xCOMET.

  • C.1 Data construction: SMAUG creates localized pathologies including added-text hallucinations, negation errors, mask in-filling, number swaps, and named-entity swaps.The material summarizes examples and category counts for zh-en and he-en data.
  • C.1 Data construction: Addition and mask-in-filling perturbations can produce fluent paraphrase-like translations, increasing the percentage of minor errors.SMAUG encourages inserted or replacement tokens to preserve fluency.
  • C.1 Data construction: Phase I training uses the COMET framework with source, translation, and reference inputs, while Phases II and III enable word-level training.
  • C.1 Data construction: The phase configurations retain shared optimization settings while changing the loss λ parameter and using minor, major, and critical error labels.
  • C.1 Data construction: Additional SMAUG perturbation results are reported for zh-en and he-en with both xCOMET-XL and xCOMET-XXL.
  • C.1 Data construction: The supplemental material includes hallucination examples alongside xCOMET-predicted error spans and examples from the hallucination benchmark.
Loading 2310.10482v1…