Source-linked AI summary
COMET: A Neural Framework for MT Evaluation
Ricardo Rei, Craig Stewart, Ana C Farinha, Alon Lavie
TL;DR
MT evaluation needs metrics that better reflect human judgments and distinguish high-performing systems. COMET trains multilingual neural evaluators from cross-lingual representations of translation inputs and different human-judgment types. Its models achieve state-of-the-art segment-level correlation and show improved differentiation of high-performing systems.
Problem
Existing MT metrics struggle to correlate accurately with human judgments at segment level and to differentiate the highest-performing systems.
Method
COMET uses pretrained cross-lingual models to build adaptable MT evaluation models that use source, hypothesis, and reference information and support multiple human-judgment types.
Results
COMET achieves new state-of-the-art segment-level correlation with human judgments and shows promising ability to better differentiate high-performing systems.
Takeaways & Limitations
The framework can serve as an automatic MT metric and be adapted to different types of human judgments of translation quality.
Takeaways & Limitations
BLEURT is limited to English target languages, while previously proposed learnable metrics focused on potentially noisy Direct Assessments.
Abstract
from arXiv · showhide
We present COMET, a neural framework for training multilingual machine translation evaluation models which obtains new state-of-the-art levels of correlation with human judgements. Our framework leverages recent breakthroughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three models with different types of human judgements: Direct Assessments, Human-mediated Translation Edit Rate and Multidimensional Quality Metrics. Our models achieve new state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems.
1 Introduction
COMET addresses weaknesses in traditional MT metrics by using multilingual neural models to predict human judgments from richer input. It targets better segment-level correlation and improved differentiation among high-performing systems.
- Traditional MT metrics mainly compare lexical overlap between an MT hypothesis and a human reference, enabling fast but limited evaluation.BLEU and METEOR count matching n-grams and remain popular because they are lightweight and fast.
- Neural MT quality increasingly deviates from monotonic lexical transfer, making BLEU less reliable for accurate quality estimation.
- MT evaluation still lacks a widely adopted modern standard despite substantial improvements in neural MT quality.The 2019 WMT News Translation Shared Task had 153 submissions, compared with 24 for the Metrics Shared Task.
- Current metrics struggle to correlate with human judgments at segment level and to differentiate the highest-performing MT systems.
- COMET trains multilingual, adaptable MT evaluation models using cross-lingual language modeling to predict Direct Assessments, HTER, and MQM judgments.
- Three COMET models show progress toward stronger segment-level correlation with human judgments and robustness to high-quality MT.
2 Model Architectures
COMET uses pretrained cross-lingual encoders and pooling to represent source, hypothesis, and reference segments, then supports regression- and ranking-based evaluation architectures. The models train either on quality scores or relative hypothesis comparisons using objectives tailored to those inputs.
- Model Architectures: The framework supports Estimator and Translation Ranking architectures that differ in training objective: direct score regression versus distance-based ranking.
- Cross-lingual Encoder: Pretrained cross-lingual encoders map source, hypothesis, and reference segments into a shared feature space, with XLM-RoBERTa used in the experiments.
- Pooling Layer: Layer-wise trainable weighting and layer dropout pool encoder information before average pooling produces a sentence embedding for each segment.
- Estimator Model: The Estimator concatenates hypothesis, reference, and embedding-combination features, then minimizes mean squared error against DA, HTER, or MQM assessments.
- Translation Ranking Model: The Translation Ranking model encodes source, reference, better hypothesis, and worse hypothesis segments, optimizing triplet margin loss relative to source and reference anchors.
- Estimator Model: The Estimator omits the raw source embedding because its added value was negligible, while source-combination features still improved correlation in experiments.Including the raw source embedding improved segment-level performance in 8 of 18 language pairs, with an average Kendall’s Tau gain of +0.0009 in those settings.
- Translation Ranking Model: At inference, Translation Ranking assigns a hypothesis score from the harmonic mean of its distances to the source and reference, then converts distance into a bounded similarity.
3 Corpora
COMET is trained on three corpora representing HTER, DARR, and MQM human judgments. These datasets vary in domains, language coverage, annotation sources, and quality labels.
- Corpora: Three COMET models target distinct human-judgment types using QT21, WMT DARR, and a proprietary MQM corpus.The models are trained for HTER, relative Direct Assessment rankings, and MQM scores, respectively.
- 3.1 The QT21 corpus: 173K tuples in QT21 pair sources, references, MT hypotheses, and post-edited translations across four language directions.HTER is computed between each MT hypothesis and its corresponding post-edited translation.
- 3.1 The QT21 corpus: HTER training learns f(s, h, r) → y to predict the human effort required to correct a hypothesis without observing its post-edited version.The inputs are the source, hypothesis, and reference; the target is the HTER score.
- 3.2 The WMT DARR corpus: WMT DARR data from 2017–2019 converts adequacy Direct Assessments into relative rankings of better and worse hypotheses across 24 language pairs.The supplied passages identify examples including Chinese-to-English and English-to-Gujarati.
- 3.3 The MQM corpus: 12K proprietary MQM tuples cover 12 English-source language pairs and include source, reference, hypothesis, and annotator-derived MQM scores.The corpus covers Style, Fluency, and Accuracy errors; English is never the target language.
- 3.3 The MQM corpus: MQM scores range from −∞ to 100 and weight minor, major, and critical errors with increasing severity.The metric is defined as MQM = 100 − IMinor + 5 × IMajor + 10 × ICrit.; training values were divided by 100 and truncated at 0.
4 Experiments
The experiments train separate Estimator and Translation Ranking models under distinct optimization setups, then evaluate COMET on the WMT 2019 Metrics Shared Task. The setup compares COMET with established and recent metrics using the official Kendall’s Tau-like evaluation.
- Training setup: COMET-HTER and COMET-MQM are Estimator models, while COMET-RANK is a Translation Ranking model trained on WMT DARR data.The Estimators use QT21 or the proprietary MQM corpus; COMET-RANK uses WMT DARR data from 2017 and 2018.
- Training setup: The two Estimators initialize a pretrained encoder, pooling layer, and feed-forward regressor, then train encoder and regressor parameter groups.Pooling-layer scalars start at zero, while feed-forward weights are randomly initialized.
- 4.2 Evaluation Setup: The official Kendall’s Tau-like score counts concordant and discordant hypothesis comparisons, including ties in the denominator.The supplied evaluation description defines the score using Concordant and Discordant pair counts.
- Motivation: Segment-level correlations were low across submitted metrics, and strong MT systems were difficult to rank correctly.The experiments therefore examine whether COMET better addresses these weaknesses, including performance on high-performing systems.
5 Results
COMET models consistently outperform established metrics across language directions and remain competitive on language pairs without English. Including the source input improves translation-ranking correlations, particularly for target-English pairs.
- COMET models outperform baseline and competing metrics across eight English-source language pairs, often by significant margins.
- The DARR Ranker outperforms both Estimators in seven of eight English-source language pairs.
- The MQM Estimator performs roughly on par with the HTER Estimator for most language pairs and surpasses other metrics for en-ru.
- The DARR model outperforms BLEURT in five of seven language pairs with English as the target.
- COMET results on language pairs without English are consistent with results on English-involving pairs.
- Including the source improves overall correlation with human judgments in the translation-ranking architecture and yields higher ∆τ for language pairs with English as target.
6 Reproducibility
The authors plan to release COMET, its trained evaluation models, and scripts for reproducing reported baselines. The implementation uses PyTorch-based tooling and was trained on a single Tesla T4 GPU.
- The COMET codebase and trained MT evaluation models will be released with scripts for running the reported baselines.
- All reported models were trained on a single Tesla T4 with 16GB of memory using PyTorch Lightning.
7 Related Work
Prior MT metrics range from fast n-gram matching and embedding-based similarity measures to learnable and reference-less approaches. The paper highlights limitations in semantic-only metrics and English-specific learnable metrics.
- Classic metrics such as BLEU, METEOR, and CHRF estimate quality using hand-crafted n-gram overlap features.
- Embedding-based metrics create soft alignments between references and hypotheses to measure semantic similarity.
- Human judgments such as DA and MQM capture more than semantic similarity, limiting the correlation achievable by semantic-similarity metrics.
- Learnable metrics directly optimize correlation with human judgments, while BLEURT reports state-of-the-art performance on recent WMT Metrics Shared Tasks.
- BLEURT is limited to evaluations where English is the target because it is built on English-BERT.
- Reference-less quality-estimation systems use multilingual pretrained encoders and have shown promising correlations with human judgments.
8 Conclusions and Future Work
COMET is a neural framework for adaptable MT evaluation models, and its three trained models achieve state-of-the-art segment-level correlation while better differentiating high-performing systems. The authors identify pretrained-model inference costs and COMET-RANK’s source/reference weighting as areas for future work.
- COMET is a neural framework for training MT evaluation models that can be adapted to different types of human judgements.
- Three distinct COMET models achieve new state-of-the-art results for segment-level correlation with human judgments.
- The models show promising ability to better differentiate high-performing MT systems.
- Pretrained models impose burdensome parameter counts and inference times, motivating future work on compact solutions such as DistilBERT.
- COMET-RANK weights source and reference differently during inference but equally in its training loss function, whose optimality remains for future investigation.
A Appendices
The appendices document COMET’s training hyper-parameters, corpus statistics, language-pair statistics, and performance tables for all and top MT systems. They also identify the metrics represented in the reported comparisons.
- Table 5 documents the hyper-parameters used to train the presented COMET models.
- Tables 6–11 provide statistics for the QT21, WMT 2017 DARR, WMT 2018 DARR, WMT 2019 DARR, and MQM corpora.
- Tables 12 and 13 report metrics performance over all and the top 10, 8, 6, and 4 MT systems for from-English and into-English language sets.
- The listed comparison metrics include COMET-RANK, COMET-HTER, COMET-MQM, BLEU, BERTSCORE, and BLEURT.