COMET: A Neural Framework for MT Evaluation

1 Introduction

• Traditional metrics compare MT hypotheses with target-language references using lexical features; BLEU and METEOR remain popular, but lexical overlap can misestimate modern neural translation quality.Source: §1 · p. 1
• MT evaluation has lagged behind neural MT advances, with no widely adopted new standard; WMT 2019 had 153 translation submissions but only 24 Metrics submissions, nearly half adapted from Quality Estimation.Source: §1 · p. 1
• WMT 2019 Metrics findings identify two challenges: weak segment-level correlation with human judgments and difficulty distinguishing the highest-performing MT systems.Source: §1 · p. 1
• COMET trains multilingual, adaptable MT evaluation models from cross-lingual language representations to predict Direct Assessments, HTER, and Multidimensional Quality Metrics judgments.The framework is PyTorch-based.Source: §1 · p. 1
• COMET adds source-language input to reference-based MT evaluation, building on QE work without references and multilingual embeddings that use all three inputs.The paper reports demonstrating the added value of source input.Source: §1 · pp. 1–2
• The authors train three models for different human judgments and report progress toward better segment-level correlation and robustness to high-quality MT.Source: §1 · p. 2

2.1 Cross-lingual Encoder

• XLM-RoBERTa (base) encodes the source, MT hypothesis, and reference into a shared feature space using a pretrained cross-lingual model.The encoder is pretrained to reconstruct masked tokens from their context; the framework identifies multilingual BERT, XLM, and XLM-RoBERTa as examples of such encoders.Source: §2.1 · p. 2

2.2 Pooling Layer

• Layer-wise attention combines information from encoder layers for each token, layer dropout reduces reliance on any single layer, and average pooling yields a sentence embedding.The layer weights are trainable, and dropout sets a selected layer’s weight to −∞ with probability p.Source: §2.2 · p. 2

2.3 Estimator Model

• The Estimator concatenates hypothesis and reference embeddings with hypothesis–source and hypothesis–reference interaction and difference features, then regresses quality scores using mean squared error.The raw source embedding is excluded because its added value was negligible in early experiments; including it improved segment-level performance in 8 of 18 language pairs, with an average Kendall’s Tau improvement of +0.0009, while source combination features were retained.Source: §2.3 · p. 3, Figure 1 · p. 3

2.4 Translation Ranking Model

• The Translation Ranking model trains on source, reference, better-hypothesis, and worse-hypothesis embeddings, using triplet margin loss to bring the better hypothesis closer to both anchors.At inference, it scores a single hypothesis using the harmonic mean of its distances to source and reference, then converts that distance into a similarity bounded between 0 and 1.Source: Figure 2 · p. 3, §2.4 · p. 4

3.1 The QT21 corpus

• QT21 provides 173K tuples from four language directions, pairing source, reference, MT hypothesis, and post-edit in IT or life-sciences domains.HTER is computed between the hypothesis and post-edit; the model predicts correction effort from the source, hypothesis, and reference without seeing the post-edit.Source: §3.1 · p. 4

3.2 The WMT DARR corpus

• WMT adequacy direct assessments from 2017–19 are mapped to relative rankings across 24 high- and low-resource language pairs.The pairs include Chinese-to-English and English-to-Gujarati.Source: §3.2 · p. 4

3.3 The MQM corpus

• The proprietary MQM corpus contains 12K customer-support chat tuples across 12 language pairs, with English always the source and never the target.Trained annotators assign scores from error annotations under Style, Fluency, and Accuracy categories; the severity-sensitive MQM metric is more fine-grained than HTER or DA.Source: §3.3 · pp. 4–5

4 Experiments

• COMET-HTER and COMET-MQM are trained on QT21 and an internal MQM corpus, respectively, while COMET-RANK uses the WMT DARR corpora from 2017 and 2018.Source: §4 · p. 5

4.1 Training Setup

• The estimators freeze the encoder for one epoch, then fine-tune with a lower encoder learning rate; COMET-RANK is fine-tuned from the outset at one learning rate.For the estimators, the encoder learning rate is 1e−5 after the first epoch; COMET-RANK uses 1e−5 throughout.Source: §4.1 · p. 5

4.2 Evaluation Setup

• Evaluation uses WMT 2019 Metrics Shared Task test data and official Kendall’s τ, comparing COMET with shared-task submissions and recent metrics; the analysis examines subsets containing the top 10, 8, 6, and 4 MT systems.Table 1 covers English-source language pairs in the WMT19 Metrics DARR corpus; the supplied caption does not provide comparison values.Source: §4.2 · pp. 5–6, Table 1 · p. 6

5.1 From English into X

• Across all eight English-source language pairs, the three COMET models outperform other metrics, often by significant margins.The comparison includes BLEU, CHRF, YISI-1, and BERTSCORE.Source: §5.1 · p. 6
• DARR Ranker beats both estimators in seven of eight pairs, while the 12K-segment MQM Estimator matches HTER on most pairs and leads other metrics on en-ru.Source: §5.1 · p. 6

5.2 From X into English

• DARR outperforms BLEURT in five of seven language pairs with English as target, and the MQM Estimator performs strongly despite having no English-target training data.The authors hypothesize that including the source input contributes to this zero-shot result.Source: §5.2 · p. 6, Table 2 · p. 7

5.3 Language pairs not involving English

• On the three WMT 2019 language pairs without English, COMET’s results are consistent with those for English-involving pairs.All three models had been trained on data involving English.Source: §5.3 · p. 6, Table 3 · p. 7

5.4 Robustness to High-Quality MT

• On top-performing MT systems, COMET is better or competitive when English is the target and generally exceeds other metrics when English is the source; even the 12K-segment MQM Estimator is competitive.The analysis includes pairs with data from at least ten systems, excluding kk-en and gu-en.Source: §5.4 · pp. 6–7, Figure 3 · p. 7

5.5 The Importance of the Source

• Including source input improves DARR Ranker’s overall human-judgment correlation and yields a higher Δτ for pairs with English as target.Both variants were trained on WMT 2017 data without English as a target; only the source-plus-reference variant saw English source embeddings.Source: §5.5 · pp. 7–8, Table 4 · p. 8

• The authors plan to release the COMET codebase, trained MT evaluation models, and scripts for running all reported baselines upon publication.Source: §6 · p. 8
• All reported models were trained on a single Tesla T4 (16GB) GPU, and COMET builds on PyTorch Lightning for flexibility and reproducibility.Source: §6 · p. 8

7 Related Work

• BLEU, METEOR, and CHRF count shared n-grams, but typically miss semantic similarity beyond the lexical level (Papineni et al., 2002; Lavie and Denkowski, 2009; Popovi´c, 2015).These metrics have been widely studied and improved (Koehn et al., 2007; Popovi´c, 2017; Denkowski and Lavie, 2011; Guo and Hu, 2019).Source: §7 · p. 8
• Embedding-based metrics such as METEOR-VECTOR, BLEU2VEC, YISI-1, MOVERSCORE, and BERTSCORE soft-align reference and hypothesis segments to score semantic similarity, but do not capture all human-judgment dimensions (Servan et al., 2016; T¨attar and Fishel, 2017; Lo, 2019; Zhao et al., 2019; Zhang et al., 2020).Human judgments such as DA and MQM capture more than semantic similarity, creating a correlation upper bound for such metrics.Source: §7 · p. 8
• Learnable metrics optimize correlation with human judgments; BLEURT claims state-of-the-art performance for the last three WMT Metrics Shared Tasks years, but its English-BERT base restricts it to English targets (Shimanaka et al., 2019; Sellam et al., 2020).Prior learnable metrics focused on DA, which can be inherently noisy because annotators are scarce.Source: §7 · p. 8
• Reference-less quality estimation has used HTER for segment-level evaluation and MQM for document-level evaluation; multilingual BERT and XLM encoders have improved correlations, while OpenKiwi facilitates stronger QE models (Bojar et al., 2013, 2014, 2015, 2016, 2017a; Specia et al., 2018; Fonseca et al., 2019; Devlin et al., 2019; Conneau and Lample, 2019; Kepler et al., 2019a, 2019b).Source: §7 · p. 8

8 Conclusions and Future Work

• COMET is a neural framework for training MT evaluation models that can serve as automatic metrics and be adapted to different types of human judgments of MT quality.Source: §8 · pp. 8–9
• The three COMET models achieve new state-of-the-art segment-level correlation with human judgments and show promising ability to better differentiate high-performing systems.They were trained to address challenges reported in the 2019 WMT Metrics Shared Task.Source: §8 · p. 9
• Future work will examine compact solutions such as DistilBERT to reduce the parameter burden and inference time of pretrained models.Source: §8 · p. 9
• Future work will test whether COMET-RANK’s equal weighting of source and reference in its training loss is optimal, given its different weighting of them during inference.It will also further examine the interdependence of the model’s inputs.Source: §8 · p. 9

A Appendices

• The training appendix lists COMET hyperparameters and states that random seed 3 was set in the torch, numpy, and random libraries.Table 5 presents the training hyperparameters.Source: A Appendices · p. 13, Table 5 · p. 14
• Corpus-statistics tables cover QT21, WMT 2017 DARR, and WMT 2019 DARR into-English language pairs.Source: Table 6 · p. 14, Table 7 · p. 14, Table 8 · p. 15
• Additional corpus-statistics tables cover MQM, WMT 2018 DARR, and WMT 2019 DARR from-English and no-English language pairs.Source: Table 9 · p. 15, Table 10 · p. 16, Table 11 · p. 16
• Supplementary performance tables report metrics over all and top-10, -8, -6, and -4 MT systems, separately for from-English and into-English language pairs.The captions list COMET variants and comparison metrics, but provide no performance values.Source: Table 12 · p. 17, Table 13 · p. 18