Source-linked AI summary
CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte M. Alves, Alon Lavie, Luisa Coheur, André F. T. Martins
TL;DR
QE systems must generalize across language pairs, including unseen ones, while supporting sentence-level, word-level, explainable, and critical-error tasks. The paper combines COMET and OpenKiwi into COMETKIWI, and its submissions achieved the best multilingual results across all tracks by considerable margins.
Problem
Unseen language pairs in QE test sets motivate systems that generalize multilingual quality estimates with few-shot training.
Method
COMETKIWI combines COMET training features with OpenKiwi’s predictor-estimator architecture, word-level sequence tagging, multilingual pretraining, and attention-gradient explanations.
Results
0.572 Spearman correlation, 0.341 MCC, and 0.486 R@K were the best submitted multilingual results for sentence-level, word-level, and Explainable QE tracks, respectively.
Takeaways & Limitations
Pretraining with references, joint sentence- and word-level objectives, and 500-example adaptation improved performance across several language pairs without harming other correlations.
Takeaways & Limitations
Large ensembles impose burdensome parameter counts and inference time, motivating distillation into smaller QE systems.
Abstract
from arXiv · showhide
We present the joint contribution of IST and Unbabel to the WMT 2022 Shared Task on Quality Estimation (QE). Our team participated on all three subtasks: (i) Sentence and Word-level Quality Prediction; (ii) Explainable QE; and (iii) Critical Error Detection. For all tasks we build on top of the COMET framework, connecting it with the predictor-estimator architecture of OpenKiwi, and equipping it with a word-level sequence tagger and an explanation extractor. Our results suggest that incorporating references during pretraining improves performance across several language pairs on downstream tasks, and that jointly training with sentence and word-level objectives yields a further boost. Furthermore, combining attention and gradient information proved to be the top strategy for extracting good explanations of sentence-level QE models. Overall, our submissions achieved the best results for all three tasks for almost all language pairs by a considerable margin.
1 Introduction
The paper introduces COMETKIWI, combining COMET and OpenKiwi to address three WMT 2022 QE tasks with multilingual generalization and explainability in mind. The systems use pretraining, joint sentence- and word-level objectives, and attention-gradient explanations, achieving leading multilingual results.
- QE assigns quality scores or labels to machine translation outputs without relying on reference translations.
- COMETKIWI combines COMET’s training features with OpenKiwi’s predictor-estimator architecture and word-level sequence tagging.
- The models are pretrained on Direct Assessment annotations and then fine-tuned on shared-task data to support multilingual generalization, including unseen languages through few-shot training.
- Pretraining on Metrics shared-task annotations is reported as important for improving QE model results.
- 500 examples improve results for new language pairs without harming correlations for other language pairs.
- The proposed interpretability method combines attention and gradient information to explain sentence-level QE predictions.
- 0.572 Spearman correlation, 0.341 MCC, and 0.486 R@K were achieved for sentence-level DA, word-level QE, and Explainable QE, respectively, with reported margins over the second-best system.
2 Background
The background distinguishes sentence-level and word-level QE and introduces transformer attention as the mechanism for contextualizing source and translation representations. COMETKIWI uses encoder hidden states to produce either sentence scores or word-level labels.
- Quality Estimation: Sentence-level QE predicts one quality score for a source sentence and its translation, whereas word-level QE predicts OK or BAD for each translated word.
- Transformers: Transformer multi-head attention contextualizes information within and across input sentences using query, key, and value representations.
- Transformers: Scaled dot-product attention takes query representations together with key and value matrices as inputs.
- Transformers: The π transformation maps attention rows to distributions, commonly using softmax, and multi-head attention applies the operation in parallel across heads.
- Transformers: Each attention head uses learnable linear transformations, and the layer output concatenates head representations before applying a learnable output transformation.
- COMETKIWI architecture: COMETKIWI’s architecture uses separate output paths: a <cls> representation yields sentence scores, while translated-token representations yield word-level predictions.
- Transformers: Encoder hidden states are refined with feed-forward blocks and residual connections; encoder-only models such as BERT and XLM use self-attention with m = n.
3 Implemented Systems
The implemented systems combine multilingual pretrained encoders, scalar-mixed representations, and task-specific objectives for sentence- and word-level QE. The submission also uses few-shot adaptation, ensembling, post-edit pretraining, and attention-based explanation methods.
- Model architecture: COMETKIWI models use scalar mixing to aggregate encoder hidden states before producing sentence or word-level predictions.The encoder processes concatenated source and translation sentences; sentence models use the first-token representation, while word models classify translated tokens as OK or BAD.
- Model architecture: Sparsemax learns which encoder layers contribute to the scalar-mixed representation, allowing irrelevant layers to be ignored.The layer weights are constrained to be nonnegative and sum to one.
- Pretraining and adaptation: Pretraining uses 657k multilingual Direct Assessment samples with references, followed by fine-tuning on DA, HTER, and MQM quality assessments without references.The Metrics corpora cover 24 language pairs from 2017–2019, while post-edit corpora provide additional word labels and HTER scores.
- Training objectives: Sentence-level models jointly optimize sentence scores and OK/BAD word labels, although sentence-only training remains available as a special loss setting.The combined objective weights sentence-level and word-level losses with λs and λw; λs = 1 and λw = 0 gives a fully sentence-level model.
- Pretraining and adaptation: Few-shot adaptation uses half of each available development set for fine-tuning on five unseen language pairs, excluding en-yo because no data were available.The official development sets contain 1K examples per language pair except en-yo.
- Explainable QE: Explainable QE combines attention and gradient information through Attention × GradNorm, while Head Mix learns separate weights for attention-head representations.The Head Mix modification extends scalar mixing with head-specific coefficients, using either softmax or sparsemax transformations.
4 Experimental Results
Experiments evaluate COMETKIWI across sentence-level, word-level, and explainable QE settings using internal splits, task-specific metrics, and multilingual or zero-shot language-pair evaluation. Results indicate gains from task-specific fine-tuning, multilingual pretraining, sentence–word multitasking, ensembling, and attention–gradient explanations.
- Experimental setup: Internal development sets were split into equal halves of approximately 500 segments per language pair for DA and MQM at both word and sentence levels.Previous shared-task systems supplied the Task 1 and Task 2 baselines.
- Experimental setup: Sentence-level submissions were ranked primarily by Spearman correlation, while word-level submissions were ranked primarily by MCC and evaluated separately on in-domain and zero-shot language pairs.Pearson correlation, MAE, RMSE, F1-OK, and F1-BAD were secondary metrics.
- Sentence-level QE: Fine-tuning on MLQE-PE improved sentence-level DA results by approximately 4%, while adding new language-pair data yielded a further approximately 1% overall improvement.Unseen pairs improved by 2–3% with only 500 samples.
- Sentence-level QE: InfoXLM produced 1.7% higher Spearman correlation than XLM-R and RemBERT for sentence-level Direct Assessments.The reported comparison concerns the three backbone transformers tested in the experiments.
- Sentence-level QE: MQM-specific DA filtering and MQM fine-tuning improved Spearman correlations over the pure DA baseline across language pairs, while combined training further improved multilingual scores.The final ensemble performed better on the submitted leaderboard than on the internal test set despite weaker internal results than individual systems.
- Word-level QE: Sentence-level multitasking boosted word-level performance, especially for out-of-English translations, while sentence-mix, language prefixes, and ensembling improved results across language pairs.The relative benefit of different ensembling methods was only 1–2% in submitted versions and below 0.1% in averaged multilingual results.
- Explainable QE: Explainable QE used continuous explanation scores evaluated by AUC, AP, and R@K on translations containing errors, with R@K as the primary metric.The experiments optimized internally for the average of all three metrics.
- Explainable QE: Attn × GradNorm outperformed the prior Attn × Norm baseline for both DA and MQM explanations, and ensembling explanations across heads produced consistent further improvements.Head Mix effects differed between DA and MQM, and Sparse Head Mix generally outperformed Soft Head Mix for DA but not MQM.
5 Official Results
Official comparisons place the submissions at or near the top across all three QE tasks and multilingual settings. The strongest coverage was achieved for sentence-level and explainable QE, while critical error detection substantially exceeded the organizers’ baseline.
- Official comparisons: The submissions achieved the best sentence-level results for 6/9 language pairs and the best word-level results for 5/9 language pairs.These comparisons are reported against the other competitors in the official results.
- Official comparisons: Explainable QE achieved the best results for all but two language pairs, km-en and ps-en.The official explainable-QE results are summarized alongside the Table 4 evaluation framework.
- Official comparisons: In constrained critical error detection, the submission vastly surpassed the organizers’ baseline despite having no other competitor.The constrained setting had no competing system for direct comparison.
- Multilingual results: The submissions obtained the best multilingual results for every task, both including and excluding en-yo, and ranked first when averaging results across language pairs.The paper reports this pattern across all three official tasks.
6 Conclusions and Future Work
The IST–Unbabel contribution combines COMET and OpenKiwi in COMETKIWI, with pretraining, joint objectives, and attention–gradient explanations supporting strong QE results. Future work targets distilling large ensembles into smaller, more practical systems.
- Pretraining with references improves downstream performance across several language pairs, while joint sentence- and word-level training provides a further boost.
- Attention weights combined with gradient information improve explainability for some language pairs and may help identify attention heads for zero-shot language pairs.
- The submissions achieve the best results for all tasks, including Critical Error Detection, for almost all language pairs by a considerable margin.
- Large ensembles impose burdensome parameter counts and inference time, motivating future work on distilling them into smaller QE systems.
A Data Information
The paper’s data information identifies the DA and MQM data used across language pairs. DA development data were split to create an internal development/test arrangement, so reported figures use the internal development split.
- The original DA development set was split to generate a new development/test split, and reported numbers correspond to the internal development split.
- Table 5 presents DA and MQM data for all language pairs.
B Official Results
The official results report separate evaluation views for sentence-level QE, word-level QE, Explainable QE, and Critical Error Detection. Table 7 organizes the first three by their respective metrics, while Table 6 reports MCC for Critical Error Detection.
- Critical Error Detection: Critical Error Detection is evaluated using ranking metrics R@K and MCC, with MCC reported as the main metric in Table 6.
- QE and Explainable QE: Table 7 reports sentence-level QE using Spearman’s correlation, word-level QE using MCC, and Explainable QE using R@K.