Source-linked AI summary
Unsupervised Quality Estimation for Neural Machine Translation
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, Lucia Specia
TL;DR
QE must inform users about MT quality, but existing systems require substantial expert-annotated data and training resources. This paper proposes unsupervised glass-box QE using NMT-derived probabilities, attention, and uncertainty quantification, and reports competitive correlation with human judgments while introducing a multilingual evaluation dataset. The authors also find that model architecture and training settings affect probability-quality relationships, and that improved translation quality does not necessarily strengthen that correlation.
Problem
Existing QE approaches require large amounts of expert-annotated data, while MT quality varies across language pairs, domains, and datasets and fluent outputs can differ in meaning.
Method
The paper estimates quality without training or additional resources by using NMT output probability distributions, attention weights, and uncertainty quantification extracted during decoding.
Results
Unsupervised indicators from carefully designed NMT probabilities correlate with human judgments competitively with supervised QE models, while model ensembles achieve optimal results for both translation quality and probability-based QE.
Takeaways & Limitations
NMT systems themselves provide a rich source of information for unsupervised QE, and the paper supplies a multilingual dataset and models for reproducing these experiments.
Takeaways & Limitations
Attention-based metrics correlate considerably less with human judgments than the other unsupervised indicators, and the authors leave combined attention- and probability-based measures for future work.
Abstract
from arXiv · showhide
Quality Estimation (QE) is an important component in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT output at test time. Existing approaches require large amounts of expert annotated data, computation and time for training. As an alternative, we devise an unsupervised approach to QE where no training or access to additional resources besides the MT system itself is required. Different from most of the current work that treats the MT system as a black box, we explore useful information that can be extracted from the MT system as a by-product of translation. By employing methods for uncertainty quantification, we achieve very good correlation with human judgments of quality, rivalling state-of-the-art supervised QE models. To evaluate our approach we collect the first dataset that enables work on both black-box and glass-box approaches to QE.
1 Introduction
The paper frames QE as necessary because MT quality varies and fluent outputs can still diverge in meaning, while existing QE systems depend on scarce annotated resources. It proposes an unsupervised glass-box approach using information produced during NMT decoding and evaluates it with a new multilingual dataset.
- Motivation: QE predicts MT output quality at test time without a gold-standard human translation, but state-of-the-art systems require substantial parallel and labeled data.Such collections are available for only a small set of languages and domains.
- Approach: NMT models provide two unsupervised QE signals: vocabulary-wide output probabilities and decoder attention weights.The approach treats the MT system as a glass box rather than relying only on its source and output.
- Approach: Uncertainty quantification is used to exploit output distributions beyond the top-1 prediction because neural probabilities can be poorly calibrated and overconfident.The experiments account for architecture, training, and search factors affecting probability reliability.
- Approach: Attention is analyzed as a soft-alignment signal, including the challenging multi-head attention of Transformer models.The paper studies how much translation-quality information can be extracted from Transformer attention.
- Evaluation: The new dataset covers 6 language pairs across high-, medium-, and low-resource scenarios, with 10K Wikipedia segments per language pair labeled using Direct Assessment.This supports evaluation of both black-box and glass-box QE approaches.
- Results: Unsupervised indicators from well-calibrated NMT probabilities rival strong supervised state-of-the-art models in correlation with human quality judgments.The paper also analyzes attention distributions and model confidence across different NMT systems.
2 Related Work
Prior QE work is dominated by supervised neural systems requiring labeled data, while earlier glass-box and unsupervised methods were limited or underperforming. The paper positions its NMT-internal, unsupervised approach alongside improved DA-based evaluation resources and uncertainty methods.
- QE: Neural QE systems learn source-target representations but are resource-heavy, requiring significant in-domain labeled data and not using internal MT information.The Predictor-Estimator model is a representative encoder-decoder and estimator architecture.
- QE: Earlier glass-box QE used statistical-MT features, while prior unsupervised approaches performed significantly worse than supervised systems.These approaches included language-model probabilities, n-best-list size, and lexical translation probabilities.
- QE: Attention-based indicators from RNN-based NMT were competitive mainly when combined with supervised QE features and were not directly applicable to multi-head Transformer attention.This motivates analyzing Transformer attention within an unsupervised glass-box setting.
- QE: The paper claims the first glass-box unsupervised QE approach for NMT that performs competitively with state-of-the-art supervised systems.This distinguishes the contribution from earlier statistical-MT and RNN-attention approaches.
- QE Datasets: Direct Assessment evaluates MT quality on a continuous 1-100 scale and has been shown to improve reproducibility and reliability relative to traditional HTER-based evaluation.HTER is a semi-automatic metric based on human-mediated translation edit rate.
- QE Datasets: Existing DA datasets contained only a few hundred segments per language pair, unavailable MT systems, and judgments from non-professional translators, limiting supervised and glass-box QE experiments.The new dataset is designed to address these limitations.
- Uncertainty quantification: Uncertainty quantification commonly uses Bayesian formulations or approximations such as Monte Carlo Dropout and model ensembling to estimate neural-model uncertainty.The paper applies this methodological family to NMT quality estimation.
3 Unsupervised QE for NMT
The paper develops unsupervised QE indicators from information produced by an NMT system, including probabilities, uncertainty under MC dropout, lexical variation, and attention weights. These indicators require no additional training resources beyond the MT system.
- 3 Unsupervised QE for NMT: Unsupervised QE indicators use output probabilities, uncertainty quantification, and attention weights extracted during NMT translation.The approach includes deterministic and uncertainty-aware output distributions, plus attention-based indicators.
- 3.1 Exploiting the Softmax Distribution: Sentence-level translation probability is normalized by length to provide a basic QE measure.The paper introduces this measure before extending probability-based QE beyond top-1 predictions.
- 3.1 Exploiting the Softmax Distribution: Entropy averages softmax uncertainty across decoding steps, while log-probability standard deviation captures dispersion lost by averaging.Concentrated probability mass suggests likely correctness, whereas near-uniform distributions suggest lower expected quality; equal means can still hide different dispersion patterns.
- 3.2 Quantifying Uncertainty: MC dropout estimates sentence-level probability expectations and variances from multiple stochastic forward passes with perturbed model parameters.The paper also considers combining these measures and notes their prior use for mitigating low-quality back-translation outputs during NMT training.
- 3.2 Quantifying Uncertainty: Lexical variation among dropout-generated translation hypotheses is measured with average pairwise Meteor similarity to capture uncertainty, ambiguity, and sentence complexity.The hypotheses are generated for the same source segment under stochastic inference.
- 3.3 Attention: Attention entropy summarizes encoder-decoder attention across Transformer heads and layers by selecting the minimum or computing the average.Attention weights represent source-target token connection strength, and the method computes entropy for each head/layer combination.
4 Multilingual Dataset for QE
The paper introduces a multilingual QE dataset spanning six language pairs and high-, medium-, and low-resource conditions. It combines Wikipedia-based source sentences, MT outputs, and professional direct-assessment annotations to support supervised and unsupervised QE.
- 4 Multilingual Dataset for QE: The dataset covers six language pairs across high-, medium-, and low-resource training conditions, with translation directions into and out of English where annotators were available.The high-resource pairs are English–German and English–Chinese; medium-resource pairs are Romanian–English and Estonian–English, with additional low-resource pairs described in the dataset.
- Document and sentence sampling: Source sentences were sampled from Wikipedia, filtered by language and length, and excluded when already present in recent Wikipedia parallel corpora.The sampling process follows FLORES and targets sentences between 50 and 150 characters.
- Document and sentence sampling: The collection uses 10K sentences per language, or 20K for German and Chinese, with minimal lexical overlap selected for high-resource pairs.The overlap criterion was used to ensure sufficient high- and low-quality translations in high-resource settings.
- NMT systems: MT systems use standard Transformers for medium- and high-resource pairs and Big-Transformer architectures for Sinhala–English and Nepali–English.Training used publicly available resources including Paracrawl and Europarl.
- DA judgments: Scores were standardized per annotator and averaged per segment, allowing genuine disagreement among annotators to contribute to final judgments.Each segment was independently evaluated by three professional translators, with additional ratings requested when disagreement exceeded 30 points.
- Data splits: The dataset is split into 7K training, 1K development, and two 1K-sentence test sets, with one test set held blind for future work.The design enables comparison between supervised and unsupervised QE approaches.
- Additional data: The released package includes the data, NMT models, and direct-assessment judgments.The paper provides these resources through the project repository.
5 Experiments and Results
The experiments compare unsupervised indicators extracted during NMT decoding with supervised QE systems using human Direct Assessment judgments across language pairs. Probability- and dropout-based indicators correlate competitively with quality, while results vary substantially by language pair and attention metrics are weaker.
- 5.2 Correlation with Human Judgments: The evaluation compares unsupervised QE indicators with PredEst and BERT-BiRNN using Pearson correlation with human DA judgments.Table 2 groups unsupervised indicators from standard decoding, MC dropout, and attention, alongside supervised models.
- 5.2 Correlation with Human Judgments: TP is consistently outperformed by D-TP, supporting uncertainty estimation as a stronger reflection of perceived translation quality.The paper reports that dropout-based indicators significantly outperform PredEst and rival BERT-BiRNN for four language pairs.
- 5.2 Correlation with Human Judgments: Performance varies considerably across language pairs, with highest correlations for medium-resource languages and drastically lower correlations for high-resource pairs.For En-De, where most translations are high quality, meaningful variation is harder to capture; low-resource results may also reflect model capacity and training-data effects.
- 5.2 Correlation with Human Judgments: Softmax-Ent and Sent-Std significantly outperform TP for four language pairs by capturing output-distribution uncertainty and token-probability variation.For Et-En examples with DA scores 62 and 1, sentence-level log-probabilities are nearly identical, whereas token-level probability sequences differ markedly.
- 5.2 Correlation with Human Judgments: D-Lex-Sim and D-TP achieve the best Group II results, while D-Var correlates much less because it ignores the model’s actual probability estimate.D-Lex-Sim measures variability among MC-dropout hypotheses and also reflects potential source-segment ambiguity.
- 5.2 Correlation with Human Judgments: Attention-based indicators correlate sensibly with human judgments but considerably below other unsupervised indicators; the best head/layer combination significantly outperforms simple averaging.Selecting the best head/layer requires validation on DA-annotated data, so it is not fully unsupervised.
6 Discussion
The discussion tests how domain shift, NMT system design, and training affect the reliability of model probabilities as unsupervised QE indicators. Better uncertainty estimation improves robustness and correlation, but translation quality and QE correlation can diverge.
- 6.1 Domain Shift: MC dropout separates in-domain from out-of-domain confidence more reliably than standard decoding for Et-En.TP averages were -0.440 and -0.445, while D-TP averages were -0.592 and -0.685; the MC dropout difference was statistically significant (p-value < 0.01).
- 6.2 NMT Calibration across NMT Systems: The system comparison evaluates RNN, MoE, ensembling, diverse beam search, sampling, and the standard Transformer using Et-En data.The study translates 400 sentences per system variant and collects Direct Assessment scores; the training-epoch experiment runs for 60 epochs and tracks Meteor and Pearson correlation.
- 6.2 NMT Calibration across NMT Systems: Diverse beam search and MoE do not improve correlation over the standard Transformer model.The experiment compares system variants that promote diversity in NMT outputs with the standard Transformer baseline.
- 6.2 NMT Calibration across NMT Systems: Ensembling achieves the best results for both translation quality and QE, while MC dropout achieves very similar results.The authors recommend either method for NMT systems with unsupervised QE.
- 6.3 NMT Calibration across Training Epochs: Training longer stabilizes test quality but deteriorates the relation between model probabilities and translation quality.The paper attributes this to increasing prediction confidence after most training examples are correctly classified, which damages calibration.
7 Conclusions
The paper presents unsupervised QE based solely on information produced by an NMT system and finds that carefully designed indicators can compete with supervised QE. It also analyzes calibration across systems and releases a multilingual QE dataset and reproducible models, while identifying extensions for future work.
- The approach requires no training or additional resources beyond the MT system.
- The method uses softmax output probabilities, attention-weight entropy, and uncertainty quantification as unsupervised QE indicators.
- Improved translation quality does not necessarily imply stronger correlation between predictive probabilities and translation quality, while ensembles achieve optimal results for both.
- The authors created a multilingual QE dataset covering low- and high-resource MT scenarios and released the dataset and MT models for reproduction.
- The sentence-level metrics could be adapted to word, phrase, and document QE, combined with supervised QE features, and extended with other uncertainty methods.