Source-linked AI summary
TransQuest: Translation Quality Estimation with Cross-lingual Transformers
Tharindu Ranasinghe, Constantin Orasan, Ruslan Mitkov
TL;DR
Neural quality-estimation systems can be expensive to train and difficult to extend across language pairs, particularly when annotated data is limited. TransQuest uses cross-lingual transformers in two architectures; the authors report state-of-the-art QE results against open-source tools and competitive transfer-learning results for low-resource pairs.
Problem
Neural QE approaches require resource-intensive training and many annotated examples, making multi-language-pair systems costly and low-resource settings difficult.
Source: §1 · p. 1Method
TransQuest is an open-source sentence-level QE framework based on cross-lingual transformers, implemented with two neural architectures.
Source: §1 · pp. 1–2Results
TransQuest outperforms other open-source tools on both aspects of sentence-level QE and yields state-of-the-art results; transfer learning also outperforms current methods for low-resource pairs with few annotations.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
A neural QE framework capable of handling more than one language pair addresses the high costs of maintaining separate multi-language-pair systems.
Source: §1 · p. 2, §5.2 · p. 7Future work includes transfer learning on language pairs without English, word- and document-level QE, and unsupervised experiments for low-resource pairs.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Recent years have seen big advances in the field of sentence-level quality estimation (QE), largely as a result of using neural-based architectures. However, the majority of these methods work only on the language pair they are trained on and need retraining for new language pairs. This process can prove difficult from a technical point of view and is usually computationally expensive. In this paper we propose a simple QE framework based on cross-lingual transformers, and we use it to implement and evaluate two different neural architectures. Our evaluation shows that the proposed methods achieve state-of-the-art results outperforming current open-source quality estimation frameworks when trained on datasets from WMT. In addition, the framework proves very useful in transfer learning settings, especially when dealing with low-resourced languages, allowing us to obtain very competitive results.
1 Introduction
Sentence-level quality estimation (QE) judges translation quality without a reference, but neural systems can be costly to train and difficult to scale across language pairs, especially where annotations are scarce. The paper introduces TransQuest, an open-source framework with two architectures, and argues it can support multi-language-pair QE and low-resource transfer learning.
1 Introduction
- QE can help select among translation engines, communicate reliability, and decide whether content needs post-editing or human translation.The paper focuses on sentence-level rather than document- or word-level estimation.Source: §1 · p. 1
- TransQuest’s two architectures outperform current state-of-the-art methods on two aspects of sentence-level quality estimation.The framework is open source.Source: §1 · p. 2
- The framework targets costly language-pair-specific QE by developing a neural method capable of estimating quality for more than one language pair.The authors describe this as the first such neural method to their knowledge.Source: §1 · p. 2
- With few annotated training instances, TransQuest transfer learning can outperform current state-of-the-art methods for low-resource language pairs.The authors also report that a tuned version won all eight tasks of the WMT 2020 direct assessment sentence-level QE shared task.Source: §1 · p. 2
- TransQuest’s model zoo offers pretrained QE models across 15 language pairs and multiple quality-estimation aspects, alongside its open-source code.The resources are described as freely available to the community.Source: §1 · p. 2
2 Related Work
Prior neural QE systems achieved strong shared-task results but often depended on complex models, parallel-data pretraining, or substantial compute. TransQuest builds on cross-lingual embeddings and transformer representations to pursue simpler sentence-level QE without parallel data.
2 Related Work
- Neural QE systems consistently topped WMT shared-task leaderboards, supplanting earlier feature-engineered approaches.Earlier systems such as QuEst and QuEst++ relied on linguistic processing and engineered features.Source: §2 · p. 2
- POSTECH’s predictor-estimator RNN was WMT 2017’s best-performing QE system, but required extensive predictor pretraining, large parallel data, and substantial computation.Source: §2 · p. 2
- OpenKiwi’s best sentence-level model stacked architectures around Predictor-Estimator, so it required extensive pretraining, parallel data, and computational resources.Source: §2 · pp. 2–3
- TransQuest uses cross-lingual embeddings to reduce reliance on parallel data and the powerful compute that such data requirements entail.The authors expected these embeddings to ease the burden of complex neural architectures.Source: §2 · p. 3
- Although multilingual BERT and DistilBERT show cross-lingual characteristics, the cited work reports reservations about language representation and weak benchmark performance (Pires et al., 2019; K et al., 2020).Source: §2 · p. 3
- XLM-R trains RoBERTa on unlabelled text in 104 languages using masked language modeling, and outperforms mBERT on several cross-lingual benchmarks.Source: §2 · p. 3
- The authors explore XLM-R with simple neural architectures for sentence-level QE, noting that such contextual cross-lingual embeddings had not previously been used for QE to their knowledge.Their initial experiments found XLM-R performed better than mBERT.Source: §2 · p. 3
3 Methodology
TransQuest uses XLM-R representations and a simple output layer in two sentence-level QE architectures. The models share training settings, with architecture-specific pooling choices selected from initial experiments.
3.1 Neural Network Architectures
- The framework represents inputs with XLM-R and sends transformer outputs to a simple quality-estimation layer, avoiding a complex output network.The transformer accepts sequences of up to 512 tokens; its [CLS] embedding represents the whole sequence.Source: §3.1 · pp. 3–4
- MonoTransQuest concatenates source and translation with [SEP], uses XLM-R, and feeds its [CLS] embedding to a softmax quality predictor.Among CLS, MEAN, and MAX pooling, initial experiments favored CLS; training uses mean-squared-error loss.Source: §3.1 · p. 4
- SiameseTransQuest encodes source and translation separately, then predicts quality from cosine similarity between their pooled representations.Initial experiments favored MEAN pooling over CLS and MAX; the objective is mean-squared-error loss.Source: §3.1 · p. 4
3.2 Training Details
- Both architectures use consistent settings across language pairs: batch size eight, Adam at 2e−5, 10% warm-up, three epochs, and early stopping after ten evaluation steps without loss improvement.XLM-R and subsequent-layer parameters are updated; training uses training data and evaluation uses a set with one fifth as many rows.Source: §3.2 · p. 4
4 Dataset
The evaluation uses publicly available WMT datasets to predict HTER and Direct Assessment (DA). The datasets cover varied language pairs and translation settings, with DA annotations drawn from professional ratings.
4 Dataset
- The study predicts two translation-quality measures, HTER and Direct Assessment, using publicly available datasets released for recent WMT QE tasks.The authors use these datasets to support reproducibility and comparison with prior results.Source: §4 · p. 5
4.1 Predicting HTER
- HTER datasets cover six language pairs across varied domains and neural or statistical translation systems.HTER estimates the percentage of edits needed to transform an automatic translation into a reference; researchers have questioned its reliability for QE evaluation.Source: §4.1 · p. 5, §4.2 · p. 5
4.2 Predicting DA
- DA datasets include high-, medium-, and low-resource language pairs, with translations rated from 0 to 100 by at least three professional translators.Scores are z-standardized, and systems predict mean DA z-scores; each language pair has 7,000 training, 1,000 development, and 1,000 test sentence pairs.Source: §4.2 · pp. 5–6
5 Evaluation and discussion
The evaluation compares TransQuest with WMT baselines across single-pair, multi-pair, and transfer-learning settings, using Pearson correlation between predictions and gold labels. Results report competitive performance across settings, including gains with transfer learning on low-resource pairs.
5.1 Supervised Single Language Pair Quality Estimation
- In single-pair evaluation, both TransQuest architectures outperformed baselines across language pairs and quality-estimation aspects, with MTransQuest slightly better than STransQuest.The evaluation compared Pearson correlation on HTER and DA tasks against OpenKiwi, QuEst++, and competition systems.Source: §5.1 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 8
- For HTER, MTransQuest gained ≈0.1-0.2 Pearson correlation over OpenKiwi in most language pairs, but did not surpass the best system on En-De SMT or En-Zh NMT.Where OpenKiwi results were unavailable, it gained ≈0.3-0.4 Pearson correlation over QuEst++ across language pairs for both NMT and SMT.Source: §5.1 · p. 7, Table 2 · p. 7
- For DA, MTransQuest gained ≈0.2-0.3 Pearson correlation over OpenKiwi across language pairs, and TransQuest won WMT 2020 QE Task 1 in all languages.The winning submission used self-ensemble and data augmentation fine-tuning.Source: §5.1 · p. 7, Table 3 · p. 8
5.2 Supervised Multi-Language Pair Quality Estimation
- Multi-pair training concatenated data either within source-English and target-English groups or across all translation directions into one model.Models were evaluated separately by language pair; the direction-agnostic models were named MTransQuest-m and STransQuest-m.Source: §5.2 · p. 8
- Multi-pair models were competitive and sometimes beat single-pair models; direction-aware models did slightly better overall and lost no more than 0.03% Pearson correlation in any pair.Multi-pair models beat single-pair models for HTER in En-Lv for SMT and NMT and En-De for SMT, and for DA in Et-En, Ru-En, and Si-En.Source: §5.2 · p. 8
- Multi-pair models with language direction considered beat prior WMT best systems except in En-De SMT and En-Zh NMT; their model size stayed at or below 2.1 GB.Single-pair pre-trained models were approximately 2 GB; the paper presents multi-pair models as an option for resource-constrained multi-language deployments.Source: §5.2 · pp. 8–9
5.3 Transfer Learning based Quality Estimation
- Transfer learning initialized low-resource Si-En and Ne-En models from a model trained on Et-En, Ro-En, and Ru-En, then tested training sets from 0 to 1,000 sentence pairs against scratch training.The transfer experiment was limited to DA, and its training pairs had English as the target language.Source: §5.3 · p. 9
- With 100 Ne-En training instances, MTransQuest achieved 0.7417 Pearson correlation with transfer learning versus 0.1242 from scratch, approaching 0.7914 with 7,000 instances.Transfer and scratch results converged as training instances increased; a similar pattern appeared for Si-En and STransQuest, with lower correlations.Source: §5.3 · pp. 9–10, Figure 2 · p. 9
6 Conclusions
TransQuest is an open-source framework based on cross-lingual transformers that outperforms other open-source tools on both aspects of sentence-level QE and yields new state-of-the-art results. Its two architectures trade accuracy for efficiency, and planned work includes additional architectures, quality-estimation levels, and transfer-learning settings.
6 Conclusions
- TransQuest outperforms other open-source tools on both aspects of sentence-level quality estimation and yields new state-of-the-art results.The framework was tested on both aspects of sentence-level QE and explores multi-language-pair models and transfer learning for low-resource pairs.Source: §6 · p. 10
- 35s for 1,000-instance inference with MTransQuest versus 16s with STransQuest; training 7,000 instances averages 4,480s versus 3,900s, respectively.The authors recommend MTransQuest when accuracy is prioritized and STransQuest when efficiency is prioritized.Source: §6 · p. 10
- Future work includes more architectures and word- and document-level models, plus transfer learning for language pairs without English and unsupervised experiments on low-resource pairs.Source: §6 · p. 10