Source-linked AI summary
OpenKiwi: An Open Source Framework for Quality Estimation
Fábio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, André F. T. Martins
TL;DR
Quality estimation evaluates translation quality without reference translations, but reproducing recent high-performing systems has been difficult. OpenKiwi packages leading QE systems in an open framework for training and experimentation. It outperforms deepQuest across word- and sentence-level tasks and achieves the best results on the word-level tasks.
Problem
Reproducing recent QE systems is difficult because they can rely on complex ensembles, complicated architectures, or poorly documented component training.
Source: §1 · p. 1Method
OpenKiwi is an open-source framework that implements four established QE systems and supports training new word- and sentence-level models.
Source: §1 · p. 1Results
OpenKiwi outperforms deepQuest on all word- and sentence-level tasks and attains the best results on all word-level tasks.
Source: §5 · p. 5Takeaways & Limitations
OpenKiwi became the WMT 2019 QE shared-task baseline, and the winning systems across word-, sentence-, and document-level tasks used it as a foundation.
Source: §6 · pp. 5–6
Abstract
from arXiv · showhide
We introduce OpenKiwi, a PyTorch-based open source framework for translation quality estimation. OpenKiwi supports training and testing of word-level and sentence-level quality estimation systems, implementing the winning systems of the WMT 2015-18 quality estimation campaigns. We benchmark OpenKiwi on two datasets from WMT 2018 (English-German SMT and NMT), yielding state-of-the-art performance on the word-level tasks and near state-of-the-art in the sentence-level tasks.
1 Introduction
Recent QE systems were difficult to reproduce, while existing open-source frameworks lagged the best shared-task systems. OpenKiwi addresses this gap with an open framework that implements leading systems and supports experimentation and new model training.
1 Introduction
- Reproducing recent QE systems was hindered by complex ensembles, complicated architectures, and poorly documented component pre-training and fine-tuning.Existing open-source frameworks were helpful but behind the recent best systems in WMT QE shared tasks.Source: §1 · p. 1
- OpenKiwi implements the best QE systems from WMT 2015–18 and lets researchers combine or modify their key components within one framework.The project aims to make experimentation easier.Source: §1 · p. 1
- The framework implements QUETCH, NUQE, Predictor-Estimator, and a stacked ensemble with a linear system.These are four QE systems proposed in recent years.Source: §1 · p. 1
- Researchers can train new QE models on new data and use the API as a package or from the command line.The project is implemented in Python with PyTorch.Source: §1 · p. 1
- YAML configuration files and optional MLflow support experiment tracking and reproducibility, and the project is released under the Affero GPL.Source: §1 · p. 1
2 Quality Estimation
Quality estimation evaluates translation quality without reference translations. It labels errors at the word level or estimates the quality of a whole sentence, with prior approaches using neural, linear, and multi-task methods.
2 Quality Estimation
- Word-level QE assigns OK or BAD labels to translated words, gaps for needed insertions, and source words that were mistranslated or omitted.The example also relates word-level labels to a sentence-level HTER score based on edits needed to produce the post-edition.Source: §2 · pp. 1–2, Figure 1 · p. 2
- Recent word-level systems combine linear and neural models, use automatic post-editing as an intermediate step, or develop specialized neural architectures.Sentence-level approaches include conversions from word-level predictions and joint training with multi-task learning.Source: §2 · p. 2
- Sentence-level QE predicts whole-sentence quality, for example from human post-editing time or the number of edits required, measured as HTER.HTER is Human Translation Error Rate.Source: §2 · p. 2
3 Implemented Systems
OpenKiwi implements four QE systems, ranging from a simple feedforward model to recurrent architectures and a stacked ensemble. Their inputs, training data, and prediction targets differ.
3 Implemented Systems
- QUETCH uses a one-hidden-layer multilayer perceptron with target and aligned-source word windows, trained separately for source, gap, and target tags.It relies only on shared-task datasets, without external auxiliary training data.Source: §3 · p. 2
- NuQE concatenates target-word and source-aligned embeddings, then feeds them through feedforward layers and a bidirectional GRU to predict OK/BAD labels.Like QUETCH, it trains separate models for source, gap, and target tags and uses shared-task data only.Source: §3 · p. 2
- Predictor-Estimator uses a predictor trained on source and target context, then an estimator that classifies words from predictor features.The predictor uses bidirectional source encoding, target LSTMs, and attention; multi-task training also predicts sentence-level HTER, while reverse-direction prediction produces source labels.Source: §3 · pp. 2–3
- The stacked ensemble combines the systems with a feature-based linear system, using lexical, contextual, alignment, syntactic, and language-model features.It produces labels only for MT words.Source: §3 · p. 3
4 Design, Implementation and Usage
OpenKiwi organizes inputs, datasets, and training in a reusable pipeline, with experimentation largely configurable through model classes and parameters. It also supports testing, experiment tracking, and both command-line and Python-package use.
Design
- Fields specify tokenization, vocabulary construction, ID mapping, and padding, while Datasets group input and output fields into minibatches.The training loop processes minibatches, computes loss, backpropagates, validates, and saves snapshots; by default, it keeps the best model and validation probabilities.Source: §4 · p. 3
- Quick experimentation usually requires changing configuration parameters or the model class, rather than changing the QE training flow.New models can focus on a torch.nn.Module-based class; new flags or options can be added in the CLI parser.Source: §4 · p. 3
Implementation
- OpenKiwi uses Poetry for deterministic dependency management, has tests with close to 80% code coverage, and supports MLflow experiment tracking.MLflow supports comparing runs and searching for metrics and parameters.Source: §4 · p. 3
Usage
- Training can be launched from the command line using a configuration file, and OpenKiwi can also be installed as a Python package or used from Python.The configuration file contains training and model options; after training, the model can predict on new data.Source: §4 · pp. 3–4
- The framework includes an interactive visualization of predictions, showing BAD words in red and BAD gaps as red underscores.The paper points to a Jupyter Notebook that produces the visualization.Source: Figure 2 · p. 4, §4 · p. 4
5 Benchmark Experiments
On the WMT 2018 test set, OpenKiwi outperforms deepQuest across word- and sentence-level tasks and achieves the best results on all word-level tasks. The experiments compare its systems on English–German SMT and NMT data, including ensemble and stacked variants.
Datasets
- The benchmark uses English–German WMT 2018 SMT and NMT datasets with sentence triplets, word-level labels, and sentence-level scores derived from post-editions using TERCOM.The SMT dataset has 26,273 training and 1,000 development triplets; the NMT dataset has 13,442 training and 1,000 development triplets.Source: §5 · p. 4
- The experiments use 526,368 artificially generated sentence triplets, built through cross-entropy filtering of a larger monolingual corpus for in-domain sentences, round-trip translation, and stratified sampling, alongside 3,396,364 in-domain parallel sentences for predictor-estimator pre-training.Source: §5 · p. 4
Systems
- APE-QE supplies pseudo-post-editions for generating word-level labels and sentence-level scores, using separate SRC→PE and MT→PE translation models.The SRC→PE model is trained on in-domain data and fine-tuned on shared-task data; the MT→PE model uses artificial triplets and oversampled shared-task data.Source: §5 · pp. 4–5
Test results
- OpenKiwi outperforms deepQuest on all word- and sentence-level tasks and achieves the best results on all word-level tasks in the WMT 2018 test comparison.The reported OpenKiwi models use STACKED for MT tags and ENSEMBLED for the other tasks.Source: Table 2 · p. 5, §5 · p. 5
Development results
- On development data, ensembled systems perform best overall, and stacking is effective for word-level MT labels; the predictor-estimator leads among single systems except on source and gap tags.APE-QE is superior for source and gap word-level tags.Source: §5 · p. 5, Table 1 · p. 5
6 Conclusions
OpenKiwi is a PyTorch framework for training word- and sentence-level QE systems, outperforming other open-source toolkits and achieving state-of-the-art word-level results. It was subsequently used as a WMT 2019 baseline and as a foundation for winning systems.
Conclusion
- OpenKiwi supports training word- and sentence-level QE systems on new data, outperforms other open-source toolkits on both levels, and yields state-of-the-art word-level results.Source: §6 · p. 5