Source-linked AI summary
Nematus: a Toolkit for Neural Machine Translation
Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, Maria Nădejde
TL;DR
Nematus addresses the need for an accurate, usable, and extensible Neural Machine Translation toolkit. It provides an attentional encoder–decoder system with a novel conditional GRU with attention and experimentation features. The toolkit supported top-performing WMT and IWSLT systems and has also been used for production MT systems.
Problem
Neural Machine Translation had established a new state of the art, motivating a toolkit that supports high translation quality, usability, and extensibility.
Method
Nematus implements an attentional encoder–decoder architecture with a novel conditional GRU with attention, minimum risk training, and configurable experimentation and ensemble features.
Results
Nematus was used to build top-performing systems for WMT and IWSLT shared translation tasks and to train MT systems currently in production.
Takeaways & Limitations
Nematus is presented as an accessible, well-documented toolkit for NMT research and production use.
Abstract
from arXiv · showhide
We present Nematus, a toolkit for Neural Machine Translation. The toolkit prioritizes high translation accuracy, usability, and extensibility. Nematus has been used to build top-performing submissions to shared translation tasks at WMT and IWSLT, and has been used to train systems for production environments.
1 Introduction
Nematus is an extensible Neural Machine Translation toolkit built on a compact, high-quality tutorial codebase. It adds functionality for usability and performance and has supported top-performing WMT and IWSLT systems.
- Nematus is a new toolkit for Neural Machine Translation.
- The toolkit grew from the compact, simple, easy-to-extend dl4mt-tutorial codebase, which produced high translation quality.
- Nematus extends the tutorial with functionality based on recent Neural Machine Translation research.
- Nematus implements an attentional encoder–decoder architecture similar to Bahdanau et al. (2015), with implementation differences.
- The added functionality aims to enhance usability and performance.
2 Neural Network Architecture
Nematus uses an attentional encoder–decoder architecture with several implementation differences, including a novel conditional GRU with attention and flexible input and regularization options.
- Decoder architecture: Nematus initializes the decoder hidden state with the mean of the source annotations rather than the last backward-encoder annotation.
- Implementation differences: The decoder uses a feedforward tanh hidden layer before softmax, while embedding layers omit additional biases.
- Implementation differences: Nematus changes the decoder phase order to Look, Update, Generate, simplifying the implementation relative to Bahdanau et al.’s Look, Generate, Update sequence.
- Implementation differences: Nematus supports recurrent Bayesian dropout, multiple source-side factors, and tied embedding matrices as optional architectural features.Factor embeddings are concatenated to form the input representation at each source position.
- Conditional GRU with attention: Its conditional GRU with attention, cGRUatt, combines two GRU transition blocks with an intervening attention mechanism.The first block produces an intermediate representation, attention computes a context vector, and the second block produces the decoder hidden state.
- Conditional GRU with attention: The attention mechanism uses the entire source annotation set and the intermediate hidden state to compute a context vector.The alignment weights are normalized between source position i and target position j.
3 Training Algorithms
Nematus trains with cross-entropy minimization using stochastic-gradient methods and also supports minimum risk training and loss-based early stopping.
- The default objective is cross-entropy minimization on a parallel training corpus, optimized with stochastic gradient descent or adaptive-learning-rate variants.Supported variants include Adadelta, RmsProp, and Adam.
- Nematus supports minimum risk training to optimize an arbitrary sentence-level loss function, including several machine-translation metrics.Supported metrics include smoothed sentence-level BLEU, METEOR, BEER, and interpolations of implemented metrics.
- Training can use early stopping based on cross-entropy or an arbitrary user-defined loss function.
4 Usability Features
Nematus provides configurable and documented tooling for experimentation, model use, decoding, and visualization. Its utilities include ensemble support and beam-search inspection.
- Experimentation support: Command-line configuration, extensive documentation, and sample training setups support experimentation with Nematus models.These features are presented alongside tools for model experimentation and visualization.
- Model application: Nematus supports ensembles of models with different architectures when their output vocabularies match.The ensemble probability distribution is computed as the geometric average of the individual models’ distributions.
- Decoding utilities: The toolkit includes beam-search decoding, parallel corpus scoring, and n-best-list rescoring scripts.
- Visualization: Nematus visualizes attention weights and beam-search graphs, including the DE→EN example shown in Figure 1.The figure visualizes search for translating “Hallo Welt!” with beam size 3.
5 Conclusion
Nematus is presented as an accessible, well-documented Neural Machine Translation toolkit whose implementation differences are considered of wider interest because of its empirically strong performance. It supports research and has also been used for production MT systems.
- Contribution: Nematus’s implementation differences from Bahdanau et al. are considered of wider interest because of its empirically strong performance.
- Usability: The authors hope Nematus will provide researchers with an accessible and well-documented toolkit.
- Practical use: Nematus has been used to train MT systems currently in production, so its scope extends beyond research.
- Availability: Nematus is available under a permissive BSD license.