Source-linked AI summary
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, Jeffrey Dean
TL;DR
NMT lagged phrase-based systems in speed, rare-word handling, and source-word coverage. GNMT combines a deep attention-based sequence-to-sequence system with wordpieces and quantized inference, reducing translation errors by roughly 60% against Google's previous phrase-based system.
Problem
NMT systems were slower, weaker on rare words, and sometimes failed to translate all source words, limiting accuracy relative to phrase-based systems.
Method
GNMT uses a deep LSTM encoder-decoder with attention, residual connections, wordpiece units, and low-precision inference.
Results
Roughly 60% fewer translation errors were observed than with Google's previous phrase-based production system across several language pairs.
Takeaways & Limitations
GNMT approaches average bilingual-human accuracy on some test sets and supports high-quality translation on much larger production datasets.
Takeaways & Limitations
Evaluation of particularly difficult translation cases and inputs longer than single sentences was left for future work.
Abstract
from arXiv · showhide
Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system.
1 Introduction
GNMT is a production NMT system designed to address NMT’s slower training and inference, rare-word handling, and incomplete source coverage. It combines architectural and decoding techniques with strong benchmark and production results.
- Motivation: NMT’s main weaknesses are slower training and inference, ineffective rare-word handling, and occasional failure to translate all source words.These weaknesses contributed to lower practical accuracy than phrase-based systems, especially on very large-scale datasets.
- System design: GNMT uses an 8-layer LSTM architecture with residual connections and connects decoder attention to the encoder’s top layer for parallelism.The system is presented as a production NMT implementation at Google.
- Rare words and decoding: Wordpieces balance character flexibility with word-level decoding efficiency while avoiding special treatment for unknown words.Beam search adds length normalization and a coverage penalty to encourage translation of all provided input.
- Benchmark results: 38.95 BLEU on WMT’14 English-to-French improved by 7.5 BLEU over one prior single model and by 1.2 BLEU over another.The comparisons concern single models without external alignment models reported in [31] and [45].
- Benchmark and production results: 24.17 BLEU on WMT’14 English-to-German was 3.4 BLEU better than a previous competitive baseline.On production data, human evaluations found 60% fewer translation errors than Google’s previous phrase-based system across several language pairs.
2 Related Work
SMT, particularly phrase-based systems, dominated machine translation before end-to-end NMT emerged. NMT has since approached and sometimes surpassed phrase-based quality, while research introduced mechanisms for rare words, coverage, data incorporation, and improved modeling.
- Statistical Machine Translation: SMT dominated machine translation for decades, with practical systems generally using phrase-based translation over word or phrase sequences.Phrase-based systems allow translated sequence lengths to differ.
- Neural Components in SMT: Neural networks improved SMT components, including phrase representations learned through a joint language model, but core phrase-based architectures retained their shortcomings.The joint language model produced an impressive improvement when combined with phrase-based translation.
- Early Neural Machine Translation: NMT initially had limited success but later approached phrase-based quality and surpassed it in a reported WMT’14 English-to-French result.The cited system achieved a 0.5 BLEU improvement over a state-of-the-art phrase-based system.
- Subsequent NMT Techniques: Subsequent NMT research addressed rare words, translation coverage, additional data, character and subword modeling, attention, and sentence-level loss minimization.The proposed techniques included attention, coverage modeling, multi-task and semi-supervised training, character encoders and decoders, subword units, varied attention mechanisms, and sentence-level objectives.
3 Model Architecture
GNMT uses an attention-based sequence-to-sequence architecture with encoder, decoder, and attention components. Its deep stacked LSTM design combines residual connections, a bidirectional bottom encoder layer, and a decoder RNN with softmax output.
- Core architecture: GNMT follows an attention-based sequence-to-sequence framework with encoder, decoder, and attention networks.The encoder maps each source symbol to a vector, while the decoder generates output symbols sequentially until EOS.
- Core architecture: The decoder combines an RNN, whose hidden state represents the next predicted symbol, with a softmax layer over candidate outputs.The decoder computes a probability distribution for each next output symbol.
- Deep recurrent networks: Deep stacked LSTMs are used for both encoder and decoder networks because depth captures subtle source- and target-language irregularities.Prior observations found that each additional layer reduced perplexity by nearly 10%.
- Residual connections: Residual connections improve backward-pass gradient flow, enabling very deep networks; GNMT generally uses 8 LSTM layers for both encoder and decoder.The connections allow training substantially deeper networks than ordinary stacked LSTMs, which work poorly beyond 8 layers in large-scale translation tasks.
- Encoder directionality: Only the bottom encoder layer is bidirectional, while all higher encoder layers are unidirectional to preserve computation parallelism.The forward and backward outputs are concatenated before being passed to the next encoder layer.
4 Segmentation Approaches
GNMT addresses open-vocabulary translation primarily with a data-driven wordpiece model that deterministically segments arbitrary character sequences while balancing character flexibility and word-level efficiency. It also considers a mixed word/character approach for representing out-of-vocabulary words.
- Wordpiece model: The wordpiece model is GNMT’s most successful approach for handling rare and out-of-vocabulary words, providing deterministic segmentation for any character sequence.It is data-driven and was originally developed for Japanese/Korean segmentation in Google’s speech recognition system.
- Wordpiece model: Wordpieces are generated before processing and decoded back into unambiguous words using special word-boundary symbols.At decoding, GNMT produces a wordpiece sequence that is converted into the corresponding word sequence.
- Wordpiece model: 8k to 32k wordpieces achieve good accuracy and fast decoding across the language pairs tested.The vocabulary is built by maximizing training-data language-model likelihood while limiting basic characters and mapping rarer ones to an unknown character.
- Wordpiece model: Wordpieces balance character flexibility with word efficiency and improve overall BLEU scores while avoiding the longer sequences and computation required by character-only models.A shared wordpiece model for source and target languages also guarantees identical segmentation of identical strings, facilitating copying of names and numbers.
- Mixed word/character model: The mixed word/character model retains a fixed word vocabulary but represents out-of-vocabulary words as prefixed sequences of constituent characters.The prefixes indicate character location within a word and distinguish these characters from normal in-vocabulary characters.
5 Training Criteria
Maximum-likelihood training does not directly reflect translation reward or teach robust ranking among incorrect outputs. GNMT therefore refines pretrained models with expected task reward, using GLEU for sentence-level reinforcement learning and a mixed ML/RL objective for stability.
- Motivation: Maximum-likelihood training fails to reflect BLEU-based task reward and does not explicitly rank incorrect outputs by their BLEU scores.Because incorrect outputs are never observed during training, the model is not directly trained to remain robust to its own errors.
- Task-reward refinement: Refining maximum-likelihood-pretrained models with task reward considerably improves results, even on large datasets.The refinement directly optimizes the translation task reward after maximum-likelihood pretraining.
- Sentence-level reward: GLEU uses the minimum of matching n-gram recall and precision for 1- to 4-token subsequences, with scores ranging from 0 to 1.It is designed for per-sentence reward because BLEU is a corpus-level measure with undesirable single-sentence properties.
- Sentence-level reward: GLEU correlates well with corpus-level BLEU while avoiding BLEU's drawbacks for per-sentence reward.This supports using GLEU in the reinforcement-learning experiments.
- Optimization procedure: Training first converges under maximum likelihood, then optionally refines the model with a mixed maximum-likelihood and expected-reward objective until development-set BLEU stops improving.The mixed objective is used to further stabilize training.
6 Quantizable Model and Quantized Inference
GNMT introduces training constraints and mixed-precision fixed-point arithmetic to make deep LSTM inference quantizable with minimal translation-quality impact. On WMT’14 English-to-French, TPU decoding preserves BLEU while substantially accelerating inference.
- Quantizable training: Deep LSTMs amplify quantization errors across unrolled steps and stacked layers, motivating explicit bounds on accumulator values.Both time-axis and depth-axis accumulators are constrained to [−δ, δ].
- Quantizable training: Quantization-aware training constrains RNN accumulators and softmax logits, enabling subsequent quantization without loss to translation quality.Accumulator bounds are annealed from δ = 8.0 to δ = 1.0, while γ remains fixed at 25.0.
- Evaluation: 0.0072 log-perplexity loss and no BLEU loss result when decoding with reduced-precision arithmetic on TPU.The comparison uses the WMT’14 English-to-French development set of 6003 sentences.
- Evaluation: 3.4 times faster TPU decoding than CPU decoding demonstrates the speed advantage of quantized arithmetic, while CPU decoding is 2.3 times faster than GPU decoding.The CPU–GPU difference is attributed partly to beam-search data transfer between host and GPU at every decoding step.
7 Decoder
GNMT decodes with beam search enhanced by length normalization and a coverage penalty to rank translation hypotheses. These refinements improve BLEU for ML-trained models, while offering less benefit after reinforcement-learning refinement.
- Beam-search scoring: Beam search uses length normalization and a coverage penalty to compare hypotheses of different lengths and encourage coverage of the source sentence.Without length normalization, standard beam search tends to favor shorter results; α and β control the two refinements, and α = β = 0 recovers pure probability-based beam search.
- Search efficiency: 30%−40%: pruning speeds up CPU beam-search decoding compared with no pruning, typically using beamsize = 3.0.GNMT typically keeps 8–12 hypotheses, while using 4 or 2 has only slight negative effects on BLEU scores.
- ML decoding results: 31.4 BLEU: length normalization and coverage penalty improve WMT’14 English-to-French development performance from 30.3 under pure sequence-probability scoring.The best reported improvement is up to +1.1 at α = 0.2 and β = 0.2, with many nearby settings performing similarly.
- RL-refined models: Length normalization and coverage penalty are less effective after reinforcement-learning refinement because the models already learn to attend to the full source sentence.This reduces under-translation and over-translation, which would otherwise incur BLEU or GLEU penalties.
8 Experiments and Results
GNMT was evaluated on WMT benchmarks and large Google-internal corpora using BLEU and human side-by-side assessments. Wordpieces, RL refinement, and ensembling improved benchmark results, while GNMT reduced translation errors by more than 60% against PBMT on major language pairs.
- Evaluation scope: Experiments covered WMT English-to-French and English-to-German benchmarks plus Google-internal English↔French, English↔Spanish, and English↔Chinese datasets.The study compared word-, character-, and wordpiece-based vocabularies and examined RL refinement and model ensembling.
- WMT results: WPM-32K achieved the best quality and fastest inference speed on WMT En→Fr, with a BLEU score of 38.95.The score was averaged over 8 trained models; the maximum individual-model BLEU score was 39.37.
- WMT results: WPM-32K achieved a BLEU score of 24.61 on WMT En→De, while wordpiece and mixed word/character models gained more than 2 BLEU points over word models.These models also gained about 4 BLEU points over previously reported results.
- RL refinement and ensembling: Ensembling 8 RL-refined models produced state-of-the-art results of 41.16 BLEU points on WMT En→Fr and 26.30 BLEU points on WMT En→De.RL refinement improved En→Fr BLEU by close to 1 point but slightly hurt En→De test performance despite improving development BLEU by about 0.4 points.
- Human evaluation: More than 60%: GNMT reduced translation errors compared to the PBMT model on major language pairs in side-by-side evaluations.On the sampled simplistic, isolated sentences, human and GNMT translations were sometimes nearly indistinguishable; difficult cases and longer inputs remained future work.
9 Conclusion
GNMT combines modeling, parallelism, and quantization techniques to achieve high-quality, efficient translation on public benchmarks and much larger production datasets. Human-rated comparisons show that GNMT approaches average bilingual translator accuracy on some tests and reduces errors versus the previous phrase-based system.
- GNMT’s translation quality approaches or surpasses all currently published results on the public WMT’14 benchmark.
- GNMT delivers high-quality translations on production datasets containing several orders of magnitude more data than the public benchmark.
- Wordpiece modeling handles open vocabularies and morphologically rich languages while supporting translation quality and inference speed.
- Model and data parallelism enable efficient training of state-of-the-art sequence-to-sequence NMT models in roughly a week.
- Model quantization drastically accelerates translation inference, enabling large models in a deployed production environment.
- Roughly a 60% reduction in translation errors is achieved versus the previous phrase-based production system on several popular language pairs.Human-rated side-by-side comparisons also show GNMT approaching the accuracy of average bilingual human translators on some test sets.