Source-linked AI summary
Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions
Marcin Junczys-Dowmunt, Tomasz Dwojak, Hieu Hoang
TL;DR
The paper addresses uncertainty about NMT quality in purely in-domain settings and whether its speed supports production use. It compares translation systems across the UN corpus and introduces AmuNMT, finding NMT competitive or better across all directions and practical on commodity hardware.
Problem
NMT generalization was known for out-of-domain data, but its performance in purely in-domain settings remained unclear for organizations with substantial proprietary data.
Method
The paper compares phrase-based SMT, hierarchical phrase-based MT, and NMT across 30 UN translation directions, and evaluates AmuNMT for translation speed.
Results
NMT is on par with or surpasses phrase-based SMT in all 30 directions, while AmuNMT achieves high-quality, high-performance translation on commodity hardware.
Takeaways & Limitations
A single GPU outmatches 16 CPU threads, and CPU-only AmuNMT can support demo systems and low-throughput deployment settings.
Takeaways & Limitations
The study did not cover architecture-related questions, and determining how reduced hidden-state size affects translation quality requires more experiments.
Abstract
from arXiv · showhide
In this paper we provide the largest published comparison of translation quality for phrase-based SMT and neural machine translation across 30 translation directions. For ten directions we also include hierarchical phrase-based MT. Experiments are performed for the recently published United Nations Parallel Corpus v1.0 and its large six-way sentence-aligned subcorpus. In the second part of the paper we investigate aspects of translation speed, introducing AmuNMT, our efficient neural machine translation decoder. We demonstrate that current neural machine translation could already be used for in-production systems when comparing words-per-second ratios.
1. Introduction
The paper compares phrase-based, hierarchical phrase-based, and neural machine translation across 30 directions using the United Nations Parallel Corpus, and introduces AmuNMT to study deployment speed.
- The study compares phrase-based SMT, hierarchical phrase-based MT, and NMT across fifteen language pairs and thirty translation directions.
- Experiments use the UN Parallel Corpus v1.0, including an approximately 11M-sentence subcorpus aligned across Arabic, Chinese, English, French, Russian, and Spanish.
- AmuNMT is introduced as an efficient neural machine translation decoder for evaluating translation speed.
- With one GPU per machine, the proposed neural setup could replace Moses in translation speed, while multiple GPUs would exceed it by far.
2. Training data
The experiments use a six-language, sentence-aligned UN corpus with official development and test sets, applying preprocessing and BPE-based subword segmentation before evaluation.
- The fully aligned UN subcorpus contains consistently aligned sentences across Arabic, Chinese, English, French, Russian, and Spanish.
- Official development and test sets each contain 4,000 sentences aligned across all official languages, enabling evaluation in any translation direction.
- Sentences longer than 100 words were discarded, and training data was lowercased and tokenized with the Moses tokenizer.
- BPE provides open-vocabulary translation with a fixed vocabulary of 30,000 subword units per language, which are reassembled during evaluation.
3. Phrase-based SMT baselines
The phrase-based SMT baselines use multiple aligned training partitions, a 5-gram language model, and additional operation-sequence and word-class models with pruning.
- The baseline training corpora were split into four equally sized parts and aligned with MGIZA++ using five iterations each of Model 1 and the HMM model.
- A 5-gram language model was trained from target parallel data, pruning higher-order n-grams that occurred only once.
- Additional baselines included a 5-gram operation sequence model and a 9-gram word-class language model trained with KenLM.
- Significance pruning was applied to the phrase table, compact phrase table, and reordering data structures.
4. Neural translation systems
The neural systems are attentional encoder-decoders trained with Nematus using mini-batches, bounded sentence length, large embeddings and hidden layers, gradient clipping, and Adadelta.
- The NMT system is an attentional encoder-decoder trained with Nematus.
- Training used mini-batches of 40 and a maximum sentence length of 100.
- The models used word embeddings of size 500, hidden layers of size 1024, and gradient-norm clipping at 1.0.
- Models were trained with Adadelta for 1.2M iterations, with checkpoints saved every 30,000 iterations.
5. Phrase-based vs. NMT – full matrix
Across the full 30-direction UN test matrix, NMT is generally comparable to or better than phrase-based SMT, with especially large gains for Chinese and Arabic pairs.
- NMT is comparable to or better than phrase-based SMT in every direction except fr-es.
- 7–9 BLEU points separate NMT from PB-SMT for translation directions involving Chinese.
- NMT also shows large improvements for translations into and out of Arabic, without special Arabic preprocessing.
6. Phrase-based vs. Hiero vs. NMT – language pairs with English
For English-involving directions, Hiero improves over PB-SMT for Chinese but remains below NMT, while extended NMT training produces further gains in this experimental setting.
- The figure covers all language pairs involving English and compares hierarchical translation with NMT models trained for 1.2M and 2.4M iterations.The 1.2M configuration used a batch size of 40 and approximately eight days of training.
- Hiero significantly outperforms PB-SMT for Chinese-English and English-Chinese but achieves less than half of NMT’s improvement.
- For French-English and Russian-English, Hiero is best when PB-SMT and NMT results are close.
- 0.3–1.3 BLEU gains result from training NMT for another eight days, reaching 2.4M iterations in total.The extended training increased total training time to 16 days per neural system.
7. Efficient decoding with AmuNMT
AmuNMT is designed for efficient neural decoding, with vocabulary selection and checkpoint averaging addressing speed–quality and model-efficiency trade-offs. Across throughput, startup time, memory, and latency, the experiments show deployment-relevant performance, especially with GPU parallelization.
- Decoder design: AmuNMT is a C++ neural MT toolkit centered on an efficient beam-search inference engine for Nematus-trained models.Its design focuses mainly on efficiency and usability, with multi-device execution, multithreading, model ensembling, vocabulary selection, and integrated subword segmentation.
- Checkpoint ensembling and averaging: Checkpoint ensembling improves quality by up to 2 BLEU points but reduces translation speed as ensemble size grows.Averaging the eight latest checkpoints yields similar quality to ensembling the four latest checkpoints, while averaging more models gives uneven results.
- Vocabulary selection: K = 75 and K′ = 75 reduce the average sentence vocabulary to about 1,250 items while preserving BLEU scores.Vocabulary selection is especially useful for CPU decoding, whose time is dominated by computations in the final output layer.
- Beam size versus speed and quality: A beam size of 5 is used because larger beams provide little additional quality while making decoding significantly slower.For English–French, beam size 5 is only 0.2 BLEU below the maximum observed quality.
- AmuNMT versus Moses and Nematus: 1,192 words per second is achieved by AmuNMT GPU with vocabulary selection, versus 455 words per second for the phrase-based baseline.Without vocabulary selection, AmuNMT GPU reaches 865 words per second and is nearly twice as fast as Moses; AmuNMT CPU is three times faster than Nematus CPU.
- Multi-GPU decoding: 3,368 words per second is reached with four GPUs, increasing to 4,423 with vocabulary selection.AmuNMT starts in less than 10 seconds, whereas Nematus may need several minutes before producing its first translation; the AmuNMT model remains an exact Nematus implementation.
- Serving and memory: About 24 models could fit on one 8GB GPU, and multiple GPUs can be parallelized and scheduled for query-based serving.The selected NMT model is approximately 300 MB and requires little translation-time overhead.
- Low-latency translation: AmuNMT GPU has 20 times lower single-sentence latency than Moses and Nematus under serial execution.This evaluation uses one CPU thread or process, a beam size of 5 for NMT systems, and reports milliseconds per sentence with lower values preferred.
8. Conclusions and future work
The paper finds that NMT matches or surpasses phrase-based SMT across all 30 UN translation directions, while AmuNMT enables high-quality translation on commodity hardware. CPU-only deployment remains viable for low-throughput settings, but architecture questions remain open.
- Translation quality: NMT is on par with or surpasses phrase-based SMT in all 30 translation directions, with substantial BLEU gains for language pairs involving Chinese and respectable gains for Arabic.The authors note that human evaluation is still needed to verify these results.
- Translation speed and hardware: AmuNMT achieves high-quality, high-performance neural translation on commodity GPUs, with a single GPU outperforming 16 CPU threads on server-grade Intel Xeon CPUs.The tested GPUs are broadly available for gaming PCs and graphics workstations.
- Translation speed and hardware: The CPU-only version of AmuNMT supports demo systems and can be viable in low-throughput settings.The authors propose training with one GPU and reusing existing Moses CPU machines for initial deployment.
- Future work and limitations: The study does not cover architecture-related questions, including whether halving hidden-state size would improve performance fourfold without reducing translation quality.Answering the quality question requires additional experiments.