SYSTRAN's Pure Neural Machine Translation Systems

1 Introduction

• Their system builds on the open-source seq2seq-attn project and contributes features back to the MT community.The project was initiated by the Harvard NLP group.Source: §1 · p. 1
• NMT training can take weeks, and the authors explore language-specific behavior because they do not expect one approach to work optimally across diverse languages.They report running hundreds of experiments and aim to inject existing language knowledge into training.Source: §1 · pp. 1–2
• The paper shares recipes for first-generation production-ready generic translation systems as a starting point for specialized systems.Reported features can improve accuracy, efficiency, learning speed, or model speed.Source: §1 · p. 2
• The production aims include terminology customization, preservation of format tags and placeholders, business-application integration, and cloud, hosted, or mobile deployment.Source: §1 · p. 2
• The authors prioritize language-specific settings and practical techniques for bringing NMT to more users over uniform translation options or research state-of-the-art systems.Source: §1 · p. 2

2 System Description

• The system uses seq2seq-attn’s encoder-decoder framework, with LSTM, global-general attention, and input-feeding in the decoder.Source: §2 · p. 2
• The framework supports bidirectional encoders, pretrained embeddings, unknown-word handling, and CPU or GPU training and decoding.Source: §2 · p. 2
• The authors implemented additional features and made many available in the seq2seq-attn repository.Source: §2 · p. 2
• The authors developed C-torch, a C-based decoder using Torch’s C APIs for finer runtime control and improved multithreading and CPU–GPU interaction.Source: §2 · p. 2
• The explored training space includes many meta-parameters, and optimal configurations differ by language pair.The report lists the explored meta-parameter space in Table 1.Source: §2 · p. 3, Table 1 · p. 3

3 Training Resources

• Generic-engine training is challenging because online translation covers varied domains and use cases, while available open-source corpora are domain-specific.Examples include legal, scientific-presentation, and colloquial text.Source: §3 · p. 3
• The release corpora combine available sources with weights, reducing institutional and colloquial shares for languages with large resources in favor of news and mixed webpages.Source: §3 · p. 3
• The authors defined baseline, medium, and very large corpora at 1M, 2–5M, and more than 10M sentences, respectively, to enable experiments at different scales.Most individual experimental results use baseline corpora, while online-system data amounts are reported separately.Source: §3 · p. 3
• Training extended to 18 epochs, or 22 for some languages, and optimal systems were generally reached around epoch 15, with behavior varying by language pair.English–Korean perplexity continued falling significantly through epochs 13–19, while Italian–English fell only marginally after epoch 10.Source: §3 · p. 3
• A 5M-sentence corpus trained for 10 epochs followed by an additional homogeneous 5M for 10 epochs improved scores by 0.8–1.2 over training on 5M for 20 epochs.Source: §3 · p. 4

4.1 Tokenization

• Most languages use generic tokenization, with Arabic clitic segmentation and German/Chinese splitting; enko and jaen use BPE, which does not optimally model Korean morphology.For most language pairs, the approach also combines a vocabulary shortlist and placeholder mechanism.Source: §4.1 · p. 4

4.2 Word Features

• Source and target case features improve translation quality, and neural recasing outperforms external recasers; target features particularly raise scores while also increasing perplexity.The framework supports arbitrary discrete encoder features and predicted decoder-side features shifted so each is generated for its annotated word.Source: Figure 1 · p. 5, §4.2 · p. 5

4.3 Named Entities (NE)

• The system replaces recognized source entities with placeholders, enforces placeholder output during beam search, then restores source strings or translations; substitution needs language-pair-specific rules.Entity recognition is cross-validated across parallel sentences, including matching entity types and aligned words or phrases.Source: §4.3 · pp. 5–6

4.4 Guided Alignments

• For the first five epochs, guided alignment boosts training over baseline, but decay slows training thereafter; a power outage left convergence unresolved.The method adds a loss that makes attention weights more like reference alignments.Source: §4.4 · pp. 6–7, Figure 2 · p. 7

4.5 Politeness Mode

• 70 to 80% politeness-generation accuracy was observed in a small English-to-Korean test, with meaning preserved in 43 of 50 sentences across modes.The approach adds a source token indicating the target sentence’s politeness mode; sometimes the unknown-word replacement scheme copies that token into the output.Source: §4.5 · p. 7

4.6 Customization

• Customer-facing customization includes terminology, domain, and style adaptation, and can extend translation memory for human post-editing workflows.The preliminary domain adaptation incrementally trains a generic neural model on newly available in-domain data.Source: §4.6 · p. 7, §4.6.1 · p. 8

4.6.1 Generic Specialization

• After one additional epoch, adapt comes within .91 BLEU of full while each in-domain epoch takes under 50 seconds versus over 17 hours for full training.The preliminary results report effectiveness with nearly 50k additional in-domain words; the comparison does not give full training the adapted model’s in-domain vocabulary.Source: Figure 3 · p. 8, §4.6.1 · p. 8

4.6.2 Post-editing Engine

• Both NPE systems outperform SPE; multi-source NPE improves NPE translation, while its converged accuracy is similar to pure NMT.The multi-source input concatenates the original source and MT hypothesis without source-target alignment.Source: Figure 4 · p. 9, §4.6.2 · p. 9

5.1 Pruning

• Up to 60% of parameters were pruned without performance loss after retraining; sparse float multiplication used less memory once pruning reached 35%.The sparse-matrix experiments use Eigen and target smaller-memory deployment.Source: Table 8 · p. 10, §5.1 · p. 10, Figure 6 · p. 10

5 Performance

• Runtime throughput is reported as compatible with production requirements on CPU and GPU, while training on 2x5M sentences took just under one month on their hardware.Their target is under a week; leveraging multiple GPUs remains engineering work, and they seek to understand speed/performance trade-offs while minimizing network size.Source: §5 · pp. 9–10

5.2 Distillation

• A 2x300 student gave slightly higher accuracy with 70% fewer parameters and 30% faster decoding; a same-structure student beat its teacher by nearly 1.5 BLEU.The sequence-level distillation experiment used teacher-generated translations; for its initial experiment, 35-best hypotheses were filtered by normalized sentence-level n-gram matching.Source: §5.2 · p. 11

5.3 Batch Translation

• Batch translation targets faster decoding of large texts by forwarding K×B sequences for beam size K and batch size B, then updating each sentence's beam path sequentially.Encoder outputs and source-side attention softmax must be masked for variable-length sentences; Figure 7 reports speedup using a student model, beam size 2, and float precision on a four-thread desktop Intel i7.Source: §5.3 · p. 11, Figure 7 · p. 11

5.4 C++ Decoder

• The Eigen-based C++ CPU decoder runs on par with or faster than Torch, especially with multiple threads.It uses a graph-based implementation, can load and infer from Torch models, and Figure 8 is cited for its better parallelization.Source: §5.4 · p. 11

6.1 Automatic Scoring and system comparison

• Perplexity and BLEU scores were highly correlated across language pairs, with lower perplexity associated with higher BLEU.BLEU was calculated on internal test sets, and systems used different amounts of training data; Figure 9 plots the scores for all language pairs.Source: §6.1 · p. 12, Figure 9 · p. 13

6 Evaluation

• The authors note that internal validation and test sets make direct comparison with other research systems difficult, so they emphasize human evaluation.For this first release, the human-ranking protocol covered English↔French and English→Korean.Source: §6.1 · p. 12, §6 · p. 12

6.2 Human Ranking Evaluation

• Reviewers ranked three randomized outputs per sentence, prioritizing accuracy while also considering fluency, and could assign tied ranks.Three professional translators or linguists fluent in both languages evaluated each sentence without being told which outputs were human or machine translations.Source: §6.2 · p. 12, §6 · p. 12
• English→French NMT outperformed existing online engines and best-of-breed technology, while French→English remained behind Google Translate and especially Microsoft Translator.The generic English→Korean model was closer to human translation and clearly outperformed existing online engines; an in-domain model outperformed the reference human translation.Source: §6.2 · p. 14

6.3 Qualitative Evaluation

• Missing words or sentence parts were the most salient error, although reviewers judged half of these omissions acceptable.The authors interpret acceptable omissions as cases where the system conveyed the point or summarized, while noting that unacceptable omissions need correction.Source: §6.3 · p. 14
• NMT substantially improved fluency, slightly reduced meaning-selection errors, and handled morphology better, but still struggled with quotes and some case generation.The authors also identify teaching more expressions and connecting already introduced named-entity handling as areas for improvement.Source: §6.3 · p. 14

7 Practical Issues

• Training examples were capped in length, leaving longer sentences unfamiliar; hard splitting can make each segment seem like a complete sentence.The authors are exploring alignment-based split points and continuation tokens, while short and incomplete inputs absent from training motivate adding synthetic data.Source: §7 · p. 15
• Recurring noise in training data can surface in translations, such as English→Korean outputs systematically converting currency amounts.The authors report NMT seems more robust to noise than other technologies, but argue training data should be prepared more aggressively.Source: §7 · p. 15
• Soft alignment leaves source–target alignment information absent from NMT output, although the authors use heuristics to show traditional alignments to end users.The authors describe source–target alignment as a requirement for their users.Source: §7 · p. 15

8.1.1 Additional LM

• Shallow fusion reranks neural-network word candidates with an external n-gram language model, whose scores can select out-of-vocabulary candidates absent from the neural network's scoring.The experiments use a target vocabulary of 60,000 words, while n-gram models can be trained with much larger vocabularies.Source: §8.1.1 · p. 15

8.1.2 Ensemble Decoding

• Ensemble decoding averages target-word probabilities from multiple neural models to exploit differences in their predictions.The authors describe training three to eight isolated models, balancing ensemble size against memory and training speed.Source: §8.1.2 · p. 16

8.2 Extending word embeddings

• External word embeddings can draw on larger monolingual corpora and different domains, and fixed embedding weights may support a different application vocabulary.The weights for words in the new vocabulary must come from the same embedding space as the original weights.Source: §8.2 · p. 16

8.3 Unknown word handling

• The authors are implementing and evaluating approaches that learn unknown-word handling within the attention and decoder networks.The stated aim is to let the model control decoding and unknown-word handling for single-word replacement.Source: §8.3 · p. 16

9 Conclusion

• For language pairs with abundant monolingual and bilingual training data, phrase-based MT still performs better than NMT, which remains limited by vocabulary size and use of monolingual data.Source: §9 · p. 16
• No single NMT configuration works universally across language pairs, and BPE is not always the best choice.The authors also say attention needs greater accuracy to better control translation output and user interactions.Source: §9 · p. 16
• Upcoming releases will continue experiments injecting linguistic knowledge and applying engineering solutions to practical NMT issues.Source: §9 · p. 16

• The analysis covers 50 English–French sentences and compares NMT, Google SMT, and RBMT outputs.Source: Table 11 · p. 21
• Error categories include entity handling, morphology, meaning selection, word ordering or fluency, and missing or duplicated words.Source: Table 11 · p. 21
• Errors are major when a reader cannot understand the sentence without the source; otherwise, they are minor.Each erroneous word is counted in only one category, even when multiple problems apply.Source: Table 11 · p. 21

A Remarkable Results

• The section presents “remarkable” translations, including both positive and negative examples, from evaluation across multiple languages.Source: A Remarkable Results · p. 22
• The French rendering of the report about Chinese public opinion appears in two variants in the supplied examples.Source: A Remarkable Results · p. 22
• For the Chinese “Morning News” sentence, the supplied English rendering says, “Morning News, now it’s our interactive time.”Source: A Remarkable Results · p. 22
• For the English–French example, one translation renders “Faced with the Nice decisions” as “Nice view of the decisions.”Source: A Remarkable Results · p. 23

B Online System Parameters

• The online systems use four LSTM layers, 500-dimensional word embeddings, dropout of 0.3, and bidirectional RNNs.Source: B Online System Parameters · p. 23
• Column Guided Alignment records whether guided alignments were used and the epoch at which the feature was stopped.Source: B Online System Parameters · p. 23
• Table 12 is identified as listing parameters for online systems.Source: Table 12 · p. 23