Source-linked AI summary
Training Neural Machine Translation To Apply Terminology Constraints
Georgiana Dinu, Prashant Mathur, Marcello Federico, Yaser Al-Onaizan
TL;DR
NMT often lacks adequate terminology for specialized domains, while constrained decoding adds inference overhead. The paper trains a standard NMT model to use runtime terminology annotations, achieving a better speed–performance trade-off than constrained decoding and production suitability.
Problem
NMT terminology is inadequate in specialized domains, while existing constrained-decoding methods add inference overhead and can be brittle in realistic conditions.
Method
The paper trains a standard Transformer NMT model to use runtime-provided terminology encoded as inline annotations and source factors.
Results
The method works with unseen terminology in zero-shot tests and offers a better speed–performance trade-off than constrained decoding.
Takeaways & Limitations
The approach is particularly suitable for production environments because it avoids constrained decoding’s inference-time computational overhead.
Takeaways & Limitations
The paper leaves comparison with tag-and-replace approaches to future work because those systems differ from constrained decoding and have limited contextual flexibility.
Abstract
from arXiv · showhide
This paper proposes a novel method to inject custom terminology into neural machine translation at run time. Previous works have mainly proposed modifications to the decoding algorithm in order to constrain the output to include run-time-provided target terms. While being effective, these constrained decoding methods add, however, significant computational overhead to the inference step, and, as we show in this paper, can be brittle when tested in realistic conditions. In this paper we approach the problem by training a neural MT system to learn how to use custom terminology when provided with the input. Comparative experiments show that our method is not only more effective than a state-of-the-art implementation of constrained decoding, but is also as fast as constraint-free decoding.
1 Introduction
The paper targets runtime terminology integration in NMT, where constrained decoding incurs substantial inference overhead. It instead trains a standard NMT model to use inline terminology annotations, evaluating the approach in zero-shot experiments against DBA constrained decoding.
- Motivation: Runtime terminology is needed because NMT output remains inadequate for many specialized domains, especially when domain data is too narrow for adaptation.Terminology databases maintained by language specialists provide domain-specific term translations.
- Related work: Constrained decoding injects matched target terms during inference but can increase translation time threefold with beam size 5.DBA reduces overhead relative to grid beam search but still incurs this practical latency cost.
- Contribution: The proposed method trains NMT to use runtime terminology supplied as inline annotations and additional source factors, eliminating inference-time computational overhead.The approach modifies training rather than the decoding algorithm and uses a standard Transformer architecture.
- Evaluation: Experiments use English-to-German translation with terms from two terminology dictionaries in a zero-shot setting with unseen terminology.The method is compared with DBA constrained decoding.
- Contribution: The terminology mechanism is decoupled from the NMT architecture, allowing it to operate with a standard Transformer as architectures change.The input combines running text with inline annotations.
2 Model
The model learns during training to use terminology entries supplied with the source sentence, while preserving the original sequence-to-sequence architecture. Training annotations mark matched terminology and are selectively sampled to retain baseline performance without terminology input.
- Terminology input: The model receives source sentences augmented with terminology entries triggered by matching their source sides.Terminology is integrated as inline annotations, with variants retaining or discarding the source term.
- Terminology input: Additional source factors signal whether tokens belong to running text, source terms, or target terms.The two input variants are illustrated in Table 1.
- Training objective: Training teaches the model to bias translations toward provided target terms, including terms absent from the training data.The training procedure annotates terminology pairs whose target terms occur in the reference.
- Training data creation: Randomly ignoring some matched annotations limits augmented data so the model also performs in the constraint-free condition.This sampling is needed because the term bases are large and annotating all matches would affect most sentences.
- Training data creation: Overlapping matches retain the longest source term, while approximate character matching permits inflected forms and compounds.These rules support morphological variation in terminology matching.
3 Experiments
Experiments compare terminology-aware training with constrained decoding on WMT 2018 English–German data, measuring terminology use, BLEU, and decoding latency. The trained models achieve high term-use rates with baseline-comparable speed, while constrained decoding can alter translations or incur quality and latency costs.
- Evaluation setup: The models are trained on 2.2 million Europarl and news-commentary sentences, with term-annotated sentences added at approximately 10% of the original training data.The implementation uses joint source–target BPE with a 32K-token vocabulary and a two-layer encoder and decoder Transformer.
- Evaluation setup: The evaluation uses WMT newstest 2013/2017 with Wiktionary and IATE terminology, including 727 sentences and 884 terms for Wiktionary and 414 sentences and 452 terms for IATE.Test sentences were selected when the reference contained the matched term, and decoding speed, BLEU, and term-use rates were reported.
- Terminology use: Baseline translation already uses terminology translations at 76%, while train-by-append reaches around 90% and train-by-replace reaches 93%–94%.Constrained decoding reaches 99% on Wiktionary but only 82% on IATE.
- Decoding speed: Both trained settings decode at speeds comparable to the baseline and are three times faster than constrained decoding.The timing comparison uses batch size 1 on a single GPU P3 AWS instance.
- Translation quality: All train-by models improve BLEU by 0.2 to 0.9 points, whereas constrained decoding generally leaves BLEU unchanged and decreases it on IATE with beam size 5.Constrained decoding can alter a translation to accommodate a term even when a variation or identical term is already present.
- Approximate matches: With approximate reference matches, train-by-append remains better than the baseline, while constrained decoding shows BLEU reductions of 0.9–1.3 points.The humanitarian → humanit¨ar example illustrates errors when a source-matching term requires target-side inflection.
4 Conclusion
The paper presents a black-box terminology-integration method trained to use run-time external terminology with a generic neural MT architecture. It reports zero-shot copying, flexible inflection, and a favorable speed–performance trade-off for production use, while leaving comparison with tag-based systems for future work.
- The method trains a generic neural MT architecture to use external terminology supplied at run time.
- Zero-shot experiments show that the model triggers copy behavior for terms never seen during training.
- The method can use terminology either in its provided form or with inflection, unlike less flexible constrained approaches.
- The authors report no better speed–performance trade-off among neural MT constrained-decoding algorithms and consider the method suitable for production environments.
- Comparison with tag-based systems is left for future work because those systems replace fixed placeholders without grammatical flexibility.
5 Aknowledgments
The authors thank Wael Hamza, Faisal Ladhak, Mona Diab, and the anonymous reviewers for their advice and comments.
- The authors thank Wael Hamza and Faisal Ladhak for their advice and comments.
- The authors thank Mona Diab for her advice and comments.
- The authors thank the anonymous reviewers for their advice and comments.
NMT Sockeye train parameters
The Sockeye NMT training configuration uses a two-layer transformer with 8 attention heads, 512-dimensional model states, and a 32,302-token vocabulary.
- The transformer uses ReLU activations, 8 attention heads, and 2 encoder layers.
- The model uses 512-dimensional representations and feed-forward layers with 2048 hidden units.
- Training uses cross-entropy with valid normalization, 0.1 label smoothing, and a vocabulary of 32,302 tokens.
- Source and target embeddings each use 512 dimensions, one factor, and a vocabulary of 32,302 tokens.
- Source and target embedding configurations use zero dropout and no additional factor configurations.