Source-linked AI summary
Domain Control for Neural Machine Translation
Catherine Kobus, Josep Crego, Jean Senellart
TL;DR
Machine translation quality suffers from domain differences in style, structure, and terminology. The paper introduces runtime domain control in one multi-domain NMT network, and reports consistent improvements for known and predicted domains, including an out-of-domain test.
Problem
NMT systems are sensitive to domain mismatch, motivating methods that handle specialized terminology and style across domains.
Method
The paper integrates domain information into a single multi-domain NMT network using additional tokens or word-level features.
Results
The Feature approach consistently improves translation quality across tasks, with an average improvement of 0.80 BLEU3, while sentence-level prediction and a new Dialogs domain also outperform Join.
Takeaways & Limitations
Domain-adapted translation can be performed with one network covering multiple domains without re-estimating model parameters for each available domain.
Takeaways & Limitations
Sentence-level domain classification is challenging with short context, and document-level translation remains future work.
Abstract
from arXiv · showhide
Machine translation systems are very sensitive to the domains they were trained on. Several domain adaptation techniques have been deeply studied. We propose a new technique for neural machine translation (NMT) that we call domain control which is performed at runtime using a unique neural network covering multiple domains. The presented approach shows quality improvements when compared to dedicated domains translating on any of the covered domains and even on out-of-domain data. In addition, model parameters do not need to be re-estimated for each domain, making this effective to real use cases. Evaluation is carried out on English-to-French translation for two different testing scenarios. We first consider the case where an end-user performs translations on a known domain. Secondly, we consider the scenario where the domain is not known and predicted at the sentence level before translating. Results show consistent accuracy improvements for both conditions.
1 Introduction
NMT quality is sensitive to domain mismatch because domains differ in style, structure, and terminology. The paper proposes runtime domain control to extend generic models to specific domains without lowering generic-data quality.
- Domain mismatch can dramatically deteriorate translation quality when training and testing data differ strongly.
- Word ambiguities require different translations across contexts, such as medical and political uses of “administer.”
- Domain control supplies domain information so one diverse-data model can choose appropriate terminology and sentence structure.
- The proposed goal is to cover specific domains with specialized terminology and style without reducing quality on generic data.
2 Related Work
Prior domain adaptation work includes data selection, model mixtures, fine-tuning, and neural models supplied with meta-information. This paper differs by targeting one network that covers multiple domains.
- Statistical MT adaptation has used in-domain data selection and mixtures of in-domain models.
- Neural MT adaptation has fine-tuned a generic out-of-domain model on in-domain data for limited additional training time.
- The paper instead aims to perform domain-adapted translation with one network covering multiple domains.
- Recent NMT approaches provide topic or other meta-information on the encoder, decoder, or both sides.
3 Neural MT
The system uses an encoder-decoder NMT architecture with bidirectional LSTM encoding, recurrent decoding, and global attention over encoder states.
- The NMT system follows an encoder-decoder architecture with multilayer recurrent networks using LSTM hidden units.
- A bidirectional encoder reads the source sequence in both directions and combines the encoders’ final outputs into source representations.
- The decoder predicts target tokens recurrently and stops when it predicts the end-of-sentence symbol.
- Global attention compares the current target hidden state with every source hidden state to derive alignment weights.
- The context vector is computed as a weighted average of all source hidden states.
4 Domain control
Domain control is integrated through either a sentence-level domain token or word-level domain features. The feature approach attaches domain information to source-word representations without expanding the source-word vocabulary.
- Two domain-control techniques are implemented: an additional token and a domain feature.
- Additional token: The additional-token method appends an artificial domain tag to each source sentence so the network can attend to sentence-pair domain information.
- Additional token: Domain tags are selected to avoid overlap with words in the source-language vocabulary.
- Word Feature: The feature method extends each word embedding with cells encoding domain information while keeping separate word and domain-tag vocabularies.
- Word Feature: Under the feature framework, sentence-level domain information is added to every source word on a word-by-word basis.
5 Experiments
Experiments evaluate domain-controlled English-to-French NMT across six domains, comparing single-domain, joined, token-controlled, feature-controlled, and automatically classified-domain configurations. Feature-based domain control consistently improves translation quality on covered domains, remains beneficial with sentence-level domain prediction, and outperforms the joined model on a new domain.
- Data and configurations: Experiments use English-to-French data from six domains: IT, Literature, Medical, News, Parliamentary, and Tourism.Medical, News, and Parliamentary corpora are public; IT, Literature, and Tourism data are proprietary.
- Data and configurations: Four configurations compare six single-domain models, a domain-agnostic joined model, and joined-data models using token or feature domain information.The Token and Feature networks are trained on all available training data while differing in how domain information is introduced.
- Known-domain evaluation: The joined model outperforms all single-domain models on their corresponding test sets, indicating gains from additional training data.The accuracy differences are smaller for Parliamentary and Tourism, which are more represented in the joined training data.
- Known-domain evaluation: Feature-based domain control consistently improves translation quality across tasks, whereas the additional-token approach improves some tasks but degrades others relative to the joined model.The reported average improvement is 0.80 on test sets except Parliamentary, where the improvement is 0.26.
- Translation examples: Feature control selects more domain-adapted medical translations, including suivies attentivement for “carefully observed” and effectuées for “administered.”These examples differ from the joined model’s soigneusement surveillées and à l’ordre du jour outputs.
- Unknown and new-domain evaluation: When domains are unknown, an RNN classifier predicts a domain per sentence before Feature-model translation; this condition still improves accuracy over the joined model despite some misclassification.On Dialogs, a new domain absent from training, the classifier assigns each sentence to one source domain and the approach outperforms the joined model.
6 Conclusions and Further Work
The method enables domain-adapted translation with one network covering multiple domains, without re-estimating parameters for each domain. Further work targets softer domain signals, joint classification and translation, and document-level translation.
- A single domain-aware network performs domain-adapted translation across multiple domains without parameter re-estimation.
- Future improvements would replace hard domain decisions with vectors measuring sentence proximity to each domain.
- The authors plan joint domain classification and translation to avoid dependence on the RNN classifier.
- The study identifies document-level translation as a direction for extending the current work.