Source-linked AI summary
A Survey of Domain Adaptation for Neural Machine Translation
Chenhui Chu, Rui Wang
TL;DR
NMT performs strongly when large parallel corpora are available, but domain-specific data are often scarce or absent, making domain adaptation important. This paper surveys and categorizes NMT domain-adaptation techniques, comparing them with SMT approaches and reporting findings across methods. Its supported scope is bounded by the lack of direct transferability from SMT and by uncombined NMT data-centric methods.
Problem
Domain-specific NMT needs high-quality translation despite scarce or nonexistent domain-specific corpora and limited parallel data.
Method
The paper provides a comprehensive survey that categorizes NMT domain-adaptation techniques as data centric or model centric and compares them with SMT techniques.
Results
Fine tuning concatenated multi-domain data performed best among the compared multi-domain methods, while mixed fine tuning outperformed multi-domain and ordinary fine tuning.
Takeaways & Limitations
The survey organizes prominent NMT domain-adaptation techniques and outlines practical research directions for domain-specific translation.
Takeaways & Limitations
Most SMT domain-adaptation methods cannot be directly applied to NMT, and NMT data-centric methods had not yet been combined in the surveyed work.
Abstract
from arXiv · showhide
Neural machine translation (NMT) is a deep learning based approach for machine translation, which yields the state-of-the-art translation performance in scenarios where large-scale parallel corpora are available. Although the high-quality and domain-specific translation is crucial in the real world, domain-specific corpora are usually scarce or nonexistent, and thus vanilla NMT performs poorly in such scenarios. Domain adaptation that leverages both out-of-domain parallel corpora as well as monolingual corpora for in-domain translation, is very important for domain-specific translation. In this paper, we give a comprehensive survey of the state-of-the-art domain adaptation techniques for NMT.
1 Introduction
Domain adaptation for NMT addresses the scarcity and domain mismatch of parallel data by leveraging out-of-domain parallel corpora and in-domain monolingual or parallel data. This survey categorizes prominent approaches and compares them with earlier SMT methods.
- Motivation: High-quality domain-specific MT is needed because general-purpose systems have limited applications and often perform poorly in specific domains.The paper contrasts a Chinese-English patent corpus with a much smaller spoken-language corpus, illustrating domain-specific data scarcity.
- Motivation: Domain adaptation leverages out-of-domain parallel corpora together with in-domain monolingual corpora to improve in-domain translation.The paper also discusses in-domain parallel corpora and synthetic corpora as data sources.
- Background: SMT domain adaptation methods are mainly data centric or model centric, but many cannot transfer directly to NMT because SMT and NMT differ substantially.Data-centric methods select training data or generate pseudo-parallel data, whereas model-centric methods interpolate models or instances.
- Survey scope: The survey categorizes NMT domain adaptation into data-centric and model-centric approaches and reviews methods developed or adapted for NMT.The data-centric category includes in-domain monolingual, synthetic, and parallel corpora.
- Survey scope: The paper presents a comprehensive survey because prior surveys covered general NMT or domain adaptation in other fields, not NMT domain adaptation specifically.It also compares NMT techniques with SMT techniques and discusses practical scenarios and future research directions.
2 Neural Machine Translation
Attention-based NMT uses an encoder, decoder, and attention mechanism to generate translations autoregressively. Its large parameter count makes abundant parallel data necessary to avoid overfitting, creating a bottleneck for low-resource domains and languages.
- Architecture: Attention-based NMT uses an encoder-decoder architecture in which an input sentence is converted into a generated translation.The architecture described is the recurrent RNNsearch model.
- Architecture: The model has three main components: an encoder, decoder, and attention model.The encoder produces continuous representations, while the decoder predicts target words.
- Attention: The attention mechanism computes a weighted average of encoder representations and supplies the resulting context vector to the decoder.Alignment weights determine how encoder hidden states contribute to each decoding step.
- Decoding: The decoder combines its recurrent hidden state, the previous word, and the context vector before a maxout and softmax layer predict the next word.The decoder is conceptually an RNN language model with embeddings and a GRU layer.
- Low-resource limitations: Abundant parallel corpora are required to avoid overfitting because NMT contains large numbers of parameters in its encoder, decoder, and attention model.This requirement is identified as the main bottleneck for low-resource domains and languages.
3 Domain Adaptation for SMT
SMT domain adaptation methods broadly either select or generate domain-related data, or interpolate models from different domains. While many data-centric methods transfer directly to NMT, model-centric methods generally do not because NMT is an integral model rather than a combination of independent models.
- Data Centric: SMT data-centric methods select domain-relevant out-of-domain data or generate pseudo-parallel sentences when parallel data are insufficient.Selection commonly uses language models or joint models, while generation can use information retrieval, self-enhancing, or parallel word embeddings.
- Data Centric: Most SMT data-centric methods transfer directly to NMT, but their non-NMT selection or generation criteria yield only modest improvements.The limitation is that these criteria are not closely related to NMT.
- Model Centric: SMT model-centric methods interpolate models from different domains at either the model level or the instance level.Model-level interpolation combines separately trained models, whereas instance-level interpolation weights examples or corpora during training.
- Model Centric: NMT adapts these ideas through model ensembles and objective-function instance weights, but most SMT interpolation methods cannot be applied directly.SMT combines independent models, while NMT is an integral model itself.
4.1 Data Centric
NMT data-centric adaptation uses monolingual, synthetic, and parallel corpora to improve in-domain translation. Methods include synthetic-data generation, multi-domain training, and data selection, but combining these approaches remains an open direction.
- Using In-Domain Monolingual Corpora: In-domain monolingual data supports NMT adaptation through language-model fusion, copied-source training, multitask learning, and synthetic parallel-data generation.Back translation can convert target-side monolingual sentences into synthetic parallel corpora for NMT training.
- Using In-Domain Monolingual Corpora: Synthetic data generation is reported effective using target-side monolingual data, source-side monolingual data, or both.The method strengthens NMT by generating synthetic parallel data from monolingual sentences.
- Multi-Domain: Multi-domain training concatenates corpora, appends domain tags to source sentences, and oversamples smaller corpora for balanced domain attention.The domain tag primes the decoder to generate sentences for the selected domain.
- Multi-Domain: Fine-tuning a concatenation system on in-domain data performs best among the compared multi-domain training methods, while a discriminative method performs better than multi-domain training.The comparison included concatenation, staking, selection, and ensemble methods.
- Data Selection: NMT data selection can use source-sentence embedding similarity to select out-of-domain sentences close to in-domain data.This criterion is designed to be more related to NMT than criteria transferred from SMT.
- Data Selection: No studies had combined the NMT data-centric methods, although they are considered complementary in principle.Combining these methods is identified as a future direction.
4.2 Model Centric
Model-centric domain adaptation changes NMT training objectives, procedures, architectures, or decoding to better handle domain differences. The survey covers fine tuning, regularization, fusion, domain control, discriminators, ensembling, and neural lattice search.
- Training Objective Centric: Model-centric methods modify NMT training functions or procedures to obtain a better in-domain training objective.
- Training Objective Centric: Fine tuning first trains on resource-rich out-of-domain data, then adapts parameters on resource-poor in-domain data.It can use parallel sentences extracted from comparable corpora.
- Training Objective Centric: Mixed fine tuning resumes a converged out-of-domain model on oversampled mixtures of in-domain and out-of-domain data.It addresses overfitting from small in-domain datasets and is reported to outperform both multi-domain training and conventional fine tuning.
- Training Objective Centric: Regularization methods such as dropout, L2-regularization, and tuneout address overfitting during fine tuning.The survey considers regularization complementary to mixed fine tuning.
- Architecture Centric: Architecture-centric methods include deep fusion, domain discriminators, and domain control through tags or word-level features.Deep fusion integrates an in-domain RNNLM with the NMT decoder, while discriminators predict source-sentence domains jointly with NMT.
- Decoding Centric: Decoding-centric methods combine NMT with additional decoding strategies, including shallow fusion, ensembling, and neural lattice search.Shallow fusion rescored hypotheses using weighted NMT and RNNLM probabilities, while neural lattice search was reported better than conventional decoding.
5 Domain Adaptation in Real-World Scenarios
Real-world domain adaptation must handle cases where the input domain is unknown and where different types of auxiliary data are available. The survey therefore recommends choosing or combining methods according to the scenario rather than prescribing one universal technique.
- When pseudo-parallel in-domain data exists within out-of-domain data, sentence selection is preferred; additional monolingual data supports LM and NMT fusion.
- Combining methods is possible when out-of-domain parallel data and in-domain monolingual data are both available.
- Mixed fine tuning applied to synthetic parallel data showed better performance than either method alone.
- Unknown input domains make domain prediction crucial for good translation in practical systems such as online translation engines.One strategy classifies domains first and then translates with corresponding models.
- NMT approaches for unknown domains include domain tags or features, similar-sentence retrieval, and test-specific fine tuning.
6 Future Directions
Future work targets broader and more practical domain adaptation for NMT, including newer architectures, domain-specific external knowledge, multilingual transfer, and adversarial methods.
- 6.1 Domain Adaptation for State-of-the-art NMT Architectures: Most domain adaptation studies use RNN-based NMT, while CNN-based NMT and Transformer models offer important future targets.The newer architectures are reported to outperform state-of-the-art RNN-based NMT in translation quality and speed.
- 6.2 Domain Specific Dictionary Incorporation: Using domain-specific dictionaries and knowledge bases remains a major research question for practical NMT deployment.Existing dictionary methods are described as limited, especially for terminology translation and rare or low-frequency words.
- 6.3 Multilingual and Multi-Domain Adaptation: Crosslingual transfer is needed when same-language-pair out-of-domain parallel data is unavailable, motivating combined multilingual and multi-domain adaptation.The survey notes that no existing system packages multiple language pairs and domains together.
- 6.4 Adversarial Methods: Adversarial methods represent another possible direction because they have been applied to domain adaptation in computer vision and machine learning.These methods use an adversarial objective involving a domain discriminator to minimize an approximate domain discrepancy distance.
7 Conclusion
Domain adaptation for NMT is a new and important topic for practical machine translation. The paper reviews recent techniques, compares them with SMT approaches, and outlines future research directions.
- 7 Conclusion: The survey presents a comprehensive review of domain adaptation techniques for NMT developed mainly during the preceding two years.It also compares NMT techniques with methods studied in SMT over the preceding two decades.
- 7 Conclusion: The paper identifies connections to general NLP, computer vision, and machine learning as future work.The authors state that such connections are intended to promote research in domain adaptation for NMT.