Source-linked AI summary
Fast Domain Adaptation for Neural Machine Translation
Markus Freitag, Yaser Al-Onaizan
TL;DR
NMT domain adaptation must reconcile abundant out-of-domain data with limited in-domain data. The paper continues training an existing system on in-domain data and ensembles it with the baseline, yielding substantial in-domain gains with only slight out-of-domain degradation across two language pairs.
Problem
Domain adaptation remains a challenge because large out-of-domain training data can dominate a small in-domain dataset.
Method
The method continues training an already trained baseline model only on in-domain data, then ensembles the continue model with the baseline at decoding time.
Results
Up to 7.2 BLEU points and 10 points in TER are gained on in-domain test data while out-of-domain performance does not drop significantly.
Takeaways & Limitations
The approach adapts an existing NMT system to a new domain within hours rather than retraining on combined data for days or weeks.
Abstract
from arXiv · showhide
Neural Machine Translation (NMT) is a new approach for automatic translation of text from one human language into another. The basic concept in NMT is to train a large Neural Network that maximizes the translation performance on a given parallel corpus. NMT is gaining popularity in the research community because it outperformed traditional SMT approaches in several translation tasks at WMT and other evaluation tasks/benchmarks at least for some language pairs. However, many of the enhancements in SMT over the years have not been incorporated into the NMT framework. In this paper, we focus on one such enhancement namely domain adaptation. We propose an approach for adapting a NMT system to a new domain. The main idea behind domain adaptation is that the availability of large out-of-domain training data and a small in-domain training data. We report significant gains with our proposed method in both automatic metrics and a human subjective evaluation metric on two language pairs. With our adaptation method, we show large improvement on the new domain while the performance of our general domain only degrades slightly. In addition, our approach is fast enough to adapt an already trained system to a new domain within few hours without the need to retrain the NMT model on the combined data which usually takes several days/weeks depending on the volume of the data.
1 Introduction
NMT has become competitive with SMT, but domain adaptation remains a challenge because large out-of-domain data can overwhelm limited in-domain data. The paper adapts an existing NMT system by further training on in-domain data and decoding with a baseline–adapted ensemble.
- NMT has achieved comparable or better performance than traditional SMT on some translation tasks.
- Domain adaptation is challenging when large out-of-domain bilingual data accompanies a relatively small in-domain corpus.The out-of-domain data tends to dominate training and limits performance on in-domain test data.
- The method continues training an existing out-of-domain NMT system using only the small in-domain dataset.This adapts the neural-network parameters to the new domain without combining and retraining on all data.
- Decoding with an ensemble of the baseline and adapted models improves in-domain performance without deteriorating general-domain performance.
2 Related Work
Prior SMT domain-adaptation methods select or weight out-of-domain data, interpolate domain-specific systems, or create synthetic parallel data. NMT cannot directly use phrase-table interpolation, motivating adaptation through further training of an existing neural system.
- Domain adaptation was already an active research topic in traditional SMT and comprised three broad method categories.
- SMT domain-adaptation methods include selecting or weighting out-of-domain data using in-domain scores.These methods have been applied to language-model and translation-model training.
- Another SMT strategy interpolates in-domain and out-of-domain models, often by adding an in-domain system to an existing system.NMT lacks phrase tables, so this interpolation method cannot be directly applied.
- Semi-supervised training creates additional parallel data by translating or back-translating monolingual in-domain text.The synthetic data can retrain language or translation models, including NMT systems.
- Luong and Manning adapted an existing NMT system through further training on in-domain data, reporting a 3.8 BLEU-point absolute gain.The present work uses the same basic adaptation approach but ensembles the further-trained model with the original model and evaluates out-of-domain degradation.
3 Neural Machine Translation
The NMT system uses an attention-based encoder–decoder architecture. A bidirectional RNN encodes the source into hidden states, while the decoder uses recurrent states and attention-weighted encoder information to predict target words.
- The encoder uses a bidirectional RNN to map the source sentence into a sequence of hidden states.Each hidden state concatenates left-to-right and right-to-left recurrent representations.
- The decoder predicts the correct target translation by maximizing its conditional log-probability.
- At each time step, the model computes probabilities for target-vocabulary words.
- The word-probability network uses the previous target-word embedding, decoder hidden state, and an attention-weighted sum of encoder states.
- The decoder transforms the previous state and previous target-word embedding with a GRU before computing its recurrent state and context.
- A two-layer feed-forward network computes the alignment weights used to form the attention context.
4 Domain Adaptation
The proposed adaptation reuses a trained baseline, continues training only on limited in-domain data, and ensembles the resulting model with the baseline. This is intended to adapt quickly while preserving general-domain performance.
- The adaptation objectives are rapid construction of an adapted system and strong in-domain performance without significant general-domain degradation.
- Retraining an NMT system on more than 4 million parallel sentence pairs can take several weeks, motivating a method that avoids full retraining.
- The method continues training an existing baseline model for additional epochs using only in-domain training data.The resulting system is called a continue model.
- The continue model can overfit because the in-domain training set is small.Overfitting may improve training-set performance while worsening performance on unseen data.
- Ensembling the continue model with the baseline limits reliance on the adapted model and keeps performance close to the general-domain baseline.
5 Experiments
Experiments on German→English and Chinese→English show that continuing training on in-domain data improves adaptation but can damage out-of-domain performance, while ensembling preserves general-domain quality. The method also achieves strong human-evaluation results and substantially shorter adaptation times than combined-data retraining.
- German→English: German→English adaptation gained 4.4 BLEU and 3.1 TER points in-domain after two epochs, but lost 2.1 BLEU and 3.9 TER points out-of-domain.Continuing training for 20 epochs led to overfitting and degraded performance on both test sets.
- German→English: Ensembling the adapted and baseline models gained 4.2 BLEU and 3.7 TER points in-domain while losing only 0.2 BLEU and 0.6 TER points out-of-domain after 20 epochs.Each continued-training epoch took 1.8 hours, and two epochs already produced a strong in-domain system.
- German→English: Across German→English data sizes, the ensemble significantly improved in-domain quality, with only slight out-of-domain degradation and no further in-domain loss from additional epochs.Adding more in-domain data improved in-domain scores without significant out-of-domain degradation.
- German→English: Both continued and ensemble models significantly outperformed the German→English baseline in human evaluation, while the ensemble performed better than continued training alone.The evaluation used an experienced annotator and a randomly selected in-domain sample of 50 sentences scored from 0 to 5.
- Training time: Adaptation required 3 hours 40 minutes with all 192K German→English in-domain sentences, compared with 15 days 11 hours for combined-data retraining.For Chinese→English, all available in-domain data required 23 hours, while one combined-data epoch required 77 hours 30 minutes.
- Chinese→English: Chinese→English ensembling gained 7.2 BLEU and 10 TER points in-domain after six epochs while losing only slightly out-of-domain.Continuing training alone improved in-domain scores by 9.5 BLEU and 12.2 TER points but significantly reduced out-of-domain performance.
6 Conclusion
The paper presents a fast two-step method for adapting an existing NMT system to new domains: continue training on in-domain data and ensemble the adapted model with the baseline at decoding. Experiments on German→English and Chinese→English show strong in-domain gains while preserving out-of-domain performance with limited training data.
- 6 Conclusion: The method initializes an in-domain model from an existing out-of-domain baseline, continues training on in-domain data, and ensembles both models during decoding.The ensemble is intended to retain general-domain performance while adapting to the new domain.
- 6 Conclusion: The Chinese→English combined-data training setup contains 12.2M parallel sentences.This passage reports the size of the setup including both in-domain and out-of-domain data.
- 6 Conclusion: Continuing baseline training alone can produce gains of up to 9.9 points in BLEU and 12.2 points in TER, but significantly degrades out-of-domain translation quality.The continue model can also overfit the small in-domain training set beyond one or two epochs.
- 6 Conclusion: Up to 7.2 points in BLEU and 10 points in TER are gained on the in-domain test set without a significant out-of-domain performance drop.These results are attributed to ensembling the baseline and continue models.
- 6 Conclusion: The approach is evaluated on two diverse and challenging language pairs: German→English and Chinese→English.The reported experiments therefore cover both language pairs used in the paper.