Source-linked AI summary
Dynamic Data Selection for Neural Machine Translation
Marlies van der Wees, Arianna Bisazza, Christof Monz
TL;DR
NMT training is costly, and the benefits of conventional data selection for NMT are less established than for PBMT. The paper introduces dynamic data selection, varying selected subsets across epochs, and finds gradual fine-tuning gives the strongest reported improvements. Its evaluation also includes settings without sizeable parallel in-domain text.
Problem
The paper examines how and to what extent NMT can benefit from data selection, given that conventional selection performs less reliably for NMT than PBMT.
Method
Dynamic data selection varies the selected training subset between epochs, including gradual fine-tuning that progressively reduces training size.
Results
Gradual fine-tuning improves over static data selection by up to +2.6 BLEU and over a high-resource general baseline by up to +3.1 BLEU.
Takeaways & Limitations
Gradual fine-tuning achieves competitive translation quality in approximately 20% of the training time of in-domain fine-tuning, even without parallel in-domain data.
Takeaways & Limitations
The experiments demonstrate applicability without sizeable parallel in-domain text by using independently sampled monolingual News Crawl data for WMT.
Abstract
from arXiv · showhide
Intelligent selection of training data has proven a successful technique to simultaneously increase training efficiency and translation performance for phrase-based machine translation (PBMT). With the recent increase in popularity of neural machine translation (NMT), we explore in this paper to what extent and how NMT can also benefit from data selection. While state-of-the-art data selection (Axelrod et al., 2011) consistently performs well for PBMT, we show that gains are substantially lower for NMT. Next, we introduce dynamic data selection for NMT, a method in which we vary the selected subset of training data between different training epochs. Our experiments show that the best results are achieved when applying a technique we call gradual fine-tuning, with improvements up to +2.6 BLEU over the original data selection approach and up to +3.1 BLEU over a general baseline.
1 Introduction
NMT training is costly, yet it usually uses all available data even though some sentences may be irrelevant or harmful. The paper investigates NMT data selection and introduces dynamic selection to improve efficiency and translation quality.
- NMT training can take several weeks, motivating methods that reduce training cost.
- Using all available training data may be suboptimal because some sentences are irrelevant or harmful for a given translation task.
- Data selection can retain relevant bitext while reducing model size and training time.
- NMT data selection is challenging because NMT systems can underperform with limited parallel data and lack a separate large-scale target-side language model.
- Dynamic data selection varies the selected training subset across epochs without modifying the NMT architecture or parameters.
- The standard selection method is less effective for NMT than PBMT despite using the same training-data subsets.
- Gradual fine-tuning improves over static selection by up to +2.6 BLEU and over a high-resource general baseline by up to +3.1 BLEU.
2 Static data selection
Static data selection ranks sentence pairs by bilingual cross-entropy differences between in-domain and general corpora. The method selects the lowest-scoring pairs, with equivalent PBMT and NMT experiments using identical subsets.
- The method ranks sentence pairs by their cross-entropy difference relative to in-domain and general corpora.
- Language models are built separately for the source and target sides of in-domain and sampled general corpora.
- For each sentence pair, the method computes four cross-entropy scores across corpus types and bitext sides.
- The bilingual cross-entropy difference favors sentences similar to the in-domain corpus and dissimilar to the general corpus.
- Training uses the top n sentence pairs with the lowest cross-entropy-difference scores.
- The experiments use the same selected sentence pairs for equivalent PBMT and NMT comparisons.
- LSTM language models are examined as an alternative to n-gram models because they retain longer histories and avoid backoff for out-of-vocabulary words.
3 Dynamic data selection
Dynamic data selection varies the training subset across NMT epochs to retain relevant data while addressing limited-data and vocabulary-coverage problems. The paper examines sampling and gradual fine-tuning, which progressively changes selection composition or size without modifying the NMT architecture.
- Motivation: Dynamic selection targets efficiency while confronting NMT's vulnerability to low-resource training and reduced vocabulary coverage for rare words.These issues can arise when data selection discards material from the long tail.
- Dynamic data selection: Dynamic data selection varies selected training subsets between NMT epochs, unlike fixed-data training in other machine-translation paradigms.The approach exploits NMT's repeated passes over the training corpus.
- Dynamic data selection: The method begins with a relevance-ranked bitext and favors highly relevant sentences without completely discarding less relevant ones.The ranking uses CED scores, while the experiments keep a fixed vocabulary created from the complete bitext.
- Sampling: Sampling independently draws n sentence pairs without replacement for each epoch using weights derived from domain-specific CED scores.Higher-ranked sentences are selected more often, while each epoch's sample is independent of other epochs.
- Gradual fine-tuning: Gradual fine-tuning starts with general bitext and progressively retains only the top-ranked sentence pairs for successive groups of η epochs.Its selection size follows n(i) = α · |G| · β⌊(i−1)/η⌋, with α controlling start size and β the retention rate.
4 Experimental settings
The experiments compare static and dynamic data selection for German→English translation across four domains using PBMT and an in-house NMT system. Training data combines several general and domain corpora, while evaluation uses domain-specific development and test sets.
- Data and domains: The study evaluates static and dynamic data selection on German→English translation across EMEA, Movies, TED, and WMT domains.The training mixture includes Commoncrawl, Europarl, News Commentary, EMEA, Movies, and TED.
- Machine translation systems: The PBMT comparison system is an in-house Moses-like system with test-set-specific PRO tuning and interpolated target-side language models.The target-side language model uses Kneser-Ney smoothing over 480M tokens, with interpolation weights optimized per test set.
- Machine translation systems: The NMT system is an in-house globally attentive encoder-decoder with four-layer unidirectional LSTMs, trained for 16 epochs on one NVIDIA Titan X GPU.The model uses 1,000-dimensional embeddings and layers, dropout probability 0.3, and beam size 12.
- Training and evaluation data: WMT lacks in-domain parallel training data, so its ranking language models use independently sampled sentences from WMT monolingual News Crawl corpora.The samples contain 3.3M German tokens or 3.5M English tokens.
- Training and evaluation data: Figure 2 compares Axelrod and random selections at the in-domain size and at 5%, 10%, 20%, and 50% of the complete bitext using identical subsets for PBMT and NMT.Results are averaged over three runs for four domains.
5 Results
Static data selection strongly benefits PBMT but yields smaller and less consistent gains for NMT. Dynamic selection, especially gradual fine-tuning, improves this trade-off by achieving competitive BLEU with substantially less training.
- Static data selection: PBMT selection often outperforms complete-bitext training, with selected subsets sometimes using only 5% of the bitext.Informed selections also outperform random selections of the same size.
- Static data selection: NMT selection results vary: only Movies improves over complete-bitext training, while other test sets show no noticeable improvement.The same |I|-sized selections that help PBMT never beat using exactly the available in-domain data for NMT.
- Static data selection: NMT is more sensitive than PBMT to small or low-quality selected datasets.PBMT appears better able to compensate through its large target-side monolingual language model.
- Static data selection: Static selection degrades most for NMT on EMEA and Movies, while WMT suffers considerably because the general bitext lacks in-domain data.For WMT, baseline translation quality is similar across MT systems before selection.
- Ranking models: Using LSTMs instead of n-gram language models for bitext ranking improves NMT by an average of 0.4 BLEU.The experiments subsequently use relevance rankings produced with LSTMs.
- Dynamic data selection: Weighted sampling produces no consistent pattern, whereas gradual fine-tuning consistently beats static selection and the general baseline in three of four test sets.The displayed gradual fine-tuning configuration is at least as fast as static selection using 20% of the bitext.
- Dynamic data selection: +2.6 BLEU is achieved over static selection for WMT news, and +3.1 BLEU over the complete baseline for TED talks.These gains come from the displayed gradual fine-tuning configuration with α = 0.5, β = 0.7, and η = 2.
- Comparison with fine-tuning: Domain-specific fine-tuning achieves +2.5–4.4 BLEU over the baselines but does not reduce training time and requires parallel in-domain text.Data selection does not surpass it, but reaches competitive translation quality in only 20% of the training time.
6 Further analysis
Further analyses attribute dynamic selection’s advantage to broader data coverage and show that gradual fine-tuning converges better despite less steady training. Its performance also depends on how learning rates are changed during training.
- Coverage: All methods use the same vocabulary, while dynamic selection reduces unseen test-set source word types closer to the complete-bitext baseline than static selection.The analysis links this reduction to observing more different sentence pairs during training.
- Random ranking: Random bitext ranking reduces data-selection success, but dynamic selection—especially gradual fine-tuning—still outperforms static selection.The authors attribute this residual advantage to improved exploration and, for gradual fine-tuning, repetition across epochs.
- Convergence: Static selection converges most steadily, but both dynamic techniques eventually reach lower development-set cross-entropy and higher test-set translation quality.The same behavior is reported for TED and the other test sets.
- Optimization: −1.1–2.0 BLEU results when learning-rate decay is applied after a fixed number of updates rather than after each gradual-fine-tuning epoch.Keeping the learning rate fixed leaves BLEU nearly unchanged or sometimes improves it, suggesting that altered search behavior may contribute to success.
7 Related work
Related work frames data selection as a way to reduce model size and training time or adapt to domains, with methods differing in their objectives and assumptions. NMT work applies comparable goals through selection, fine-tuning, and training-efficiency changes.
- SMT data selection: Earlier SMT data-selection methods target either smaller models and faster training or adaptation to a new domain.Domain-adaptation methods commonly rank sentences using information-theoretic relevance measures.
- SMT data selection: Similarity-based selection performs best when test and general corpora differ substantially, whereas coverage-based selection is superior when they are relatively similar.This comparison is reported for SMT data-selection methods.
- NMT adaptation: In NMT, domain adaptation typically trains on the complete bitext before fine-tuning on a smaller in-domain corpus.Other approaches combine fine-tuning with ensembles, domain tags, or backtranslated in-domain data.
- NMT efficiency: Other NMT efficiency work parallelizes models or data, changes network structure, reduces parameters through distillation, or boosts challenging data during training.Boosting challenging data is identified as most related because it adjusts training data during training.
8 Conclusions
The paper finds that conventional data selection is less reliable for NMT than PBMT and introduces dynamic selection by varying training subsets across epochs. Gradual fine-tuning consistently improves results while substantially reducing training time, including when no parallel in-domain data is available.
- Conclusions: State-of-the-art data selection performs reliably for PBMT but yields unreliable results for NMT.The comparison uses the same training-data subsets across the two paradigms.
- Conclusions: Dynamic data selection varies the selected training-data subset between epochs to make data selection profitable for NMT.The paper explores two dynamic-selection techniques.
- Conclusions: +2.6 BLEU over conventional static data selection and +3.1 BLEU over a high-resource general baseline are achieved by gradual fine-tuning.The technique gradually reduces training size.
- Conclusions: ∼20% of the training time is sufficient for gradual fine-tuning to approximate in-domain fine-tuning, even without parallel in-domain data.This result is reported as part of the paper’s conclusion.