Source-linked AI summary
Curriculum Learning and Minibatch Bucketing in Neural Machine Translation
Tom Kocmi, Ondrej Bojar
TL;DR
The paper asks whether ordering NMT training examples can improve translation quality or reduce training time. It compares linguistically homogeneous minibatches with curricula that gradually introduce complexity, finding no minibatch-bucketing effect but up to 1 BLEU point from some curricula, without shortening training time.
Problem
NMT training is computationally complex, motivating example orderings that might improve translation quality or reduce training time.
Method
The study tests minibatches grouped by linguistic similarity and curricula that progressively introduce examples of greater complexity.
Results
Some curricula improve translation quality by up to 1 BLEU point, while minibatch bucketing has no difference from randomly composed minibatches.
Takeaways & Limitations
Curriculum ordering can improve translation quality, but the tested minibatch homogeneity does not affect training and curriculum ordering does not shorten training time.
Takeaways & Limitations
The curriculum strategy cannot shorten training time because performance remains below baseline until the gradually organized epoch is finished.
Abstract
from arXiv · showhide
We examine the effects of particular orderings of sentence pairs on the on-line training of neural machine translation (NMT). We focus on two types of such orderings: (1) ensuring that each minibatch contains sentences similar in some aspect and (2) gradual inclusion of some sentence types as the training progresses (so called "curriculum learning"). In our English-to-Czech experiments, the internal homogeneity of minibatches has no effect on the training but some of our "curricula" achieve a small improvement over the baseline.
1 Introduction
The paper addresses costly NMT training by organizing minibatches and corpus order to improve translation quality or reduce training time. It motivates minibatch-based training because large NMT datasets make full-batch training impractical and padding wastes computation.
- NMT training is resource-intensive because state-of-the-art systems may require weeks on GPUs and large parallel corpora.
- Training optimizes millions of model parameters by gradient descent against an objective commonly based on cross entropy.
- Minibatch training averages errors over small example groups, stabilizing gradients while enabling flexible data handling and GPU parallelization.
- Full-batch training is impractical for NMT, while variable-length minibatches waste computation because shorter sentences require padding.
- The study organizes minibatches by linguistic similarity and orders corpus examples by increasing complexity to seek better quality or shorter training.
2 Minibatch Bucketing
Minibatch bucketing groups training examples with similar linguistic properties so each update can focus on a shared phenomenon. The method extends length-based bucketing from a computational optimization to a proposed modeling strategy.
- Length-based bucketing can improve parallel-processing speed by avoiding wasted computation on padding, whereas this work tests whether linguistic homogeneity improves learning.
- The method assumes homogeneous minibatches can highlight a particular language phenomenon and help gradients identify shared features.
- Minibatch bucketing groups sentences with similar linguistic properties, including length, conjunctions, nouns, proper nouns, or verbs.
- Training data are divided into feature-based buckets, shuffled within buckets, split into minibatch-sized groups, and then shuffled across groups.
3 Curriculum Learning
Curriculum learning presents NMT with increasingly complex examples, using linguistically motivated features to define difficulty. The paper adapts the schedule to use each example once per epoch while addressing rapid overfitting to recent example types.
- Curriculum learning starts with easier examples and gradually introduces more complex ones, a strategy previously shown to improve neural-network performance.
- The method requires continued training on easy examples because neural networks can quickly overfit, making some mixing strategy necessary.
- The paper modifies this strategy so each training example is used only once during an epoch, under a decreasing-bin-size condition.
- The selection schedule samples from the easiest bin first, progressively combines additional bins, and feeds accumulated examples as minibatches.
- Because NMT difficulty is unclear, the experiments use sentence length, coordinating conjunctions, and source- or target-side word-frequency ranks.
4 Experiments
The experiments compare minibatch bucketing and curriculum learning in English-to-Czech NMT. Bucketing follows the baseline without significant improvement, while several curricula improve later, with target-length organization performing best but creating adaptation and continuation challenges.
- Experimental setup: The experiments evaluate minibatch bucketing and curriculum learning using an NMT system and English-to-Czech data.The setup uses Neural Monkey with a standard attention-based encoder-decoder, the WMT 2017 training data, and BLEU evaluation after every 100k examples.
- Minibatch bucketing: Minibatch bucketing produces results slightly above the baseline but within the standard deviation range, so no significant improvement can be claimed.The comparison concerns bucketing examples by similar linguistic properties; the table reports results after one epoch.
- Curriculum learning: Several curriculum setups improve over the baseline, with target-side sentence-length bins achieving a gain of 1 BLEU point.The curriculum variants organize training examples by sentence length, coordinating conjunctions, word ranks, or combinations of word ranks.
- Quick adaptation or overfitting: Curriculum learning starts below the baseline and catches up later as training-example complexity becomes better matched to the fixed test-set complexity.The learning curves show bucketing closely following the baseline while curricula initially perform much worse.
- Quick adaptation or overfitting: The target-length curriculum initially trains on short sentences, adapts sharply when longer bins are added, and surpasses the baseline near the epoch’s end.Its learning curve shows very poor scores through more than half the training data, followed by large BLEU increases as longer sentences enter training.
- Quick adaptation or overfitting: Revisiting shorter sentences is necessary: simply sorting the corpus by sentence length causes training to fail and prevents reasonable performance.The reversed curriculum also deteriorates when long-sentence bins are prohibited, consistent with rapid adaptation to short-sentence data.
- Continuing the curriculum: After one curriculum epoch, restarting from easy examples harms performance early but can improve the first-epoch result, whereas shuffled continuation shows no further test-set improvement.The authors identify subsequent training as uncertain and report that the model may already be fixed in its current optimum.
5 Related Work
Prior work uses minibatch bucketing to improve training efficiency and curriculum learning to introduce complexity gradually, with reported performance benefits in several settings.
- Bucketing similar-length sentences produced up to a 4× training-time speedup, with buckets drawn randomly from the training set.The approach is also used in Nematus, an open-source NMT toolkit.
- Alternating bucket order by sentence length periodically revisited different-length buckets and improved both training speed and performance.
- Curriculum learning for a neural language model gradually increased vocabulary size by 5000 words per epoch and significantly improved performance.The final epoch used the full vocabulary and all examples.
- An automatic multitask-learning method selected examples using training signals to focus on subtasks and accelerate the main task.
- Uniform sampling from the training data remained a strong baseline for example selection.
6 Conclusion
The English-to-Czech experiments found no effect from linguistically homogeneous minibatches, while curricula gradually introducing more complex sentences improved translation quality by up to 1 BLEU point. However, the curriculum did not shorten training because performance remained below baseline until the organized epoch was completed.
- Similar linguistic properties within minibatches produced no difference from randomly composed minibatches.
- Up to 1 BLEU point improvement came from gradually including more complex sentences by length or vocabulary size.
- Curriculum learning displayed clear performance jumps as longer sentences entered the training data.
- The curriculum strategy could not shorten training because performance stayed well below baseline until the gradually organized epoch finished.
- The experiments showed quick adaptation to recent training examples, alongside a high risk of overfitting to their particular properties.