Source-linked AI summary
An Empirical Exploration of Curriculum Learning for Neural Machine Translation
Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J Martindale, Paul McNamee, Kevin Duh, Marine Carpuat
TL;DR
Neural machine translation is expensive to train, and it is unclear how to measure sample difficulty for curriculum learning. The paper explores curriculum designs probabilistically on German–English translation, finding that some improve convergence speed without BLEU loss, while outcomes depend on curriculum choices and hyperparameters.
Problem
NMT training is prohibitively slow, and it is unclear how to quantify sample difficulty when designing curricula for the task.
Source: §1 · p. 1Method
The authors use a probabilistic view of curriculum learning to explore difficulty criteria and schedules on a German–English translation task.
Source: §3 · p. 3, §5.2 · p. 5, §8 · p. 9Results
20 of 100 curricula improved convergence speed at no loss in BLEU.
Source: §8 · p. 9Takeaways & Limitations
Curriculum learning can train expensive NMT models effectively, but careful design of curriculum hyperparameters matters in practice.
Source: §1 · p. 2No clear single best curriculum strategy emerges from the experiments.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Machine translation systems based on deep neural networks are expensive to train. Curriculum learning aims to address this issue by choosing the order in which samples are presented during training to help train better models faster. We adopt a probabilistic view of curriculum learning, which lets us flexibly evaluate the impact of curricula design, and perform an extensive exploration on a German-English translation task. Results show that it is possible to improve convergence time at no loss in translation quality. However, results are highly sensitive to the choice of sample difficulty criteria, curriculum schedule and other hyperparameters.
1 Introduction
The paper explores curriculum learning for neural machine translation, where training is prohibitively slow and the right difficulty measure and curriculum design remain unclear. Its experiments find that curricula can improve convergence without reducing translation quality, but results depend on design choices.
1 Introduction
- NMT training is prohibitively slow, while it remains unclear how to quantify sample difficulty or combine curricula with existing data-organization methods.Prior work examined one-epoch training, leaving the effect of curricula through convergence unresolved.Source: §1 · p. 1
- The paper uses probabilistic sampling to explore model-score and linguistic difficulty criteria alongside easy-to-hard, dynamic-sampling, and boosting schedules.Sampling probabilities change with example difficulty and the curriculum schedule.Source: §1 · p. 1
- Curriculum learning improved convergence speed without loss of translation quality, and strategies beyond strictly easy-to-hard ordering also showed benefits.The experiments also found that results were highly sensitive to hyperparameters, with no single best strategy emerging.Source: §1 · pp. 1–2
- The empirical exploration uses a German-English translation task and implements its training strategies in Sockeye.The modification is publicly available.Source: §1 · p. 1
2 Related Work
Prior work applies curriculum learning and related ordering strategies across learning tasks, while NMT studies have not established how curricula behave through convergence. This paper revisits sample ordering in a unified framework on a German-English task.
2 Related Work
- Curriculum learning, introduced by Bengio et al. (2009), selects examples and their order, with easier examples hypothesized to benefit training.Prior applications include vocabulary growth for neural language models and feature-based paragraph ordering for word embeddings.Source: §2 · p. 2
- Related ordering strategies include spaced repetition, sequencing tasks by difficulty, and multi-task learning schedules.This work focuses on organizing samples within a single task.Source: §2 · p. 2
- NMT training commonly randomizes sentence-pair order or groups similar-length samples for computational efficiency, unlike curriculum learning’s statistical-efficiency aim.Curriculum learning seeks fewer training batches to reach a given performance.Source: §2 · p. 2
- The first NMT curriculum study tested one epoch and reported quality gains from easy-to-hard sentence-length and vocabulary-frequency ordering.Whether those findings hold through convergence remained unclear.Source: §2 · p. 2
- Prior dynamic-sampling work selects examples using model scores, domain relevance, or training-cost changes, while the balance with randomization and bucketing remains unclear.The paper tests these ideas in a unified framework on German-English data through convergence.Source: §2 · p. 2
3 A Probabilistic View of Curriculum Learning
The paper represents curriculum learning as phase-dependent sampling probabilities over training examples. This view supports varied schedules and criteria while requiring only changes to the data-sampling procedure.
3 A Probabilistic View of Curriculum Learning
- At each training phase, a distribution assigns sampling probabilities to examples, which can shift from favoring easy samples toward including harder ones and eventually uniform sampling.Phases may correspond to checkpoints or epochs; Figure 1 illustrates an example progression.Source: §3 · p. 3, Figure 1 · p. 3
- The probabilistic framework supports easy-to-difficult, reverse, and uniform-sampling schedules by specifying different distributions across phases.Uniform sampling corresponds to standard training with random mini-batches.Source: §3 · p. 3
- The method changes only data sampling and is modular with respect to the optimizer’s learning-rate schedule and minibatch shuffling.The optimizer needs sampling probabilities, which abstracts away from the chosen difficulty criteria.Source: §3 · p. 3
- Examples can be grouped into shards of similar difficulty, with distributions defined over shards rather than individual examples.Figure 2 depicts shards as blocks, with darker shades indicating greater difficulty; patch width does not encode sample count.Source: §3 · p. 3, Figure 2 · p. 3
4 Sample Difficulty Criteria
The paper evaluates sample difficulty using translation-model scores and linguistic features. Linguistic criteria include sentence length and word-frequency ranks, with multiple source- and target-side variants.
4 Sample Difficulty Criteria
- Difficulty criteria comprise how well an auxiliary translation model captures a pair and linguistic features independent of a translation model.Figure 3 presents score distributions and Jenks Natural Breaks shardings for selected criteria on 151,627 German-English TED Talks sentence pairs.Source: §4 · pp. 3–4, Figure 3 · p. 4
- The one-best score measures the auxiliary model’s probability for its top translation given the source sentence, indicating its confidence in that prediction.The prediction need not be the correct translation.Source: §4 · p. 4
- Sentence-length criteria rank source and target sentences separately or by their combined length, reflecting the challenge of longer sentences for NMT.The paper describes long-distance dependencies as one difficulty associated with longer sentences.Source: §4 · p. 4
- Word-frequency criteria use either the rarest word’s frequency rank or the average word-frequency rank, computed for source, target, or concatenated sentences.Using the maximum rank is equivalent to gradually increasing vocabulary size.Source: §4 · p. 4
5 Methods
The methods group samples by difficulty, vary which difficulty shards are available over successive curriculum phases, and preserve shuffled, length-based mini-batching. They compare several schedules that change the order or frequency with which difficulty classes are presented.
5.1 Data Sharding
- Jenks Natural Breaks groups samples by difficulty while minimizing within-class variance and maximizing between-class variance.The authors use it instead of score thresholds or even-sized shards, which respectively make balanced breaks difficult or risk difficulty fluctuations within and across shards.Source: §5.1 · pp. 4–5
5.2 Curriculum Schedule
- The default schedule adds harder shards over phases, reverse starts with the hardest shards, and noshuffle visits available shards deterministically from easiest to hardest.The schedules test easy-first ordering and whether deterministic shard visits change training compared with shuffled visits.Source: §5.2 · p. 5
- Boost adds a copy of the hardest shard after all difficulty classes have been processed, while reduce repeatedly removes and restores easy shards so harder shards are seen more often.These variants combine easy-first training with increased emphasis on difficult shards.Source: §5.2 · pp. 5–6
5.3 Training Strategy
- Training uses shards as the schedule’s visible data, shuffles samples within each shard, and buckets them by length before drawing mini-batches.Visible shards are shuffled except under noshuffle; the schedule updates at a fixed batch frequency, and training continues until validation perplexity fails to improve for 32 checkpoints.Source: §5.3 · p. 6
6 Experiment Setup
The experiments use German–English TED Talks data and Sockeye LSTM translation models, comparing curriculum training with a no-curriculum baseline. The setup also includes an auxiliary model for producing sample scores and sets a fixed curriculum update frequency.
Data
- Experiments use the German–English Multitarget TED Talks Task corpus, with about 150k training sentence pairs and about 2k sentences each for development and test.The data are tokenized and split into subwords with separate 30k-symbol BPE models for source and target.Source: §6 · p. 6
NMT Setup
- Sockeye models use 512-dimensional embeddings, one LSTM layer per encoder and decoder, 4096-word-count batches, Adam, and a dev-set validation set for early stopping.The initial learning rate is either 0.0002 or 0.0008.Source: §6 · p. 6
- The baseline has no curriculum, accesses the full training set, and uses length-bucketed mini-batches.Its training data are randomly split into the same five shards used by curriculum models.Source: §6 · p. 6
- An auxiliary model with 200 rather than 512 hidden units generates one-best scores for training samples as a reference for studying curriculum learning.Its training time may cancel out time saved by curriculum learning in practice.Source: §6 · pp. 6–7
Curriculum Learning Setup
- The curriculum implementation is integrated into Sockeye, and its update frequency is set to 1000 batches, matching the checkpoint frequency.Curriculum training is enabled using a file of sentence-level scores.Source: §6 · p. 7
7 Results
Curriculum learning can shorten convergence without sacrificing BLEU, but performance depends on difficulty criteria, schedules, and other hyperparameters. Vocabulary frequency performs as well as the more expensive one-best score, while sentence length offers little benefit beyond early training.
7 Results
- In early training, all curricula improved over baseline validation BLEU by checkpoint seven with the smaller learning rate, whereas only the reverse schedule beat baseline with the higher rate.Similar early-training trends across other difficulty criteria were also sensitive to initial learning rate.Source: §7 · p. 7, Figure 5 · p. 7
- 20 of 100 curriculum strategies converged earlier than the baseline without loss in BLEU.Average source word frequency with the reverse schedule reduced training time by 19% to 30%; one-best score with the boost schedule processed 19% fewer mini-batches at learning rate 0.002, with comparable BLEU (28.4 vs. 28.1).Source: §7 · p. 7, Table 2 · p. 8, Table 3 · p. 8
- Vocabulary frequency performs as well as the more expensive one-best score, while sentence length helps convergence time only slightly and in one of ten configurations.The sentence-length comparison was 74,000 vs. 79,000 mini-batches; its curricula could outperform the baseline early, but the baseline caught up by convergence.Source: §7 · pp. 7–8
- For the average-word-frequency criterion, default and reverse schedules can yield close performance, while ordered data shards (noshuffle) do not improve over default sampling.The authors report that training-time conclusions from Zhang et al. (2017) do not hold for their dataset and schedules; that earlier study trained for one epoch with a different strategy.Source: §7 · p. 8
- Increasing the number of mini-batches between curriculum updates raised validation BLEU more quickly, but convergence time showed no clear trend and BLEU at convergence stayed the same.Table 1 studies update frequency for the default schedule, average German source-word-frequency ranking, and initial learning rate 0.0002.Source: Table 1 · p. 7, §7 · p. 8
- No curriculum schedule consistently outperforms the others, and the optimal schedule for a complexity criterion can change with the initial learning rate.The default schedule yielded better results at learning rate 0.0008 but not 0.0002.Source: §7 · pp. 7–8
8 Conclusion
The conclusion reports an extensive curriculum-design exploration for German-English TED translation, where 20 of 100 curricula improved convergence speed without loss in BLEU. It also notes that results on other language pairs and datasets remain to be studied.
8 Conclusion
- The impact on other language pairs and datasets remains to be studied.Source: §8 · pp. 8–9
- In a controlled exploration of curriculum learning for expensive NMT models, 20 of 100 curricula tried improved convergence speed without BLEU loss; “easy to hard” was not the only useful sample-ordering strategy.Source: §8 · p. 9
- The authors use a probabilistic view of curriculum learning to explore curriculum designs in controlled settings.The exploration covers different difficulty criteria and curriculum schedules.Source: §8 · p. 9
A Supplementary Material
The supplementary material provides figures comparing training-set statistics and validation BLEU curves, along with decoding-performance tables for two initial learning rates.
A Supplementary Material
- Figure 6 presents statistics for the 151,627-sample GE-EN TED Talks training set, split into five Jenks Natural Breaks shards from easiest to harder.Source: Figure 6 · p. 11
- Figures 7–10 show validation BLEU curves for different ranking criteria or curriculum schedules at initial learning rates 0.0002 and 0.0008.Source: Figure 7 · p. 12, Figure 8 · p. 13, Figure 9 · p. 14
- Table 4 reports decoding performance at the 7th checkpoint with initial learning rate 0.0002, while Table 5 reports it with 0.0008.Source: Table 4 · p. 16, Table 5 · p. 16