Source-linked AI summary
Automated Curriculum Learning for Neural Networks
Alex Graves, Marc G. Bellemare, Jacob Menick, Remi Munos, Koray Kavukcuoglu
TL;DR
Curriculum learning needs an effective task progression, but fixed orderings and hand-chosen advancement rules can be unsuitable when task difficulty is unknown or multidimensional. The paper learns a stochastic syllabus by feeding per-example learning-progress rewards to a nonstationary multi-armed bandit. On LSTM networks across three curricula, this can significantly accelerate learning; prediction gain was most consistent for maximum likelihood, while gradient variational complexity gain performed best for variational inference.
Problem
Fixed curriculum orderings and hand-chosen advancement thresholds are difficult to specify when tasks vary along multiple difficulty axes or lack a predefined order.
Method
A nonstationary multi-armed bandit uses per-example learning-progress signals to adapt a stochastic policy over curriculum tasks.
Results
Across three curricula, stochastic syllabus selection produced significant curriculum-efficiency gains; prediction gain was most consistent for maximum likelihood, while gradient variational complexity gain performed best for variational inference.
Takeaways & Limitations
Instantaneous progress signals can support efficient curriculum selection when external evaluation is difficult, favoring local assessment of learning progress.
Takeaways & Limitations
Gradient prediction gain is biased toward tasks whose loss has higher variance, while two-sample signals have higher variance than prediction gain.
Abstract
from arXiv · showhide
We introduce a method for automatically selecting the path, or syllabus, that a neural network follows through a curriculum so as to maximise learning efficiency. A measure of the amount that the network learns from each data sample is provided as a reward signal to a nonstationary multi-armed bandit algorithm, which then determines a stochastic syllabus. We consider a range of signals derived from two distinct indicators of learning progress: rate of increase in prediction accuracy, and rate of increase in network complexity. Experimental results for LSTM networks on three curricula demonstrate that our approach can significantly accelerate learning, in some cases halving the time required to attain a satisfactory performance level.
1. Introduction
Curriculum learning can accelerate training, but manually ordering tasks and tuning advancement rules is difficult when task difficulty is multidimensional or unknown. The paper instead adapts a stochastic task-selection policy using learning-progress rewards and a nonstationary bandit.
- 1. Introduction: Curriculum effectiveness depends strongly on how progression through tasks is scheduled.Hand-chosen thresholds and probabilities introduce hard-to-tune parameters.
- 1. Introduction: Fixed advancement rules assume tasks have a known difficulty ordering, which may fail when tasks vary along multiple difficulty axes or lack a predefined order.
- 1. Introduction: The proposed approach treats the next-task decision as a stochastic policy continuously adapted to maximize learning progress.The paper considers prediction gain and introduces complexity gain as progress signals.
- 1. Introduction: The approach connects adaptive curriculum selection with intrinsic motivation, active learning, and adaptive rather than precomputed training orderings.
- 1. Introduction: A nonstationary multi-armed bandit uses per-example progress rewards to learn a stochastic policy over curriculum tasks.It is nonstationary because the network’s behavior and the optimal task policy evolve during training.
2. Background
The paper formalizes curricula as collections of task distributions and syllabi as time-varying task-selection distributions. It frames syllabus selection as a nonstationary adversarial bandit problem and adapts rewards to changing scales.
- 2. Background: The framework models each batch as an example and each task as a distribution over input-target sequences.
- 2. Background: A curriculum is an ensemble of task distributions, while a syllabus is a time-varying sequence of distributions over those tasks.
- 2. Background: The multiple-tasks setting minimizes performance loss across all task distributions, whereas the target-task setting minimizes loss only on the final task.
- 2. Background: An N-task curriculum becomes an N-armed bandit whose adaptive policy selects tasks and receives payoff only from the selected arm.
- 2.2. Adversarial Multi-Armed Bandits: Exp3.S is used because task rewards change as the network learns, making a piecewise-stationary strategy more appropriate than one fixed best arm.
- 2. Background: Rewards are adaptively rescaled to [−1, 1] because their magnitude depends on the progress measure and changes during training.Approximate quantiles are computed from a reservoir sample rather than the entire reward history.
3. Learning Progress Signals
The paper defines learning-progress rewards from changes in predictive performance or model complexity, then uses them to guide curriculum selection. These signals differ in computational cost, bias, and variance, motivating comparisons among several alternatives.
- Learning Progress Signals: Training samples receive rewards based on learning progress, computed as a progress measure divided by processing time and supplied to the curriculum-selection algorithm.The resulting reward is rescaled to [−1, 1] before updating Exp3.S.
- Loss-driven Progress: The loss-driven signals compare model predictions before and after training on a sample, with some instantaneous measures requiring only x and others requiring an additional x′.Prediction gain uses an additional forward pass for direct comparison, while gradient prediction gain uses a first-order approximation.
- Loss-driven Progress: Self prediction gain estimates progress on a new sample from the same task, addressing the bias caused by evaluating the training sample itself.Mean prediction gain extends target prediction gain to multiple tasks by averaging progress across tasks, but incurs additional sampling variance.
- Complexity-driven Progress: Complexity-driven signals measure how quickly network complexity increases, drawing on the MDL principle that useful complexity should compress the data more than it costs.Variational complexity gain measures complexity change after an update, while gradient variational complexity gain uses a directional derivative of the KL term.
- Complexity-driven Progress: The gradient variational complexity gain is a directional derivative of the KL divergence along the gradient-descent direction, with a linear approximation considered more reliable than for prediction gain.The stated rationale is that model complexity has less curvature than the loss surface.
- Prediction Gain Bias: Prediction gain contains true expected learning progress plus gradient variance, whereas self prediction gain is an unbiased estimate of expected learning progress.Gradient prediction gain worsens the variance-related problem because it relies on a Taylor approximation.
4. Experiments
Across n-gram, repeat-copy, and bAbI curricula, adaptive gain-based syllabuses often accelerated learning by selecting tasks according to measured progress rather than uniform sampling. The experiments also show that effective policies can exploit structure across multiple difficulty dimensions and discover useful task orderings.
- Experimental setup: Three curricula—n-gram language modelling, repeat copy, and bAbI—were evaluated with stacked unidirectional LSTM networks using multiple learning-progress gains.The gains included loss-driven and complexity-driven signals, with experiments using maximum-likelihood and variational-inference training where specified.
- N-gram language modelling: Complexity-based gains rapidly traversed the n-gram curriculum before strongly favouring the 10-gram task, while loss-based gains moved toward higher n more slowly and initially favoured simpler tasks.GVCG later revisited the 0-gram task, whereas VCG followed a similar path with less confidence.
- Repeat copy: GVCG solved the repeat-copy target about twice as fast as uniform sampling for variational-inference training, while PG, SPG, and TPG were somewhat faster than uniform for maximum-likelihood training.Direct training on the target task failed to learn, and GVCG produced strongly non-uniform policies with higher network complexity than uniform sampling.
- Repeat copy: GVCG first focused on short sequences with high repeat counts, then long sequences with low repeat counts, reducing loss across many tasks without visiting all 169 tasks.This strategy decoupled the repeat-copy curriculum’s two memory-related difficulty dimensions.
- bAbI: On bAbI, prediction gain clearly outperformed uniform sampling in learning speed and completed-task count, while self-prediction gain showed the same benefits less strongly.GVCG was initially faster than uniform for variational-inference training but slightly worse later, and variational inference generally hampered progress.
- bAbI: PG and GVCG accelerated bAbI by selectively focusing on tasks, including early concentration on Time Reasoning and sustained attention to the harder Path Finding task.The syllabuses also progressed from Single Supporting Fact to Three Supporting Facts, indicating discovery of implicit orderings in an unsorted curriculum.
5. Conclusion
The experiments indicate that adaptively choosing a stochastic syllabus to maximize learning progress can substantially improve curriculum efficiency when paired with a suitable progress signal. Prediction gain was most consistent for maximum likelihood training, while gradient variational complexity gain performed best for variational inference training.
- Suitable progress signals enabled stochastic syllabuses to produce significant gains in curriculum-learning efficiency.
- Uniform sampling from all tasks was a surprisingly strong benchmark, possibly because fastest-progress tasks dominate learning gradients.
- Prediction gain was the most consistent signal for maximum likelihood training.
- Gradient variational complexity gain performed best for variational inference training.
- Instantaneous progress signals can be evaluated from training samples alone, supporting applications where external evaluation is difficult.