Source-linked AI summary
On The Power of Curriculum Learning in Training Deep Networks
Guy Hacohen, Daphna Weinshall
TL;DR
The paper asks how neural-network training can benefit from non-uniformly sampling examples by difficulty and pace. It decomposes curriculum learning into scoring and pacing functions, evaluates transfer and bootstrapping strategies, and analyzes the resulting optimization landscape. The study reports faster learning and improved final CNN performance, while the theoretical analysis preserves the original global minimum under stated conditions.
Problem
Curriculum learning requires both ranking training examples by difficulty and constructing mini-batches whose difficulty increases over training.
Method
The paper separates scoring from pacing, using transfer learning or bootstrapping for scoring and evaluating fixed exponential, varied exponential, and single-step pacing functions.
Results
Transfer and bootstrapping speed learning and improve CNN generalization, while curriculum modifies the optimization landscape and preserves the global minimum under stated conditions.
Takeaways & Limitations
Curriculum learning can improve CNN training across scoring and pacing choices, and its theoretical effect can make the global optimum more pronounced without changing it.
Takeaways & Limitations
The theoretical preservation of the global optimum depends on assumption (5), which follows immediately when Var[U_ϑ] is constant across parameters.
Abstract
from arXiv · showhide
Training neural networks is traditionally done by providing a sequence of random mini-batches sampled uniformly from the entire training data. In this work, we analyze the effect of curriculum learning, which involves the non-uniform sampling of mini-batches, on the training of deep networks, and specifically CNNs trained for image recognition. To employ curriculum learning, the training algorithm must resolve 2 problems: (i) sort the training examples by difficulty; (ii) compute a series of mini-batches that exhibit an increasing level of difficulty. We address challenge (i) using two methods: transfer learning from some competitive ``teacher" network, and bootstrapping. In our empirical evaluation, both methods show similar benefits in terms of increased learning speed and improved final performance on test data. We address challenge (ii) by investigating different pacing functions to guide the sampling. The empirical investigation includes a variety of network architectures, using images from CIFAR-10, CIFAR-100 and subsets of ImageNet. We conclude with a novel theoretical analysis of curriculum learning, where we show how it effectively modifies the optimization landscape. We then define the concept of an ideal curriculum, and show that under mild conditions it does not change the corresponding global minimum of the optimization function.
1. Introduction
The paper frames curriculum learning as structured presentation of training examples, addressing how to rank example difficulty and control the pace of increasingly difficult mini-batches. It evaluates scoring and pacing strategies for CNNs and analyzes how curriculum learning changes optimization while preserving the global minimum under conditions.
- Motivation: Curriculum learning structures training by presenting easier examples before harder ones, but requires ordering examples by difficulty and choosing an appropriate presentation pace.These challenges parallel curriculum design for human learners, where both material order and pacing matter.
- Approach: The paper separates curriculum learning into a scoring function that ranks example difficulty and a pacing function that controls how data is presented.Scoring uses transfer learning or bootstrapping, while pacing may depend on the data and learner.
- Scoring methods: Transfer learning and bootstrapping provide alternative scoring methods that speed learning and improve CNN generalization without requiring manual difficulty labels or additional resources, respectively.Transfer uses networks trained on ImageNet; bootstrapping ranks examples with a classifier trained without curriculum.
- Pacing methods: Fixed exponential, varied exponential, and single-step pacing have comparable performance in the empirical setup.The protocols differ in how the amount of sampled data and the duration of each pacing stage change over training.
- Theory: The theoretical analysis shows that curriculum learning modifies the optimization landscape, making it steeper while maintaining the original problem’s global minimum.The analysis also provides a framework for understanding why different dynamic sampling heuristics can be beneficial.
- Related work: Curriculum learning has been studied as a way to order examples by difficulty during neural-network training, alongside related dynamic-sampling approaches.The paper distinguishes curriculum learning from methods whose example rankings depend on the model’s current hypothesis.
2. Curriculum Learning
Curriculum learning structures training by ranking examples with a scoring function and controlling how quickly progressively larger subsets are introduced through a pacing function. The paper compares curriculum, anti-curriculum, and random controls, and evaluates transfer-based scoring alongside several pacing designs.
- Curriculum components: A curriculum combines a scoring function that ranks examples by difficulty with a pacing function that controls the subset presented over training.The scoring function sorts examples in ascending difficulty, while the pacing function determines the sequence of subsets used to sample mini-batches.
- Curriculum components: The curriculum algorithm sorts the data by scoring function and generates a sequence of mini-batches from progressively selected subsets.The method returns mini-batches produced after sorting the training data and applying the pacing function at each iteration.
- Control conditions: The anti-curriculum presents harder examples before easier ones, whereas random curriculum randomizes the example ordering.These provide control conditions for isolating the effects of ascending-difficulty scoring.
- Scoring functions: Transfer scoring uses ImageNet-pretrained Inception features and a classifier confidence score to assign difficulty to each training image.The paper also considers self-taught scoring, and reports similar results with alternative confidence scores, classifiers, and teacher networks.
- Pacing functions: Pacing functions are restricted to monotonically increasing staircase functions whose hyper-parameters include step length, increase, and starting percent.This restriction ensures that the likelihood of easier examples can only decrease as training proceeds.
- Pacing functions: Fixed exponential, varied exponential, and single-step pacing differ in whether step lengths vary and how many pacing steps they use.Varied exponential pacing adds hyper-parameters but can remove the need to retune learning-rate parameters, while single-step pacing uses fewer hyper-parameters.
3. Empirical Evaluation
The evaluation tests curriculum learning across datasets, architectures, scoring methods, and pacing functions. Across these settings, curriculum generally accelerates learning and improves final accuracy, with transfer-based scoring especially robust.
- Experimental setup: The study evaluates curriculum learning across CIFAR-10, CIFAR-100, ImageNet subsets, and multiple network architectures, using tuned and cross-validated hyper-parameters.The test cases include moderate-size and VGG networks, while hyper-parameters are tuned separately to account for pacing-related changes in effective learning rate.
- Curriculum by transfer: Curriculum learning starts faster and reaches better final test performance than vanilla training on CIFAR-100, CIFAR-10, and a cats subset of ImageNet.The benefit is reported as larger on the more challenging CIFAR-100 task and remains significant, though smaller, with the VGG network.
- Curriculum by transfer: Transfer-based curriculum retains its advantage when learning rates are tuned separately, indicating that the improvement is attributed to effective transfer scoring rather than learning-rate settings.Random scoring performs similarly to vanilla, while curriculum and random conditions use very similar tuned learning-rate hyper-parameters.
- Curriculum by bootstrapping: Bootstrapping curriculum improves test accuracy throughout training, whereas self-paced scoring decreases accuracy and can significantly delay early learning.The self-paced decline is most prominent at the beginning, where the curriculum’s beneficial effects are observed.
- Analysis of scoring function: Transfer-scored examples produce more similar mean gradient directions and lower total gradient variance than some random subsets, helping explain their distinct optimization behavior.Random subsets approximate the exact empirical gradient direction, whereas transfer-scored examples point toward nearby local minima and exhibit lower variance.
- Summary of results: Different curriculum pacing functions improve learning accuracy during training and converge to similar performance, except self-paced scoring, which impairs learning.Transfer-based curriculum is described as easier to obtain and more robust than the other curriculum conditions.
4. Theoretical Analysis
The theoretical analysis models curriculum learning as a modification of the optimization landscape induced by a sampling prior. Under stated covariance conditions, the modified landscape preserves the original global optimum while making it more pronounced.
- Curriculum as a sampling prior: Curriculum scoring and pacing induce a Bayesian prior over training examples, with pacing controlling how the sampled subset grows.The induced probability is non-increasing with example difficulty, while the pacing function increases the number of included points.
- Landscape modification: The change in expected utility caused by the curriculum prior equals the covariance between example utility and the induced prior.This covariance determines how the prior reshapes the utility function over hyper-parameters.
- Landscape modification: If the curriculum prior is positively associated with the optimal utility, it can make the optimization landscape steeper toward the optimal parameters.The analysis relates this effect to the correlation between the induced prior and utility across examples.
- Global optimum: Under the proposition’s curriculum condition, the modified optimization landscape has the same global optimum as the original problem and a more pronounced maximum there.Thus curriculum learning changes the optimization function without necessarily changing its global solution.
- Ideal curriculum: The ideal curriculum is the prior corresponding to an optimal hypothesis, and its guarantee holds when utility variance is roughly constant across plausible parameters.The constant-variance condition is given as a sufficient condition for the proposition to apply.
- Implications: The analysis suggests both self-paced and hard-data-mining strategies can be effective when their curriculum priors are positively correlated with optimal utility.The paper also suggests a curriculum directly correlated with optimal utility could outperform both strategies.
A. Additional Empirical Results
Additional experiments test curriculum learning across CIFAR-100 super-classes, transfer-scoring choices, classifiers, and pacing variants. The results indicate that the observed benefits are qualitatively robust, while pacing can also regulate learning-rate effects.
- Transfer scoring: Transfer scoring produces similar qualitative results with Inception, VGG-16, and ResNet teachers and with alternative classifiers.The reported comparisons include RBF SVM, linear SVM, and other classifier conditions.
- Pacing functions: Varied exponential pacing changes step lengths, while tuning only the first two avoids the otherwise impractical number of additional hyper-parameters.The paper motivates this restriction because most curriculum power lies in the first few steps.
- Pacing functions: Pacing can control the number of epochs spent at each dataset size and mitigate pacing-induced learning-rate effects when tuned correctly.This allows curriculum learning without separately tuning the learning rate.
- Empirical results: With the vanilla learning rate fixed, curriculum training improves accuracy throughout learning, and tuning curriculum parameters can nearly reproduce fixed exponential pacing.The latter result suggests the varied pacing function can largely nullify the indirect learning-rate manipulation.
B. Extended Discussion
Repeated self-taught bootstrapping did not provide additional observed benefit and could impair performance after many repetitions. Robustness analyses compared pacing and scoring choices using final accuracy and AUC-based selection.
- Bootstrapping: Repeated self-taught bootstrapping produced no observed benefit and impaired performance after a large number of repetitions.The procedure could be repeated in principle, but accumulated scoring errors were identified as a concern.
Fair comparison in parameter tuning
The experiments tuned learning-rate and pacing parameters while using comparable search ranges across curriculum, anti-curriculum, random, and vanilla conditions. Vanilla received a broader, finer learning-rate search because it had no pacing parameters.
- Fair comparison in parameter tuning: Curriculum, anti-curriculum, and random conditions used identical ranges for pacing and learning-rate hyperparameters during coarse grid search.The vanilla condition had no pacing hyperparameters, so its learning-rate range was expanded and refined.
- Fair comparison in parameter tuning: For the public competitive network, the published learning-rate schedule was retained while pacing hyperparameters were grid-searched.The vanilla condition was repeated as many times as the total grid-search experiments to support a fair comparison.
Learning Rate Tuning
The study tested cyclic learning-rate scheduling to determine whether curriculum results depended on the commonly used learning-rate scheduling method. The comparison covered vanilla and curriculum conditions in cases 2 and 3.
- Learning Rate Tuning: Cyclic learning rates were tested in cases 2 and 3 as a robustness check against artifacts from learning-rate scheduling.The schedule periodically increases and decreases the learning rate, following Smith (2017).
C. Methodology, additional details
The methodology uses exponentially increasing data exposure, a moderate convolutional architecture, specified grid-search ranges, and a seven-class ImageNet cat subset. Robustness was also evaluated with AUC-based selection and cyclic scheduling.
- Exponential Pacing: Pacing functions increase the data size exponentially at each step, matching the customary exponential change in learning rate.This defines the pacing schedule used throughout the work.
- Architecture Details: The moderate-size network is a convolutional neural network with eight convolutional layers using 32 to 256 filters, pooling, dropout, and a 512-unit fully connected layer.The batch size was 100.
- Learning-rate robustness: With cyclic learning-rate scheduling, case 3 was evaluated under conditions similar to test case 3.The comparison is presented in Figure 13.
- Grid-search hyper-parameters: For case 6, the grid-search parameters include initial learning rate, exponential decrease, learning-rate step size, pacing step size, increase, and starting percentage.The supplied ranges include an initial learning rate of 0.2 ∼ 0.01 and starting percentage of 2% ∼ 15%.
- ImageNet Dataset Details: Case 6 uses a seven-class ImageNet subset containing cat hyponyms, with images resized to 56 × 56 and each channel normalized to mean 0 and standard deviation 1.Each class contains 1300 training images and 50 test images.
- Robustness Of Results: AUC-based selection retains significantly higher performance for curriculum conditions than controls, with fixed exponential pacing and Inception scoring significantly above other curriculum methods.AUC is described as more robust because it uses the full learning curve rather than only a few final points.