Source-linked AI summary
Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks
Daphna Weinshall, Gad Cohen, Dan Amir
TL;DR
The paper asks how curriculum learning can accelerate optimization when ideal example rankings are difficult to obtain. It proves convergence results for convex linear regression and uses transfer learning from a pretrained network to rank examples for CNN training. Curriculum learning speeds convergence early and can improve generalization on harder tasks, while the empirical ranking only approximates the ideal curriculum.
Problem
Reliable difficulty rankings are hard to obtain, and human judgments may not match the difficulty experienced by machine learners.
Method
The paper proves curriculum-learning properties for stochastic gradient descent on convex linear regression and ranks CNN training examples using confidence from a classifier trained on another task.
Results
Curriculum learning consistently increases early convergence speed, and improves generalization when tasks are more difficult.
Takeaways & Limitations
Transferred example rankings can provide a practical curriculum when human difficulty scores are unavailable or unreliable.
Takeaways & Limitations
The empirical ranking is only an estimate derived from another classifier rather than the ideal difficulty definition used in the theory.
Abstract
from arXiv · showhide
We provide theoretical investigation of curriculum learning in the context of stochastic gradient descent when optimizing the convex linear regression loss. We prove that the rate of convergence of an ideal curriculum learning method is monotonically increasing with the difficulty of the examples. Moreover, among all equally difficult points, convergence is faster when using points which incur higher loss with respect to the current hypothesis. We then analyze curriculum learning in the context of training a CNN. We describe a method which infers the curriculum by way of transfer learning from another network, pre-trained on a different task. While this approach can only approximate the ideal curriculum, we observe empirically similar behavior to the one predicted by the theory, namely, a significant boost in convergence speed at the beginning of training. When the task is made more difficult, improvement in generalization performance is also observed. Finally, curriculum learning exhibits robustness against unfavorable conditions such as excessive regularization.
1. Introduction
Curriculum learning orders training examples by estimated difficulty, reflecting how human teaching structures learning. This paper studies its theoretical basis and proposes transfer-based ranking for cases where reliable human difficulty scores are unavailable.
- Curriculum learning presents higher-ranked or easier examples earlier, using difficulty-based ordering to guide training.The ranking may be supplied by a teacher and can also be interpreted as weighting examples.
- The paper theoretically analyzes curriculum learning for stochastic gradient descent optimizing convex linear regression loss.It defines ideal difficulty using each training point’s loss relative to the optimal classifier.
- Ideal difficulty rankings are difficult to obtain and may not match what makes examples difficult for machine learners rather than humans.This mismatch is documented in visual object recognition and limits practical use of human-provided curricula.
- The paper ranks examples without a human teacher by transferring difficulty estimates from another classifier trained on a related task.This transfers knowledge between teacher and student classifiers rather than transferring instance representations.
- Curriculum learning consistently accelerates early learning, while final generalization sometimes improves when tasks are difficult, networks are small, or regularization is strong.The method constructs a presentation schedule from the transferred ranking and compares multiple scheduling controls.
2. Theoretical analysis
The theoretical analysis defines stochastic curriculum learning for SGD and shows that easier examples yield faster expected convergence, especially early in training. It further characterizes how current loss, gradient geometry, and assumptions about the data affect this result, with deep-learning evidence agreeing qualitatively with the theory.
- Definitions and setup: Stochastic curriculum learning biases SGD toward easier examples early, then gradually reduces this bias until sampling matches vanilla SGD.The method requires both a difficulty score for training points and a scheduling procedure.
- Definitions and setup: The analysis studies how difficulty affects convergence toward the global minimum of expected least-squares loss for linear hypotheses.The regression model uses hypotheses h(x) = a^t x + b with y ∈ R.
- Convergence rate decreases with difficulty: Theorem 1 shows that expected convergence rate decreases monotonically with a point’s difficulty score Ψ when the step size is sufficiently small.The result concerns the gradient step used by SGD and assumes a specified conditional data distribution.
- Convergence rate increases with current loss: For fixed difficulty, expected convergence increases with the sampled point’s current loss, although without fixing difficulty there are hypotheses where convergence decreases with current loss.Thus the current-loss relationship depends on whether difficulty is held constant.
- Deep learning: simulation results: The deep-learning test uses an empirical ranking rather than the ideal difficulty definition, so it evaluates whether theory transfers under an estimated curriculum.The ranking is derived from another classifier rather than directly from the optimal hypothesis.
- Deep learning: simulation results: The empirical gradient-direction error is largest for difficult examples at epoch 0, shrinks substantially by epoch 10, and becomes insignificant by epoch 20.The comparison uses the easiest and most difficult examples and supports the theoretical corollaries in the deep-learning setup.
3. Curriculum learning in deep networks
The paper develops a CNN curriculum that ranks examples through transfer from a more powerful pretrained classifier, then schedules training from easier to harder examples. Experiments find faster early learning, stronger final generalization on harder tasks, and reduced sensitivity to excessive regularization.
- Ranking examples by knowledge transfer: The method estimates example difficulty using a confidence measure from a classifier trained on a pretrained network’s penultimate-layer representation, then sorts training data accordingly.Unlike conventional transfer, it transfers example rankings rather than representations for use at test time.
- Scheduling mechanisms: Curriculum scheduling prioritizes easier examples initially and gradually increases the sampling weight of more difficult examples.The study evaluates fixed schedules, adaptive schedules based on current training loss, random-order controls, and anti-curriculum ordering.
- Empirical evaluation: Curriculum learning consistently accelerates early training, while reduced final generalization error appears primarily when the task is relatively difficult.The study varies task difficulty through class separability, network size, and regularization.
- Empirical evaluation: Using a Small CNN on CIFAR100 makes the task harder and enables curriculum learning to improve final generalization performance, unlike the Large-network setting.With the Large network, curriculum learning speeds initial training but converges to the same performance as regular training; similar results appear on STL-10.
- Controlling for task difficulty: Across three CIFAR100 five-class tasks, the improvement in final accuracy is larger for more difficult tasks, and curriculum learning is least affected by excessive regularization.Too much regularization harms performance overall, but curriculum learning shows greater robustness to that degradation.
4. Summary and Discussion
The paper combines theory and experiments to show that curriculum learning can accelerate early training, with additional generalization benefits on more difficult tasks. It also identifies conditions under which example selection improves convergence and suggests a mechanism for non-convex optimization.
- In non-convex optimization, oversampling coherent easier examples may increase the likelihood of escaping low-quality local minima.The proposed mechanism is linked to empirically higher variance in gradient direction for more difficult points.
- When training-point difficulty is fixed, convergence is faster for points with higher loss under the current hypothesis.The paper also notes that fixing the difficulty score is necessary for this result.
- Curriculum learning accelerates early training across theoretical linear-regression analysis and deep-network experiments.The deep-network experiments found increased convergence rates at the beginning of training under varied task and control conditions.
- With more difficult tasks, curriculum learning improves generalization performance.