Source-linked AI summary
Curriculum Dropout
Pietro Morerio, Jacopo Cavazza, Riccardo Volpi, Rene Vidal, Vittorio Murino
TL;DR
Fixed dropout probability is presented as suboptimal because training dynamics change over time. Curriculum Dropout schedules neuron retention to increase suppression as training proceeds, and experiments across seven image-classification tasks report better performance than standard and anti-Curriculum dropout.
Problem
A fixed dropout parameter cannot adapt to increasing levels of negative co-adaptation during training.
Source: §3 · p. 3Method
Curriculum Dropout varies the neuron-retention probability over training, gradually reducing the expected number of active units while respecting a lower bound.
Source: §3 · p. 3Results
Takeaways & Limitations
The schedule provides adaptive regularization, a smoother initialization for weight optimization, and a mathematically sound curriculum interpretation.
Source: §6 · p. 8Evaluation covered image-classification tasks only; applicability to other domains remains a conjecture and future-work direction.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Dropout is a very effective way of regularizing neural networks. Stochastically "dropping out" units with a certain probability discourages over-specific co-adaptations of feature detectors, preventing overfitting and improving network generalization. Besides, Dropout can be interpreted as an approximate model aggregation technique, where an exponential number of smaller networks are averaged in order to get a more powerful ensemble. In this paper, we show that using a fixed dropout probability during training is a suboptimal choice. We thus propose a time scheduling for the probability of retaining neurons in the network. This induces an adaptive regularization scheme that smoothly increases the difficulty of the optimization problem. This idea of "starting easy" and adaptively increasing the difficulty of the learning problem has its roots in curriculum learning and allows one to train better models. Indeed, we prove that our optimization strategy implements a very general curriculum scheme, by gradually adding noise to both the input and intermediate feature representations within the network architecture. Experiments on seven image classification datasets and different network architectures show that our method, named Curriculum Dropout, frequently yields to better generalization and, at worst, performs just as well as the standard Dropout method.
1. Introduction
Deep neural networks remain vulnerable to overfitting, and fixed dropout may impose too much difficulty from the start. The paper proposes Curriculum Dropout, which increases suppression over training and connects dropout to regularization theory and curriculum learning; experiments report generally superior classification performance.
1. Introduction
- Overfitting remains a concern because network weights can specialize to training examples and generalize poorly.Dropout addresses excessive co-adaptation by randomly suppressing neurons during training.Source: §1 · p. 1
- The method dynamically increases the expected number of suppressed units as training progresses, adapting dropout to the training timeline.The authors’ motivating intuition is that excessive feature-detector co-adaptation is unlikely in early training.Source: §1 · p. 2
- The paper connects dropout with regularization theory and curriculum learning to justify its training approach.It presents these connections as an improved justification for (Curriculum) Dropout training.Source: §1 · p. 2
- Curriculum Dropout generally outperforms original Dropout [13] and anti-Curriculum in image classification.The evaluation compares these approaches on standard datasets using neural-network architectures.Source: §1 · p. 2
2. Related Work
Prior work proposed many dropout variations, but their relative superiority and theoretical justification remained unclear. This paper distinguishes its approach by offering justifications for deep neural networks alongside experimental evaluation.
2. Related Work
- Drop-Connect suppresses connections, data-dependent and Evolutional dropout vary unit probabilities, and annealed dropout decreases suppression during training [20].Other cited variations accelerate dropout, apply it to recurrent or convolutional networks, or combine it with a binary belief network [1].Source: §2 · p. 2
- The relative superiority of dropout variations remained unclear, with comparisons also complicated by unavailable code [20].Some methods offered empirical results without a theoretical justification, while formal analyses relied on approximations biased toward shallow models [28].Source: §2 · pp. 2–3
- Unlike prior approaches characterized as empirical-only or approximation-based, this paper provides natural justifications for dropout generalization in deep neural networks.The authors contrast their approach with methods that omit justification or analyze shallow models.Source: §2 · p. 3
3. A Time Scheduling for the Dropout Rate
Curriculum Dropout replaces a fixed retain probability with a schedule that starts without suppression and gradually introduces dropout as training proceeds. The paper motivates this timing by arguing that overfitting from excessive co-adaptation emerges later, and interprets the schedule as both adaptive regularization and a smoother start to dropout training.
3. A Time Scheduling for the Dropout Rate
- Overfitting from excessive co-adaptations is argued to arise later in training, motivating delayed dropout rather than a fixed retain probability throughout.The authors connect this timing to statistically independent random initial weights and growing risk of over-specialization as optimization proceeds.Source: §3 · p. 3
- A time-varying retain probability θ(t) starts at 1 and decreases toward θ, with θ(t) ≥ θ; D·θ(t) represents the average active units in a layer.The authors prescribe θ between 0.5 and 0.9, with lower retain probability for higher layers.Source: §3 · pp. 3–4
- The selected exponential schedule avoids polynomial schedules’ manual threshold and extra parameters, while reaching θ faster than the compared exponential curves.The paper compares the functions in Figure 2; polynomial schedules require choosing degree and threshold, whereas the selected schedule is Eq. (1).Source: Figure 2 · p. 3, §3 · p. 4
- The rule γ = 10/T makes θcurriculum(T) differ from θ by less than 10^-4 and is used for all experiments.T is the total number of gradient updates; the heuristic is intended for a known order of magnitude of T.Source: §3 · p. 4
- The schedule is also interpreted as a soft transition from dropout-free training to dropout training, providing a smarter initialization; unlike a step switch, it avoids an objective discontinuity that can damage performance.The paper connects smoothly increasing training difficulty to Curriculum Learning and states that its selected function adds no parameter to be tuned.Source: §3 · pp. 4–5
- As θ(t) decreases, the regularizer’s effect increases over training, whereas it is zero initially; the authors present this as time-adaptive regularization for deep networks.The paper says the initial zero regularization is appropriate because weights remain close to their random initialization and overfitting is unlikely early.Source: §3 · pp. 4–5
4. Curriculum Learning, Curriculum Dropout
Curriculum Dropout recasts scheduled neuron retention as a curriculum over training examples: the examples become progressively harder as more input information is suppressed. Unlike fixed-probability Dropout, this schedule increases difficulty smoothly over training.
Curriculum Learning, Curriculum Dropout
- Curriculum learning orders training by difficulty, requiring examples sampled later from Qλ+ε to be harder than those sampled from Qλ.The paper formalizes example difficulty through weights Wλ(z), which increase with λ and reach 1 at the end of training.Source: §4 · p. 5
- Scheduled Dropout realizes this curriculum by treating a dropped-out input as a clean example masked with Bernoulli noise, with the mask probability set by θ(t).The analysis considers input-layer Dropout; the authors say the same reasoning applies to intermediate layers because each layer supplies features to the next.Source: §4 · p. 5
- The authors show that their choice of Qλ satisfies the curriculum-learning definition, interpreting progressively corrupted inputs as increasingly difficult examples.At λ = 0, no entries are suppressed; increasing λ reduces θ and removes more information from the training examples.Source: §4 · pp. 5–6
- The retain probability θ(t) starts high and decreases over gradient updates, progressively suppressing more units and increasing training difficulty.At the start, θ(0) = 1 leaves all entries intact; as θ decreases, more entries are set to zero on average.Source: §4 · p. 6
- Fixed-probability Dropout uses a fixed training-example difficulty, risking examples that are too simple or too difficult, whereas Curriculum Dropout increases difficulty smoothly.The paper also reports that anti-Curriculum performs slightly better or worse than no-curriculum training, but worse than curriculum implementations.Source: §4 · p. 6
5. Experiments
Across the tested image-classification settings, Curriculum Dropout improves generalization over standard Dropout in almost all cases, with parity on MNIST using an MLP. Experiments also compare anti-Curriculum and abrupt switch schedules, and report drawbacks for both.
5. Experiments
- Experiments use LeNet, a deeper CNN, and MLPs, and omit data augmentation, special preprocessing, and deeper architectures to focus on comparison with standard Dropout.The deeper CNN is called CNN-2; LeNet is called CNN-1.Source: §5 · p. 6
- The Figure 3 accuracy curves report means over 10 retrainings with standard-deviation shadows; Table 1 compares percentage accuracy improvement against a no-suppression baseline.The reported Table 1 metric averages the 10 highest mean accuracies per paradigm over 10 runs.Source: Figure 3 · p. 7, §5 · pp. 7–8, Table 1 · p. 8
- On Double MNIST, abrupt Switch-Curriculum changes produce accuracy spikes and can lose learned concepts; late switches are deleterious.The authors report no heuristic for choosing the switch epoch, making the approach less powerful than a smoothly scheduled curriculum.Source: §5 · p. 8
- Curriculum Dropout improves generalization over standard Dropout in almost all cases, matching it on MNIST with an MLP.The authors conjecture that curriculum learning may be less effective on simpler tasks; they report no additional cost or training-time requirement.Source: §5 · p. 8
- Anti-Curriculum is improved by a more significant gap by the proposed schedule, and sometimes performs worse than an unregularized network.Anti-Curriculum often requires early stopping, unlike regular and Curriculum Dropout.Source: §5 · p. 8
6. Conclusions and Future Work
Curriculum Dropout gradually increases suppressed units to provide adaptive regularization and smoother initialization for weight optimization. Across image-classification tasks, it outperformed standard Dropout and Annealed, though testing covered image classification only.
6. Conclusions and Future Work
- Gradually increasing suppressed units provides adaptive regularization, smoother weight-optimization initialization, and a mathematically sound curriculum.Source: §6 · p. 8
- Across 7 image classification tasks and various architectures, Curriculum Dropout outperformed original Dropout and Annealed.The authors report that it always outperformed original Dropout across the tested architectures.Source: §6 · p. 8
- Testing covered image-classification tasks only; applications to other domains and tasks remain future work.Source: §6 · p. 8