Source-linked AI summary
Improving neural networks by preventing co-adaptation of feature detectors
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov
TL;DR
Small training sets can make feedforward neural networks overfit, limiting held-out performance. The paper introduces random dropout to prevent co-adaptation, improving results across speech, object, and text benchmarks, including record recognition errors.
Problem
Limited labeled training data allows neural networks to fit training cases while performing worse on held-out test data.
Method
Dropout randomly omits each hidden unit during training, preventing co-adaptation while efficiently averaging predictions across many networks.
Results
Dropout improves generalization across speech, object, and text benchmarks, reaching a record 42.4% error on one object-recognition dataset.
Takeaways & Limitations
Random dropout is a broadly effective regularization method for neural networks across multiple benchmark tasks.
Takeaways & Limitations
Architecture choices were selected using validation performance with dropout before evaluating dropout on the real test sets.
Abstract
from arXiv · showhide
When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.
6 King’s College Rd, Toronto, Ontario M5S 3G4, Canada
Dropout reduces overfitting by randomly omitting hidden units, preventing co-adaptation while efficiently averaging many shared-weight networks. It improves generalization across benchmark tasks, including MNIST, speech, object recognition, and Reuters classification.
- Method: 50% dropout randomly omits each hidden unit during every training case, preventing units from relying on specific co-adaptations.The procedure can also be viewed as efficient model averaging over many neural networks.
- Method: Dropout efficiently trains a huge number of different networks because each presentation creates another network while active units share weights.This avoids the computational expense of separately training and testing many networks.
- Inference: The mean network uses all hidden units with halved outgoing weights and performs similarly to averaging many dropout networks.For classification, it is exactly equivalent to the geometric mean of the probability distributions over all 2^N possible networks.
- Inference: Dropout improves ensemble predictions: the mean network has higher correct-answer log probability than the average individual network, and lower regression squared error.These guarantees assume dropout networks do not all make identical predictions for classification.
- Benchmark results: 92 errors were obtained by fine-tuning a pretrained deep belief net with 50% dropout, versus 118 errors with standard back-propagation.A deep Boltzmann machine achieved a mean of 79 errors with dropout, a record without prior knowledge or enhanced training sets.
- Benchmark results: 31.05% error on Reuters fell to 29.62% with dropout, while 48.6% CIFAR-10 error matched comparable performance using a single neural network.For speech and object-recognition evaluations, architectures were selected using a separate validation set before testing dropout.
A Experiments on MNIST · A.1 Details for dropout training
The MNIST experiments evaluated dropout across four neural-network architectures for digit classification. Training used specified dropout and learning-rate settings, with learning-rate decay and fixed-length input constraints improving optimization and controlling weight growth.
- A Experiments on MNIST: MNIST contains 60,000 training and 10,000 testing images of handwritten digits classified into their correct digit classes.
- A Experiments on MNIST: The experiments evaluated dropout sensitivity across four architectures: 784-800-800-10, 784-1200-1200-10, 784-2000-2000-10, and 784-1200-1200-1200-10.
- A.1 Details for dropout training: The training configuration used ϵ0 = 10.0, f = 0.998, pi = 0.5, pf = 0.99, and T = 500.
- A.1 Details for dropout training: A constant learning rate improved performance over standard backpropagation, while starting high and decaying the rate provided a significant additional boost.
- A.1 Details for dropout training: Constraining input vectors to a fixed length prevents weights from increasing arbitrarily in magnitude regardless of the learning rate.
- A.1 Details for dropout training: Fixed-length input constraints give the network substantial opportunity to search for a good configuration in weight space.
A.2 Details for dropout finetuning
Dropout can finetune pretrained neural networks, and a small learning rate substantially improves performance over standard backpropagation finetuning. Experiments applied dropout finetuning to pretrained Deep Belief Networks and Deep Boltzmann Machines.
- A.2 Details for dropout finetuning: Dropout finetuning with a small learning rate performed much better than standard backpropagation finetuning on pretrained models.The authors report this as a general finding for finetuning pretrained networks.
- A.2 Details for dropout finetuning: For a pretrained Deep Belief Network with 784-500-500-2000 architecture, dropout used 50% hidden-unit and 20% visible-unit rates with a constant learning rate of 1.0.The model was pretrained using greedy layer-wise Contrastive Divergence learning, with no constraint on incoming weight-vector length.
- A.2 Details for dropout finetuning: 79 errors with dropout versus 94 errors with usual finetuning were obtained for a pretrained Deep Boltzmann Machine.The DBM used a 1784-500-1000-10 architecture, and its finetuning hyperparameters matched those used for the Deep Belief Network.
A.3 Effect on features
Dropout is examined as a mechanism that encourages hidden units to learn useful features independently rather than relying on specific other units. The analysis studies first-level features learned by a 784-500-500 neural network without generative pre-training.
- A.3 Effect on features: Dropout encourages each hidden unit to learn a useful feature without relying on specific other hidden units to correct its mistakes.This is presented as one reason dropout gives major improvements over backpropagation.
- A.3 Effect on features: The feature-learning effect is investigated to better understand why dropout produces major improvements over backpropagation.
- A.3 Effect on features: The analysis examines first-level features learned by a 784-500-500 neural network without generative pre-training.The learned features are shown in Figure 5, whose panels each display 100 random features.
B Experiments on TIMIT
The experiments evaluate speech-recognition networks on TIMIT using standardized log filter-bank inputs and varied architectures. Across tested configurations, dropout consistently produced lower validation error.
- Dataset: TIMIT contains recordings from 630 speakers across 8 American English dialects, with each speaker reading 10 phonetically rich sentences.The corpus includes word- and phone-level transcriptions for converting speech signals into phone-transcription sequences.
- Preprocessing: Inputs were normalized log filter-bank responses extracted from 25 ms speech windows with 10 ms strides.Kaldi was used to preprocess the data for exact reproducibility, and minibatches contained 100 examples.
- Architectures: The study varied input frames of 15 or 31, network depths of 3, 4, or 5 layers, and hidden-layer widths of 2000 or 4000 units.These architecture combinations were evaluated using validation-error curves.
- Results: Dropout consistently led to lower validation error across the tested architecture combinations.The passage attributes this result to the validation-error curves shown for multiple configurations.
B.1 Pretraining
The TIMIT networks were pretrained layerwise with a Deep Belief Network, using a Gaussian RBM for real-valued inputs and binary RBMs for subsequent layers. Training used Contrastive Divergence with momentum and specified initialization, learning-rate, and epoch schedules.
- Pretraining approach: All TIMIT experiments used Deep Belief Network pretraining, with the first layer implemented as a Gaussian RBM for real-valued inputs.The network was pretrained before subsequent training.
- Gaussian RBM configuration: The Gaussian RBM used zero visible biases, weights sampled from a zero-mean normal distribution with standard deviation 0.01, and fixed visible-unit variance 1.0.Learning minimized Contrastive Divergence with momentum increasing from 0.5 to 0.9 over 20 epochs.
- Optimization: The Gaussian RBM used a learning rate of 0.001 on the average gradient, subsequently multiplied by 1-momentum.Momentum was used to speed learning, and an L2 weight decay term was specified.
- Layerwise RBM training: Subsequent layers were trained as binary RBMs for 50 epochs each using a learning rate of 0.01.Visible biases were initialized as log(p/(1 −p)), where p was the dataset mean activation.
B.2 Dropout Finetuning
Dropout finetuning initialized the network from pretrained RBMs and used dropout backpropagation with a gradually increased momentum. Compared with standard backpropagation, dropout consistently reduced error and cross-entropy, controlled overfitting, and supported larger networks without early stopping.
- Dropout finetuning: Pretrained RBMs initialized the neural network before finetuning with dropout backpropagation.Momentum increased linearly from 0.5 to 0.9 over 10 epochs, with a learning rate of 1.0.
- Dropout finetuning: About 200 epochs were required for the model to converge.
- Dropout finetuning: Dropout consistently achieved lower frame classification error and cross-entropy than standard backpropagation across network architectures and input representations.It significantly controlled overfitting, enabled much larger networks to be trained, and removed the need for early stopping.
C Experiments on Reuters · D Tiny Images and CIFAR-10
The Reuters experiments used a 50-class, 402,738-document corpus to compare dropout with standard backpropagation, while the next section introduced Tiny Images and its CIFAR-10 subset. Dropout improved Reuters generalization and enabled smooth learning without early stopping.
- C Experiments on Reuters: Reuters Corpus Volume I contains 804,414 manually categorized newswire stories spanning 103 topics organized in a typically three-level tree.The topics cover corporate/industrial, economics, government/social, and markets groups.
- C Experiments on Reuters: The Reuters data were reduced to 50 classes and 402,738 documents, then randomly split into equal-sized training and test sets.Categories with no data, only four training examples, or 25% of examples were removed.
- C Experiments on Reuters: Each Reuters document was represented using the 2000 most frequent non-stopwords in the dataset.The representation followed the class and document filtering described for the experimental corpus.
- C Experiments on Reuters: A 2000-2000-1000-50 network trained for 500 epochs with dropout-backpropagation was compared against standard backpropagation.The training hyperparameters matched those used for MNIST dropout training.
- C Experiments on Reuters: Dropout produced significant Reuters improvements in generalization across the tested architectures and allowed learning to proceed smoothly without early stopping.Figure 7 reports training and test-set errors as learning progresses for networks trained with and without dropout.
- D Tiny Images and CIFAR-10: The Tiny Images dataset contains 80 million 32 × 32 color images collected from the web using English-noun image searches.Each image has a highly unreliable label corresponding to the noun used in the search.
- D Tiny Images and CIFAR-10: CIFAR-10 is a Tiny Images subset containing 60000 images across ten classes, with 5000 training images and 1000 testing images per class.Its classes are airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
- D Tiny Images and CIFAR-10: CIFAR-10 was filtered to remove incorrectly labeled images and contains highly varied images without canonical object viewpoint or scale.Images were included when they contained one dominant, easily identifiable instance of a CIFAR-10 class.
E ImageNet … F.5 Weight initialization
The experiments use deep feed-forward CNNs on a large ImageNet object-recognition dataset, combining convolutional architecture, pooling, normalization, rectifying nonlinearities, multinomial logistic regression, and carefully chosen weight initialization. These design choices reduce overfitting, support efficient training, and prevent inactive neurons during learning.
- E ImageNet: ImageNet contains roughly 1.3 million training images across 1000 object categories, with 50000 validation images.The experiments use the 2010 competition version, based on roughly 1000 images per class.
- F Convolutional Neural Networks: The models for CIFAR-10 and ImageNet are deep, feed-forward CNNs composed of layered neurons applying linear filters, biases, and nonlinear activations.Each neuron passes its transformed output to the next layer.
- F Convolutional Neural Networks: CNNs organize neurons topographically and use local filters centered at corresponding input locations, matching spatial structure in images.Shared filtering treats positions equally when similar structures can occur throughout an image.
- F Convolutional Neural Networks: Shared-filter convolution drastically reduces parameters and overfitting capacity, making dropout far less advantageous in convolutional layers.The reduced parameter count also lowers representational capacity relative to networks with distinct filters for every neuron.
- F.1 Pooling: Pooling layers subsample local neighborhoods of convolutional-unit outputs within each neuron bank.Pooling units are arranged topographically and compute a function over their local neighborhoods.
- F.2 Local response normalization: Response normalization layers encourage competition among large activations from neurons in different banks at the same topographic position.They implement a form of lateral inhibition, with N, α, and β selected as validation-set hyperparameters.
- F.3 Neuron nonlinearities: All neurons use the max-with-zero nonlinearity, which reduces training time and the need for contrast normalization because activities do not saturate.Their activities instead scale up when present.
- F.4 Objective function: The networks maximize the multinomial logistic regression objective, equivalent to minimizing average training-case cross-entropy.The cross-entropy is computed between the true label distribution and the model’s predicted label distribution.
F.6 Training · F.7 Learning rates · G Models for CIFAR-10
The models are trained with momentum-based stochastic gradient descent, heuristically selected layerwise learning rates, and scheduled reductions. CIFAR-10 architectures differ by dropout: the regularized model adds a locally connected layer with 50% dropout and more parameters.
- F.6 Training: Training uses stochastic gradient descent with batches of 128 examples and momentum of 0.9.The weight updates use a momentum variable and the average objective derivative over each batch.
- F.6 Training: All models are trained with cuda-convnet on a single NVIDIA GTX 580 GPU.Training takes roughly 90 minutes on CIFAR-10; ImageNet takes roughly four days with dropout and two days without.
- F.6 Training: CIFAR-10 training takes roughly 90 minutes, while ImageNet training takes roughly four days with dropout and two days without.These durations are reported for training with the publicly available cuda-convnet package.
- F.7 Learning rates: Each layer uses the same learning rate, selected heuristically as the largest power of ten that reduces the objective function.Typical learning rates are of the order 10−2 or 10−3.
- F.7 Learning rates: The learning rate is reduced twice by a factor of ten shortly before training terminates.The initial value is typically of the order 10−2 or 10−3.
- G Models for CIFAR-10: The CIFAR-10 model without dropout has three convolutional layers, each followed by pooling, with max-pooling first and average-pooling thereafter.Pooling summarizes a 3×3 neighborhood with stride 2; response normalization follows the first two pooling layers.
- G Models for CIFAR-10: The CIFAR-10 model with dropout adds a fourth weight layer because dropout permits more parameters.This layer is locally connected but not convolutional, contains 16 banks of 3 × 3 filters, and uses 50% dropout.
H Models for ImageNet
The ImageNet model used a seven-weight-layer CNN trained with augmented image patches and dropout, while the no-dropout counterpart omitted the globally connected layers that caused serious overfitting. The authors emphasized that dropout substantially helps even highly complex object-recognition networks.
- Architecture and training: The dropout ImageNet model was a seven-weight-layer CNN trained on randomly extracted 224×224 patches and their horizontal reflections from 256×256 images.The first five weight layers were convolutional and the final two globally connected; max-pooling followed the first, second, and fifth convolutional layers.
- Architecture and training: The no-dropout model was similar but omitted the two globally connected layers because they caused serious overfitting without dropout.
- Result: The authors used the complex architecture to achieve state-of-the-art validation performance while demonstrating that dropout helps highly complex object-recognition networks.They stated that the architecture’s complexity was not the paper’s main point.