PaperlayerView exact revision on arXiv

Source-linked AI summary

Dropout Training as Adaptive Regularization

Stefan Wager, Sida Wang, Percy Liang

arXiv:1307.1493v2stat.MLcs.LGstat.ME

TL;DR

Dropout’s theoretical basis is not well understood despite its success in controlling overfitting. This paper frames dropout as adaptive regularization, connects it to AdaGrad, and uses unlabeled data to improve the regularizer, consistently boosting document-classification performance and surpassing prior state-of-the-art on IMDB reviews.

  • Problem

    Despite dropout’s success in controlling overfitting, its theoretical reasons for success remain insufficiently understood.

  • Method

    The paper analyzes dropout as Fisher-scaled adaptive L2 regularization, connects it to AdaGrad, and develops a discriminative semi-supervised scheme using unlabeled data.

  • Results

    The semi-supervised approach consistently improves dropout training across document-classification tasks and outperforms previous state-of-the-art on IMDB reviews.

  • Takeaways & Limitations

    Viewing dropout as adaptive regularization unifies it with balanced L2 regularization and AdaGrad while yielding an effective semi-supervised training method.

  • Takeaways & Limitations

    The quadratic surrogate for the dropout regularizer is nonconvex, although the authors report no major optimization issues when training logistic regression.

Abstract

from arXiv · show

Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling the features by an estimate of the inverse diagonal Fisher information matrix. We also establish a connection to AdaGrad, an online learning algorithm, and find that a close relative of AdaGrad operates by repeatedly solving linear dropout-regularized problems. By casting dropout as regularization, we develop a natural semi-supervised algorithm that uses unlabeled data to create a better adaptive regularizer. We apply this idea to document classification tasks, and show that it consistently boosts the performance of dropout training, improving on state-of-the-art results on the IMDB reviews dataset.

1 Introduction

The paper explains dropout as adaptive regularization: feature corruption yields an L2-type effect, with dropout scaling features using inverse Fisher information. This view also connects dropout to AdaGrad and motivates a semi-supervised method using unlabeled data to improve document classification.

  • 1 Introduction: Dropout controls overfitting by randomly omitting feature subsets during training, but its theoretical basis was previously unclear.The paper studies feature dropout as a generic training method applicable to any learning algorithm.
  • 1 Introduction: Artificial feature corruption is connected to regularization, including an L2-type effect for additive Gaussian noise in the low-noise limit.This motivates analyzing dropout through regularization rather than only as data corruption.
  • 1 Introduction: Dropout is first-order equivalent to L2 regularization after transforming inputs by diag(ˆI)−1/2, where ˆI estimates the Fisher information matrix.The transformation makes objective level curves more spherical and balances regularization across features.
  • 1 Introduction: For logistic regression, dropout acts as adaptive L2 regularization that favors rare but useful features.This addresses the setting where rare but useful features create challenges for learning algorithms.
  • 1 Introduction: A close relative of AdaGrad advances by repeatedly solving linearized dropout-regularized problems, paralleling SGD’s solution of linearized L2-regularized problems.The paper presents this as an intimate connection between AdaGrad and dropout training.
  • 1 Introduction: The adaptive-regularization formulation yields a fully discriminative semi-supervised method that uses unlabeled data to learn a better dropout regularizer and consistently improves dropout training on document classification.The approach is applied to several document classification problems, including the benchmark IMDB reviews dataset.

2 Artificial Feature Noising as Regularization

Artificial feature noising in GLMs is equivalent to adding a positive, label-independent regularization penalty, with dropout and Gaussian noise as key instances. A quadratic surrogate accurately approximates this penalty and supports practical logistic-regression training while justifying prediction on clean features.

  • Feature-noising schemes: Artificial feature noising replaces each feature vector with an independently corrupted version and averages over many noisy datasets.The framework considers additive Gaussian noise and dropout noise.
  • Feature-noising schemes: Dropout sets each feature to zero with probability δ and rescales it by 1/(1 − δ) otherwise, while Gaussian noise adds ξ_i ∼ N(0, σ^2I_d×d).Dropout uses independent scaled Bernoulli components; Gaussian noise is zero-mean.
  • Noising as regularization: For GLMs, artificial feature noising reduces to a positive regularizer R(β) that is independent of the labels and penalizes model complexity rather than classifier accuracy.Convexity of the log-partition function makes R positive by Jensen’s inequality, enabling comparison with ridge and lasso penalties.
  • Prediction with clean features: For GLMs, predicting with clean features is formally justified because clean-feature and noise-averaged prediction rules are equivalent when noising induces a label-independent likelihood penalty.The paper notes that the two prediction expressions are generally not equivalent outside this condition.
  • Quadratic approximation: The quadratic surrogate R_q is generally very accurate, slightly overestimates the true penalty near p ≈ 0.5, and underestimates it for highly confident predictions.On a real classification dataset, the approximation remained accurate through most training and deteriorated only slightly near convergence; fitting with R_q gave similar results to dropout regularization.

3 Regularization based on Additive Noise

Additive feature noise yields a quadratic regularization penalty whose effect depends on the likelihood. It exactly recovers ridge regression for linear models, while logistic models receive both parameter shrinkage and pressure toward confident predictions.

  • Linear regression: With isotropic noise Var[ε] = σ2I_d×d, the noisy linear predictor has variance σ2∥β∥2.The variance calculation supplies the quadratic penalty underlying the ridge correspondence.
  • Linear regression: Additive feature noise in linear regression is equivalent to ridge regression with L2 regularization.For linear regression, the quadratic approximation is exact, so the L2 correspondence is exact.
  • Logistic regression: For logistic regression, the curvature is A′′(x_i · β) = p_i(1 − p_i), where p_i is the predicted probability of y_i = 1.Here p_i = (1 + exp(−x_i · β))^-1.
  • Logistic regression: The logistic noising penalty encourages parsimonious modeling by shrinking ∥β∥2 and confident predictions by moving p_i away from 1/2.The supplied passage states that the penalty encourages small ∥β∥2 and pushes the probabilities away from 1/2.

4 Regularization based on Dropout Noise

Dropout induces an adaptive regularizer whose penalty depends on feature activity and prediction uncertainty, approximating L2 regularization after feature normalization by inverse diagonal Fisher information. This explains its advantage for rare, discriminative features and its superior simulated performance over L2 regularization.

  • Dropout independently zeros each feature with probability δ and rescales it by 1/(1 −δ) otherwise, thereby corrupting training examples.
  • At the maximum-likelihood estimate, dropout approximates an L2 penalty after normalizing features by diag(I)^−1/2, where I estimates Fisher information.This normalization makes the local loss geometry more spherical across feature directions.
  • Linear Regression: For linear regression, dropout is equivalent to ridge regression after normalizing each design-matrix column.
  • Logistic Regression: In logistic regression, dropout favors confident predictions and small β, while permitting large β_j when associated predictions have small p_i(1 −p_i).
  • Logistic Regression: Dropout is especially suited to rare, discriminative features because observations with x_ij = 0 do not penalize β_j, provided the feature supports confident predictions when active.The benefit specifically concerns rare features positively co-adapted with other features.
  • In simulations with 50 features active 4% of the time and 1000 always-active nuisance features, dropout outperformed L2 regularization.The study used d = 1050 features and found that inactive rare-feature examples prevent penalty interactions with uncertain predictions.

5 Dropout Regularization in Online Learning

The section connects dropout regularization to online learning by showing that dropout descent is an AdaGrad-like adaptive procedure. Both methods scale features using estimates of Fisher information, favoring rare and useful features over ordinary L2 regularization.

  • SGD and adaptive regularization: SGD can be viewed as repeatedly solving linearized L2-regularized problems.The update’s first two terms linearize the loss, while the third is an L2 regularizer.
  • AdaGrad and rare features: AdaGrad modifies SGD with an online-learned transformation to address slow learning for rare but highly discriminative features.Its adaptive transformation uses A_t = diag(G_t)^1/2 and achieves desirable regret bounds for rare, useful features.
  • Dropout descent: Dropout descent is first-order equivalent to adaptive SGD with A_t = diag(H_t).This equivalence arises when dropout replaces the L2 penalty in the linearized online-learning update.
  • Fisher-information connection: For generalized linear models, G_t and H_t consistently estimate the Fisher information as β̂_t converges to β*.Both corresponding expressions equal the Fisher information, linking dropout-based online learning to AdaGrad-like updates.
  • Relationship and distinction: AdaGrad and dropout share the goal of scaling features by Fisher information, although AdaGrad uses diag(G_t)^−1/2 rather than diag(G_t)^−1 for a more aggressive learning rate.This scaling aims to make objective level-curves more circular, unlike ordinary L2 regularization.

6 Semi-Supervised Dropout Training

The section develops semi-supervised dropout training by estimating its label-independent regularizer from unlabeled data, motivated by confidence on both labeled and unlabeled examples. Experiments show that this approach consistently improves dropout training, including on IMDB text classification.

  • Semi-supervised method: Because the dropout regularizer is independent of labels, unlabeled examples can estimate it more accurately.The method introduces a discount factor α for unlabeled data.
  • Semi-supervised method: α ∈[0.1, 0.4] works well empirically, with the discount selected by crossvalidation and a quadratic surrogate used for optimization.The quadratic surrogate is optimized instead of the exact regularizer.
  • Motivation: The approach seeks weights that produce confident predictions on unlabeled data as well as labeled data, unlike methods relying primarily on generative models or input-distribution assumptions.This intuition is shared by entropy regularization and transductive SVMs.
  • Experiments: Using unlabeled data to learn a better regularizer consistently improves dropout-training performance across several text-classification datasets.The experiments include results in Table 4a and a figure illustrating improvement on one dataset.
  • IMDB experiment: On IMDB, the method uses 50,000 additional unlabeled examples, including neutral reviews, to tune a dropout-regularized logistic-regression classifier with unigram/bigram features.The labeled train and test sets each contain 25,000 positive or negative examples.

7 Conclusion

The paper frames dropout training as adaptive regularization, revealing connections to adaptively balanced L2 regularization and AdaGrad. This framework also motivates a simple semi-supervised training method and suggests further study of dropout’s regularizer.

  • Conclusion: Dropout training is analyzed as a form of adaptive regularization.The authors identify multiple opportunities to investigate the connection further.
  • Conclusion: The framework uncovers close connections between dropout training, adaptively balanced L2 regularization, and AdaGrad.
  • Conclusion: The adaptive-regularization view leads to a simple yet effective method for semi-supervised training.
  • Conclusion: MNB with EM failed to give an improvement in the authors’ semi-supervised MNB implementation.

A Appendix

The appendix examines quadratic approximations to the logistic loss, showing how approximation quality depends on the expansion point while curvature remains symmetric in the natural parameter.

  • Quadratic approximations: At η = 0, where p = 0.5, the quadratic approximation always lies above the actual logistic loss.This approximation is represented by the red curve in Figure A.1.
  • Quadratic approximations: At the more extreme points p = 0.05 and p = 0.95, quadratic approximations undershoot the true loss over a large range.The approximation behavior is equivalent at p and 1 − p because loss curvature is symmetric in η.

A.1 Description of Simulation Study

The simulation study uses 1050-dimensional features with 50 structured discriminative features and 1000 Gaussian nuisance features, then generates labels from a logistic model with signal only in the first 50 coordinates.

  • Generating features: The simulation has 1050 features: 50 discriminative features arranged into five groups of 10 and 1000 nuisance features.Each feature vector is generated independently.
  • Generating features: For each example, one of 25 groups and a sign are selected; groups 1–5 activate one discriminative group, whereas groups 6–25 activate none.The active group’s entries are drawn from a signed exponential distribution, while the remaining discriminative features are zero.
  • Generating features: The 1000 nuisance features are independently drawn from N(0, 1), and the procedure gives all columns of X equal expected second moments.The nuisance-feature distribution and equal-moment property are specified separately in the study description.
  • Generating labels: Labels are sampled from Bernoulli(σ(x_i · β)), with β equal to 0.057 on the first 50 coordinates and 0 on the remaining 1000.The coefficient value was chosen so the average |x_i · β| in the presence of signal is 2.
  • Training: Each simulation run uses a training set of n = 75, with groups cycled deterministically and dropout set to δ = 0.9 versus L2 penalization at λ = 32.The penalization parameters were set to roughly optimal values.
Loading 1307.1493v2…