Generalized Dropout

• Large CNNs risk overfitting despite large datasets; Dropout was proposed heuristically to reduce neuron co-adaptation, and later received an approximate Bayesian interpretation [3].Source: §1 · p. 1
• The paper motivates enforcing independence among CNN features rather than weights when using variational inference.Source: §1 · p. 1
• Generalized Dropout derives a Bayesian approach to neural-network inference and introduces a rich family of regularizers.Source: §1 · p. 2
• The authors experimentally analyze Dropout++ behavior and test Stochastic Architecture Learning for selecting network width.Source: §1 · p. 2
• Experiments report that Dropout++ can boost performance on standard networks, while Stochastic Architecture Learning can select neural-network width.Source: §1 · p. 2

2 Bayesian Neural Networks

• Variational inference approximates the intractable neural-network posterior, while mean-field independence is problematic for correlated parameters and motivates the gated approach.The method uses a delta distribution for weights, avoiding Bayesian inference over them, and approximates the objective with one Monte Carlo sample; the authors note Classic Dropout makes the same approximation.Source: §2 · p. 2, §2.1 · p. 3

2.1 Bayesian Neural Networks with Gates

• Stochastic gates in [0, 1] modulate neuron outputs, and their learned probabilities control each feature’s relative importance.The formulation places priors on gates and uses variational distributions; the same parameter μ2 is used for two distributions for simplicity.Source: §2.1 · pp. 2–3

2.2 Generalized Dropout

• The beta-prior objective over gates and k defines Generalized Dropout, whose variants include Dropout++ and Stochastic Architecture Learning.Dropout++ (0.5) becomes indistinguishable from Classic Dropout at α = β → ∞; other Dropout rates use α ≠ β.Source: §2.2 · pp. 3–4

2.3 Architecture Learning

• Stochastic Architecture Learning is a stochastic version of Architecture Learning, with a similar objective but Bernoulli-sampled gates rather than threshold-selected gates.The authors say their Bayesian route reaches a similar formulation to a method motivated by minimizing neuron count.Source: §2.3 · p. 4

2.4 A Practitioner’s Perspective

• The gate priors shape learned probabilities: SAL favors values near 0 or 1, while Dropout++ (flat) imposes no restriction and needs no hyperparameters.Increasing Dropout++ (0.5)’s regularization penalizes deviations from 0.5 more heavily, recovering Dropout in the limit.Source: §2.4 · p. 5

2.5 Estimating gradients for binary stochastic gates

• The method uses a straight-through estimator for gradients and clips k to [0, 1] before Bernoulli gate sampling.The authors report that clipping worked better than a sigmoid and describe it as a linearized sigmoid.Source: §2.5 · p. 5

2.6 Applying to Convolution Layers

• For convolutional architecture selection, one gate per feature map can suppress entire maps and identify which filters are essential.Generalized Dropout otherwise follows classical dropout’s pointwise multiplication over the output feature map.Source: §2.6 · p. 5

Dropout extensions

• DropConnect stochastically drops weights rather than neurons, but is reported to work only on fully connected layers.The cited work aims for better accuracy through ensembles of networks.Source: §3 · p. 6
• Standout makes the dropout rate depend on layer activations, while Variational Dropout gives Gaussian Dropout a Bayesian interpretation [10].Source: §3 · p. 6

Bayesian interpretations

• Gal and Ghahramani give binary Dropout a Bayesian interpretation and report improved test performance with Monte Carlo averaging rather than test-time rescaling [3].This paper extends that work with a hyper-prior and Bernoulli prior, while using rescaling at test time.Source: §3 · p. 6

Bayesian neural networks

• Unlike Bayesian neural-network methods over weights, this paper is Bayesian with respect to gates [13] [14].Source: §3 · p. 6

4 Experiments

• The experiments first analyze the Generalized Dropout family and then study specific applications.Source: §4 · p. 6

4.1 Analysis of Generalized Dropout

• The Generalized Dropout analysis uses MNIST and a standard LeNet-like network with two convolutional and two fully connected layers.Source: §4.1 · p. 6

4.1.1 Effect of data-size

• Generalized Dropout methods are not always strictly more accurate than Dropout, while Dropout++ (0) seems advantageous on small data.Source: §4.1.1 · p. 6, Figure 2 · p. 7

4.1.2 Effect of Layer-width

• Gate values adapt to layer width, and Dropout++ adjusts on small layers where fixed Dropout at p = 0.5 tends to remove too many neurons.The experiments confirm the expectation that larger layers learn lower gate values and smaller layers higher values.Source: §4.1.2 · pp. 6–7, Figure 2 · p. 7

4.2 Dropout++ on standard models

• Adding Dropout++ after each layer decreases error rates on ResNet32, ResNet56, and a generic VGG-like network trained on CIFAR-10.The table reports that learned values depend on initialization; Dropout++ values initialized at 1.0 are close to 0.98, for which fixed Dropout performs similarly.Source: Table 2 · p. 8, §4.2 · p. 9

4.1.3 Effect of Initialization

• Generalized Dropout parameter initialization is less crucial than the Dropout value, but initializing at p = 1.0 helps training time.That initialization starts with no Dropout and lets the model learn how much to add.Source: Figure 2 · p. 7, §4.1.3 · p. 8

4.1.4 Visualization of Learnt Parameters

• On convolutional layers, Dropout++ learns to attend to image centers rather than corners, potentially reducing evaluation time and breaking translation equivariance.The authors suggest this spatial bias could help with face images, where a filter need not find an eye everywhere.Source: Figure 2 · p. 7, §4.1.4 · p. 8

4.1.5 Architecture Selection

• SAL’s near-binary gates enable pruning low-gate neurons to select layer width; larger β/α ratios yield smaller architectures, sometimes at a performance cost.Source: Table 1 · p. 7, §4.1.5 · p. 8

5 Conclusion

• Dropout++ is generally beneficial when added flat after every layer of a deep network.Source: §5 · p. 9
• Generalized Dropout is a family of methods that generalize Dropout-like behavior.Source: §5 · p. 9
• Dropout++ is an adaptive version of Dropout that helps set a useful parameter value regardless of layer width and initialization.Source: §5 · p. 9
• Stochastic Architecture Learning performs architecture selection.Source: §5 · p. 9
• An uninformed choice of the Dropout parameter usually hurts performance.Source: §5 · p. 9