Source-linked AI summary
Adversarial Dropout for Supervised and Semi-supervised Learning
Sungrae Park, Jun-Keon Park, Su-Jin Shin, Il-Chul Moon
TL;DR
Deep networks can overfit as complexity grows, motivating regularization beyond standard dropout and input-layer adversarial perturbations. The paper introduces adversarial dropout, which optimizes dropout masks to maximize divergence from supervision, then trains the resulting sub-network; it reports improved supervised and semi-supervised generalization, including 3.55% on SVHN and 9.22% on CIFAR-10, alongside increased sparsity.
Problem
As deep-network complexity increases, overfitting motivates regularization, while existing adversarial training perturbs inputs and does not directly alter network structure.
Method
Adversarial dropout optimizes hidden-layer dropout masks against the model’s label assignment, constrains them near random masks, and trains the resulting reconfigured sub-network.
Results
Adversarial dropout improves supervised and semi-supervised generalization, achieving 3.55% on SVHN and 9.22% on CIFAR-10 with VAdD and VAT together.
Takeaways & Limitations
The approach combines adversarial training with dropout to induce sparser neural-network structures and use poor ensemble models to reduce overfitting.
Takeaways & Limitations
The adversarial dropout mask is restricted to remain close to the random dropout mask to avoid disconnected layers that do not support back propagation.
Abstract
from arXiv · showhide
Recently, the training with adversarial examples, which are generated by adding a small but worst-case perturbation on input examples, has been proved to improve generalization performance of neural networks. In contrast to the individually biased inputs to enhance the generality, this paper introduces adversarial dropout, which is a minimal set of dropouts that maximize the divergence between the outputs from the network with the dropouts and the training supervisions. The identified adversarial dropout are used to reconfigure the neural network to train, and we demonstrated that training on the reconfigured sub-network improves the generalization performance of supervised and semi-supervised learning tasks on MNIST and CIFAR-10. We analyzed the trained model to reason the performance improvement, and we found that adversarial dropout increases the sparsity of neural networks more than the standard dropout does.
Introduction
The paper combines dropout with adversarial training to select dropout masks that challenge the model’s label assignment. Unlike additive adversarial perturbations, adversarial dropout changes hidden-layer connectivity and is reported to improve performance while producing sparser networks.
- Deep networks can overfit as increasing depth raises model complexity, motivating regularization methods such as dropout.
- Dropout randomly disconnects neural units to reduce feature co-adaptation and can be interpreted as training an ensemble of subnetworks.
- Adversarial dropout optimizes dropout-mask direction against the model’s label assignment rather than relying on randomly sampled masks.
- Unlike additive adversarial training, adversarial dropout induces sparse neural-network structure by changing connectivity directly.
- Adversarial dropout improves MNIST supervised performance and achieves state-of-the-art semi-supervised performance on SVHN and CIFAR-10 against recent dropout and adversarial-training methods.
- Its regularization strength is specified by a rank-valued parameter, whereas adversarial training uses a conventional continuous-valued scale.
Preliminaries
The preliminaries review noise injection, self-ensembling dropout, and adversarial training as foundations for adversarial dropout. These components differ in whether they perturb hidden representations, compare sampled subnetworks, or optimize input perturbations against a target distribution.
- Noise Layers: Noise layers replace hidden variables with perturbed versions, including additive Gaussian noise and multiplicative dropout noise.
- Noise Layers: Dropout noise applies an elementwise Bernoulli mask, while additive Gaussian noise adds sampled Gaussian perturbations controlled by variance.
- Self-ensembling: The Π model minimizes divergence between outputs from two independently sampled dropout networks with the same input.
- Self-ensembling: Adversarial dropout is a special case of the Π model in which one dropout network is sampled adversarially.
- Adversarial Training: Adversarial training adds a loss comparing the model output on an adversarially perturbed input with a target distribution.
- Adversarial Training: Standard adversarial training uses the label encoding as its target, whereas virtual adversarial training uses the current model estimate and does not use labels in its adversarial loss.
Method
Adversarial dropout selects dropout masks that maximize output divergence under a constraint, then uses the resulting masks as a training regularizer. The method supports supervised and semi-supervised variants and is approximated through Taylor expansion and integer programming.
- Adversarial dropout formulation: Adversarial dropout selects a mask that maximizes divergence from an adversarial target while remaining close to a sampled dropout mask.The constraint limits the number of changed dropout conditions and prevents a layer from losing all connections.
- Relation to adversarial training: Unlike additive adversarial training, adversarial dropout perturbs hidden units through masks and can produce nonlinear perturbations when applied across multiple layers.The method changes the network's sparse structure, whereas additive perturbation does not directly change network structure.
- Supervised and virtual variants: Supervised adversarial dropout compares the adversarially dropped network output with the label target, whereas virtual adversarial dropout compares two network outputs.The virtual variant uses the sampled dropout output as its target and is designed for semi-supervised regularization.
- Training objective: The training objective combines the sampled-dropout loss with a weighted adversarial-dropout divergence term, and it can also include adversarial-training regularization.Separate trade-off parameters control the adversarial-dropout and adversarial-training terms.
- Fast approximation: The adversarial mask is approximated by relaxing binary dropout variables to continuous values, applying a first-order Taylor expansion, and solving an integer programming problem.The resulting linearized objective can be treated as a 0/1 knapsack problem for selecting the mask.
14 end
The MNIST test-performance table covers supervised and semi-supervised settings, while the approximation discussion addresses zero-gradient behavior in stochastic networks.
- MNIST evaluation: Table 1 reports MNIST test performance using 1,000 labeled examples for semi-supervised learning and 60,000 labeled examples for supervised learning.Each setting is repeated eight times.
- Stochastic-layer condition: The zero-gradient situation does not occur when the model contains additional stochastic layers with independently sampled noises.In that setting, different stochastic-layer samples produce different model outputs.
Experiments
Experiments evaluate adversarial dropout on supervised and semi-supervised classification across MNIST, SVHN, and CIFAR-10, alongside analyses of feature structure and activation sparsity. Adversarial dropout performs strongly against dropout and adversarial-training baselines, especially when combined with VAT, while producing sparser representations than standard dropout.
- Benchmark evaluation: Adversarial dropout was evaluated against standard dropout, Π model, and adversarial training on supervised and semi-supervised benchmark tasks.The experiments covered MNIST, SVHN, and CIFAR-10, with additional sparsity analyses.
- Supervised learning: Adversarial dropout achieved top supervised-learning performance on both SVHN and CIFAR-10, with further improvements when combined with adversarial training.
- Semi-supervised learning: VAdD combined with VAT achieved top semi-supervised performance on SVHN and CIFAR-10 compared with prior reported methods.
- Semi-supervised learning: The combined VAdD-and-VAT model reached 3.55% test error on SVHN and 10.04% and 9.22% test error on CIFAR-10.
- Feature analysis: Adversarial dropout prevented dead filters and standardized nonlocalized features while preserving localized features more effectively than standard dropout.
- Sparsity analysis: Adversarial dropout produced fewer highly activated units and lower mean activation values than no-dropout and standard-dropout models.These activation patterns indicate greater model sparsity than standard dropout.
Disucssion
The discussion contrasts adversarial training with adversarial dropout through a linear-regression analysis. Adversarial dropout selects feature dropouts that cause large loss changes, yielding feature-dependent and selectively activated regularization controlled by the dropout degree.
- Linear-regression analysis: The linear-regression analysis examines how adversarial training and adversarial dropout produce different regularization effects.
- Adversarial training: Adversarial training uses additive input perturbations generated by gradient direction, with noise magnitude controlled by δ.
- Adversarial dropout: Adversarial dropout selects a constrained mask that maximizes the loss after elementwise feature masking, with k controlling the dropout degree.
- Regularization differences: When k is zero, adversarial-dropout penalty terms disappear because the constraint permits no dropout.
- Regularization differences: Adversarial-dropout regularization depends on each example’s feature scale and selectively activates for the largest loss changes, unlike adversarial training’s static and always-active penalties.
Conclusion
The conclusion frames adversarial dropout as a hidden-layer combination of adversarial perturbation and dropout. It reports improved generalization and state-of-the-art semi-supervised results, while relating mask differences to graph-edit and Jaccard distances.
- Conclusion: Adversarial dropout combines adversarial perturbation with dropout on hidden layers rather than applying additive perturbations only at the input layer.
- Conclusion: Training with an adversarially poor ensemble structure was reported to prevent overfitting using a few effective features.
- Conclusion: VAdD with VAT achieved 3.55% error on SVHN and 9.22% on CIFAR-10 in semi-supervised learning.
- Distance measures: Graph edit distance and Jaccard distance were used to relate differences between dropout masks to constraints on their Euclidean distance.
- Distance measures: For binary dropout masks, Euclidean distance provides the number of differently dropped nodes.
Appendix B. Detailed Experiment Set-up
The appendix describes the network architectures and experimental settings used for the paper’s results and points to publicly available TensorFlow implementations.
- Implementation: The appendix documents network architectures and experimental settings for reproducing the reported results.
- Implementation: TensorFlow implementations are available through the paper’s cited GitHub repository.
B.1. MNIST : Convolutional Neural Networks
This section specifies the MNIST CNN experiment settings and situates the supervised comparison among CNN-based classifiers. It also records implementation details for the training procedure and adversarial-dropout regularization.
- MNIST experimental setup: The MNIST experiment uses a CNN architecture documented in Table 3.The dataset contains 70,000 28 × 28 handwritten-digit images, with 60,000 for training and the remainder for testing.
- MNIST experimental setup: Networks are trained with Adam for 100 epochs using minibatches of 128.The reported optimizer settings include learning rate 0.001 and momentum parameters β1 = 0.9 and β2 = 0.999.
- MNIST experimental setup: Adversarial-dropout regularization uses δ = 0.005, corresponding to 10 adversarial changes from a randomly selected dropout mask.The regularization weight is ramped up during the first 30 epochs with a Gaussian curve.
- Comparison scope: The supervised comparison covers additional close-family CNN classifiers but excludes recent architectures such as ResNet and DenseNet.The stated comparison scope is therefore limited to the included CNN-based models.
D.2. CIFAR-10 : Semi-supervised classification results with additional baselines
This section reports additional baseline comparisons for semi-supervised CIFAR-10 classification. The implementation reproduces the reported baseline performance and reports improvement from adversarial dropout.
- Baseline comparison: Additional baseline models are compared for the semi-supervised learning task.The comparison extends beyond the primary baseline set described elsewhere in the paper.
- Results: The implementation reproduces the reported baseline performance and shows performance improvement from adversarial dropout.The supplied passage characterizes this as a comparison against the reported results rather than providing numerical values.
- Related evaluation: Supervised CIFAR-10 performance is also tabulated, with each setting repeated five times.Table 6 is identified as the supervised-learning performance table.
- Semi-supervised CIFAR-10: The semi-supervised CIFAR-10 evaluation uses 4,000 labeled examples and repeats each setting five times.These conditions are stated in the table caption for the semi-supervised task.
Appendix E. Proof of Linear Regression Regularization
The appendix derives regularization terms for adversarial training and adversarial dropout in linear regression. It formulates adversarial dropout through a constrained mask-selection problem and reorganizes the resulting objective.
- Linear-regression setup: The proof begins with linear regression on data points x_i and targets y_i, using a weight vector w and loss l(w).The setup defines the regression variables before introducing adversarial perturbations.
- Adversarial training: Adversarial examples are formed by adding r_i^adv = δ sign(∇_x_i l(w)) to x_i.The parameter δ controls the degree of adversarial noise, and the construction uses FGSM.
- Adversarial dropout: Adversarial dropout selects a mask perturbation by maximizing the regression loss subject to a constraint ∥ϵ_i − 1∥_2 ≤ k.The hyperparameter k controls the degree of adversarial dropout, and the derivation uses one vector as the base condition.
- Adversarial dropout: The selected condition is characterized through the kth lowest element of x_i ⊙ ∇_x_i l(w) and satisfies the dropout constraint.The derivation then defines the adversarial-dropout objective and reorganizes its regularization term.