PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributional Smoothing with Virtual Adversarial Training

Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, Shin Ishii

arXiv:1507.00677v9stat.MLcs.LG

TL;DR

The paper addresses overfitting and the need for smooth model outputs by introducing local distributional smoothness and its VAT regularizer. VAT finds adversarial directions from the model distribution without labels, making it suitable for semi-supervised learning at relatively low computational cost. Experiments report strong performance across MNIST, SVHN, and NORB, while the method uses an approximation that ignores the perturbation derivative with respect to model parameters.

  • Problem

    Finite-sample supervised training can overfit, motivating regularization that encourages models to produce smooth outputs for inputs such as images and time series.

  • Method

    VAT regularizes KL-divergence-based local distributional smoothness, using label-independent perturbations and a Hessian-based approximation of the adversarial direction.

  • Results

    VAT was effective in supervised and semi-supervised learning: it trailed only Ladder network on MNIST and outperformed the current state-of-the-art semi-supervised method on SVHN and NORB.

  • Takeaways & Limitations

    VAT combines applicability to supervised and semi-supervised training with parametrization invariance, relatively low computational cost, and only two hyperparameters.

  • Takeaways & Limitations

    The method ignores the derivative of the approximated adversarial direction with respect to model parameters because its numerical evaluation was volatile.

Abstract

from arXiv · show

We propose local distributional smoothness (LDS), a new notion of smoothness for statistical model that can be used as a regularization term to promote the smoothness of the model distribution. We named the LDS based regularization as virtual adversarial training (VAT). The LDS of a model at an input datapoint is defined as the KL-divergence based robustness of the model distribution against local perturbation around the datapoint. VAT resembles adversarial training, but distinguishes itself in that it determines the adversarial direction from the model distribution alone without using the label information, making it applicable to semi-supervised learning. The computational cost for VAT is relatively low. For neural network, the approximated gradient of the LDS can be computed with no more than three pairs of forward and back propagations. When we applied our technique to supervised and semi-supervised learning for the MNIST dataset, it outperformed all the training methods other than the current state of the art method, which is based on a highly advanced generative model. We also applied our method to SVHN and NORB, and confirmed our method's superior performance over the current state of the art semi-supervised method applied to these datasets.

1 INTRODUCTION

The paper introduces VAT, which regularizes local smoothness of a model’s output distribution using label-independent perturbations. It argues that VAT is broadly applicable, parametrization invariant, computationally efficient, and effective across supervised and semi-supervised benchmarks.

  • The method is motivated by the observation that useful model outputs should often vary smoothly with natural inputs such as images and time series.
  • VAT defines local distributional smoothness as the model distribution’s sensitivity to input perturbations measured by KL divergence.The resulting regularization term is based on the model distribution itself.
  • VAT’s regularized objective is parametrization invariant, unlike popular Lq regularization.The paper states that the optimal model distribution is unique regardless of parametrization.
  • Unlike conventional adversarial training, VAT determines perturbation directions without label information, enabling semi-supervised learning.
  • For neural networks, the approximated LDS gradient requires no more than three pairs of forward and back propagations.
  • VAT outperformed contemporary methods on MNIST except the state-of-the-art generative-model method, and exceeded the leading semi-supervised method on SVHN and NORB.

2 METHODS

VAT regularizes a model by measuring and reducing the KL-divergence sensitivity of its output distribution to local input perturbations. It estimates the most sensitive direction using Hessian-based power iteration and computes the resulting regularization gradient efficiently.

  • Local distributional smoothness: LDS measures the model distribution's sensitivity to local input perturbations using KL divergence, and adding LDS terms regularizes the likelihood objective.Because it measures smoothness of the model distribution itself, the formulation is parametrization invariant.
  • Virtual adversarial perturbation: VAT determines the virtual adversarial direction from the model distribution without label information, enabling semi-supervised learning.This contrasts with adversarial training, which identifies perturbations using observed input-label pairs.
  • Perturbation evaluation: Under a second-order approximation, the virtual adversarial perturbation is the dominant eigenvector of the KL-divergence Hessian with magnitude ϵ.The Hessian is defined as the second derivative of the KL divergence with respect to the perturbation at zero.
  • Perturbation evaluation: Power iteration and finite differences approximate the perturbation without explicitly computing the Hessian, with accuracy improving monotonically as iterations increase.A randomly initialized unit vector converges toward the dominant eigenvector when it is not perpendicular to it.
  • Gradient computation: The LDS gradient with respect to parameters ignores the perturbation's parameter derivative because numerical evaluation was volatile and reduced effective regularization.This modification produced better generalization performance and higher LDS-related objective values in the authors' experiments.
  • Computational cost: For neural networks, VAT requires two forward and two backward propagations, plus one additional forward propagation for unlabeled samples when likelihood and LDS use the same dataset.Thus the supervised computation uses four propagation passes, while semi-supervised training adds one forward pass.

3 EXPERIMENTS

Experiments compare VAT with other regularization methods on synthetic binary datasets and benchmark supervised and semi-supervised tasks. VAT produced smoother, more accurate models and strong benchmark performance, while retaining low computational cost.

  • Synthetic datasets: The synthetic ‘Moons’ and ‘Circles’ datasets contain 16 training samples and 1000 test samples, making MLE vulnerable to overfitting.The data are generated from two trajectories in R^2 and linearly embedded into 100 dimensions.
  • Synthetic datasets: During training, VAT’s average LDS decreased slowly while MLE’s fell rapidly; both reached zero training error, but VAT had significantly lower final test error.The comparison used ϵ = 0.5 and Ip = 1 for training, with LDS evaluation using Ip = 5.
  • Synthetic datasets: VAT consistently produced intended decision boundaries for both ‘Moons’ and ‘Circles’, whereas other methods were inconsistent or decisively wrong.VAT imposes local smoothness around each data point; globally smoothing the model can reduce accuracy.
  • Synthetic datasets: VAT and adversarial training achieved much lower test errors than the other regularization methods on both synthetic datasets.Figure 5 compares average test errors for ‘Moons’ and ‘Circles’.
  • Benchmark datasets: For supervised permutation-invariant MNIST, VAT outperformed contemporary methods except the Ladder network, a highly advanced generative-model-based method.Test errors were selected using validation performance and averaged over repeated training runs.
  • Benchmark datasets: VAT also outperformed contemporary semi-supervised methods on permutation-invariant MNIST and strongly outperformed the current state-of-the-art method on SVHN and NORB.Its label-independent LDS enables semi-supervised training.

4 DISCUSSION AND RELATED WORKS

VAT smooths the model distribution using local input-output sensitivity, distinguishing itself from label-based adversarial training and global L2 smoothing. Its principal-direction perturbations outperform random perturbation approaches, while related methods differ in their use of input distributions or Jacobian penalties.

  • Related methods: VAT defines local robustness through the model’s input-output relationship, whereas L2 regularization provides global rather than local smoothing.Adversarial training and VAT use local relationships; L2 regularization does not.
  • Related methods: VAT outperforms PEA and random perturbation training, indicating the importance of selecting perturbations using the Hessian’s principal direction.Random directions can increase loss variance in unnecessary dimensions, while VAT follows the direction of greatest model sensitivity.
  • Experimental comparison: VAT outperformed contemporary methods except Ladder network on permutation-invariant MNIST and strongly outperformed the current state-of-the-art semi-supervised method on SVHN and NORB.Table 2 reports averaged test errors for the three datasets, with stars marking methods dependent on generative models or pre-training.
  • Related methods: Deep contractive networks penalize the Frobenius norm of output Jacobians, but were unable to significantly decrease test error.The method approximates the full Jacobian using adjacent hidden-layer pairs.
  • Related methods: Ladder network uses layer-wise denoising autoencoders, while VAT models only p(y|x, θ) and can therefore complement methods that explicitly model the input distribution.The paper identifies manifold-learning extensions as a possible direction for improving VAT.

5 CONCLUSION

The experiments indicate that VAT is effective for supervised and semi-supervised learning across synthetic data and MNIST, SVHN, and NORB. The method is also presented as computationally simple and requiring few hyperparameters.

  • Empirical findings: VAT was effective in supervised and semi-supervised learning experiments on synthetic datasets, MNIST, SVHN, and NORB.On MNIST it trailed only Ladder network, while on SVHN and NORB it outperformed the current state-of-the-art semi-supervised method.
  • Simplicity: VAT uses two hyperparameters, ϵ and λ, and the experiments worked sufficiently well while optimizing ϵ with λ fixed at 1.The conclusion contrasts this with models relying heavily on generative models, which depend on many hyperparameters.
  • Simplicity: VAT has relatively low computational cost because its approximated LDS gradient requires no more than three pairs of forward and back propagations.This computational property is emphasized as part of the method’s simplicity.

A.1 SUPERVISED BINARY CLASSIFICATION FOR SYNTHETIC DATASETS

The synthetic-data experiments used searched regularization strengths and fixed λ = 1 for random perturbation training, adversarial training, and VAT. Training used specified momentum and learning-rate schedules for 1,000 updates.

  • Hyperparameter search: The synthetic experiments searched λ from 1e-4 through 200 for L2 regularization and p(z) from 0.05 through 0.95 for input dropout.These ranges were part of the hyperparameter search space.
  • Hyperparameter search: The searched perturbation ranges were ϵ = 0.2 through 4.0 for random perturbation, 0.01 through 0.2 for L∞ adversarial training, and 0.1 through 2.0 for L2 adversarial training and VAT.All random perturbation, adversarial training, and VAT experiments used λ = 1.
  • Optimization: Training used momentum µ_i = 0.9, an exponentially decreasing learning rate with rate 0.995, initial learning rate γ_1 = 1.0, and 1,000 parameter updates.The schedule and update count were fixed in the reported experiment.

A.2 SUPERVISED CLASSIFICATION FOR THE MNIST DATASET

The supervised MNIST experiments searched perturbation strengths for adversarial training and VAT under λ = 1, using one power iteration for VAT. Optimization used mini-batch ADAM with 50,000 parameter updates.

  • Hyperparameter search: The searched ϵ ranges were 0.05 through 0.1 for L∞ adversarial training and 1.0 through 3.0 for L2 adversarial training and VAT.VAT used I_p = 1 during the search.
  • Optimization: All supervised MNIST experiments used λ = 1 and mini-batch stochastic gradient descent based on ADAM.The mini-batch size was 100, with ADAM’s default tunable parameters.
  • Optimization: The networks were trained for 50,000 parameter updates.This update count was used for the supervised MNIST training procedure.

A.3 SEMI-SUPERVISED CLASSIFICATION FOR THE MNIST DATASET

The MNIST semi-supervised experiments selected perturbation hyperparameters by grid search and used fixed VAT settings with ADAM-based minibatch SGD.

  • For N_l = 100, the best ϵ was searched over {0.2, 0.3, 0.4}.
  • For all other cases, the best ϵ was selected from {1.5, 2.0, 2.5}.
  • The experiments used λ = 1, I_p = 1, and ADAM-based minibatch SGD with the supervised-setting hyperparameters.

A.4 SEMI-SUPERVISED CLASSIFICATION FOR THE SVHN AND NORB DATASET

The SVHN and NORB experiments used downsampled, whitened image inputs and deeper ReLU networks, with VAT hyperparameters selected by grid search and fixed training settings.

  • SVHN images were downsampled from 32 × 32 × 3 to 16 × 16 × 3 and vectorized into 768-dimensional vectors before whitening.
  • Both datasets used ReLU networks with hidden-layer sizes (1200, 600, 300, 150, 150), deeper than the MNIST network.
  • The perturbation hyperparameter ϵ was grid-searched over {1.0, 1.5, · · ·, 4.5, 5.0}, with λ = 1 and I_p = 1.
  • Training used ADAM-based minibatch SGD with minibatches of 100, an initial learning rate of 0.002, 0.9 exponential decay per epoch, and 100 epochs.
Loading 1507.00677v9…