PaperlayerView exact revision on arXiv

Source-linked AI summary

Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Shin Ishii

arXiv:1704.03976v2stat.MLcs.LG

TL;DR

Semi-supervised learning needs regularization that can operate without labels while addressing overfitting. VAT smooths the model along virtual adversarial directions, and experiments found it effective across supervised and semi-supervised benchmarks, greatly outperforming state-of-the-art methods on SVHN and CIFAR-10.

  • Problem

    Semi-supervised learning requires regularization that can address overfitting without relying on supervisory signals for every input.

  • Method

    VAT promotes local distributional smoothness by efficiently regularizing the model against virtual adversarial directions defined without label information.

  • Results

    VAT was effective in supervised and semi-supervised learning, outperforming recent methods on MNIST and greatly outperforming state-of-the-art semi-supervised methods on SVHN and CIFAR-10.

  • Takeaways & Limitations

    VAT offers a regularization approach applicable to semi-supervised learning because its local smoothness objective does not require supervisory signals.

  • Takeaways & Limitations

    K = 1 might not be sufficient depending on the dataset and model, because the Hessian spectrum is not guaranteed to be skew.

Abstract

from arXiv · show

We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input. Virtual adversarial loss is defined as the robustness of the conditional label distribution around each input data point against local perturbation. Unlike adversarial training, our method defines the adversarial direction without label information and is hence applicable to semi-supervised learning. Because the directions in which we smooth the model are only "virtually" adversarial, we call our method virtual adversarial training (VAT). The computational cost of VAT is relatively low. For neural networks, the approximated gradient of virtual adversarial loss can be computed with no more than two pairs of forward- and back-propagations. In our experiments, we applied VAT to supervised and semi-supervised learning tasks on multiple benchmark datasets. With a simple enhancement of the algorithm based on the entropy minimization principle, our VAT achieves state-of-the-art performance for semi-supervised learning tasks on SVHN and CIFAR-10.

1 INTRODUCTION

The paper introduces virtual adversarial training (VAT), which promotes isotropic smoothness by selectively smoothing the model along directions that most alter its output distribution, without supervisory signals. VAT is designed for semi-supervised learning, broad parametric-model applicability, efficient computation, and strong performance across benchmark datasets.

  • Motivation: Finite samples create overfitting risk, motivating regularization that incorporates prior beliefs favoring smooth conditional output distributions.Regularization augments the loss and can be interpreted from a Bayesian standpoint as a prior over models.
  • Motivation: Random-noise smoothing can leave predictors vulnerable to adversarial directions where label probabilities are especially sensitive.Adversarial training improves generalization and robustness by addressing these directions.
  • Method: VAT selectively smooths each input along a virtual adversarial direction that maximally changes the output distribution according to distributional divergence.The method defines local distributional smoothness as divergence-based robustness against this direction and promotes it while maximizing model likelihood.
  • Advantages: VAT supports semi-supervised learning, applies to differentiable parametric models, uses few hyperparameters, and provides parametrization-invariant regularization.For neural networks, an approximation enables implementation at no more than three times the computational cost of standard regularization-free training.
  • Results and extensions: VAT outperformed contemporary methods on permutation-invariant MNIST and achieved superior or comparable performance to state-of-the-art methods on semi-supervised CIFAR-10 and SVHN.The article extends earlier work through objective clarification, RPT comparison, additional experiments, virtual-adversarial-example evaluation, and entropy minimization.

2 RELATED WORKS

Prior work regularizes neural networks by smoothing model behavior with random or adversarial perturbations, while semi-supervised methods also use noise processes, image augmentation, label propagation, and generative models. VAT follows this smoothing philosophy but replaces random noise with virtual adversarial perturbations and directly augments the function with a regularization term.

  • Random perturbation regularization: Classic neural-network regularization applies random perturbations to inputs or hidden layers to smooth the input-output relation.Gaussian input perturbations induce an extra objective regularizer and, for small perturbations, behave similarly to Tikhonov regularizers.
  • Random perturbation regularization: Dropout regularizes neural networks with random masks and can be interpreted as Bayesian model ensembles, complementing VAT’s direct function regularization.The masks are applied to inputs or hidden layers during training.
  • Semi-supervised learning: Semi-supervised neural-network methods use random perturbations, reconstruction constraints, and image augmentation to stabilize outputs, improve generalization, or propagate labels.Pseudo Ensemble Agreement constrains layer outputs under noise, while ladder networks reconstruct lower-layer signals; image augmentation smooths models around data points and extrapolates labels to unlabeled examples.
  • VAT’s relation to prior work: VAT is philosophically similar to prior noise-based semi-supervised work but uses virtual adversarial perturbations as its noise process.The paper reports that this perturbation choice improved generalization performance in experiments.
  • Generative approaches: A separate semi-supervised family uses sophisticated generative models, including variational-autoencoder-based methods and their variants, without explicitly defining smoothness.This family is distinguished from the previously introduced smoothing-based methods.

3 METHODS

VAT regularizes the model by enforcing local smoothness of predicted label distributions around both labeled and unlabeled inputs, using virtual adversarial perturbations derived without unknown labels. Its efficient implementation uses power iteration and a small hyperparameter set, while producing a more targeted and stable smoothing direction than random perturbation training.

  • Problem setup: VAT trains p(y|x, θ) using labeled dataset Dl and unlabeled dataset Dul, enabling learning from both data types.The model is defined through the conditional output distribution p(y|x, θ).
  • Virtual adversarial perturbation: VAT replaces the unavailable q(y|x) at unlabeled inputs with the current model estimate p(y|x, ˆθ), creating virtual labels for perturbation construction.This substitution motivates the term “virtual” and supports semi-supervised learning.
  • VAT objective: The virtual adversarial loss LDS measures local smoothness by comparing output distributions at x∗ and x∗ + rvadv, and VAT averages this loss over all input points as regularizer Rvadv.Reducing LDS makes the model smoother at each data point, alongside the labeled-data negative log-likelihood.
  • Hyperparameters: VAT has two scalar hyperparameters, ϵ and α, and in all experiments fixed α = 1 while tuning only ϵ.ϵ constrains the adversarial direction, whereas α balances the negative log-likelihood and Rvadv.
  • Efficient computation: VAT computes rvadv with a second-order approximation because the divergence gradient at zero perturbation vanishes, then uses power iteration to approximate the dominant adversarial direction.The method assumes twice differentiability almost everywhere and uses K ≥ 1 power iterations from a random sphere sample.
  • VAT versus RPT: Compared with RPT’s uniformly distributed isotropic smoothing, VAT selects a more deterministic proximal direction and reduces variance, while RPT falls behind VAT in reducing generalization error.VAT smooths along points aligned with the virtual adversarial direction rather than across the entire isotropic neighborhood.

4 EXPERIMENTS · 4.1 Testing the Efficacy of VAT on Benchmark Tasks

The experiments evaluate VAT against competing methods, study its hyperparameters, robustness, and perturbation mechanism, and test supervised and semi-supervised image classification. VAT performs strongly on MNIST, SVHN, and CIFAR-10, including state-of-the-art semi-supervised results with entropy minimization.

  • 4 EXPERIMENTS: Experiments compare VAT with RPT and recent supervised and semi-supervised algorithms while examining hyperparameters, robustness, and the virtual-versus-random perturbation mechanism.The evaluations use MNIST, CIFAR-10, and SVHN benchmarks.
  • 4.1.1 Supervised Learning on MNIST and CIFAR-10: VAT training made the MNIST model smoother than the baseline, as its Rvadv exceeded that of the network trained without VAT during training.Rvadv was evaluated with ϵ = 2.0 for both models.
  • 4.1.1 Supervised Learning on MNIST and CIFAR-10: In supervised MNIST experiments, VAT outperformed all contemporary regularization methods except ladder networks.The MNIST network used four hidden layers with 1200, 600, 300, and 150 units.
  • 4.1.1 Supervised Learning on MNIST and CIFAR-10: Increasing the number of power iterations K beyond one did not produce substantial performance improvement.The study explicitly tested VAT with K > 1.
  • 4.1.2 Semi-Supervised Learning on MNIST, SVHN, and CIFAR-10: VAT applies to semi-supervised learning because LDS(x, θ) does not require the supervisory signal for x, unlike adversarial training.The method was evaluated on semi-supervised MNIST, SVHN, and CIFAR-10.
  • 4.1.2 Semi-Supervised Learning on MNIST, SVHN, and CIFAR-10: 14.82(%) was the test error rate achieved by VAT in semi-supervised SVHN and CIFAR-10 experiments, outperforming state-of-the-art methods on CIFAR-10.The experiments used CNN architectures from recent semi-supervised learning methods.
  • 4.1.2 Semi-Supervised Learning on MNIST, SVHN, and CIFAR-10: VAT+EntMin outperformed state-of-the-art semi-supervised methods on both SVHN and CIFAR-10, including with moderate image data augmentation on CIFAR-10.VAT+EntMin combines Rvadv with conditional entropy minimization, which exaggerates predictions on each data point.

4.2 Effects of Perturbation Size ϵ and Regularization Coefficient α

VAT uses only two hyperparameters, α and ϵ, but experiments show that fixing α = 1 and tuning ϵ alone can provide effective performance in appropriate settings. For small ϵ, regularization depends on α and ϵ2 jointly, whereas larger ϵ may require searching their best pair.

  • Hyperparameter Search: VAT involves only two hyperparameters, α and ϵ, and competitive results were achieved while fixing α = 1 in Section 4.1 experiments.This reduces the hyperparameter-optimization burden for large neural networks and datasets.
  • Hyperparameter Search: For small ϵ, VAT regularization strength is proportional to αϵ2, so searching either α or ϵ alone can suffice.This equivalence follows from the small-ϵ Taylor expansion of the virtual adversarial loss.
  • Hyperparameter Search: For relatively large ϵ, α and ϵ cannot be combined, so the best-performing pair should be searched jointly.The small-ϵ simplification does not apply when the perturbation size is relatively large.
  • Empirical Findings: On MNIST, tuning ϵ alone achieved satisfactory VAT performance, leading to the recommendation to prioritize searching ϵ over α.The supervised MNIST parameter search found optimal performance at α = 1, which was then fixed in benchmark experiments.
  • Empirical Findings: Tuning ϵ alone was sufficient for good VAT performance and achieved state-of-the-art performance for several tasks.This summarizes the reported benchmark outcome after fixing α = 1.

4.3 Effect of the Number of the Power Iterations K

Increasing the power-iteration count K sharply improves Rvadv from random perturbations at K = 0 to virtual adversarial perturbations at K = 1, after which Rvadv saturates. Increasing K beyond 1 yielded no notable performance improvement on semi-supervised CIFAR-10, although K = 1 may be insufficient for some datasets and models.

  • Effect of the Number of the Power Iterations K: Rvadv increases significantly from K = 0 to K = 1 and then saturates for supervised MNIST and semi-supervised CIFAR-10.K = 0 uses random perturbations, whereas K = 1 uses virtual adversarial perturbations.
  • Effect of the Number of the Power Iterations K: Increasing K beyond 1 produced no notable performance improvement on the semi-supervised CIFAR-10 task.Table 6 reports test accuracies for different K values.
  • Effect of the Number of the Power Iterations K: K = 1 may not suffice when the Hessian spectrum is not skewed, depending on the dataset and model.The sufficiency of one power iteration is therefore not guaranteed universally.

4.4 Visualization of Virtual Adversarial Examples · 4.5 Experimental Assessment of the Difference between VAT and RPT

Sections 4.4–4.5 visualize virtual adversarial examples and assess why VAT outperforms random perturbation training (RPT). VAT-trained models show stronger robustness to relevant virtual adversarial perturbations, while its advantage is attributed to both power-iteration dynamics and smoothing along the model’s most sensitive direction.

  • 4.4 Visualization of Virtual Adversarial Examples: VAT performance across ϵ values was aligned with the corresponding virtual adversarial examples generated after training on SVHN and CIFAR-10.For small ϵ, the examples were difficult for humans to distinguish from clean images; the best validation-performing ϵ was separately identified.
  • 4.4 Visualization of Virtual Adversarial Examples: For adversarial examples around ϵ ∼100, further corruption would make images unidentifiable by humans, defining a range where classifiers should avoid mistakes.The examples were almost identifiable but already heavily corrupted.
  • 4.4 Visualization of Virtual Adversarial Examples: The VAT-trained model had a much lower misidentification rate than the non-VAT model on virtual adversarial examples in the human-identifiable range.The comparison used models trained with and without VAT and perturbations generated from both training conditions.
  • 4.4 Visualization of Virtual Adversarial Examples: The VAT-trained model correctly identified adversarial examples generated by both itself and the model trained without VAT.This result appears in the comparison’s bottom panel.
  • 4.5 Experimental Assessment of the Difference between VAT and RPT: VAT’s advantage over RPT may arise because power iteration accelerates learning by reducing derivative-objective variance and because sensitivity-directed smoothing improves generalization more effectively.These are the two possible reasons proposed for VAT’s performance difference.
  • 4.5 Experimental Assessment of the Difference between VAT and RPT: 3 times: on MNIST, the normalized SD norm of RPT became as large as three times that of VAT.The study compared VAT with K = 1 against RPT with K = 0, with additional gradient averaging used in a performance comparison.
  • 4.5 Experimental Assessment of the Difference between VAT and RPT: VAT’s superior performance was attributed largely to its objective function, particularly its maximum-based smoothing of the model’s most sensitive direction rather than expectation-based smoothing.The paper argues that this direction is the one most needed to repair missing isotropic smoothness around an input.

5 CONCLUSIONS

Experiments on MNIST, SVHN, and CIFAR-10 indicate that VAT is effective for supervised and semi-supervised learning. The method combines strong benchmark performance with a simple, computationally efficient implementation.

  • VAT outperformed recent popular MNIST methods other than ladder networks, which use a special network structure.
  • VAT greatly outperformed current state-of-the-art semi-supervised learning methods on SVHN and CIFAR-10.
  • The approximation of R_vadv avoids internal optimization when selecting the adversarial direction and keeps computational cost small.
  • VAT has only two hyperparameters, ϵ and α, and performs sufficiently well on the benchmark dataset when only ϵ is optimized.
  • VAT applies to a wide variety of models regardless of architecture.

APPENDIX A SUPERVISED CLASSIFICATION FOR THE MNIST DATASET

The appendix describes supervised MNIST classification experiments, including the dataset structure, hyperparameter search ranges, and neural-network training setup.

  • Dataset: MNIST contains 28 × 28 handwritten-digit images, giving 784 input dimensions and labels from 0 to 9.The dataset consists of images and corresponding digit labels.
  • Hyperparameter search: The searched perturbation ranges were ϵ = [0.05, 0.1] for L∞ adversarial training and ϵ = [0.05, 5.0] for both L2 adversarial training and VAT.All experiments used α = 1 except when evaluating α effects in Section 4.2.
  • Training setup: Training used mini-batch SGD with ADAM, batch size 100, 60,000 parameter updates, initial learning rate 0.002, and exponential decay of 0.9 per 600 updates.ADAM tunable parameters used its default values.

APPENDIX B SUPERVISED CLASSIFICATION FOR CIFAR-10 DATASET

The supervised CIFAR-10 experiment used RGB object images with a 50,000-example training set and a 10,000-example test set. The setup reserved 10,000 training examples for validation, applied ZCA whitening, augmented images, and trained the Conv-Large model.

  • Dataset and preprocessing: CIFAR-10 contains 32 × 32 × 3 RGB images of categorized objects, with 50,000 training examples and 10,000 test examples.The categories include cars, trucks, planes, animals, and humans.
  • Dataset and preprocessing: 10,000 of the 50,000 training examples were used for validation, and ZCA whitening was applied before the experiment.
  • Dataset and preprocessing: Training data were augmented with random 2 × 2 translations and random horizontal flips.
  • Model: The experiment trained the Conv-Large model.The passage refers to Table 7 for the model definition.

APPENDIX C SEMI-SUPERVISED CLASSIFICATION FOR THE MNIST DATASET

The MNIST semi-supervised experiments used the supervised network with Gaussian noise added to hidden variables, evaluated two labeled-sample sizes, and tuned VAT training configurations. Training used separate labeled and mixed-data mini-batches for likelihood and regularization terms, respectively.

  • Model and data: Adding zero-mean Gaussian noise with 0.5 standard deviation to hidden variables stabilized semi-supervised VAT training on MNIST.The experiments used the same network as supervised learning and tested N_l ∈ {100, 1000} labeled samples.
  • Optimization: The experiments searched ϵ over [0.05, 10.0] while fixing α = 1 and K = 1.ADAM-based mini-batch SGD used the same hyperparameter values as the supervised setting, and likelihood was computed from labeled data only.
  • Optimization: Each update used a labeled mini-batch of size 64 for likelihood and a mixed labeled-unlabeled mini-batch of size 256 for regularization.Networks were trained for 100,000 parameter updates, with linear ADAM learning-rate decay beginning after the 50,000-th update.
  • Evaluation: The experiments were repeated 3 times with different random seeds for weight initialization and labeled-sample selection.The supplied passage also describes linear learning-rate decay after the 50,000-th update.

APPENDIX D SEMI-SUPERVISED CLASSIFICATION FOR THE SVHN AND CIFAR-10 DATASETS

The appendix describes semi-supervised SVHN and CIFAR-10 experiments, including dataset splits, preprocessing, optimization settings, and training-duration differences. It also reports dataset-specific augmentation and repeated runs with different random seeds.

  • Datasets and splits: SVHN contains 73,257 training and 26,032 test images; 1,000 samples were reserved for validation and another 1,000 labeled for semi-supervised training.The images are 32 × 32 × 3 RGB house-number images with labels 0–9.
  • Optimization: ADAM training started at learning rate 0.001 and linearly decayed it during the final 16,000 updates.Reported CNN-Small and CNN-Large results used data augmentation and the specified ϵ choices.
  • Experimental variations: SVHN experiments evaluated the algorithm with and without data augmentation using the validation settings for Conv-Small and Conv-Large.CIFAR-10 models did not appear to converge with 48,000 updates, so results used 200,000 updates instead.
  • Reproducibility: Experiments were repeated 3 times with different random seeds for weight initialization and labeled-sample selection.The reported results were means and standard deviations.
Loading 1704.03976v2…