Source-linked AI summary
Spectral Norm Regularization for Improving the Generalizability of Deep Learning
Yuichi Yoshida, Takeru Miyato
TL;DR
The paper asks why deep-learning models with similar training behavior can generalize differently, focusing on sensitivity to input perturbations. It proposes spectral norm regularization, which limits the spectral norms of weight matrices, and experimentally finds better generalizability than baseline methods. The results support input-perturbation insensitivity as an important factor in generalization.
Problem
Deep-learning models can have similar local-minimum loss values but substantially different test performance, and large-minibatch training can generalize poorly.
Method
Spectral norm regularization penalizes large spectral norms of neural-network weight matrices to obtain models less sensitive to input perturbations.
Results
Models trained with spectral norm regularization exhibit better generalizability than models trained with other baseline methods.
Takeaways & Limitations
Experimental comparisons indicate that insensitivity to test-data perturbations is an important factor for determining generalizability.
Takeaways & Limitations
The work leaves a theoretical understanding of spectral norm regularization’s effect on generalization as future work.
Abstract
from arXiv · showhide
We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spectral norm regularization, which penalizes the high spectral norm of weight matrices in neural networks. We provide supportive evidence for the abovementioned hypothesis by experimentally confirming that the models trained using spectral norm regularization exhibit better generalizability than other baseline methods.
1 Introduction
The paper frames deep-learning generalizability as poorly understood despite similar training losses, with practical concerns arising when large-batch synchronous SGD harms test performance. It proposes spectral norm regularization to reduce sensitivity to test-data perturbations and reports better generalizability than baseline methods.
- Deep networks can reach local minima with similar loss values yet differ significantly in test-data performance, leaving generalizability insufficiently understood.
- Large-minibatch synchronous SGD enables parallel training but has been reported to produce models with poor generalization.
- The paper studies generalizability through models’ sensitivity to input perturbations, seeking insensitivity to perturbations of test data.
- Spectral norm regularization penalizes large spectral norms in neural-network weight matrices to reduce sensitivity to unknown test-data perturbations.
- Experiments on several real-world datasets find better generalizability than other baseline methods and identify test-data perturbation insensitivity as a more important predictor than Hessian maximum eigenvalue.
2 Related Works
Related work commonly explains generalizability through parameter-space flatness or sharpness, while this paper emphasizes sensitivity to input-data perturbations. Prior work also questions whether classical complexity measures adequately explain deep-network generalizability.
- Flatness and sharpness characterize how much a loss function changes when model parameters are perturbed around a local minimum.
- Minimum description length theory relates flatter minima to better generalization through the number of bits required to describe statistical models.
- Large-minibatch SGD is associated with poorer generalization, while smaller minibatches tend to converge to flatter minima according to prior experiments.
- Unlike parameter-perturbation analyses, this paper considers sensitivity of the loss function to input-data perturbations; adversarial training perturbs training data to increase loss.
- Classical Rademacher complexity and VC dimension have been reported inadequate for understanding deep-neural-network generalizability.
3 Spectral Norm Regularization
Spectral norm regularization reduces sensitivity to input perturbations by controlling the spectral norms of neural-network weight matrices. The method is formulated as an empirical-risk objective and compared with weight decay, adversarial training, and Jacobian regularization.
- 3.1 General idea: For piecewise-linear networks, input sensitivity is governed by the spectral norm of the network’s input-output linear operator.A small operator spectral norm implies that the model is insensitive to input perturbations.
- 3.1 General idea: Bounding each layer weight matrix’s spectral norm suffices to bound the spectral norm of the network operator.The feed-forward network is represented as cascaded layer computations, with activation Jacobian factors having spectral norm at most one in the ReLU case.
- 3.2 Details of spectral norm regularization: Spectral norm regularization penalizes the squared spectral norms of layer weight matrices alongside the training loss.The regularization factor λ controls the added penalty, which decreases the spectral norms of the weight matrices.
- 3.2 Details of spectral norm regularization: The method estimates dominant singular values and vectors with power iteration during stochastic-gradient training.The procedure alternates matrix-vector and transpose-matrix-vector products; one iteration was adequate in the reported experiments, and convolutional weights are reshaped into matrices.
- 3.3 Comparison with other regularization methods: Unlike weight decay, spectral norm regularization focuses on the largest singular value rather than shrinking all singular directions.The paper contrasts this with Frobenius-norm regularization, which reduces the sum of squared singular values and can shrink weights in all directions.
- 3.3 Comparison with other regularization methods: Spectral norm regularization is contrasted with adversarial and Jacobian regularization as an approach to reducing sensitivity to input perturbations.The paper states that adversarial training targets training-data perturbations, while Jacobian regularization can be computationally expensive; a layer-wise alternative may coincide with weight decay when activation effects are neglected.
4 Experiments
Experiments across multiple datasets, architectures, batch sizes, and regularization methods show that spectral norm regularization improves test accuracy and generalization gaps. Its results support input-perturbation sensitivity as a useful predictor of generalizability, whereas Hessian maximum eigenvalues are not.
- Experimental setup: Experiments compare vanilla, weight decay, adversarial training, and spectral norm regularization across four model–dataset settings and two minibatch regimes.The settings include VGGNet/CIFAR-10, NIN/CIFAR-100, DenseNet/CIFAR-100, and DenseNet/STL-10, with batch sizes B = 64 and B = 4096.
- Test accuracy: Spectral achieves the best test accuracy in the large-batch regime for every model, while its accuracy decreases less with larger minibatches than the other methods.In the small-batch regime, decay and spectral outperform the other two methods.
- Generalization gap: Except for DenseNet on STL-10, spectral has the smallest generalization gap in every setting, followed by decay.The gap is defined as the minimum training–test accuracy difference when test accuracy exceeds a selected threshold.
- Generalization gap: For DenseNet on CIFAR-100, spectral maintains the smallest generalization gaps in both batch regimes, while decay’s gap increases substantially with batch size.The spectral gap does not increase significantly as the minibatch size grows.
- Input sensitivity: Test-data gradient ℓ2-norm tracks the generalization gap, whereas training-data gradient ℓ2-norm does not predict it well.The test-data norm and gap both increase during training, while the training-data norm decreases as the gap increases.
- Input sensitivity: Spectral suppresses only the spectral norm, preserving greater model complexity than decay while reducing input sensitivity and achieving better training fit.Its singular-value spectrum is flatter, and its largest singular value is low; spectral models consequently show less sensitivity to input perturbations.
- Hessian analysis: Spectral models can have the largest Hessian eigenvalues while achieving the best generalizability, indicating that this eigenvalue is not a suggestive generalizability measure.The experiment therefore favors test-data perturbation insensitivity over Hessian maximum eigenvalues as a predictor.
5 Conclusions
The paper hypothesizes that input sensitivity harms generalizability and proposes spectral norm regularization to reduce test-data sensitivity. Experiments support better generalizability than baseline methods, while theoretical understanding remains an open direction.
- Spectral norm regularization reduces sensitivity to test-data perturbations by constraining the spectral norms of neural-network weight matrices.
- Experiments show that spectral norm regularization exhibits better generalizability than other baseline methods.
- Comparisons indicate that insensitivity to test-data perturbations plays a crucial role in determining generalizability.
- A theoretical explanation of spectral norm regularization’s effect on generalization remains an open future direction.
A Accuracy
Across VGGNet, NIN, and DenseNet on STL-10, spectral achieves the smallest generalization gap or the best test accuracy in every setting.
- Spectral achieves the smallest generalization gap or the best test accuracy in every STL-10 setting.This reduces the generalization gap without significantly suppressing model complexity.
B Sensitivity to the perturbation of the input
The test-data gradient norm is well correlated with the generalization gap, whereas the training-data gradient norm does not predict it well for VGGNet and NIN. Figure 6 distinguishes small- and large-batch results with solid and dashed lines.
- The ℓ2-norm of the loss gradient with respect to test data is well correlated with the generalization gap.
- Figure 6 uses solid lines for the small-batch regime and dashed lines for the large-batch regime.
- For VGGNet and NIN, the training-data gradient norm does not predict the generalization gap well.