Source-linked AI summary
Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, Michael W. Mahoney
TL;DR
Large-batch neural-network training can lose accuracy, while the reasons for this behavior remain incompletely understood. The paper analyzes true Hessian spectra and robust optimization, finding that large batches reach higher-spectrum regions and are more vulnerable to adversarial attacks, whereas robust training favors flatter regions.
Problem
Large-batch training can incur accuracy loss, and the paper asks how batch size changes convergence geometry and adversarial robustness.
Method
The paper backpropagates second derivatives to compute the true Hessian spectrum and studies its relationship with large-batch training and white-box adversarial attacks.
Results
Large-batch training reaches areas with higher Hessian spectra and is more prone to adversarial attacks, while robust training favors smaller spectra; the robust inner loop is saddle-free almost everywhere.
Takeaways & Limitations
Optimization hyperparameters such as batch size can substantially affect network robustness, linking robust training with flatter regions of the Hessian landscape.
Takeaways & Limitations
The paper does not propose a solution that fully resolves large-batch training.
Abstract
from arXiv · showhide
Large batch size training of Neural Networks has been shown to incur accuracy loss when trained with the current methods. The exact underlying reasons for this are still not completely understood. Here, we study large batch size training through the lens of the Hessian operator and robust optimization. In particular, we perform a Hessian based study to analyze exactly how the landscape of the loss function changes when training with large batch size. We compute the true Hessian spectrum, without approximation, by back-propagating the second derivative. Extensive experiments on multiple networks show that saddle-points are not the cause for generalization gap of large batch size training, and the results consistently show that large batch converges to points with noticeably higher Hessian spectrum. Furthermore, we show that robust training allows one to favor flat areas, as points with large Hessian spectrum show poor robustness to adversarial perturbation. We further study this relationship, and provide empirical and theoretical proof that the inner loop for robust training is a saddle-free optimization problem \textit{almost everywhere}. We present detailed experiments with five different network architectures, including a residual network, tested on MNIST, CIFAR-10, and CIFAR-100 datasets. We have open sourced our method which can be accessed at [1].
1 Introduction
The paper revisits why large-batch training can converge poorly, analyzing its local loss geometry and connection to adversarial robustness. It argues that large batches reach higher-spectrum regions rather than being primarily hindered by saddle points.
- Motivation: Large-batch training is studied because it enables data parallelism but can produce poor convergence and test performance across some networks and datasets.Existing methods to avoid large-batch performance loss do not work universally.
- Research questions: The paper compares large- and small-batch training by examining differences in the local geometry surrounding their converged models.It also connects batch size with robustness to adversarial perturbations.
- Approach: Second-derivative backpropagation computes the true Hessian spectrum during training, avoiding the approximate curvature metric used in earlier work.The spectrum is analyzed across eigenvalues rather than only the dominant eigenvalue.
- Findings: Large batches progressively become trapped in regions with noticeably larger Hessian spectra, while saddle points are not identified as the cause of their training problem.The analysis considers batch sizes up to the gradient-descent limit.
- Robustness: Robust optimization favors areas with small spectra, and large-batch solutions are significantly more prone to adversarial attacks than small-batch solutions.The paper reports empirical and theoretical evidence that the robust-training inner loop is saddle-free almost everywhere.
- Limitations: The main limitation is that the paper provides analytical insights but does not propose a solution that fully resolves large-batch training.Experiments were repeated across multiple datasets and models, including a residual network.
2 Large Batch, Generalization Gap and Hessian Spectrum
Direct Hessian-spectrum analysis shows that increasing batch size leads to convergence in regions of higher curvature, rather than saddle-point trapping. These higher-curvature regions are visible in both loss measurements and parameter-space visualizations.
- After a certain point, increasing batch size degrades test-dataset performance for C1 on CIFAR-10 and M1 on MNIST.All batches were trained under the same conditions until reaching a training loss of 0.001 or better.
- The C1 loss landscape is visualized by changing model parameters along the first two dominant Hessian eigenvectors with perturbation magnitudes ϵ1 and ϵ2.
- Large-batch models converge to points with noticeably larger Hessian spectra, including both the dominant eigenvalue and the remaining 19 eigenvalues.The experiments compute the true Hessian spectrum directly and compare models trained with different batch sizes.
- Perturbing parameters along the dominant Hessian eigenvector shows that large-batch models occupy higher-curvature regions for both training and test losses.The paper also visualizes the C1 landscape by perturbing parameters along the first two dominant Hessian eigenvectors.
- Large-batch training is progressively attracted to areas with larger Hessian spectra but does not get stuck in saddle points.This conclusion is based on tracking the dominant Hessian eigenvalue together with the total gradient during training.
3 Large Batch, Adversarial Attack and Robust training
The paper connects adversarial attacks and robust training to Hessian geometry, showing that large-batch models are more vulnerable while robust optimization favors flatter regions.
- Robust optimization and adversarial attack: Adversarial attacks seek bounded input perturbations that maximize the model loss, while robust training approximately solves the resulting min-max problem.The inner maximization is approximated with one or more gradient ascents because exact perturbation search is infeasible at every iteration.
- Robust optimization and adversarial attack: The adversarial-perturbation inner problem is saddle-free almost everywhere under strictly ReLU networks with convolutional or fully connected layers, enabling Newton-CG methods.The corresponding input Hessian is positive semidefinite almost everywhere under the stated assumptions.
- Large-batch training and robustness: Large-batch models are considerably more prone to adversarial attacks than small-batch models across the reported experiments.Adversarial performance is measured as the fraction of correctly classified perturbed inputs for ε = 0.02 and ε = 0.01 under the L∞ norm.
- Large-batch training and robustness: The weight-Hessian spectrum does not generally determine input robustness, although the neural-network experiments show a strong empirical association between larger spectrum and weaker robustness.A least-squares example demonstrates that weight and input Hessians can vary independently, while some reported network metrics did not correlate with adversarial accuracy.
- Adversarial training and Hessian spectrum: Adversarial training improves robustness on adversarial datasets but can reduce clean-data accuracy.For example, FGSM-trained models achieved 63.46% against L2Grad versus 14.32% for the original MORI model in the cited MNIST result.
- Adversarial training and Hessian spectrum: Robust optimization drives training toward points with smaller Hessian spectrum, providing a systematic way to bias optimization away from sharp minima.The paper reports preliminary evidence of better large-batch performance but emphasizes that more extensive tests are needed before claiming superiority over other methods.
4 Conclusion
The paper links large-batch training to higher Hessian spectra and greater vulnerability to adversarial attacks, while robust training favors smaller spectra. It also finds that adversarial perturbation search is saddle-free almost everywhere under the stated criterion.
- Large-batch training tends to converge to areas with higher Hessian spectrum.
- Large-batch models are more prone to adversarial attacks across multiple datasets.
- Robust training favors areas with noticeably smaller Hessian spectrum than large-batch training.
- The Hessian with respect to inputs does not directly correlate with model robustness.
- Under the stated criterion, the adversarial-perturbation problem is saddle-free almost everywhere because the input Hessian is positive semidefinite.
A.1 Proof of Theorem 1
The proof explains why the input-space Hessian can be positive semidefinite almost everywhere for networks with linear or ReLU layers before softmax. This establishes saddle-free adversarial-perturbation optimization without implying input convexity.
- The resulting Hessians are positive semidefinite almost everywhere, making the adversarial-perturbation problem saddle-free.
- Positive semidefinite input Hessians do not make these ReLU networks convex with respect to inputs because ReLU causes discontinuities.
- The chain rule decomposes the input Hessian through derivatives of the loss and softmax-layer input.
- For networks with linear or ReLU layers before softmax, the relevant second derivatives with respect to the input vanish almost everywhere.
- The cross-entropy Hessian with softmax structure contributes a matrix whose rank is at most the number of classes.
A.2 Attacks Mentioned in Paper
This section points to the paper’s attack definitions and directs readers to Table 5 for their details.
- The attacks used in the paper are detailed in this section and summarized in Table 5.
- The paper’s attack definitions are provided in Table 5.
A.3 Models Mentioned in Paper
This section describes the notation used for neural-network architectures and directs readers to Table 6 for the model definitions.
- The paper denotes convolution, max-pooling, fully connected, and softmax layers using compact architectural notation.
- ReLU activations are omitted from the written architecture descriptions but implemented in practice.
- Convolution layers use padding to avoid dimension reduction.
- The definitions of all models used in the paper are provided in Table 6.
A.4 Discussion on Second Order Method
On CIFAR-10, second-order adversarial attacks are generally weaker than gradient-based variants, while their robustness behavior is linked to limitations of the input Hessian.
- Second-order adversarial attacks perform well on MNIST but are weaker than gradient-based variants in most CIFAR-10 experiments.Second-order-trained robust models are also more vulnerable on CIFAR-10, particularly MF HSM and ML2HESS.
- The CIFAR-10 input Hessian is low rank, with rank at most ten despite having a 3K × 3K matrix representation.This low-rank structure creates numerical problems for the inexact Newton and Conjugate Gradient procedure.
- The input Hessian does not directly correlate with network robustness, so maximizing loss may be less effective than perturbing inputs toward the decision boundary.
A.5 More Numerical Result for §2 and 3
Additional experiments reproduce the paper’s main patterns: large-batch training reaches sharper regions, while robust optimization favors flatter regions and can improve large-batch performance.
- A.5 More Numerical Result for §2 and 3: Increasing λθ1 coincides with decreasing adversarial accuracy for fixed ϵ.
- A.5 More Numerical Result for §2 and 3: The figures compare loss landscapes along dominant Hessian eigenvectors across training, batch sizes, datasets, and model architectures.They include MNIST and CIFAR-10 examples, with one- and two-eigenvector perturbations.
- A.5 More Numerical Result for §2 and 3: Robust models converge to regions with smaller curvature in the plotted loss landscapes.
- A.5 More Numerical Result for §2 and 3: For the C1 model, robust optimization achieves better performance than baseline large-batch training, although more extensive testing is needed.
- A.5 More Numerical Result for §2 and 3: The dominant Hessian eigenvalue increases during training, while robust optimization pushes training toward flatter areas.