PaperlayerView exact revision on arXiv

Source-linked AI summary

PyHessian: Neural Networks Through the Lens of the Hessian

Zhewei Yao, Amir Gholami, Kurt Keutzer, Michael Mahoney

arXiv:1912.07145v3cs.LGmath.NA

TL;DR

The paper addresses the limited scalability of Hessian-based analysis for deep neural networks and introduces PYHESSIAN as a practical framework for studying curvature and architectural effects. Its experiments show that BN and residual connections do not have uniformly smoothing effects: BN's relationship with loss-landscape sharpness depends on depth, while removing residual connections slightly coarsens the landscape.

  • Problem

    Hessian-based studies often test small or simple networks, leaving the effects of architectural components on measurable loss-landscape properties insufficiently characterized for deep models.

  • Method

    PYHESSIAN uses scalable approximate Hessian computations to obtain top eigenvalues, the trace, and eigenvalue spectral density for analyzing neural-network behavior.

  • Results

    BN does not necessarily smooth the loss landscape: removing it flattens the spectrum in shallow ResNet20 but produces sharper minima in deeper ResNets, while removing residual connections slightly increases Hessian-spectrum measures.

  • Takeaways & Limitations

    Loss-landscape conclusions about BN and residual connections should be evaluated with depth and multiple Hessian-spectrum measures rather than treated as universal architectural effects.

Abstract

from arXiv · show

We present PYHESSIAN, a new scalable framework that enables fast computation of Hessian (i.e., second-order derivative) information for deep neural networks. PYHESSIAN enables fast computations of the top Hessian eigenvalues, the Hessian trace, and the full Hessian eigenvalue/spectral density, and it supports distributed-memory execution on cloud/supercomputer systems and is available as open source. This general framework can be used to analyze neural network models, including the topology of the loss landscape (i.e., curvature information) to gain insight into the behavior of different models/optimizers. To illustrate this, we analyze the effect of residual connections and Batch Normalization layers on the trainability of neural networks. One recent claim, based on simpler first-order analysis, is that residual connections and Batch Normalization make the loss landscape smoother, thus making it easier for Stochastic Gradient Descent to converge to a good solution. Our extensive analysis shows new finer-scale insights, demonstrating that, while conventional wisdom is sometimes validated, in other cases it is simply incorrect. In particular, we find that Batch Normalization does not necessarily make the loss landscape smoother, especially for shallower networks.

I. INTRODUCTION

The paper introduces PYHESSIAN to make scalable Hessian analysis practical for deep neural networks and applies it to residual connections and Batch Normalization. Its findings show that architectural effects on loss-landscape smoothness depend on network depth, challenging blanket claims about BN and residual connections.

  • Contributions: PYHESSIAN provides direct, efficient computation of top Hessian eigenvalues, the Hessian trace, and the full eigenvalue spectral density.The framework uses approximate numerical and randomized numerical linear-algebra techniques and supports distributed execution.
  • Findings: Removing BN flattens the Hessian spectrum in shallower ResNet20, contrary to the common belief that BN universally smooths the loss landscape.The paper reports this behavior on both Cifar-10 and Cifar-100, while observing the conventional smoothness pattern only in deeper models.
  • Findings: For deeper ResNets, removing BN leads to sharper local minima, with the effect becoming more pronounced as model depth increases.The reported comparisons include ResNet32/38 and connect the sharper minima with the Hessian spectrum.
  • Findings: Removing residual connections slightly increases the top eigenvalue, Hessian trace, and ESD support range across both shallow and deep models.The increase is reported consistently for ResNet20/32/38/56.
  • Findings: BN is generally more important in final ResNet stages than earlier stages, and removing it from the last stage significantly degrades testing performance.The degradation is reported as strongly correlated with the Hessian trace.
  • Motivation: Prior Hessian-analysis studies often used small or simple networks, limiting how representative their conclusions were for practical workloads.The paper identifies the lack of a scalable, programmable framework as a main reason for this restriction.

III. METHODOLOGY

The methodology formulates supervised learning as empirical-loss minimization over model parameters and then uses Hessian statistics to analyze model behavior. The specified setup defines the loss, architecture, data size, and learnable parameters used in that analysis.

  • Objective: Supervised learning is formulated as minimizing the average loss over N training examples with respect to the learnable parameter vector θ.The model architecture is represented by M, and each example contributes l(M(x_i), y_i, θ) to the objective.
  • Notation: The formulation treats θ as the learnable weight parameter, l as the loss function, M as the neural-network architecture, and N as the training-data size.The input pair is denoted by (x, y).

A. Neural Network Hessian Matvec

The Hessian is usually too large to form explicitly for neural networks, but its matrix-vector product can be computed efficiently. This Hessian matvec enables scalable estimation of eigenvalues and other spectral properties.

  • Hessian matrix-vector products: For a neural network with m parameters, the Hessian is an m×m second-derivative matrix that is often infeasible to form explicitly.The gradient is a vector, while the second derivative with respect to model parameters is the Hessian matrix.
  • Hessian matrix-vector products: Applying the Hessian to a random vector v provides an oracle for extracting second-order spectral information without constructing the full matrix.This matrix-free approach is central to scalable Hessian analysis.
  • Hessian matrix-vector products: A Hessian matvec costs about the same as one gradient backpropagation.The equality follows from differentiating the gradient-vector product and using the Hessian definition.
  • Hessian matrix-vector products: Power iteration can use the matvec oracle to compute the top k Hessian eigenvalues.The framework additionally computes the Hessian trace and eigenvalue spectral density because top eigenvalues may not represent the full landscape.
  • Hessian trace estimation: Hutchinson’s method estimates the Hessian trace from repeated random-vector quantities E[vT Hv], each requiring a Hessian matvec and a dot product.The random vectors can use Rademacher or standard Gaussian entries.

C. Full Eigenvalue Spectral Density

The full Hessian eigenvalue spectral density is approximated through successive kernel, quadrature, Lanczos, and sampling steps. Stochastic Lanczos Quadrature makes this finer-grained spectral analysis computationally tractable.

  • Spectral-density construction: The full empirical spectral density provides finer-grained information about Hessian eigenvalues than the top eigenvalues or the trace.The density is smoothed with a Gaussian kernel before approximation.
  • Spectral-density construction: The ideal ESD is a distribution over Hessian eigenvalues, with λ_i denoting the ith eigenvalue in descending order.The Gaussian-kernel density converges toward the unsmoothed density as σ approaches zero.
  • Spectral-density construction: The ESD computation applies four approximations: Gaussian smoothing, Gaussian quadrature, stochastic Lanczos estimation, and finite-sum approximation.These steps progressively replace unknown eigenvalue-dependent quantities with computable estimates.
  • Computational challenge: Computing the smoothed trace separately for every t remains intractable because the number of evaluation points scales with the number of model parameters.The Lanczos-based relaxation addresses this repeated-trace bottleneck.
  • Stochastic Lanczos Quadrature: Stochastic Lanczos Quadrature estimates the spectral density using approximate eigenpairs and weights obtained from Lanczos iterations over random vectors.For each run, a normalized random vector produces a tridiagonal matrix whose approximate eigenpairs contribute to the estimate.

IV. RESULTS

The experiments evaluate how Batch Normalization and residual connections affect the Hessian spectrum throughout SGD training. They examine top eigenvalues, trace, and full ESD across multiple ResNet architectures and datasets.

  • Experimental scope: The study measures top Hessian eigenvalues, Hessian trace, and full ESD throughout training with SGD and momentum.These metrics are evaluated as curvature descriptors during optimization.
  • Experimental scope: Experiments compare ResNet architectures with and without Batch Normalization or residual connections across multiple depths.The results section separates analysis of the entire model from analysis of individual ResNet stages.
  • Experimental scope: The experimental analysis focuses on how architectural components change the Hessian spectrum during training.The supplied results-section passage describes the study design but does not report a specific outcome.

A. Experimental Setting

The experiments evaluate ResNet variants across depths on Cifar-10 and Cifar-100, selecting the best result across initial learning rates and tracking spectra throughout training. Removing Batch Normalization harms accuracy increasingly with depth, while removing residual connections causes a smaller degradation.

  • Model and dataset setup: The study evaluates ResNet20/32/38/56 and variants with or without Batch Normalization and residual connections on Cifar-10 and Cifar-100.The Cifar-100 experiments produced observations consistent with those on Cifar-10.
  • Training protocol: Each model is trained with multiple initial learning rates, and the best-performing result is selected for analysis.The Hessian spectrum is examined at all training checkpoints.
  • Observed accuracy effects: Removing Batch Normalization lowers accuracy, with larger degradation in deeper models; ResNet−BN 56 cannot be trained at all.The table reports this pattern for Cifar-10, with Cifar-100 results provided separately.
  • Observed accuracy effects: Removing residual connections also causes slight performance degradation.This effect is smaller than the reported degradation from removing Batch Normalization.

B. Full Network Hessian Analysis

Full-network Hessian analysis shows that Batch Normalization affects curvature differently by depth, while removing residual connections consistently sharpens the landscape. These effects are reflected in Hessian spectra, loss-landscape visualizations, and generalization behavior.

  • Batch Normalization: Removing BN makes training harder, and ResNet56 on Cifar-10 could not be trained without BN even after hyperparameter tuning.The reported training difficulty does not imply that the Hessian spectrum must exceed the baseline in every depth regime.
  • Batch Normalization: Removing BN produces depth-dependent curvature: shallower ResNet20 has a flatter spectrum, whereas deeper models develop larger Hessian traces and eigenvalue ranges.For ResNet20, training initially concentrates around many small or degenerate Hessian directions; deeper models show substantially larger spectra without BN.
  • Batch Normalization: For ResNet20, removing BN converges to a flatter local minimum than the BN baseline, contrary to the claim that BN always smooths the loss landscape.For ResNet38, removing BN instead converges to a point with higher loss and non-degenerate directions.
  • Residual Connections: Residual-connection removal causes slightly poorer generalization, but its accuracy degradation is smaller than that caused by removing BN.The paper reports the same broad curvature behavior across shallow and deep models and across Cifar-10 and Cifar-100.
  • Residual Connections: Removing residual connections consistently increases the Hessian trace and, for deeper models, expands the top eigenvalues and ESD support range.The corresponding loss-landscape converging point becomes sharper as depth grows.

C. Stage-wise Hessian Analysis

Stage-wise analysis finds that BN is more important in later ResNet stages, while residual-connection removal has a smaller curvature and accuracy effect. The third stage is especially consequential for both Hessian trace and generalization.

  • Batch Normalization by Stage: Removing BN from the last ResNet stage increases the Hessian trace more rapidly than removing it from the first or second stage.The stage-wise comparison is reported for ResNet32 on Cifar-10, with related plots for other depths.
  • Batch Normalization by Stage: Removing BN from the third stage causes a larger accuracy drop than removing it from earlier stages, matching the stage-wise Hessian-trace pattern.A similar trend is reported for other models and on Cifar-100.
  • Batch Normalization by Stage: Stages that strongly affect accuracy also exhibit substantial increases in Hessian trace.The stage definition groups blocks with the same activation resolution.
  • Residual Connections by Stage: Removing residual connections produces a relatively smaller Hessian-trace increase and smaller accuracy impact than removing BN.Table III reports accuracy for baseline ResNet models and architectures with residual connections removed at different stages.

D. Summary of Results

The paper develops PYHESSIAN for scalable Hessian analysis and applies it to neural-network architecture changes. Results show that BN and residual connections affect loss-landscape curvature in distinct ways.

  • PYHESSIAN computes top Hessian eigenvalues, the Hessian trace, and the full Hessian eigenvalue spectral density without forming the full Hessian.It supports distributed-memory execution on cloud and supercomputer systems.
  • The framework analyzes how architectural changes, including BN and residual connections, affect neural-network loss landscapes.
  • Adding BN does not necessarily produce a smoother loss landscape: the effect appears in deeper models but not shallower ones.Without BN, deeper models converge to sharp local minima with high training loss and poor generalization.
  • Removing residual connections produces a slightly coarser loss landscape, consistently supported by the three Hessian spectrum metrics.

APPENDIX

The appendix documents additional experiments, computational procedures, training settings, and accuracy results for the ResNet variants. These results show performance degradation when BN or residual connections are removed.

  • The appendix extends the main-text results with additional figures, tables, algorithms, and experimental details.
  • ResNet20 is illustrated with three stages, while the appendix provides pseudo-code for power iteration, Hutchinson trace estimation, and stochastic Lanczos quadrature.
  • The experiments train ResNet, ResNet−BN, and ResNet−Res for 180 epochs using SGD with momentum and scheduled learning-rate decay.Cifar-10 uses five initial learning rates, while Cifar-100 uses ten.
  • Removing BN or residual connections causes performance degradation on Cifar-100 across the reported ResNet depths.
  • Testing curves show much worse generalization without BN and a similar but substantially smaller loss after removing residual connections.

D. Loss Landscape Details

The loss-landscape plots evaluate parameter perturbations along the Hessian’s two dominant eigenvectors and compute loss over 4,096 data points.

  • Loss landscapes are plotted by perturbing model parameters along the first two dominant Hessian eigenvectors, v1 and v2.The loss is computed over K = 4096 data points.

E. Extra Results

The appendix adds stage-wise accuracy and Hessian analyses across datasets, depths, and architectural variants. The results reinforce that BN and residual connections can change curvature and generalization in depth-dependent ways.

  • Additional Cifar-100 tables report baseline accuracy and architectures with BN removed from individual stages or residual connections removed at different stages.
  • Removing BN from the third stage significantly increases the Hessian trace relative to removing it from the first or second stage, correlating with final generalization performance.
  • On Cifar-100, removing BN rapidly increases the whole-network Hessian trace, while removing residual connections produces a sharper loss landscape throughout training.
  • Without BN, the Hessian ESD centers around zero during early training in the examined depths, indicating harder training in the reported comparisons.
  • Removing residual connections produces a sharper loss landscape for depth 56 and slightly increases Hessian curvature measures across the reported depths.
  • For depth 20, removing BN can yield a smoother loss landscape than standard ResNet, whereas for depths 32 and 38 it yields a sharper landscape.
Loading 1912.07145v3…