Source-linked AI summary
Visualizing the Loss Landscape of Neural Nets
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein
TL;DR
The paper addresses why neural networks with highly non-convex losses can be trainable and why different architectures and training choices produce different minimizer geometries. It develops visualization methods, especially filter normalization, and uses them to study loss landscapes across architectures and training settings. The results connect sharpness, depth, skip connections, trainability, and generalization in the examined networks.
Problem
The effects of architecture and training choices on neural loss-surface structure, trainability, and generalization are not well understood.
Method
The paper develops high-resolution loss-landscape visualizations, including filter-wise normalized directions, to compare minimizers and examine architecture and optimization effects.
Results
Filter-normalized sharpness correlates with generalization error; sufficient depth produces chaotic, poorly trainable landscapes, while skip connections promote flat minimizers and prevent this transition.
Takeaways & Limitations
Loss-landscape visualization offers an empirical way to connect neural-network design choices with trainability and generalization, supporting more general understanding of network structure.
Takeaways & Limitations
Apparent convexity in a low-dimensional visualization does not establish that the full high-dimensional loss function is convex.
Abstract
from arXiv · showhide
Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effects on the underlying loss landscape, are not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.
1 Introduction
The paper investigates why neural networks can optimize highly non-convex losses and how architecture, optimization choices, and minimizer geometry affect trainability and generalization. It introduces visualization methods to characterize these relationships, including filter normalization and trajectory visualizations.
- Neural-network trainability depends strongly on architecture, optimizer, initialization, and other choices whose effects on loss-surface structure remain unclear.
- The authors use high-resolution visualizations to study loss landscapes, architecture choices, trainability, and the relationship between minimizer geometry and generalization.
- Filter normalization enables meaningful side-by-side comparisons of minimizer sharpness across architectures and training methods, with sharpness correlating well with generalization error.
- Sufficient depth changes loss landscapes from nearly convex to highly chaotic, coinciding with a dramatic drop in generalization error and eventual lack of trainability.
- Skip connections promote flat minimizers and prevent the transition to chaotic behavior, helping explain their necessity for training extremely deep networks.
- Optimization trajectories occupy an extremely low-dimensional space that can be explained by large, nearly convex regions of the loss landscape.
2 Theoretical Background
Prior work studies neural-network optimization and the relationship between local-minimum sharpness or flatness and generalization. These analyses often rely on restrictive assumptions or measures whose invariance properties limit their interpretation.
- Theoretical optimization results commonly assume restricted sample distributions, nonlinearities, loss functions, or network classes.
- Flatness has been defined through the size of low-training-loss regions surrounding minima, while other work uses Hessian eigenvalues or neighborhood maximum loss.
- Quantitative sharpness measures may fail to determine generalization because network symmetries can change measured sharpness without changing the network.
3 The Basics of Loss Function Visualization
Loss visualization projects high-dimensional neural-network objectives into one- or two-dimensional plots. Linear interpolation is simple but can miss non-convexity and produce misleading sharpness comparisons, while contour methods provide broader surface views at substantial computational cost.
- Neural networks optimize losses over very high-dimensional parameter spaces, whereas visualizations are limited to one or two dimensions.
- One-dimensional interpolation plots the loss along the line between two parameter configurations using a scalar weighted average.
- One-dimensional plots make non-convexities difficult to visualize and can give misleading sharpness comparisons because they ignore batch normalization and network symmetries.
- Two-dimensional contour methods select a center point and direction vectors, then plot loss over a projected surface.
- Because two-dimensional plotting is computationally expensive, conventional methods often show low-resolution, small regions that miss complex non-convexity.
4 Proposed Visualization: Filter-Wise Normalization
Scale invariance can make loss-surface sharpness comparisons meaningless when perturbations are applied without accounting for filter magnitudes. The paper addresses this with filter-wise normalized directions, whose sharpness correlates with generalization error.
- Random-direction plots may fail to capture intrinsic loss-surface geometry and cannot compare different minimizers or networks because of scale invariance.
- For rectified networks, rescaling weights across layers can leave network behavior unchanged while altering apparent loss-surface sharpness.
- Filter-wise normalization rescales each direction filter to match the corresponding filter norm in the model parameters.
- The normalization applies to convolutional and fully connected layers, treating fully connected neuron-generating weights as filters.
- Filter-normalized sharpness correlates well with generalization error, whereas plots without normalization can be misleading.
5 The Sharp vs Flat Dilemma
Unnormalized visualizations can make batch-size effects appear to reverse because weight scaling distorts sharpness. Filter normalization removes this scaling artifact, revealing subtler sharpness differences that correlate with generalization error.
- Unnormalized interpolation plots suggest that small-batch minima are wide and large-batch minima sharp, but weight decay can reverse this apparent relationship.Small batches nevertheless generalize better in both experiments, so the unnormalized sharpness comparison does not track generalization.
- Weight histograms and norm trajectories show that batch size and weight decay change parameter scale, which batch normalization makes functionally irrelevant.Large-batch training yields smaller weights without weight decay but larger weights with it; weight decay shrinks weights more when smaller batches perform more updates per epoch.
- Filter-normalized plots remove geometry differences caused by weight scaling and expose more subtle sharpness differences between minimizers.The method evaluates each minimizer separately along random filter-normalized directions and supports side-by-side comparisons.
- Figure 3 visualizes one-dimensional and two-dimensional loss landscapes for SGD solutions under different weight-decay and batch-size settings, with test error reported in each subfigure title.
- With filter normalization, large-batch solutions appear sharper and have higher test error, while small-batch solutions have wider contours.The reported relationship is between visually sharper minima and higher generalization error, although the sharpness difference is not dramatic.
6 What Makes Neural Networks Trainable? Insights on the (Non)Convexity Structure of Loss Surfaces
The loss landscape becomes increasingly chaotic as networks deepen without skip connections, while skip connections and increased width preserve flatter, more convex regions. These geometric patterns coincide with trainability and generalization differences, although low-dimensional plots require cautious interpretation.
- The Effect of Network Depth: Without skip connections, increasing depth transforms the landscape from nearly convex to chaotic, with steep directions and increasingly sharp or ill-conditioned minimizers.ResNet-20-noshort is relatively benign, whereas ResNet-56-noshort and ResNet-110-noshort exhibit dramatic non-convexity.
- Shortcut Connections to the Rescue: Skip connections prevent the depth-related transition to chaotic behavior and preserve similarly shaped low-loss contours even from 20 to 110 layers.Their effect is most pronounced in deep networks; for shallow networks, the difference is comparatively small.
- Shortcut Connections to the Rescue: DenseNet shows no noticeable non-convexity, extending the observed stabilizing effect beyond residual skip connections.The comparison is made using the loss surfaces of ResNet-110-noshort and DenseNet on CIFAR-10.
- Wide Models vs Thin Models: Increasing width with k = 2, 4, or 8 produces flatter minima and wider apparently convex regions, while skip connections further widen minimizers.The wider models show no noticeable chaotic behavior compared with narrow CIFAR-optimized ResNets.
- Landscape Geometry Affects Generalization: Flatter and more convex landscapes consistently correspond to lower test error, whereas chaotic landscapes in deep networks without skip connections have worse training and test error.Wide-ResNets in the top row of Figure 6 generalize best and show no noticeable chaotic behavior.
- A note of caution: Are we really seeing convexity?: Low-dimensional plots can reveal non-convexity but cannot establish full-dimensional convexity, because apparent convexity may reflect dominant positive average curvature.The study therefore also maps |λmin/λmax| using Hessian eigenvalues to detect hidden negative curvature.
7 Visualizing Optimization Paths
Random projections obscure optimization trajectories because high-dimensional random directions are nearly orthogonal to their low-dimensional variation. PCA directions instead capture trajectory variation and reveal how stochasticity and learning-rate changes shape descent.
- Why Random Directions Fail: Random-direction projections capture almost none of the optimizer trajectory because random vectors are nearly orthogonal in high-dimensional spaces.The expected cosine similarity between Gaussian random vectors in n dimensions is roughly 2/(πn).
- Effective Trajectory Plotting using PCA Directions: PCA directions provide non-random trajectory visualizations and quantify the variation captured along each direction.The method applies PCA to parameter displacements from the final model and selects the two most explanatory directions.
- Trajectory Behavior: At early training stages, trajectories follow gradient directions, while later stochasticity is stronger with weight decay and small batches.These settings increase gradient noise and produce greater departures from deterministic descent directions.
- Trajectory Behavior: Large stepsizes can make noisy trajectories orbit solutions, whereas learning-rate drops reduce effective noise and redirect paths into nearby local minimizers.The learning-rate reduction is marked by a kink in the trajectory.
- Effective Trajectory Plotting using PCA Directions: 40% to 90% of descent-path variation lies in only 2 dimensions, consistent with movement dominated by nearby attractors and wide, nearly convex regions.This low dimensionality supports the presence of broad, structured regions in the loss landscape.
8 Conclusion
The paper presents visualization techniques for understanding how architecture, optimizer choice, and batch size affect neural-network loss landscapes. It argues that visualization, together with theory, may support faster training, simpler models, and better generalization.
- Conclusion: The visualization technique provides insights into consequences of network architecture, optimizer selection, and batch size.These choices are examined as factors affecting neural-network loss landscapes.
- Conclusion: The authors hope effective visualization combined with advances in theory will contribute to faster training, simpler models, and better generalization.This is presented as a goal for continued progress rather than an established outcome.
A.1 The Change of Weights Norm during Training
Weight decay changes how weight norms evolve during VGG-9 training, while fixed-epoch comparisons make update-count differences important when comparing batch sizes.
- Weight Norm Dynamics: Without weight decay, VGG-9’s weight norm grows steadily during training without constraints.The trajectory is shown over training epochs and iterations.
- Weight Norm Dynamics: With nonzero weight decay, the weight norm decreases rapidly at first and then stabilizes until the learning rate is decayed.The stabilization persists before the scheduled learning-rate reduction.
- Weight Norm Dynamics: Under a fixed number of epochs, differences between large-batch and small-batch weight-norm changes are mainly caused by the larger number of updates.The comparison therefore depends on both epochs and iterations.
A.2 Comparision of Normalization Methods
The appendix compares normalization schemes and visualization settings, emphasizing filter normalization for comparing minima, while documenting trajectory repeatability and implementation conditions across experiments.
- Normalization Methods: Filter normalization rescales each filter direction to match the corresponding filter’s weight norm, whereas layer normalization performs this matching at the layer level.No normalization adds the direction directly without processing.
- Normalization Methods: Filter normalization is the paper’s preferred approach for plotting loss surfaces.The authors use it extensively in their visualizations.
- Normalization Methods: Without normalization, the x-axis range must be chosen carefully and flatness does not consistently track generalization error.The enlarged plots use the range [-0.2, 0.2] rather than [-1.0, 1.0].
- Normalization Methods: Filter normalization is reported as more accurate than layer normalization for relating sharpness to generalization error.A layer-normalized example has a flatter minimum but worse generalization error.
- ResNet-56 Comparisons: ResNet-56 visualizations compare optimizer, batch size, and weight decay, including zero and 5e-4 weight-decay settings.The corresponding test errors are reported in Table 1.
- Repeatability: Ten random filter-normalized directions produce closely shaped VGG-9 plots, while ResNet-56-noshort plots vary but retain consistent qualitative chaotic behavior.The ResNet-56-noshort example reports final training loss 0.192, training error 6.49, and test error 13.31.
- Implementation Details: Batch-normalization running statistics must be included in linear interpolation parameters, but filter normalization removes the effect of weight scaling.This permits batch-normalization parameters to be ignored in filter-normalized visualizations.