PaperlayerView exact revision on arXiv

Source-linked AI summary

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson

arXiv:1802.10026v4stat.MLcs.AIcs.LG

TL;DR

DNN loss surfaces are poorly understood, with optima that appear isolated despite high-dimensional non-convexity. The paper finds simple near-constant-accuracy paths between modes, develops a procedure to discover them, and introduces FGE, which achieves state-of-the-art results across CIFAR-10, CIFAR-100, and ImageNet.

  • Problem

    DNN loss surfaces are highly non-convex and poorly understood, with high loss along line segments between optima suggesting that modes are isolated.

  • Method

    The paper optimizes parameterized curves between independently trained networks and uses the resulting mode connectivity to construct Fast Geometric Ensembling.

  • Results

    FGE achieves state-of-the-art results on CIFAR-10, CIFAR-100, and ImageNet, including a 0.56% top-1 error-rate improvement on ImageNet after 5 epochs.

  • Takeaways & Limitations

    Deep-network optima can be viewed as connected by simple low-loss pathways rather than only as isolated modes, enabling efficient high-performing ensembles.

  • Takeaways & Limitations

    FGE averages networks with lower diversity and slightly lower individual performance than fully trained independent networks, but can produce more high-performing networks for a fixed computational budget.

Abstract

from arXiv · show

The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these high-accuracy pathways between modes. Inspired by this new geometric insight, we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in the time required to train a single model. We achieve improved performance compared to the recent state-of-the-art Snapshot Ensembles, on CIFAR-10, CIFAR-100, and ImageNet.

1 Introduction

Although DNN loss surfaces appear highly non-convex and optima can seem isolated, the paper finds simple near-constant-accuracy paths connecting independently trained modes. This geometric insight motivates a new ensembling procedure.

  • DNN loss surfaces can have exponentially many local optima and high loss along line segments between optima, suggesting isolated modes.
  • A training procedure discovers near-constant-loss paths between modes, including one-bend polygonal chains across modern DNN architectures.The paths are demonstrated using affine-combination planes of three weight vectors.
  • Figure 1 contrasts isolated optima in one plane with quadratic Bezier and one-bend polygonal paths connecting two lower optima.The direct linear paths between modes incur high loss.
  • The geometric discovery motivates Fast Geometric Ensembling, which seeks multiple high-performing but diverse networks efficiently.
  • The paper reports improved performance over Snapshot Ensembles on CIFAR-10 and CIFAR-100, plus a 0.56% top-1 error-rate improvement on ImageNet after 5 epochs of FGE.

2 Related Work

Prior work studies both local minima structure and global loss geometry, but existing methods face theoretical or empirical limitations. This paper introduces a simpler procedure for finding high-accuracy paths and uses the resulting insight for fast ensembling.

  • Local and global loss structure: Loss-surface research examines local minima properties and global structure, including how neural networks overcome poor local optima.Local studies contrast sharp and wide minima, while global studies analyze relationships among optima and loss landscapes.
  • Local minima: Flat minima have been linked to strong generalization, but later work argues that common flatness notions cannot directly explain generalization.Sharp minima have instead been associated with poor test results in the cited prior work.
  • Global loss structure: A theoretical curve-connectivity result for one-hidden-layer ReLU networks does not readily generalize to multilayer networks.The bound depends on network parameter count and data smoothness.
  • Prior path-finding methods: A dynamic-programming method constructs polygonal chains for MNIST and PTB but struggles to reach 80% test accuracy for AlexNet on CIFAR-10 and omits ensembling.These limitations motivate a simpler approach for modern architectures.
  • This paper: The proposed procedure finds near-constant-accuracy polygonal chains with one bend, while Fast Geometric Ensembling trains ensembles in single-model time and reaches 96.4% accuracy on CIFAR-10.Experiments compare the geometric approach with Snapshot Ensembles and report improved performance on CIFAR-10, CIFAR-100, and ImageNet.

3 Finding Paths between Modes

The paper learns a parametric path between two independently trained network modes by minimizing loss along the curve. It uses a tractable uniform-in-parameter objective and considers polygonal chains and Bezier curves as path parameterizations.

  • General procedure: The method minimizes training error along a parametric curve connecting two independently trained weight vectors.The curve is continuous and piecewise smooth, with its endpoints fixed at the two networks’ weights.
  • Curve objective: The ideal objective averages loss uniformly along the curve, but its parameter-dependent sampling distribution makes stochastic gradients generally intractable.The curve-uniform distribution is normalized by the path length.
  • Tractable objective: A tractable alternative averages L(φθ(t)) over t sampled uniformly from [0, 1], rather than uniformly over the curve.The two objectives coincide for a two-segment polygonal chain with equal-length segments and linear segment parameterizations.
  • Optimization: Optimization repeatedly samples t from U(0, 1), takes a gradient step on L(φθ(t)), and repeats until convergence.This procedure yields unbiased gradient estimates for the tractable objective.
  • Curve parameterizations: A one-bend polygonal chain uses the trained networks as endpoints and the bend as the curve parameter.The paper also considers smooth quadratic Bezier curves with the same endpoints and generalizes the parameterizations to multiple bends.

4 Curve Finding Experiments

The proposed procedure finds high-accuracy paths between independently trained modes across several architectures and datasets. These paths maintain nearly constant train loss and test error, unlike direct line segments, and can yield diverse representations.

  • The experiments cover VGG-16, Wide ResNet, ResNet, fully connected, and recurrent networks on CIFAR-10 and CIFAR-100.
  • For each model and dataset, two independently initialized networks are connected using quadratic Bezier curves, one-bend polygonal chains, and direct line segments.
  • Train loss and test error remain nearly constant along both Bezier curves and polygonal chains for ResNet-164 on CIFAR-100, with similar trends across other tested models.
  • Different random seeds produce nonunique connecting curves: VGG-16 on CIFAR-10 has turning points separated by 29.6 versus endpoint distance 50.
  • Direct line segments can reach 90% test error for VGG-16, 80% for ResNet-158, and 66% for Wide ResNet-28-10.
  • A point around t ≥ 0.4 along a connecting curve ensembles with its endpoint as effectively as the two independently trained endpoints, indicating meaningfully different predictions.

5 Fast Geometric Ensembling

Fast Geometric Ensembling uses short cyclical learning-rate steps to collect diverse networks near a low-error region without explicitly constructing a connecting curve. It avoids pre-trained endpoints and fits within the time required to train one network.

  • Fast Geometric Ensembling (FGE) collects diverse networks using cyclical learning rates motivated by mode connectivity.
  • FGE does not require explicitly finding a connecting curve or pre-trained endpoints, so it can be trained in the time required to train a single network.
  • FGE saves checkpoints when the cyclical learning rate reaches its minimum and ensembles the collected models after training.
  • The learning-rate cycle alternates exploration with larger steps and exploitation with smaller steps, typically using cycles of 2 to 4 epochs.
  • A standard learning-rate schedule first trains the model for about 80% of single-model training time before FGE uses the remaining computational budget.
  • FGE uses smaller weight-space steps than Snapshot Ensembles: about 7 versus 40 for Preactivation-ResNet-164 on CIFAR-100.

6 Fast Geometric Ensembling Experiments

The experiments evaluate FGE against independent and Snapshot ensembles under matched computational budgets, finding especially strong results on CIFAR-100 and consistent gains over Snapshot Ensembles. Additional ImageNet experiments show that FGE improves a pretrained ResNet-50 within five epochs.

  • Experimental setup: Under matched computational budgets, the study compares independent, FGE, and Snapshot ensembles across CIFAR-100 and CIFAR-10.The methods use the same runtime per epoch, so kB budgets entail the same total computation for all approaches.
  • CIFAR experiments: FGE outperformed Snapshot Ensembles in all conducted experiments, particularly as the computational budget increased.On CIFAR-100, FGE outperformed all other methods across experiments; on CIFAR-10, it consistently improved upon Snapshot Ensembles.
  • CIFAR experiments: FGE’s advantage over independent training was most noticeable on CIFAR-100, where it outperformed all other methods in every experiment.On CIFAR-10, FGE improved against independent training with VGG but was more similar to independent training with ResNets.
  • CIFAR experiments: 22 epochs of FGE gathered sufficiently diverse networks to outperform Snapshot Ensembles at both one- and two-training-budget settings for ResNet-164 on CIFAR-100.The training budget B was 150 epochs, and FGE was applied after 125 epochs of standard independent training.
  • Diversity and trade-offs: FGE networks had lower prediction diversity and slightly lower individual performance than independently trained networks, but FGE produced more high-performing networks for a fixed budget.On CIFAR-100, prediction disagreement was 14.57% for two FGE networks versus 19.97% for two independent networks; individual accuracy was 78.0% versus 78.5%.
  • ImageNet: 23.31 top-1 test error was achieved by a four-model FGE ensemble on ImageNet after five epochs, improving a pretrained model’s 23.87 error by 0.56.This result was comparable to the reported 23.33 error from Snapshot Ensembles using ResNet-50.

7 Discussion and Future Work

The paper concludes that deep-network optima are connected by simple, near-constant-accuracy pathways rather than behaving as isolated modes. This insight motivates FGE and suggests future applications to Bayesian inference and optimization.

  • Discussion: The optima of deep neural networks are connected by simple pathways, including polygonal chains with a single bend, with near-constant accuracy.The paper introduced a training procedure that finds pathways using a user-specified curve choice.
  • Discussion: The geometric discovery motivates FGE, which achieves state-of-the-art results on CIFAR-10, CIFAR-100, and ImageNet.FGE is presented as a practical ensembling approach inspired by mode connectivity.
  • Future work: The authors propose viewing low-loss regions as valleys connecting optima rather than as isolated modes.This reframing is presented as a direction for thinking about the structure of complex neural-network loss surfaces.
  • Future work: These pathways could support approximate Bayesian methods that move between modes and optimization procedures targeting convergence, stability, and accuracy.The paper specifically mentions stochastic MCMC, variational-inference proposals, and SGD-related procedures.

A Supplementary Material

The supplementary material contains complexity analysis, batch-normalization details, curve formulas, and additional curve-finding experiments.

  • Supplementary material: The supplement discusses computational complexity, test-time batch normalization, formulas for polygonal and Bezier curves, and further curve-finding experiments.It also includes a table summarizing the path-finding experiments.

A.1 Computational complexity of curve finding

Curve finding adds complexity from evaluating the curve and applying its Jacobian, while the network’s ordinary forward and backward passes remain unchanged. For the one-bend family, the total additional complexity is O(|net|), with an observed epoch-time increase usually below 50%.

  • Method: The forward pass computes a curve point φ_θ(t), then evaluates the DNN at that point; the backward pass computes its gradient and multiplies by the curve Jacobian.The DNN-specific steps are exactly the same as in single-model training.
  • Curve parametrization: A one-bend curve uses fixed endpoint weights ˆw1 and ˆw2, trainable parameters θ, and coefficient functions c1(t), c2(t), and c.The curve is represented as φ_θ(t) = ˆw1 · c1(t) + θ · c(t) + ˆw2 · c2(t).
  • Computational complexity: O(|net|) is the complexity of evaluating the one-bend curve because it requires a weighted sum over the network parameters.The backward-pass Jacobian multiplication also has O(|net|) additional complexity.
  • Computational complexity: Below 50% is the usual observed gap in epoch time between training one model and applying the proposed method with the same architecture.The total additional computational complexity is O(|net|) for the one-bend family.

A.2 Batch Normalization

Batch normalization uses mini-batch statistics during training and aggregated statistics at test time. For networks evaluated along a newly found curve, the required statistics are recomputed with an additional data pass.

  • Batch normalization re-parameterizes each layer’s output using its mean, standard deviation, scale γ, and offset β.A positive ε provides numerical stability.
  • During training, μ(x) and σ(x) are computed separately for each mini-batch, while test-time evaluation uses statistics aggregated during training.
  • For a network on a connecting curve φ(t), μ(x) and σ(x) are recomputed over mini-batches for each curve point during training.
  • At test time, curve networks require one additional pass over the data because running batch-normalization statistics were not collected during training.

A.3 Formulas for curves with n bends

The paper parameterizes polygonal chains and Bezier curves connecting endpoints through n learned bends. Polygonal-chain segments are selected by the interval containing t, while Bezier curves use a separate n-bend formulation.

  • Polygonal chains: A polygonal chain with n bends θ = {w1, w2, …, wn} connects endpoints w0 and wn+1 through piecewise linear segments.
  • Polygonal chains: For each interval of t, the polygonal-chain parameterization selects the corresponding adjacent pair of vertices and linearly interpolates between them.
  • Bezier curves: The paper also gives a Bezier-curve parameterization connecting endpoints w0 and wn+1 with n intermediate bends.

A.4 Curve Finding Experiments

Curve-finding experiments evaluate simple paths between independently trained modes across architectures and tasks. The paths maintain near-constant accuracy, support diverse curve ensembles, and expose practical costs for batch normalization and temperature scaling.

  • Visualizations: ResNet-164 on CIFAR-100 shows quadratic Bezier and one-bend polygonal paths with near-constant loss, whereas direct linear connections incur high loss.
  • Experimental setup: Table 2 reports minimum, maximum, and integral statistics for train loss and train or test error along curves, estimated from 121 equally spaced points.The integral is approximated with the trapezoidal rule and represents a mean over a uniform distribution on the curve.
  • Experimental setup: The study also evaluates recurrent networks on next-word prediction using perplexity on the PTB dataset.
  • Curve ensembling: A 50-point ResNet-164 curve ensemble reaches 21.03% CIFAR-100 test error, versus 22.0% for its endpoints and 21.01% for three independent networks.
  • Curve ensembling: Temperature scaling reduces the curve ensemble’s error from 21.03% to 20.7% by flattening loss along the curve and correcting overconfident predictions.
  • Practical limitations: Direct curve ensembling requires manual temperature scaling, an additional training-data pass for each network’s batch normalization, and at least two endpoint models.
Loading 1802.10026v4…