PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Ensembles: A Loss Landscape Perspective

Stanislav Fort, Huiyi Hu, Balaji Lakshminarayanan

arXiv:1912.02757v2stat.MLcs.LG

TL;DR

Deep ensembles are studied as a response to the difficulty of representing useful uncertainty with tractable Bayesian approximations. The paper measures prediction diversity across trajectories and subspaces, finding that random initializations explore distinct function-space modes and achieve a better diversity–accuracy trade-off than subspace sampling methods.

  • Problem

    Exact Bayesian posterior inference is computationally expensive for deep neural networks, while scalable approximations may fail to explore multiple modes despite capturing uncertainty within one mode.

  • Method

    The paper measures prediction-function similarity within optimization trajectories, trajectory-derived subspaces, and across independently randomized trajectories, then compares their diversity–accuracy trade-offs and ensemble effects.

  • Results

    Random initializations explore different function-space modes, whereas trajectory and subspace samples remain similar in predictions; random initializations therefore outperform subspace methods on the diversity–accuracy trade-off.

  • Takeaways & Limitations

    Subspace sampling and ensembling provide complementary benefits, with ensembling offering greater relative benefits when predictions are averaged over more diverse solutions.

  • Takeaways & Limitations

    The modeling setup assumes a parametric neural-network conditional distribution with a prior over weights and biases.

Abstract

from arXiv · show

Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable variational Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods. Finally, we evaluate the relative effects of ensembling, subspace based methods and ensembles of subspace based methods, and the experimental results validate our hypothesis.

1 Introduction

The introduction frames deep ensembles as a practical alternative to point-estimate and single-mode Bayesian approaches, and proposes that random initialization explores diverse function-space modes. The paper investigates this hypothesis by measuring prediction-function diversity across trajectories and subspaces.

  • Motivation: Exact Bayesian posterior inference is computationally expensive for deep neural networks, whereas MAP estimation is usually tractable but returns only a point estimate.This motivates methods that can represent uncertainty beyond a single posterior mode.
  • Motivation: Deep ensembles train multiple networks from different initializations, potentially obtaining different solutions because the loss is non-convex.The introduction focuses on random-initialization ensembles rather than adversarially trained variants.
  • Motivation: Finite data can support multiple parameter explanations, making diverse solutions important for quantifying epistemic uncertainty.Bayesian neural networks target distributions over weights, while ensembles can produce multiple fitted solutions.
  • Hypothesis: The paper hypothesizes that ensembles sample different function-space modes, while variational Bayesian methods may remain within one mode despite modeling within-mode uncertainty.The authors note that a MAP optimum for training loss need not be a local optimum for validation loss.
  • Approach: The study measures function-space diversity within trajectories, within trajectory-derived subspaces, and across randomly initialized trajectories.It compares prediction similarity across datasets, architectures, and dataset shift, finding within-trajectory functions similar and independently initialized functions diverse.

2 Background

The background describes neural-network loss landscapes as high-dimensional but empirically structured, including low-obstacle paths and low-loss connections between optima. These weight-space properties motivate examining whether connected solutions also differ in their functions.

  • Loss landscapes: The neural-network loss landscape is the high-dimensional space of weights and biases traversed during training.Despite its potentially complicated geometry, empirical studies report surprisingly simple loss-surface properties.
  • Loss landscapes: Optimization from initialization to an optimum can follow a linear path whose loss decreases monotonically without significant obstacles.This observation concerns the loss along a path in parameter space.
  • Mode definition: The paper defines a mode by a unique function, so different parameter vectors can represent the same mode because of weight-space symmetries.Thus, parameter-space separation does not by itself establish function-space diversity.
  • Mode connectivity: Independent optima can be connected by continuous low-loss paths even when their direct linear interpolation passes through high loss.These low-loss tunnels provide a way to study relationships among distinct optima.

3 Experimental setup

The experiments span CIFAR-10, CIFAR-100, and ImageNet, using multiple convolutional architectures and training configurations. Evaluation includes both ordinary test data and corrupted datasets for studying behavior under dataset shift.

  • Datasets: The study evaluates CIFAR-10, CIFAR-100, and ImageNet to test whether its observations generalize across dataset complexity.CIFAR-10 contains 50K training examples across 10 classes; CIFAR-100 has 100 classes, and ImageNet roughly 1M examples across 1000 classes.
  • Architectures: CIFAR-10 experiments use SmallCNN, MediumCNN, and ResNet20v1 to assess architectural generality.The reported test accuracies are 64%, 71%, and 90%, respectively.
  • Training: Training uses Adam with batch size 128, while additional validation checks whether findings also hold with vanilla SGD.SmallCNN and MediumCNN use dropout 0.1, and learning-rate schedules are specified for similarity analyses.
  • Dataset shift: The evaluation includes CIFAR-10-C and ImageNet-C, which vary corruption type and intensity to measure uncertainty calibration under dataset shift.The benchmarks contain 19 corruption types for CIFAR-10-C and 15 for ImageNet-C, with intensities from 1 to 5.

4 Visualizing Function Space Similarity

The experiments compare weight-space and prediction-space similarity within optimization trajectories, across random initializations, and among subspace samples. Random initializations produce more distinct functions, while subspace samples remain near their originating trajectory and yield a worse diversity–accuracy trade-off.

  • Within and Across Randomly Initialized Trajectories: Checkpoints along a single optimization trajectory are largely similar in both weight space and function space.Weight alignment is measured by cosine similarity, while function disagreement is the fraction of labels receiving different predictions.
  • Within and Across Randomly Initialized Trajectories: Solutions from different random initializations have much lower function similarity than functions within a single trajectory.Across architectures, independently initialized weight vectors are essentially orthogonal and their predictions are approximately as dissimilar as other solution pairs.
  • Within and Across Randomly Initialized Trajectories: t-SNE prediction plots place checkpoints from different trajectories far apart, while checkpoints from one trajectory cluster more closely.The plotted prediction representations are formed from flattened softmax outputs for a set of examples.
  • Within Subspace from Each Trajectory and Across Trajectories: Subspace-sampled functions remain near the prediction-space neighborhood of their originating trajectory and do not overlap with another randomly initialized trajectory.This pattern holds across the four subspace generation methods shown for SmallCNN on CIFAR-10.
  • Within Subspace from Each Trajectory and Across Trajectories: Along radial paths toward two independent optima, solutions and Gaussian-subspace samples are more similar to their respective optima than to the other optimum.The radial visualization also shows that independently initialized trajectories eventually achieve similar accuracy.
  • Diversity versus Accuracy: Independently initialized and optimized solutions achieve a better diversity–accuracy trade-off than four subspace sampling methods across the evaluated architectures and datasets.The separation between subspace methods and independent optima becomes more pronounced for more difficult problems and more powerful networks.

5 Evaluating the Relative Effects of Ensembling versus Subspace Methods

The paper compares independently trained ensembles, subspace sampling, and combinations of both across standard and shifted-data evaluations. Results indicate complementary benefits, with independent random initializations producing the greatest prediction diversity.

  • Controlled comparisons: The controlled comparison evaluates a single-trajectory baseline, subspace sampling, random-initialization ensembles, and ensembles that combine both strategies.The combined method samples within multiple independently trained trajectories.
  • CIFAR-10 results: The experiments validate that subspace sampling and ensembling provide complementary benefits on CIFAR-10.The relative benefit of ensembling increases when it averages predictions from more diverse solutions.
  • Dataset shift: On CIFAR-10-C, ensembles and subspace sampling again provide complementary benefits as corruption intensity varies.The evaluation reports accuracy, Brier score, and Jensen-Shannon divergence.
  • Dataset shift: Jensen-Shannon divergence is highest between independent random initializations and lower for subspace sampling, with the difference increasing under dataset shift.On SVHN, the reported divergence is 0.384 for independent runs versus 0.034 for diagonal Gaussian sampling.
  • ImageNet results: On ImageNet and ImageNet-C, random subspace performs best among subspace methods and provides complementary benefits to random-initialization ensembles.The ImageNet evaluation varies corruption intensity on the test set and ImageNet-C.

6 Discussion

The discussion concludes that random initialization explores distinct function-space modes, whereas subspace-based methods remain insufficiently diverse despite weight-space variation. The diversity–accuracy analysis shows that current variational Bayesian methods do not match independently trained models.

  • Discussion: Randomly initialized training trajectories explore different modes in function space, explaining why random-initialization deep ensembles work well.The conclusion attributes ensemble effectiveness to function-space diversity rather than bootstrap resampling alone.
  • Discussion: Weight averaging, Monte Carlo dropout, and local Gaussian approximations can move far in weight space while producing insufficiently diverse predictions.These methods remain similar in function space despite their weight-space displacement.
  • Discussion: Current variational Bayesian methods do not achieve the diversity–accuracy trade-off reached by independently trained models.The comparison is framed through the paper’s diversity–accuracy plane.

A Identical loss does not imply identical functions in prediction space

Low-loss connectivity between optima does not imply that the connected models implement the same function. The paper visualizes this distinction by comparing loss landscapes with prediction agreement along optimized paths.

  • Loss landscapes: Different optimization trajectories can reach similar loss values while remaining distinct in their functional forms.The loss-surface analysis compares trajectories and optima in weight space and prediction space.
  • Mode connectivity: The paper constructs low-loss tunnels between independent optima by optimizing a connector from a linear interpolation path toward the low-loss manifold.The connector procedure follows Fort and Jastrzebski’s simplification of earlier mode-connectivity methods.
  • Prediction-space analysis: Prediction similarities are computed along linear segments of the curved connector to visualize function-space behavior across the loss-landscape cut.Each segment is combined with the origin of weight space to form a triangular visualization region.

B Effect of randomness: random initialization versus random shuffling

Random initialization and data-order shuffling are complementary randomness sources, but initialization is dominant. Mini-batch shuffling contributes more randomness at higher learning rates.

  • Sources of randomness: Random initialization is the dominant source of prediction diversity, while random training-batch shuffling provides a complementary contribution.The experiments decouple initialization from data-order effects.
  • Sources of randomness: Random mini-batch shuffling adds more randomness at higher learning rates because of increased gradient noise.This effect is reported alongside the stronger influence of initialization.
  • Hardware dependence: On GPUs, identical initialization and training batches can still produce non-identical predictions, whereas the corresponding TPU runs learn the same function.The figure reports zero prediction diversity for those TPU runs.

C Comparison to cSG-MCMC

The authors compare cSG-MCMC with independently initialized deep ensembles and develop accuracy–diversity bounds for interpreting function-space differences. Their results show that independent solutions achieve higher practical diversity than the modeled limits, leaving room for improvement.

  • Comparison to cSG-MCMC: cSG-MCMC was evaluated against independently initialized and trained models using function diversity on ResNet-18 with CIFAR-10 and CIFAR-100.The experiments used three random initializations and 150 epochs for cSG-MCMC, comprising three learning-rate cycles.
  • Comparison to cSG-MCMC: Deep ensembles produced statistically significantly more diverse functions than cSG-MCMC on both CIFAR-10 and CIFAR-100.The comparison reports means and 1σ confidence intervals from four experiments per method.
  • Accuracy–diversity bounds: The theoretical upper-bound curve models disagreement between a reference function and another function using their accuracies and the number of classes.The derivation considers cases where predictions are correct or incorrect and either agree or disagree.
  • Accuracy–diversity bounds: Practical diversity remained below the theoretical optimum even for independently initialized and optimized solutions.This gap provides scope for future work.
  • Accuracy–diversity bounds: The lower-limit curve models a function near the reference in weight space by perturbing its predictions with label-flip probability p.The resulting accuracy relationship is inverted to express disagreement as a function of accuracy.

D.3 Correlated predictions: 1-parameter family

The paper extends the correlated-prediction model by assigning separate flip probabilities to correctly and incorrectly classified examples. A one-parameter exponent family fits the observed accuracy–diversity trade-off, with e = 0.22 providing the closest match.

  • D.3 Correlated predictions: 1-parameter family: The refined model uses p+ for correctly labeled examples and p− for incorrectly labeled examples.This separates the probabilities that correct and incorrect predictions flip.
  • D.3 Correlated predictions: 1-parameter family: The lower limit is recovered when p+ = p− = p, while the absolute worst case flips only correctly labeled examples.The latter corresponds to p− = 0.
  • D.3 Correlated predictions: 1-parameter family: The one-parameter family ties the flip probabilities through an exponent e and generates a realistic accuracy–diversity trade-off.The resulting functions are shown for several exponent values.
  • D.3 Correlated predictions: 1-parameter family: e = 0.22 provided the closest match to the observed data.For e < 1, the model assigns a larger flip probability to wrong labels than to correct labels, representing robustness to perturbations.
Loading 1912.02757v2…