PaperlayerView exact revision on arXiv

Source-linked AI summary

Repulsive Deep Ensembles are Bayesian

Francesco D'Angelo, Vincent Fortuin

arXiv:2106.11642v3cs.LGstat.ML

TL;DR

Deep ensembles lack guarantees of member diversity and convergence to the true Bayesian posterior, while weight-space inference can waste computation on distinct weights representing the same function. The paper adds kernelized repulsion to ensemble updates and studies it in weight and function space. The authors report improved uncertainty estimates and closer approaches to the true posterior, including for OOD detection on real-world data.

  • Problem

    Deep ensembles provide no guarantees of member diversity or convergence to the true Bayesian posterior, and weight-space inference can spend computation on redundant modes representing equivalent functions.

    Source: §1 · pp. 1–2

  • Method

    The paper adds kernelized repulsion to deep-ensemble updates and derives weight- and function-space terms that approximate Wasserstein gradient flows of KL divergence.

    Source: §1 · p. 2, §3.2 · p. 6, §3.3 · p. 7

  • Results

    The authors report that functional repulsion improves uncertainty estimates on synthetic examples and OOD detection on real-world data, and approaches the true Bayesian posterior more closely.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The authors frame kernelized repulsion as transforming deep ensembles' MAP inference into proper Bayesian inference through Wasserstein gradient descent on KL divergence.

    Source: §6 · p. 10

    The impact of the Jacobian in the function-space update and explicit convergence bounds remain topics for future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

Deep ensembles have recently gained popularity in the deep learning community for their conceptual simplicity and efficiency. However, maintaining functional diversity between ensemble members that are independently trained with gradient descent is challenging. This can lead to pathologies when adding more ensemble members, such as a saturation of the ensemble performance, which converges to the performance of a single model. Moreover, this does not only affect the quality of its predictions, but even more so the uncertainty estimates of the ensemble, and thus its performance on out-of-distribution data. We hypothesize that this limitation can be overcome by discouraging different ensemble members from collapsing to the same function. To this end, we introduce a kernelized repulsive term in the update rule of the deep ensembles. We show that this simple modification not only enforces and maintains diversity among the members but, even more importantly, transforms the maximum a posteriori inference into proper Bayesian inference. Namely, we show that the training dynamics of our proposed repulsive ensembles follow a Wasserstein gradient flow of the KL divergence with the true posterior. We study repulsive terms in weight and function space and empirically compare their performance to standard ensembles and Bayesian baselines on synthetic and real-world prediction tasks.

1 Introduction

The introduction argues that deep ensembles lack guarantees of functional diversity and Bayesian convergence, while weight-space inference can waste computation on equivalent functions. It presents kernelized repulsion as a route to Bayesian convergence and improved uncertainty estimation.

1 Introduction

  • Deep ensembles perform well in prediction and uncertainty estimation, but lack guarantees of member diversity or convergence to the true Bayesian posterior.The paper proposes repulsion between members as a way to address this gap.Source: §1 · p. 1
  • Weight-space inference can treat distinct weights that encode the same function as diverse, creating redundant posterior modes and poor uncertainty estimates.The paper calls this neural-network non-identifiability and notes that several inference methods can spend computation on equivalent functions.Source: §1 · p. 2
  • Kernelized repulsion regularizers are proposed to prevent ensemble members from collapsing and to give deep ensembles Bayesian convergence properties.The paper derives several such terms for gradient updates.Source: §1 · pp. 1–2
  • The proposed terms approximate Wasserstein gradient flows of KL divergence in both weight and function space, and the methods are compared theoretically with deep ensembles and SVGD.The paper also assesses the methods on synthetic and real-world tasks, reporting competitive performance and improved uncertainty estimation.Source: §1 · p. 2

2 Repulsive Deep Ensembles

The section frames deep ensembles as MAP solutions used for Bayesian model averaging, then develops kernel-mediated repulsion in weight and function spaces and contrasts it with SVGD.

2 Repulsive Deep Ensembles

  • Deep ensembles combine predictions from independent MAP solutions, but do not guarantee sampling from the true posterior or prevent members from collapsing to the same mode.Their ability to average hypotheses does not replace the Langevin noise identified as crucial for true-posterior sampling.Source: §2 · p. 3

2.1 Repulsive force in weight space

  • Kernel-gradient repulsion moves weight-space members away from one another; a positive scaling factor sets the repulsive term, with convergence depending on its choice.The paper also describes an RBF-kernel intuition in which the gradient drives a member away from neighboring members.Source: §2.1 · pp. 3–4

2.2 Repulsive force in function space

  • Function-space repulsion interacts over neural-network regression functions, then projects the update through a canonical input-space projection and each particle’s Jacobian.The projection uses a subset of inputs, such as a batch of training points, before the update is mapped back to parameter space.Source: §2.2 · p. 4

2.3 Comparison to Stein variational gradient descent

  • Unlike SVGD’s kernel-weighted averaging of posterior gradients, the proposed update avoids that averaging; kernel methods may struggle in high dimensions, and projected function distances may misrepresent true distances.In function space, the update is projected back using only the particle’s own Jacobian.Source: §2.3 · pp. 4–5

3 Repulsive deep ensembles are Bayesian

The section derives repulsive ensemble updates from particle approximations and Wasserstein gradient flow, with kernel-based gradient estimators for parameter- and function-space inference. It also specifies kernel constraints and compares uncertainty captured by function- and weight-space methods.

3.1 Particle approximation

  • Particle approximations represent a target measure with weighted samples chosen to minimize a distance between the empirical and target distributions.Equal weights of 1/N are one example, obtained with MCMC samples.Source: §3.1 · p. 5

3.2 Gradient flow in parameter space

  • Wasserstein gradient flow evolves probability measures to minimize functionals such as KL divergence, with the posterior as the unique stationary distribution in the stated dynamics.The flow is expressed through a Liouville equation, and the KL case recovers the Fokker–Planck equation.Source: §3.2 · pp. 5–6
  • In the infinite-particle limit, KDE-based repulsion follows a Wasserstein KL gradient flow and converges to the true Bayesian posterior.The repulsive force approximates the gradient of the empirical particle measure, encouraging diversity while avoiding collapse.Source: §3.2 · p. 6

3.3 Gradient flow in function space

  • Function-space updates use function particles and a functional posterior; implicit weight-space priors require an additional gradient estimator, while analytical priors such as Gaussian processes can encode function properties.The authors note this may be useful for out-of-distribution detection.Source: §3.3 · p. 7

3.4 The choice of the kernel

  • For asymptotic Bayesian convergence, SGE and SSGE require Stein-class kernels, whereas KDE permits any symmetric probability density; L1 and L2 KDE kernels showed no substantial difference in the reported experiments.Further investigation of the L1-versus-L2 choice is left for future work.Source: §3.4 · p. 7
  • Figure 3 compares predictive-posterior entropy and reports that function-space methods capture uncertainty better than weight-space methods, approaching the HMC posterior.Figure 2 displays samples from SVGD, KDE-WGD, SGE-WGD, and SSGE-WGD, with KDE marginals for samples and particles.Source: Figure 2 · p. 7, Figure 3 · p. 8

4 Experiments

Experiments compare Wasserstein-gradient methods with deep ensembles and SVGD on synthetic and real-world prediction tasks, evaluating predictive performance, uncertainty, and functional diversity. Results vary by setting, with functional methods especially strong on uncertainty and OOD detection in some tasks.

Evaluation setup

  • The evaluation uses accuracy, NLL, ECE, entropy- and disagreement-based OOD metrics, and functional diversity measured by model disagreement.The experiments use an RBF kernel with the median heuristic by default; adaptive bandwidth performed better than a fixed constant in their experiments.Source: §4 · p. 8

Synthetic distribution sampling

  • On a two-dimensional Gaussian, SGE-WGD, SSGE-WGD, and SVGD fit the target almost perfectly; SGE-WGD and SVGD also perform best on the Funnel distribution.Source: §4 · p. 8

BNN 1D regression

  • In one-dimensional regression, weight-space methods fail to capture epistemic uncertainty between training clusters, whereas f-SVGD and fWGD closely match the HMC predictive posterior.Source: BNN 1D regression · p. 8

FashionMNIST classification

  • Across image-classification tasks, WGD methods achieve the best accuracy, while fWGD methods yield the best OOD detection and functional diversity; sge-WGD leads accuracy on FashionMNIST.On FashionMNIST, function-space methods also achieve the lowest NLL.Source: Table 1 · p. 9, FashionMNIST classification · p. 9

BNN 2D classification

  • In synthetic classification, fWGD is confident near training data but remains uncertain out of distribution, unlike overconfident weight-space methods and f-SVGD.The authors interpret this pattern as better representation of epistemic uncertainty and a basis for principled OOD detection.Source: BNN 2D classification · pp. 8–9

CIFAR classification

  • On CIFAR-10 with SVHN as OOD data, weight-space methods outperform function-space methods in accuracy and entropy-based OOD detection, while deep ensembles achieve the best entropy-based OOD detection.All repulsive ensembles improve functional diversity, accuracy, and OOD detection over standard SVGD.Source: CIFAR classification · p. 9

5 Related Work

Related work places the paper among gradient-flow interpretations of inference, particle methods, and neural-network ensembles. It also distinguishes its focus from earlier Bayesian interpretations and diversity approaches.

Gradient-flow methods

  • SVGD has been studied theoretically and can be viewed as a KL-divergence Wasserstein gradient flow in Stein geometry.Source: §5 · p. 9
  • MCMC-type algorithms also admit gradient-flow interpretations, and SGE-based WGD has an alternative smoothed-test-function formulation.WGD also relates to particle inference frameworks and projected variants.Source: §5 · p. 10

Ensembles and Bayesian interpretations

  • Prior work has interpreted deep ensembles through several Bayesian lenses, while other studies have examined repulsive interactions, ensemble methods, and function-space variational inference.Source: §5 · p. 10

6 Conclusion

The conclusion presents kernelized repulsion as a way to improve deep ensembles and interpret their training as Wasserstein gradient descent on KL divergence. It highlights functional repulsion's uncertainty and OOD results while identifying theoretical questions for future work.

6 Conclusion

  • Kernelized repulsion improves prediction accuracy and recasts deep-ensemble MAP training as Wasserstein gradient descent on the KL divergence.Source: §6 · p. 10
  • Functional repulsion improves uncertainty estimates on simple synthetic examples and OOD detection on real-world data, and can approach the true Bayesian posterior more closely.Source: §6 · p. 10
  • Future work should study the fWGD update's Jacobian and derive explicit convergence bounds for the proposed method.The authors also propose comparing those bounds with existing SVGD bounds.Source: §6 · p. 10

A Non-identifiable neural networks

Neural networks can represent the same function with different weights, making the weight-to-function map non-injective. This redundancy can create equivalent posterior modes, while function-space inference avoids that redundancy but is often impractical.

A Non-identifiable neural networks

  • Non-identifiability means multiple weight configurations can produce the same network function, including through prediction-preserving transformations such as certain MLP weight permutations and sign flips.Source: A Non-identifiable neural networks · p. 16
  • Equivalent functions can correspond to multiple posterior modes in weight space, although prior invariance is required for those modes to be equally likely.Source: A Non-identifiable neural networks · p. 16
  • Function-space inference avoids this redundancy because each optimal function has a unique posterior mode, but inference is generally prohibitive in an infinite-dimensional space.The passage notes Gaussian processes as a limited setting where exact function-space Bayesian inference is possible.Source: A Non-identifiable neural networks · p. 16

B Quantify functional diversity

This section relates functional diversity among predictions to epistemic uncertainty, defines the functional derivative used for the KL derivation, and interprets SVGD as a Wasserstein gradient flow under a kernel-induced Stein geometry.

B Quantify functional diversity

  • Functional diversity among hypotheses informs epistemic uncertainty, which can quantify whether a test point resembles the training distribution and support out-of-distribution detection.Predictive entropy is presented as a common uncertainty measure, but it cannot distinguish confident disagreement from shared lack of confidence.Source: B Quantify functional diversity · pp. 16–17
  • Model disagreement is zero when predictions agree and grows with disagreement, making it a direct measure of epistemic uncertainty.The section proposes taking an expectation over the output space to obtain a scalar from the disagreement quantity.Source: B Quantify functional diversity · p. 17

C Functional derivative of the KL divergence

  • The functional derivative describes how a functional’s value varies when its function changes, and the section derives this quantity for the KL divergence.The derivation begins with definitions of a functional and the KL divergence.Source: C Functional derivative of the KL… · pp. 17–18

D SVGD as Wasserstein gradient flow

  • SVGD can be interpreted as a Wasserstein gradient flow under a Stein geometry induced by its kernel, with the kernel operator acting as a perturbation of the gradient.The section relates the operator form of the dynamics to the KL-divergence functional derivative approximated in the kernel’s RKHS.Source: D SVGD as Wasserstein gradient flow · pp. 18–19

E Kernel density estimation

The section presents KDE as a smoothed empirical distribution and states its limiting relation to the true density; the accompanying funnel caption reports the strongest fits.

E Kernel density estimation

  • With an RBF kernel, KDE smooths the empirical distribution by replacing each data-centered Dirac delta with a kernel.The empirical distribution is described as a mixture of Dirac deltas centered at the training data.Source: E Kernel density estimation · p. 19
  • As bandwidth h approaches zero and sample size N grows without bound, the KDE is unbiased and equals the true density.In that limit, the kernels approach Dirac deltas and the estimate approaches the empirical distribution.Source: E Kernel density estimation · p. 19
  • On Neal’s funnel, SGE-WGD and SVGD are reported to fit the distribution best.The caption identifies this as Figure F.1.Source: Figure F.1 · p. 20

F Additional experiments

The additional experiments examine sampling from a two-dimensional Funnel distribution; the text identifies its target density and points to Figure F.1 for results.

F Additional experiments

  • The additional experiments assess different methods by sampling from the Funnel distribution.The passage frames these as additional results for the methods.Source: F Additional experiments · p. 20
  • The Funnel target is p(x, y) = N(y|µ = 0, σ = 3)N(x|0, exp(y/2)).The experiment samples from this two-dimensional distribution.Source: F Additional experiments · p. 20
  • The Funnel sampling results are reported in Figure F.1.The passage does not state further result values or comparisons.Source: F Additional experiments · p. 20

G Implementation details

The implementation section reports shared computing details and separate configurations for synthetic sampling, regression, and classification experiments on two image datasets.

G Implementation details

  • The experiments used the ETH Leonhard cluster with NVIDIA GTX 1080 Ti GPUs and took roughly 150 GPU hours.The code is partially based on von Oswald et al. [73].Source: G Implementation details · p. 20

G.1 Sampling from synthetic distributions

  • Synthetic Gaussian sampling used 100 particles for 5,000 Adam iterations, while Funnel sampling used 500 particles for 2,000 iterations.Both used a fixed learning rate of 0.1; the Gaussian kernel bandwidth used the median heuristic, while Funnel used bandwidth 0.5.Source: §G.1 · p. 20

G.2 1D regression

  • The 1D regression experiment used 90 training points from two input intervals and a two-hidden-layer, 50-unit ReLU network.The model used a Gaussian likelihood and standard normal prior, with 50 particles optimized for 15,000 gradient steps.Source: §G.2 · pp. 20–21

G.3 2D classification

  • The 2D classification experiment used 200 points from a five-Gaussian mixture and a two-hidden-layer, 50-unit ReLU network.It used a softmax likelihood, standard normal prior, and 100 particles optimized for 10,000 steps.Source: §G.3 · p. 21

G.4 Classification on FashionMNIST

  • FashionMNIST experiments used a three-hidden-layer, 100-unit ReLU network and averaged results over five random seeds.The setup used 50 particles and 50,000 optimization steps; the passage also describes hyper-deep-ensemble tuning.Source: §G.4 · p. 21

G.5 Classification on CIFAR-10

  • CIFAR-10 experiments used ResNet32 and averaged results over five random seeds.The setup used 20 particles and 50,000 optimization steps; the passage also describes hyper-deep-ensemble tuning.Source: §G.5 · p. 21
Loading 2106.11642v3…