PaperlayerView exact revision on arXiv

Source-linked AI summary

Tighter risk certificates for neural networks

María Pérez-Ortiz, Omar Rivasplata, John Shawe-Taylor, Csaba Szepesvári

arXiv:2007.12911v3cs.LGcs.CVstat.ML

TL;DR

Probabilistic neural networks need risk certificates that are informative about out-of-sample classification error. This paper trains them with objectives derived from PAC-Bayes bounds and evaluates certificates using part of the learning data. On MNIST and CIFAR-10, the methods yield competitive test performance and tighter, non-vacuous certificates than previous results, supporting their promise for self-certified learning and model selection.

  • Problem

    Risk certificates are informative only when their values match or closely follow classification error rates measured on a test set.

    Source: §1 · p. 3

  • Method

    The paper studies three PAC-Bayes-derived objectives for probabilistic neural networks and computes certificates using part of the data used to learn the predictor.

    Source: §1 · p. 2

  • Results

    Across MNIST and CIFAR-10, PBB objectives yield competitive test performance and tighter non-vacuous certificates than previous results; on MNIST, a small convolutional neural network achieves 1% test set error and a risk certificate of 1.5%.

    Source: §1 · p. 3, §8 · p. 34

  • Takeaways & Limitations

    The certificates can guide learning and model selection, and their evaluation does not require a held-out test set.

    Source: §7.9 · p. 33, §8 · p. 34

    The experiments leave covariance structures, broader weight-distribution choices, and more datasets for future validation.

    Source: §8 · p. 34

Abstract

from arXiv · show

This paper presents an empirical study regarding training probabilistic neural networks using training objectives derived from PAC-Bayes bounds. In the context of probabilistic neural networks, the output of training is a probability distribution over network weights. We present two training objectives, used here for the first time in connection with training neural networks. These two training objectives are derived from tight PAC-Bayes bounds. We also re-implement a previously used training objective based on a classical PAC-Bayes bound, to compare the properties of the predictors learned using the different training objectives. We compute risk certificates for the learnt predictors, based on part of the data used to learn the predictors. We further experiment with different types of priors on the weights (both data-free and data-dependent priors) and neural network architectures. Our experiments on MNIST and CIFAR-10 show that our training methods produce competitive test set errors and non-vacuous risk bounds with much tighter values than previous results in the literature, showing promise not only to guide the learning algorithm through bounding the risk but also for model selection. These observations suggest that the methods studied here might be good candidates for self-certified learning, in the sense of using the whole data set for learning a predictor and certifying its risk on any unseen data (from the same distribution as the training data) potentially without the need for holding out test data.

1. Introduction

The paper studies principled PAC-Bayes training objectives for probabilistic neural networks and evaluates whether they yield useful risk certificates alongside competitive prediction performance. It proposes two new objectives and reports tighter certificates than prior work, motivating self-certified learning.

1. Introduction

  • PAC-Bayes with Backprop (PBB) is a strategy for deriving probabilistic neural-network training methods from PAC-Bayes bounds and optimizing them with stochastic gradient descent.The paper studies three objectives: fquad, flambda, and fclassic.Source: §1 · pp. 2, 4
  • PBB objectives achieve competitive test-set errors and tighter risk certificates than prior results, narrowing the gap between certificates and test-set error on MNIST and CIFAR-10.The authors attribute this improvement to the tightness of the PAC-Bayes bounds, established analytically and corroborated experimentally.Source: §1 · pp. 3–4
  • The authors motivate self-certified learning as using available data to learn a predictor and certify its risk on unseen examples, provided the certificate tracks test error.Their approach uses a data-dependent prior learned on a subset that does not overlap the data used to compute the certificate.Source: §1 · pp. 3–4
  • The classification-error guarantee is obtained by evaluating a risk bound after training; the training objective itself guarantees risk under the surrogate loss.The authors identify calibration of surrogate cross-entropy loss as a more ambitious goal.Source: §1 · p. 4
  • The paper proposes objectives derived from PAC-Bayes-quadratic and PAC-Bayes-lambda bounds, and re-implements a classic-bound objective for comparison.The comparisons cover test-set accuracy and risk certificates.Source: §1 · p. 4
  • Unlike Bayes-by-Backprop, the PBB methods provide a risk certificate as well as an error estimate and require fewer heuristics.The paper presents its results as competitive in test-set accuracy with Blundell et al. (2015).Source: §1 · p. 4

2. Generalisation through Risk Upper Bounds

Empirical risk is observable, but a small training loss does not always imply small risk on unseen data. The section motivates minimizing a tight upper bound on risk as an alternative to empirical-risk minimization alone.

2. Generalisation through Risk Upper Bounds

  • Under certain conditions, small empirical risk guarantees a small risk gap, but empirical-risk minimization can also yield substantially higher unseen-data risk through overfitting.Small function-class capacity relative to sample size and stability are given as examples of conditions supporting a small risk gap.Source: §2 · p. 7
  • Minimizing a tight surrogate objective that upper-bounds risk can provide a small-risk guarantee and a self-bounding learning method.The paper follows this approach with two training objectives derived from PAC-Bayes bounds.Source: §2 · p. 7
  • The proposed objectives yield test-set performance comparable to Blundell et al. (2015) and tighter non-vacuous bounds than the classic objective used by Dziugaite and Roy.The classic objective is essentially equivalent to the one used by Dziugaite and Roy.Source: §2 · p. 7

3. PAC-Bayes Bounds

The section presents PAC-Bayes bounds that relate the expected loss of a distribution over weights to empirical loss and other quantities. It describes classic, quadratic, and lambda variants and the conditions that affect their use.

3. PAC-Bayes Bounds

  • For probabilistic networks, the predictor is a distribution Q over weights, and its loss is the expected loss over draws from Q.PAC-Bayes bounds relate this population loss to empirical loss and other quantities with high probability.Source: §3 · p. 8
  • The PAC-Bayes-kl bound gives a high-probability risk guarantee for all weight distributions Q, provided the reference distribution Q0 is fixed independently of the evaluated sample.The paper uses a version with sharp dependence on sample size, as clarified by Maurer (2004).Source: §3 · p. 9
  • The classic PAC-Bayes bound relaxes the PAC-Bayes-kl bound and permits optimizing its upper bound over Q to select a data-dependent distribution.The paper describes the classic form as following from Pinsker’s inequality.Source: §3 · p. 9
  • The refined Pinsker inequality is tighter when population loss L(Q) is below 1/4, but its appearance on the right-hand side prevents direct optimization.Solving the resulting quadratic inequality yields the PAC-Bayes-quadratic bound, which holds uniformly over Q.Source: §3 · pp. 9–10
  • The PAC-Bayes-lambda bound holds uniformly over Q and lambda in (0, 2), so searching a grid of lambda values does not worsen the bound.The source identifies this bound with Thiemann et al. (2017).Source: §3 · p. 10

4. The Bayes by Backprop (BBB) Objective

The Bayes by Backprop objective learns a distribution over network weights by approximating the Bayesian posterior. Rewriting its KL divergence yields a data-fit term and a prior-complexity term, with a tunable scaling factor in the implemented objective.

4. The Bayes by Backprop (BBB) Objective

  • BBB minimizes the KL divergence from a learned weight distribution to the Bayesian posterior, equivalently balancing expected negative log-likelihood against divergence from the prior.The prior-dependent term is KL(Qθ∥Q0).Source: §4 · p. 11
  • PAC-Bayes and Bayesian learning are closely connected for negative log-likelihood, but PAC-Bayes permits more flexible losses and parameter distributions.Its prior is a reference distribution, and its posterior need not result from a likelihood update.Source: §4 · p. 11
  • The objective’s scaling factor η is heuristic; tuning it searches KL balls around the prior and controls the model-space complexity penalty.The variational Bayes argument gives η = 1, while BBB uses SGD and a validation set to choose η.Source: §4 · p. 11

5. Towards Practical PAC-Bayes with Backprop (PBB) Methods

PAC-Bayes with Backprop (PBB) trains probabilistic neural networks by minimizing PAC-Bayes-derived objectives with stochastic gradient descent. The section develops two objectives based on tighter bounds, compares them with a classical-bound objective, and describes the loss, optimization, and prior choices used to make them practical.

5. Towards Practical PAC-Bayes with Backprop (PBB) Methods

  • PBB trains probabilistic neural networks by minimizing PAC-Bayes bounds with SGD; two objectives are introduced for the first time for this purpose and compared with a third.The objectives use the bounded zero-one or surrogate loss setting.Source: §5 · p. 12
  • Cross-entropy replaces zero-one loss because the latter is piecewise constant and unsuitable for gradient optimization; bounding probabilities makes the surrogate loss PAC-Bayes-compatible.The paper lower-bounds probabilities by p_min and rescales bounded cross-entropy to [0,1].Source: §5 · pp. 12–13

5.1 Optimisation Problem

  • fquad and fclassic optimize over Q, whereas flambda alternates optimization over Q and λ; PBB uses pathwise gradients to make weight-sampling gradients computable by backpropagation.The experiments use diagonal Gaussian posteriors reparameterized by a mean and coordinatewise scale; Laplace noise uses a similar procedure.Source: §5.1 · p. 13
  • fquad and flambda relax the PAC-Bayes-kl bound using refined Pinsker, while fclassic uses Pinsker; the compared inequalities are tighter on opposite sides of p = 1/4.The inequality in Eq. (14) is tighter when p > 1/4, and Eq. (15) when p < 1/4; they match at p = 1/4.Source: §5.1 · pp. 13–14

5.2 The Choice of the PAC-Bayes Prior Distribution

  • The experiments use Gaussian or Laplace priors and matching posterior families; data-dependent priors are learned by ERM on a separate subset, with dropout used to limit prior overfitting.The prior-learning subset is independent of the subset used to compute the risk certificate.Source: §5.2 · p. 14

5.2.1 Formulas for the KL: Laplace and Gaussian

  • The paper gives KL formulas for one-dimensional Laplace and Gaussian distributions and extends them to independent-component product distributions by summing component KL divergences.For the Laplace formula’s absolute value, autodifferentiation uses left or right derivatives.Source: §5.2.1 · p. 15

6. Computing Risk Certificates

Risk certificates are computed by inverting the PAC-Bayes-kl bound, but the posterior’s empirical loss is not directly computable. The paper estimates that loss by Monte Carlo weight sampling and combines the estimate with the bound to produce high-probability certificates.

6. Computing Risk Certificates

  • The PAC-Bayes-kl bound yields an upper bound on stochastic-predictor risk by inverting binary KL; evaluating the posterior empirical loss is the remaining difficulty.The inversion function f⋆ satisfies kl(x∥y) ≤ b precisely when y ≤ f⋆(x,b).Source: §6 · p. 15

6.1 Estimating the Empirical Loss via Monte Carlo Sampling

  • Monte Carlo weight samples estimate the posterior empirical loss, with a tail bound controlling the estimate; the resulting upper bound applies to zero-one or cross-entropy loss.The Monte Carlo estimate is valid with probability at least 1 − δ′ over random weight samples.Source: §6.1 · p. 16

6.2 Final Expression for Evaluating the Risk Certificate

  • The experiments combine PAC-Bayes-kl inversion with Monte Carlo sampling to evaluate cross-entropy and zero-one risk certificates for stochastic predictors, with confidence over data and weight samples.The numerical implementation evaluates certificates for distributions learned by the training methods.Source: §6.2 · p. 16, §6.1 · p. 16

7. Experimental Results

The experiments compare PAC-Bayes-derived training objectives with BBB and, where possible, dropout ERM under shared conditions. They vary weight distributions and priors, evaluate several prediction strategies and metrics, and include model-selection analyses across hyperparameter runs.

7. Experimental Results

  • Experiments on MNIST and CIFAR-10 compare fquad and flambda with fclassic and fbbb, and with dropout ERM where possible, under matched training conditions.The comparisons hold initialization, prior, optimizer, and architecture constant.Source: §7 · pp. 16–17

7.1 Choice of Distribution over Weights

  • The study tests Gaussian and Laplace weight distributions, using a posterior family matched to the chosen prior family.Both data-free random priors and data-dependent priors are tested; data-dependent prior centers are ERM-trained on a subset, with dropout used during prior learning to limit overfitting.Source: §7.1 · p. 17

7.2 Experimental Setup

  • Risk certificates use PAC-Bayes-kl with δ = 0.025, δ′ = 0.01, and 150,000 Monte Carlo samples; inputs are standardized.The stated confidence settings are used across the PBB objectives.Source: §7.2 · p. 17

7.2.1 Hyperparameter selection

  • Models are selected by the best zero-one risk certificate after a hyperparameter grid search; ERM instead uses held-out validation data because no risk certificate is used.Validation data comprise 4% of MNIST training data and 5% of CIFAR-10 training data; the authors report no empirical evidence for risk-bound model selection of ERM.Source: §7.2.1 · pp. 17–18

7.2.2 Predictors and metrics reported

  • The study reports test cross-entropy and zero-one error for stochastic, posterior-mean deterministic, and 100-sample majority-vote ensemble predictors.It also reports training empirical risks, posterior-prior KL, and PAC-Bayes-kl certificates for the stochastic predictor.Source: §7.2.2 · p. 18

7.2.3 Architectures

  • MNIST uses a 3-layer fully connected network and a 4-layer CNN; CIFAR-10 uses CNNs with 9, 13, or 15 learnable-parameter layers.For a reduced 10% MNIST subset, a figure plots risk certificates against test error for stochastic and deterministic predictors; Figure 1 covers model selection across more than 600 hyperparameter runs.Source: §7.2.3 · pp. 18–19, Figure 1 · p. 19

7.3 Hyper-parameter and Architecture Search through PAC-Bayes Bounds

On MNIST, risk certificates tracked stochastic test error and showed promise for selecting hyperparameters and fully connected architectures. The authors then used the ℓ01 certificate for subsequent hyperparameter tuning.

7.3 Hyper-parameter and Architecture Search through PAC-Bayes Bounds

  • Small-error MNIST runs showed relatively stable risk certificates that correlated positively with stochastic test 0-1 error, especially for the ℓ01 certificate.The hyperparameter search used 10% of the training data while retaining the test set; certificate variability increased toward higher error values.Source: §7.3 · pp. 19–20
  • Across fully connected MNIST architectures trained with fquad, the ℓ01 risk certificate followed a linear trend with test 0-1 error, supporting its use for model selection.The networks varied in layer and neuron counts while hyperparameters were fixed.Source: Figure 2 · p. 20, §7.3 · p. 20
  • The authors used the ℓ01 risk certificate for subsequent hyperparameter tuning, avoiding a held-out set for that purpose.This strategy followed the observed model-selection trends in Figures 1 and 2.Source: §7.3 · p. 20

7.4 Comparison of Different Training Objectives and Priors

Across MNIST experiments, fquad produced the tightest 0-1 risk certificates, while flambda generally achieved the best stochastic test performance. The methods also compared favorably with prior work, and data-dependent priors and CNNs improved results across the tested settings.

7.4 Comparison of Different Training Objectives and Priors

  • fquad consistently achieved the best 0-1 risk certificates, while flambda was the best PAC-Bayes objective for stochastic test performance but generally had looser certificates.The reported comparisons cover MNIST architectures, priors, and training objectives.Source: Table 1 · p. 21, §7.4 · pp. 21–22, Figure 3 · p. 22
  • Risk certificates guided model selection for all objectives except ferm, which used a validation set; fbbb selection chose a KL attenuating coefficient of 0.1.For fbbb, larger KL attenuating coefficients could yield better test performance but much looser certificates.Source: §7.4 · p. 21
  • Data-dependent priors and CNNs consistently improved test performance and risk certificates, while data-free priors still yielded non-vacuous certificates.PBB methods also showed competitive performance against fbbb and ferm.Source: Table 1 · p. 21, Figure 3 · p. 22, §7.4 · p. 23
  • 0.0279 risk certificate versus Dziugaite and Roy (2018)'s tightest 0.210; the PBB methods also achieved close to 98% test accuracy (0.0202 test error).The comparison used the same FCN architecture, though optimizers, priors, and weight initialization differed.Source: §7.4 · p. 23, Table 2 · p. 23

7.5 KL Attenuating Trick

With a small KL penalty during training, stochastic test error decreased while the KL term and final risk certificate increased. Laplace and Gaussian weight distributions showed no significant, consistent differences in certificate tightness or test error.

7.5 KL Attenuating Trick

  • A KL penalty of 0.0001 during training lowered stochastic-classifier 0-1 error but increased the KL term and final risk certificate.This trade-off was tested with a CNN; the authors advise weighing test performance against certificate tightness.Source: §7.5 · p. 24

7.6 Laplace Weight Distributions

  • Laplace and Gaussian weight distributions showed no significant, consistent differences in risk certificates or test error; both achieved good results on these measures.The distribution choice may depend on the problem.Source: §7.6 · p. 24, Table 4 · p. 25
  • For MNIST, flambda generally gave the lowest test error but worse certificates than fquad, while fclassic had the worst test performance and loosest bounds.fquad offered a reasonable trade-off, with a slightly curved relationship between ℓ01 certificate and stochastic test error.Source: §7.6 · pp. 24–25, Figure 4 · p. 25

7.7 CIFAR-10 with Larger Architectures

On CIFAR-10, the authors evaluate PAC-Bayes training objectives with three deep CNN architectures and data-dependent priors. The objectives yield non-vacuous certificates, with performance and certificate tightness varying by objective and prior-data amount.

7.7 CIFAR-10 with Larger Architectures

  • The evaluation used 9-, 13-, and 15-layer CNNs with about 6M, 10M, and 13M parameters, respectively; probabilistic networks doubled those counts.The authors kept training simple to isolate the objectives, while noting that more competitive, deeper CIFAR-10 baselines and advanced training strategies exist.Source: §7.7 · pp. 25, 28
  • 2 points in test 0-1 error for flambda when using 50% of the data for learning the prior; risk certificates were non-vacuous and tight, though less tight than for MNIST.Figure 5 presents certificate tightness for three architectures and priors learned from 50% or 70% of the data.Source: §7.7 · p. 26, Figure 5 · p. 27
  • flambda showed better test performance but less tight certificates; fclassic and fquad were closer in both measures, with fclassic sometimes tighter but often worse in test performance.The authors attribute fclassic's tighter bounds in some cases to the Pinsker inequality when true loss exceeds 0.25.Source: §7.7 · p. 26
  • On CIFAR-10, results using 70% of the data were very close to ERM across all three deep CNN architectures (9, 13, and 15 layers), with data-dependent PAC-Bayes priors.Source: Table 5 · p. 26, §7.7 · p. 27
  • The KL generally decreased as architectures deepened, especially from 9 to 13 layers, despite the larger parameter counts.The authors suggest smaller weight-update norms in higher-dimensional spaces may explain this pattern.Source: §7.7 · p. 28

7.8 Additional Miscellaneous Experiments

Additional experiments examine training duration, predictor types, posterior scales, and ensemble uncertainty. They find that posterior training can continue without apparent overfitting, while ensemble disagreement highlights unusual MNIST digits.

Training epochs for prior and posterior

  • Training both prior and posterior improved loss and risk certificates; a better-trained prior required fewer posterior epochs, while a random prior did not reach similarly competitive posteriors.Only a few prior-training epochs were needed for competitive posteriors; training the posterior for up to 1000 epochs did not worsen test error or certificate values.Source: §7.8 · pp. 28–29, Figure 7 · p. 29, Figure 8 · p. 30

Predictor comparison

  • For CIFAR-10, stochastic, deterministic, and ensemble predictors had similar test performance; on MNIST, the stochastic predictor performed significantly worse.Figure 9 compares the predictor types; the authors suggest, without establishing, that CIFAR-10 solutions may occupy a broad region of similarly good solutions.Source: §7.8 · p. 30, Figure 9 · p. 31

Posterior scale parameters

  • Optimisation changed posterior scale parameters, especially reducing those associated with the input and output layers.The authors propose exploring layer-specific scale initialisations and alternative covariance structures.Source: §7.8 · p. 31, Figure 10 · p. 32

Uncertainty quantification

  • The 100-member ensemble’s most uncertain MNIST digits looked unusual and could confuse a human, while its most certain digits were easily identifiable as 4, 6, and 9.The authors regard this visual experiment as suggestive for uncertainty quantification but note that more experiments are needed.Source: §7.8 · pp. 31–32, Figure 11 · p. 32

7.9 Further Discussion

The discussion highlights the practical potential of PBB predictors for risk certification and model selection without a held-out test set. It also frames uncertainty quantification and other probabilistic-model advantages as promising but not exhaustively tested.

Practical usefulness

  • PBB randomised predictors receive tight population-level performance guarantees evaluated on data used to train the posterior, so certificate evaluation does not require a held-out test set.The authors also report using the same bound for model selection and present self-certified learning as a data-efficient possibility.Source: §7.9 · p. 33

Probabilistic-model advantages

  • Probabilistic networks support uncertainty quantification and classification reject options; the paper’s ensemble experiment is one simple example of these potential advantages.The discussion also notes possible goals such as model pruning and distillation.Source: §7.9 · p. 33

Scope and future validation

  • The authors have not exhaustively tested whether these broader advantages extend to PBB-trained networks and say the hypotheses require thorough validation.They contrast the tight certificates studied here with bounds for point-estimator models, described as notoriously vacuous for these experiments.Source: §7.9 · p. 33

8. Conclusion and Future Work

PBB methods combine competitive test performance with non-vacuous, comparatively tight risk certificates, and the paper highlights data-dependent priors as useful for both. The authors frame the approach as promising for self-certified learning while identifying several directions for further study.

8. Conclusion and Future Work

  • PBB methods offer a theoretically grounded strategy with population-level guarantees for unseen data from the training distribution; data-dependent priors showed empirical usefulness for competitive performance and tight certificates.Source: §8 · p. 33
  • 1% MNIST test set error accompanied a 1.5% risk certificate; on CIFAR-10, a stochastic predictor had a 14.6% test set error and an 18% risk certificate.The authors caution that establishing self-certified learning would require broader experimental validation or theoretical characterization.Source: §8 · p. 34
  • Future work targets broader weight-distribution and dataset choices, KL-term dominance, deeper architectures, and risk certificates and methods for ensemble predictors.Source: §8 · p. 34
Loading 2007.12911v3…