PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach

Tim Pearce, Mohamed Zaki, Alexandra Brintrup, Andy Neely

arXiv:1802.07167v3stat.ML

TL;DR

Neural networks need to quantify uncertainty for individual regression predictions, not merely achieve strong average accuracy. The paper derives a distribution-free loss from the principle that prediction intervals should be narrow while meeting target coverage, and uses ensembles to account for model uncertainty. Across ten benchmark datasets, the method generally improved prediction-interval quality and reduced average width by 11.6%, with a training-process fragility caveat.

  • Problem

    Neural networks often perform well on aggregated accuracy metrics, but many applications require uncertainty quantification for each prediction.

  • Method

    The paper derives a loss directly from the narrow-interval, target-coverage principle and uses ensembles to account for model uncertainty and data noise variance without distributional assumptions.

  • Results

    11.6% narrower MPIW on average; QD-Ens outperformed MVE-Ens on all but one dataset, with PICP generally closer to the 95% target.

  • Takeaways & Limitations

    On ten benchmark regression datasets, the ensemble method reduced prediction-interval widths by over 10% while generally improving coverage alignment.

  • Takeaways & Limitations

    QD-Ens required a lower learning rate, greater sensitivity to decay rate, and two to ten times more training epochs than MVE-Ens.

Abstract

from arXiv · show

This paper considers the generation of prediction intervals (PIs) by neural networks for quantifying uncertainty in regression tasks. It is axiomatic that high-quality PIs should be as narrow as possible, whilst capturing a specified portion of data. We derive a loss function directly from this axiom that requires no distributional assumption. We show how its form derives from a likelihood principle, that it can be used with gradient descent, and that model uncertainty is accounted for in ensembled form. Benchmark experiments show the method outperforms current state-of-the-art uncertainty quantification methods, reducing average PI width by over 10%.

1. Introduction

Neural networks provide strong average accuracy, but many applications also require uncertainty for each prediction. The paper develops QD, including QD-Ens, to produce narrow, well-covered prediction intervals without distributional assumptions.

  • Motivation: Prediction intervals provide lower and upper bounds with a specified probability of containing the realised data point.They can support better-informed decisions when point estimates alone are insufficient.
  • Related limitations: Bayesian and dropout-based uncertainty methods can require high computational demand or strong assumptions.
  • Proposed method: QD formulates prediction-interval output as constrained optimisation, minimising width while targeting a specified coverage proportion.Its objective follows the high-quality PI principle.
  • Proposed method: QD derives a loss function from the high-quality PI principle and uses an ensemble to account for model uncertainty and data noise variance.
  • Limitations addressed: LUBE accounts only for data-noise variance, implicitly assuming training data fully populates the input space.The paper identifies this as an oversimplification because that assumption is seldom satisfied.
  • Evaluation: The paper reports that QD outperforms the compared method on PI quality metrics, achieving closer target coverage and reducing average PI width by around 10%.The comparison uses ten benchmark regression datasets and MVE-Ens as the competing model.

2. Related Work

Related work quantifies neural-network uncertainty through confidence or prediction intervals, probabilistic outputs, resampling, Bayesian methods, and ensembles. These approaches differ in their treatment of model uncertainty, data noise, computational cost, and evaluation metrics.

  • Classical and probabilistic methods: The Delta method estimates model uncertainty using nonlinear-regression confidence-interval theory but requires the computationally demanding Hessian matrix.
  • Classical and probabilistic methods: MVE uses neural-network outputs for the mean and variance of a normal distribution to estimate data-noise variance.Its loss is the negative log likelihood of the predicted distribution given the data.
  • Resampling and ensembles: Bootstrap estimates model uncertainty by training multiple networks with different initialisations on resampled training datasets.It can be combined with MVE to estimate total variance.
  • Bayesian methods: Bayesian neural networks treat model parameters as distributions and can predict distributions, but Markov chain Monte Carlo may be prohibitively costly.Dropout has also been interpreted as variational inference.
  • Resampling and ensembles: MVE-Ens modernises earlier ensemble work by combining individual MVE networks without dataset resampling and adding adversarial training examples.
  • Evaluation conventions: Modern uncertainty studies commonly evaluate NLL and RMSE, whereas PI studies report coverage proportion and PI width.
  • Prediction-interval methods: LUBE applies the high-quality PI principle but has mainly used simulated annealing and other non-gradient-based training methods.Its application-focused use has not accounted for model uncertainty in single-network implementations.
  • Uncertainty framework: Regression uncertainty includes model uncertainty in the estimated function and irreducible data noise, which may be constant or input-dependent.Prediction intervals therefore account for uncertainty in observations, unlike confidence intervals focused on the function estimate.

3. A Quality-Driven, Distribution-Free Loss Function

The paper derives a distribution-free loss from the principle that prediction intervals should be narrow while meeting target coverage, then adapts it for gradient descent and uncertainty estimation.

  • 3.1. Derivation: Prediction intervals should minimise mean width subject to capturing at least the desired proportion, 1−α, of observations.Captured width is used so missed observations do not encourage further interval shrinkage.
  • 3.1. Derivation: The coverage model treats each capture indicator as an iid Bernoulli variable, making total coverage binomial and enabling a likelihood-based penalty.The iid assumption is considered reasonable for randomly sampled mini-batches, though clustered points may violate it.
  • 3.1. Derivation: The resulting loss combines captured interval width with a one-sided squared coverage penalty weighted by λ.The penalty applies when PICP falls below 1−α; n and α scale the coverage term.
  • 3.2. Comparison to LUBE: Compared with LUBE, the loss uses additive width, likelihood-derived squared coverage, unnormalised width, and captured MPIW to avoid zero-width or missed-point incentives.The additive form addresses LUBE’s global zero-width minimum, while captured MPIW prevents shrinking intervals for missed observations.
  • 3.3. Training QD with Gradient Descent: The hard capture indicators make direct LossQD gradient descent fail because the weight shrinks without convergence, so sigmoid-softened indicators define LossQD−soft.With an appropriate softening factor s, steepest descent reaches a minimum; s = 160 worked across the reported datasets.

4. Ensembles to Estimate Model Uncertainty

The paper uses ensembles to estimate model uncertainty alongside data noise variance, addressing uncertainty that single QD networks underestimate. Parameter and training-data resampling create predictive diversity whose variance estimates model uncertainty.

  • Motivation: LUBE and QD estimate data noise variance but not uncertainty in those estimates, leaving predictions poorly grounded for unfamiliar inputs.The paper illustrates this with a network that estimates noise at x = 1.0 but predicts an unsupported upper bound at x = 2.0.
  • Ensemble approach: Ensembling addresses this gap by training multiple neural networks with varied parameter initialisations or resampled training subsets.Parameter resampling targets parameter uncertainty, while bootstrap resampling targets training-data uncertainty.
  • Resampling choice: Previous reports found parameter resampling superior to bootstrap resampling or their combination, but offered no robust justification.This comparison motivates caution when selecting the resampling strategy.
  • Ensemble approach: The variance of predictions across a diverse neural-network ensemble provides an estimate of model uncertainty.The ensemble produces upper and lower PI estimates by combining individual bounds with their uncertainty.

5. Qualitative Experiments

Qualitative experiments compare gradient descent with PSO, QD with MVE under different noise distributions, and QD ensembles for model uncertainty. The experiments show advantages for GD, distribution-free QD under exponential noise, and ensemble uncertainty behaviour in sparsely represented input regions.

  • 5.1. Training method: PSO vs. GD: GD produced slightly better PICP and MPIW than PSO, with smoother and tighter PI boundaries, more consistency, and lower computational effort.The comparison used synthetic data generated with heteroscedastic normal noise, ϵ ∼N(0, x4).
  • 5.2. Loss function: QD vs. MVE: QD makes no distributional assumption, whereas MVE assumes Gaussian noise; experiments therefore compare them under normal and exponential synthetic noise.The exponential-noise dataset used ϵ ∼ exp(1/x2).
  • 5.2. Loss function: QD vs. MVE: MVE closely matched ideal PIs for normal noise but struggled with exponential noise, while QD approximated both cases reasonably.QD did not learn boundaries well where data was sparse.
  • 5.3. Model uncertainty: A QD ensemble estimated model uncertainty whose behaviour increased in input regions absent from training data.The training region was x ∈[−1, 1], while the highlighted unrepresented regions were x > 4 and x < −4.

6. Benchmarking Experiments

Benchmarking evaluates QD-Ens against MVE-Ens across ten datasets using prediction-interval metrics, with comparisons designed around each method’s output type. QD-Ens generally achieved closer 95% coverage and narrower intervals, but required a more fragile and expensive training process.

  • Experimental setup: The benchmark followed prior uncertainty-evaluation procedures across ten open-access datasets, using 95% PIs and ensembles of five neural networks.NLL and RMSE were reported in the appendix, while the main comparison focused on PI quality metrics.
  • Comparison methodology: QD-Ens outputs PI bounds directly, whereas MVE-Ens outputs distributions, so NLL and RMSE comparisons for QD-Ens require imposing a distribution.For a fairer comparison, MVE-Ens distributions were converted into PIs and evaluated using MPIW, PICP, and LossQD−soft.
  • Results: QD-Ens outperformed MVE-Ens on all but one dataset, with PICP generally closer to 95% and MPIW averaging 11.6% narrower.The exception was Kin8nm, where the authors suspect Gaussian simulation noise may explain MVE-Ens’s superior performance.
  • Limitations: QD-Ens training was more fragile than MVE-Ens, requiring a lower learning rate, greater sensitivity to decay rate, and two to ten times more epochs.This is the main reported training-cost drawback of the benchmarked method.
  • Discussion: Bootstrap resampling performed worse than parameter resampling, although the authors suspect larger ensembles could improve bootstrap performance.Attempts to relate residual normality to QD-Ens improvement over MVE-Ens were unreliable because normality tests had variable power.

7. Conclusions and Future Work

The paper derives a distribution-free PI loss from the principle of narrow intervals subject to target coverage, enables gradient-descent training, and uses ensembles to address model uncertainty. Across ten benchmark datasets, the resulting model reduced PI widths by over 10%, while several questions remain open for future work.

  • Conclusions: The proposed loss function directly encodes the principle that high-quality PIs should be narrow while meeting a specified coverage proportion.The paper contrasts this formulation with prior work and reports successful use with gradient descent after slight modification.
  • Conclusions: The paper argues that a single QD network underestimates uncertainty and addresses this limitation by using the model in an ensemble.The ensemble incorporates model uncertainty alongside PI estimation.
  • Conclusions: Over 10%: the new model reduced prediction-interval widths on ten benchmark regression datasets.This is the paper’s headline benchmark result.
  • Future work: Future work includes explaining resampling differences, estimating model uncertainty with dropout or conformal prediction, and studying neural-network architecture effects.These are identified as areas requiring further investigation.

A. Experimental details

The experiments used single-layer neural networks for consistency with previous work, while the developed methods can also be applied to deeper architectures without modification.

  • The experiments used single-layer neural networks to remain consistent with previous work.
  • The developed methods may be applied to deeper architectures without modification.

A.1.1. TRAINING METHOD: PSO VS. GD

The qualitative comparison examined particle swarm optimisation and gradient descent using neural networks trained on uniformly sampled synthetic data.

  • The comparison evaluates PSO versus GD for training neural networks.
  • The networks used ReLU activations and 50 nodes in one hidden layer.
  • GD used LossQD−soft for 2,000 epochs, while PSO used 50 particles over 2,000 iterations.
  • The data consisted of 200 points sampled uniformly from [−2, 2].

A.1.2. LOSS FUNCTION: QD VS. MVE

The experiments compare loss functions and training methods on synthetic and benchmark regression data, including individual and ensembled neural networks and metrics with stated caveats.

  • The synthetic experiments tested permutations of LUBE, QD, and MVE with GD and PSO using individual neural networks.
  • The synthetic-data setup used 200 training points, 2,000 validation points, ten repetitions, and a target coverage of 95%.
  • GD and PSO used softened loss versions, with 2,000 epochs and five-times greater computational effort for PSO than GD.
  • NLL and RMSE for LUBE and QD should be interpreted cautiously, while slight overfitting produced PICP values below 95%.
  • MVE produced significantly wider prediction intervals than LUBE and QD under exponential noise, but comparable widths under normal noise.
  • The benchmarking experiments covered ten open-access datasets with random 90%/10% train/test splits and repeated trials.
  • Five neural networks formed each ensemble through parameter resampling, with Adam optimisation used for training.
  • NLL for QD-Ens was computed from an equivalent Gaussian distribution, breaking the prediction intervals’ distribution-free assumption for comparison purposes.
Loading 1802.07167v3…