PaperlayerView exact revision on arXiv

Source-linked AI summary

Tight Prediction Intervals Using Expanded Interval Minimization

Dongqi Su, Ying Yin Ting, Jason Ansel

arXiv:1806.11222v1cs.LGstat.ML

TL;DR

Prediction intervals must achieve target coverage without becoming unnecessarily wide. EIM is a neural-network technique for generating prediction intervals, evaluated after scaling model outputs to match target coverage. Across the two datasets, EIM produced tighter intervals than other techniques; it was 1.06x tighter than the next-best technique at 90% coverage.

  • Problem

    Prediction intervals must achieve target coverage without becoming unnecessarily wide.

    Source: §1 · p. 1

  • Method

    EIM is a neural-network technique for generating prediction intervals, evaluated after scaling model outputs to match target coverage.

    Source: §5.3 · p. 6, §7 · p. 8

  • Results

    Across the two datasets, EIM produced tighter intervals than other techniques; it was 1.06x tighter than the next-best technique at 90% coverage.

    Source: §6 · p. 7, §7 · p. 8

  • Takeaways & Limitations

    The authors describe EIM as a natural fit for applications with asymmetrically distributed error.

    Source: §7 · p. 8

    The maximum-likelihood approach assumes residual noise ϵ(x) follows a normal distribution.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Prediction intervals are a valuable way of quantifying uncertainty in regression problems. Good prediction intervals should be both correct, containing the actual value between the lower and upper bound at least a target percentage of the time; and tight, having a small mean width of the bounds. Many prior techniques for generating prediction intervals make assumptions on the distribution of error, which causes them to work poorly for problems with asymmetric distributions. This paper presents Expanded Interval Minimization (EIM), a novel loss function for generating prediction intervals using neural networks. This loss function uses minibatch statistics to estimate the coverage and optimize the width of the prediction intervals. It does not make the same assumptions on the distributions of data and error as prior work. We compare to three published techniques and show EIM produces on average 1.37x tighter prediction intervals and in the worst case 1.06x tighter intervals across two large real-world datasets and varying coverage levels.

1 Introduction

Prediction intervals must be both correct at a target coverage and tight, but Gaussian-motivated baselines struggle with asymmetric distributions. EIM directly predicts both bounds and, across two datasets, reports tighter intervals than prior techniques.

1 Introduction

  • Useful prediction intervals must meet a target coverage while minimizing mean bound width; infinite bounds satisfy coverage but are uninformative.The paper uses three published neural-network techniques as baselines.Source: §1 · p. 1
  • Maximum-likelihood and ensemble methods are motivated by Gaussian error assumptions and struggle with asymmetric distributions in the datasets.Maximum likelihood uses separate networks for the predicted value and error; ensembles use variance across models.Source: §1 · p. 1
  • Quantile regression avoids a specific distribution assumption, but restricts intervals to spans between two chosen quantiles.The paper selects those quantiles by exhaustive grid search for each target coverage.Source: §1 · p. 1
  • EIM directly outputs lower and upper bounds and scales them to target minibatch coverage before minimizing their width.The scaling is described as ensuring coverage on training data while optimizing tightness.Source: §1 · pp. 1–2
  • On the Domain Valuation Dataset and Million Song Dataset, EIM produces at least 1.26x tighter prediction intervals than the next-best method at 70% coverage, 1.21x at 80%, and 1.06x at 90%.Source: §1 · p. 2

2 Related Work

The paper reviews interval methods with different assumptions and flexibility, then implements maximum likelihood, ensemble, quantile regression, and fixed bounds as comparison baselines.

2 Related Work

  • The reviewed methods include delta, variance-prediction, and quantile-regression approaches; the delta method is excluded because its constant-variance and homogeneous-noise assumptions do not fit the datasets.The paper implements maximum likelihood, ensemble, quantile regression, and fixed bounds as baselines.Source: §2 · p. 2

2.4 Fixed Bounds Method

  • The experiments include fixed bounds as a naive baseline alongside the proposed and learned interval methods.The paper characterizes this baseline as a ceiling representing a trivial score without an advanced technique.Source: §2 · p. 2, §2.4 · p. 3

2.1 Maximum Likelihood Method

  • Maximum likelihood uses separate networks for regression and total prediction variance, then forms bounds using a confidence-dependent factor motivated by Gaussianity.The two networks are trained on disjoint halves of the training data.Source: §2.1 · p. 2

2.2 Ensemble Method

  • The ensemble method uses bootstrap models to estimate regression and variance, and its residual-noise model assumes normally distributed errors.The implementation uses M = 200 bootstrap networks, M2 = 8 groups, and P = 1000 resamples.Source: §2.2 · p. 3

2.3 Quantile Regression Method

  • Quantile regression estimates a conditional target quantile and constructs bounds from lower and upper quantile models, with the quantiles selected by exhaustive grid search.The paper reports that the simple 10%/90% choice was suboptimal.Source: §2.3 · p. 3

3 Assessment Metrics

The paper assesses intervals along two competing goals: achieving target coverage and minimizing mean width. Its evaluation scales model outputs to make coverage comparisons exact.

3 Assessment Metrics

  • Prediction interval coverage probability (PICP) measures how often an interval contains the observed target, while mean prediction interval width (MPIW) measures average interval size.The text also uses MPI as a name for mean prediction interval width.Source: §3 · pp. 3–4
  • For target T, the objective is to minimize MPIW while maintaining PICP ≥ T.The paper notes that some literature treats results below the target coverage as invalid.Source: §3 · p. 4
  • The evaluation linearly scales each technique’s output by a constant factor so it reaches the target PICP exactly.This is used to compare methods at matching coverage levels.Source: §3 · p. 4

4 Expanded Interval Minimization

EIM optimizes neural-network interval bounds by using minibatch statistics to estimate coverage and scale bounds to the target before minimizing width. Its training procedure uses larger batches, optional fixed-bound pretraining, and a final holdout-set scaling step.

4 Expanded Interval Minimization

  • Directly optimizing network parameters for lower and upper interval bounds is difficult and expensive, motivating EIM’s minibatch-based objective.The objective is to produce bounds that satisfy target coverage while minimizing width.Source: §4 · p. 4
  • EIM treats minibatch coverage and width as noisy estimates of population metrics, then scales bounds so minibatch PICP equals target T before minimizing width.A minibatch-specific scaling factor expands or shrinks the predicted bounds to reach the target coverage.Source: §4 · p. 4
  • The method finds each instance’s minimum scale to cover its target, then uses the Tth-percentile scale so the minibatch reaches PICP T.A percentile here selects the value at the corresponding position in the sorted set.Source: §4 · p. 4
  • Averaging scale values within δ of the Tth percentile addresses the single-instance gradient problem; the paper uses δ from 1% to 3%.The original formulation gives nonzero loss gradients to only one minibatch value.Source: §4 · p. 5

4.1 Training Procedure and Testing

  • Larger minibatches stabilize coverage estimates, fixed-bound pretraining helps prevent early divergence, and holdout-set scaling sets production outputs to target PICP.The final scaling factor grows or shrinks future model outputs around their center.Source: §4.1 · p. 5

5 Evaluation Datasets

The evaluation uses domain-name sale prices and song release years, scales intervals to exact target coverage for fair comparison, and adapts network outputs for interval prediction.

5 Evaluation Datasets

5.1 Aftermarket Domain Valuation Dataset

  • The Domain Valuation task predicts aftermarket sale prices from domain names, using a tokenizer and language model built from word embeddings and crawled-site tokenizations as an input.Source: §5.1 · p. 5

5.2 Year Prediction Million Song Dataset

  • The Million Song subset predicts song release year from 90 audio features: 12 timbre averages and 78 timbre covariances.Source: §5.2 · p. 6

5.3 Scaled Intervals

  • For fair comparison, each model's raw intervals are scaled to exact target PICP; the required scaling factor shrinks or expands the intervals.Source: §5.3 · p. 6

5.4 Dataset Specific Neural Network Structures

  • Figures 2(a,c) show point-output structures for fixed bounds, MLE, and ensembles; (b,d) show modified structures outputting both bounds for EIM and quantile regression.Source: Figure 2 · p. 6, §5.4 · p. 6

6 Results and Discussion

EIM produces tighter prediction intervals than the compared techniques, with the advantage varying by target coverage. Its intervals adapt to data distributions and target levels, while its single-network training is simpler than ensemble or quantile-regression implementations.

6 Results and Discussion

  • Across both datasets, EIM yields 1.33x–2.23x tighter bounds than fixed bounds, 1.26x–2.1x than MLE, 1.21x–1.61x than ensemble, and 1.06x–1.52x than quantile regression.Against the next-best technique, EIM is 1.26x tighter at 70% coverage and 1.06x tighter at 90%.Source: §6 · p. 7
  • Target-specific EIM and quantile regression outperform methods supporting only symmetric bounds; each EIM version specializes to its trained PICP target.MPIW changes when versions trained for one PICP target are scaled to other targets on the Domain Valuation Dataset.Source: §6 · p. 7, Figure 4 · p. 7
  • On lower-concentrated datasets, EIM shifts its prediction distribution more across target percentiles; on other datasets, EIM and quantile-regression densities skew toward the data mass.Figure 5 shows target histograms and normalized densities for how often each technique's predicted range contains each value.Source: §6 · p. 8, Figure 5 · p. 8
  • As target coverage rises, prediction distributions spread and methods cluster more closely; mean shifts are most pronounced for EIM.Differences between techniques are larger at lower PICP targets.Source: §6 · p. 8
  • EIM is fastest and simplest, requiring one network; ensembles require 200 networks, while quantile regression's two grid-searched hyperparameters make it slowest to train.For Million Song, grid search chose quantile pairs (0.2, 0.65), (0.2, 0.75), and (0.5, 0.75) at 70%, 80%, and 90% PICP; scaled choices outperformed naive symmetric choices.Source: §6 · p. 8

7 Conclusions

The conclusion presents EIM as a neural-network approach to prediction intervals, with tighter intervals than the next-best technique at the tested coverage targets. It identifies asymmetrically distributed error as a setting for which EIM is a natural fit.

7 Conclusions

  • At 70%, 80%, and 90% coverage, EIM produces intervals respectively 1.26x, 1.21x, and 1.06x tighter than the next-best technique.Source: §7 · p. 8
  • The authors describe EIM as a natural fit for prediction-interval applications with asymmetrically distributed error.Source: §7 · p. 8
  • The authors hope EIM will help others generate tighter prediction intervals and advance what deep learning machines can do.Source: §7 · p. 8
Loading 1806.11222v1…