Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification

1 Introduction

• Quantiles can represent complex distributions without parametric assumptions, use interpretable target-space units, form prediction intervals, and support inverse-transform sampling.These properties motivate quantile-based uncertainty quantification.Source: §1 · p. 1
• The paper proposes four approaches to improve quantile estimates, including model-agnostic learning, tunable calibration and sharpness, centered-interval calibration, and group batching.The methods are described as addressing limitations of pinball-loss optimization.Source: §1 · p. 2
• Model Agnostic QR uses conditional density estimation to learn all quantiles simultaneously and can be applied with any regression model.The proposed procedure addresses restrictions on model-family choice associated with pinball-loss optimization.Source: §1 · p. 2
• The proposed tunable loss targets calibration and sharpness separately, letting users choose their balance rather than relying on pinball loss's implicit balance.Source: §1 · p. 2
• The authors introduce training aimed at improving quantile predictions beyond average calibration and demonstrate its efficacy via adversarial group calibration.They also propose an alternative loss for centered intervals, which pinball loss does not target.Source: §1 · p. 2

2.1 Notation

• The section defines a quantile model as estimates of conditional quantiles across input values and quantile levels, under an assumed true conditional distribution.Source: §2.1 · p. 2

2.2 Assessing the Quality of Predictive UQ

• Calibration compares observed probabilities below predicted quantiles with their expected probabilities; sharpness measures prediction concentration, and average calibration can coexist with uninformative predictions.For non-parametric predictions, centered 95% interval width is one sharpness measure; average calibration and sharpness generally trade off.Source: §2.2 · pp. 2–3
• Adversarial group calibration requires average calibration within every positive-measure input subset, while individual calibration concerns each input and average calibration pools the input distribution.Individual calibration is typically unverifiable with finite data in the assumption-less case.Source: §2.2 · p. 3
• Pinball loss is a proper score combining calibration and sharpness, but its implied balance depends on model expressivity and can favor sharpness enough to yield miscalibrated uncertainty.The authors argue that sharpness should be optimized subject to calibration.Source: §2.2 · p. 4
• Pinball loss can decrease while test calibration worsens; in the synthetic example, SQR is sharper than the true noise level, while Cali is better calibrated but less sharp.The proposition states that a model with lower pinball loss can have worse average calibration; Figure 1 illustrates the practical pattern.Source: Figure 1 · p. 3, §2.2 · p. 4

3.1 Utilizing Conditional Density Estimation for Model Agnostic QR

• MAQR estimates conditional quantiles from nearby points, then regresses onto those estimates, allowing any regression model to target true conditional quantiles.It assumes nearby inputs have similar conditional distributions.Source: §3.1 · pp. 4–5

3.2 Explicitly balancing calibration and sharpness with the combined calibration loss

• The combined calibration loss pairs an objective minimized by true quantiles with a sharpness penalty, while λ explicitly sets the balance between them.On a finite dataset, the empirical calibration objective is minimized by an average-calibrated solution.Source: §3.2 · pp. 6–7

3.3 Encouraging calibration of centered intervals with the interval score

• The interval score is optimized across centered intervals and has expected score minimized at the true conditional quantiles.Training across all expected interval probabilities covers all quantile levels.Source: §3.3 · p. 7

3.4 Inducing adversarial group calibration with group batching

• Feature-based group batching takes gradient steps on deliberately grouped subsets, targeting adversarial group calibration rather than only average calibration.Uniform batches tend to preserve the marginal input distribution FX; deliberately grouped batches need not.Source: §3.4 · pp. 7–8

4 Experiments

• The evaluation covers 8 UCI datasets and 10 fusion targets, comparing MAQR, Cali, Interval, SQR, and mPAIC across calibration, sharpness, interval, and scoring metrics.The fusion targets have 468-dimensional inputs; MAQR is omitted there because of the computational costs and statistical challenge of high-dimensional nonparametric density estimation.Source: §4 · pp. 8–9, §4.1 · p. 9

4.1 UCI and Fusion Experiments

• MAQR has the best average calibration on 7 of 8 UCI datasets; Cali and Interval are competitive on 9 of 10 fusion datasets, while SQR is sharpest, often at calibration cost.On fusion datasets, SQR has worse average and adversarial group calibration except on betan.Source: §4.1 · pp. 9–10
• MAQR ranks best on UCI check score, interval score, and centered interval calibration; Cali and Interval produce competitive centered prediction intervals on 9 of 10 fusion datasets.The authors note that MAQR’s UCI result is notable because SQR and Interval explicitly minimize check and interval scores, respectively.Source: §4.1 · p. 10

4.2 Ablation Study: Effect of Group Batching

• With group batching, average calibration improves on 6 of 8 datasets for Cali and 7 of 8 for SQR, although sharpness tends to worsen.The ablation also reports improved adversarial group calibration; less sharp predictions may be desirable when the true distribution has high noise.Source: Figure 4 · p. 9, §4.2 · p. 10

5 Discussion

• The four proposed methods aim to improve quantile estimates, and a model-agnostic method can leverage existing regression procedures and training infrastructure for uncertainty quantification.The authors describe this as potentially useful without much additional overhead.Source: §5 · p. 10
• No single metric captures all aspects of uncertainty-procedure performance, motivating a holistic review and the Uncertainty Toolbox for evaluation, visualization, and recalibration.The toolbox includes the full suite of metrics discussed in the paper.Source: §5 · p. 10
• Modeling epistemic uncertainty is orthogonal to the paper’s goals; the discussion points to bootstrapped ensembles of its methods in Appendix G.Source: §5 · p. 10

A.1 Proof of Proposition 1

• An average-calibrated model can be replaced by one with lower pinball loss but worse average calibration at every quantile level.The proof constructs a model by changing one prediction while leaving the others fixed.Source: §A.1 · pp. 15–16

A.2 Proof of Proposition 2

• The true quantile minimizes the population calibration objective, while an average-calibrated solution minimizes its empirical counterpart on a finite dataset.Both minima are zero.Source: §A.2 · p. 16

A.3 Derivation of Gradients of Calibration Objective

• The calibration-objective derivation expresses the overprediction contribution using the conditional gap and its probability, with Y's CDF and PDF denoted FY and fY.Source: §A.3 · pp. 16–17

A.4 Optimum of Interval Score

• Setting the interval-score endpoint derivatives to zero places the interval bounds at their respective true quantiles.Source: §A.4 · p. 18

B.1 General Algorithm for Model Agnostic Quantile Regression

• The model-agnostic procedure estimates conditional quantiles to build a training dataset, then fits a regression model to those estimates.Any regression model can be used; gradient-boosted trees produced UCI performance described as very similar to the neural-network implementation.Source: §B.1 · p. 19

B.2 Algorithm Complexity

• Creating the quantile dataset takes O(N^2 log N) time and O(N^2) space in the stated worst case; fitting the regression model is not analyzed.Source: §B.2 · p. 19

• Each UCI dataset uses a 72%/18%/10% train/validation/test split, with inputs and targets centered and scaled to unit variance.Source: §C.1 · p. 20
• All methods share a two-layer, 64-unit ReLU network, learning rate 1e−3, batch size 64, and early stopping with a 10,000-epoch maximum.Source: §C.1 · p. 20
• IndvCal tunes α using Pareto-optimal test NLL and adversarial group calibration, while Cali tunes λ against adversarial group calibration.The group-batching frequency for Interval and Cali is also cross-validated using adversarial group calibration.Source: §C.1 · p. 20
• MAQR trains a mean model, constructs a quantile dataset, then fits a quantile model, tuning its distance threshold over k ∈ [10, 20, 30, 40, 50].Its quantile-model batch size is 1024 because the dataset can become large.Source: §C.1 · p. 20

C.2 Fusion Experiment Details

• The fusion task predicts next-state changes from a 468-dimensional feature vector built from 10 plasma-state variables and 8 neutral-beam action variables.Features summarize means, variances, and slopes over frames in 200 ms state and action windows.Source: §C.2 · pp. 20–21

C.5 Discussion of Compute and Resources

• The fusion data contain 100K training points and 10K each for validation and testing; one model took roughly one hour to converge.Fusion used three 100-unit hidden layers and batch size 500.Source: §C.2 · p. 21, §C.5 · p. 22

C.3 Group Batching Implementation Details

• The implemented group batching sorts points by one input dimension, batches consecutive points, and cycles through dimensions; it is described as simple, inexpensive, and effective.More elaborate threshold-based subsets can be computationally demanding as dimensionality increases.Source: §C.3 · pp. 21–22

C.4 Calculation of Evaluation Metrics

• Adversarial group calibration is estimated by recording the worst error among 20 random test groups at each of 10 group sizes from 1% to 100%.Sharpness is mean 95% centered-interval width, and check and interval scores are averaged over the test set.Source: §C.4 · p. 22

D Full Experimental Results

• The full-results section reports average calibration-sharpness, adversarial group calibration, check score, interval score, and centered interval calibration.Average calibration-sharpness is also visualized with calibration and sharpness on the axes.Source: D Full Experimental Results · p. 23

D.1 Full UCI Experiment Results

• On UCI, MAQR tends to achieve the best check score and interval score, and the best centered-interval calibration.Source: Figure 9 · p. 24, Figure 10 · p. 24, Figure 11 · p. 25

D.2 Full Fusion Experiment Results

• On fusion, Cali tends to lead in average calibration, SQR in sharpness, and Cali and Interval in adversarial group calibration; Cali and Interval achieve the best centered-interval calibration on 9 of 10 datasets.Source: Figure 12 · p. 26, Figure 14 · p. 26, Figure 17 · p. 27
• On fusion, SQR achieves the best check score, while Interval achieves the best interval score.Source: Figure 15 · p. 27, Figure 16 · p. 27

E.1 Details of Synthetic Experiment (Figure 1)

• The synthetic experiment relabels Boston-dataset outputs with a learned mean and adds zero-mean uniform noise whose support is 5% of the mean-value range.Source: §E.1 · p. 28

E.2 Regularizing the Pinball Loss

• Regularization of SQR tends to bias predictions further toward sharpness by shrinking the range of predictions across quantile levels.Source: §E.2 · p. 28

E.3 Comparison on 95% Prediction Interval Task

• MAQR consistently reached 0.95 coverage, including where baselines could not, and produced much sharper intervals when baselines matched coverage.On Naval, its mean prediction interval width was an order of magnitude smaller than all other baselines.Source: §E.3 · p. 29, Figure 19 · p. 30

E.4 Discussion on Recalibration [35]

• Recalibration generally improved average calibration but reduced sharpness, with recalibrated predictions moving upper-left in Figure 21.The discussion reports the same pattern across methods, including baselines.Source: §E.4 · p. 30, Figure 21 · p. 31
• Adversarial group calibration changed little except on Naval; proper scores improved on average, while centered interval calibration tended to worsen.Figures 22–25 show these metric-specific patterns.Source: §E.4 · p. 30, Figure 22 · p. 31, Figure 23 · p. 31
• Across these metrics, recalibration’s overall benefit remained inconclusive, though it may suit practitioners primarily concerned with average calibration.The authors say it does not seem a robust remedy when convergence to the true conditional distribution is the primary objective.Source: §E.4 · p. 30

F.1 Ablation Study Details

• The ablation compares group batching for Cali and baseline SQR, tuning batching frequency by cross-validation against uniform-draw batches without batching.The no-batching condition uses the training-data default of uniformly drawn batches.Source: §F.1 · p. 33

F.2 Full Ablation Study Experiment Results

• Group batching tended to improve average and adversarial group calibration while worsening sharpness for both Cali and SQR.Figures 27–30 report these patterns separately for the proposed Cali method and baseline SQR.Source: Figure 27 · p. 34, Figure 28 · p. 34, Figure 29 · p. 35
• Group batching generally worsened check score and split interval-score outcomes between Cali and SQR; changes were often insignificant except on Naval.The check-score pattern is attributed to worsened sharpness, which the score favors.Source: Figure 31 · p. 35, Figure 32 · p. 36
• Group batching improved centered interval calibration for both methods, significantly in several cases including Naval and Energy for both and Concrete with SQR.The overall change was often insignificant when standard errors were considered.Source: Figure 33 · p. 36

G.1 Sources for Epistemic Uncertainty

• The described epistemic ensemble trains quantile models with random initializations on bootstrapped training samples, addressing parameter and data uncertainty.The discussion treats model misspecification as ignorable given rich function approximators.Source: §G.1 · p. 37

G.2 Expressing and Utilizing Epistemic Uncertainty

• For a centered total-uncertainty interval, bootstrap dispersion widens the upper and lower bounds to form conservative limits.The construction uses a chosen critical value for the bootstrap distribution.Source: §G.2 · pp. 37–38

G.3 Metrics for Total Uncertainty Evaluation

• Epistemic coverage evaluates sharpness subject to sufficient coverage, penalizing undercoverage but accepting overcoverage.The paper notes that standard regression-setting experiments and metrics for epistemic uncertainty do not exist, and that ordinary distribution-fit metrics can reward a lucky prediction.Source: §G.3 · p. 38

G.4 Demonstrating Effect of Bootstrap Ensembling and the Epistemic Coverage Metric

• The epistemic experiment swaps train and test sets, leaving the training set roughly one-seventh the test-set size.The design is intended to create high epistemic uncertainty on the test set.Source: §G.4 · pp. 38–39
• Bootstrap ensembling reduced epistemic coverage error significantly to or near zero, while increasing interval width.The experiment uses Interval Boot-Ens; the metric penalizes intervals that are too narrow.Source: §G.4 · p. 39, Figure 34 · p. 39
• Quantifying and evaluating epistemic uncertainty in regression remains open, and alternative methods are left for future work.The reported practical effect of ensembling is to make distributional predictions more underconfident.Source: §G.4 · p. 39

H Additional Discussions

• The authors acknowledge that expanded autonomous-system use may have disruptive effects in relevant industries.They connect this concern to developing robust and accurate uncertainty-quantification methods.Source: H Additional Discussions · p. 40
• Calibration information could be used with ill intent, including to intentionally avoid calibrated or fair decisions.The paper notes calibration’s relevance to fairness.Source: H Additional Discussions · p. 40