PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Calibrated Probabilistic Forecasts for Events of Interest via Outcome-Conditional Recalibration

Jakob Benjamin Wessel, Sam Allen

arXiv:2610.10076v1stat.MLcs.LG

TL;DR

Standard recalibration can leave forecasts miscalibrated for outcomes of particular interest, despite being calibrated overall. The paper introduces outcome-conditional recalibration, which recalibrates conditional forecast distributions and rescales their event probabilities. Across regression benchmarks and electricity price forecasting, it improves calibration for outcomes of interest while retaining competitive overall calibration.

  • Problem

    Standard recalibration may be miscalibrated when evaluation is restricted to particular outcomes, such as extreme events important for decision making.

    Source: §1 · pp. 1–2

  • Method

    Outcome-conditional recalibration applies quantile recalibration to conditional forecast distributions on user-defined regions, then rescales them to match empirical event frequencies.

    Source: §1 · p. 2

  • Results

    Across regression benchmarks, existing methods do not necessarily calibrate forecasts for particular outcomes, while the proposed approach improves outcome-conditional calibration and retains competitive overall calibration.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The method yields valid, continuous forecasts calibrated within regions of interest, including reliable predictions of negative electricity prices where alternative recalibration methods are not calibrated.

    Source: §1 · p. 2, §5.3 · p. 7

    Outcome-conditional recalibration does not guarantee probabilistic calibration overall, regardless of the intervals used.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Calibration is an essential requirement for probabilistic predictions to be useful for decision making. While state-of-the-art prediction methods often yield miscalibrated predictive distributions, several post-hoc recalibration schemes have been proposed to generate calibrated predictions. However, popular recalibration schemes can conceal miscalibration in specific regions of the outcome space. Since particular outcomes, such as extreme events, often matter most for decision making, probabilistic predictions should be calibrated when evaluation is restricted to these outcomes. Hence, in this paper, we introduce outcome-conditional recalibration, a post-hoc method to recalibrate probabilistic predictions on user-defined regions of the outcome space. The method is simple, easy to implement, and can be applied to arbitrary predictive distributions. It works by applying the quantile recalibration approach of Kuleshov et al. (2018) to forecast conditional distributions, before rescaling these conditional distributions so that forecast event probabilities match empirical occurrence frequencies. This produces valid and continuous predictive distributions that are calibrated within each region of interest. Across regression benchmarks, we demonstrate that existing recalibration schemes do not necessarily yield calibrated predictions when interest is on particular outcomes, and that our approach improves outcome-conditional calibration relative to existing conditional and unconditional recalibration methods, while retaining competitive calibration overall. In an application to day-ahead electricity price forecasting, the approach substantially improves calibration when predicting negative prices, at negligible cost to forecast accuracy.

1 INTRODUCTION

Standard recalibration can produce forecasts that are calibrated on average but miscalibrated for outcomes of particular interest. The paper introduces outcome-conditional recalibration and reports improved calibration for such outcomes, including negative electricity prices.

1 INTRODUCTION

  • Standard recalibration may leave predictions miscalibrated when evaluation is restricted to particular outcomes, including extreme events that matter for decision making.Unconditional calibration aligns probabilities and frequencies on average but does not ensure calibration on an outcome subset.Source: §1 · pp. 1–2
  • Outcome-conditional recalibration applies quantile recalibration to conditional forecast distributions on outcome-space subsets, then rescales them to match empirical event frequencies.The resulting distributions are valid, continuous, and calibrated within the selected regions.Source: §1 · p. 2
  • Across regression benchmarks, standard and stronger conditional recalibration methods do not necessarily calibrate extreme-event forecasts, while the proposed method improves extreme-event calibration and remains competitive overall.For electricity prices, it yields reliable predictions of negative prices where alternatives do not.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries establish the forecast and outcome notation, then position outcome-conditional recalibration among existing approaches. Prior work had not considered calibration conditional on observed outcomes, while related conditional methods rely on modeling assumptions.

2.1 Notation

  • A probabilistic forecast F is a cumulative distribution function for scalar outcome Y, treated as random through its dependence on covariates; forecast–observation pairs may come from training or calibration data.The forecast PIT is F(Y).Source: §2.1 · p. 2

2.2 Related Work

  • Prior recalibration work includes quantile recalibration and training-time regularization, with extensions to multivariate forecasts and stronger conditional notions.Covariate-conditional approaches target calibration conditional on forecasts but typically require fairly strong modeling assumptions and lack finite-sample guarantees.Source: §2.2 · pp. 2–3
  • Earlier studies had not considered calibration conditional on observed outcomes; the proposed post-hoc scheme targets that notion and provides theoretical in-sample calibration guarantees.Wessel et al. (2026) instead use tail-miscalibration regularization, which only weakly enforces tail calibration.Source: §2.2 · p. 3

3 CALIBRATION

Probabilistic calibration concerns the forecast PIT, whereas outcome-conditional calibration assesses calibration within a chosen outcome region and whether its event probability is correctly forecast. The stronger auto-calibration notion implies outcome-conditional calibration but its recalibration methods require restrictive modeling assumptions.

3 CALIBRATION

  • Probabilistic calibration requires the forecast PIT F(Y) to follow a standard uniform distribution, corresponding to correct unconditional prediction-interval coverage.This is the conventional calibration notion described in the section.Source: §3 · p. 3
  • Outcome-conditional calibration on a region requires both a calibrated conditional forecast distribution there and agreement between predicted and empirical region-occurrence probabilities.The two requirements are called CPIT calibration and occurrence calibration.Source: §3 · p. 3
  • Probabilistic calibration generally does not imply outcome-conditional calibration, so standard recalibration may not yield calibrated predictions for a chosen outcome region.The section motivates outcome-conditional recalibration on this basis.Source: §3 · p. 3
  • Auto-calibration implies outcome-conditional calibration, but methods designed to enforce it require somewhat restrictive modeling assumptions.The paper reports that its method improves outcome-conditional and global probabilistic calibration relative to these methods.Source: §3 · p. 3

4 FROM GLOBAL TO LOCAL RECALIBRATION

The section first describes quantile recalibration as a global transformation of the forecast CDF, then extends it across outcome-space regions. The regional method recalibrates conditional forecasts and rescales their probabilities, yielding valid, continuous distributions calibrated on the selected intervals.

4.1 Quantile Recalibration

  • Quantile recalibration transforms a forecast CDF using a function estimated from forecast PIT values; the transformed forecast is calibrated by construction.In practice, the transformation is estimated from a finite sample, using isotonic regression in the described implementation.Source: §4.1 · p. 4

4.2 Outcome-conditional Recalibration

  • The regional method rescales predicted event probabilities to match occurrence calibration while appropriately handling probability mass outside the region.It applies the conditional recalibration separately to intervals partitioning the outcome space.Source: §4.2 · p. 4
  • Outcome-conditional recalibration applies quantile recalibration separately to conditional forecasts on each interval and combines them using rescaling weights.The resulting piecewise distribution is valid, continuous, and outcome-conditionally calibrated on the intervals by construction.Source: §4.2 · p. 4
  • The intervals may be any partition of the outcome space, and disconnected subsets can also be used; more intervals generally improve overall calibration when sufficient data are available.The forecasting problem can guide interval selection, such as focusing on negative electricity prices.Source: §4.2 · p. 5
  • Outcome-conditional recalibration does not guarantee probabilistic calibration overall, regardless of the chosen intervals.A global-local variant applies quantile recalibration after the regional method, but does not necessarily preserve outcome-conditional calibration.Source: §4.2 · p. 5

5 EXPERIMENTS

The experiments compare outcome-conditional recalibration with global and covariate-conditional methods on regression benchmarks and German day-ahead electricity prices. Outcome-conditional recalibration improves calibration within regions of interest, while its overall calibration and accuracy trade-offs vary by setting.

5.2 UCI and OpenML experiments

  • Across UCI and OpenML datasets (N = 768–45,730), results are averaged over 10 stratified random splits with 75% training and 25% testing, using five outcome intervals.For DDNN predictions, recalibration parameters are estimated weekly using a trailing 182-day window of past out-of-sample predictions and applied across the following week.Source: §5.2 · p. 5, §5.3 · p. 7
  • Outcome-conditional recalibration outperforms the other methods in occurrence error and outcome-conditional calibration error across all benchmark datasets and both prediction models.Figure 1 summarizes outcome-conditional and overall calibration errors across datasets and models.Source: Figure 1 · p. 6, §5.2 · p. 6
  • On benchmarks, outcome-conditional recalibration has slightly worse overall calibration than quantile recalibration, while adding quantile recalibration gives the best overall calibration without a large outcome-conditional sacrifice.The discussion reports good overall calibration and transfer of outcome-conditional calibration to other partitions; covariate-conditional recalibration is competitive but uses a slower density fit, whereas outcome-conditional recalibration is hyperparameter-free and needs no additional validation dataset.Source: §5.2 · p. 6

5.3 Electricity Price Forecasting

  • For German day-ahead prices, outcome-conditional recalibration yields good overall calibration and calibration for negative prices, unlike quantile and covariate-conditional recalibration.The raw DDNN is not calibrated for negative prices, and neither unconditional quantile nor covariate-conditional recalibration corrects this.Source: §5.3 · p. 7
  • For negative prices, other methods have roughly correct average occurrence probabilities but CPIT patterns suggest they over-predict severity; all methods have very good overall calibration.Figure 2 shows the PIT and negative-group CPIT diagnostics alongside event-occurrence ratios.Source: §5.3 · p. 7, Figure 2 · p. 8
  • Recalibration methods have slightly worse average CRPS than the raw DDNN, although a small accuracy decrease may be tolerable when calibration improves for high-impact outcomes.The raw DDNN has the most accurate predictions by average CRPS.Source: §5.3 · pp. 7–8

6 DISCUSSION AND CONCLUSIONS

The paper argues that recalibration should address outcomes important to decisions, not only average calibration. It presents outcome-conditional recalibration and reports its practical scope, empirical motivation, and open connections to conformal prediction and auto-calibration.

6 DISCUSSION AND CONCLUSIONS

  • Existing recalibration methods typically do not ensure calibrated forecasts for particular outcomes, while the proposed method targets calibration for those outcomes.The discussion identifies extreme events as especially important for decision making.Source: §6 · p. 8
  • The method is hyperparameter-free and can be estimated almost instantaneously, using training data or, ideally, an out-of-sample calibration dataset.The authors note potential for use in other application domains.Source: §6 · p. 8
  • Because its forecasts are calibrated in-sample, outcome-conditional recalibration could potentially be conformalised to obtain out-of-sample outcome-conditional guarantees.The paper presents this as scope for future work, not an established guarantee.Source: §6 · p. 8
  • The authors describe the method as the first post-hoc recalibration approach targeting outcome-dependent calibration, while noting that its relationship to auto-calibration needs further study.Their empirical results suggest that schemes designed to enforce auto-calibration do not necessarily calibrate forecasts for particular outcomes.Source: §6 · p. 8

Code and Data availability

The paper reports that experiment code will be provided upon acceptance and discloses the authors’ use of generative AI tools.

Code and Data availability

AI Use Statement

  • Claude Opus 4.8 and Claude Opus 5 in Claude Code supported method implementation and manuscript editing; the authors say they checked outputs and tested the code.They take responsibility for the final content, including AI-assisted text, claims, and artifacts.Source: AI Use Statement · p. 9

A PROOFS

The proofs establish that the recalibrated distribution is non-decreasing and bounded, continuous, and locally calibrated within each interval.

A PROOFS

  • The recalibrated predictive distribution is non-decreasing and bounded between zero and one.The proof invokes the non-decreasing behavior and endpoint values of the component maps.Source: A PROOFS · p. 11
  • Continuity within each interval follows from continuity of the interval-specific maps and their endpoint values.Source: A PROOFS · p. 11
  • Conditional on the outcome falling in an interval, the transformed conditional forecast follows a standard uniform distribution, establishing the first requirement of local calibration.The proof applies this result to all intervals.Source: A PROOFS · p. 11

B SAMPLE-LEVEL ESTIMATION

The sample-level procedure partitions the outcome space into intervals, corrects conditional calibration and group occurrence probabilities, then combines these corrections into a recalibrated CDF.

B SAMPLE-LEVEL ESTIMATION

  • For each interval, isotonic regression fits a non-decreasing map to conditional PIT values, correcting probabilistic or quantile calibration within that group.The map has boundary values zero and one.Source: B SAMPLE-LEVEL ESTIMATION · p. 12
  • Positive group scalings adjust each interval’s forecast mass so its average recalibrated mass matches the empirical occurrence rate.The procedure’s occurrence probabilities are estimated from group sample counts.Source: B SAMPLE-LEVEL ESTIMATION · p. 12
  • Because the normalization couples groups, the scalings are found by iterative proportional fitting; setting them all to one recovers the uncorrected case.The overall scale cancels and may be fixed by normalization.Source: B SAMPLE-LEVEL ESTIMATION · p. 12
  • The combined recalibrated CDF is valid, continuous, and in-sample outcome-conditionally calibrated.The paper states these properties are established by Propositions 4.1 and 4.2.Source: B SAMPLE-LEVEL ESTIMATION · p. 12

C PER-DATASET UCI/OPENML RESULTS

Across ten UCI and OpenML datasets, outcome-conditional recalibration leads on subset and occurrence calibration, while KFE18 and OC+QR are often strongest on overall calibration.

C PER-DATASET UCI/OPENML RESULTS

D OUT-OF-SAMPLE FITTING FOR THE UCI/OPENML EXPERIMENTS

Out-of-sample fitting largely preserves outcome-conditional calibration gains. Transfer depends on partition granularity, while iterated recalibration composes corrections across subsets to target calibration on many partitions.

D OUT-OF-SAMPLE FITTING FOR THE UCI/OPENML EXPERIMENTS

E EVALUATING ON PARTITIONS NOT USED FOR FITTING

  • With fixed G = 5 fitting partitions, calibration error worsens on finer evaluation partitions, while fitting on finer partitions also performs well on coarser ones.Outcome-conditional recalibration still outperforms all baselines when evaluated on finer partitions.Source: E EVALUATING ON PARTITIONS NOT USED… · p. 17, Table 6 · p. 17
  • For interest across many partitions, iterated recalibration performs well but offers little additional benefit over the more parsimonious approach.The evaluation assesses transfer to finer partitions and performance under different partition combinations.Source: E EVALUATING ON PARTITIONS NOT USED… · p. 17

E.1 Iterated recalibration

  • Iterated recalibration applies outcome-conditional recalibration repeatedly across different partitions and composes the resulting transformations.It is introduced because a single application is in-sample calibrated on only its training family of subsets.Source: §E.1 · p. 17
  • The method defines candidate subsets from outcome-space breakpoints; with nb = 17, the set contains 136 subsets, including the real line.Breakpoints may be placed at equally spaced outcome quantiles.Source: §E.1 · p. 17
  • Each round selects the subset with the largest calibration error, recalibrates it, and leaves its complement untouched apart from distribution-validity rescaling.The final prediction composes the one-subset recalibrations.Source: §E.1 · pp. 17–18
  • The number of rounds is usually of minor importance; ten fixed rounds performed well in the application, and held-out early stopping added little.Single-step outcome-conditional recalibration is described as extremely fast, making this computationally feasible.Source: §E.1 · p. 18
Loading 2610.10076v1…