Source-linked AI summary
Evaluating epidemic forecasts in an interval format
Johannes Bracher, Evan L. Ray, Tilmann Gneiting, Nicholas G. Reich
TL;DR
Epidemic forecasts are often issued as central predictive intervals, making full-distribution metrics such as the logarithmic score unavailable. The paper reviews interval-based scoring methods, especially the weighted interval score, and describes its CRPS approximation, absolute-error interpretation, and sharpness–penalty decomposition. The discussion supports distance-sensitive scores such as CRPS or WIS in low-predictability settings, while recognizing trade-offs with logarithmic scoring.
Problem
Central predictive intervals do not provide the full predictive distributions required to evaluate metrics such as the logarithmic score.
Method
The paper reviews proper quantile- and interval-based scoring methods and explains how the weighted interval score aggregates predictive medians and multiple central intervals.
Results
The weighted interval score approximates the CRPS and can be interpreted as a probabilistic generalization of absolute error with sharpness and penalty components.
Takeaways & Limitations
Distance-sensitive scores such as CRPS or WIS are attractive for epidemic contexts with low predictability, while score choice can depend on the cost of different errors.
Takeaways & Limitations
Approximating the logarithmic score from interval forecasts would require interpolation within intervals and still face difficulties for observations outside the highest-coverage interval.
Abstract
from arXiv · showhide
For practical reasons, many forecasts of case, hospitalization and death counts in the context of the current COVID-19 pandemic are issued in the form of central predictive intervals at various levels. This is also the case for the forecasts collected in the COVID-19 Forecast Hub (https://covid19forecasthub.org/). Forecast evaluation metrics like the logarithmic score, which has been applied in several infectious disease forecasting challenges, are then not available as they require full predictive distributions. This article provides an overview of how established methods for the evaluation of quantile and interval forecasts can be applied to epidemic forecasts in this format. Specifically, we discuss the computation and interpretation of the weighted interval score, which is a proper score that approximates the continuous ranked probability score. It can be interpreted as a generalization of the absolute error to probabilistic forecasts and allows for a decomposition into a measure of sharpness and penalties for over- and underprediction.
Author summary
The paper presents scoring methods for evaluating epidemic forecasts issued as central predictive intervals, focusing on the weighted interval score and its relationship to the CRPS.
- The paper reviews scoring methods suitable for the interval/quantile format used by the COVID-19 Forecast Hub.It focuses on methods that can evaluate forecasts without full predictive distributions.
1 Introduction
Probabilistic epidemic forecasts quantify uncertainty, but acute outbreaks make common outcome-binning schemes difficult to define. The COVID-19 Forecast Hub therefore stores predictive quantiles and intervals, requiring adapted scoring methods.
- Probabilistic forecasts report predicted outcomes together with their uncertainty.
- Acute outbreak settings make reasonable binning schemes difficult to define because outcomes vary across space and time and forecasts are produced under time pressure.
- The Forecast Hub stores predictive medians and central prediction intervals at nominal levels from 10% through 98%.This format stores predictive distributions in reasonable detail independently of the expected range of outcomes.
- Quantile- and interval-based forecasts require adapted scoring methods because the logarithmic score cannot be evaluated from quantiles alone.
2 Forecast evaluation using proper scoring rules
Proper scoring rules evaluate probabilistic forecasts, but interval-formatted forecasts require interval-based alternatives. The paper develops the weighted interval score as a proper, interpretable approximation to the CRPS.
- 2. Forecast evaluation using proper scoring rules: Proper scoring rules encourage honest probabilistic forecasting because forecasters have no incentive to depart from their true beliefs.
- 2.1 Common scores to evaluate full predictive distributions: The logarithmic score uses the probability assigned to the observed outcome, but it can become −∞ when that probability is zero.Truncation at −10 prevents this divergence but makes the score improper.
- 2.1 Common scores to evaluate full predictive distributions: The CRPS generalizes absolute error to probabilistic forecasts and remains finite when the forecast assigns zero probability to the observed outcome.
- 2.2 Scores for forecasts provided in an interval format: The interval score combines prediction-interval width with penalties for observations below the lower or above the upper endpoint.Penalty severity increases for intervals with higher nominal coverage.
- 2.2 Scores for forecasts provided in an interval format: The weighted interval score aggregates a predictive median and multiple central prediction intervals using proper non-negative weights.It is a special case of the quantile score and can approximate the CRPS.
- 2.2 Scores for forecasts provided in an interval format: For the Forecast Hub, evaluation uses the predictive median and 11 intervals spanning nominal coverages from 10% to 98%.
- 2.2 Scores for forecasts provided in an interval format: The weighted interval score decomposes into weighted interval widths and penalty terms representing forecast sharpness and calibration.
3 Qualitative comparison for different scores
The scores behave similarly near the predictive center but can disagree substantially for tail observations. WIS favors distance-sensitive forecasts, whereas the logarithmic score penalizes forecasts with overly light tails more severely.
- 3.1 Illustration for an integer-valued outcome: For a negative binomial forecast with expectation 60 and size 4, logarithmic score is lowest at the predictive mode, while AE, CRPS, and interval scores are best at the median.The 80% interval score can be flat across observations inside its prediction interval.
- 3.2 Differing behaviour if agreement between predictions and observations is poor: Tail observations produce the main qualitative differences between logarithmic and interval-based scores.The comparison uses forecasts F and G with expectations 60 and 80, respectively, and observed value y = 190.
- 3.2 Differing behaviour if agreement between predictions and observations is poor: −9.37 versus −9.69: the logarithmic score favors F over G when y = 190 because F is more dispersed and has slightly heavier tails.The values are logS(F, 190) = −9.37 and logS(G, 190) = −9.69.
- 3.2 Differing behaviour if agreement between predictions and observations is poor: 103.9 versus 87.8: WIS favors G over F because G’s quantiles are generally closer to y = 190.The values are WIS(F, 190) = 103.9 and WIS(G, 190) = 87.8.
- 3.2 Differing behaviour if agreement between predictions and observations is poor: The WIS is sensitive to distance, whereas the logarithmic score is local and ignores distance.The paper relates this distinction to different inferential and decision-making settings.
4 Application to FluSight forecasts
The FluSight examples use historical forecasts translated from binned to quantile format and decompose WIS into sharpness and coverage penalties. KCDE and SARIMA had similar sharpness, but SARIMA incurred larger penalties for intervals missing observations.
- 4 Application to FluSight forecasts: Historical 2016/2017 FluSight forecasts were originally binned but translated to quantile format for the examples.The section uses these forecasts to illustrate practical evaluation aspects.
- 4 Application to FluSight forecasts: WIS decomposes into average prediction-interval width and penalties for observations outside intervals.Separating over- and underprediction penalties can reveal systematic biases or asymmetries.
- 4 Application to FluSight forecasts: KCDE and SARIMA issued forecasts with similar sharpness, but SARIMA was more strongly penalized for intervals not covering observations.The decomposition distinguishes interval-width contributions from over- and underprediction penalties.
- 4 Application to FluSight forecasts: KCDE and SARIMA forecasts were on average off by 0.25 and 0.35 percentage points, respectively.These averages summarize the one-week-ahead FluSight comparison.
SARIMA
The FluSight application compares interval scores and PIT histograms for KCDE and SARIMA. PIT histograms assess calibration, while score decompositions show whether differences arise from sharpness or prediction errors.
- SARIMA: Figure 4 compares one-week-ahead KCDE and SARIMA forecasts using interval score and WIS across the 2016/2017 FluSight season.The top row uses α = 0.2 and the bottom row uses WIS with K = 11.
- SARIMA: Figure 4 decomposes scores into interval-width contributions and penalties for over- and underprediction.The right-hand panels show mean scores over the season.
- SARIMA: PIT histograms assess calibration by showing the empirical distribution of probability integral transform values across forecasts.A calibrated forecast should produce an approximately uniform histogram.
- SARIMA: L-shaped and J-shaped PIT histograms indicate downward and upward bias, while U-shaped and inverse-U-shaped histograms indicate under- and overdispersion.These shapes provide visual diagnostics for forecast bias and dispersion.
- SARIMA: For discrete binned forecasts, non-randomized corrections account for observations coinciding with reported quantiles when constructing PIT histograms.The correction splits such counts between neighboring histogram bins.
5 A brief remark on evaluating point forecasts
Point forecasts submitted to the COVID-19 Forecast Hub are evaluated with absolute error, making the predictive median the appropriate point forecast. Absolute error is interpretable alongside WIS but is dominated by larger targets, so relative alternatives require caution.
- 5 A brief remark on evaluating point forecasts: The Forecast Hub applies absolute error to point forecasts, implying that teams should report the predictive median.Absolute error and WIS are reported on the same scale as the observations.
- 5 A brief remark on evaluating point forecasts: Absolute error averages are dominated by larger states and weeks with high activity, as are CRPS and WIS.This weighting reflects the larger magnitude of high-activity targets.
- 5 A brief remark on evaluating point forecasts: MAPE treats forecasting 200 instead of 400 deaths like forecasting 2 instead of 4, despite the different absolute discrepancies.The paper therefore argues that emphasizing targets with higher expected values can be meaningful.
- 5 A brief remark on evaluating point forecasts: MAPE does not encourage reporting predictive medians or means and should therefore be used with caution.It instead favors less interpretable types of point forecasts.
6 Discussion
The paper frames interval-based epidemic forecast evaluation as an approximate CRPS problem and recommends distance-sensitive scores such as CRPS or WIS, especially when predictability is low. Its scope is restricted to central prediction intervals, while logarithmic-score approximation remains difficult because of interpolation and tail assumptions.
- The discussion restricts attention to central prediction intervals because noncentral intervals create additional conceptual complexity.
- The proposed method evaluates epidemic forecasts in interval or quantile format through an approximate CRPS based on prediction intervals.
- Approximating the logarithmic score would require interpolation within intervals and strong assumptions about tails beyond the highest nominal coverage.
- CRPS and WIS are presented as attractive distance-sensitive scores for contexts with low predictability, including the COVID-19 pandemic.
A Relationship between quantile score, interval score and CRPS
The paper connects quantile scores, interval scores, and the WIS as related ways to approximate the CRPS from predictive quantiles. The resulting score is translation-invariant, while the practical quantile grid places slightly more weight on the tails.
- The quantile score uses the forecast quantile qτ and observed outcome y, and the interval score for a central (1 −α) prediction interval combines scores at α/2 and 1 −α/2.
- The interval-based score is invariant under translation, so one-step evaluations of incident and cumulative counts give the same results.
- The WIS equals an average of quantile scores over many approximately equally spaced levels spanning the unit interval.
- The WIS representation matches the alternative WIS expression when paired quantile levels correspond to the interval levels and the median is included.
- The practical quantile levels are not equally spaced in the tails, producing slightly greater tail weight relative to the CRPS.
Reproducibility
The paper supports reproducibility by releasing code for Figures 1–6 and using data from the public CDC FluSight ensemble repository.
- Code reproducing Figures 1–6 is publicly available, and all paper data come from the public CDC FluSight ensemble repository.