PaperlayerView exact revision on arXiv

Source-linked AI summary

Prediction of infectious disease epidemics via weighted density ensembles

Evan L. Ray, Nicholas G. Reich

arXiv:1703.10936v1stat.ML

TL;DR

The paper examines whether ensemble methods can improve infectious-disease predictions, which may inform public-health planning. It combines component predictive distributions using weighted ensembles and evaluates influenza-season forecasts. Ensembles performed about as well as individual component models overall, with more stable performance across seasons.

  • Problem

    The paper examines whether ensemble methods can improve infectious-disease predictions, which are relevant to public-health planning.

    Source: Introduction · p. 2, Discussion · p. 12

  • Method

    The study combines component predictive distributions using weighted ensembles, including feature-dependent weights based on observed data and component distributions.

    Source: Introduction · pp. 2–3

  • Results

    Ensemble methods performed about as well as individual component models overall, with increased stability across seasons.

    Source: Discussion · p. 12

  • Takeaways & Limitations

    Greater stability in predictive accuracy may give decision makers more confidence when using predictions as inputs to policy.

    Source: Discussion · p. 12

    The study used a small selection of component models, including two that shared seasonal terms and recent-incidence observations.

    Source: Discussion · p. 13

Abstract

from arXiv · show

Accurate and reliable predictions of infectious disease dynamics can be valuable to public health organizations that plan interventions to decrease or prevent disease transmission. A great variety of models have been developed for this task, using different model structures, covariates, and targets for prediction. Experience has shown that the performance of these models varies; some tend to do better or worse in different seasons or at different points within a season. Ensemble methods combine multiple models to obtain a single prediction that leverages the strengths of each model. We considered a range of ensemble methods that each form a predictive density for a target of interest as a weighted sum of the predictive densities from component models. In the simplest case, equal weight is assigned to each component model; in the most complex case, the weights vary with the region, prediction target, week of the season when the predictions are made, a measure of component model uncertainty, and recent observations of disease incidence. We applied these methods to predict measures of influenza season timing and severity in the United States, both at the national and regional levels, using three component models. We trained the models on retrospective predictions from 14 seasons (1997/1998 - 2010/2011) and evaluated each model's prospective, out-of-sample performance in the five subsequent influenza seasons. In this test phase, the ensemble methods showed overall performance that was similar to the best of the component models, but offered more consistent performance across seasons than the component models. Ensemble methods offer the potential to deliver more reliable predictions to public health decision makers.

Introduction

The paper asks whether weighted ensembles can use changing model performance to improve infectious-disease forecasts, which matter for public-health planning. It compares weighting approaches and applies them to influenza forecasts, including a feature-weighted method.

Introduction

  • Historical-incidence models performed better early in the U.S. influenza season, while time-series models improved in midseason, motivating ensembles that could use past performance.Source: Introduction · p. 2
  • All ensemble methods use stacking: component models are fit separately, then a second-stage model learns to combine their predictive densities.The weighted density ensemble forms a weighted sum of component predictive densities, with weights that may depend on covariates.Source: Introduction · p. 2
  • The study compares equal weights, constant model-specific weights, and feature-dependent weights to assess the benefits of increasing ensemble complexity.The feature-weighted approach uses observations at prediction time and aspects of component predictive distributions.Source: Introduction · p. 3
  • Unlike prior infectious-disease stacking with constant component weights, this work compares more flexible ensembles whose weights depend on observed features.The prior approach is attributed to Yamana et al..Source: Introduction · p. 3
  • The paper applies ensemble forecasting to U.S. influenza season attributes at each week of the season.The authors present the application as a case study of ensemble forecasting in a setting with complex population dynamics.Source: Introduction · p. 3

Methods

The methods section outlines a comparison of weighted-density ensemble approaches for forecasting influenza-season features.

Methods

  • The paper compares methods for determining weights in weighted density ensembles.Source: Methods · p. 3
  • The application forecasts specific features of U.S. influenza seasons.Source: Methods · p. 3
  • The methods are introduced through the influenza data and targets, the component models, and the ensemble specifications.Source: Methods · p. 3

Data and prediction targets

The study uses CDC weekly influenza-like illness data for U.S. regions and the nation to define seasonal targets and evaluate predictive distributions.

Data and prediction targets

  • CDC wILI data cover the 10 U.S. Health and Human Services regions and the nation, with weekly updates during the influenza season.For this analysis, models use final reported wILI measures rather than interim reports.Source: Data and prediction targets · pp. 3–4
  • The targets are onset week, defined by three successive weeks at or above a region-season threshold, peak incidence, and peak week.Peak incidence is the season's maximum wILI; peak week is when that maximum occurs.Source: Data and prediction targets · p. 4
  • Predictive distributions assign probabilities to target-specific bins, including a no-onset bin and an open-ended peak-incidence bin above 13.05.The peak-incidence bins are rounded to one decimal place.Source: Data and prediction targets · p. 4
  • Accuracy is measured by the log score, the natural log of the probability assigned to the bin containing the observed outcome.For tied peak-incidence maxima across weeks, probabilities for those weeks are summed before scoring.Source: Data and prediction targets · p. 4

Component models

Three component models generate probabilistic predictions for seasonal influenza targets; their forecasts are trained and evaluated using season-based splits and cross-validation.

Component models

  • The component set comprises seasonal-average KDE, KCDE with copulas, and seasonal ARIMA models, each fit independently by region.KCDE conditions future incidence on recent observations, while SARIMA uses a standard time-series specification.Source: Component models · p. 4

Kernel Density Estimation (KDE)

  • KDE estimates target distributions from prior seasons within a region, including an onset-no-onset probability estimated from training seasons.The empirical distributions are formed by sampling and discretizing outcomes; predictions do not change as new in-season data arrive.Source: Kernel Density Estimation (KDE) · pp. 4–5

Kernel Conditional Density Estimation (KCDE)

  • KCDE estimates conditional distributions for future weekly incidence, uses a copula to model dependence, and derives seasonal targets from simulated trajectories.The target distributions are obtained by computing onset, peak week, and peak incidence for simulated incidence paths.Source: Kernel Conditional Density Estimation… · p. 5

Seasonal auto-regressive integrated moving average (SARIMA)

  • SARIMA forecasts future wILI trajectories, from which predictive distributions for the seasonal targets are computed.The models use log-transformed wILI and seasonal differencing.Source: Seasonal auto-regressive integrated… · p. 5

Component model training

  • Models were trained on 14 seasons and tested prospectively on five later seasons, with leave-one-season-out predictions used to estimate ensemble weights.Some training-season predictions were unavailable for SARIMA and KCDE because required lagged observations were missing.Source: Component model training · p. 6

Ensemble models

These stacked ensembles combine component-model predictive densities while varying how their weights are estimated. The six methods range from equal or constant weights to feature-dependent weights, including regularized variants.

Ensemble models

Overview of ensemble models

  • Feature-weighted variants use season week and model uncertainty; regularized variants use week alone, week plus uncertainty, or recent wILI.The feature sets were selected before test-phase predictions.Source: Overview of ensemble models · p. 7
  • The six ensembles span equal weights, region- and target-specific constant weights, feature-dependent weights, and regularized feature-dependent variants.Comparisons among the variants examine the flexibility and regularization of the weights, as well as the contribution of different features.Source: Overview of ensemble models · pp. 7–8
  • The evaluation generates predictions from six ensembles and three component models across 11 regions, five test seasons, and three targets, assuming perfect reporting.Table 1 summarizes the ensemble methods and what their weights depend on.Source: Overview of ensemble models · p. 8, Table 1 · p. 8

Feature-weighted stacking framework

The feature-weighted framework maps observed features to model weights using softmax-transformed functions estimated with gradient tree boosting. Regularization controls how flexibly those weights vary, with tuning based on leave-one-season-out cross-validation.

Feature-weighted stacking framework

  • Softmax transforms latent weight functions into nonnegative weights summing to one, while gradient tree boosting estimates the functions by minimizing the stacked predictions’ negative log-score.The score uses each component model’s cross-validated predictive density at the realized outcome.Source: Feature-weighted stacking framework · pp. 8–9
  • Each latent weight function is built from regression trees that partition feature space into regions and assign constants to those regions.The implementation uses a gradient-descent step when the loss is locally concave, since the loss is not guaranteed to be convex.Source: Feature-weighted stacking framework · p. 9
  • Regularization controls tree count, tree leaves, and regression constants; grid search selects settings using leave-one-season-out cross-validated performance.More boosting iterations permit more extreme and rapidly changing weights, while stronger penalties reduce flexibility or weight changes.Source: Feature-weighted stacking framework · p. 9

Software and code

The analyses used R 3.2.2, and the authors made the analysis data and code available in an R package. They also describe steps taken to support reproducibility and access to real-time predictions.

Software and code

  • The analyses used R version 3.2.2, and the data and code were made freely available in an R package.The package can be installed directly in R.Source: Software and code · p. 10
  • Predictions from early model versions during the 2016/2017 influenza season were made viewable online.Source: Software and code · p. 10
  • The authors set seeds for stochastic simulations and dynamically generated the manuscript using RMarkdown.Source: Software and code · p. 10

Results

Across test seasons, several methods had similar aggregate performance, but ensembles were more consistent than component models. Feature-weighted ensembles tracked changing relative model performance, and regularization improved early-season accuracy.

Results

  • Overall log scores were computed across test-phase regions and seasons, with a separate examination of predictions made before each target’s onset or peak week.The early-prediction comparison covered peak timing, peak incidence, and onset timing.Source: Results · p. 10

Feature-weighted ensemble model weights reflect trends in component model log scores

  • SARIMA’s weight rose and KDE’s fell as the national influenza season progressed, while the size of these shifts differed by prediction target.Feature-weighted ensemble weights generally tracked trends in relative component-model performance.Source: Feature-weighted ensemble model… · p. 10

Best models have similar aggregate performance

  • KCDE, SARIMA, CW, and feature-weighted variants had similar aggregate performance, with no statistically significant mean-performance differences detected.KDE generally scored lower, pulling EW slightly below the other methods; unregularized FW-w also had some poor peak-incidence cases.Source: Best models have similar aggregate… · pp. 10–11

Ensembles show stable performance across seasons for early-season predictions

  • Component models swung between best and worst across season-target pairs, while ensemble methods showed more stable performance across test seasons.Only FW-reg-w stayed in the top half for onset timing; EW and FW-reg-w also did so for peak timing, while FW-reg-wu and FW-reg-wui did so for peak incidence. EW and regularized FW methods had top-half worst-case log scores for all three targets.Source: Ensembles show stable performance… · p. 11
  • For peak timing, EW and FW-reg-w had slightly lower average log scores than SARIMA overall, but ranked among the best methods in all five test seasons.In 2015/2016, when SARIMA performed worst, the ensembles assigned about 60 to 70 percent more probability to the realized peak week than SARIMA.Source: Ensembles show stable performance… · p. 11

Regularization improves feature-weighted ensemble models

  • Regularization improved early-season prediction accuracy for all three metrics; FW-wu beat its regularized counterpart in only one of 15 target-season combinations.The sole exception was onset timing in 2015/2016.Source: Regularization improves… · pp. 11–12

Discussion

Ensembles performed about as well as individual component models overall and were more stable across seasons. The discussion also notes that the evaluation and feature-weighting exploration were constrained by limited data, component-model diversity, and restricted feature sets.

Discussion

  • Ensembles performed about as well as individual component models overall while improving cross-season stability; component and ensemble models also outperformed a seasonal-average baseline.Source: Discussion · pp. 12–13
  • More stable predictive accuracy can give decision makers greater confidence using forecasts in policy, especially for unusually severe or unusually timed seasons.Source: Discussion · p. 12
  • No ensemble dominated: FW-reg-w had good average and worst-case performance across targets and seasons, while equal weights had better worst-case peak-timing performance.CW's worst-case log scores and model rankings were slightly lower than FW-reg-w's; equally weighted models had lower mean log scores than CW and FW-reg-w across all three targets.Source: Discussion · p. 12
  • The feature-weighting scheme estimates weights that sum to 1, and smoothing the weights consistently improved performance over an unsmoothed implementation.The authors note that the approach could combine distribution estimates beyond infectious disease or time-series applications.Source: Discussion · p. 12
  • Only 14 seasons were available for training and five for testing; limited training data may have weakened weight estimation, while correlated scores reduced effective sample size.Source: Discussion · p. 13
  • Interpretation is bounded by a small, partly similar component-model set and restricted weighting features; additional model diversity or other weighting features may improve performance or informativeness.The authors suggest alternative structures or covariates to increase component diversity and note that untested weighting features may be more informative.Source: Discussion · p. 13
Loading 1703.10936v1…