PaperlayerView exact revision on arXiv

Source-linked AI summary

Accurate estimation of influenza epidemics using Google search data via ARGO

Shihao Yang, Mauricio Santillana, S. C. Kou

arXiv:1505.00864v2stat.APcs.SIstat.ML

TL;DR

CDC influenza reports can be revised after publication, complicating real-time estimation. ARGO combines autoregression with Google search data, and it outperforms alternative estimation methods, while remaining vulnerable to abrupt changes in search behavior or presentation.

  • Problem

    CDC influenza reports are revised after publication, so information available during a given week may be inaccurate.

  • Method

    ARGO is an autoregressive model using Google search queries as exogenous variables, with regularization for automatic information selection and a hidden-Markov-model motivation.

  • Results

    ARGO outperformed all alternatives across accuracy metrics for the whole study period and was estimated to be at least twice as efficient as any alternative.

  • Takeaways & Limitations

    ARGO provides robust, highly accurate influenza activity estimates using publicly available Google search data, including low-quality sources.

  • Takeaways & Limitations

    ARGO’s accuracy can be affected by abrupt search-engine changes or changes in how health-related search information is displayed.

Abstract

from arXiv · show

Accurate real-time tracking of influenza outbreaks helps public health officials make timely and meaningful decisions that could save lives. We propose an influenza tracking model, ARGO (AutoRegression with GOogle search data), that uses publicly available online search data. In addition to having a rigorous statistical foundation, ARGO outperforms all previously available Google-search-based tracking models, including the latest version of Google Flu Trends, even though it uses only low-quality search data as input from publicly available Google Trends and Google Correlate websites. ARGO not only incorporates the seasonality in influenza epidemics but also captures changes in people's online search behavior over time. ARGO is also flexible, self-correcting, robust, and scalable, making it a potentially powerful tool that can be used for real-time tracking of other social events at multiple temporal and spatial resolutions.

0.1 Our contribution

ARGO addresses shortcomings of earlier Google-search-based influenza tracking by dynamically using CDC updates, selecting useful queries, modeling seasonality, and adapting to recent search behavior. It is reported to improve accuracy over alternative methods, including GFT-based approaches, despite using publicly available low-quality search data.

  • 0.1 Our contribution: ARGO dynamically incorporates new CDC reports and automatically selects the most useful Google search queries for estimation.These features address model drift and changing query usefulness over time.
  • 0.1 Our contribution: ARGO includes long-term seasonal information from past flu seasons and trains on a two-year moving window to capture recent search-pattern changes.The moving window immediately precedes the date being estimated.
  • 0.1 Our contribution: ARGO is twice as accurate as the method combining GFT with autoregressive terms and significantly improves on the latest version of GFT using publicly available low-quality search data.The reported improvements are statistically significant.
  • 0.1 Our contribution: ARGO is an autoregressive model with Google search queries as exogenous variables and uses L1, and potentially L2, regularization for automatic information selection.The model is named ARGO, short for AutoRegression with GOogle search data.

1 Results

Retrospective estimates from March 2009 through July 2015 were evaluated against delayed CDC weighted ILI reports across multiple accuracy metrics and periods. ARGO outperformed alternative methods broadly, remained robust to CDC report revisions, and was more stable under changing Google Trends data.

  • 1 Results: From 2009-03-29 to 2015-07-11, ARGO estimates were compared with CDC weighted ILI activity using multiple accuracy metrics.The evaluation assumed access only to historical CDC reports available before each estimation date.
  • 1 Results: ARGO outperformed all alternatives on every accuracy metric over the full evaluation period.Table 1 also reports results for the 2009 off-season H1N1 outbreak and regular flu seasons since 2010.
  • 1 Results: In post-2009 regular flu seasons, ARGO uniformly outperformed alternatives in root mean squared error, mean absolute error, mean absolute percentage error, and correlation.ARGO also avoided GFT’s reported over-shooting problem.
  • 1 Results: ARGO was estimated to be at least twice as efficient as every alternative, with the accuracy improvement highly statistically significant.The comparison used 95% confidence intervals for relative efficiency.
  • 1 Results: Using unrevised CDC information, ARGO still outperformed all other methods and its five accuracy metrics essentially did not change.This evaluation tested robustness to the revisions that CDC reports receive after initial publication.
  • 1 Results: Despite variation in Google Trends data, ARGO was threefold more stable than the method of and still outperformed other methods on average.The stability test used search frequencies collected on 25 different days in April 2015.

2 Discussion

ARGO combines seasonal influenza information with dynamically reweighted Google search data, producing accurate and robust estimates despite low-quality inputs. Its time-series and search components complement one another, while self-correction helps respond to unusual outbreaks and search surges.

  • Accuracy and robustness: ARGO’s enhanced accuracy appears to arise from combining seasonal flu information with dynamic reweighting of search information.Seasonality pulls estimates toward historical levels, while query-term weights can change over time.
  • Complementary information: ARGO captures sudden ILI changes similarly to search-based methods while outperforming AR(3) in increment correlation.Time-series models respond slowly to sudden changes, whereas searches respond quickly but can be sensitive to public over-reaction.
  • Statistical comparison: ARGO was at least twice as efficient as every alternative model according to the estimated relative-efficiency confidence intervals.Relative efficiency above one indicates greater predictive power for ARGO.
  • Self-correction: During unusual episodes, ARGO self-corrected by shifting weights between time-series and Google-search terms, missing a 2012–13 peak by only one week.The method missed the peak by two weeks for method and about four weeks for GFT.
  • Limitations and next steps: ARGO’s accuracy remains dependent on public search behavior and may be affected by abrupt search-engine changes or altered presentation of health information.The authors also note that future performance is not guaranteed by past performance.

3 Materials and Methods

ARGO combines historical CDC influenza activity with Google search data in a hidden-Markov-model-motivated autoregressive framework, using transformed variables, regularization, and rolling estimation. The methods also define accuracy assessment and bootstrap confidence intervals.

  • Data sources: The study uses publicly available data, including CDC ILI reports and Google search information, without requiring IRB approval.Search-term selection is designed to avoid forward-looking information in out-of-sample predictions.
  • Formulation: ARGO models logit-transformed CDC ILI activity autoregressively and uses log-transformed Google search frequencies as exogenous variables.The model is motivated by a hidden Markov model in which searches depend on contemporaneous ILI activity.
  • Data and estimation: The model uses a two-year rolling training window, with N = 52 weeks for seasonality and K = 100 Google search terms.The rolling window immediately precedes each desired estimation date.
  • Parameter estimation: L1 and L2 regularization address the setting with more independent variables than observations, while hyper-parameters are selected through cross-validation.The reported setup fixes ηα = ηβ = 0 and uses L1 penalization for autoregressive and search-term coefficients.
  • Accuracy metrics: RMSE, MAE, MAPE, correlation, and correlation of increments are used to evaluate estimates against CDC weighted ILI activity.Relative efficiency compares methods through their true mean squared errors, estimated from observed values.
  • Uncertainty: Stationary bootstrap confidence intervals use geometrically distributed residual blocks with mean length 52 to account for temporal dependence.The resulting non-parametric intervals are recovered on the original scale by exponentiation.

A SI Methods and Robustness Analysis

The supporting methods describe the predictive formulation, hyper-parameter strategy, and robustness analyses used to evaluate ARGO. These analyses address CDC report revisions and variation in online search behavior.

  • Methods: The supporting methods first describe ARGO’s predictive distribution and corresponding assumptions.
  • Methods: They next explain the statistical strategy for determining ARGO’s hyper-parameters.
  • Robustness Analysis: Two sensitivity analyses test robustness to CDC ILI report revisions and observed variation in internet search behavior.

B Predictive distribution in the formulation of ARGO model

ARGO transforms ILI and search variables and specifies a predictive distribution within a hidden-Markov-model framework. The predictive distribution has a linear mean in lagged ILI activity and contemporaneous search variables, with constant variance.

  • Transformations: CDC ILI activity is logit-transformed, while normalized Google search volumes are log-transformed after adding δ = 0.5 to avoid log(0).
  • Assumptions: Conditional on current ILI activity yt, the search vector Xt is independent of other times’ ILI activity and search vectors.
  • Assumptions: The search model uses β as a coefficient vector, μx as a mean vector, and Q as the covariance matrix.
  • Predictive distribution: ARGO’s predictive distribution is normal, with mean linear in y(t−N):(t−1) and Xt and constant variance.

C Determination of the hyper–parameters for ARGO

ARGO’s hyper-parameters are selected using a two-year rolling window and cross-validation across L1, L2, and elastic-net specifications. The reported comparisons favor L1 penalization and support the simpler shared-penalty specification.

  • Training strategy: Each weekly estimate uses the preceding 104 weeks as a rolling training window for selecting hyper-parameters.
  • L1 specification: The proposed specification sets ηα = ηβ = 0 and λα = λβ, then cross-validates the common L1 penalty.
  • Alternative specifications: The alternatives include separate L1 penalties, shared or separate L2 penalties, and shared elastic-net penalties.
  • Specification comparison: L1 penalties generally outperform L2 penalties because they can shrink unnecessary coefficients exactly to zero.
  • Specification comparison: The elastic-net model does not provide error reduction, and separate L1 penalties outperform separate L2 penalties on RMSE and MAE.
  • Specification comparison: The shared-L1 ARGO specification performs within one standard deviation of the smallest cross-validation mean while retaining greater simplicity.

D Revision of CDC’s ILI activity reports

The evaluation simulates real-time use by training ARGO only on CDC ILI reports available at each estimation week, excluding future revisions. Under this schedule, ARGO remains accurate and robust.

  • D Revision of CDC’s ILI activity reports: ARGO and alternative models were trained under a schedule designed to mimic real-time tracking amid weekly CDC report revisions.The schedule addresses the fact that previously reported ILI values can change as new information is incorporated.
  • D Revision of CDC’s ILI activity reports: ARGO training uses CDC ILI values available by week j, while accuracy is evaluated against finalized ILI values.This prevents forward-looking information from entering the training data while retaining a finalized target for evaluation.
  • D Revision of CDC’s ILI activity reports: ARGO still outperforms all alternative models when training excludes forward-looking CDC revisions.Its four accuracy metrics essentially match those obtained when finalized ILI values are used for training.

E Variations of Google Trends data

The study tests ARGO against changing Google Trends inputs because historical search data are revised through re-normalizations and algorithm updates. ARGO remains substantially more stable than the comparison method despite these variations.

  • E Variations of Google Trends data: Google Trends historical data change through re-normalizations and algorithm updates, motivating a robustness test across 25 accessed versions.The test covers Sep 28, 2014 to Mar 29, 2015, a partial 2014–15 flu season.
  • E Variations of Google Trends data: Despite variation in low-quality Google Trends data, ARGO is threefold more robust than the method of Santillana et al..The study attributes this stability to incorporating time-series information.
  • E Variations of Google Trends data: ARGO outperforms GFT uniformly across all tested Google Trends versions in both RMSE and MAE.This comparison uses the partial 2014–15 season and the revised search-data inputs.

F Detailed description of Google Correlate data

Google Correlate identified search phrases for ARGO at different dates, and the earlier list included several spurious terms that disappeared from the later version.

  • F Detailed description of Google Correlate data: The March 2009 Google Correlate list included spurious phrases such as “college.basketball.standings” and “march.vacation.”These examples were identified among the search phrases used by the model.
  • F Detailed description of Google Correlate data: The spurious phrases in the March 2009 version did not appear in the May 2010 Google Correlate version.The comparison shows that the identified query set changed between the two dates.

G Dynamic coefficients for ARGO

ARGO dynamically estimates coefficients for autoregressive lags and Google search terms, using regularized specifications and changing query inputs. The supplied tables and figure materials describe its evaluation targets, penalties, query lists, and coefficient encoding.

  • G Dynamic coefficients for ARGO: The previous week’s ILI activity significantly affects current activity, while half-year and one-year lags can provide additional information.These effects are shown among the dynamically trained time-series coefficients.
  • G Dynamic coefficients for ARGO: ARGO selected 14 Google Correlate terms out of 100 on average each week.The dynamic procedure therefore uses a changing subset of the available query terms.
  • G Dynamic coefficients for ARGO: Table 3 compares model estimates against finalized CDC ILI activity while excluding forward-looking information from report revisions.RMSE, MAE, and MAPE are reported relative to a naive method, with the naive method’s absolute error in parentheses.
  • G Dynamic coefficients for ARGO: Table 5 compares five ARGO penalty specifications across the full in-sample period and selected partial or full flu seasons.The specifications vary same versus separate L1 or L2 penalties and an elastic-net penalty.
  • G Dynamic coefficients for ARGO: Tables 6 and 7 list Google Correlate search phrases identified using data from March 28, 2009 and May 22, 2010.These lists document the query sets associated with the two dates.
  • G Dynamic coefficients for ARGO: ARGO’s heatmap uses red for positive coefficients, blue for negative coefficients, white for zero, and grey for missing values.A black dashed line separates Google queries from autoregressive lags, while a yellow dashed line separates coefficient groups.
Loading 1505.00864v2…