Source-linked AI summary
Combining Search, Social Media, and Traditional Data Sources to Improve Influenza Surveillance
Mauricio Santillana, Andre T. Nguyen, Mark Dredze, Michael J. Paul, John S. Brownstein
TL;DR
Timely influenza surveillance is difficult because CDC ILI reports are not immediately available. The paper combines independent signals from healthcare, search, social-media, and participatory-surveillance data with machine-learning ensembles, producing accurate real-time estimates and forecasts up to four weeks ahead of CDC releases.
Problem
Predicting seasonal and non-seasonal influenza dynamics and obtaining timely ILI incidence estimates remain challenging, despite their importance for outbreak preparation and response.
Method
The methodology independently predicts ILI from multiple data sources and combines those estimates into one prediction using machine-learning ensemble approaches.
Results
The ensemble outperforms independently constructed real-time predictors and produces accurate weekly ILI predictions for up to four weeks ahead of CDC report releases.
Takeaways & Limitations
Combining multiple flu predictors is advantageous over choosing the best individual predictor across real-time, one-week, two-week, and three-week predictions.
Takeaways & Limitations
The CDC weekly reports used as the national flu-activity gold standard may not be ideal, and the best ensemble methodology may change across future flu seasons.
Abstract
from arXiv · showhide
We present a machine learning-based methodology capable of providing real-time ("nowcast") and forecast estimates of influenza activity in the US by leveraging data from multiple data sources including: Google searches, Twitter microblogs, nearly real-time hospital visit records, and data from a participatory surveillance system. Our main contribution consists of combining multiple influenza-like illnesses (ILI) activity estimates, generated independently with each data source, into a single prediction of ILI utilizing machine learning ensemble approaches. Our methodology exploits the information in each data source and produces accurate weekly ILI predictions for up to four weeks ahead of the release of CDC's ILI reports. We evaluate the predictive ability of our ensemble approach during the 2013-2014 (retrospective) and 2014-2015 (live) flu seasons for each of the four weekly time horizons. Our ensemble approach demonstrates several advantages: (1) our ensemble method's predictions outperform every prediction using each data source independently, (2) our methodology can produce predictions one week ahead of GFT's real-time estimates with comparable accuracy, and (3) our two and three week forecast estimates have comparable accuracy to real-time predictions using an autoregressive model. Moreover, our results show that considerable insight is gained from incorporating disparate data streams, in the form of social media and crowd sourced data, into influenza predictions in all time horizons
1. Introduction
Influenza surveillance requires timely incidence estimates, but predicting outbreak dynamics remains challenging. The paper proposes combining multiple Internet-based and healthcare data sources to produce real-time and future ILI estimates.
- Motivation: Influenza outbreaks remain difficult to predict, while their substantial mortality makes timely ILI estimates important for public-health preparation and response.The paper notes up to 500,000 deaths worldwide and an estimated 3,000 to 50,000 US deaths annually.
- Prior approaches: Existing approaches estimate US ILI activity ahead of CDC reports using statistical, mechanistic, search, social-media, participatory-surveillance, and clinical-query data.Examples include Google, Yahoo, Baidu, Twitter, Wikipedia, Flu Near You, and UpToDate queries.
- Contribution: The proposed methodology combines independently generated ILI estimates from Google searches, hospital visits, Twitter posts, and Flu Near You using machine-learning ensembles.The component predictors may capture different influenza-incidence signals and are shown to complement one another when combined.
- Contribution: The ensemble produces weekly ILI predictions for up to four weeks ahead of CDC report release, evaluated across the 2013-2014 and 2014-2015 flu seasons.The four weekly horizons include real-time estimation and forecasts extending three weeks into the future.
2. Data
The study combines CDC ILI reports with five independent data sources spanning healthcare visits, searches, Twitter, participatory surveys, and Google Flu Trends. These sources differ in timing, construction, and population coverage, supporting nowcasting and forecasting comparisons.
- Predictor sources: Five weak predictors are built from athenahealth visits, Google Trends, influenza-related Twitter posts, FluNearYou surveys, and Google Flu Trends.The datasets combine healthcare, search, social-media, participatory-surveillance, and proprietary search-based estimates.
- Reference data: The study uses revised CDC ILI reports as the gold standard for validation and includes CDC virology data only as a comparison source.CDC ILI reports are revised as additional reports arrive and become the official %ILI values.
- Healthcare data: athenahealth provides nationally aggregated weekly counts of people seeking ILI-related medical attention from July 2009 to February 2015.Its data are typically available at least one week before CDC ILI reports and are dynamically mapped to CDC ILI for out-of-sample prediction.
- Search data: Google Trends predicts flu activity from query-search volumes for 100 terms using a dynamic multivariate approach covering July 2013 through February 2015.The identity transformation was used for out-of-sample predictions because it performed better than the logit transformation in the cited methodology.
- Social-media data: Twitter predictions use a classifier trained on approximately 12,000 annotated tweets to identify messages expressing influenza infection.The resulting normalized weekly influenza-tweet volumes are available through HealthTweets.org.
- Participatory surveillance: FluNearYou collects weekly self-reports from voluntary participants about ILI symptoms and records participant location, vaccination status, gender, and age.The surveys are Internet-based and conducted weekly throughout the year.
3. Methods
The study combines independently generated influenza-like illness predictors with machine-learning ensembles to produce real-time estimates and forecasts. Across evaluation horizons, the ensemble improves on individual predictors and autoregressive baselines while retaining useful forecast accuracy.
- Ensemble methodology: Three machine-learning approaches—stacked linear regression, Support Vector Machine regression, and AdaBoost with Decision Trees Regression—combine five independently generated ILI estimates.The five estimates are produced from separate data sources, and the algorithms were selected for distinct strengths in combining information.
- Real-time evaluation: 0.989 Pearson correlation and about 0.176 %ILI RMSE characterize the real-time ensemble’s accuracy against revised CDC ILI.Its worst-case MAPE was 23.6%, comparable to LASSO’s 20.2%.
- Forecast evaluation: AdaBoost has the best accuracy, robustness, and correlation across the three forecast horizons, although hit rate varies by horizon.One-week ensemble forecasts have accuracy comparable to real-time Google Flu Trends predictions by RMSE.
- Forecast evaluation: Ensemble forecasts improve on the AR3 autoregressive model across all similarity metrics and horizons, with correlations increasing from 0.845 to 0.960, 0.759 to 0.927, and 0.683 to 0.904.The forecast horizons are one, two, and three weeks, respectively; ensemble RMSE nearly halves autoregressive errors.
- Forecast limitations: Peak-timing and magnitude accuracy decays with longer forecast horizons, reaching two-to-three-week lags and errors up to 25–30% for three-week forecasts.The three-week forecasts show about 20% error for the 2013–2014 peak and up to 25–30% for the 2014–2015 peaks.