Source-linked AI summary
OMG U got flu? Analysis of shared health messages for bio-surveillance
Nigel Collier, Nguyen Truong Son, Ngoc Mai Nguyen
TL;DR
Traditional physician surveillance can take about two weeks to disseminate data, motivating interest in earlier epidemic signals. This study classifies Twitter messages about self-protective behavior and influenza diagnosis, reporting moderately strong correlation with CDC A(H1N1) laboratory data.
Problem
Traditional sentinel networks can take about two weeks to disseminate data, creating interest in earlier signals that could help detect outbreaks.
Source: Introduction · pp. 1–2Method
The study applies annotation guidelines and supervised classifiers to Twitter reports of self-protective behavior and self-reported influenza diagnosis, then compares message counts with laboratory data.
Source: Annotation · p. 3, Results 1: classification experiments · p. 5, Results 2: comparison to CDC data · p. 8Results
The study reports moderately strong correlation between classified message counts and CDC A(H1N1) laboratory data, with the strongest correlation when avoidance, sanitation, and pharmaceutical-intervention messages were combined.
Source: Conclusions · p. 8, Results 2: comparison to CDC data · p. 8Takeaways & Limitations
The authors see the modeled signals as potentially applicable to diseases beyond influenza and as a possible resource for evaluating official prevention campaigns.
Source: Conclusions · p. 9The study found no evidence that pre-diagnostic messages predicted laboratory case rises and largely ignored reporting bias and Twitter users’ demographic characteristics.
Source: Conclusions · p. 8
Abstract
from arXiv · showhide
Background: Micro-blogging services such as Twitter offer the potential to crowdsource epidemics in real-time. However, Twitter posts ('tweets') are often ambiguous and reactive to media trends. In order to ground user messages in epidemic response we focused on tracking reports of self-protective behaviour such as avoiding public gatherings or increased sanitation as the basis for further risk analysis. Results: We created guidelines for tagging self protective behaviour based on Jones and Salath\'e (2009)'s behaviour response survey. Applying the guidelines to a corpus of 5283 Twitter messages related to influenza like illness showed a high level of inter-annotator agreement (kappa 0.86). We employed supervised learning using unigrams, bigrams and regular expressions as features with two supervised classifiers (SVM and Naive Bayes) to classify tweets into 4 self-reported protective behaviour categories plus a self-reported diagnosis. In addition to classification performance we report moderately strong Spearman's Rho correlation by comparing classifier output against WHO/NREVSS laboratory data for A(H1N1) in the USA during the 2009-2010 influenza season. Conclusions: The study adds to evidence supporting a high degree of correlation between pre-diagnostic social media signals and diagnostic influenza case data, pointing the way towards low cost sensor networks. We believe that the signals we have modelled may be applicable to a wide range of diseases.
Introduction
Traditional sentinel networks can be slow, while social communication offers a lower-cost opportunity for timely outbreak surveillance. Micro-blogs may help surface local or pre-diagnostic signals that aggregated reports can miss.
Introduction
- Web-based crisis surveillance already includes Twitter event maps, Flutracking, and Ushahidi, while operational systems have relied on agency collaboration for verification and risk assessment.The paper notes that multiple sensor sources can provide verification, robustness, and redundancy despite risks of false reporting.Source: Introduction · pp. 1–2
- Traditional sentinel physician networks take about two weeks to receive and disseminate data, motivating interest in faster surveillance signals.The paper frames social communication as a lower-cost source that could complement traditional sensor networks.Source: Introduction · pp. 1–2
- Micro-blogs may reveal local outbreak signals or pre-diagnostic trends that are obscured in country-level aggregates.The authors also cite mobile-device messaging and automatic geo-coding as advantages for early detection.Source: Introduction · p. 2
Background
Social-media and query signals can track influenza activity, but messages are ambiguous and can react to media coverage rather than report personal experience. This study focuses on self-protective behavior as a candidate signal for epidemic monitoring.
Background
- Tweets can reflect personal experience, but ambiguity, risk-perception responses, and media reactivity complicate their use as epidemic signals.The authors note that posts may redistribute links or requests for information instead of describing user experience.Source: Background · p. 2
- The study focuses on precautionary actions identified in Jones and Salathé’s behaviour response survey as candidate features for detecting epidemic trends.The authors propose aberration detection on sharp rises in reliable features and suggest local risk-perception signals may add to event-based models.Source: Background · p. 2
- Prior studies reported strong influenza correlations for Twitter and Google Flu Trends, including Twitter data from 49 UK urban centres and Google query data during the 2009 A(H1N1) pandemic [11].The cited studies compared signals with ILI or acute respiratory infection data from sentinel networks.Source: Background · pp. 2–3
- Google Flu Trends correlated less strongly with influenza test-positive results than with ILI, and searches deviated from ILI during a high-activity, high-coverage season.The discussion identifies reporting bias during media storms as an issue for interpreting query and social-networking data.Source: Background · p. 3
Methods
The authors define five message categories and annotation rules, then build a manually labelled corpus and compare supervised classifiers with rule-based features. Annotation agreement was high, while the sanitation class lacked enough examples for hand-built rules.
Annotation
- The annotation scheme covers avoidance, increased sanitation, pharmaceutical intervention, mask wearing, and self-reported influenza diagnosis.The categories were adapted from Jones and Salathé’s behaviour responses; low-frequency initial categories were combined into four protective-behavior classes.Source: Annotation · p. 3
- A message was positive only for the user or a close family member, non-hypothetical action, and an event within the previous week; messages could receive multiple labels.The authors say anonymized examples are shown in Table 1.Source: Annotation · p. 3
- Keyword filtering reduced the initial pool to about 225,000 influenza-topic tweets, then pattern filtering and deduplication yielded 5,283 messages for manual classification.The 5,283 messages were sampled from 7,412 tweets selected across the five classes.Source: Annotation · pp. 3–4
- A second annotator achieved 0.88 simple agreement and kappa 0.86 on 2,116 balanced messages, supporting the reproducibility of the annotation scheme.The second annotator had not helped create the guidelines and was not a study co-investigator.Source: Annotation · p. 4
Models
- Naive Bayes and SVM classifiers used unigram and bigram features, with SVM tuned by grid search; SRL regular-expression rules added structured features when they matched.Rules were built from held-out tweets and covered lexical variants, entities, verbs, time expressions, modality, and negation; sanitation lacked enough confident examples for meaningful hand-built rules.Source: Models · p. 5
Results 1: classification experiments
SVM generally outperformed Naive Bayes except for mask-wearing classification, and undersampling improved performance across classes. Rule features helped most with unigrams, while classification errors also varied with message length.
Results 1: classification experiments
- Undersampling negative examples improved performance uniformly, including the otherwise poorly performing mask-wearing class; SVM outperformed NB overall except for that class.The authors attribute the low mask-class F-scores to strongly biased negative data; undersampling results were not shown.Source: Results 1: classification experiments · p. 5
- Model performance generally followed training-data volume, although self-diagnosis F-score was lower than that trend despite many positive examples.NB tended toward higher recall than precision, while SVM generally had higher precision than recall.Source: Results 1: classification experiments · p. 5
- SRL rules provided substantial benefits with unigrams, but improvements were less certain when bigrams were added.The comparison is reported across the classification experiments.Source: Results 1: classification experiments · p. 5
F1 results
- For NB, false positives fell from 0.23 on 34–56-character messages to 0.08 on 122–144-character messages, while false negatives remained about 0.2–0.25 across 34–144 characters.For SVM, both false positives and false negatives generally decreased as message length increased.Source: F1 results · p. 6
- Misspellings, abbreviations, slang, and missing punctuation complicated classification and required compensation within the SRL rules.The authors specifically note missing auxiliary verbs and articles as forms that rules needed to accommodate.Source: F1 results · p. 6
Results 2: comparison to CDC data
The SVM UNI model was applied to keyword-filtered Edinburgh Corpus tweets and compared with U.S. laboratory data. Aggregated self-protection counts showed a moderately strong association with CDC data, though positive-message volume was small and false positives remained.
Results 2: comparison to CDC data
- The comparison used the SVM UNI model on 52,193 keyword-filtered tweets from the Edinburgh Corpus.The corpus contains 97 million tweets from November 11, 2009, to February 1, 2010.Source: Results 2: comparison to CDC data · p. 6
- Positive tweets were relatively few; wearing a mask was absent, while aggregated self-protection counts appeared to track CDC results closely.The table compares positively identified tweets with influenza-positive tests; week 46 had only partial Twitter data.Source: Results 2: comparison to CDC data · p. 7, Table 5 · p. 7
- The strongest Spearman’s Rho correlation with CDC A(H1N1) laboratory data came from combining avoidance, increased sanitation, and personal protection; increased sanitation alone was weakest, and wearing a mask yielded no data.The comparison covered weeks 47 to 5 of the 2009–2010 U.S. influenza season.Source: Results 2: comparison to CDC data · pp. 7–8
- False positives included questions, hypotheticals, distant-past events, and comments relaying influenza advice, motivating stronger linguistic features within the 140-character limit.Source: Results 2: comparison to CDC data · p. 8
Conclusions
The authors report moderately strong correlations with CDC A(H1N1) data but no evidence that the messages predicted laboratory case rises. They identify unmodeled biases and propose broader training and future applications beyond influenza.
Conclusions
- The authors report moderately strong correlation with CDC A(H1N1) data and plan to expand training data, address class imbalance, strengthen linguistic features, and test outbreak detection.They also plan to examine tracking over longer periods.Source: Conclusions · p. 8, Table 6 · p. 8
- The study found no evidence that pre-diagnostic self-protection or self-reporting messages predicted laboratory case rises, and positively classified message volume was lower than expected.Source: Conclusions · p. 8
- Reporting bias, Twitter users’ location and age, and possible links between preventive behavior and perceived disease severity were not addressed.Source: Conclusions · p. 8
- The authors propose exploring these signals for diseases beyond influenza and for evaluating prevention campaigns and informing risk communication.Source: Conclusions · p. 9
Author details
The authors list their institutional affiliations and divide study responsibilities across conception, corpus and guideline work, and machine-learning experiments. They report no competing interests.
Author details
- The listed affiliations are the National Institute of Informatics, Japan Science and Technology Agency, and Vietnam National University at HCMC.Source: Author details · p. 9
- NC conceived and directed the study; NM handled corpus collection, analysis, and guideline development; NS conducted the machine-learning experiments.All authors contributed to and agreed on the final manuscript.Source: Author details · p. 9
Competing interests
- The authors declared no competing interests.Source: Competing interests · p. 9