Source-linked AI summary
Catching Zika Fever: Application of Crowdsourcing and Machine Learning for Tracking Health Misinformation on Twitter
Amira Ghenai, Yelena Mejova
TL;DR
Health rumors on social media can hamper public-health crisis response, yet tools and guidance for monitoring them were lacking. This paper combines expert-designed retrieval, crowdsourced annotation, and machine learning to track Zika rumors on Twitter. It reports bursty and varied rumor activity and finds that expert supervision is needed to identify health misinformation in social media.
Problem
Health rumors can hamper public-health crisis efforts, while social media is difficult to monitor and tools for tracking health misinformation were not yet available.
Source: §I · p. 1Method
The paper proposes a health-rumor tracking pipeline combining expert knowledge, handcrafted retrieval queries, crowdsourced tweet annotation, and machine-learning classifiers.
Source: §I · p. 1, A. Query Construction · p. 4, B. Crowdsourced Annotation · p. 4Results
Zika rumors showed bursty and varied patterns, and tracking health misinformation in social media required expert supervision, supplemented by crowd annotation.
Source: §VI · p. 8, §VII · p. 9Takeaways & Limitations
The study finds that expert supervision and crowd annotation can support tracking rumor-related tweets and clarifications in the Zika case.
Source: §VII · p. 9The study cannot establish how representative its social-media users are of the general population and examined only a handful of WHO- and Snopes-highlighted rumors.
Source: §VI · p. 9
Abstract
from arXiv · showhide
In February 2016, World Health Organization declared the Zika outbreak a Public Health Emergency of International Concern. With developing evidence it can cause birth defects, and the Summer Olympics coming up in the worst affected country, Brazil, the virus caught fire on social media. In this work, use Zika as a case study in building a tool for tracking the misinformation around health concerns on Twitter. We collect more than 13 million tweets -- spanning the initial reports in February 2016 and the Summer Olympics -- regarding the Zika outbreak and track rumors outlined by the World Health Organization and Snopes fact checking website. The tool pipeline, which incorporates health professionals, crowdsourcing, and machine learning, allows us to capture health-related rumors around the world, as well as clarification campaigns by reputable health organizations. In the case of Zika, we discover an extremely bursty behavior of rumor-related topics, and show that, once the questionable topic is detected, it is possible to identify rumor-bearing tweets using automated techniques. Thus, we illustrate insights the proposed tools provide into potentially harmful information on social media, allowing public health researchers and practitioners to respond with a targeted and timely action.
I. INTRODUCTION
The paper addresses the difficulty of monitoring health misinformation on social media, which can hamper public-health crisis response. It proposes tools for tracking Zika rumors on Twitter and applies social-media analytics to health-rumor tracking.
I. INTRODUCTION
- Health rumors can hamper efforts to address public-health crises, while social media is difficult for public-health institutions to monitor, track, and analyze.Although the WHO treats social media as important for crisis monitoring, guidance and tools for doing so were not yet available.Source: §I · p. 1
- The paper proposes tools and a methodology for tracking health-related misinformation on Twitter, identifying WHO- and Snopes-outlined rumors in nearly 13 million tweets using topic discovery and expert-led retrieval.Crowdsourcing distinguishes rumor from clarification tweets, which are then used to build automatic classifiers.Source: §I · p. 1
- The study presents its work as a first application of state-of-the-art social-media analytics to health-rumor tracking.The Zika crisis involved emerging concerns about birth defects, international spread, and the Rio Olympics; communication was important because there was no cure or vaccine at the time.Source: §I · p. 1
II. RELATED WORK
Prior work used social media to study health attitudes and detect rumors, but little research focused on tracking health-related rumors. This paper situates its broader Zika rumor-tracking pipeline against those strands of work.
Health-related attitudes on social media
- Earlier social-media studies examined health behaviors and attitudes, including drug use, adverse drug interactions, and food-related attitudes, often combining expert knowledge with data analytics.The cited work includes a Twitter marijuana-use classifier, studies of Xanax and Adderall attitudes, drug-interaction mining, and food attitudes on Instagram.Source: §II · p. 2
Rumor detection and analysis
- Rumor research used credibility analysis, machine learning, and labeled Twitter or Weibo datasets, while this paper addresses a setting where labeled data are unavailable and rumor documents must first be found.Related work also used content, network, and Twitter-specific features to track urban legends.Source: §II · p. 2
Health Related misinformation detection
- Health-rumor research included vaccine information tracking and crowdsourced crisis annotation, but the authors describe little work focused on detecting and tracking health rumors.Compared with a study of two Zika-vaccine memes, this work uses health professionals to expand the rumor set to six stories and examines rumors and clarification efforts.Source: §II · p. 2
III. DATA COLLECTION
The study assembled a multilingual Twitter collection about Zika spanning the outbreak’s early interest and the Rio Olympics, then focused its analysis on English tweets. It also estimated tweet locations using profile and metadata fields.
III. DATA COLLECTION
- The collection contains 13,728,215 Zika-related tweets gathered from January 13 through August 22, 2016, using medical and everyday-language keywords.The period includes the early-February peak in interest and the August 5–21 Olympic Games; Figure 1 presents volume by language.Source: §III · p. 2, Figure 1 · p. 3
- English, Spanish, and Portuguese account for 46%, 27%, and 17% of the dataset, respectively, and the analysis focuses on English-language tweets.No language restriction was imposed, although the English keywords introduced some bias.Source: §III · p. 2
- GPS coordinates were missing for 99% of tweets, so location estimation also used tweet place names, user profiles, and other geolocated tweets from the same user.The procedure achieved 68% coverage for English data; the most frequent locations included the United States, United Kingdom, India, Canada, Nigeria, and Brazil.Source: §III · p. 3
IV. RUMOR SELECTION
Rumor selection began with claims identified by WHO and Snopes, then retained topics that could be described clearly enough to classify tweets. The final set comprised six Zika rumor stories.
IV. RUMOR SELECTION
- The final selection contains six Zika rumor stories: four from WHO and two from Snopes.The topics were supervised by health experts to cover issues considered important and influential in the outbreak.Source: §IV · p. 3
- Of eight WHO statements debunking rumors, four were discarded because they were not topically cohesive or did not clearly state the rumor being debunked.The selection process required an unambiguous rumor description that would allow a tweet to be classified.Source: §IV · p. 3
- Snopes articles were included when the associated rumors were not specific to the United States.WHO served as an authority for identifying and verifying Zika rumors, while Snopes was selected for its expert sourcing.Source: §IV · p. 3
V. RUMOR TRACKING
Automated LDA topic discovery did not surface the selected Zika rumors, illustrating the need for expert knowledge to obtain a high-precision view. The accompanying table covers rumor queries and retrieved-tweet counts.
V. RUMOR TRACKING
- LDA topics did not capture the selected rumors; most were informational, spam, or jokes, motivating expert knowledge for high-precision analysis.The authors manually examined between 5 and 50 topics.Source: §V · p. 3
- The accompanying table lists rumor queries and the number of tweets retrieved.Source: Table III · p. 4
A. Query Construction
The authors built rumor-specific Boolean search queries using keywords, synonyms, and exclusions, then iteratively refined them against returned tweets. The resulting collection included false positives requiring later classification.
A. Query Construction
- Each rumor query used Boolean keyword combinations with synonyms and replacements, refined over at least three rounds of labeling top results.The queries used AND, OR, and NOT operators.Source: A. Query Construction · p. 4
- Query construction added medical synonyms and rumor-discriminating terms; the authors avoided automatic query expansion because it performed poorly on informal Twitter text.For R2, excluding “rash” distinguished the rumor from general information about Zika symptoms.Source: A. Query Construction · p. 4
- 89,572 tweets were retrieved across rumors, ranging from 73,832 to 202, but matches could be rumors, clarifications, or other tweets.Source: A. Query Construction · p. 4
B. Crowdsourced Annotation
Crowdflower workers labeled tweets as supporting a rumor, clarifying it, or neither, under quality controls and majority voting. Agreement averaged 87.7%, while the labeled set contained both rumors and clarifications.
B. Crowdsourced Annotation
- Workers assigned one of three labels—supporting a rumor, debunking it, or neither—with at least three annotations and majority-vote classification per tweet.Annotators below 70% accuracy on at least 20 gold-standard tweets were banned, and their annotations discarded.Source: B. Crowdsourced Annotation · p. 4
- Tweets were de-duplicated before labeling, with up to 1,000 unique tweets annotated per rumor; larger sets included 700 most-retweeted tweets and 300 sampled tweets.Labels were propagated to duplicates within each set.Source: B. Crowdsourced Annotation · p. 4
- Table IV reports class distributions for the six rumors, with propagated-label counts shown in parentheses.Source: B. Crowdsourced Annotation · p. 4
- Rumors averaged 51% of final tweets and clarifications 15%; annotator agreement averaged 87.7%, ranging from 76% to 93%.Source: B. Crowdsourced Annotation · p. 5
C. Temporal Tracking
The six rumors showed bursty patterns and uneven clarification responses. Their paths ranged from quickly fading spikes to longer-lived everyday concerns.
C. Temporal Tracking
- R4–R6 clarification volumes tracked rumor volumes fairly closely (r around 0.5), while R1–R3 showed mismatches.Figure 2 plots the rumor and clarification volumes for all six rumors.Source: C. Temporal Tracking · p. 5, Figure 2 · p. 5
- R1’s July spike followed a widely retweeted article linking chemical exposure to microcephaly, then quickly faded without detected authoritative-source intervention.The article gained thousands of retweets within days.Source: C. Temporal Tracking · p. 5
- The R2 flu-or-cold confusion generated many jokes and lighthearted tweets, although a large proportion of symptom-related tweets had that character.The paper notes that confusion between common illnesses and Zika could pose a problem for health authorities.Source: C. Temporal Tracking · p. 5
- R3’s April vaccine rumor receded quickly after a February–March WHO campaign, despite no detected authoritative clarification around the April spike.The rumor originated in an advocacy-site article claiming a Tdap vaccine caused microcephaly.Source: C. Temporal Tracking · p. 5
- R4–R6 had quick clarification interactions, including mainstream-media and WHO coverage for R4, Snopes links for R5, and sensationalized coffee claims for R6.The R6 tweets still linked to the original correct information.Source: C. Temporal Tracking · p. 5
- Across the six rumors, mainstream-covered R4–R6 declined quickly, as did advocacy-site-originated R1 and R3 without official response; everyday R2 and possibly R6 concerns lasted longer.The paper characterizes the longer-lived concerns as ones tied to daily occurrences that propagated in Twitter lore.Source: C. Temporal Tracking · p. 5
D. Rumor Classification
The section develops and evaluates tweet-level rumor classifiers using crowdsourced labels and Twitter, linguistic, readability, and medical-domain features. Performance was strong in the within-topic evaluation, but uneven when a topic was excluded from training.
D. Rumor Classification
- After removing exact duplicates, the labeled dataset contained 26,728 tweets: 8,488 rumors and 18,240 non-rumors, which combine clarifications and other tweets.Clarifications alone were too under-represented to serve as the non-rumor class.Source: D. Rumor Classification · p. 6
- The 48-feature set covered Twitter, sentiment, linguistic, readability, and medical features; readability and medical features were new to this work.Readability features measured text ease, while medical features included domain-specific lexical and URL-source signals.Source: D. Rumor Classification · pp. 6–7
- The medical lexicon contrasted term frequencies from 113 infectious-disease Wikipedia pages with an equally sized general-Wikipedia corpus, retaining the top 13,300 terms by probability difference.The URL features classified domains into advocacy, social media, news, informational, and non-informative groups.Source: D. Rumor Classification · p. 7
- Feature selection favored medical-domain signals such as advocacy and Wikipedia domains, alongside syntax, sentiment, and Twitter features.The selected top features are reported for Information Gain and Greedy Backward Elimination.Source: D. Rumor Classification · pp. 7–8, Table VII · p. 8
- Advocacy websites supplied 35.0% of URLs in rumor tweets versus 3.1% in non-rumors, while informational domains supplied 25.9% versus 64.0%.News URLs accounted for 39.1% in rumor tweets and 32.3% in non-rumors.Source: D. Rumor Classification · p. 7
- Precision 0.946 and recall 0.944 were achieved by the Random Tree classifier with the top 10 GBE features in 10-fold cross-validation.The authors caution that these results are overfitted because of limited data, feature selection on the test set, manual labels, and topical specialization.Source: D. Rumor Classification · p. 8, Table VIII · p. 8
- When training on five topics and testing on the sixth, precision was worst for topics 1 and 5, while recall fell below 0.500 for topics 3 and 4.The authors identify topically matched expert-labeled data as important in this setting.Source: D. Rumor Classification · p. 8, Table IX · p. 8
VI. DISCUSSION
The Zika analysis found that automatic topic discovery alone was too coarse for identifying concerning rumors, while expert queries and further review were needed. The discussion also highlights varied rumor lifetimes, uncertainty in identifying health rumors, and limits in the study’s coverage and integration with public-health workflows.
VI. DISCUSSION
- LDA was too coarse to identify the concerning rumors, and roughly half of the captured tweets were actual rumors, exposing the limits of keyword- or hashtag-based topic definitions.The authors therefore used expert knowledge to compose high-precision queries and called for additional examination of captured tweets.Source: §VI · p. 8
- Rumor longevity varied: everyday topics such as flu or coffee could persist through humor, while advocacy-site rumors could spike and dissipate without correction.Advocacy sites could adapt their stances to trending topics while maintaining their core message.Source: §VI · p. 8
- Mainstream news sites appeared in 39.1% of rumor tweets and 32.3% of other tweets, including clarifications; the authors emphasize authoritative sources beyond mainstream news for setting the record straight.They also cite prior work linking mainstream news to fear and misinformation during Ebola.Source: §VI · pp. 8–9
- Timely detection may help officials respond before rumors spread, but overreacting can amplify misconceptions, so awareness campaigns need real-time, international monitoring.The discussion notes examples in which rumors or vaccine-related concerns had serious consequences,,,.Source: §VI · p. 9
- The study covered rumors deemed certain by authoritative sources amid substantial uncertainty, and its scope was limited to a handful of WHO- and Snopes-raised rumors and users not known to represent the general population.The authors also note that identifying at-risk users and examining language- and culture-specific differences could provide further insight.Source: §VI · p. 9
- The authors call for closer integration with public-health communication workflows and note that volunteer labeling, as in AIDR, may support ongoing health emergencies.In that approach, volunteers provide labeled social-media data to train automated methods.Source: §VI · p. 9
VII. CONCLUSION
The paper presents a manually evaluated pipeline combining expert knowledge, crowdsourcing, and machine learning to track health-related rumors. Its conclusion emphasizes expert oversight, varied rumor behavior, authoritative information sources, and further work on multilingual analysis and related tools.
VII. CONCLUSION
- The pipeline combines expert knowledge, crowdsourcing, and machine learning for health-rumor discovery, with each analysis step tested by manual evaluation.The evaluation provides qualitative and quantitative insight into a process intended to collect data relevant to health-communication professionals.Source: §VII · p. 9
- Tracking health misinformation is not trivial and requires expert supervision, with crowd workers able to add annotations to captured rumor-related tweets.The authors describe Zika rumors as bursty and varied, with some provoked by advocacy groups and others propagated through humor or light banter.Source: §VII · p. 9
- The authors find traditional media sources were not prominent in clarifying rumors, while authoritative informational sources were important.Source: §VII · p. 9
- Future work will expand cross-language rumor analysis, study informational-campaign detection, and explore a user-friendly interface for data scientists and health-communication professionals.The proposed interface may involve expert input, as in AIDR.Source: §VII · p. 9