Source-linked AI summary
"I Wanted to Predict Elections with Twitter and all I got was this Lousy Paper" -- A Balanced Survey on Election Prediction using Twitter Data
Daniel Gayo-Avello
TL;DR
Twitter-based election prediction lacks balanced, sound evidence. This paper surveys the existing research and concludes that elections cannot be consistently predicted from Twitter.
Problem
Current Twitter election-prediction research lacks balanced, sound, and reproducible evidence.
Method
The paper surveys and summarizes the main problems in state-of-the-art election prediction using Twitter data.
Results
The review concludes that elections cannot be consistently predicted from Twitter.
Takeaways & Limitations
Serious research should address demographic bias and other unresolved problems before relying on Twitter for election prediction.
Takeaways & Limitations
Existing studies generally analyze whether predictions could have been made after the fact rather than predicting future election results.
Abstract
from arXiv · showhide
Predicting X from Twitter is a popular fad within the Twitter research subculture. It seems both appealing and relatively easy. Among such kind of studies, electoral prediction is maybe the most attractive, and at this moment there is a growing body of literature on such a topic. This is not only an interesting research problem but, above all, it is extremely difficult. However, most of the authors seem to be more interested in claiming positive results than in providing sound and reproducible methods. It is also especially worrisome that many recent papers seem to only acknowledge those studies supporting the idea of Twitter predicting elections, instead of conducting a balanced literature review showing both sides of the matter. After reading many of such papers I have decided to write such a survey myself. Hence, in this paper, every study relevant to the matter of electoral prediction using social media is commented. From this review it can be concluded that the predictive power of Twitter regarding elections has been greatly exaggerated, and that hard research problems still lie ahead.
A Balanced Survey on Election Prediction using Twitter Data · Introduction
The survey argues that Twitter’s electoral predictive power has been exaggerated, highlighting post-hoc evaluation, weak baselines, inconsistent vote and truth definitions, biased data, and unreliable sentiment analysis. It recommends future research centered on genuinely prospective prediction, validated methods, credibility, demographic adjustment, and the silent majority.
- Introduction: The author’s position is that elections cannot be predicted with Twitter, motivating a critical survey of the field.The survey also examines data biases, research flaws, and reproducibility concerns in prior work.
- Flaws in Current Research regarding Electoral Predictions using Twitter Data: No paper reviewed predicts a future election; instead, studies claim predictions could have been made retrospectively, while negative results are rare.This makes the literature’s positive predictive claims difficult to assess prospectively.
- Recommendations for Future Research regarding Electoral Predictions using Twitter Data: Chance is an invalid baseline when incumbency strongly influences elections; researchers should compare against incumbent-win predictions across prior elections.A method that is not substantially better than this baseline is merely a convoluted version of it.
- Flaws in Current Research regarding Electoral Predictions using Twitter Data: Current studies lack accepted standards for counting Twitter votes and interpreting electoral reality, comparing outputs with tweets, users, sentiment, polls, popular vote, or representation shares.The survey calls for explicit definitions of both votes and the golden truth, preferably using actual election results rather than polls.
- Flaws in Current Research regarding Electoral Predictions using Twitter Data: Most sentiment-based classifiers are only slightly better than random classifiers, because sentiment analysis is often applied as a naïve black box.The author recommends dedicated political sentiment research, including humor and sarcasm detection.
- Recommendations for Future Research regarding Electoral Predictions using Twitter Data: Twitter election studies ignore rumors, propaganda, misleading information, demographic underrepresentation, and self-selection by politically active users.The author recommends credibility checks, removal of disinformation and automated accounts, and adjustment for demographic participation and group membership.
- Recommendations for Future Research regarding Electoral Predictions using Twitter Data: The silent majority is a huge problem and should become a central topic of future research.The survey notes that very little has been studied about users who do not participate on Twitter.
- Core Lines of Future Research: The proposed research agenda includes accurate political sentiment analysis, automated propaganda, disinformation and sock-puppet detection, demographic profiling, and study of participation and self-selection bias.These are identified as major lines of work for forecasting from political tweets.
Relevant Prior Art
Prior research produced mixed and often overstated evidence for predicting elections from Twitter. Studies identified demographic and participation biases, methodological fragility, and performance generally below traditional polls or only slightly above chance.
- Early evidence: Early studies linked Twitter mood or sentiment to socioeconomic, consumer-confidence, and presidential-approval measures, but found no correlation with electoral polls or predictive election conclusions.Bollen et al. reported delayed mood fluctuations from socioeconomic turmoil, while O’Connor et al. found no correlation between electoral polls and Twitter sentiment.
- Tweet-counting claims: Tumasjan et al. claimed that counting party or candidate mentions accurately reflected election results, but Jungherr et al. found arbitrary choices and time-window sensitivity.Jungherr et al. refuted Tumasjan et al.’s claim, while a later response weakened the original interpretation by presenting Twitter as complementary to polls.
- Data limitations: Twitter’s user base and activity are nonrepresentative: highly populated counties and some demographic groups are overrepresented, while a vocal minority dominates content over a silent majority.Political vocal minorities can act as resonance chambers, and demographic bias should be acknowledged and corrected in predictions.
- Predictive performance: Across elections, Metaxas et al. found Twitter prediction only slightly better than chance, while Livne et al. reported 88% precision with Twitter versus 81% without it.Livne et al. modeled elections as binary outcomes, omitting information about tight elections and possible coalitions.
- Methodological and external problems: The literature also highlights publication bias, naïve sentiment classifiers, failed election forecasts, and credibility problems because tweet content and user heuristics can be manipulated.Truthy addressed astroturfing and disinformation detection, while other work examined smear campaigns and credibility assessment.
- Predictive performance: Sentiment analysis sometimes improved performance, but Bermingham and Smeaton found their approach uncompetitive with traditional polls, and Dutch-election methods remained below polls.The Dutch study also required polling data to correct demographic differences, sharing a limitation with Bermingham and Smeaton’s work.