Source-linked AI summary
The spread of low-credibility content by social bots
Chengcheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Kaicheng Yang, Alessandro Flammini, Filippo Menczer
TL;DR
Digital misinformation poses a major global risk, but evidence about the mechanisms behind its spread has been limited. This study analyzes Twitter diffusion during and after the 2016 U.S. presidential campaign and finds that social bots disproportionately amplify low-credibility content through early activity and targeted interactions, with humans retweeting them. The findings suggest that curbing social bots may help mitigate this spread, while deployment of detection tools risks false positives and censorship concerns.
Problem
The study addresses the limited systematic evidence about how social bots contribute to the spread of low-credibility content online.
Method
The study analyzes Twitter diffusion networks and messages linking to low-credibility articles, including bot activity, timing, targeting, and retweet interactions.
Results
Social bots disproportionately amplify low-credibility content through early spreading activity and replies or mentions targeting influential users, while humans retweet bot-shared articles.
Takeaways & Limitations
Curbing social bots may be an effective strategy for mitigating the spread of low-credibility content, with bot scores potentially helping prioritize accounts for review.
Takeaways & Limitations
Detection-tool deployment can produce false positives, including suspension of legitimate accounts and concerns about censorship.
Abstract
from arXiv · showhide
The massive spread of digital misinformation has been identified as a major global risk and has been alleged to influence elections and threaten democracies. Communication, cognitive, social, and computer scientists are engaged in efforts to study the complex causes for the viral diffusion of misinformation online and to develop solutions, while search and social media platforms are beginning to deploy countermeasures. With few exceptions, these efforts have been mainly informed by anecdotal evidence rather than systematic data. Here we analyze 14 million messages spreading 400 thousand articles on Twitter during and following the 2016 U.S. presidential campaign and election. We find evidence that social bots played a disproportionate role in amplifying low-credibility content. Accounts that actively spread articles from low-credibility sources are significantly more likely to be bots. Automated accounts are particularly active in amplifying content in the very early spreading moments, before an article goes viral. Bots also target users with many followers through replies and mentions. Humans are vulnerable to this manipulation, retweeting bots who post links to low-credibility content. Successful low-credibility sources are heavily supported by social bots. These results suggest that curbing social bots may be an effective strategy for mitigating the spread of online misinformation.
1 Introduction
Digital misinformation is widespread and potentially harmful, but its influence is difficult to establish systematically. Online vulnerability reflects interacting cognitive, social, and algorithmic biases, while social bots can exploit these vulnerabilities and amplify false content.
- Misinformation is widespread on social media and has been identified as a major global risk, although its effects on elections and democracies are difficult to prove.Reported harms include dangerous health decisions and stock-market manipulation.
- Online vulnerability to misinformation reflects interacting cognitive, social, and algorithmic biases.Examples include information overload, novelty, selective exposure, popularity bias, confirmation bias, and motivated reasoning.
- Social bots are software-controlled profiles or pages that can imitate real users, interact through social connections, and target people likely to believe misinformation.Their strategies exploit tendencies to attend to popular content and trust information or contacts in social settings.
- Systematic evidence is needed to assess whether misinformation spreads mainly through cognitive limitations, echo chambers, or malicious bots.The literature on bots’ role in misinformation has been largely anecdotal or limited.
2 Results
Across Twitter, low-credibility content showed distinctive, bot-associated spreading patterns despite popularity distributions similar to fact-checking content. Bots were especially active early, targeted influential users, and were frequently retweeted by humans.
- Data and overall diffusion: Low-credibility articles had popularity distributions indistinguishable from fact-checking articles, although massive numbers of people encountered low-credibility content.Most articles went unnoticed, while a significant fraction became viral across several orders of magnitude in popularity.
- Anomalous spreading patterns: Low-credibility content spread mainly through original tweets and retweets, with few replies, and increasingly concentrated posting activity among a small group of super-spreaders as popularity rose.A single account could post the same article hundreds or thousands of times, suggesting amplification through automated means.
- Bot involvement: Super-spreaders were more likely to be bots than a random sample of accounts posting low-credibility links.Botometer scores were used to evaluate accounts and classify automation with a 0.5 threshold.
- Bot strategies: Bots were prevalent in the first few seconds after viral low-credibility articles first appeared on Twitter.The authors conjecture that this early intervention exposes users to articles and increases their chances of becoming viral.
- Bot strategies and human interaction: Accounts with the largest bot scores tended to mention users with more followers, while humans did most of the retweeting and retweeted bot-posted articles as much as human-posted articles.These patterns suggest targeting of influential users and difficulty distinguishing bot-shared from human-shared low-credibility content.
- Network impact: Disconnecting high-bot-score accounts was the second-best strategy for reducing unique low-credibility articles, behind disconnecting influential accounts.For reducing overall post volume, the bot-score strategy performed well when more than a certain number of nodes were removed.
- Source-level bot support: Popular low-credibility sources had many bots among their promoters, unlike satire and fact-checking websites, while beforeitsnews.com showed especially high automation.Low-credibility sources also had greater bot support and greater median and/or total volume in many cases.
3 Discussion
The analysis finds that social bots play a central role in amplifying low-credibility content, using early amplification and targeting influential users. The authors discuss robustness, methodological novelty, platform countermeasures, and trade-offs in mitigation.
- Findings: Social bots amplify low-credibility content early, target influential users through replies and mentions, and are retweeted by humans.These mechanisms help low-credibility content achieve reach statistically indistinguishable from fact-checking articles.
- Novelty: The findings provide new evidence because the study covers a broader set of low-credibility articles and includes bots resharing links first posted by humans.The comparison contrasts this analysis with work based on fact-checked articles that omitted this amplification mechanism.
- Robustness: The conclusions remain qualitatively similar across stricter source selection, active-spreader thresholds, and bot-score thresholds.Activity and bot scores were also uncorrelated with account activity volume.
- Scope: The study’s Twitter focus limits direct generalization, although the authors note that similar abuse may occur on other platforms.Limited access to spreading data on platforms such as Facebook and ephemeral services constrains future studies.
- Implications: Curbing social bots may mitigate low-credibility content, but automated detection risks false-positive suspensions and censorship concerns.Human-in-the-loop review avoids some errors but does not scale to software-enabled abuse.
- Implications: CAPTCHAs could increase the cost of automatic posting or resharing, but may add friction to legitimate automation.The authors identify this as a trade-off requiring careful study.
4 Methods
The study combines complete Twitter-link data with source-based identification of low-credibility content, URL normalization, account metadata, and Botometer scores. It analyzes source robustness, propagation networks, targeting, and bot activity using operational thresholds and graph measures.
- Data collection: Researchers collected 389,569 articles from 120 low-credibility sites and tracked 15,053 stories from independent fact-checking organizations.The collection ran from mid-May 2016 through the end of March 2017, and satire was retained.
- Source selection: The source-based proxy assumes that many articles from listed low-credibility sources are misinformation or unsubstantiated information.A random article sample was checked using an industry-convention definition of misinformation.
- Data collection: The dataset contains 13,617,425 public posts linking to low-credibility articles and 1,133,674 posts linking to fact checks.These were described as the complete set of tweets linking to the tracked articles during the study period.
- Preprocessing: Researchers canonicalized URLs by resolving redirects and removing analytics parameters to merge links referring to the same article.The passage reports that 44% of links were redirected and 34% contained tracking parameters.
- Bot classification: Botometer assigns account bot scores using more than a thousand public-data and metadata features, with a 0.5 threshold used to classify accounts.The score is calibrated as a confidence level using recent tweets and mentions retrieved through Twitter’s APIs.
- Network analysis: The network analysis uses retweet links, weighting directed edges by retweet counts and measuring activity and influence with in-strength and out-strength.The pre-election network contains 630,368 accounts and 2,236,041 directed edges.
Appendix: Supplementary Background
The supplementary background situates the study within systems for monitoring misinformation, rumor propagation, meme promotion, and social-bot detection. It emphasizes source-based proxies and the gray area between human and fully automated accounts.
- Monitoring systems: Existing misinformation-monitoring systems include tools for propagation analysis, rumor detection, verification, and interactive exploration.These systems differ in automation and generally require a seed rumor or keyword.
- Measurement: Using sources as proxies for misinformation labels is increasingly adopted because fact-checking millions of articles is impossible.The present analysis follows this source-based approach rather than selecting individually labeled stories.
- Related methods: Machine learning has been used to distinguish organic trending memes from memes artificially promoted through advertising.This work addresses artificial promotion rather than the paper’s specific source-based bot analysis.
- Social bots: Bot-based manipulation spans a gray area between human and completely automated accounts, including cyborgs that amplify human-generated content.The paper notes that some of the studied manipulation may involve this type of account.
List of Sources
The source list records which independent organizations or experts classified each low-credibility site and when Hoaxy began tracking it. A consensus subset identifies sources appearing on at least three lists.
- Consensus subset: Consensus sources are those appearing in at least three lists, yielding a 65-source subset for robustness analysis.The subset accounts for 327,840 articles and 10,663,818 tweets.
- Registry notes: The source registry includes mirrored domains and notes that climatefeedback.org was added in April 2017 without affecting the analysis.Mirror relationships are explicitly marked for several listed domains.
- Source registry: Table 1 maps each low-credibility source to the third-party lists that include it.The listed organizations include Fake News Watch, OpenSources, Daily Dot, NPR, Snopes, BuzzFeed News, and PolitiFact.
- Tracking dates: The table’s date records whether Hoaxy began following a source on June 29 or December 20, 2016.The table caption identifies these as the two tracking-start dates.
Hoaxy Data
The Hoaxy system collected public posts linking to tracked fact-checking and low-credibility websites from mid-May 2016 through March 2017. Low-credibility sources produced roughly 100 articles weekly, while article popularity varied widely.
- The study period ran from mid-May 2016 to the end of March 2017, with a brief collection interruption in October 2016.
- The low-credibility source set expanded from 70 to 120 websites in December 2016.
- 13,617,425 public posts linked to 389,569 low-credibility articles, compared with 1,133,674 posts linking to 15,053 fact-checking articles.
- Low-credibility websites published approximately 100 articles per week on average, and those articles received approximately 30 tweets per article per week near the study’s end.
- Popularity distributions were broad and essentially indistinguishable between low-credibility and fact-checking articles, whether measured by tweets or sharing accounts.
Content Analysis
The content analysis sampled articles from source-based and tweet-weighted perspectives, retrieved early versions when possible, and used independent reviewer labeling. The estimated false-positive rate was below 15% in both samples.
- The source-based approach assumes most articles from compiled low-credibility websites are misinformation, because individually fact-checking millions of articles is infeasible.
- A source-weighted sample selected sources uniformly before selecting articles, whereas the N = 50 tweet-weighted sample favored articles from more popular sources.
- The earliest available Wayback Machine snapshot was used to reduce overestimation from articles updated after debunking.
- Below 15% of articles were verified as false positives in both samples after independent review and majority labeling.
Concentration
The Gini coefficient measures how concentrated an article’s posting activity is across accounts. Higher values indicate that a small subset of accounts generated a large share of posts.
- The Gini coefficient is calculated from the Lorenz curve relating cumulative tweets to cumulative posting accounts for each article.
- A high coefficient indicates that a small subset of accounts was responsible for a large portion of an article’s posts.
Bot Score Calibration
Bot score calibration maps classifier outputs to probabilistic scores that can be interpreted as confidence levels. Reliability diagrams compare predicted scores with true-positive fractions, with better calibration closer to the diagonal.
- Platt’s scaling uses logistic regression trained on classifier outputs to calibrate Botometer’s bot scores.
- Calibration changes scores within the unit interval while preserving the ranking among accounts.
- The reliability analysis divides the unit interval into 20 bins before comparing predicted scores with true-positive fractions.
- Reliability curves compare each bin’s mean predicted score with its fraction of true-positive cases; well-calibrated points align with the diagonal.
Bot Classification
The paper classifies Twitter accounts by bot score to examine whether highly active spreaders of low-credibility content are automated. Multiple analyses compare bot activity, amplification, targeting, and robustness across source types and classification thresholds.
- Classification method: Botometer scores sampled accounts sharing low-credibility articles to estimate their level of automation.The study evaluated 915 inspectable accounts from a random sample of 1,000; 85 were unavailable because they were suspended, deleted, or private.
- Super-spreaders: Accounts that spread low-credibility articles most actively are more likely to be social bots.Super-spreaders were selected by ranking accounts according to the number of tweets linking to low-credibility sources.
- Super-spreaders: A single account sometimes posted the same low-credibility article hundreds or thousands of times, producing a long tail of repeated sharing.Figure 11 plots the cumulative distribution of repetitions by individual accounts.
- Targeting: Bots repeatedly mentioned influential users in tweets linking to low-credibility content, including 19 posts mentioning @realDonaldTrump for one article.The targeting analysis examines replies and mentions in the diffusion network.
- Amplification: Bot activity is higher for low-credibility than fact-checking articles and selectively amplifies low-credibility content.For low-credibility articles, human tweets grow faster than bot tweets; fact-checking articles show a linear relationship, and conclusions remain robust across bot-score thresholds.
- Robustness: The study’s robustness checks found qualitatively similar results under a stricter source criterion and no correlation between account activity volume and bot scores.Different bot-score thresholds preserve the conclusions, although estimated percentages and tweet counts vary with the threshold.