Source-linked AI summary
Botometer 101: Social bot practicum for computational social scientists
Kai-Cheng Yang, Emilio Ferrara, Filippo Menczer
TL;DR
Deceptive social bots can manipulate online discussions and complicate research using social-media data, motivating accessible and reliable detection tools. The paper addresses this need with an introductory tutorial on Botometer, covering its operation, access methods, a case study, and recommended practices; Botometer-V4 achieves an AUC of 0.99 on annotated datasets, while the case study finds differing automation patterns across stock-related discussions. The authors caution that scores are transient, classifiers can mislabel accounts, and conclusions from small, nonrepresentative samples should not be generalized.
Problem
Deceptive bots can manipulate online discussions, and their presence may distort social-media analyses, creating a need for reliable tools to detect them.
Method
The paper presents an introductory tutorial on Botometer, explaining its machine-learning approach, access methods, case-study workflow, and recommended research practices.
Results
Botometer-V4 achieves an AUC of 0.99 on annotated datasets, while the case study reports more automated activity for $SHIB and $FLOKI discussions but more highly automated bot-like accounts among $AAPL tweeters.
Takeaways & Limitations
Researchers can use Botometer through a web interface or API, adapt the shared case-study code, and validate thresholds for their specific datasets.
Takeaways & Limitations
Botometer scores are transient, individual accounts may be misclassified, and small nonrepresentative case-study samples should not be generalized.
Abstract
from arXiv · showhide
Social bots have become an important component of online social media. Deceptive bots, in particular, can manipulate online discussions of important issues ranging from elections to public health, threatening the constructive exchange of information. Their ubiquity makes them an interesting research subject and requires researchers to properly handle them when conducting studies using social media data. Therefore, it is important for researchers to gain access to bot detection tools that are reliable and easy to use. This paper aims to provide an introductory tutorial of Botometer, a public tool for bot detection on Twitter, for readers who are new to this topic and may not be familiar with programming and machine learning. We introduce how Botometer works, the different ways users can access it, and present a case study as a demonstration. Readers can use the case study code as a template for their own research. We also discuss recommended practice for using Botometer.
1 Introduction
Malicious social bots can manipulate online discussions and distort social-media research, creating a need for reliable detection tools. This tutorial introduces Botometer, explains its operation and access methods, demonstrates its use, and offers recommended practices.
- Bot activity has been documented across major political discussions, while deceptive bots continue evolving tactics to evade detection.The paper cites interference in elections and referendums across multiple countries.
- Malicious bots can amplify narratives, manipulate markets, target popular users, inflate popularity metrics, and spread inflammatory or hateful content.They may also form dense networks that boost engagement and amplify one another’s messages.
- Researchers need bot detection because bots remain prevalent and their presence may distort analyses of social-media data.Studying bot behavior and impact is also presented as an important research topic.
- The practicum provides an introductory tutorial on Botometer, a machine learning tool for detecting bots on Twitter.It is aimed at readers who may be unfamiliar with Botometer, machine learning methods, programming interfaces, or result interpretation.
- Botometer is emphasized because it is maintained, routinely upgraded, and accessible through a free web interface and an API for large-scale analysis.The API is available to researchers with Twitter developer accounts and charges a nominal fee for heavy use.
- The tutorial explains how Botometer works and how to access it, then provides a replicable case study and recommended practices.The case-study source code is shared as a template for readers’ own research.
2 How Botometer works
Botometer is a supervised machine-learning tool that classifies Twitter accounts using account features and trained classifiers. Its versions, scores, evaluation, and fast endpoint involve important interpretation and scope considerations.
- Versions and architecture: Botometer’s versions evolve through new training data, features, and architectures, so results and input/output formats across versions are usually not comparable.This tutorial focuses on V4, which introduced a new architecture, while BotometerLite was later added for fast detection.
- Supervised classification: Botometer is a supervised classifier that distinguishes bot-like from human-like accounts using features extracted from Twitter data.V4 uses specialized Random Forest classifiers for bot types and humans, whose outputs are aggregated into a final result.
- Training data: The model relies on labeled training data, although social-bot ground truth is difficult because no objective operational definition exists and some accounts fall between human and bot behavior.Training labels may come from human annotation, automated methods, or suspicious botnets, and serve as practical proxies for ground truth.
- Accuracy and limitations: V4 achieves an AUC of 0.99 in 5-fold cross-validation on annotated datasets, but performance can decline for inactive, novel, differently situated, or non-English accounts.Conflicting training labels and missing distinguishing features can also contribute to misclassification.
- Score interpretation: Botometer scores indicate relative bot-likeness rather than a direct probability that an account is automated.CAP provides a probabilistic interpretation by incorporating classifier results and prior knowledge about bot prevalence.
- Fast bot classification: BotometerLite improves scalability through selective training-data optimization, using five of eight datasets while performing well across its evaluation metrics.Its speed supports large-volume or historical analysis, but limited training data may reduce accuracy for certain bot classes relative to V4.
3 Botometer interface
Botometer can be accessed through a website or APIs, supporting both quick individual checks and programmatic bulk analysis. Researchers must supply appropriate credentials or account data and plan around API rate limits.
- Website access: The Botometer website supports quick account checks through a web browser after users authorize access to Twitter data.It is designed for straightforward use, including on mobile devices.
- API access: The Botometer Pro API supports programmatic bulk account checking for research applications.It requires a Twitter app’s keys, a RapidAPI account, and an API usage subscription.
- API workflow: API users submit an account’s 200 most recent tweets and mentioning tweets in a specified HTTPS format, then receive processed results.The maintained Python package can fetch Twitter data, format it, and query the API.
- API options: Botometer provides both Botometer-V4 and BotometerLite endpoints, with their inputs, outputs, and limitations compared in Table 2.The paper also summarizes shared Botometer resources in Table 3.
- Operational constraints: API query capacity is limited by rate limits, so researchers should consult documentation and estimate the time required for analysis.The listed rate-limit values are upper bounds for a single app key, and actual numbers also depend on factors such as internet speed.
4 Case study
The case study demonstrates a Botometer workflow from data collection and account querying through language filtering, bot-score analysis, and interpretation. Comparing cashtags shows that conclusions depend on the analysis measure and threshold, while the small, nonrepresentative sample limits generalization.
- Data collection: The case study collects 2,000 tweets for each of $FLOKI, $SHIB, and $AAPL, then queries Botometer while avoiding repeated account requests.The workflow uses Tweepy for collection and records already queried accounts to improve efficiency.
- Data preparation: The analysis uses tweets as units, so accounts posting the same cashtag repeatedly contribute more, and it uses the overall score for English-speaking accounts.The overall and language-independent scores come from different classifiers and are not mixed.
- Bot-score analysis: Bot-score distributions are bimodal, with repeated cryptocurrency-promoting replies producing a spike near 0.89 for $SHIB and $FLOKI.The spike is attributed to a bot-like account replying with the same message to many NFT-related tweets.
- Bot-score analysis: Mann–Whitney tests find no significant distribution difference between $SHIB and $FLOKI (p = 0.56), while each has higher bot activity than $AAPL (p < 0.001).These comparisons use raw bot-score distributions.
- Threshold analysis: Using a 0.5 threshold, likely-bot tweet percentages are higher for $SHIB than $FLOKI (p = 0.009) and $AAPL (p < 0.001), while a 0.7 threshold reverses the $SHIB/$AAPL and $FLOKI/$AAPL ordering.At 0.7, $SHIB and $FLOKI are not significantly different (p = 0.38), and both are lower than $AAPL (p < 0.001).
- Interpretation: The authors conclude that $SHIB and $FLOKI discussions show more automated activity than $AAPL, but $AAPL includes more highly automated bot-like accounts.They caution that the small, nonrepresentative samples reflect only the collected data and should not be generalized.
5 Recommended practice
Recommended practice emphasizes treating Botometer scores as time-sensitive and imperfect, validating threshold-based analyses, and avoiding misuse of account-level classifications. Researchers should favor distributional comparisons, assess robustness, and interpret automated-account labels cautiously.
- 5.1 Transient nature of Botometer scores: Botometer scores can fluctuate as accounts’ recent activity and mentions change, so a single score reflects only the evaluation moment.The tool analyzes an account’s 200 most recent tweets and other tweets mentioning it.
- 5.1 Transient nature of Botometer scores: Researchers should conduct bot analysis soon after data collection to reduce unavailable accounts and keep scores relevant.Accounts may be suspended or removed, making later analysis impossible.
- 5.2 Analyzing bot activity: Comparing bot-score distributions across account groups can reveal information while reducing uncertainty from Botometer’s imperfection and transient scores.The recommended approach is especially suited to large datasets and statistical analysis.
- 5.3 Validating thresholds: When thresholding is necessary, researchers should validate the threshold with annotated accounts or test multiple thresholds for robustness.Annotated accounts can support selecting thresholds using accuracy, precision, recall, or F1, depending on the objective.
- 5.4 Using Botometer in a civil way: Botometer should not be used to label opponents as bots: classifiers can mislabel accounts, and automation does not imply deception or maliciousness.The authors characterize such name-calling as a misuse that harms healthy and informative conversations.