Source-linked AI summary
TwiBot-20: A Comprehensive Twitter Bot Detection Benchmark
Shangbin Feng, Herun Wan, Ningnan Wang, Jundong Li, Minnan Luo
TL;DR
Existing Twitter bot benchmarks lack user diversity, comprehensive information, and sufficient scale for stable supervised detection evaluation. TwiBot-20 constructs a large, diverse benchmark with semantic, property, and neighborhood data, then evaluates existing detectors. Existing methods perform worse on TwiBot-20 than on earlier datasets, indicating that Twitter bot detection remains challenging.
Problem
Existing benchmarks lack user diversity, comprehensive user information, and sufficient data scale for training and stable benchmarking of supervised Twitter bot detectors.
Method
TwiBot-20 constructs a large benchmark of diverse users with semantic, property, and neighborhood information, including follow relationships.
Results
Existing bot detection measures achieve significantly lower performance on TwiBot-20 than on cresci-17 or PAN-193.
Takeaways & Limitations
Twitter bot detection remains a challenging open problem requiring continued research, while TwiBot-20 supports community-aware and multimodal approaches.
Takeaways & Limitations
Bot-detection annotation is heavily influenced by underlying assumptions and subject to bias despite the paper’s trustworthiness procedures.
Abstract
from arXiv · showhide
Twitter has become a vital social media platform while an ample amount of malicious Twitter bots exist and induce undesirable social effects. Successful Twitter bot detection proposals are generally supervised, which rely heavily on large-scale datasets. However, existing benchmarks generally suffer from low levels of user diversity, limited user information and data scarcity. Therefore, these datasets are not sufficient to train and stably benchmark bot detection measures. To alleviate these problems, we present TwiBot-20, a massive Twitter bot detection benchmark, which contains 229,573 users, 33,488,192 tweets, 8,723,736 user property items and 455,958 follow relationships. TwiBot-20 covers diversified bots and genuine users to better represent the real-world Twittersphere. TwiBot-20 also includes three modals of user information to support both binary classification of single users and community-aware approaches. To the best of our knowledge, TwiBot-20 is the largest Twitter bot detection benchmark to date. We reproduce competitive bot detection methods and conduct a thorough evaluation on TwiBot-20 and two other public datasets. Experiment results demonstrate that existing bot detection measures fail to match their previously claimed performance on TwiBot-20, which suggests that Twitter bot detection remains a challenging task and requires further research efforts.
1 INTRODUCTION
Existing Twitter bot benchmarks lack stable coverage because they often underrepresent user diversity, omit modalities of user information, and remain small. TwiBot-20 addresses these gaps with diverse sampling, multimodal data, and a substantially larger benchmark.
- Existing bot detection datasets generally fail to provide a stable benchmark because of user diversity, information, and data scarcity problems.
- TwiBot-20 collects users through breadth-first traversal from many seeds across different topics, producing geographic and interest-domain diversity.
- TwiBot-20 retrieves semantic, property, and neighborhood information, including follow relationships for community-aware detection.
- 229,573 users, 33,488,192 tweets, 8,723,736 user property items, and 455,958 follow links make TwiBot-20 the largest benchmark described.
- The paper evaluates existing detection measures on TwiBot-20 after reviewing prior datasets, defining the task, and describing collection, analysis, and experiments.
2 RELATED WORK
Prior Twitter bot detection work spans feature engineering and neural methods, while datasets differ substantially in scale, composition, and available information. TwiBot-20’s selection algorithm is presented as a benchmark-construction method.
- Early Twitter bot detectors engineered features from URLs, websites, campaigns, homepages, social networks, and temporal behavior.
- Deep-learning approaches use generative adversarial, recurrent, LSTM, and graph convolutional networks with progressively richer user information.
- Earlier datasets include caverlee-2011’s content polluters and legitimate users and cresci-2015’s genuine and fake-follower accounts.
- TwiBot-20’s user-selection algorithm takes a seed user and builds a user-information set through layered expansion over follow relationships.
3 PROBLEM DEFINITION
The paper defines Twitter bots as automated software controlling accounts through programs and the Twitter API, and frames detection as prediction from three information modalities. The objective is to approximate ground-truth labels accurately.
- A Twitter bot is software that controls a Twitter account through automated programs and the Twitter API.
- A Twitter user is represented by semantic, property, and neighborhood information, covering posts, user features, and follower-following graph structure.
- Given a user and its three information modalities, the detector learns a function whose predicted label approximates the ground-truth label.
4 DATA COLLECTION
TwiBot-20 samples a broad Twittersphere through controlled breadth-first search, retrieves three information modalities, and uses layered annotation procedures. The resulting release includes large-scale labeled and support data.
- TwiBot-20 was collected from July to September 2020 by selecting users, retrieving multimodal information, and deriving annotations.
- Breadth-first search expands follow-graph clusters from diverse seed users through layer 3 without restricting users to a topic or follow pattern.
- Seed users span politics, business, entertainment, and sports, supplemented by commenters and hashtag-active users to broaden community coverage.
- The Twitter API provides up to 200 recent tweets, 38 property items, and follower-following relationships for each user.
- TwiBot-20 includes all directly retrievable user information, addressing benchmarks’ frequent omission of neighborhood data and partial property or tweet coverage.
- Annotation combines literature-based criteria, five annotators per user, majority agreement, direct messages, manual review, and consensus-based discard of disputed cases.
- The release partitions labeled users 7:2:1 into train, validation, and test sets and provides unsupervised users as a support set.
- TwiBot-20 is publicly available, with a small GitHub sample and encouragement to use the full dataset.
5 DATA ANALYSIS
TwiBot-20 is analyzed as a large, information-complete benchmark designed to represent diverse users and support graph-based bot detection. Its annotations generally align with established bot characteristics, although some behavioral patterns differ from earlier findings.
- 5.2 User Information Analysis: TwiBot-20 contains semantic, property and neighborhood information, making it the first publicly available benchmark to include user follow relationships.Its dense follow graph supports community-based methods such as graph neural networks.
- 5.3 User Diversity Analysis: TwiBot-20 represents geographically and topically diverse users, including communities associated with politics, sports, business, marketing, love and travel.Users span countries including India, the United States, Europe and Africa, while Figure 3 summarizes the 15 most frequent merged hashtags.
- 5.4 Annotation Quality Analysis: Bot users in TwiBot-20 can have fewer tweets because they tweet intensively during certain periods and then hibernate for long intervals.This pattern is offered as a possible explanation for discrepancies from earlier reputation-related findings.
- 5.4 Annotation Quality Analysis: The annotation analysis finds that TwiBot-20 generally matches previously proposed bot characteristics.The analysis examines account reputation, tweet activity and screen-name likelihood against prior literature.
6 EXPERIMENTS
Experiments compare established bot detectors across TwiBot-20 and two public datasets, then study how dataset size, user-information modalities, and user diversity affect evaluation. Existing methods perform worse on TwiBot-20, while its scale and multimodal structure support broader and more challenging benchmarking.
- 6.1 Experiment Settings: The experiments use Accuracy, F1-score, and MCC and compare competitive and state-of-the-art methods across TwiBot-20, cresci-17, and PAN-193.The datasets share a random 7:2:1 training, validation, and test partition across the reported experiments.
- 6.2 Benchmark performance: All bot detection baselines achieve significantly lower performance on TwiBot-20 than on cresci-17 or PAN-193.The authors interpret this result as evidence that TwiBot-20 is more challenging.
- 6.4 User Information Study: Performance gaps between methods using semantic information alone and methods also using user properties are larger on TwiBot-20 than on cresci-17.The comparison involves Wei et al. versus Kudugunta et al., whose models use different information modalities.
- 6.3 Dataset Size Study: Competitive baselines such as Yang et al. maintain performance when trained on a small portion of TwiBot-20.The authors argue that this indicates TwiBot-20 can sufficiently train detectors, unlike much smaller datasets that may not stably benchmark them.
- 6.4 User Information Study: TwiBot-20 provides semantic, property, and neighborhood information, enabling community-based detectors that the other datasets cannot support.Its follow relationships allow methods such as graph convolutional networks to use user neighborhoods.
- 6.5 User Diversity Study: Alhosseini et al. performs worse when trained on one user-interest domain than when trained on the full TwiBot-20.The result supports evaluating detectors on diversified bots and genuine users rather than on a specific bot scenario.
7 CONCLUSION AND FUTURE WORK
TwiBot-20 is presented as a public benchmark intended to represent the diversified Twittersphere and support further bot-detection research. Experiments find that state-of-the-art detectors do not match their previously reported performance on TwiBot-20, underscoring the task’s continuing difficulty.
- Conclusion: TwiBot-20 is a comprehensive benchmark designed to represent diversified bots co-existing on major social media platforms.The authors collected and annotated Twitter data and made the benchmark public.
- Conclusion: Extensive experiments show that state-of-the-art bot detectors fail to match their previously reported performance on TwiBot-20.
- Future Work: Twitter bot detection remains challenging and demands continual research efforts.The authors plan to study novel Twitter bots and propose robust bot detectors.