PaperlayerView exact revision on arXiv

Source-linked AI summary

TwiBot-22: Towards Graph-Based Twitter Bot Detection

Shangbin Feng, Zhaoxuan Tan, Herun Wan, Ningnan Wang, Zilong Chen, Binchi Zhang, Qinghua Zheng, Wenqian Zhang, Zhenyu Lei, Shujie Yang, Xinshun Feng, Qingyue Zhang, Hongrui Wang, Yuhan Liu, Yuyang Bai, Heng Wang, Zijian Cai, Yanbo Wang, Lijing Zheng, Zihan Ma, Jundong Li, Minnan Luo

arXiv:2206.04564v6cs.SIcs.AI

TL;DR

Twitter bot detection needs graph-based benchmarks because existing graph datasets are scarce and limited in scale, structure, and annotation quality. TwiBot-22 addresses this gap with a large heterogeneous benchmark, broad baseline evaluation, and an integrated evaluation framework. It establishes a larger, more diverse, and better-annotated resource while identifying further directions involving coordinated bot groups and multimodal user information.

  • Problem

    Existing graph-based Twitter bot-detection datasets are few and suffer from limited scale, incomplete graph structure, and low annotation quality.

  • Method

    TwiBot-22 combines two-stage network expansion, weak-supervision annotation, evaluation of 35 baselines on 9 datasets, and a consolidated evaluation framework.

  • Results

    TwiBot-22 is the largest benchmark to date, provides diversified entities and relations, and has considerably improved annotation quality over existing datasets.

  • Takeaways & Limitations

    The benchmark and framework support consistent comparison of graph-based bot-detection models and datasets.

  • Takeaways & Limitations

    The paper leaves temporal and subgraph-level detection of coordinated bot campaigns, along with multimodal user-feature modeling, for future work.

Abstract

from arXiv · show

Twitter bot detection has become an increasingly important task to combat misinformation, facilitate social media moderation, and preserve the integrity of the online discourse. State-of-the-art bot detection methods generally leverage the graph structure of the Twitter network, and they exhibit promising performance when confronting novel Twitter bots that traditional methods fail to detect. However, very few of the existing Twitter bot detection datasets are graph-based, and even these few graph-based datasets suffer from limited dataset scale, incomplete graph structure, as well as low annotation quality. In fact, the lack of a large-scale graph-based Twitter bot detection benchmark that addresses these issues has seriously hindered the development and evaluation of novel graph-based bot detection approaches. In this paper, we propose TwiBot-22, a comprehensive graph-based Twitter bot detection benchmark that presents the largest dataset to date, provides diversified entities and relations on the Twitter network, and has considerably better annotation quality than existing datasets. In addition, we re-implement 35 representative Twitter bot detection baselines and evaluate them on 9 datasets, including TwiBot-22, to promote a fair comparison of model performance and a holistic understanding of research progress. To facilitate further research, we consolidate all implemented codes and datasets into the TwiBot-22 evaluation framework, where researchers could consistently evaluate new models and datasets. The TwiBot-22 Twitter bot detection benchmark and evaluation framework are publicly available at https://twibot22.github.io/

1 Introduction

Twitter bot detection matters because automated users contribute to disinformation and other societal problems, yet graph-based research lacks datasets with sufficient scale, structure, and annotation quality. TwiBot-22 addresses these gaps with a large heterogeneous benchmark, broad baseline evaluation, and a reproducible framework.

  • Automated Twitter bots contribute to disinformation, election interference, extremism campaigns, and conspiracy-theory propagation, motivating automatic detection.
  • Only two of 18 listed datasets explicitly provide Twitter-user graph structure, limiting support for graph-based bot detection.
  • Existing graph-based datasets suffer from limited scale, incomplete heterogeneous relations, and noisy or false-positive-prone annotations.TwiBot-20 and cresci-15 contain 11,826 and 7,251 labeled users, respectively, while real-world discussions can involve hundreds of thousands of users.
  • TwiBot-22 uses controlled network expansion and weak supervision to create a benchmark five times larger than the largest existing dataset with diversified entities, relations, and improved annotations.It provides 4 entity types and 14 relation types, forming a heterogeneous Twitter graph.
  • The authors re-implement 35 bot-detection baselines across 9 datasets and consolidate datasets and code into the TwiBot-22 evaluation framework.

2 Related Work

Twitter bot-detection research spans feature, text, and graph-based methods, supported by datasets that often target political contexts. However, only a small fraction of existing datasets expose graph structure, and those datasets remain limited for benchmarking graph-based approaches.

  • Twitter bot-detection methods mainly use engineered user features, text representations, or graph structure.
  • Feature-based methods extract signals from metadata, tweets, descriptions, temporal patterns, and follow relationships for traditional classification.
  • Text-based methods encode tweets and descriptions with embeddings, recurrent networks, attention, and pretrained language models.
  • Graph-based methods apply network science, representation learning, GNNs, and heterogeneous GNNs to Twitter bot detection.
  • Existing datasets often focus on United States and European politics, while only 2 of 18 listed datasets explicitly provide Twitter graph structure.
  • Social-network structure supports analysis with graph neural networks, including heterogeneous-graph models for social-media detection tasks.

3 TwiBot-22 Dataset

TwiBot-22 constructs a large, heterogeneous Twitter graph through diversity-aware user sampling and collection of additional entities and relations, then produces labels using expert-guided weak supervision. Its resulting benchmark is designed to improve scale, graph coverage, and annotation quality over existing graph-based datasets.

  • User network collection: Diversity-aware BFS samples users across metadata distributions and values, starting from @NeurIPSConf, to include varied bots and genuine users.Each expansion randomly adopts one metadata field and one sampling strategy.
  • Heterogeneous graph building: The collection process enriches the user-follow graph with tweets, lists, hashtags, and 12 additional relations, forming a heterogeneous network.The resulting graph contains 92,932,326 nodes and 170,185,937 edges.
  • Data annotation: Weak supervision combines 1,000 expert annotations with labeling functions and seven feature-based or neural models to generate TwiBot-22 labels.Eight handcrafted labeling functions and uncertainty filtering remove the top 40% most uncertain predictions for each classifier.
  • Data annotation: 90.5% annotation accuracy exceeds the 80% accuracy standard reported for TwiBot-20.Snorkel cleans probabilistic labels before an MLP produces the final annotations.
  • Benchmark analysis: TwiBot-22 is approximately five times larger than TwiBot-20 and includes 4 entity types and 14 relation types, versus 2 and 3 in the compared graph-based datasets.These comparisons address dataset scale and graph-structure limitations identified in existing benchmarks.

4 Experiments

Experiments benchmark 35 Twitter bot detection baselines across 9 datasets, examining graph contributions, scalability, and generalization. Graph-based methods generally perform better, but results also expose scalability and generalization challenges.

  • Benchmark setup: 35 baseline methods are evaluated on 9 Twitter bot detection datasets using five runs with reported average performance and standard deviation.Table 2 reports accuracy and distinguishes feature-, text-, and graph-based methods.
  • Benchmark results: Graph-based methods outperform feature- and text-based methods, with top-five averages exceeding all-baseline averages by 13.8% on TwiBot-20 and 8.2% on TwiBot-22.All top-five models on both datasets are graph-based.
  • Scalability: Dehghan et al. achieves near-state-of-the-art performance on TwiBot-20 but does not scale to TwiBot-22 because the implementation encounters an out-of-memory problem.The result illustrates a scalability issue on the largest benchmark.
  • Benchmark results: Performance on TwiBot-22 is 2.7% lower on average than on TwiBot-20 across all baseline methods, indicating that bot detection remains an open problem.The paper attributes this difference to evolving bot behavior and evasion.
  • Role of graph structure: Removing graph components causes performance drops across all baselines, with generally larger declines for BotRGCN and RGT.The comparison covers TwiBot-20 and TwiBot-22.
  • Generalization study: BotRGCN achieves the best generalization average, outperforming RGT by 3.66, while GAT’s higher accuracy does not translate into higher average generalization.Generalization is evaluated by training on one network fold and testing on another; GAT’s avg is 2.55 lower than LOBO’s despite higher accuracy.

5 Evaluation Framework

The TwiBot-22 evaluation framework consolidates datasets, preprocessing code, and 35 implemented baselines. It provides a unified, reproducible interface that can be extended with future datasets and methods.

  • Framework components: The framework consolidates Twitter bot detection datasets, preprocessing code, and all 35 implemented baselines.It is made publicly available to facilitate further research.
  • Framework components: A unified interface supports different types of Twitter bot detection datasets.
  • Framework components: The framework provides 35 representative baselines with well-documented implementations.
  • Extensibility: Researchers can extend the evaluation framework with new datasets and methods proposed in future work.

6 Conclusion and Future Work

The paper consolidates TwiBot-22, its evaluation framework, and broad baseline evaluations, then identifies future work on coordinated bots, multimodal features, generalization, and scalability.

  • TwiBot-22 combines a graph-based benchmark, 35 re-implemented models evaluated on 9 datasets, graph analysis, generalization analysis, and a reproducible evaluation framework.The framework is intended to support reproducing experiments and testing new datasets and models.
  • Future Work: Future work targets identifying bot clusters and coordinated campaigns through temporal and subgraph-level detection approaches.The paper characterizes this area as having scarce existing literature.
  • Future Work: Future work also targets multimodal bot detection using images and videos, which TwiBot-22 provides but the 35 baselines do not leverage.
  • Future Work: The paper proposes quantitative evaluation of bot-detection generalization because existing work mainly focuses on detection performance.
  • Future Work: The paper identifies scalability as an open issue because graph-based methods require more computation and execution time than feature-based models.This concern is especially relevant as the Twitter network continues expanding.

Checklist

The checklist records the paper’s claims about contributions, reproducibility, data governance, ethical considerations, and released heterogeneous-graph assets.

  • The paper states that its claims and scope accurately reflect the work and that it follows ethics review guidelines.
  • Reproducibility: The paper reports discussing its limitations and providing code, data, instructions, training details, error bars, and compute information for reproduction.The checklist points readers to the TwiBot-22 GitHub repository and Section B.6 for supporting details.
  • Assets: The paper cites existing assets, discusses their licenses, and provides new assets through URLs.
  • Data Governance: The dataset excludes private and protected users but may contain offensive content because some bots are designed to be offensive.
  • Dataset Assets: TwiBot-22’s released graph includes four entity types: user, tweet, list, and hashtag.The detailed entity information is presented in Table 4, alongside a complete relation inventory in Table 5.

A.2 Data Collection Details

TwiBot-22 is collected in two stages: diversity-aware user-network expansion followed by heterogeneous graph construction, with expert annotations guiding weak supervision.

  • User Network Collection: The first collection stage starts from @NeurIPSConf and expands through 1,000 followers and 1,000 followees using breadth-first search.Sampling strategies and metadata are used to select six neighborhood users per expansion round.
  • Heterogeneous Graph Building: The second stage builds the heterogeneous graph by collecting users’ tweets, interactions, mentions, lists, hashtags, and related tweets.The process includes pinned tweets, recent liked tweets, retweets, quotes, replies, and list members, followers, and tweets.
  • Sampling and Statistics: The diversity-aware sampling process uses user metadata listed in Table 6, while Table 7 reports TwiBot-22 statistics.
  • Collection Timeline: The two collection stages ran from January 20 to February 1, 2022, and from February 1 to March 15, 2022, respectively.
  • Expert Annotation: Seventeen researchers assigned each user to five experts, whose majority-voted labels for 1,000 users guided weak-supervision learning.Experts classified users as human, bot, or not sure.

A.4 Dataset Statistics

This section presents benchmark statistics and an expert study comparing annotation quality between TwiBot-20 and TwiBot-22.

  • Benchmark Statistics: Table 7 presents important statistics about the TwiBot-22 benchmark.
  • Annotation Quality: An expert study randomly selected 500 users from each of TwiBot-20 and TwiBot-22 and assigned each user to three experts.Experts used five ordered labels ranging from definitely bot to definitely human, including not sure.
  • Annotation Quality: The study reports Randolph’s Kappa Coefficient to model agreement between experts.

A.6 Other TwiBot-22 Details

TwiBot-22 provides documentation, hosting, licensing, and maintenance information for the dataset and evaluation framework. The paper also identifies missing stored media and possible misuse as limitations or risks.

  • Intended use: TwiBot-22 is intended for research in Twitter bot detection and social network analysis.
  • Documentation and access: The TwiBot-22 evaluation framework is documented through its official website and repository, which hosts preprocessing code and 35 bot-detection methods.The repository is described as supporting reproducible research.
  • Hosting and maintenance: The dataset is hosted on Google Drive, while the evaluation framework is hosted on GitHub and intended for continued maintenance and community expansion.The research group plans to add new datasets and baselines with community help.
  • Licensing: The dataset uses CC BY-NC-ND 4.0, while the implemented evaluation-framework code uses the MIT license.
  • Limitations: The dataset does not store user images or videos, although researchers can download them using the media links when necessary.
  • Potential negative societal impact: The dataset and framework might be misused to study evasive bots and design more difficult-to-detect Twitter bots.

B Experiment Details

The experiments cover diverse Twitter bot-detection baselines and classify them by whether they use features, text, graph structure, or combinations of these inputs. Results are reported across nine datasets, with tables documenting F1-score and community examples.

  • Reported metrics: Table 8 reports average F1-score and standard deviation for 35 methods across nine datasets, using notation for unsupported datasets and methods that cannot scale to TwiBot-22.
  • Graph-based baselines: RGT models influence and relation heterogeneity with graph transformers followed by semantic attention over relation-specific user representations.
  • Baseline categorization: Feature-based baselines use user metadata and engineered features, text-based baselines use tweets or descriptions, and graph-based baselines use Twitter network structure.
  • Baseline categorization: Baselines may receive multiple labels when they combine input types; BotRGCN is categorized as FTG because it uses metadata, text, and a relational graph.
  • Sub-community analysis: The study documents ten sub-communities, including five centered on closely connected users and five identified by clustering hashtag representations.Example hashtags are provided for the hashtag-based communities, and community statistics are reported in Table 10.

B.3 F1-score Results

The experiments compare baseline performance, isolate the contribution of graph information, test generalization across Twitter sub-communities, and examine annotation behavior using labeling-function and expert-label evaluations.

  • Benchmark evaluation: The study re-implements 35 baselines and evaluates them on nine datasets, reporting detection accuracy and F1-score.
  • Graph ablation: Graph contributions are examined by removing graph components or graph-derived features from graph-based methods and comparing the resulting models.
  • Generalization analysis: Generalization experiments identify ten TwiBot-22 sub-communities using connected neighborhoods and hashtag-based clustering.Five communities are centered on selected accounts, while five are formed from users associated with similar hashtags.
  • Experimental protocol: Each experiment is run five times, with average performance and standard deviation reported from a server equipped with eight GeForce RTX 2080 Ti GPUs.
  • Annotation analysis: The annotation study removes individual labeling functions and measures how many labels change relative to the full Snorkel-based annotation process.
  • Expert-label evaluation: Expert-label evaluation uses TwiBot-22 training and validation data with expert labels as the test set, including tests on 1,000 and 500 manually annotated users.
Loading 2206.04564v6…