Source-linked AI summary
Detection of Novel Social Bots by Ensembles of Specialized Classifiers
Mohsen Sayyadiharikandeh, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, Filippo Menczer
TL;DR
Supervised bot detectors struggle with novel behaviors because bot classes use heterogeneous behavioral features, while retraining demands costly annotations. The paper proposes an ensemble of specialized classifiers, reports better cross-domain generalization with a 56% average F1 improvement, and deploys ESC in Botometer.
Problem
Novel bots with unseen, heterogeneous behaviors evade supervised detectors, while adapting models typically requires substantial new labeled data.
Method
ESC trains specialized models for distinct bot classes, aggregates human and bot models, and combines their outputs through a voting scheme.
Results
56% average F1 improvement was achieved across six datasets, with recall increasing from 42% to 84% and precision from 52% to 64%.
Takeaways & Limitations
ESC generalizes better across domains, learns about new domains with fewer annotated examples, and was deployed in Botometer v4.
Takeaways & Limitations
Kaiser-3 is not independent, its small bot class makes F1 sensitive to imbalance, and verified accounts bias comparisons toward tweetbotornot2.
Abstract
from arXiv · showhide
Malicious actors create inauthentic social media accounts controlled in part by algorithms, known as social bots, to disseminate misinformation and agitate online discussion. While researchers have developed sophisticated methods to detect abuse, novel bots with diverse behaviors evade detection. We show that different types of bots are characterized by different behavioral features. As a result, supervised learning techniques suffer severe performance deterioration when attempting to detect behaviors not observed in the training data. Moreover, tuning these models to recognize novel bots requires retraining with a significant amount of new annotations, which are expensive to obtain. To address these issues, we propose a new supervised learning method that trains classifiers specialized for each class of bots and combines their decisions through the maximum rule. The ensemble of specialized classifiers (ESC) can better generalize, leading to an average improvement of 56\% in F1 score for unseen accounts across datasets. Furthermore, novel bot behaviors are learned with fewer labeled examples during retraining. We deployed ESC in the newest version of Botometer, a popular tool to detect social bots in the wild, with a cross-validation AUC of 0.99.
1 INTRODUCTION
Social bots can scale misinformation, popularity amplification, and polarization across domains, making social media vulnerable to manipulation. Existing supervised detectors miss novel bot characteristics, motivating ESC and its deployment in Botometer.
- Motivation: Algorithmically controlled social media accounts can scale misinformation, popularity amplification, and polarization.Bot activity has been reported in politics, health, and business, alongside broader risks to news consumption and elections.
- Motivation: Novel social bots emerge and evade platform detection, so tools identifying them remain important for information-ecosystem authenticity.Legislative disclosure measures do not guarantee effective protection against malicious bots.
- Research gap: Supervised detectors can miss bots with unseen characteristics, while addressing weak generalization typically requires retraining on newly labeled data.The paper focuses particularly on Botometer, a widely adopted Twitter-account evaluation tool.
- Approach: ESC trains specialized classifiers for different bot types and combines their assessments to improve cross-domain detection without sacrificing in-domain accuracy.The method evaluates on datasets excluded from training and is designed to learn efficiently from new-domain examples.
- Deployment: ESC was deployed in Botometer v4 as a publicly available tool for detecting social bots in the wild.
2 THE CHALLENGE OF GENERALIZATION
The section examines why supervised bot detectors generalize poorly across datasets: bot classes rely on different behavioral features, while human accounts are more consistent. Cross-domain testing exposes lower bot recall and motivates analyzing dataset and bot-class heterogeneity.
- 2.1 Datasets: The study draws on annotated datasets from the Bot Repository and additional German-language and politician-account collections.Dataset annotations combine human labeling with automated procedures based on behavior, metadata, or high-precision rules.
- 2.1 Datasets: The models use over 1,200 features spanning account metadata, networks, temporal behavior, content, and sentiment.
- 2.2 Cross-domain performance comparison: Cross-domain evaluation trains Random Forests on one dataset and tests them on another, revealing weaker generalization than in-domain evaluation.The experiment uses 100-tree Random Forests, Platt-scaled scores, a 0.5 threshold, and 5-fold in-domain cross-validation.
- 2.2 Cross-domain performance comparison: Cross-domain bot recall is more impaired than precision because bot accounts receive lower scores, whereas human score distributions remain comparatively consistent.In-domain scores separate humans and bots more clearly, with human distributions left-skewed and bot distributions right-skewed.
- 2.3 Predictability of different bot classes: Different bot classes depend on different discriminative features: product-oriented spambots use adjectives, political spambots show sentiment, and fake followers exhibit aggressive following patterns.These distinctions support the premise that bot behaviors are heterogeneous across classes and datasets.
3 METHODS
The method builds on Botometer-v3 by adding specialized Random Forest classifiers for distinct bot classes and combining their scores through a maximum-rule voting module. ESC also outputs an explanatory bot-class label and produces scores comparable to the prior Botometer version.
- Baseline bot detection models: Botometer-v3 merges bots from all datasets into one positive class and uses a Random Forest bot score based on the fraction of positive tree votes.The baseline uses 1,209 features; Botometer-v3.1 uses 1,160 after removing unavailable time-zone and language metadata.
- Proposed method: Figure 4 contrasts RF0, representing the previous Botometer version, with the voting module, representing the new ESC version.The figure therefore provides a direct architectural comparison between the baseline score and the ensemble score.
- Proposed method: ESC trains specialized Random Forest classifiers for distinct bot classes alongside a human-detection classifier corresponding to the Botometer baseline.Each specialized classifier is trained on a balanced bot-class set and an equally sized sample of human accounts pooled across datasets.
- Proposed method: The voting module applies the maximum rule, selecting the classifier with the highest transformed score as the winning class.The human classifier contributes through a low bot score, which signals that an account is likely human.
- Proposed method: ESC calibrates the winning score with Platt’s scaling and returns the winning bot class as an explanatory outcome.The maximum rule shifts likely-bot scores toward one and likely-human scores toward zero.
4 RESULTS
ESC improves cross-domain bot detection by combining classifiers specialized for coherent bot behaviors, while preserving strong in-domain performance and enabling efficient adaptation to new domains.
- 4.1 In-domain performance: ESC achieves an AUC of 0.96 in-domain, close to Botometer’s 0.97 AUC baseline.
- 4.1 In-domain performance: Manual inspection found that most of 50 sampled accounts flagged by ESC were no longer accurately labeled as human.Many were inactive, devoted to spam diffusion, or controlled by third-party applications.
- 4.2 Cross-domain performance: ESC produces higher bot scores for cross-domain bots through the maximum rule, leading to higher recall and better generalization.
- 4.2 Cross-domain performance: 100% recall improvement across six datasets raises recall from 42% to 84%, while F1 increases 56% from 47% to 73%.Precision increases from 52% to 64%.
- 4.2 Cross-domain performance: On combined-test, recall rises from 77% to 86%, precision remains 70%, and F1 rises 5% from 73% to 77%.AUC comparisons are similar.
- 4.2 Cross-domain performance: The kaiser-1 and kaiser-3 evaluations are not completely hold-out because botwiki bots overlap with ESC training data.Even excluding botwiki, ESC obtains F1 scores of 0.84 and 0.80 on kaiser-1 and kaiser-3, respectively.
- 4.2 Cross-domain performance: ESC performs better or comparably to tweetbotornot2 in F1 on all datasets except those from Rauchfleisch and Kaiser.On combined-test, ESC outperforms tweetbotornot2 in both F1 and AUC.
- 4.3 Model adaptation: ESC adapts by adding a classifier trained on new data, avoiding retraining of existing classifiers.
5 RELATED WORK
Related work spans crowdsourced, graph-based, supervised, unsupervised, and adversarial bot-detection methods, while prior studies document evolving bot classes and Botometer’s errors.
- 5 RELATED WORK: Crowdsourcing collects annotations conveniently but is limited by scalability and user privacy, motivating automatic methods for large platforms.
- 5 RELATED WORK: Supervised methods use profile, network, and content features but require annotated data and adaptation to continuously evolving bot classes.
- 5 RELATED WORK: Unsupervised methods can be less vulnerable to cross-domain performance decay and are especially suited to detecting coordination among bots.
- 5 RELATED WORK: Bot detection can identify coordination through tweet content, temporal timeline features, or retweeting behavior.
- 5 RELATED WORK: Prior work reports that future systems may need to anticipate evolving spambots, while current systems and annotators can fail on some accounts.
- 5 RELATED WORK: Existing studies distinguish multiple bot classes, but assigning one class to an account is difficult and some social spambots resemble humans.
- 5 RELATED WORK: Rauchfleisch and Kaiser criticized Botometer-v3 for high false positives and false negatives, while the paper notes biases in their assumptions and unavailable-account handling.
6 BOTOMETER-V4 DEPLOYMENT
ESC was deployed as Botometer v4, with a modular ecosystem, six specialized classifier outputs, and an API designed for efficient, scalable responses.
- 6 BOTOMETER-V4 DEPLOYMENT: Botometer v4 deploys ESC through the Observatory on Social Media and provides website and public API access.
- 6 BOTOMETER-V4 DEPLOYMENT: The deployed system uses Python, MKL, and scikit-learn Random Forest implementations.
- 6 BOTOMETER-V4 DEPLOYMENT: Figure 10 tracks daily Botometer API requests and annotates the model versions deployed over time.
- 6 BOTOMETER-V4 DEPLOYMENT: Load tests with 20 simultaneous jobs produced an average response time below 100ms per query.
- 6 BOTOMETER-V4 DEPLOYMENT: The deployed model trains specialized classifiers on reorganized bot classes, including financial bots and expanded other-bot categories.
- 6 BOTOMETER-V4 DEPLOYMENT: Botometer reports scores from six specialized classifiers: fake followers, spammers, self-declared, astroturf, financial, and others.
- 6 BOTOMETER-V4 DEPLOYMENT: The interface exposes component scores and interpretable class labels for greater transparency.
7 CONCLUSION
The paper attributes cross-domain deterioration to heterogeneous bot behaviors and presents ESC as a modular, adaptable alternative, while warning that evaluation is dataset-sensitive.
- 7 CONCLUSION: Dynamic platforms, changing account behavior, and invalidated ground-truth data make online bot detection difficult.
- 7 CONCLUSION: ESC generalizes better than a monolithic classifier and is more robust to mislabeled training examples.
- 7 CONCLUSION: Cross-domain performance is highly sensitive to the datasets used for training and testing, making cherry-picked evaluations potentially misleading.
- 7 CONCLUSION: ESC’s modular classifiers can be substituted independently and expanded when new annotated datasets become available.
- 7 CONCLUSION: The architecture learns new domains efficiently by requiring fewer annotated examples.
- 7 CONCLUSION: Future work includes detecting when a new bot type warrants another classifier, automatic clustering, and active-learning query strategies.