Source-linked AI summary
BotOrNot: A System to Evaluate Social Bots
Clayton A. Davis, Onur Varol, Emilio Ferrara, Alessandro Flammini, Filippo Menczer
TL;DR
Social bots can automate social-media activity and support coordinated manipulation, creating a need to assess whether accounts are human- or machine-controlled. BotOrNot addresses this need with a public Twitter-account evaluation service using more than 1,000 features and supervised classifiers. The service had served over one million queries, while its classifier achieved 0.95 AUC in ten-fold cross-validation, a result the authors caution may overestimate current performance.
Problem
Social bots automatically produce content and interact with people, and coordinated campaigns have been used for manipulation and artificial grassroots support.
Method
BotOrNot retrieves recent Twitter activity and computes a bot-likelihood score using more than 1,000 network, user, social-contact, timing, content, and sentiment features.
Results
0.95 AUC was achieved in ten-fold cross-validation, while the service served over one million queries after its public API became available.
Takeaways & Limitations
The free website and API lower the entry barrier for researchers, reporters, and enthusiasts who want to evaluate social-media accounts.
Abstract
from arXiv · showhide
While most online social media accounts are controlled by humans, these platforms also host automated agents called social bots or sybil accounts. Recent literature reported on cases of social bots imitating humans to manipulate discussions, alter the popularity of users, pollute content and spread misinformation, and even perform terrorist propaganda and recruitment actions. Here we present BotOrNot, a publicly-available service that leverages more than one thousand features to evaluate the extent to which a Twitter account exhibits similarity to the known characteristics of social bots. Since its release in May 2014, BotOrNot has served over one million requests via our website and APIs.
1. INTRODUCTION
Social bots are automated social-media agents whose coordinated use can support manipulation and other harmful campaigns. BotOrNot is introduced as a publicly available service for evaluating whether Twitter accounts are controlled by humans or machines.
- Social bots, also called sybil accounts, automatically produce content and interact with humans on social media.
- Coordinated bot accounts have been used for astroturfing and to create artificial market buzz.One reported campaign produced chatter that automated trading algorithms acted on, causing a spurious 200-fold increase in market price.
- BotOrNot evaluates whether a Twitter account is controlled by a human or a machine.The service retrieves recent activity and returns a bot-likelihood score with feature plots for website users.
2. RELEASE TIMELINE
BotOrNot evolved from a website-only service into a publicly accessible API. Usage exceeded one million queries as access expanded and system stability improved.
- The service initially handled users through its website, while capacity concerns delayed release of a public API.Press coverage drove about 18k uses during the first eight months.
- Rate limits were implemented after power users generated more than 8k requests per day through an internal API endpoint.The authors report that system stability increased after these changes.
- Over one million BotOrNot queries had been served after the public API became available.More than 540k requests were served in the month after the API announcement, bringing the total above one million.
3. SYSTEM DESIGN
BotOrNot retrieves recent Twitter activity, computes a bot-likelihood score, and returns reports or machine-readable results. Its classifier combines more than 1,000 features with supervised Random Forest models trained on labeled bot and human accounts.
- 3.1 BotOrNot Service: The service retrieves recent tweets and mentions through Twitter’s REST API before forwarding the data to the BotOrNot server.Website requests return reports with plots, while API requests return classification results in JSON.
- 3.1 BotOrNot Service: Over 900k unique user account classifications had been stored, and their bot-score distribution was shown in Figure 2.
- 3.2 Classification System: More than 1,000 features cover network, user, friend, timing, content, and sentiment information extracted from metadata and activity.
- 3.2 Classification System: The training dataset contained 15k manually verified social bots, 16k legitimate accounts, and more than 5.6 million tweets.The data were collected from recent tweets and mentions using the Twitter Search API.
- 3.2 Classification System: 0.95 AUC was obtained with ten-fold cross-validation using Random Forest classifiers.The paper cautions that this performance likely overestimates current performance because the training data are old.
4. CONCLUSION
The authors position BotOrNot as a free service that lowers the barrier to social-bot evaluation and supports applications built on its public classification service.
- The free website and API let researchers, reporters, and enthusiasts evaluate accounts without setting up their own classifiers.API use still requires some scripting experience.
- A proposed browser plugin could fetch a BotOrNot report for a selected Twitter username.