Source-linked AI summary
Breaking and Defending LLM-Powered Social Media Bot Detection Systems
Nof Orenstein, Yoni Birman
TL;DR
LLM-based bot detectors improve semantic analysis but create new adversarial attack surfaces. This paper evaluates attacks and defenses in a black-box bot-detection pipeline, introducing LSABRE, which maintains 86% detection accuracy under adversarial manipulation.
Problem
The study addresses the limited evaluation of both offensive and defensive uses of LLMs in bot detection despite their vulnerability to adversarial attacks.
Method
The authors evaluate content-rewrite and prompt-based attacks and defenses in a black-box LLM bot-detection pipeline, then combine diverse model behaviors in LSABRE.
Results
86% detection accuracy: LSABRE improves robustness against adversarial manipulations.
Takeaways & Limitations
Ensemble approaches show potential for securing social media bot detection systems against adversarial manipulation.
Takeaways & Limitations
The study focuses on zero-shot rewriting generated by three models and leaves more sophisticated content-manipulation strategies for future work.
Abstract
from arXiv · showhide
The rise of social media bots poses a persistent threat, enabling misinformation, opinion manipulation, and the erosion of trust in online platforms. To combat this, machine learning systems have been developed to detect and limit bot activity, but attackers continuously adapt through techniques such as adversarial learning and behavior imitation, fueling an ongoing arms race between bots and detection tools. Recent advances in large language models (LLMs) have significantly improved bot detection by enabling deeper semantic and contextual analysis of accounts and their content. However, this shift also introduces new attack surfaces, allowing adversaries to craft exploits that directly target the reasoning and generation mechanisms of LLM-based classifiers. Industry tools such as Anthropic's Claude Code Security similarly leverage LLMs for security-critical decisions, further motivating a careful study of their attack surfaces. In this work, we investigate both the offensive and defensive aspects of LLM-powered, threat-specific cybersecurity applications. While centered on the challenge of social media bot detection, our methodology and insights generalize to a broad class of LLM-powered cybersecurity systems, including phishing detection, email classification, and fraud analysis. We introduce two novel adversarial attack strategies that systematically exploit the semantic and contextual weaknesses of LLM-based classifiers, degrading their detection accuracy by up to 48%. To counter these threats, we propose a robust multi-LLM defense architecture designed to preserve detection reliability under adaptive adversarial conditions. Our solution, LSABRE (LLM-powered Social Adversarial Bot Recognition Ensemble), is a multi-LLM framework that substantially improves robustness across a range of attacks, maintaining 86% detection accuracy even under strong, adaptive adversarial pressure.
1. Introduction
LLM-based bot detection addresses a growing threat from automated accounts but introduces adversarial attack surfaces. This work studies attacks and defenses together, proposing new attack methods, defense strategies, LSABRE, and a benchmark dataset.
- Bots amplify misinformation, manipulate public opinion, and support malicious cyber activities, threatening discourse integrity, user safety, and platform credibility.
- LLM adoption improves scalable bot detection and explanation quality but exposes security systems to adversarial attacks targeting model behavior.
- The study evaluates offensive and defensive aspects of LLM-powered cybersecurity applications, using bot detection as a practical case with broader applicability across cyber-threat domains.
- Contributions: Feature-engineered Guidance Rewrite strengthens rewrite-based evasion by injecting task-relevant salient features into LLM prompts.
- Contributions: The authors systematically evaluate generic defenses and introduce self-examination, in-context learning, and feature-guided reasoning against semantic and prompt-based attacks.
- Contributions: LSABRE combines diverse LLM detection and prevention actors, achieving ~86% detection accuracy under attack while maintaining a low falsepositive rate (~13%).
- Contributions: The benchmark rewrite-attack dataset is constructed over TwiBot20 with attack variants generated using Llama, Mistral, and Gemma.
2. Background and Related Work
Prior bot detectors use metadata, text, and graph signals, while LLMs offer generalization and interpretable outputs with reduced training needs. This work addresses the limited study of jointly attacking and defending LLMs used for bot classification under black-box, inference-time constraints.
- Bot Detection: Earlier bot detection methods analyzed profile metadata, tweet content, or user connections using supervised, NLP, and graph-based models.
- LLM-Based Detection: LLM-based detectors reduce training requirements and provide generalization and interpretable outputs, but remain vulnerable to adversarial attacks.
- Datasets: The literature includes bot-human datasets such as TwiBot-20, TwiBot-22, and Cresci 2017 containing profile, tweet, and network data.
- Research Gap: Prior studies generally separate LLM-based detection improvements from adversarial robustness or address generic NLP vulnerabilities rather than bot classification.
- LLM Attacks: Inference-time attacks manipulate inputs, whereas training-time attacks poison model data and system attacks exploit external plugins, tools, or libraries.
- LLM Attacks: The study examines red-team, template-based, and neural prompt-to-prompt attack categories, including jailbreaks and automatically generated evasive prompts.
- Study Scope: The evaluation assumes black-box access and focuses on classification rather than safety-alignment tasks in conversational LLMs.
- Defenses: The authors adapt prior inference-time defense evaluations to bot detection, covering both prompt injection and rewrite attacks.
3. Offensive and Defensive Methods
The paper formulates black-box LLM bot detection, defines content and prompt-based attacks, and evaluates input defenses including the LSABRE ensemble. Its methods target both rewritten profile content and injected instructions while preserving the detection pipeline.
- Detection Pipeline: The system combines Twitter profile metadata and all user tweets in a structured prompt, then outputs a binary bot-or-human classification with an explanation.The pipeline uses modular data ingestion, prompt construction, and classification under black-box access.
- Attack Formulation: The attack formulation defines the target task as bot detection and the detector as a function mapping prompts to binary labels, with 1 denoting bot and 0 denoting human.The full prompt concatenates instruction components, profile data, and a suffix around the user content.
- Attack Taxonomy: Inference-time attacks are categorized as Content Manipulation, LLM Manipulation, or Mixed Manipulation, with detailed analysis focused on rewriting and prompt injection.Content Manipulation alters Twitter content, whereas LLM Manipulation modifies prompts to bypass safety policies or disrupt execution.
- Rewrite Attacks: Rewrite attacks replace original profile content with more legitimate-sounding text using zero-shot, few-shot, or classifier-guided strategies.Classifier Guidance Rewrite uses a pretrained model or LLM for bot classification to improve the rewritten content.
- Prompt Injection: Prompt injection adds adversarial instructions or data to profile content to divert the detector from bot classification, including reasoning, safety-alignment, and out-of-service attacks.The formalization represents injected content as XI and the compromised input as XT ⊕XI.
- Defense Methods: The defense study uses prevention methods to neutralize adversarial inputs and detection methods to identify them, implementing five novel defenses and integrating six additional techniques.The authors exclude paraphrasing, re-tokenization, and perplexity defenses because of clean-data degradation, unreliable removal, or black-box incompatibility.
- Ensemble Defense: LSABRE combines multiple LLMs and complementary defense behaviors to reduce single-model weaknesses against both content manipulation and LLM manipulation attacks.The ensemble aggregates decisions from diverse detection and prevention actors while maintaining performance across varying attack types.
4. Experiments
Experiments show substantial variation in how LLM detectors withstand content and LLM manipulation attacks, while defense effectiveness depends strongly on the target model and attack type. Combining complementary defenses improves robustness, but no single technique consistently restores clean performance.
- Content Manipulation: ∼35% degradation affected Gemma under content manipulation, compared with only ∼10% for Llama.Feature Guidance was consistently effective across models, while Mistral’s apparent 5% improvement reflected its tendency to classify most inputs as human.
- LLM Manipulation: ∼46-48% degradation affected Llama and Gemma under LLM manipulation, while Mistral showed a ∼-6% reduction.A minus sign indicates that Mistral’s prediction score improved; reasoning-based prompt injections were the most effective overall.
- Attacker Effectiveness: Mistral was the strongest attacker, producing a ∼16% average detection reduction, followed by Llama at ∼13% and Gemma at ∼6%.Mistral consistently caused the largest drop in detection accuracy across target models.
- Content Manipulation Defenses: ∼21.75% average improvement was achieved for Mistral using Self Examination with ICL, while Llama improved ∼4.567% using Self Examination with Feature Guidance.Gemma showed minimal change, with approximately ∼0.00037% reduction under Self Examination Zero-Shot.
- Defense Limitations: No single defense fully mitigated attacks while preserving original detection performance, motivating combinations of complementary defenses.Combining a high-TPR method with one that lowers FPR was suggested for more balanced protection, and combined defenses consistently improved robustness in evaluation.
- LLM Manipulation Defenses: ∼34.5% and ∼9.3% detection improvements were achieved for Llama and Gemma, respectively, using Feature Guidance against LLM manipulation.Known Answer produced the highest Llama gain at ∼41.3%, but safety injection attacks remained resistant to all prevention techniques.
5. Proposed Ensemble Architecture LSABRE: LLM based Social Adversarial Bot Recognition Ensemble
LSABRE is a black-box defense ensemble that detects potentially adversarial prompts, applies prevention measures to flagged inputs, and then performs final classification. Its architecture combines multiple specialized LLM actors across detection, prevention, and classification layers.
- LSABRE defends black-box Twitter bot detectors without requiring access to or modification of their implementations.
- The ensemble uses a two-stage pipeline: it detects suspicious inputs, then applies prevention-augmented classification to flagged prompts.Potentially adversarial prompt P′ is processed first by detection and then by prevention-augmented classification.
- Actor selection combines Feature Guidance Llama and ICL Llama for balanced content-manipulation prevention, while Known Answer and Feature Guidance actors address different LLM-manipulation attacks.
- The architecture has detection, prevention, and classification layers that respectively vote on adversarial inputs, add defensive instructions, and send hardened prompts to the detector.
- Detection-before-attack and detection-after-attack results exclude Mistral because it is not well-suited for this task.
6. Discussion
The study evaluates LLM-based Twitter bot detectors against content and LLM manipulation attacks, introducing attacks, defenses, an ensemble, and a benchmark dataset. Results show that defenses vary by attack type, while LSABRE improves robustness and reaches 86% detection accuracy.
- The evaluation covers content manipulation and LLM manipulation attacks against LLM-based Twitter bot detectors.
- The paper introduces Feature-engineered Guidance Rewrite, self-examination defenses, LSABRE, and a benchmark Rewrite Attack dataset.
- No single defense method consistently mitigates every attack type while preserving detection performance.
- Llama shows the highest resilience among the evaluated models, and self-examination defenses show particular promise against Rewrite Attacks.
- 86% detection accuracy is achieved by LSABRE alongside improved robustness against adversarial manipulations.
7. Appendix
The appendix documents bot-detection prompts, attack and defense experiment categories, evaluation metrics, dataset scale, and token-cost and latency estimation procedures. It also describes tables reporting results for content manipulation, LLM manipulation, and prevention or detection defenses.
- Bot-detection prompts combine account metadata with tweets and require a one-word Human-or-Bot output.
- LLM manipulation uses prompt injection to divert the detector from its original bot-detection task.
- The defense detection method classifies tweets as Rewritten or Original using cues such as unnatural language, repetition, and subtle manipulation.
- The appendix reports accuracy, TPR, FPR, and average prediction score as evaluation measures.Average prediction score aggregates model outputs across models or defense and attack techniques.
- The dataset contains more than 3M tweets from 2,000 users, with almost 200K tweets selected for rewriting across Llama, Mistral, and Gemma.
- The appendix defines baseline detection, detection under rewriting and prompt-injection attacks, and detection under defenses.The defense setting covers both rewriting and injection attacks, plus a baseline without attacks.
- Reported tables cover attack predictions, prevention-based defenses, and detection-based defenses for content and LLM manipulation.