PaperlayerView exact revision on arXiv

Source-linked AI summary

Anatomy of an AI-powered malicious social botnet

Kai-Cheng Yang, Filippo Menczer

arXiv:2307.16336v1cs.CYcs.AIcs.SI

TL;DR

Evidence of malicious LLM use in the field has been limited, despite concerns about realistic deceptive and harmful content. This case study identifies and analyzes a Twitter botnet that appears to use ChatGPT. Its coordinated activity is observable, but content classifiers and conventional bot-detection methods face challenges distinguishing these accounts in the wild.

  • Problem

    Evidence of malicious LLM deployment in the field remains limited, even as LLMs can generate realistic content and raise concerns about misuse.

    Source: §1 · p. 2

  • Method

    The authors analyze a Twitter botnet identified through self-revealing tweets and heuristics, validating 1,140 accounts through manual annotation.

    Source: §1 · p. 2

  • Results

    Current LLM-content detectors cannot effectively distinguish the bot accounts from human accounts in the wild, although their coordinated patterns can be detected.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The case study documents LLM-enabled malicious bots in social media and provides insights and data for further investigation.

    Source: §6 · p. 19

    Because the study focuses on one Twitter botnet, its findings might not represent other LLM-powered bots, and not all fox8 content can be guaranteed to be LLM-generated.

    Source: §6 · p. 19

Abstract

from arXiv · show

Large language models (LLMs) exhibit impressive capabilities in generating realistic text across diverse subjects. Concerns have been raised that they could be utilized to produce fake content with a deceptive intention, although evidence thus far remains anecdotal. This paper presents a case study about a Twitter botnet that appears to employ ChatGPT to generate human-like content. Through heuristics, we identify 1,140 accounts and validate them via manual annotation. These accounts form a dense cluster of fake personas that exhibit similar behaviors, including posting machine-generated content and stolen images, and engage with each other through replies and retweets. ChatGPT-generated content promotes suspicious websites and spreads harmful comments. While the accounts in the AI botnet can be detected through their coordination patterns, current state-of-the-art LLM content classifiers fail to discriminate between them and human accounts in the wild. These findings highlight the threats posed by AI-enabled social bots.

1 Introduction

Although LLM misuse in the field had been supported mainly by anecdotal evidence, this paper examines a Twitter botnet that appears to use ChatGPT for harmful content. Its 1,140 accounts form a coordinated network of fake personas, and current LLM-content detectors do not effectively distinguish them from humans.

1 Introduction

  • Field evidence of malicious LLM deployment remained largely anecdotal, despite concerns about deceptive content, harmful outputs, and challenges for automated detection.Prior work was described mainly as predictions and laboratory experiments.Source: §1 · p. 2
  • Traditional social bots typically lack the intelligence to create realistic personas, convincing content, or natural automated conversations, while LLMs can enhance bots across these dimensions.Social bots are accounts controlled in part by software and have been associated with distorted conversations and misinformation.Source: §1 · p. 2
  • 1,140 accounts identified as the “fox8” botnet form a dense network, post machine-generated content and stolen selfies, and interact through retweets and replies.Self-revealing tweets suggest ChatGPT-generated content promotes suspicious websites and spreads harmful comments; state-of-the-art LLM-content detectors cannot effectively distinguish the bots from human accounts in the wild.Source: §1 · p. 2
  • The case study offers real-world insight into adversarial use of LLMs and shares botnet data for further research.The authors anticipate more advanced bot accounts serving diverse purposes as AI technologies advance.Source: §1 · pp. 2–3

2 Related work

Related work describes LLM-enabled cyber-social threats and the difficulty of detecting generated text, while noting that empirical study of advanced-LLM social bots remains limited. Social-bot detection draws on account, network, and content signals, but context-dependent coordination methods can generalize poorly.

2.1 LLM-powered cyber threats

  • LLMs can produce human-like, personalized content at low cost, but empirical evidence of malicious field use remains rare.Prior studies include a reported increase in machine-generated text on low-credibility news sites after ChatGPT’s release.Source: §2.1 · p. 3

2.2 LLM-generated content detection

  • Black-box classifiers seek human-versus-machine text patterns, while white-box methods use embedded signals or watermarks; paraphrasing and watermark removal can undermine detection.The cited work characterizes LLM-generated-text detection as a formidable challenge.Source: §2.2 · p. 4

2.3 Social bot detection

  • Supervised social-bot detectors use labeled accounts and features such as metadata, network structure, and posted content.These approaches distinguish social-media bots from text-only chatbots by considering profiles and varied interactions.Source: §2.3 · p. 4
  • Unsupervised detection can find human-mimicking bots through coordinated activity, but varying similarity measures limit generalizability across contexts.Signals include temporal activity, common retweets, and shared URLs.Source: §2.3 · p. 4

2.4 Bots supercharged by LLMs

  • Empirical study of social bots using advanced LLMs remains limited despite earlier work on GPT-2-generated text and self-disclosed algorithmic bots.This study contributes by analyzing a more up-to-date botnet using state-of-the-art AI models.Source: §2.4 · p. 5

3 Identification of the fox8 botnet

The authors used accidental self-revealing tweets as a clue to find candidate LLM-powered bots, then examined accounts linked to recurring suspicious websites. They assembled 1,140 candidate bots and a matched-size set of human accounts for behavioral comparison.

3 Identification of the fox8 botnet

  • ChatGPT’s refusal safeguards can produce standardized self-identifying messages, which bots may post accidentally without suitable filtering.The authors used this self-revealing content as a search clue.Source: §3 · p. 5
  • 76% of 100 manually checked accounts were likely humans posting or retweeting ChatGPT outputs, while the remaining accounts were likely LLM-using bots.The sample came from 12,226 tweets by 9,112 unique accounts found by searching Twitter for “as an ai language model” between Oct. 1, 2022, and Apr. 23, 2023.Source: §3 · p. 5
  • The authors extracted 1,140 accounts linking to three recurring suspicious websites and named the resulting likely ChatGPT-using botnet “fox8.”They report strong indications that the accounts originated from the same botnet.Source: §3 · p. 6
  • The fox8-23 benchmark combines 1,140 bot accounts with 1,140 human accounts sampled from four existing datasets.Up to 200 recent tweets and friend and follower lists were collected for fox8 accounts; the human accounts’ tweets predated ChatGPT, reducing the possibility of data contamination.Source: §3 · p. 6

4 Characterizations

The fox8 accounts exhibit coordinated networks, constrained behavioral patterns, and cryptocurrency-oriented activity. Their shared websites and self-revealing tweets point to suspicious promotion and varied ChatGPT uses, though the authors caution that harmful prompts may be overrepresented in the evidence.

4.1 Profiles

  • Fox8 bots averaged 74.0 followers, 140.4 friends, and 149.6 tweets; most accounts were created more than seven years ago, though some date to 2023.Many profiles mentioned cryptocurrencies and blockchains.Source: §4.1 · p. 6, Figure 1 · p. 7

4.2 Social networks

  • Fox8 bots formed a dense follow network; within-group reply pairs occurred at 0.2%, versus 0.016% for the baseline, while cross-group interactions were rare.The authors suggest the bots follow and interact with each other to boost engagement metrics, while also engaging with accounts outside the botnet.Source: §4.2 · p. 8, Figure 2 · p. 9

4.3 Content type

  • Bots averaged 25.6% original tweets, 36.1% replies, and 38.4% retweets or quotes, concentrating in a narrower behavioral region than human accounts.Many bots also posted photos, often appropriated selfies, to create fake personas.Source: §4.3 · p. 10, Figure 3 · p. 10

4.4 Amplified hashtags and accounts

  • Most external accounts the bots engaged with related to cryptocurrency, blockchain, or NFTs; the authors infer that the bots mainly amplified cryptocurrency and blockchain information.Most shared hashtags were also associated with cryptocurrency or blockchain.Source: §4.4 · pp. 10–11, Figure 4 · p. 11

4.5 Shared websites

  • Three linked websites were unusually prominent, and shared infrastructure, registration timing, pop-ups, and absent editorial details raised red flags.Tweets linking to them often did not cohere with the articles; the authors speculate that ChatGPT-generated text promoted the sites.Source: §4.5 · pp. 11–12, Figure 5 · p. 12

4.6 Self-revealing tweets

  • 81.3% of self-revealing tweets reflected harmful, hateful, or negative-content instructions; 12.3% involved requests beyond the model’s capabilities.Another 4.1% involved other prohibited instructions, and 2.0% contained positive content.Source: §4.6 · p. 12, Table 1 · p. 13
  • Some replies requesting additional information suggest the operator used ChatGPT to make bots act as chatbots; harmful prompts may be overrepresented because they more often elicit self-revealing responses.The authors also report that the bots sometimes chatted with each other.Source: §4.6 · p. 14

5 Detection

The section tests conventional bot detection, LLM-content detectors, and account-level scoring. Botometer misses the fox8 bots; OpenAI’s detector shows promise in controlled comparisons but produces many false positives in field use.

5 Detection

  • Coordinated links, hashtags, and interactions offer unsupervised detection signals, but may not generalize to other LLM-powered bots.The section therefore explores approaches intended to apply across bot types.Source: §5 · p. 14

5.1 Botometer

  • Botometer’s fox8 score distributions are left-skewed, with recall nearly zero at 2.5, so it does not identify the bots.The authors note that Botometer predates ChatGPT and relies on account characteristics despite the bots’ sophisticated, human-like behavior.Source: §5.1 · pp. 14–15, Figure 6 · p. 15

5.2 LLM-generated content detectors

  • OpenAI’s detector scores are typically very high for self-revealing tweets, but span the full range for other tweets; GPTZero probabilities are near zero.Concatenated tweets mostly receive near-zero OpenAI scores, potentially because the tweets were not produced in a single session.Source: §5.2 · pp. 16–17, Figure 7 · p. 16

5.3 Detecting LLM-powered bots

  • A 52.7 threshold maximizes account-level F1 at 0.84 on fox8-23, where bots have higher average OpenAI detector scores than humans.The means are 57.7 for bots and 48.6 for humans; human scores have the larger standard deviation.Source: §5.3 · p. 17, Figure 8 · p. 18
  • 815 random accounts are labeled bots at the 52.7 threshold, but many may be false positives, especially accounts with very short tweets.The random sample’s scores are higher than fox8-23 humans; among accounts with at least four qualified tweets, scores are not significantly different from those humans.Source: §5.3 · p. 18, Figure 8 · p. 18

6 Conclusion and discussion

The case study documents a Twitter botnet that appears to use ChatGPT for human-like content, including harmful comments and promotion of dubious websites. Detection remains difficult: classical bot methods are inadequate, and text scoring has field-use challenges.

6 Conclusion and discussion

  • The fox8 bots use human-like profiles, stolen images, mutual following, replies, and retweets; self-revealing tweets suggest ChatGPT generates harmful comments and original posts or replies.The study also finds coordinated promotion of dubious websites and speculates that the accounts follow a single probabilistic model for activity types and frequencies.Source: §6 · p. 19
  • Classical bot detection methods prove inadequate, while OpenAI’s text classifier shows potential in controlled tests but faces field-use challenges and high false positives.The authors caution that it is premature to rely solely on this classifier for field detection.Source: §6 · p. 19
  • Findings may not represent other LLM-powered bots because the analysis covers one Twitter botnet and cannot guarantee all fox8 content is LLM-generated.The authors also warn that suspended free API access may make replication or discovery of new bots impossible.Source: §6 · p. 19
  • Future bots may stop posting self-revealing tweets, use less safeguarded or purpose-trained models, and become more autonomous or multimodal.The fox8 bots currently use pre-established rules and ChatGPT for content generation and dialogue; the case study bots use language models for text only.Source: §6 · pp. 19–20
  • The authors call for better short-text detection, bot-specific regulation, and public awareness, while cautioning that awareness efforts can amplify existing cognitive biases.They state that precise regulation proposals are beyond the paper’s scope.Source: §6 · p. 20

Related papers

Loading 2307.16336v1…