Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities

• LLMs support applications from code creation and email writing to university coursework, while achieving strong performance across several fields.Source: §1 · pp. 1–2
• LLM text generation has been associated with research-practice concerns, fraud, social engineering, misinformation, malware, illicit drugs, and cyber weapons.Source: §1 · p. 2
• WormGPT and FraudGPT, based on existing language models, have been developed and distributed online, while Europol identified fraud, disinformation, and cybercrime as affected criminal areas.Source: §1 · p. 2
• The paper reviews safety and security work through a taxonomy linking generative threats, prevention measures, and vulnerabilities that arise from imperfect prevention.Source: §1 · pp. 2–3, Figure 1 · p. 2
• Threats include phishing, malware, and misinformation; prevention includes content filtering, RLHF, and red teaming; vulnerabilities include jailbreaking and prompt injection.Source: Figure 1 · p. 2, §1 · p. 3
• The paper concludes by discussing public perceptions of LLM-enabled threats, limitations of prevention strategies, and future concerns from continued development.Source: §1 · p. 3

• Weidinger et al. (2022) classify 21 LLM risks into six areas and distinguish observed risks from anticipated risks.Source: §2 · p. 3
• This paper overlaps with prior taxonomies on misinformation and malicious use but emphasizes newer security concepts such as prompt injection attacks.Source: §2 · p. 3
• Huang et al. (2023) distinguish inherent issues, intended attacks, and unintended bugs, whereas this paper focuses exclusively on deliberate attacks for potentially illicit purposes.Source: §2 · p. 3
• Other reviews organize LLM risks around privacy, security, responsibility, fairness, foundation-model impacts, or prompt injection rather than the paper’s broader security-literature overview.Source: §2 · p. 3
• Gupta et al. (2023) cover ChatGPT threats across attacks, cyber offense, cyber defense, and social, legal, and ethical issues, relying mainly on news and blog reports.Source: §2 · pp. 3–4

3.1 Adversarial attacks against ML models

• Adversarial examples are small, semantics-preserving input modifications that can drastically change predictions, first demonstrated through imperceptible image perturbations.Source: §3.1 · p. 4
• Research spans adversarial attacks, detection, and defenses across vision, language, and reinforcement learning, although practical applications have received comparatively little attention.Source: §3.1 · p. 4

3.2 LLMs and adversarial attacks

• ChatGPT shows better zero-shot adversarial robustness than DeBERTa, BART, and BLOOM, but performance remains far from perfect and attack success rates against ChatGPT are high.Source: §3.2 · p. 4
• Character-level perturbations to few-shot examples substantially reduce GPT2-XL and LLaMA-7B performance, showing brittleness in in-context learning.Source: §3.2 · pp. 4–5
• Adversarial visual inputs caused MiniGPT-4 to generate harmful content despite unperturbed-input safety behavior, circumventing safeguards through PGD-based optimization.Source: §3.2 · p. 5
• Prefix-tuned GPT-2 models were vulnerable to adversarial attacks across multiple text-classification datasets, extending concerns to parameter-efficient adaptation.Source: §3.2 · p. 5
• GPT-4 was used to generate code circumventing the AI Guardian defense, illustrating LLMs’ potential role as assistants for constructing attacks.Source: §3.2 · p. 5

3.3 Security issues beyond adversarial attacks

• Beyond adversarial examples, newer LLM security attacks include jailbreaking and prompt injection and generally require substantially more human intervention.Source: §3.3 · p. 5

• The literature collection includes peer-reviewed articles, preprints, and news articles identified through searches guided by the authors’ knowledge and expertise.Source: §4 · p. 5
• The authors do not guarantee a complete collection of existing work and instead aim to outline threats and considerations relevant to LLM users and practitioners.Source: §4 · p. 5
• Among 36 works in the Threats section, 27 were peer-reviewed, representing 75% of the sample.Source: §4 · p. 6
• Peer-review coverage was lower for Prevention measures, with 20 of 42 works reviewed, or 48%.Source: §4 · p. 6
• Vulnerabilities had the lowest peer-review coverage, with 3 of 15 works reviewed, or 20%.Source: §4 · p. 6, Figure 2 · p. 6

5 Threats

• LLMs enable threats through direct misuse of generated content and through manipulation of models or their training data.The reviewed threat categories include fraud, malware, scientific misconduct, misinformation, data memorization, and data poisoning.Source: §5 · p. 6

5.1 Fraud, impersonation, social engineering

• 600 UK MPs were targeted with personalized phishing emails, while Claude could generate 1,000 phishing emails for $10 in around two hours.Hazell (2023) conditioned GPT models on MPs’ Wikipedia articles; the demonstration was qualitative rather than a quantitative evaluation of email quality.Source: §5.1 · p. 6

5.2 Generating malware

• LLMs can generate malicious computer code, lowering the programming-skill barrier for producing malware, and tools such as WormGPT and FraudGPT have been distributed for cybercriminal use.WormGPT is based on GPT-J, while FraudGPT offers phishing-email and malware-generation functionality.Source: §5.2 · p. 7

5.3 Scientific misconduct

• ChatGPT can generate original content that tends to circumvent plagiarism detection, raising concerns about plagiarism and scientific-practice integrity.The paper treats this misuse as misconduct rather than necessarily as a criminal act.Source: §5.3 · p. 7

5.4 Misinformation

• GPT-2 models generated misinformation at scale that appeared credible to human readers, while GPT-3-generated tweets were generally indistinguishable from human-written tweets.Kreps et al. (2022) found that most crowdworkers rated generated news as credible, and Spitale et al. (2023) found participants could not generally distinguish AI-generated from human-written tweets.Source: §5.4 · pp. 7–8

5.5 Data memorization

• LLM memorization can leak sensitive information, with GPT-2 Large reaching 23% recall and 30% precision for PII extraction on law cases without differential privacy.Differentially private fine-tuning reduced the same model’s precision and recall to around 3%; larger models and duplicated data increase leakage risks.Source: §5.5 · pp. 8–9

5.6 Data poisoning

• Poisoned training data can implant targeted behaviors, including backdoors, misinformation, sentiment steering, code injection, and over-refusal, while black-box defenses remain challenging.AutoPoison altered behavior without compromising fluency; VPI was effective with 1% poisoned training data, and data filtering outperformed an additional inference-time instruction as a defense.Source: §5.6 · pp. 10–11

6.1 Preventing misuse of LLMs via content detection

• AI-generated-content detection is intended to flag potentially malicious misinformation, plagiarism, and fabricated content that appears human-written.Source: §6.1 · p. 11
• Detection methods include watermarking, discriminative classifiers, and zero-shot approaches that use token probabilities or perturbation behavior.Watermarking modifies generation for algorithmic detection, while DetectGPT compares log-probability changes after perturbing text.Source: §6.1 · pp. 11–12
• A detector achieved AUC 0.97 in-distribution but only 0.66 on an out-of-distribution challenge set, and newer LLMs could deceive it.Source: §6.1 · p. 12
• Paraphrasing can circumvent AI-text detectors, while watermarking and retrieval-based systems may misclassify human text as machine-generated.Source: §6.1 · p. 12
• Detection systems can misclassify non-native English writing as AI-generated, creating risks of unfair disadvantage in evaluative and educational settings.Source: §6.1 · p. 12

6.2 Red teaming

• Human red teams generated 38,961 attacks against models ranging from 2.7B to 52B parameters and spanning four safety configurations.Source: §6.2 · p. 13
• Across model sizes and LLM types, red-team experiments found persistent susceptibility and clear failure modes, although rejection sampling was especially difficult to red-team.RLHF-trained models became harder to red-team as model size increased, but remained vulnerable overall.Source: §6.2 · p. 13
• Automated red teaming uses one LLM to generate test questions for another, replacing the need for a human workforce.Source: §6.2 · p. 13, Figure 6 · p. 13
• Reinforcement-learning-generated attacks produced around 40% offensive texts, compared with 3.7% for zero-shot prompting across 500,000 completions.Source: §6.2 · p. 13
• Automated red teaming also measured training-data memorization and distributional bias, including Gopher’s generation of real phone numbers and email addresses.Source: §6.2 · p. 13

6.3 LLM content filtering

• Content filters fine-tune Transformer models to detect sexual, hateful, violent, self-harm, harassment, or toxic content, including with LLMs acting as moderators.Source: §6.3 · p. 14

6.4 Safeguarding via RLHF

• RLHF uses human preferences to condition LLM behavior toward helpful and harmless outputs, with crowdworkers separately ranking helpfulness and harmfulness.Source: §6.4 · p. 14
• Helpfulness and harmlessness were anti-correlated in RLHF experiments, indicating a potential trade-off between the two objectives.Source: §6.4 · p. 14
• Constitutional AI replaces human feedback with AI-generated critiques and revisions, and models combined with chain-of-thought performed on par with human-feedback-trained models.Source: §6.4 · pp. 14–15
• RLHF implementation is difficult because policy, behavior policy, reward, and value models interact, requiring extensive hyperparameter tuning.Source: §6.4 · p. 15
• XSTEST found exaggerated safety behavior: LLaMA-2 fully refused almost 40% of safe prompts, compared with 6% for GPT-4.The reported behavior was attributed to over-reliance on lexical cues.Source: §6.4 · p. 15

6.5 Safety via instruction-following

• Larger RLHF-trained models increasingly self-corrected biased or discriminatory responses when given explicit safety instructions, with this behavior emerging around 22B parameters.Source: §6.5 · p. 15
• Instruction-based safety avoids additional model training by directly adding requests for unbiased, non-stereotyped answers to input prompts.Source: §6.5 · p. 15

6.6 Methods to avoid memorization

• PPO fine-tuning can reduce exact training-data sequence generation by encouraging suffixes that are dissimilar to memorized examples without sacrificing general generation quality.Source: §6.6 · pp. 15–16
• Larger GPT-Neo models generated more diverse suffixes, with dissimilarity scores increasing alongside model size.Source: §6.6 · p. 16

Privacy-preservation through prompt-tuning

• RAPT combines privatized data, prompt- and prefix-tuning, and token reconstruction to improve robustness against embedding-inversion and attribute-inference attacks.Source: Privacy-preservation through… · p. 16
• Stronger privacy constraints reduced downstream accuracy, while privatized token reconstruction improved performance on privatized datasets.Source: Privacy-preservation through… · p. 16

Scope limitation

• Most existing mitigation methods focus on BERT-sized models rather than billion-parameter LLMs, so defending larger models against data poisoning remains an open research challenge.Existing work nevertheless shows that larger models are vulnerable to data poisoning.Source: §6.7 · p. 16

Perplexity-based defense

• ONION effectively defended BERT-based models against all five evaluated backdoor attacks across sentiment, hateful-content, and news-classification datasets.It detects suspicious tokens through changes in perplexity after single-word removal, then removes tokens exceeding a threshold.Source: §6.7 · pp. 16–17

Perturbation-based defense

• RAP classifies examples as poisoned using robustness-aware perturbations and outperformed existing defenses on BERT-based sentiment-analysis and toxicity-detection tasks.Its motivation is that poisoned examples are more robust to adversarial perturbations.Source: §6.7 · p. 17

Representation-based defense

• DAN detects poisoned examples through their separation from clean examples in feature space and surpassed existing detection baselines.Other feature-based methods use lexical and syntactic correlations to remove compromised training data, achieving near-perfect defense particularly for insertion-based attacks.Source: §6.7 · p. 17

Gradient-based defense

• 97% attack-success-rate reduction was achieved by IMBERT against multiple insertion-based attacks while retaining competitive clean-dataset accuracy.IMBERT identifies triggers through their large-magnitude gradients relative to other tokens.Source: §6.7 · p. 17

Attribution-based defense

• AttDef outperformed two baselines while preserving comparable clean-dataset accuracy and significantly reducing attack success rates.The method combines ELECTRA-based poisoned-instance detection with partial layer-wise relevance propagation for trigger identification.Source: §6.7 · p. 17

7.1 Prompt injection

• Prompt injection divides into prompt leaking, which extracts system prompts, and goal hijacking, which manipulates models into ignoring them.These vulnerabilities can provide access to or overwrite security instructions that models should follow.Source: §7.1 · p. 18, Figure 7 · p. 18
• Prompt leaking exceeded 60% success across all tested models and datasets, while its n-gram-overlap defense remained circumventable through symbols or Caesar-cipher encryption.The evaluation tested GPT-3.5, GPT-4, and Vicuna-13B using paired secret prompts and user queries.Source: §7.1 · pp. 18–19
• Indirect prompt injection can hide adversarial instructions in retrieved sources such as website HTML, while multimodal attacks embed instructions in any image or audio recording.Multimodal attacks include targeted-output attacks and dialog poisoning, which manipulates behavior across a conversation.Source: §7.1 · pp. 19–20

7.2 Jailbreaking

• GCG-generated nonsensical suffixes successfully attacked LLaMA-2-7B-Chat and Vicuna-7B and transferred to Pythia, Falcon, Guanaco, GPT-3.5, GPT-4, and PaLM-2.This transfer shows that alignment approaches can be circumvented across multiple architectures.Source: §7.2 · p. 20
• Jailbreaking bypasses safeguards without requiring system-prompt access, and instruction-embedded attacks made all three tested instruction-following models vulnerable to varying degrees.Examples include DAN-style prompts and malicious instructions embedded in translation tasks.Source: §7.2 · pp. 20–21
• Kang et al. showed that obfuscation, code injection, and virtualization bypassed OpenAI content filters for most tested malicious-use cases.ChatGPT also scored highly when human annotators evaluated generated phishing and scam emails for convincingness and personalization.Source: §7.2 · p. 21, Figure 9 · p. 21
• Wei et al. attribute jailbreaking to competing objectives and mismatched generalization, finding that several attack combinations successfully elicited unwanted behavior.They argue that scaling alone will not ensure safety and propose safety-capability parity between safety mechanisms and base-model pretraining.Source: Figure 8 · p. 19, §7.2 · p. 22
• JAILBREAKER achieved greater efficacy against OpenAI models and produced the first successful jailbreaks for Bard and Bing Chat, revealing previously unnoticed vulnerabilities.The method infers internal defense architectures from response times and autonomously generates universal jailbreak prompts.Source: §7.2 · p. 22

8.1 Public concerns around LLMs

• ChatGPT’s popularity has produced public concern about bias, robustness, reliability, toxicity, and AI safety across users and governments.An analysis of around 300,000 tweets identified these recurring concern categories.Source: §8.1 · p. 23, §8 · p. 23
• Around 46% of the reviewed papers were not peer-reviewed, leading the authors to emphasize peer review for prioritizing pressing threat-enabling issues.The paper reports 43 of 93 discussed papers as not peer-reviewed.Source: §8.1 · p. 23

8.2 Limitations of LLM safety

• Theoretical work argues that incomplete alignment leaves LLMs susceptible to adversarial prompts and that stronger security may trade off against standard accuracy.The discussion frames this as a conflict between LLM utility and safety.Source: §8.2 · p. 24

8.3 An outlook on future LLM-enabled security concerns

• Future concerns include personalization-related addiction, dependency, privacy risks, access disparities, polarization, and influence operations.The discussion links personalization to echo chambers and notes that LLMs can show deceptive behavior and human-like susceptibility to persuasion.Source: §8.3 · p. 24
• Human-like generated digital content may increase distrust in online media, while personalization could intensify the targeting of misleading information.CounterCloud is cited as an example of an autonomous news platform requiring no human intervention.Source: §8.3 · p. 24
• The outlook deliberately excludes several possible future crime areas, including robotics and financial-market disruption, because they are beyond the paper’s scope.The authors focus instead on personalization and digital-information dissemination.Source: §8.3 · p. 24

9 Conclusion

• The paper’s conclusion organizes existing work around LLM threats, prevention strategies, and vulnerabilities associated with illicit purposes.It covers risks arising in both academic and real-world settings.Source: §9 · p. 24
• The authors argue that peer review is important for identifying and prioritizing relevant concerns in this rapidly developing field.The conclusion connects this priority to awareness of current and future LLM-related risks.Source: §9 · p. 25