Spear Phishing With Large Language Models

• Spear phishing uses personalised information to manipulate targets into revealing sensitive information.Source: §1 · p. 1
• The paper analyses how LLMs can be integrated into multiple stages of the cyber kill chain, focusing on spear phishing.Source: §1 · p. 1
• LLMs can assist reconnaissance by gathering targets’ personal information for effective spear-phishing messages.Source: §1 · p. 1
• Basic prompt engineering can bypass LLM safety measures and elicit guidance on spear-phishing attacks, malware, and persuasive email principles.Source: §1 · p. 1

• Social engineering uses deception and manipulation to obtain sensitive information or unauthorised access to systems.Source: §2 · p. 2
• Humans’ vulnerability makes social engineering a critical component of successful cyberattacks and the most common form of internet crime.Source: §2 · p. 2
• Phishing mimics an authentic party to build trust and persuade people to share passwords, click malicious links or attachments, or transfer funds.Source: §2 · p. 2
• Spear phishing has enabled major breaches, including the compromise of Sony executives and a presidential campaign chairperson’s private email account.Source: §2 · p. 2

• GPT-3, PaLM, and GPT-4 can generate coherent text, answer detailed questions, reason through complex problems, and write code.Source: §3 · p. 2
• Transformer architectures use attention mechanisms and next-token prediction over large datasets to learn language and generate text.Source: §3 · p. 2
• Scaling trains progressively larger models on more data and computation, improving performance across a wider range of tasks.Source: §3 · p. 2
• Scaling can produce emergent abilities and risks that existing defenses may not manage, including GPT-4’s improved assistance with social engineering.Source: §3 · p. 2
• Publicly accessible LLMs have made realistic, inexpensive spear-phishing email generation feasible for cybercriminals.Source: §3 · p. 2

• Earlier research warned that increasingly capable AI could automate social engineering, including tailored messages that mimic trusted contacts at scale.Source: §4 · pp. 2–3
• Modern LLMs can identify targets, conduct background research, craft personalised messages, and design basic forms of malware.Source: §4 · p. 3
• SNAP_R generated targeted phishing tweets six times faster than a comparable human while achieving a similar click rate.Source: §4 · p. 3
• Recent language-modelling advances have made automated social-engineering attacks increasingly feasible compared with earlier systems.Source: §4 · p. 3
• Darktrace observed a 135% increase in novel social-engineering attacks among thousands of active customers between January and February 2023, attributing the rise to LLM-powered chatbots.Source: §4 · p. 3

5 How LLMs could be used for cyberattacks

• $10 for 1,000 emails in under 2 hours illustrates how LLMs can make personalised spear phishing economically scalable.Using Claude, the paper reports that a hacker could generate 1,000 emails for $10 USD in under 2 hours.Source: §5 · p. 3

5.2 Collect

• LLMs can automate target collection by converting biographical information into personalised messages referencing an MP’s region, party, interests, and other details.The demonstration scraped Wikipedia pages for British MPs, summarised them, and used the resulting information to generate emails.Source: §5.2 · p. 4, §5.1 · p. 4

5.3 Contact

• Combining personalisation, contextual relevance, psychological appeals, and authority cues with target details provides a prompt-based route to generating spear phishing emails at scale.The author first elicited effective-email principles, then combined those principles and biographical details in a single prompt.Source: §5.3 · p. 4
• GPT-4 generated more sophisticated messages than GPT-3.5 and GPT-3, drawing on recipients’ work histories, backgrounds, values, and political priorities.The examples were described as human-like despite some exaggerated or contrived passages.Source: §5.3 · pp. 5–6

5.3.1 Open-source models

• The open-source oasst-sft-6-llama-30b model appeared comparable to GPT-3.5 but varied substantially in quality and remained less sophisticated than GPT-4.Some outputs were convincing, while others were subpar or irrelevant.Source: §5.3.1 · p. 6

5.4 Compromise

• GPT-4 generated a VBA macro that downloaded and executed an external payload, demonstrating a basic compromise capability and the difficulty of preventing malicious jailbreaking.The example was rudimentary and did not establish that LLMs can design sophisticated malware.Source: §5.4 · p. 7

6 Discussion

• LLMs can help scale spear phishing by reducing cognitive workload, financial costs, and skill requirements for attackers.They can generate human-sounding personalised emails, operate continuously, process unstructured data, lower per-email costs, and support relatively low-skilled attackers.Source: §6 · pp. 7–8
• Less than one cent per email with GPT-3.5 and a few cents with GPT-4 lower the marginal financial cost of spear phishing attempts.Source: §6 · p. 8

6.1 Governance challenges with LLMs

• LLM-based spear phishing raises governance difficulties because models are dual-use, prompt safeguards are bypassable, and data filtering may impair benign email generation.The paper argues that model-level interventions may produce an unfavorable Misuse-Use Tradeoff by stifling both legitimate use and misuse.Source: §6.1 · p. 8
• Completely stopping phishing is likely unrealistic in a vast, adaptive cyberspace, so mitigation is a more attainable policy objective.Source: §6.1 · p. 8

6.2 AI-based cyberattacks in the future

• Email is only one channel for AI-enabled cybercrime, with future research needed to assess risks and attack vectors across additional communication channels.The paper notes evidence that scammers have used AI-generated voice clones to deceive targets.Source: §6.2 · p. 8
• AI agents may automate increasingly sophisticated hacking and deception campaigns with little or no human involvement.Agentic systems could pursue open-ended goals, conduct natural-language dialogue, and scale spear phishing beyond current campaigns.Source: §6.2 · p. 8

7.1 Structured access schemes

• Structured access schemes can control interactions, support compliance, and enable cooperation with law enforcement to trace problematic user-generated content.APIs can facilitate governance strategies that ban or sanction malicious users.Source: §7.1 · p. 8
• Layered LLM screening can classify prompt risk and subject high-risk queries to more advanced analysis for malicious activity.A smaller fine-tuned model can categorize prompts before sophisticated models flag suspected terms-of-use violations.Source: §7.1 · p. 9
• Open-source LLMs threaten structured-access governance because unmonitored users can evade provider intervention and fine-tune models for malicious use.The same openness enabling beneficial applications can let malicious actors operate more covertly.Source: §7.1 · p. 9

7.2 LLM-based defensive systems

• LLM-based defensive systems can inspect incoming emails for deceptive URLs, writing inconsistencies, suspicious addresses, and other malicious content.They can compare messages with prior correspondence from recognized contacts to identify anomalies.Source: §7.2 · p. 9
• Fine-tuning on phishing examples could improve detection beyond what human recipients can feasibly achieve, although further development is needed for sophisticated attacks.The paper notes emerging defensive adoption, including Google’s 2023 Sec-PaLM announcement.Source: §7.2 · p. 9

8 Conclusion

• LLMs can assist multiple cyber kill-chain stages, and outsourcing personalized email generation could improve cybercriminal effectiveness.The paper specifically discusses the collect, contact, and compromise phases.Source: §8 · p. 9
• Realistic spear phishing emails can be generated quickly and inexpensively, enabling large campaigns with limited technical expertise and financial resources.Once infrastructure exists, it can be reused with little additional effort, while per-email costs approach declining inference costs at scale.Source: §8 · p. 9
• The dual-use nature of LLMs makes governance difficult, and basic prompt engineering can circumvent model-level safeguards.The paper calls for empirically informed interventions balancing beneficial applications against misuse prevention.Source: §8 · pp. 9–10
• Structured access can control interactions, support compliance and law-enforcement cooperation, while LLMs can screen emails for malicious content.These approaches are presented as possible ways to navigate the tradeoff between beneficial use and misuse prevention.Source: §8 · p. 10

Appendix A: Examples of spear phishing emails across various LLMs

• The examples target varied roles, including legal-firm associates, US government analysts, congressional staffers, the Secretary of Commerce, and corporate directors.Prompts provide role-specific contexts and requests to increase message plausibility.Source: Appendix A · pp. 12–16
• Password-reset examples ask recipients to click links after a purported security incident or network-security request.One prompt mentions suspicious activity requiring users to change passwords, while another generated email asks users to reset them.Source: Appendix A · p. 12
• A generated conference email uses named officials, a specific event, and existing-credential login instructions to request attendance confirmation.The example identifies the Emerging Disruptive Technology for Defense conference and directs the recipient to an account page.Source: Appendix A · p. 14
• Reported generation times range from 2 seconds to 40 seconds across the examples.The appendix reports 2 seconds, 3 seconds, 7 seconds, 14 seconds, and 40 seconds for different messages.Source: Appendix A · pp. 12–16
• Other examples imitate institutional outreach by inviting a congressional staffer to a Brookings climate panel or presenting a semiconductor petition to the Commerce Secretary.The messages use attachments or links and institution-specific policy contexts.Source: Appendix A · pp. 15–16