Source-linked AI summary
Getting pwn'd by AI: Penetration Testing with Large Language Models
Andreas Happe, Jürgen Cito
TL;DR
Penetration testing requires substantial expertise and manual analysis, motivating the question of whether LLMs can augment testers as AI sparring partners. The paper evaluates high-level planning and a low-level GPT3.5 feedback loop with a vulnerable virtual machine, finding promising initial behavior while identifying grounding, stability, moderation, and ethical limitations.
Problem
Penetration testing requires high expertise and many manual testing and analysis steps, while practitioners need sparring partners for alternative ideas and approaches.
Method
The paper evaluates LLMs for high-level penetration-test planning and low-level vulnerability hunting through GPT3.5 interacting over SSH with a vulnerable virtual machine.
Results
The experiments produced realistic high-level attack plans and routinely gained root privileges in the vulnerable virtual machine, with GPT3.5 also suggesting context-linked attack vectors.
Takeaways & Limitations
LLMs can act as penetration-testing sparring partners that provide attack suggestions and introductory vulnerability explanations within the tested scope.
Takeaways & Limitations
The prototype’s single runs were unstable, and GPT3.5 sometimes hallucinated commands or relied on pattern matching rather than deeper system understanding.
Abstract
from arXiv · showhide
The field of software security testing, more specifically penetration testing, is an activity that requires high levels of expertise and involves many manual testing and analysis steps. This paper explores the potential usage of large-language models, such as GPT3.5, to augment penetration testers with AI sparring partners. We explore the feasibility of supplementing penetration testers with AI models for two distinct use cases: high-level task planning for security testing assignments and low-level vulnerability hunting within a vulnerable virtual machine. For the latter, we implemented a closed-feedback loop between LLM-generated low-level actions with a vulnerable virtual machine (connected through SSH) and allowed the LLM to analyze the machine state for vulnerabilities and suggest concrete attack vectors which were automatically executed within the virtual machine. We discuss promising initial results, detail avenues for improvement, and close deliberating on the ethics of providing AI-based sparring partners.
1 INTRODUCTION
The paper asks whether LLMs can serve as sparring partners that augment penetration testers amid workforce shortages and manual testing demands. It examines high-level planning, low-level vulnerability hunting, and related ethical boundaries.
- Motivation: Cybersecurity workforce growth lagged behind the expanding workforce gap, while penetration testers reported needing human sparring partners for alternative ideas.The gap increased 26.2% YoY while the workforce grew 11.1% YoY.
- Motivation: AI sparring partners could support vulnerability intuition, knowledge transfer, and novice penetration-tester training while retaining human involvement.The paper links sparring to alternative approaches, CTF-derived intuition, and potential benefits for low-skilled workers.
- Research question and approach: The research question is whether security testing can be automated with LLMs, using MITRE ATT&CK as a structure for tactics, techniques, and procedures.The paper evaluates both generic and organization-specific penetration-test planning and low-level attack guidance.
- Scope: The paper limits its scope by excluding phishing and vishing attacks intended to deceive humans, while identifying automated penetration-test reporting as another possible application.The exclusion is attributed to ethical reasons; report-generation experimentation is not pursued here.
2 BACKGROUND
The background introduces LLMs, autonomous agents, and MITRE ATT&CK as the conceptual and technical foundations for AI-assisted penetration testing. It frames a sparring partner as needing coverage from high-level tactics to low-level procedures.
- MITRE ATT&CK: MITRE ATT&CK organizes adversary knowledge hierarchically into tactics, techniques, and procedures, from objectives to concrete execution details.Examples include reconnaissance, privilege escalation, collection, Sudo abuse, and Kerberoasting.
- MITRE ATT&CK: The paper assumes an effective penetration-testing sparring partner should select suitable tactics and techniques, then derive feasible techniques and procedures for a chosen tactic.This requirement spans both high-level planning and low-level attack execution.
- Large Language Models: LLMs are self-supervised neural networks whose capabilities depend partly on parameter scale, while foundation models can be refined for specific uses.The background contrasts billion- to trillion-parameter models and notes that training new models is prohibitively expensive for most researchers.
- Large Language Models: Prompt engineering concerns the careful preparation of prompts and the iterative exchange of questions and answers with systems such as ChatGPT.The background presents prompting as a distinct discipline associated with LLM use.
- Pre-trained Autonomous AI Agents: Autonomous-agent systems use LLMs to generate, decompose, prioritize, execute, and contextualize tasks, reducing manual prompt engineering and supporting iterative execution.AutoGPT and BabyAGI are presented as examples of these patterns.
3 LLM-BASED PENETRATION TESTING
The paper evaluates LLM assistance at two levels: strategic penetration-test planning and concrete attack guidance against a vulnerable system. The low-level prototype closes a feedback loop between GPT3.5 and an SSH-connected virtual machine.
- High-Level Guidance: The high-level use case asks LLMs for attack methodologies that map goals such as Active Directory attacks to suitable tactics and techniques.This differs from low-level guidance, where the tactic is already selected and the model searches for system-specific techniques and procedures.
- High-Level Guidance: AgentGPT generated realistic Active Directory attack vectors, including password spraying, Kerberoasting, AS-REP roasting, and delegation abuse.The paper reports that these vectors were realistic, feasible, and commonly used in penetration testing.
- High-Level Guidance: AutoGPT produced an external penetration-testing plan containing vulnerability scanning, OSINT and user enumeration, and phishing against identified users.The plan used operations typically performed during external penetration tests and could further inspect the target company’s web page.
- Low-Level Guidance: The low-level system connected GPT3.5 through SSH to a deliberately vulnerable Linux virtual machine and executed model-generated shell commands.The model received command output as feedback and was asked to identify vulnerabilities and provide exploitation examples as verification commands.
- Low-Level Guidance: The prototype routinely gained root privileges by using findings such as sudoers entries, GTFObins, and exposed password files.A modified prompt requesting a reverse shell also successfully produced root shells, while SUID findings were not actively exploited.
4 DISCUSSION
The prototype showed that GPT3.5 could use system feedback to suggest plausible privilege-escalation paths, but its behavior was unstable, pattern-based, and ethically difficult to constrain.
- GPT3.5’s suggestions indicated context-sensitive reasoning after retrieving sudoers or passwd information.Sudoers output led to vulnerable sudo-command suggestions, while passwd output led to attacks on weakly configured accounts.
- Some exploit suggestions were reasonable from Linux context but appeared without prior enumeration, suggesting reliance on learned patterns.The paper specifically cites dirty_cow as an example of a system exploit suggested without earlier enumeration.
- Pure hallucinations were infrequent, but “exploit.sh” was the most common repeated hallucinated suggestion and may reflect training-data memorization.The authors considered it reasonable that security write-ups containing this script appeared in GPT3.5’s training data.
- Single runs varied in command sequences and identified vulnerabilities, while repeated or longer runs converged; compared with linpeas.sh, LLM behavior was less deterministic.The authors attribute instability partly to overfocus on individual system aspects and note that linpeas.sh uses a curated checklist.
- Prompt wording could reduce ethical refusals, and local models would remove server-side ethics checks.The optional vulnerability-explanation step was more often denied, but this did not affect overall hacking progress.
5 A VISION OF AI-AUGMENTED PEN-TESTING
The authors envision combining high-level planning and low-level exploitation in one human-supervised system, while improving memory, prompting, and model deployment.
- Integration of High- and Low-Level: Integrating high-level task planning with low-level exploitation could provide a more uniform user experience and enable synergy from shared system knowledge.Operators could move from questions about Active Directory attacks to privilege escalation on a concrete system.
- Integration of High- and Low-Level: The prototype separated high- and low-level tasks across two different LLMs.The envisioned system would instead keep the interaction within a single system.
- Local Deployment: Local models could avoid cloud costs and sensitive-data sharing while enabling customer-specific training and fine-tuning.The authors suggest training on engagement data or across subsequent penetration tests.
- Memory: The prototype’s memory stored executed command outputs in prompts until the model’s context limit was reached.The authors propose reflected summaries and separate memory streams for commands, findings, and system context.
- Prompting: Automatically generating and optimizing prompts may improve effectiveness, but sensitive-use prompts require close human monitoring.The current prototype relied on static, manually written prompts.
- Prompting: Studying the questions penetration testers ask themselves could inform better prompts and understanding of the industry.The proposed research connects prompt design to empirical studies of practitioner workflows.
6 FINAL ETHICAL CONSIDERATIONS
The paper frames AI-augmented penetration testing as dual-use research: attackers are expected to explore automated LLM-based hacking, so defenders must prepare for that possibility.
- The paper studies LLM augmentation in benign penetration-testing settings, while acknowledging that the tools can be subverted for malicious purposes.The authors connect this concern to increasing reported malicious AI use by APTs and common criminals.
- Server-side supervision cannot reliably contain misuse because models can run locally and leaked models can be reused by malicious actors.The authors state that fine-tuning leaked models for malicious activities is within APT reach.
- The authors assume attackers will explore LLM possibilities, including fully automated approaches, because experimentation has low entry costs.They conclude that this development can no longer be contained.
- The paper concludes that defenders need to prepare for LLM-enabled attacks and that LLMs can play a significant defensive role.