Source-linked AI summary
AutoPentester: An LLM Agent-based Framework for Automated Pentesting
Yasod Ginige, Akila Niroshan, Sajal Jain, Suranga Seneviratne
TL;DR
Pentesting is time-consuming and demands expertise amid a shortage of cybersecurity professionals. AutoPentester automates pentesting with an LLM-agent framework; evaluations report higher subtask completion and vulnerability coverage than PentestGPT.
Problem
Rising cyber threats and pentesting demand outstrip skilled professionals, while assessments are repetitive, time-consuming, and require substantial manual effort and expertise.
Source: §I · p. 1Method
AutoPentester is an LLM-agent framework that automates pentesting through iterative strategy generation and security-tool command execution.
Source: §I · pp. 1–2Results
27.0% higher subtask completion rate and 39.5% greater vulnerability coverage than PentestGPT; AutoPentester also required 18.7% fewer steps and 92.6% less human intervention.
Source: §I · p. 2, §VI · p. 11Takeaways & Limitations
The authors describe its structured automation as suitable for enterprise security assessments and large-scale red-team drills, while cautioning that automated exploitation requires care in production.
Source: §VI · p. 11Fully automated mode struggles with GUI-based interfaces accessed through CLI tools, and RAG-focused commands can miss exploit corner cases.
Source: §VI · p. 11
Abstract
from arXiv · showhide
Penetration testing and vulnerability assessment are essential industry practices for safeguarding computer systems. As cyber threats grow in scale and complexity, the demand for pentesting has surged, surpassing the capacity of human professionals to meet it effectively. With advances in AI, particularly Large Language Models (LLMs), there have been attempts to automate the pentesting process. However, existing tools such as PentestGPT are still semi-manual, requiring significant professional human interaction to conduct pentests. To this end, we propose a novel LLM agent-based framework, AutoPentester, which automates the pentesting process. Given a target IP, AutoPentester automatically conducts pentesting steps using common security tools in an iterative process. It can dynamically generate attack strategies based on the tool outputs from the previous iteration, mimicking the human pentester approach. We evaluate AutoPentester using Hack The Box and custom-made VMs, comparing the results with the state-of-the-art PentestGPT. Results show that AutoPentester achieves a 27.0% better subtask completion rate and 39.5% more vulnerability coverage with fewer steps. Most importantly, it requires significantly fewer human interactions and interventions compared to PentestGPT. Furthermore, we recruit a group of security industry professional volunteers for a user survey and perform a qualitative analysis to evaluate AutoPentester against industry practices and compare it with PentestGPT. On average, AutoPentester received a score of 3.93 out of 5 based on user reviews, which was 19.8% higher than PentestGPT.
I. INTRODUCTION
AutoPentester addresses the time and expertise burden of pentesting and limitations in existing automation with an iterative LLM-agent framework. The paper reports higher benchmark performance than PentestGPT and positive professional survey results.
I. INTRODUCTION
- Pentesting is time-consuming and expertise-intensive, with a typical test taking over two weeks amid a shortage of skilled professionals.Source: §I · p. 1
- Prior LLM-based pentesting tools often need human guidance to plan strategies, execute commands, and process outputs, limiting automation.Earlier reinforcement-learning approaches mainly suggested attack vectors rather than executing them or producing comprehensive reports.Source: §I · p. 1
- AutoPentester uses five modules to plan from prior findings, generate commands, execute CLI tools, verify results, and prevent repetitive loops.The modules are Strategy Analyzer, RAG-based Generator, Agent-Computer Interface, Results Verifier, and Repetition Identifier.Source: §I · pp. 1–2
- Given a target IP, AutoPentester iteratively performs reconnaissance, scanning, vulnerability assessment, and exploitation, then generates a report.The evaluation uses Hack-The-Box machines and custom-built vulnerable VMs.Source: §I · p. 2
- 27.0% higher subtask completion rate and 39.5% higher vulnerability coverage than PentestGPT are reported for AutoPentester.The evaluation spans multiple pentesting and threat-assessment tasks.Source: §I · p. 2
- Subtask completion improves by 20.0%; the Repetition Identifier reduces looping by 90.5%, and the Results Verifier decreases incomplete commands by 80.1%.Source: §I · p. 2
- The paper reports 3.93/5 average professional-survey score for AutoPentester, a 19.8% advantage over PentestGPT.Participants also believed it could save time in initial pentesting and suggested red-team use.Source: §I · p. 2
II. RELATED WORK
Prior automated-pentesting work ranges from attack-path selection to LLM-based frameworks, but the surveyed approaches often cover only part of the process or retain substantial human involvement. The authors position AutoPentester against gaps in automation, scope, reporting, and professional evaluation.
Machine Learning-based methods
- Earlier reinforcement-learning methods selected attack paths or network routes, while some approaches depended on prior application knowledge or omitted software vulnerability assessment.Hu et al. and HA-DRL [10] focused on attack-vector selection; NIG-AP focused on network information gathering, and Casola et al. targeted cloud applications.Source: §II · pp. 2–3
LLM-based methods
- ScriptKiddie [23] generated high-level steps for human execution, while PentestGPT dynamically planned but required operators to run commands and conduct security research.Source: §II · p. 3
- AutoAttacker [13] could execute generated commands through Metasploit but was limited to that framework and evaluated only on Metasploitable II.PenHeal is described as having comparable limitations.Source: §II · p. 3
II. RELATED WORK
- The authors characterize existing frameworks as limited in automation and use cases, dependent on human expertise, and lacking report generation and industry-professional testing.Source: §II · p. 3
III. AUTOPENTESTER FRAMEWORK
AutoPentester mirrors an iterative pentesting workflow: it interprets prior tool outputs, updates a findings-based attack plan, and selects the next action. Its framework uses agents to summarize, strategize, generate commands, verify results, and report findings.
III. AUTOPENTESTER FRAMEWORK
- AutoPentester executes one task per iteration and adapts its attack strategy based on prior outcomes, following the described human pentester workflow.It generally begins with an nmap scan, then assesses and exploits identified services.Source: §III · p. 3
A. Summarizer Agent
- The Summarizer splits verified tool output into 6,000-character chunks with 500-character overlaps, summarizes each, and merges the summaries for the Strategy Analyzer.The approach addresses outputs that may exceed LLM input limits.Source: A. Summarizer Agent · p. 3
B. Strategy Analyzer Agent
- The Strategy Analyzer stores workflow steps and findings as PTT attributes, then uses findings-oriented chain-of-thought reasoning to update the strategy and choose the next step.For example, a scan finding of vsftpd 2.3.4 leads to vulnerability assessment of that version and possible Metasploit exploit testing.Source: B. Strategy Analyzer Agent · pp. 3–4
- The PTT is saved as a text file and supplied at each iteration to help the Strategy Analyzer retain focus across longer runs.The authors say this supports more iterations without forgetting the attack environment or hallucinating.Source: B. Strategy Analyzer Agent · p. 4
C. Repetition Identifier Module
AutoPentester checks proposed steps for repetition and uses retrieval-supported command generation to carry out the Strategy Analyzer’s selected step. The Generator also receives information about the attack machine to produce executable commands.
C. Repetition Identifier Module
- The Repetition Identifier checks whether a proposed step resembles an earlier one and offers the operator options when it detects a repetition.The options include continuing with a different path, exiting, using interactive mode, or providing general input.Source: C. Repetition Identifier Module · p. 4
- A cosine-similarity score below the empirically chosen 0.15 threshold marks a step as a repetition.Steps are represented as vector embeddings of descriptions of the service, exploitation method, and tool.Source: C. Repetition Identifier Module · p. 4
D. Generator Agent
- The Generator uses retrieval-augmented generation to retrieve relevant attack knowledge and combines it with machine-specific information to create executable tool commands.The machine information includes paths to files used by tools and the local IP address.Source: D. Generator Agent · p. 4
- The retrieval database uses 500-character chunks and cosine similarity to return the ten most relevant chunks for a selected step.The retrieved chunks are supplied to the Generator as supporting information for command generation.Source: D. Generator Agent · p. 5
E. Agent Computer Interface (ACI)
The Agent-Computer Interface extracts and runs commands for security tools, while the Results Verifier checks and can refine tool commands. The Report Generator uses a human-readable log to produce a CSV summary of findings.
E. Agent Computer Interface (ACI)
- The ACI extracts commands from Generator output, maps them to the relevant tool inputs, and invokes the corresponding Python function.It uses subprocess for general CLI tools and pexpect for interactive CLIs; its design can be extended to security tools on static or interactive CLIs.Source: E. Agent Computer Interface (ACI) · p. 5
F. Results Verifier Agent
- When tool output is not as expected, the Results Verifier reasons over the commands and outcomes to refine commands before forwarding valid results to the Summarizer.The paper gives filtered ports as an example where adjusted nmap commands may reveal open ports.Source: F. Results Verifier Agent · p. 5
G. Report Generator Agent
- When pentesting ends, the Report Generator converts the human-readable process log into a CSV summary of findings.The listed fields include CVE number, CVSS score, risk level, vulnerability details, remediation, host information, and VPR.Source: G. Report Generator Agent · pp. 5–6
IV. EXPERIMENT SETUP
The evaluation uses installed security tools and two kinds of targets: Hack The Box machines for strategic pentesting and custom vulnerable VMs for vulnerability and threat assessment. PentestGPT is the baseline, with picoCTF excluded from the comparison.
IV. EXPERIMENT SETUP
- The framework was installed in a Linux VM with ten security tools, including Nmap, Metasploit, Netcat, Nikto, and Sqlmap.The selected tools were based on prior work and domain knowledge about commonly used pentesting tools.Source: §IV · p. 6
A. Target Machines
- Ten HTB machines—six easy and four intermediate—were used to assess strategic pentesting, including five machines from PentestGPT’s study and five additional unseen machines.The machines cover subtasks extracted from official write-ups and grouped into nine categories.Source: A. Target Machines · p. 6
- Four custom VMs containing OWASP Top 10 vulnerabilities were designed to assess vulnerability and threat analysis, with emphasis on identifying vulnerabilities and covering the attack surface.VM4 uses Metasploitable II.Source: A. Target Machines · p. 6
B. PentestGPT Baseline
- PentestGPT was selected as the baseline because its purpose was similar and its code was public; picoCTF testing was omitted because reported performance could require substantial operator guidance.For XtraORdinary, PentestGPT did not generate a solution without significant assistance on the approach and task breakdown.Source: B. PentestGPT Baseline · p. 6
C. Performance Metrics
The evaluation defines quantitative metrics for task coverage, tool efficiency, and human intervention, and pairs these with a qualitative study of professional reviews. Quantitative metrics were manually checked, while volunteers assessed reports and tool recordings.
C. Performance Metrics
- Subtask completion measures the percentage of machine subtasks completed, based on requirements extracted from official HTB write-ups.The evaluation also counts services covered and steps or repetitive loops used.Source: C. Performance Metrics · p. 6
- Human interaction counts steps requiring intervention, including running security-tool commands, correcting incomplete commands, and observing visual results.Incomplete commands are also counted separately as a quantitative metric.Source: C. Performance Metrics · pp. 6–7
- Vulnerability coverage counts vulnerabilities covered and is used only in custom-VM experiments, where the total vulnerabilities are known.The authors manually counted metrics from logs, cross-checked counts independently, and averaged three runs per target machine.Source: C. Performance Metrics · p. 7
- The qualitative study recruited ten cybersecurity professionals with more than five years of experience to review reports and tool recordings.The group comprised seven pentesters and three other cybersecurity professionals; the questionnaire included 13 MCQs and 15 short-answer questions.Source: C. Performance Metrics · p. 7
- Review materials included reports from two HTB machines and two VMs, with questions addressing attack strategies, vulnerability coverage, reporting, and shared aspects of attack-surface coverage and efficiency.The study also asked about disclosed information and progression of attack steps.Source: C. Performance Metrics · p. 7
D. Choice of LLM
AutoPentester can use any API-providing LLM, whose cybersecurity knowledge may affect performance. The authors selected GPT-4-turbo for subsequent experiments after testing three LLMs on Hack The Box machines.
D. Choice of LLM
- AutoPentester can integrate with any LLM that provides an API, but cybersecurity knowledge may differ across models and affect task performance.Source: D. Choice of LLM · p. 7
- GPT-4-turbo was selected for the remaining experiments because it achieved the highest subtask completion percentage among the three tested LLMs.The comparison tested Gemini-2.0flash, GPT-3.5-turbo, and GPT-4-turbo on Hack The Box machines.Source: D. Choice of LLM · p. 7
- Table III compares LLM performance and marks Linux as L and Windows as W.Source: Table III · p. 8
V. RESULTS
Across the reported benchmarks, AutoPentester outperforms PentestGPT on key performance measures while using fewer steps and requiring fewer human interactions. The results also report its vulnerability coverage on custom VMs and its runtime and cost trade-offs on Hack The Box machines.
A. Quantitative Analysis
- 27.0% better average subtask completion (59.92% vs 47.18%) accompanied approximately 18.7% fewer steps than PentestGPT on Hack The Box machines.AutoPentester outperformed PentestGPT in 49 scenarios, matched it in seven, and underperformed in four.Source: A. Quantitative Analysis · p. 8, Table IV · p. 9
- 85.7% fewer loops per machine for AutoPentester than PentestGPT, which averaged 2.1 loops compared with AutoPentester’s 0.3.AutoPentester averaged 1.13 human interactions compared with PentestGPT’s 15.36.Source: A. Quantitative Analysis · p. 8, Table IV · p. 9
- 97.7% fewer incomplete commands per machine for AutoPentester than PentestGPT, averaging 0.1 versus 4.46.The authors attribute command completeness to RAG examples and the Results Verifier’s corrections.Source: A. Quantitative Analysis · p. 8, Table IV · p. 9
- 98.14% average vulnerability coverage on custom VMs compared with PentestGPT’s 70.37%, with fewer steps and almost zero human interactions and incomplete commands for AutoPentester.PentestGPT averaged 18.16 human interactions and 6.25 incomplete commands per VM.Source: A. Quantitative Analysis · p. 8, Table V · p. 9, Table VI · p. 9
- AutoPentester cost $3.86 more on average across Hack The Box machines and took 71.9% longer, while achieving 12.74% higher subtask completion than PentestGPT.The authors cite interactive tool waiting periods as one reason for longer runtime and note that automated runs can occur outside regular working hours.Source: A. Quantitative Analysis · p. 8
B. Qualitative Analysis
Professionals rated AutoPentester more positively overall, citing its attack-surface coverage, information gathering, workflow, and reports. They also saw its automation as useful for reducing manual effort in red-team drills, while preferring PentestGPT’s flexibility in some situations.
B. Qualitative Analysis
- Survey responses favored AutoPentester for attack-surface coverage, relevant information gathering, and the advancement of its pentesting steps.Source: B. Qualitative Analysis · p. 9
- Professionals rated AutoPentester around 4 across questions, above PentestGPT’s mean of 3.3, but rated PentestGPT slightly higher for alignment with human pentesters.The survey asked about attack-surface coverage, information, strategy, efficiency, remediation, clarity, and time savings.Source: B. Qualitative Analysis · pp. 9–10, Figure 5 · p. 10
- Participants praised AutoPentester’s structured workflow and clear remediation, and noted that automation could reduce manual effort in red-team drills.They also valued executive summaries and impact assessments for communicating technical risks to non-technical stakeholders.Source: B. Qualitative Analysis · p. 10
- PentestGPT was viewed as offering more manual flexibility, but its workflow was also described as relying heavily on user command and tool selection.Source: B. Qualitative Analysis · p. 10
- The word-cloud analysis associates more green-shaded words with AutoPentester feedback, indicating more positive sentiment than for PentestGPT.Font size represents word frequency and color represents sentiment context.Source: B. Qualitative Analysis · p. 10
C. Ablation Study
The ablation study evaluates how AutoPentester’s modules contribute to performance. Its results identify gains associated with strategy reasoning, command guidance, loop reduction, and command verification.
C. Ablation Study
- The ablation baseline comprised a Summarizer, Analyzer, and Generator; the study then replaced the Analyzer with the Strategy Analyzer and added RAG, Repetition Identifier, and Results Verifier variants.RAG plus the Repetition Identifier had the highest performance among the two-module combinations reported.Source: C. Ablation Study · p. 10
- The reasoning-based Strategy Analyzer improved subtask completion by 25.0% over the baseline by using previous findings to derive strategies.Source: C. Ablation Study · p. 10
- Adding RAG gave the Generator relevant guidance for precise commands, including correct target IPs, port numbers, and file paths.Source: C. Ablation Study · p. 10
- Adding the Repetition Identifier reduced loops per step by 90.5%, from 0.48 to 0.05.Source: C. Ablation Study · p. 10
- Adding the Results Verifier reduced the incomplete-command ratio by 80.14%, from 0.43 to 0.13.Source: C. Ablation Study · p. 10
D. Failure cases
AutoPentester outperforms PentestGPT on reported benchmark metrics, while failure cases expose strategy and web-application gaps. The discussion also identifies operating constraints and future research directions.
D. Failure cases
- Only Lame and Bashed were fully completed; AutoPentester failed at a subtask on the other eight of 10 HTB machines.Failures included four strategy-identification misses, two failures from not finding required exploit information, and two web-application subtasks where curl use lost information in large responses.Source: D. Failure cases · p. 11
VI. DISCUSSION AND CONCLUDING REMARKS
- 27.0% higher subtask completion rate and 39.5% greater vulnerability coverage, with 18.7% fewer steps and 92.6% less human intervention than PentestGPT.The discussion also reports that experts caution automated exploitation requires careful handling in production environments.Source: §VI · p. 11
- The user survey highlighted AutoPentester’s structured automation approach as suitable for enterprise assessments and large-scale red-team drills, while experts cautioned about production exploitation.Its CLI interaction can let testers multitask.Source: §VI · p. 11
- Fully automated interaction with GUI-based interfaces is challenging with CLI tools such as curl, and RAG-focused commands can miss corner cases when the knowledge base is not current.The user study would also have benefited from a larger sample, though recruiting industry-professional volunteers is difficult.Source: §VI · pp. 11–12
- Future work could fine-tune LLMs for pentesting strategies using RL, RLHF, or DPO, and add GUI interaction and web-focused tools such as ZAP and OpenVAS.The paper notes that current LLMs cannot identify complex strategies in penetration testing tasks.Source: §VI · p. 12