PaperlayerView exact revision on arXiv

Source-linked AI summary

PentestAgent: Incorporating LLM Agents to Automated Penetration Testing

Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, Wei Ruan

arXiv:2411.05185v3cs.CR

TL;DR

LLM-based penetration testing is limited by gaps in pentesting knowledge and automation. PentestAgent combines a multi-agent architecture with LLM techniques to automate and validate testing actions, and achieves strong task-completion performance and overall efficiency on its benchmark.

  • Problem

    LLM-based penetration testing requires substantial manual effort to find vulnerability information, configure tools, and select exploits, while existing frameworks have limited knowledge and automation.

    Source: §2.2 · p. 3, §6 · p. 12

  • Method

    PentestAgent uses task-specific LLM agents, RAG, and validation and debugging mechanisms to automate penetration testing actions.

    Source: §1 · p. 2

  • Results

    PentestAgent achieves strong performance in task completion and overall efficiency on a benchmark based on VulHub environments and HackTheBox CTF challenges.

    Source: §6 · p. 12

  • Takeaways & Limitations

    PentestAgent can identify and validate exposed vulnerabilities such as SSRF, providing a starting point for more advanced exploitation in complex scenarios.

    Source: §5.2 · p. 12

    PentestAgent focuses on exploiting individual vulnerable applications and services; sophisticated attack planning that combines vulnerabilities is beyond this paper’s scope.

    Source: §5.2 · p. 12

Abstract

from arXiv · show

Penetration testing is a critical technique for identifying security vulnerabilities, traditionally performed manually by skilled security specialists. This complex process involves gathering information about the target system, identifying entry points, exploiting the system, and reporting findings. Despite its effectiveness, manual penetration testing is time-consuming and expensive, often requiring significant expertise and resources that many organizations cannot afford. While automated penetration testing methods have been proposed, they often fall short in real-world applications due to limitations in flexibility, adaptability, and implementation. Recent advancements in large language models (LLMs) offer new opportunities for enhancing penetration testing through increased intelligence and automation. However, current LLM-based approaches still face significant challenges, including limited penetration testing knowledge and a lack of comprehensive automation capabilities. To address these gaps, we propose PentestAgent, a novel LLM-based automated penetration testing framework that leverages the power of LLMs and various LLM-based techniques like Retrieval Augmented Generation (RAG) to enhance penetration testing knowledge and automate various tasks. Our framework leverages multi-agent collaboration to automate intelligence gathering, vulnerability analysis, and exploitation stages, reducing manual intervention. We evaluate PentestAgent using a comprehensive benchmark, demonstrating superior performance in task completion and overall efficiency. This work significantly advances the practical applicability of automated penetration testing systems.

1 Introduction

Manual penetration testing is costly and expertise-intensive, while earlier automated approaches and current LLM-based systems retain gaps in implementation, penetration-testing knowledge, and automation. PentestAgent combines multi-agent design and RAG with a benchmark intended to evaluate automated penetration testing.

1 Introduction

  • Manual penetration testing requires skilled specialists and can take an average of 80 hours, with substantial outliers, making it costly for many organizations.The cited report gives an average of 80 hours and notes that some tests take several hundred hours.Source: §1 · p. 1
  • Current LLM-based approaches have limited penetration-testing knowledge and insufficient automation for adapting, validating, and debugging procedures across environments.The stated knowledge gap includes limited coverage of techniques and an outdated action space.Source: §1 · p. 2
  • PentestAgent combines multi-agent design and RAG with validation and debugging to update its knowledge base and automate penetration-testing tasks.Its agents can use supplementary data during response synthesis and customize toolsets across tasks.Source: §1 · p. 2
  • Table 1 compares LLM-based pentesting systems, with PentestAgent described as fully automated and having a larger state and action space, online search augmentation, and debugging capabilities.The comparison contrasts it with AutoAttacker and PentestGPT, described as relying on manual or limited approaches.Source: §1 · p. 2, Table 1 · p. 2
  • The proposed benchmark uses VulHub vulnerable Docker environments and HackTheBox CTF challenges to provide a practical evaluation framework across difficulty levels and vulnerabilities.The authors identify this benchmark as addressing a gap in current research.Source: §1 · p. 2
  • PentestAgent’s benchmark evaluation reports superior performance in completing the full penetration-testing process and individual tasks.The benchmark is based on VulHub vulnerable Docker environments and spans multiple difficulty levels and common weaknesses and vulnerabilities.Source: §1 · p. 2

2 Background and Related Work

The background describes penetration testing as a multi-stage process and reviews gaps in existing tools and LLM-based approaches. It details challenges in knowledge, memory, and workflow integration, then surveys LLM techniques proposed to address them.

2.1 Penetration Testing

  • External penetration testing covers internet-exposed assets, but coordinating specialized tools across intelligence gathering, vulnerability analysis, and exploitation still requires expertise and manual effort.The paper focuses on automating web penetration testing, which it identifies as an underexplored external-assessment area.Source: §2.1 · pp. 2–3

2.2 Challenges of Applying LLM to Pentesting

Limited Pentesting Knowledge Example

  • Limited pentesting knowledge leaves users to find CVEs, analyze vulnerabilities, set up tools, and select and configure exploits themselves.The paper presents these steps as requiring penetration-testing expertise and significant human effort.Source: §2.2 · p. 3

Repetition of Tasks Example

Loss of Context Example

  • Context loss can undermine sequential exploitation, while weak working-memory transitions can restart exploit execution and disrupt workflow continuity.The paper also notes that current LLMs do not inherently support working-memory management within and between sessions.Source: Repetition of Tasks Example · p. 4, Loss of Context Example · p. 4

2.3 LLM Techniques for Overcoming Challenges

  • RAG retrieves indexed external information for response synthesis and can provide long-term memories that are dynamically queried and stored.The section also describes CoT, role-playing, self-reflection, and structured outputs as techniques for reasoning, output control, and workflow integration.Source: §2.3 · pp. 4–5

3 System Design

PentestAgent coordinates reconnaissance, search, planning, and execution agents across intelligence gathering, vulnerability analysis, and exploitation. RAG, iterative agent workflows, and stored intermediate knowledge support this process.

3.1 System Overview

  • The four agents coordinate the three testing stages, from target reconnaissance and vulnerability analysis through exploit execution and report generation.Environmental information and execution history are stored in databases for later stages and reporting.Source: §3.1 · p. 5

3.2 Reconnaissance Agent

  • The reconnaissance agent iteratively generates commands and analyzes results until stopping, then summarizes findings and stores them in a database.Expert-defined workflows and RAG provide external knowledge for selecting procedures and tools.Source: §3.2 · p. 6

Reconnaissance System Message (Simplified)

  • Reconnaissance prompts use role-play, chain-of-thought, RAG, and structured output to guide task decomposition, tool use, stopping, and handoff.Structured responses support subsequent pipeline steps; RAG also enables tool-document retrieval and persistent storage of collected environmental information.Source: Reconnaissance System Message… · p. 6

3.3 Search Agent

  • The search agent runs two hierarchical online-search rounds: first for attack surfaces, then for procedure-level attack knowledge, storing each separately.Search functions include general web, vulnerability-database, and exploit-repository searches; RAG-based question answering extracts key information from raw results into concise databases.Source: §3.3 · p. 6

3.4 Planning Agent

  • The planning agent uses RAG and the pentesting knowledge base to identify attack surfaces relevant to detected services, then generates a list of exploits.Source: §3.4 · p. 7

Execution Preparation System Message (Simplified)

  • Before execution, the agent identifies exploit parameters and required information, retrieves context with RAG, and queries the environmental database to fill those requirements.It uses chain-of-thought and structured JSON output to prepare the transition to exploitation.Source: §3.5 · pp. 7–8, CoT · p. 8

Execution Exploit System Message (Simplified)

  • The execution agent uses RAG to break exploit plans into step-by-step instructions and iteratively executes them, applying self-reflection to analyze and fix errors.It records error history to avoid repeating errors.Source: Execution Exploit System Message… · p. 8

4 Evaluation

The evaluation benchmarks PentestAgent on vulnerable environments and measures end-to-end success, stage completion, time, and API cost. It reports strong overall automation, with performance and consistency varying by model, task difficulty, and setting.

4.1.1 Benchmark Dataset; 4.1.3 Environment setup

  • The benchmark includes 67 VulHub targets across 32 CWE categories and 11 HackTheBox challenges; VulHub difficulty and realism are assessed using CVSS and EPSS.VulHub provides pre-built vulnerable Docker environments; attacker and victim VMs were connected through NAT.Source: §4.1.1 · pp. 8–9, §4.1.3 · p. 9

4.1.2 Metric

  • Effectiveness requires all three stages to finish automatically with a functional exploit; stage metrics assess identification, exploit discovery, and execution, while efficiency tracks cycle time and API cost.For HackTheBox, success means obtaining initial access; stage completion is evaluated assuming preceding stages succeeded.Source: §4.1.2 · p. 9

4.2 Effectiveness of the Entire Framework

  • 74.2% overall success for GPT-4 versus 60.6% for GPT-3.5; both exceeded 60% on automated penetration-testing tasks.The success-rate comparison is reported in Figure 7.Source: Figure 7 · p. 9, §4.2 · p. 9

4.3 Completion level of Penetration Testing Stages

  • GPT-4 completed easy-task intelligence gathering and vulnerability analysis, with 81.8% exploitation; its hard-task intelligence-gathering completion was 50%.GPT-3.5 completed no hard-task intelligence gathering or exploitation; its easy-task completion was 92% for intelligence gathering, 96% for vulnerability analysis, and 72% for exploitation.Source: §4.3 · pp. 9–10, Figure 8 · p. 10

4.4 Ablation Study

  • Vulnerability analysis reached 100% completion across tested backbones; o1-mini led exploitation, while GPT-3.5 was fastest and most cost-effective but weaker at exploitation.The ablation used six easy, five medium, and two hard VulHub targets because of hardware limits; o1-mini took longer during vulnerability analysis, while Llama 3.1-8B-Instruct had no cost but higher time overhead and lower exploitation performance.Source: §4.4 · p. 10

4.5 Practicality Study; 4.6 Comparison with PentestGPT

  • PentestAgent exploited six HackTheBox machines; against PentestGPT with GPT-3.5, it took 220 seconds versus 1199 for intelligence gathering and 172 versus 364 for exploitation.PentestGPT was slightly faster in vulnerability analysis, but human involvement slowed its other stages.Source: §4.5 · p. 10, §4.6 · p. 11

4.7 Failure Analysis

  • Failures include missed web-component enumeration and exploitation blocked by extra expertise, user actions, or hallucinations; zero temperature, retry limits, and alternate exploits mitigate hallucinations.The paper also describes stop conditions intended to prevent loops or unintended actions.Source: §4.7 · pp. 11–12

5 Discussion

PentestAgent is contrasted with frameworks focused on post-breach exploitation or human-guided CTFs: it covers multiple penetration-testing stages autonomously. Its scope centers on individual vulnerable applications and services, while sophisticated multi-vulnerability attack strategies remain beyond the paper’s scope.

5.1 Comparison with Existing Frameworks

  • PentestAgent covers reconnaissance, vulnerability analysis, and exploitation, whereas AutoAttacker automates only the post-breach exploitation phase.The paper argues that early-stage tasks also matter in real-world security assessment.Source: §5.1 · p. 12
  • Enigma’s interactive CTF interfaces still rely on human intervention, while PentestAgent is designed to operate autonomously across diverse attack stages.Enigma focuses primarily on crypto and reverse engineering.Source: §5.1 · p. 12
  • PentestAgent offers an end-to-end automated solution, unlike AutoAttacker’s post-breach focus and Enigma’s human-in-the-loop CTF support.AutoAttacker targets exploitation after compromise; Enigma primarily targets crypto and reverse-engineering challenges.Source: §5.1 · p. 12

5.2 Limitations on Performing Sophisticated Pentesting

  • PentestAgent focuses on individual vulnerable applications and services; complex scenarios may require chaining vulnerabilities, such as SSRF to reach an internal application and root privileges.The paper says its system can identify and validate exposed vulnerabilities such as SSRF as a starting point for further exploitation, while sophisticated strategies remain outside its scope.Source: §5.2 · p. 12

6 Conclusion

PentestAgent combines multi-agent architecture with RAG and CoT to address limited pentesting knowledge and insufficient automation. Its benchmark evaluation reports strong task-completion performance and overall efficiency, and the appendix describes prompts for structured reconnaissance summaries.

6 Conclusion

  • PentestAgent combines a multi-agent architecture with RAG and CoT to improve knowledge integration and automation in penetration testing.The framework is presented as addressing limited pentesting knowledge and insufficient automation.Source: §6 · p. 12
  • The VulHub vulnerable Docker and HackTheBox CTF benchmark reports strong task completion and overall efficiency for PentestAgent.The passage provides no numerical values for these outcomes.Source: §6 · p. 12

A Prompts

Reconnaissance Summary Prompt (Simplified)

  • The appendix’s reconnaissance prompt specifies a structured summary format and example to guide relevant outputs and reduce hallucination.The prompt requests valid JSON; a following prompt summarizes search results into structured output for parsing and storage.Source: A Prompts · p. 13, Reconnaissance Summary Prompt… · pp. 13–14

Search Results Summary Prompt

The appendix prompts structure the extraction and transfer of information about target applications, vulnerabilities, exploit repositories, and execution environments.

Search Results Summary Prompt

  • The search-results prompt requests CVE numbers, URLs, keywords, and applicable versions, with few-shot examples added because the LLM was unfamiliar with software versioning.The results are requested in valid JSON for structured output.Source: Search Results Summary Prompt · p. 14

Exploit Procedure Analysis Prompt (Simplified)

  • The exploit-procedure prompt uses RAG and CoT to summarize repository contents, exploit effects, applicable versions, and execution requirements in structured JSON.It asks whether a repository contains an exploit for the target service or app and what effect it has.Source: Exploit Procedure Analysis Prompt… · p. 14

Attack Surface Suggestion Prompt (Simplified)

  • The attack-surface prompt ranks vulnerabilities for an application and version by confidence, checking version applicability and vulnerability type.It asks for supporting details and reasons in valid JSON.Source: Attack Surface Suggestion Prompt… · p. 14

Exploit Suggestion Prompt (Simplified)

Execution Information Query Prompt (Simplified)

B.1 VulHub Benchmark Construction

The benchmark assigns vulnerability difficulty using CVSS exploitability scores and selects among multiple CVEs using EPSS scores. Exploitability-score cutoffs distinguish easy, medium, and hard cases.

B.1 VulHub Benchmark Construction

  • Most exploitability scores are above 3.0, and scores of 2.0 and 3.0 are natural cutoffs for easy, medium, and hard difficulty levels.Figure 11 shows the distribution of exploitability scores.Source: §B.1 · p. 14, Figure 11 · p. 15
  • CVSS 3.x exploitability scores serve as the difficulty reference because the score reflects how easily and by what technical means a vulnerability can be exploited.A higher exploitability score indicates an easier vulnerability to exploit.Source: §B.1 · p. 14
  • When an application or service has multiple CVEs, the benchmark selects a CVE using its EPSS score, which indicates likelihood of exploitation in the wild.A higher EPSS score indicates greater likelihood of real-world exploitation; Figure 12 is captioned as showing EPSS-score coverage.Source: §B.1 · p. 14, Figure 12 · p. 15

B.2 HackTheBox Benchmark Construction

The HackTheBox challenges were selected to cover operating systems, vulnerability types, and difficulty levels, supporting assessment across varied penetration-testing scenarios.

B.2 HackTheBox Benchmark Construction

  • The challenge set was chosen to represent real-world scenarios through operating-system diversity, vulnerability relevance, and difficulty levels.The selection is intended to assess automation, performance across vulnerability types, and effectiveness in progressively complex scenarios.Source: §B.2 · p. 15
  • The selected machines span Linux and Windows and include well-known vulnerabilities from different years, including EternalBlue and MS08-067.Blue features EternalBlue (CVE-2017-0144), while Legacy includes MS08-067 (CVE-2008-4250).Source: §B.2 · p. 15
  • The benchmark includes easy, medium, and hard challenges, with harder examples requiring deeper reconnaissance, multi-step attacks, and advanced reasoning.Lame and Optimum test fundamental exploitation; Stratosphere and Reel represent medium and hard scenarios.Source: §B.2 · p. 15

B.3 Benchmark Coverage

The benchmark uses known vulnerabilities selected for high real-world exploit likelihood, while acknowledging the difficulty of evaluating on zero-day or one-day vulnerable environments.

B.3 Benchmark Coverage

  • Tables 3 and 4 report PentestAgent’s and PentestGPT’s performance, respectively, across HackTheBox CTF challenges.Source: Table 3 · p. 16, Table 4 · p. 16
  • The benchmark uses known vulnerabilities, and its mean EPSS score is 79.58 while its median is 97.19.The authors state that open datasets containing zero-day or even one-day vulnerable environments remain challenging to find.Source: §B.3 · pp. 15–16
  • The authors selected environments using EPSS scores to represent vulnerabilities likely to exist and be exploitable in practical settings.They frame the known-vulnerability benchmark as a realistic context for assessing risks from vulnerable and outdated components.Source: §B.3 · pp. 15–16

C Additional Evaluation Results

On VulHub targets, PentestAgent completed more tasks than PentestGPT across all three penetration-testing stages. It also spent less time in intelligence gathering and exploitation, but more time in vulnerability analysis.

C Additional Evaluation Results

  • 80% versus 10% completion in intelligence gathering, 100% versus 10% in vulnerability analysis, and 70% versus 30% in exploitation favored PentestAgent over PentestGPT.Figure 15 compares completion levels on VulHub targets.Source: Figure 15 · p. 16, C Additional Evaluation Results · p. 16
  • 212.9 seconds versus 658.7 seconds in intelligence gathering and 58.6 seconds versus 283.5 seconds in exploitation favored PentestAgent over PentestGPT.Figure 16 compares average time spent on VulHub targets.Source: Figure 16 · p. 16, C Additional Evaluation Results · p. 16
  • 698.8 seconds versus 433.5 seconds in vulnerability analysis meant PentestAgent took longer than PentestGPT, with the extra time linked to higher success.Source: C Additional Evaluation Results · p. 17
  • The authors characterize PentestAgent as more effective and efficient overall, while noting the vulnerability-analysis time trade-off.They connect its lower intelligence-gathering and exploitation overhead with scalability and practical use.Source: C Additional Evaluation Results · p. 17

Related papers

Loading 2411.05185v3…