PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes a Good LLM Agent for Real-world Penetration Testing?

Gelei Deng, Yi Liu, Yuekang Li, Ruozhao Yang, Xiaofei Xie, Jie Zhang, Han Qiu, Tianwei Zhang

arXiv:2602.17622v1cs.CRcs.SE

TL;DR

LLM penetration-testing agents vary widely because capability gaps and persistent planning barriers affect performance. This paper analyzes these failures, introduces PENTESTGPT V2 with difficulty-aware planning, and reports stronger benchmark and enterprise results than prior systems.

  • Problem

    Penetration testing faces a global cybersecurity workforce shortfall, while existing LLM agents exhibit capability gaps and planning failures that limit reliable automation.

  • Method

    The paper analyzes 28 systems and five implementations, then combines a Tool and Skill Layer with Task Difficulty Assessment integrated into Evidence-Guided Attack Tree Search.

  • Results

    91% task completion on CTF benchmarks and 4 of 5 GOAD hosts compromised, versus 2 by prior systems.

  • Takeaways & Limitations

    The findings distinguish engineering-addressable capability gaps from complexity barriers requiring architectural innovation in LLM penetration-testing agents.

  • Takeaways & Limitations

    The evaluation omits binary exploitation, mobile security, and cloud-specific attacks, while retired machines with public walkthroughs may inflate absolute benchmark results.

Abstract

from arXiv · show

LLM-based agents show promise for automating penetration testing, yet reported performance varies widely across systems and benchmarks. We analyze 28 LLM-based penetration testing systems and evaluate five representative implementations across three benchmarks of increasing complexity. Our analysis reveals two distinct failure modes: Type A failures stem from capability gaps (missing tools, inadequate prompts) that engineering readily addresses, while Type B failures persist regardless of tooling due to planning and state management limitations. We show that Type B failures share a root cause that is largely invariant to the underlying LLM: agents lack real-time task difficulty estimation. As a result, agents misallocate effort, over-commit to low-value branches, and exhaust context before completing attack chains. Based on this insight, we present Excalibur, a penetration testing agent that couples strong tooling with difficulty-aware planning. A Tool and Skill Layer eliminates Type A failures through typed interfaces and retrieval-augmented knowledge. A Task Difficulty Assessment (TDA) mechanism addresses Type B failures by estimating tractability through four measurable dimensions (horizon estimation, evidence confidence, context load, and historical success) and uses these estimates to guide exploration-exploitation decisions within an Evidence-Guided Attack Tree Search (EGATS) framework. Excalibur achieves up to 91% task completion on CTF benchmarks with frontier models (39 to 49% relative improvement over baselines) and compromises 4 of 5 hosts on the GOAD Active Directory environment versus 2 by prior systems. These results show that difficulty-aware planning yields consistent end-to-end gains across models and addresses a limitation that model scaling alone does not eliminate.

1 Introduction

This paper investigates why LLM penetration-testing systems vary widely in performance and identifies capability gaps and complexity barriers as distinct failure modes. It presents PENTESTGPT V2, combining typed tools and skills with difficulty-aware planning, and reports gains across increasingly realistic benchmarks.

  • Motivation: Reported completion rates range from single digits under naive prompting to 40–80% with more sophisticated architectures, motivating analysis of the underlying performance differences.
  • Failure analysis: Analysis of 28 systems and five implementations identifies Type A capability gaps and Type B complexity barriers requiring different solutions.Type A failures involve missing tools and knowledge, whereas Type B failures persist because of planning and state-management limitations.
  • Failure analysis: Type B failures arise because agents cannot assess task difficulty in real time, causing premature branch commitment, inadequate exploitation transitions, and context forgetting.Difficulty assessment reduced Type B failures from 58% to 27%, while Type A failure rates remained unchanged.
  • System design: PENTESTGPT V2 addresses Type A failures with typed interfaces and skills, and Type B failures with TDA integrated into EGATS to guide attack-path decisions.TDA estimates tractability using horizon, evidence confidence, context load, and historical success; the system also maintains structured external state.
  • Evaluation: 91% peak task completion on XBOW, 12/13 machines rooted on the PentestGPT Benchmark, and 4/5 GOAD hosts compromised, outperforming prior systems across three benchmarks.With Claude Opus 4.5, XBOW mean completion was 89% versus 61% for the best baseline; prior systems compromised at most 2 GOAD hosts.

2 Background

Penetration testing follows phased attack workflows and requires sequential tool use, adaptive strategy, and decision-making across heterogeneous outputs. Benchmarks enable standardized evaluation but only approximate real-world engagements, while LLM agents face challenges shared with other agentic domains.

  • Penetration testing workflow: Penetration testing proceeds through reconnaissance, enumeration, exploitation, and post-exploitation, typically exploring attack surfaces broadly before pursuing promising paths deeply.
  • Benchmarking: CTF, VulnHub, and HTB benchmarks provide standardized vulnerable environments spanning web, cryptographic, binary, and machine-based challenges.
  • Benchmarking: GOAD is the closest current benchmark approximation to enterprise environments, requiring chained attacks across multi-domain Windows networks while omitting social engineering and time pressure.
  • Benchmarking: Benchmark results measure specific technical capabilities rather than predicting overall real-world penetration-testing effectiveness.
  • LLM-based agents: LLM penetration-testing agents combine tool use and agentic scaffolding to execute reconnaissance, exploitation, and post-exploitation while maintaining strategies across many steps.These challenges resemble limitations reported in software engineering and web navigation, suggesting they are not unique to penetration testing.

3 Understanding LLM Agent Failures

The analysis evaluates representative penetration-testing agents and finds that their apparent architectural advantages compress with stronger models, while persistent failures arise from difficulty assessment, planning, and state management. It distinguishes tooling-addressable capability gaps from complexity barriers that remain during long attack chains.

  • Agent Architecture Convergence: Existing systems’ performance gaps compress with stronger models, indicating that many architectural techniques address transient model limitations rather than persistent task challenges.On XBOW, the spread narrows from 27–39% with GPT-4o to 40–49% with GPT-5, while the PentestGPT Benchmark spread narrows from 4–6 to 7–8 machines.
  • Agent Architecture Convergence: Specialized AD capabilities improve Cochise’s GOAD performance but reduce generality, with lower XBOW and PentestGPT Benchmark results than VulnBot under GPT-4o.Cochise reports 34% on XBOW and 4/13 on the PentestGPT Benchmark, compared with VulnBot’s 39% and 6/13.
  • Two Distinct Failure Categories: Trace analysis identifies context forgetting, premature commitment, and exploration–exploitation imbalance, which can cascade into chain errors when agents lose state across attack phases.
  • Two Distinct Failure Categories: 79% of GOAD failures persist despite tooling improvements, whereas 68% of XBOW failures resolve with improved tooling as task depth increases.XBOW tasks typically require 1–3 exploitation steps, while GOAD attacks require 5–10 steps across multiple hosts.
  • Two Distinct Failure Categories: Type A failures are capability gaps addressable through missing tools and knowledge, whereas Type B failures are search-strategy and state-management failures that persist despite adequate capabilities.
  • Root Cause: Missing Difficulty Assessment: Type B failures share a root cause: agents cannot assess task difficulty in real time to estimate path length, judge evidence sufficiency, or monitor context consumption.This limitation causes agents to persist on unproductive branches, delay exploitation, and exhaust context before completing attack chains.
  • Implications: Effective systems must combine capability engineering with planning that assesses task difficulty in real time, because either approach alone leaves one failure category unresolved.

4 Design of PENTESTGPT V2

PENTESTGPT V2 addresses capability gaps and persistent planning failures with structured tooling, task-difficulty assessment, and evidence-guided attack-tree search. Its design uses measurable difficulty signals to choose reconnaissance or exploitation, prune intractable branches, and preserve attack-state information.

  • 4 Design of PENTESTGPT V2: PENTESTGPT V2 combines a Tool and Skill Layer for Type A failures with TDA-guided EGATS for Type B complexity barriers.The system also includes a memory subsystem for preserving state across attack phases.
  • 4.4 EGATS: The attack procedure initializes a target-rooted tree, selects goals with TDA, executes tool-mediated actions, and feeds results back into tree and memory state.Successful exploitation can spawn pivots, while branches can be pruned when difficulty and action-count conditions indicate intractability.
  • 4.2 Tool and Skill Layer: The Tool and Skill Layer improves XBOW completion from 54% to 68% on capability-limited tasks.Typed interfaces, skill composition, and retrieval-augmented knowledge reduce tool invocation errors that can cascade into attack failures.
  • 4.3 Task Difficulty Index: TDA estimates tractability from horizon, evidence confidence, context load, and historical success, combining them into a Task Difficulty Index.The index uses normalized horizon estimates and weights selected through validation, with completion varying within ±3% across tested weight configurations.
  • 4.3.2 Task Difficulty Index: TDI selects reconnaissance for difficult branches, exploitation for easy branches, and LLM-guided decisions for intermediate cases.The reported thresholds are TDI > 0.6 for exploration and TDI < 0.3 for exploitation.
  • 4.4 EGATS: EGATS integrates external attack-tree structure, evidence-based pruning, and TDA-guided mode selection to guide penetration-testing search.Its tree represents observations, hypotheses, and actions, while promise scores are updated from execution outcomes.

5 Evaluation

The evaluation compares PENTESTGPT V2 with prior systems across increasingly realistic benchmarks and finds consistent gains, while ablations and behavior analyses link those gains to tooling, difficulty-aware search, and memory. The remaining failure case shows that difficulty assessment cannot solve undocumented exploits requiring creative reasoning.

  • 5.1 Experimental Setup: 91% peak task completion on XBOW and 4 of 5 GOAD hosts compromised, versus 2 for prior systems, demonstrate PENTESTGPT V2’s strongest cross-benchmark gains.On the PentestGPT Benchmark, it roots 12 of 13 machines versus 9 for the best baseline.
  • 5.1 Experimental Setup: PENTESTGPT V2 roots 12 of 13 machines versus 9 for the best baseline and solves both Hard-rated targets where baselines stop at initial steps.The result is consistent across GPT-5.2 and Opus 4.5 thinking trials.
  • 5.1 Experimental Setup: PENTESTGPT V2 compromises 4 of 5 GOAD hosts versus at most 2 for baselines, doubling the compromise rate from 40% to 80%.The pattern holds across models and reasoning modes, including 3 hosts for Gemini 3 versus 1–2 for baselines.
  • 5.2 RQ1: Overall Performance: The Tool Layer adds 14 XBOW points but no GOAD improvement, while TDA-EGATS adds 9 XBOW points, 2 machines, and 1 GOAD host.These results separate capability improvements on short-horizon tasks from planning gains on harder environments.
  • 5.2 RQ1: Overall Performance: Memory adds 8 XBOW points, 1 machine, and 1 GOAD host, enabling credential persistence needed for the fourth GOAD compromise.The result connects explicit state management to extended attack campaigns.
  • 5.3 RQ2: Ablation Study: TDA-EGATS prunes 4.2 branches per machine and backtracks when success rates fall, unlike baselines that commit deeply to unproductive paths.PentestGPT explores 3.2 branches and reaches depth 12.4 before pivoting, versus 7.8 branches and depth 5.1 for PENTESTGPT V2.
  • 5.4.3 Failure Case: PlayerTwo: PENTESTGPT V2 fails on PlayerTwo because TDA treats difficult-but-tractable and novel creative exploitation similarly when documentation and model knowledge are absent.After three unsuccessful fuzzing attempts, the branch is pruned as apparently intractable.
  • 5.5 Resource Efficiency: PENTESTGPT V2 uses 23% fewer XBOW calls while achieving 39% higher success rates, and is 1.8× and 1.7× more cost-effective on XBOW and GOAD.GOAD uses 18% more calls but produces twice as many compromised hosts.

6 Discussion

The discussion bounds the findings by benchmark coverage, model and baseline choices, and persistent barriers involving creativity, adversarial environments, and long-horizon continuity. These boundaries show that current gains do not establish fully autonomous real-world penetration testing.

  • 6.1 Limitations and Threats to Validity: The evaluation omits binary exploitation, mobile security, and cloud-specific attacks, so its findings do not cover those challenge types.Binary exploitation is identified as requiring precise memory-layout reasoning not captured by the benchmarks.
  • 6.1 Limitations and Threats to Validity: Retired machines with public walkthroughs may inflate absolute benchmark results through data contamination, although the authors argue planning components target orthogonal challenges.Real-world engagements also include active defenses and novel vulnerability classes absent from historical benchmarks.
  • 6.1 Limitations and Threats to Validity: Results may interact differently with model families because Opus 4.5 reaches the highest XBOW performance, while future models may shift the current easy–hard boundary.The evaluation uses GPT-5.2, Claude-Opus-4.5, and Gemini-3.0-Pro.
  • 6.2 What Remains Hard: Novel exploitation remains a creativity barrier: systematic search helps difficult known tasks but not undocumented vulnerabilities requiring genuine invention.PlayerTwo illustrates this boundary when no documented exploitation pattern exists.
  • 6.2 What Remains Hard: Evidence grounding offers limited defense against honeypots and deceptive services that create convincing but false beliefs about the environment.Defenders can exploit predictable agent reasoning patterns, while agents lack equivalent meta-awareness of manipulation.
  • 6.2 What Remains Hard: EGATS and Memory improve within-session reasoning and state preservation, but cross-session campaigns still require hierarchical abstraction, goal decomposition, and progress monitoring.The temporal-scale barrier differs from simply processing longer contexts.

7 Conclusion

The conclusion frames PENTESTGPT V2 as a system that combines typed tooling with difficulty-aware search to address distinct capability and complexity failures. Its reported gains support difficulty-aware planning as an architectural contribution beyond model improvements alone.

  • 7 Conclusion: PENTESTGPT V2 separates Type A capability gaps from Type B complexity barriers and assigns engineering and architectural remedies accordingly.The Tool and Skill Layer addresses Type A failures, while TDA-EGATS addresses Type B failures.
  • 7 Conclusion: 91% CTF task completion and 4 of 5 GOAD hosts compromised, versus 2 for prior systems, summarize the main reported end-to-end gains.The conclusion reports a 49% improvement over baselines on CTF benchmarks.
  • 7 Conclusion: The ablations indicate that TDA-guided exploration contributes beyond attack-tree structure alone, producing value that model improvements cannot replicate.The system integrates typed interfaces, retrieval-augmented knowledge, TDA, EGATS, and memory.
  • 7 Conclusion: The Tool and Skill Layer exposes 38 security tools through typed interfaces specifying inputs, outputs, and preconditions or postconditions.Tool selection follows standard penetration-testing methodology and commonly used professional tools.
  • 7 Conclusion: Evidence confidence is computed along a root-to-current-node path from deterministically assigned node scores, excluding the initial root state.Tool outputs are parsed against expected patterns to determine evidence types automatically.

D Parameter Derivation and Validation

The appendix documents how PENTESTGPT V2’s hyperparameters were derived and tested for sensitivity.

  • D Parameter Derivation and Validation: The appendix provides hyperparameter derivation and sensitivity analysis for PENTESTGPT V2.

D.1 Validation Dataset

Hyperparameters are tuned on a held-out validation set of 30 retired HTB execution traces, separate from the evaluation benchmark and models.

  • D.1 Validation Dataset: 30 retired HTB execution traces form the held-out validation set for hyperparameter tuning.The traces come from machines retired during 2022–2023.
  • D.1 Validation Dataset: The validation set spans 10 Easy, 12 Medium, and 8 Hard machines across web, Linux privilege-escalation, and Windows/AD attacks.Its attack coverage includes 12 web-exploitation, 10 Linux privilege-escalation, and 8 Windows/AD machines.
  • D.1 Validation Dataset: GPT-4o is used for validation, while GPT-5.2, Opus 4.5, and Gemini 3 are reserved for evaluation.The sets are disjoint to avoid overlap with evaluation models.

D.2 TDI Weight Selection

TDI weights are selected by grid search on validation-set mean subtask completion, with performance remaining stable across a broad weight range.

  • D.2 TDI Weight Selection: Grid search evaluates mean subtask completion across validation-set traces, while threshold sensitivity is reported separately.Table 14 presents mode-selection threshold sensitivity rather than TDI weight selection.
  • D.2 TDI Weight Selection: Performance varies within ±3% across configurations with every TDI weight between 0.1 and 0.4.This indicates robustness to precise weight selection.
  • D.2 TDI Weight Selection: The selected weights assign 0.3 each to horizon and evidence confidence, and 0.2 each to context load and success rate.Horizon and evidence confidence are treated as primary difficulty signals, with context load and success rate providing secondary modulation.

D.4 Pruning Parameters

Pruning and difficulty-penalty parameters balance avoiding tractable-path errors against wasting effort on intractable branches.

  • D.4 Pruning Parameters: The selected pruning threshold is 0.8 with a minimum of 3 attempts, preventing premature and excessively delayed pruning.Lower thresholds increase false pruning, whereas higher thresholds waste attempts on intractable paths.
  • D.4 Pruning Parameters: The difficulty penalty coefficient λ = 0.5 controls how strongly TDI affects node selection in the UCB formula.It determines the influence of difficulty assessment during search.
  • D.4 Pruning Parameters: λ = 0 recovers standard UCB, while λ = 1.0 over-penalizes difficult nodes and blocks exploration of challenging but tractable paths.The selected intermediate penalty avoids both insufficient difficulty awareness and excessive suppression of difficult paths.

D.6 Context Load Degradation Study

The context-load study finds stable instruction-following through 40% of the context window, followed by approximately linear degradation and characteristic failure modes.

  • D.6 Context Load Degradation Study: The study evaluates 50 GOAD-derived instruction-following tasks with context variants from 10% to 90% across three frontier models.The models are GPT-4o, Claude-3-Sonnet, and Gemini-1.5-Pro, evaluated at temperature 0.
  • D.6 Context Load Degradation Study: Performance remains above 90% through 40% context load, then degrades approximately linearly.The 40% point is identified as the inflection beyond which added context begins harming performance.
  • D.6 Context Load Degradation Study: Beyond 40% load, failures involve ignored earlier information in 42% of cases, hallucinated tool outputs in 31%, and incorrect plausible commands in 27%.These categories characterize the observed degradation under accumulated penetration-testing context.
  • D.6 Context Load Degradation Study: The appendix includes tables for integrated tools, evidence scoring, TDI weights, mode thresholds, pruning parameters, and UCB difficulty penalties.These tables document the surrounding system configuration and sensitivity analyses.
Loading 2602.17622v1…