Source-linked AI summary
Big Enough to Break Out: Tracking the Rising Capability of LLM Penetration-Testing Agents
Victoria Lovelace, Cameron Berryman, Yuhan You, Suhas Reddy Adavelly, Joel Sadler, Daniel Graham
TL;DR
LLM penetration-testing agents can automate parts of assessments, but their capabilities and failure limits remain insufficiently understood. The paper compares legacy and autonomous PentestGPT systems, evaluates coverage memory, and analyzes failures across public targets. The autonomous system completes all three attack chains, while memory does not improve outcomes and reviewed stalls point toward planning and commitment rather than forgetting.
Problem
Existing evidence shows LLM agents can handle penetration-testing subtasks but provides limited understanding of complete capabilities and failure limits.
Method
The paper compares legacy and autonomous PentestGPT harnesses, adds coverage-memory layers, and uses repeated-trial scoring plus transcript-based failure analysis.
Results
The autonomous configuration completes all three attack chains, while the legacy configuration reaches roughly half the subtasks on machines it does not solve; coverage memory improves neither harness.
Takeaways & Limitations
Reviewed stalls suggest that future capability progression may depend more on planning and commitment than on improved memory engineering.
Takeaways & Limitations
The cross-setup comparison cannot attribute the capability gap to the models because model, harness, execution, autonomy, and memory all change together.
Abstract
from arXiv · showhide
Large language model (LLM) agents are increasingly applied to penetration testing, but we still know little about what they can do or how they fail. We compare two PentestGPT-based systems: a legacy human-in-the-loop system running the open-weight Kimi K2.5, and a newer autonomous system running Claude Opus 4.8. Across three public targets, the autonomous system solves all three, including the two the legacy system never finishes. The legacy result is the more surprising of the two. Even on the machines the legacy system fails to solve, it completes about half the subtasks, while running on ordinary university GPUs with no provider guardrails. We can describe the trend but not explain it, since model, harness, autonomy, and memory architecture all change together. Its direction still points to the next question: what will limit these agents as they take on more complex tasks? The usual answer is long-horizon memory, the loss of access to earlier findings during long attack chains. We test it by adding a coverage-memory layer to both systems, and neither improves outcomes. In the legacy stalled runs we could review, the limiting factor appeared to be planning and commitment rather than lost memory: agents held the evidence for a route forward and never turned it into a concrete exploitation hypothesis, which may suggest that offensive capability will advance with agents' ability to plan rather than with better memory. The same subtask scoring that tracks this capability is available to defenders, who can measure it as it rises instead of waiting to meet it in the field.
1 Introduction
The paper tracks rising LLM penetration-testing capability by comparing a legacy human-in-the-loop system with a newer autonomous system, then tests whether coverage memory limits progress.
- 1 Introduction: The study compares two capability points: a legacy human-in-the-loop PentestGPT system and a newer autonomous configuration.The comparison is descriptive because model, framework, execution, autonomy, and memory differ between setups.
- 1 Introduction: The autonomous configuration completes attack chains that the legacy system does not, while the legacy system still handles roughly half the subtasks on unsolved machines.This makes the accessible legacy result significant despite its weaker overall completion outcome.
- 1 Introduction: The paper adds a coverage-memory layer to both frameworks, but neither intervention improves outcomes.The legacy version nearly triples reasoning time per cycle, while autonomous trials never encounter the retrieval failure targeted by the intervention.
- 1 Introduction: The results motivate tracking capability progression through subtask scoring rather than waiting for complete attack-chain success.The comparison provides a concrete starting point for measuring how quickly capability is advancing.
- 1 Introduction: The paper extends subtask evaluation with repeated-trial scoring and transcript-based failure analysis to identify remaining bottlenecks.Its analysis suggests that long-horizon progress may depend more on planning than memory.
2 Background and Related Work
PentestGPT established an LLM framework for automating parts of penetration-testing workflows, while LLM Wiki provides a persistent knowledge-base pattern adapted for coverage memory.
- 2 Background and Related Work: PentestGPT automates parts of penetration testing through reasoning, command generation, and tool-output parsing modules.Its Reasoning Module maintains a Pentesting Task Tree, while the other modules execute and compress workflow information.
- 2 Background and Related Work: Karpathy’s LLM Wiki is a persistent, self-maintained knowledge base whose linked Markdown pages accumulate information for later queries.The paper adapts this pattern for its legacy coverage-memory implementation.
3 Systems
The paper evaluates legacy and autonomous PentestGPT harnesses with different divisions of labor, state representations, and bounded views of prior findings.
- 3 Systems: The legacy harness uses three model sessions for reasoning, command generation, and parsing, while a human executes each proposed command.Planning remains with the LLM; the operator does not select actions or alter the plan.
- 3 Systems: The autonomous harness uses Supervisor and Executor roles inside a deterministic controller that validates tasks, traces, evidence, and state transitions.Its task types include discover, enumerate, test, exploit, verify, and recover.
- 3 Systems: The autonomous system stores authoritative state in a SQLite Memory Kernel, while provider transcripts and summaries remain diagnostic traces.Evidence entering the record must trace to exact contiguous slices of real tool output.
- 3 Systems: The Supervisor sees a bounded projection of the full history, so early observations can remain stored while leaving the planner’s working view.This is a retrieval limit rather than a storage-capacity limit.
- 3 Systems: Controller convergence denotes repeated discovery or enumeration after earlier coverage leaves the Supervisor’s projection, although maintainers also identify planning as a competing explanation.The paper carries both limited retrieval and over-decomposition forward as hypotheses.
4 The Coverage-Memory Intervention
The coverage-memory intervention records explored attack surfaces and exposes them during planning, using framework-specific storage and recall mechanisms.
- 4 The Coverage-Memory Intervention: The intervention tests whether persistent access to earlier findings reduces repeated work in both PentestGPT frameworks.Each framework is evaluated separately because its state-management architecture differs.
- 4 The Coverage-Memory Intervention: Both implementations separate storage from recall: notes record examined surfaces and are made available to the planner before its next action.The store is advisory, lasts one trial, starts empty, and does not satisfy evidence or dependency requirements.
- 4 The Coverage-Memory Intervention: The autonomous layer automatically copies committed observations into a vault, while the Supervisor deliberately consults that vault before new discovery or enumeration.This restores access to dropped coverage without changing the canonical record.
- 4 The Coverage-Memory Intervention: The legacy implementation adapts Karpathy’s LLM Wiki inside the reasoning session to store confirmed findings and dead ends.A manager prepends stored summaries before reasoning and asks the model for new entries after task-tree updates.
- 4 The Coverage-Memory Intervention: The legacy comparison includes an added reasoning-model call for writing wiki entries, so memory changes both visibility and per-cycle model work.Time and token effects are reported separately from the baseline outcome comparison.
5 Methodology
The study evaluates two PentestGPT versions on three vulnerable machines using controlled prompts, target-specific scoring, transcript review, and efficiency measures. It also defines admissible outcomes and a failure taxonomy to distinguish agent reasoning from infrastructure or framework problems.
- Experimental Design: Two PentestGPT versions are evaluated on three public vulnerable machines, with comparisons within each version made between baseline and coverage-memory trials.The legacy system uses Kimi K2.5, while the autonomous system uses Claude Opus 4.8; cross-version differences describe capability changes rather than causal effects.
- Experimental Design: Trials begin from fixed target information without an attack plan, and success requires direct evidence such as verified root access, a proof file, or remote code execution.The evaluation prohibits end-to-end vulnerability scanners but permits targeted tools after a specific attack surface is found.
- Measurements: The study combines trial outcomes, subtask completion, reasoning cycles, decisions, time, tokens, cost, wall-clock time, and repeated-discovery measures from saved runs and transcripts.Efficiency comparisons are made when conditions share the same outcome, and repeated discovery or failure modes are descriptive rather than proof of controller convergence.
- Scoring: Bob and Tr0ll are scored with pre-specified checklists granting credit only for transcript-supported completion, while Metasploitable 2 is assessed through direct evidence because it lacks one intended route.Alternate valid attack paths receive credit, so scoring measures progress without enforcing a single procedure.
- Outcomes and Failure Modes: Outcomes distinguish success, stall, generation failure, and external abort, while the failure taxonomy records context loss, invalid commands, unsupported interpretations, and failure to identify a solution path.Stalls require sustained no progress and are assigned through terminal-transcript review rather than a fixed cycle or decision threshold.
- Admissibility: Only trials with functioning targets that end because of agent behavior count toward reasoning-capability statistics; infrastructure, framework, provider, and other external failures are excluded.This admissibility rule separates agent performance from failures caused by the experimental environment or platform.
6 Results
The autonomous Claude system completed every benchmark chain, while the legacy Kimi system reached roughly half the subtasks on targets it failed to solve. Coverage memory did not improve outcomes, and exploratory failures point more toward planning and commitment than forgotten evidence.
- 6.1 Capability Progression Across Generations: Claude solved all three admitted trials on each benchmark target, while Kimi solved Metasploitable 2 but stalled on Bob and Tr0ll.The comparison is descriptive because model, harness, execution, autonomy, and memory differ.
- 6.1 Capability Progression Across Generations: Kimi reached 7 of 14 subtasks on Bob and 9 and 7 of 14 across two Tr0ll trials, whereas Claude completed all 14 in every admitted baseline run.Legacy reasoning time alone exceeded the autonomous runs’ full wall-clock time.
- 6.2 Coverage Memory Did Not Improve Outcomes: Coverage memory did not improve outcomes: all Metasploitable 2 and Bob trials still succeeded, while both conditions solved all Tr0ll trials.Tr0ll averaged 6.3 versus 9.0 Supervisor decisions, but repeated discovery was zero and the difference was not treated as evidence of benefit.
- 6.2 Coverage Memory Did Not Improve Outcomes: Across the benchmark targets, repeated discovery was zero under the strict measure, so the intervention’s target behavior never occurred and controller convergence was not tested.The trials show no outcome improvement on short chains but cannot establish whether recall helps when earlier coverage leaves the projection.
- 6.2 Coverage Memory Did Not Improve Outcomes: The legacy wiki left mean cycle count unchanged but increased mean reasoning time per cycle 2.9×, from 99.3 to 286.6 seconds.The legacy coverage-memory condition was run only on Metasploitable 2.
- 6.3 Failure Patterns in Legacy Stalls: In reviewed legacy stalls, agents retained relevant findings but failed to connect and commit to a route forward, suggesting planning and hypothesis selection as the binding constraint.The sample is small, failure-mode counts are lower bounds, and comparable autonomous failures were exploratory.
- 6.5 Over-Decomposition on a Long Target: On the longer Enigma target, all three exploratory Claude runs exhausted their decision budgets without root, producing mostly enumeration and testing tasks rather than exploitation.The runs used Claude Opus 5 and were outside the controlled comparison.
7 Discussion
The coverage-memory intervention did not improve outcomes because the targeted retrieval failure did not occur; reviewed stalls instead point toward planning and commitment as the likely bottleneck. This suggests future progress may depend more on hypothesis selection and commitment than on expanding memory.
- Coverage memory should be added only when evidence shows that the planner is losing information, because retrieval was not failing in the evaluated runs.The legacy wiki increased reasoning time per cycle by 2.9×, while autonomous recall increased Supervisor output by roughly 18% per decision.
- The reviewed stalls contained evidence for a route forward, but agents kept decomposing, re-enumerating, or varying tactics instead of acting on it.Three legacy stalls do not establish weak commitment as the general cause of controller convergence, but they point away from memory capacity as the binding constraint.
- If broader evaluation confirms this pattern, future systems may need to rank hypotheses, commit to exploitation paths, and pivot when evidence contradicts them.A planning-focused mechanism would change how the agent chooses among available findings, whereas coverage recall changes only which findings remain accessible.
- The same lab-based subtask checklists and failure categories can help defenders track where new agents stop and why.The authors place the observed stopping point at hypothesis selection rather than reconnaissance or memory, and suggest rerunning the checks on each new model.
8 Threats to Validity
The evaluation is constrained by possible benchmark contamination, small samples, confounded system changes, non-identical controller limits, and judgment in scoring. These limitations prevent causal attribution and weaken efficiency comparisons, although within-target outcome comparisons remain more robust.
- Public walkthroughs may have supplied exact solutions, so the evaluated agents may have reproduced known solutions rather than derived every step from evidence.Subtask checklists and transcript review show progress but cannot eliminate this contamination threat.
- With one to five trials per condition, the study reports means and ranges without statistical inference, making efficiency differences provisional.Categorical outcome results are more robust because they were unanimous within each condition.
- The model, framework, execution paradigm, autonomy, and memory system all change together, so the capability gap cannot be attributed to any single factor.The open-weight model appears only in the weaker harness and the frontier model only in the stronger one.
- Controller settings varied across machines, making within-target comparisons valid but cross-target comparisons invalid.The exploratory Enigma runs also used a different model and a six-turn, 60-decision, four-attempt configuration.
- Subtask and failure-mode scoring requires human judgment for broader approaches and whether findings support a solution path.Keeping supporting transcript segments makes these judgments reviewable, while independent scoring remains a future improvement.
9 Future Work
Future work should test memory on a target long enough for findings to leave the planner’s visible context and should investigate planning-focused interventions across broader transcripts. These studies also need cleaner comparisons that separate model effects from harness effects and avoid public walkthrough contamination.
- Future evaluations should use custom targets without public walkthroughs and place multiple models in one autonomous framework.This would hold framework, execution, and memory constant while varying only the model.
- A longer target with the coverage layer enabled is the direct next test of the retrieval hypothesis.The benchmark machines were too short, while Enigma was long enough and all three baseline runs stalled.
- Broader transcript analysis should test whether weak hypothesis selection and fixation recur across more models, targets, and runs.If they do, future work could retain a concrete exploitation hypothesis, cap repeated effort, and force a pivot when that budget expires.
10 Conclusion
The study finds a substantial capability progression from a legacy open-weight, human-in-the-loop system to an autonomous frontier-model system, while cautioning that the gap is not causally explained. Coverage memory did not improve outcomes, and observed failures instead point toward planning and commitment.
- The autonomous frontier-model configuration finished every attack chain, including two the legacy configuration never finished, while the open-weight system completed roughly half the subtasks on failed machines.The legacy system ran on commodity university GPUs without provider approval or refusal.
- Coverage memory improved neither harness and nearly tripled legacy reasoning time per cycle, while autonomous runs showed no finding leaving the planner’s view.Reviewed failures looked less like forgetting than failure to commit despite evidence for a route forward.