PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediation

• Manual penetration testing is time-consuming and expertise-intensive, while existing LLM approaches either require human intervention or target narrow security tasks.Source: §1 · p. 1
• Integrating penetration testing with remediation is essential because exploitation methods and software versions provide information needed to address discovered weaknesses.Source: §1 · p. 1
• The study evaluates whether LLMs can autonomously discover multiple vulnerabilities and remediate them effectively and cost-efficiently.Source: §1 · p. 2
• PenHeal combines a fully automated Pentest Module for detecting multiple vulnerabilities with a Remediation Module that generates and selects actionable strategies.Counterfactual Prompting explores multiple attack paths, while Instructor, Adviser, and Evaluator LLMs support guidance and recommendation selection.Source: §1 · p. 2
• PenHeal establishes a benchmark for vulnerability coverage, remediation effectiveness, and remediation efficiency.Source: §1 · p. 2

2.1 Cybersecurity Frameworks

• Penetration testing simulates attacks through reconnaissance, scanning, vulnerability assessment, exploitation, and post-exploitation to identify exploitable weaknesses.Source: §2.1.1 · p. 2

2.1.1 Penetration Testing.

• Existing AI penetration-testing studies often use simplistic or artificial cases and struggle with high-dimensional environments and large action spaces.Source: §2.1.1 · p. 2

2.1.2 Vulnerability Remediation.

• Vulnerability-remediation research largely targets source-code flaws, leaving remediation for computer hosts, systems, and network infrastructures comparatively underexplored.Source: §2.1.2 · p. 2

2.2 LLMs and Their Cybersecurity Application

• LLM advances, retrieval augmentation, role-play prompting, and tool-using agents provide the technical basis for applying language models to specialized cybersecurity tasks.Source: §2.2.1 · p. 2, §2.2.2 · p. 2, §2.2.3 · p. 2

2.2.4 LLM’s application in cybersecurity.

• Prior attack-automation systems demonstrate progress but remain focused on specific tasks or require human intervention, leaving fully automated multi-vulnerability testing insufficiently explored.Source: §2.2.4 · p. 3
• Existing benchmarks are either closed-source or task-specific, and no benchmark evaluates remediation effectiveness, motivating a benchmark covering vulnerability coverage, remediation effectiveness, and efficiency.Source: §2.2.4 · p. 3

• The objective is to maximize vulnerability coverage while maximizing remediation value under a cost budget.Source: §3 · p. 3
• Detection coverage measures the percentage of system vulnerabilities identified, reducing the chance that critical weaknesses are overlooked.Source: §3 · p. 3
• Remediation effectiveness is measured by the reduction in vulnerability score after applying a strategy and represented by its value score.Source: §3 · p. 3
• Remediation efficiency measures the time, money, and other resources required to implement recommendations through a cost score.Source: §3 · p. 3

4.1 Overview

• The Pentest Module scans a target, passes discovered vulnerabilities to remediation, and accepts user-defined remediation costs and budgets.Source: §4.1 · pp. 3–4, Figure 1 · p. 4

4.2.1 Planner and Attack Plan.

• The Planner orchestrates reconnaissance, scanning, assessment, and exploitation tasks, updating an attack plan from summarized outcomes to avoid duplicated effort.Source: §4.2 · p. 4, §4.2.1 · p. 4

4.2.2 Counterfactual Prompting.

• Counterfactual Prompting asks the Planner to assume identified vulnerabilities are absent, encouraging exploration of alternative vulnerabilities and attack vectors.Source: §4.2.2 · p. 4

4.2.3 Executor.

• The Executor generates parseable terminal commands, while the Instructor retrieves relevant external knowledge to improve command precision and the Summarizer condenses verbose outputs.Source: §4.2.3 · p. 4, §4.2.4 · pp. 4–5, §4.2.5 · p. 5

4.2.5 Summarizer.

• The Extractor converts attack history into structured vulnerability information for downstream remediation.Source: §4.2.6 · p. 5

4.3.1 Estimator and CVSS Lookup.

• The Remediation Module retrieves CVE and CVSS attributes for documented vulnerabilities and estimates descriptors for undocumented ones before generating recommendations.Source: §4.3 · p. 5, §4.3.1 · p. 5

4.3.2 Advisor.

• The Advisor proposes concrete fixes such as updates, patches, configuration changes, and protective measures, including commands or procedures where appropriate.Source: §4.3.2 · pp. 5–6

4.3.3 Evaluator.

• The Evaluator scores recommendations by value and cost, then uses a Group Knapsack formulation to select a high-value set under budget constraints.Source: §4.3.3 · pp. 6–7

5.1 Experimental Setup

• PenHeal is evaluated on Metasploitable2, an openly available victim environment containing 10 terminal-exploitable vulnerability types.The attacking host uses Kali Linux with Metasploit and Nmap on the same private network.Source: §5.1.1 · p. 7, Table 1 · p. 7

5.1.1 Environment.

• GPT-4 powers reasoning-intensive components, while GPT-3.5-turbo handles utility modules to reduce computational costs.The configuration assigns GPT-4 to the Planner, Executor, Advisor, and Evaluator, and GPT-3.5-turbo to the Summarizer, Extractor, and Estimator.Source: §5.1.2 · p. 7

5.1.2 Use of LLM.

• The Instructor uses LangChain-based retrieval to connect the LLMs with external penetration-testing knowledge.The knowledge base includes a general penetration-testing book and a practical Metasploit cookbook.Source: §5.1.2 · p. 7

5.2 Benchmark

• The benchmark scores detection coverage, remediation effectiveness, and remediation cost, then combines them into a holistic overall score.All three scores are unitless and range from 0 to 10; average value and cost scores reduce the influence of recommendation counts.Source: §5.2 · pp. 7–8
• The overall benchmark is designed to balance vulnerability detection, remediation quality, and remediation efficiency in realistic conditions.The evaluation uses the diverse vulnerabilities of Metasploitable2 to assess practical model capabilities.Source: §5.2 · p. 8

5.3 Numerical Results and Observations

• 31% higher detection coverage, 32% greater remediation effectiveness than PentestGPT, and 46% lower costs establish PenHeal’s strongest benchmark result.PenHeal records S_D=5.67, S_R=7.6, and S_overall=9.87; Figure 4 averages results over three runs.Source: §5.3 · p. 8, Figure 4 · p. 8
• PenHeal requires only the victim machine’s IP address, whereas PentestGPT and GPT-4 require human intervention to interpret outputs and advance tasks.The comparison concerns the experimental workflow described for the three models.Source: §5.3 · p. 8

5.4 Ablation Study

• Removing Counterfactual Prompting causes repeated attack paths and a 71% drop in detection coverage.The ablation compares PenHeal-noCounterfactual with the original model using the same benchmark.Source: §5.4 · pp. 8–9, Figure 5 · p. 8
• Removing the Instructor produces incorrect commands and reduces detection coverage by 59%.The Instructor supplies external knowledge that helps the Executor generate accurate commands.Source: §5.4 · p. 9
• Removing the Evaluator yields costly or ineffective recommendations, including unnecessary shutdowns or audits, with C=6.40 and overall score 5.48.This variant has the highest cost score and lowest overall score among the reported ablations.Source: §5.4 · p. 9

5.5 Demonstration of Model Outputs

• The Planner marks successful tasks complete, starts new tasks, and adds information-gathering subtasks after failed attacks.The attack-plan demonstration presents this behavior as evidence of runtime adaptability.Source: §5.5.1 · p. 9, Figure 6 · p. 9

5.5.1 Demonstration of the Attack Plan.

• The Instructor retrieves external knowledge for Samba exploitation, including the Metasploit module and affected Samba versions.The demonstrated excerpt concerns exploiting Samba smbd 3.X–4.X on ports 139 and 445.Source: §5.5.2 · p. 9

5.5.2 Demonstration of the Instructor’s Excerpt.

• The attack-plan visualization encodes reconnaissance in orange and exploits on ports 21, 6667, and 80 in green, red, and purple.Each tree node represents a specific task.Source: Figure 6 · p. 9

5.5.3 Demonstration of the Remediation recommendations.

• The Evaluator and Group Knapsack Algorithm narrow broad Adviser outputs into targeted recommendations under a budget.For the Samba example, updating Samba or configuring the firewall is adopted, while audits and service shutdown are rejected.Source: §5.5.3 · pp. 9–10
• The filtering rejects recommendations that do not directly address the vulnerability or that are highly disruptive in operational environments.Regular audits are rejected as indirect, while shutting down Samba is avoided because of its high cost and disruptiveness.Source: §5.5.3 · pp. 9–10

6.1 Answer to the Research Questions

• Approximately 60% detection coverage on Metasploitable2 lets PenHeal surpass PentestGPT and GPT-4 while autonomously discovering multiple vulnerabilities.This result answers RQ1 affirmatively without human intervention.Source: §6.1 · p. 10
• PenHeal’s remediation recommendations significantly exceed PentestGPT and GPT-4 in effectiveness and efficiency, answering RQ2 positively.The reported measures are remediation effectiveness and cost, denoted S_R and C.Source: §6.1 · p. 10

6.2 Limitations and Future Work

• The framework remains a proof of concept requiring refinement before it becomes a mature, plug-and-play solution.This limits the interpretation of the reported promise as a deployment-ready system.Source: §6.2 · p. 10
• Hallucinations sometimes produce incorrect commands, including nonexistent Metasploit modules or commands that violate task requirements.The authors identify this as a limitation shared with AUTOATTACKER [14].Source: §6.2 · p. 10
• Effectiveness depends on the attack tools in the knowledge base, limiting testing of more sophisticated systems or environments with restricted tool access.The authors propose integrating more diverse cybersecurity tools to broaden applicability.Source: §6.2 · p. 10
• Testing was limited to the Metasploitable2 machine, so performance across multiple machines and broader network scenarios may vary.The paper calls for extensive evaluation across varied network settings.Source: §6.2 · p. 10
• PenHeal generates remediation strategies but does not execute them because it operates from the attacker’s host machine.A secure bridge between attacker and victim systems is proposed for future end-to-end remediation.Source: §6.2 · p. 10

6.3 Ethical Considerations

• Automated attacks create security and privacy concerns because an unsecured model could be misused to discover and exploit vulnerabilities.The authors intentionally avoid exploitation after discovery and recommend cautious monitoring and usage.Source: §6.3 · p. 10

7 Conclusion

• PenHeal automates penetration testing and generates selected remediation recommendations from an input victim IP with minimal human intervention.The pipeline ends with recommendations tailored to user preference and budget.Source: §7 · p. 10
• PenHeal outperforms PentestGPT and GPT-4 in detection coverage, remediation effectiveness, and cost efficiency.These results support the feasibility of LLMs for advanced cybersecurity tasks.Source: §7 · p. 10
• The model helps address the gap in fully automated penetration testing for multiple vulnerabilities and their remediation, but still needs updates before becoming plug-and-play.Future directions include reducing tool dependence, improving remediation knowledge, and enabling end-to-end remediation.Source: §7 · p. 10

A.1 Planner System Prompt

• The Planner prompt organizes attack tasks in a layered sequence, assigns to-do, completed, or failed states, and updates results after each outcome.It permits adding subtasks when uncertainty or missing information requires further investigation.Source: §A.1 · pp. 11–12
• Listing 6 identifies the Planner System Prompt, while Listing 7 identifies the Executor System Prompt.The listings label the prompt specifications corresponding to the planning and execution roles.Source: §A.1 · p. 12, §A.2 · p. 12

A.2 Executor System Prompt

• The Executor prompt requires parseable command delimiters, logically sequenced commands, explicit Metasploit formatting, and a Kali Linux environment with pre-installed tools.These constraints standardize command generation and execution assumptions.Source: §A.2 · p. 12

A.3 Estimator System Prompt

• The Estimator prompt produces a CVSS 3.0 vector string and defines attack-vector choices such as Network, Adjacent, Local, and Physical.It also begins specifying attack complexity as conditions beyond the attacker’s control.Source: §A.3 · p. 12
• Listing 8 identifies the Estimator System Prompt, Listing 9 the Advisor System Prompt, and Listing 10 the Evaluator System Prompt.These labels correspond to the scoring, recommendation, and assessment stages.Source: §A.3 · p. 12, §A.4 · p. 12, §A.5 · p. 12

A.4 Advisor System Prompt

• The Advisor prompt requires actionable, effective remediation, preferring specific commands and added protection over general advice or service shutdowns.For example, it favors concrete update and firewall commands.Source: §A.4 · p. 12

A.5 Evaluator System Prompt

• The Evaluator prompt scores proposed remediation approaches for risk mitigation and cost using supplied definitions of value and cost.Its inputs include the vulnerabilities already discovered.Source: §A.5 · p. 12

B Attack Plan Script

• The figure shows a sample attack plan organized as a layered structure.The supplied caption identifies the content but does not specify additional axes, encodings, or comparisons.Source: Figure 7 · p. 12

C Group Knapsack Algorithm

• The algorithm selects at most one item from each group while maximizing total value under capacity C.This extends the classic knapsack objective by partitioning items into groups with a one-item-per-group restriction.Source: C Group Knapsack Algorithm · p. 12
• It initializes a dynamic-programming array of size C+1 with zeros for the knapsack capacity C.The array stores values for capacity states from zero through C.Source: C Group Knapsack Algorithm · p. 12
• For each group, the procedure scans capacities from C down to 0 and evaluates every item in that group.The nested loops organize computation by group, capacity, and candidate item.Source: C Group Knapsack Algorithm · p. 12
• For feasible items, it updates dp[j] with the maximum of its current value and the value obtained by adding the item.An item is considered only when j is at least its weight.Source: C Group Knapsack Algorithm · p. 12
• After processing all groups, the algorithm returns dp[C] as the computed result.The pseudocode labels this procedure as the Group Knapsack problem.Source: C Group Knapsack Algorithm · p. 12