LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testing

1 Introduction

• Threat actors vary from users of accessible scripts and common exploits to sophisticated, well-resourced groups that may use private tools and zero-days.Sophisticated actors may deliberate, consult external resources, and target strongly defended networks.Source: §1 · p. 1
• Across expertise levels, threat actors make sequential decisions by interpreting new information and choosing what to do next.Tool knowledge and understanding the effects of tool outputs can support more successful decisions.Source: §1 · p. 1
• The paper asks whether an LLM can reason about threats, recommend and adapt actions, and potentially support novice actors or automate a campaign.These questions are explored in the context of improving defensive systems through realistic, controlled threat exercises.Source: §1 · pp. 1–2
• Initial ChatGPT exploration on a simple cyber task produced positive, startling findings and led the authors to test an end-to-end three-stage campaign.They then investigated prompt engineering, chain-of-thought reasoning, and assistive or autonomous LLM use.Source: §1 · p. 2

2 First Impressions and Challenges Ahead

• A scan can return hundreds of lines, making manual interpretation or a tool-specific parser laborious, especially when output must be related to an objective.The authors focus on an LLM decision process for executing and interpreting command-line tools.Source: §2 · p. 2
• ChatGPT summarized an NMap scan and returned executable Metasploit commands with correct parameters for exploiting vsftpd to spawn a remote root shell.The authors describe the scan summary as concise and accurate.Source: Figure 2 · p. 2, §2 · p. 3, Figure 3 · p. 3
• A Docker network sandbox and controller let the authors execute recommended tools and return their responses to ChatGPT for further experiments.The setup was used to verify response executability and automatically collect tool output.Source: §2 · p. 3
• Similar simple experiments produced predominantly accurate results and suggested the LLM could reason and plan from raw tool responses.The authors characterize this as resembling a more advanced “script kiddie” capability.Source: §2 · p. 3
• Prompt chaining was used to represent a single-action decision process and produce consistent, executable actions.The process prompts the LLM to observe network state, recommend actions, and execute them on the sandbox.Source: §2 · p. 3
• Campaign guidance is constrained by context-window size and stateless requests, which make it challenging to fit campaign history and tool responses into longer interactions.The authors identify scaling to longer campaigns and more complex network conditions as a primary challenge.Source: §2 · p. 3
• Domain-specific prompts and prompt chaining enabled the authors to automate multiple steps in a campaign, removing the human from the process.The automated agent process follows the single-action decision process described by the authors.Source: §2 · p. 3

3.1 A Single-Step Decision Process

• The single-step loop selects a tactic, generates and executes a command, then interprets its response; discrete interactions are intended to improve transparency and consistency over zero-shot prompting.The human tracks network state and prior actions because the LLM has no memory across prompts.Source: §3.1 · p. 4

3.2 Prompt Engineering

• Each decision-stage prompt combines SETUP, CONTEXT, and INSTRUCTION sub-prompts to establish the task, supply history, and constrain response content or format.Campaign context must be supplied explicitly because LLMs do not maintain memory across interactions.Source: §3.2 · pp. 4–5

3.2.1 Executable Action Stage Prompt

• The executable-action prompt names available tools, supplies prior commands and outputs plus tactic-specific guidance, and formats commands to support retries and parsing.Sequential commands are requested to improve error checking, while the prompt can request an alternate action after failure and prevent repeated actions.Source: §3.2.1 · pp. 5–6

3.2.2 Output Translation Prompt

• Output translation summarizes tool responses while preserving network details and syntax errors or corrections, then classifies the action as SUCCESS or FAILURE.The reported details are intended to inform subsequent tactical planning.Source: §3.2.2 · p. 6

3.2.3 Tactic Selector Prompt

• The tactic selector uses campaign preconditions and prior-action context to choose among bounded tactics, while an explicit end condition and objective define when the campaign should stop.Some judgments, such as whether enough information is known, are left to the LLM.Source: §3.2.3 · pp. 6–7

3.3 Automated Agent Prompting

• A finite-state controller automates the prompt chain, recording action history and the selected tactic while cycling through execution, translation, and tactic selection.The design also limits total actions, failures, and repeated actions to guard against runaway behavior.Source: §3.3 · pp. 7–8

4 Experiments

• The experiments evaluate the sandbox, demonstrate an automated campaign, test service-to-exploit matching, and examine prompt components.The execution-stage assessment repeated each single-service campaign ten times.Source: §4 · p. 8, §4.3 · p. 10

4.1 Sandbox and Network Environment

• The Docker sandbox used a Kali container controlled by an external agent and a single intentionally vulnerable Metasploitable 2 target.The VLAN could support more complex networks, but this demonstration used only one target device.Source: §4.1 · p. 8, Figure 12 · p. 8

4.2 Automation Demonstration

• The agent chained reconnaissance, exploitation, and exfiltration, but failed to transfer the archive; the demonstration used an easy-to-exploit vsftpd service.It summarized scan output, opened a remote root shell, and read sensitive files, while file-system navigation and exfiltration remained unreliable.Source: §4.2 · pp. 8, 10

4.3 Execution Stage Evaluation

• Exploit recommendations varied by service: vsftpd was most successful, while SQL sometimes failed, and Telnet, Apache, and exposed Port 513 elicited incorrect or speculative actions.The authors also report repeated recommendations for a vulnerability that was absent in the configured SMTP service; the prompt could not produce exploits for every service.Source: §4.3 · pp. 10–11, Table 1 · p. 10

4.4 Prompt Evaluation

• Prompt experiments found that common cybersecurity terminology and explicit command-only instructions helped produce structured, actionable responses.The authors note that prompt grammar, statement order, and pronoun use remain open questions.Source: §4.4 · p. 11

5 Discussion

• The paper frames the demonstration as prompting discussion of the nature of the work and dual use.Source: §5 · p. 11

5.1 Reflections on the Nature of Work

• Prompt engineering still requires acumen and skill, but shifts effort from figuring out what to do to figuring out how to instruct the LLM.The authors report spending about a month exploring prompt engineering and developing the demonstration.Source: §5.1 · p. 11

5.2 Abuse and Dual Use

• The design is intended to bolster defenses through penetration testing, but programmers could adapt it to guide malicious campaigns, lowering the entry bar for illegitimate actors.Source: §5.2 · p. 11
• LLMs could help low-competence threats challenge stronger defenses, potentially prompting defensive adaptation and accelerating a cybersecurity arms race.The authors say the effects on mercenary or state actors are unknown and describe autonomous, adaptive agents as a possible future scenario.Source: §5.2 · p. 12

6.1 Method Limitations

• The design depends on GPT-3.5’s biases and its not-yet-fully-understood ability to interpret command-line output and generate executable commands.Source: §6.1 · p. 12
• The demonstration does not establish how LLM access will affect mercenary or state actors pursuing more competent, better-resourced campaigns.The authors describe those effects as unknown.Source: Figure 14 · p. 12
• GPT-3.X training data gathered as of June 2021 leaves the system without knowledge of later tools and actions, a gap for testing against frequently discovered vulnerabilities and exploits.The authors say regularly updating an LLM is necessary to address this gap.Source: §6.1 · p. 12
• The kill-chain abstraction can impose a rigid tactical repertoire that misses opportunistic behavior and other human reasoning, while the agent tends to launch exploits rather than explore insecure configurations through normal service use.Source: §6.1 · pp. 12–13

6.2 Threat Scenario Demonstration and Experiment Limitations

• The proof-of-concept uses a simple, easily exploitable environment and does not address tactical planning in complex multi-hop networks.The current tactical selector is ill-equipped to choose targets or decide when to pivot; the demonstration also begins with network access.Source: §6.2 · p. 13

• Earlier language-model work covered translation, classification, and sentiment analysis, while later large models demonstrated broader topic understanding.The section describes “large” models as typically having more than 10B parameters and lists both closed- and open-source examples.Source: §7 · p. 13
• Prompt-engineering strategies used in this work include few-shot prompting, chain-of-thought, and self-consistency; prior work also explored LLM agents and planners.Source: §7 · p. 13
• The authors describe LLM application to cyber agents as limited and contrast cyber ranges and fixed-action systems with simulators that abstract network attributes, actions, or the network itself.Examples include CyberVAN, RIVALS-Deception, CASCADES, CyberEvo, and CyberBattleSim.Source: §7 · p. 13

• The paper contributes a Docker sandbox, prompt-engineering approaches, a prompt-chained kill-chain agent, an automated campaign demonstration, and an evaluation of cyber knowledge and prompt effects.Source: §8 · p. 13
• The work is preliminary, and future investigations will examine LLM capability on more complex multi-hop networks and more sophisticated or nuanced vulnerabilities.Source: §8 · p. 13
• Further work may require fine-tuning or interfaces to outside tools, and the authors plan to study defensive capabilities and prompt sensitivity.They also describe generative AI in cybersecurity as promising but concerning.Source: §8 · pp. 13–14

A Execution Prompt Engineering Analysis

• The prompt uses an Nmap scan containing vsftpd to elicit responses to statements in the execution stage.Source: Table 2 · p. 16
• Combining the statements produces a concise set of Metasploit commands that can be parsed and executed.Source: Table 2 · p. 16
• One statement was added to demonstrate LLM content moderation and excluded from subsequent prompts.Source: Table 2 · p. 16