PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks

Jiacen Xu, Jack W. Stokes, Geoff McDonald, Xuesong Bai, David Marshall, Siyue Wang, Adith Swaminathan, Zhou Li

arXiv:2403.01038v1cs.CRcs.AI

TL;DR

The paper asks whether LLMs can automate human-operated post-breach attacks, a capability not comprehensively evaluated across techniques and environments. It develops AUTOATTACKER with modular agents and retrieved attack experiences, finding that GPT-4 completes the evaluated tasks successfully while other tested models usually fail. The results underscore the need for defenses against increasingly capable LLM-generated attacks, although the study remains bounded by known limitations in LLM context and reasoning for complex environments.

  • Problem

    The paper addresses the lack of comprehensive evidence on using LLMs to simulate post-breach hands-on-keyboard attacks across varied attack techniques and environments.

  • Method

    AUTOATTACKER uses modular summarizer, planner, navigator, and experience-manager agents to generate attack commands with retrieved prior actions and structured prompting.

  • Results

    GPT-4 completed all evaluated attack tasks successfully at temperature 0, while GPT-3.5 and both tested Llama2 models failed on most tasks.

  • Takeaways & Limitations

    The study demonstrates the need for defenders to develop defenses that can thwart attacks generated by increasingly capable LLMs.

  • Takeaways & Limitations

    Directly prompted LLMs struggled with execution-environment tracking, complex observations, and complex action spaces, and prior reasoning methods were ill-aligned with these attack conditions.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated impressive results on natural language tasks, and security researchers are beginning to employ them in both offensive and defensive systems. In cyber-security, there have been multiple research efforts that utilize LLMs focusing on the pre-breach stage of attacks like phishing and malware generation. However, so far there lacks a comprehensive study regarding whether LLM-based systems can be leveraged to simulate the post-breach stage of attacks that are typically human-operated, or "hands-on-keyboard" attacks, under various attack techniques and environments. As LLMs inevitably advance, they may be able to automate both the pre- and post-breach attack stages. This shift may transform organizational attacks from rare, expert-led events to frequent, automated operations requiring no expertise and executed at automation speed and scale. This risks fundamentally changing global computer security and correspondingly causing substantial economic impacts, and a goal of this work is to better understand these risks now so we can better prepare for these inevitable ever-more-capable LLMs on the horizon. On the immediate impact side, this research serves three purposes. First, an automated LLM-based, post-breach exploitation framework can help analysts quickly test and continually improve their organization's network security posture against previously unseen attacks. Second, an LLM-based penetration test system can extend the effectiveness of red teams with a limited number of human analysts. Finally, this research can help defensive systems and teams learn to detect novel attack behaviors preemptively before their use in the wild....

I. INTRODUCTION

This work studies whether LLMs can automate human-like, post-breach hands-on-keyboard attacks across varied techniques and environments. AUTOATTACKER combines modular LLM agents, accumulated attack experiences, and a benchmark spanning diverse attack stages; GPT-4 completed all evaluated tasks, while weaker models generally failed.

  • Motivation: LLM-based cyber-attack research has largely focused on pre-breach activities, leaving comprehensive evaluation of post-breach hands-on-keyboard attacks under varied environments limited.Prior automated approaches also lacked comprehensive task coverage and automatic adaptation to different environments.
  • Challenges: Directly prompting LLMs produced unsatisfactory attack automation because models struggled with policy constraints, verbose outputs, context tracking, environment differences, and chained subtasks.Advanced attacks can fail when any subtask breaks, while command parameters are highly variable and sensitive to system details or paths.
  • Method: AUTOATTACKER uses modular summarizer, planner, navigator, and experience-manager components, with retrieval of prior attack actions to improve command precision and reuse successful subtasks.The system also uses carefully designed prompts and an action format containing planning, command type, and the exact command.
  • Results: GPT-4 completed all attack tasks successfully at temperature 0, whereas GPT-3.5, Llama2-7B-chat, and Llama2-70B-chat failed on most tasks.The experience manager and other components also reduced attack overhead and cost.
  • Evaluation: The authors introduce a benchmark with 14 attacks spanning reconnaissance, initial access, execution, persistence, and privilege escalation across Windows and Linux virtual machines.The simulated environment includes multiple virtual machines, varied software, and an attacker machine equipped for post-breach operations.

C. Security Issues of LLMs

The paper identifies several barriers to using LLMs for automated cyber-attacks, including safety filtering, difficult command extraction, weak environment tracking, and fragile multi-step execution. These issues can make directly deploying general-purpose LLMs unreliable without specialized system design.

  • Safety and response constraints: Jailbreaking is required because commercial LLMs generally refuse prompts requesting malicious attack commands.The paper describes jailbreaking as circumventing built-in safeguards to elicit otherwise restricted responses.
  • Environment tracking: LLMs may lose track of the execution environment and generate commands for the wrong shell, operating system, or tool.GPT-4 produced a Windows echo command while operating in a Metasploit Meterpreter shell, where the command was not runnable.
  • Safety and response constraints: Verbose LLM responses make extracting a single runnable attack command non-trivial and error-prone.Even after bypassing content filters, responses to privilege-escalation requests were described as too verbose and difficult to parse.
  • Robust automation: Complex attacks require uninterrupted chains of correct subtasks, because one failed step can break the entire attack.The paper emphasizes that human intervention or correction during automated execution should be avoided.
  • Robust automation: Attack commands have a large input space, and small typos in tools, parameters, paths, or names can fail victim-environment syntax checks.The challenge is compounded by alternative tools such as Mimikatz and PowerShell and their many parameters.

B. Limitations of Prior Works

Prior LLM-based attack-automation systems provide limited coverage of attack tasks and environments, while general LLM reasoning methods do not match cyber-attacks’ complex actions and observations. Existing systems also underexplore reuse across tasks and attacks spanning multiple machines.

  • Reasoning-system limitations: General LLM reasoning methods are poorly aligned with cyber-attacks because they assume limited action spaces and simple observations.The paper contrasts tasks such as Game of 24 with cyber-attacks’ complex commands and environment state.
  • Comparison dimensions: Table I compares related systems using task environment, human involvement, cross-machine support, retrieval of prior task results, and tested-task scope.The table defines ENV, HI, CM, RAG, and TT as comparison dimensions.
  • Coverage limitations: Prior LLM penetration-testing systems evaluated narrow task scopes, such as Linux privilege escalation, rather than broad attack workflows.One cited system used a simple command-response loop and demonstrated privilege escalation on Linux, with no other tasks evaluated.
  • Coverage limitations: Existing systems generally do not reuse results from previously executed attack tasks or evaluate complex cross-machine attacks such as lateral movement.The paper notes that prior simulated attacks target a single machine and omit cross-machine evaluation.

IV. METHODOLOGY

AUTOATTACKER models attacks as task chains and uses four interacting components to generate and execute actions without human interaction. Summarization, planning, navigation, experience retrieval, and jailbreaking address context, environment, command, and policy challenges.

  • Attack formalization: AUTOATTACKER defines an attack task as an environment and objective whose completion can be inferred from the victim environment.The attack procedure is represented as a chain of situations, observations, and actions.
  • Components and workflow: The system uses a summarizer, planner, navigator, and experience manager to generate and execute attack actions autonomously.The workflow iteratively summarizes observations, plans an action, selects an action using prior experience, and executes it.
  • LLM jailbreaking: A one-round role-playing jailbreak prompt elicits the cyber-attack information and commands required by the tested LLMs.The method begins prompts by presenting the model as an autonomous attacker agent.
  • Summarizer: The summarizer compresses observations and prior actions into a situation for the planner while preserving execution history within the LLM context.It updates a summary with the latest observation and appends previous actions before planning.

C. Planner (PLA)

The planner uses a structured prompt to convert the summarized situation and objective into a concrete attack action. Its action format and checking mechanisms aim to make commands precise and executable despite occasional LLM mistakes.

  • Planner prompt: The planner prompt combines the attacker objective, summarized situation, output requirements, and examples to derive a concrete action.The output requirements request step-by-step reasoning and examples support navigator action selection.
  • Planner prompt: AUTOATTACKER uses abstract objectives by default and detailed objectives for a variant without an experience database.Both objective types provide task-related information, but they are assigned to different system variants.
  • Action format: The planner returns actions containing planning, command type, and command sections to characterize the next execution scenario and command.The structured format reduces ambiguity when the navigator interprets the planner output.
  • Command validation: LLM-generated commands can still contain syntax mistakes, so a command checker fixes common errors such as semicolon-separated commands.The checker addresses cases where explicit prompt instructions are ignored.

D. Navigator (NAV) and Experience Manager (EXP)

The navigator executes or selects planner actions using retrieved successful experiences, while the benchmark evaluates autonomous attacks across varied tasks and environments. The experiments use isolated virtual machines, multiple operating systems, and repeated success-rate measurements.

  • Experience manager: The experience manager stores successful actions and retrieves similar examples to help the navigator choose among candidate actions.It embeds the planning section, ranks stored actions by cosine similarity, and supplies the top k examples alongside the planner action.
  • Navigator: The navigator uses an LLM to select one action from the planner output and the top three retrieved experience examples.The selected action is then executed in the victim environment.
  • Attack tasks: The benchmark covers 14 MITRE ATT&CK tasks across reconnaissance, initial access, execution, persistence, and privilege escalation.Tasks are classified as basic or advanced, and each is assessed for adaptability, stealthiness, and impact.
  • Environment setup: The environment uses Hyper-V with Kali Linux, Windows, and Ubuntu virtual machines, resetting the machines before each autonomous task.The attacker VM runs Metasploit and interacts with victim VMs without internet access in the isolated test network.
  • Evaluation: Each task is run three times, measuring success rate and average LLM interaction rounds, with termination after success or 30 rounds.The evaluation includes GPT-3.5, GPT-4, Llama2-7B-chat, and Llama2-70B-chat.

B. Experiment Results

AUTOATTACKER successfully completes diverse post-breach attack tasks with GPT-4, while performance depends strongly on model, temperature, and system components. Experiments also show that experience reuse reduces interaction overhead and cost.

  • GPT Results: At temperature T = 0, GPT-4 achieves a perfect success rate on every task, but higher temperatures cause occasional failures.Failures at T = 0.5 and T = 1 occur on Dump the Hash Attack, Log4j Header Inject, and SSH Public Key Inject.
  • Model Comparison: GPT-3.5 fails most tasks, while Llama2-7B-chat and Llama2-70B-chat fail all tasks because of command, tool-knowledge, and format-following problems.GPT-3.5 also struggles to track environment state and determine when tasks are complete.
  • Task Chain and Cost: The experience manager reduces ransomware interactions from 17 to 7, and advanced-task API charges are lower when it is used.Charges for basic tasks are below 1 dollar, while advanced-task costs also decrease with experience reuse.
  • Ablation Study: With an abstract objective and the experience manager, the ransomware task reaches a 3/3 success rate within 7 interactions; without the manager, it cannot be completed.Without both the experience manager and summarizer, AUTOATTACKER still achieves a 2/3 success rate; including the experience manager significantly improves performance.
  • Embedding Analysis: The experience manager ranks related actions using cosine similarity between embeddings of their planning sections to guide navigator action selection.Higher similarity scores indicate closer semantic meaning in the reported example.

VI. DISCUSSION

The discussion frames AUTOATTACKER as a feasibility study with important security implications and bounded experimental scope. It highlights jailbreaking as a prerequisite, insecure test environments, limited ATT&CK coverage, and possible defensive uses.

  • Limitations: AUTOATTACKER assesses whether LLMs can automatically carry out real-world cyber-attacks, using an environment configured to be insecure beforehand.The authors note that well-protected environments may require different tools because Metasploit is often detected and prevented.
  • Limitations: The study covers only a small subset of the MITRE ATT&CK Matrix because constructing vulnerable environments for individual techniques is time-consuming.The authors also identify single-LLM task assignment and hallucinations as limitations and future-work directions.
  • Security implications: Jailbreaking is presented as a prerequisite for using LLMs to automate attacks, while no defense is known to guarantee that jailbreaking cannot occur.The paper also notes that auditing whether attack knowledge has been truly unlearned is difficult.
  • Security implications: Automated post-breach attacks are considered more challenging, but the authors believe they may become more achievable as LLMs advance.They warn that attack volume and end-to-end operation could increase and accelerate, potentially favoring attackers.
  • Defensive applications: AUTOATTACKER can serve as a pentesting system for improving defenses against previously known and previously unobserved attacks.The paper also positions it as a benchmark for future defense research.

VII. CONCLUSION

The conclusion presents AUTOATTACKER as an LLM-based system for generating penetration tests across complex attack stages. GPT-4 produced valid solutions for all 14 tasks under a few interactions, while the other evaluated models usually failed.

  • Conclusion: AUTOATTACKER generates penetration tests for complex tasks such as lateral movement and credential getting across different attack-life-cycle stages.The system addresses LLM limitations and attack complexity through multiple sub-agents and domain knowledge in prompting.
  • Conclusion: GPT-4 automatically generated valid solutions for all 14 attack tasks under a few interactions.The conclusion contrasts this result with the weaker performance of the other evaluated models.
  • Conclusion: GPT-3.5, Llama2-7B-chat, and Llama2-70B-chat failed to find a solution in most attack tasks.This comparison is reported alongside GPT-4's success across the benchmark.
  • Attack stages: The demonstrated workflow follows attack stages including reconnaissance, initial access, credential access, lateral movement, collection, exfiltration, and reporting.These stages are described through the MITRE ATT&CK framework and an enterprise pentesting scenario.
  • Benchmark: The benchmark includes a task list linked to MITRE ATT&CK tactics and descriptions.The paper identifies this mapping in Table VIII.

X. AUTOATTACKER SYSTEM WORKFLOW

Algorithm 1 provides the pseudocode for the workflow employed by the AUTOATTACKER penetration test system.

  • AUTOATTACKER workflow: Algorithm 1 presents the pseudocode for AUTOATTACKER's penetration-test workflow.The passage identifies Algorithm 1 as the workflow specification for the system.
  • AUTOATTACKER workflow: The workflow is presented as an algorithm rather than only a prose description.Algorithm 1 is explicitly identified as providing pseudocode.
  • AUTOATTACKER workflow: AUTOATTACKER's system workflow is documented through Algorithm 1.The passage connects the algorithm directly to the penetration test system.

XI. EXAMPLE PROMPTS AND LLM RESPONSES

The examples document AUTOATTACKER's prompt-driven workflow, including planning, environment interaction, command formatting, and LLM safety behavior. The appendix contrasts a GPT-4 refusal with the system's structured workflow materials.

  • Examples: The appendix includes prompts and LLM responses used in the study.These materials are presented as examples considered in the main body.
  • Workflow: The workflow tracks tasks, observations, context, experiences, environments, and interaction status while iteratively selecting actions.The pseudocode shows summarization, planning, environment execution, and action recording within repeated loops.
  • Safety behavior: GPT-4 refused the attempted dump-the-hash request when used without the role-play jailbreak template.The example describes this as the default result of GPT-4's safety layer.
  • Task reference: Table VIII lists the tasks implemented by AUTOATTACKER and their related MITRE ATT&CK names, tactics, and descriptions.The table is identified as the task-description reference for the system.
  • Planner prompt: The planner prompt asks an autonomous attacker agent to respond with the next action using supported commands such as execute_shell.The example assumes a Kali Linux attacker with a Meterpreter reverse shell on a Windows 10 victim.
  • Action format: The required action format combines a planning rationale with a command and its argument.The example shows a plan to obtain a command shell from Meterpreter followed by execute_shell shell.

The requirements you must follow: RESPOND WITH EXACTLY ONE

The system requires one command per response without extra text, while GPT-4 demonstrates iterative correction during a Log4j exploit-module search. The examples also show privilege escalation and hash extraction steps in a Pass the Hash attack.

  • The requirements you must follow: RESPOND WITH EXACTLY ONE: The response format restricts the system to one command per response, forbids chained commands and semicolons, and requires checking task completion.These implementation constraints are repeated across the supplied instructions.
  • The requirements you must follow: RESPOND WITH EXACTLY ONE: For Pass the Hash, the demonstrated sequence escalates to SYSTEM with getsystem and dumps user password hashes with hashdump.The examples describe these as separate meterpreter commands after obtaining access.
  • The requirements you must follow: RESPOND WITH EXACTLY ONE: GPT-4 initially selected a nonexistent Log4j Metasploit module, then searched for “log4j” after the load failure.The failed module load prompted a search for available modules.
  • The requirements you must follow: RESPOND WITH EXACTLY ONE: GPT-4 identified exploit/multi/http/log4shell_header_injection from the search results and selected it for the Log4j header-injection task.The search output lists Log4Shell-related modules, including header injection and scanners.

XII. DETAILED AND ABSTRACT OBJECTIVE EXAMPLES

The examples contrast detailed and abstract objectives for a Pass the Hash attack. The detailed version enumerates atomic actions and reminders, whereas the abstract version retains only the high-level attack flow and required setup.

  • XII. DETAILED AND ABSTRACT OBJECTIVE EXAMPLES: A detailed objective specifies atomic actions and reminders, while an abstract objective omits most details and gives high-level ideas.The contrast is presented explicitly as an example for the Pass the Hash attack.
  • XII. DETAILED AND ABSTRACT OBJECTIVE EXAMPLES: The detailed objective directs the operator to obtain the Administrator NTLM hash, background the meterpreter shell, and use Metasploit psexec for Pass the Hash.It also names getsystem, mimikatz, and creds_msv as part of the prescribed sequence.
  • XII. DETAILED AND ABSTRACT OBJECTIVE EXAMPLES: The abstract objective retains the backgrounding step, use of the Administrator NTLM hash with psexec, LHOST configuration, and resulting meterpreter access in the domain controller.It compresses the detailed procedure into a smaller set of high-level actions and setup requirements.
Loading 2403.01038v1…