PaperlayerView exact revision on arXiv

Source-linked AI summary

Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks

Brian Singer, Keane Lucas, Lakshmi Adiga, Meghna Jain, Lujo Bauer, Vyas Sekar

arXiv:2501.16466v4cs.CRcs.AI

TL;DR

Multi-host red-team exercises are costly and existing LLM-assisted offense systems have limited success in these challenges. Incalmo assigns high-level task planning to an LLM and execution to expert agents, with auxiliary services for context and assets. On MHBench, Incalmo succeeded in 37 out of 40 environments, compared with 3 out of 40 for ExpertPromptShell.

  • Problem

    Multi-host red-team exercises are costly and expertise-intensive, while existing LLM offense systems have very limited success on multi-host challenges.

    Source: §1 · p. 1

  • Method

    Incalmo has an LLM plan high-level declarative tasks, delegates execution to expert agents, and uses auxiliary services to manage context and acquired assets.

    Source: §1 · p. 2

  • Results

    Incalmo succeeded in 37 out of 40 MHBench environments, while ExpertPromptShell with Sonnet 4 succeeded in 3 out of 40.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Incalmo demonstrates the feasibility of LLM-assisted red teaming in complex multi-host settings across a majority of diverse MHBench environments.

    Source: §9 · p. 14

    The study is scoped to exercises with known vulnerabilities; future versions could add 0-day-specific task agents.

    Source: §7 · p. 13

Abstract

from arXiv · show

Security operators use red teams to simulate real attackers and proactively find defense gaps. In realistic enterprise settings, this involves executing multi-host network attacks spanning many "stepping stone" hosts. Unfortunately, red teams are expensive and entail significant expertise and effort. Given the promise of LLMs in CTF challenges, we first analyze if LLMs can autonomously execute multi-host red team exercises. We find that state-of-the-art LLM-assisted offense systems (e.g., PentestGPT, CyberSecEval3) with leading LLMs (e.g., Sonnet 4, Gemini 2.5 Pro) are unable to do so. Building on our observations in understanding the failure modes of state-of-the-art systems, we argue the need to improve the abstractions and interfaces for LLM-assisted red teaming. Based on this insight, we present the design and implementation of Incalmo, an LLM-assisted system for autonomously red teaming multi-host networks. Incalmo uses LLMs to plan red team exercises in terms of high-level declarative tasks that are executed by domain-specific task agents. Incalmo also uses auxiliary services to manage context and acquired assets. For our evaluation, we develop MHBench, a novel multi-host attack benchmark with 40 realistic emulated networks (from 22 to 50 hosts). We find that Incalmo successfully acquires critical assets (i.e., key hosts or data) in 37 out of 40 MHBench environments. In contrast, state-of-the-art LLM-assisted systems succeed in only 3 out of 40 environments. We show that Incalmo is efficient-successful attacks took 12-54 minutes and cost <$15 in LLM credits.

1. Introduction

Multi-host red-team exercises are costly and expertise-intensive, while existing LLM-assisted systems struggle to execute them autonomously. Incalmo raises the planning abstraction, delegates execution to expert agents, and succeeds across most MHBench environments.

1. Introduction

  • Red teams test network defenses across many hosts, helping defenders prioritize vulnerabilities, evaluate detection rules, and test response strategies, but exercises require substantial expertise and effort.Source: §1 · p. 1
  • MHBench contains 40 multi-host environments and shows that state-of-the-art LLM offense systems achieve very limited success in these challenges.The benchmark draws on public attack reports, reference topologies, and prior work.Source: §1 · p. 1
  • Existing systems waste effort on irrelevant tasks, execute tasks incorrectly, use brittle post-exploitation techniques, and suffer context bloat in long challenges.Source: §1 · p. 2
  • Incalmo uses an LLM to plan high-level declarative kill-chain tasks, expert domain-specific agents to execute them, and auxiliary services to manage context and acquired assets.The services support environment-state, attack-graph, and command-and-control functions.Source: §1 · p. 2
  • The demonstrated setting covers unforeseen topologies and attack paths involving known or public vulnerabilities; the design could be extended to support 0-day capabilities.Source: §1 · p. 2
  • For Success, Incalmo achieved 37 out of 40 environments versus ExpertPromptShell’s 3 out of 40; for TotalAcquisition, it led in 37 and tied in three.Successful attacks took 12–54 minutes and cost ≤$15 in LLM credits.Source: Figure 2 · p. 2, §1 · p. 2
  • Incalmo’s abstractions matter more than model size in the reported ablation; even smaller models such as Haiku 3.5 succeed in most environments.Source: §1 · p. 2

2. Motivation and background

The section motivates autonomous red teaming by the cost of manual multi-host exercises and the limited evidence on whether LLM-assisted systems can handle them. An Equifax example illustrates the stepping-stone nature of these attacks, while evaluations find existing systems largely unable to realize them.

2.1. Motivating example: Red teaming Equifax

  • The 2017 Equifax breach involved compromising web servers, using found credentials to reach database servers, and exfiltrating data from 48 databases.The example illustrates an attack spanning network segments and vulnerabilities to acquire critical assets.Source: §2.1 · p. 3
  • Red teams can help defenders prioritize vulnerabilities, evaluate detection rules, and test response strategies, but exercises require specialized, expensive teams and manual effort.Automated red teaming is presented as a potential way to lower the cost and effort of continuous testing.Source: §2.1 · p. 3

2.2. Approaches to offense systems

  • The paper focuses on autonomous LLM-based red teaming of multi-host attacks, which span hosts and subnetworks and require intermediate subgoals, strategic planning, and coordinated actions.Prior LLM work includes CTF-style and single-host challenges, while the paper identifies limited understanding of multi-host performance.Source: §2.2 · pp. 3–4

2.3. Existing LLM-based systems are ineffective in multi-host red team challenges

  • The authors create MHBench and evaluate autonomous systems including CyberSecEval3, ExpertPromptShell, and CAI, plus human-in-the-loop PentestGPT, using selected LLMs and 10 illustrative environments.The evaluations include five trials per environment for baseline systems; PentestGPT is run for three trials because it requires manual effort.Source: §2.3 · p. 4
  • Red-team success means acquiring at least one key asset; TotalAcquisition measures how many critical assets are captured.The section uses Success and TotalAcquisition to evaluate baseline systems.Source: §2.3 · p. 4
  • Across the evaluated systems, only ExpertPromptShell partially succeeded: Sonnet 4 in the Equifax-inspired environment and Sonnet 4 and Gemini 2.5 Pro in the chain environment.The figure summarizes Success and TotalAcquisition across 10 environments.Source: Figure 4 · p. 4, §2.3 · pp. 4–5

3. Failure analysis

The failure analysis combines an iterative model of red-team work, reference attack solutions, and execution-log review. It identifies irrelevant task choices, incorrect execution, brittle host-control methods, and growing context as problems in existing systems.

3.1. Methodology

  • The analysis models red teaming as a loop of choosing and executing tasks, then updating knowledge and assets, and compares system logs with manually implemented reference solutions.Reference tasks are defined against states in an attack graph; relevant tasks are checked for correct implementation.Source: §3.1 · p. 5

3.2. Observations

  • Across evaluated environments, 47–90% of ExpertPromptShell’s commands were irrelevant, including brute-force attempts and attacks on non-exploitable services.Caldera also performed irrelevant actions, while PentestGPT sometimes tried to delete command history on the attacker’s host.Source: §3.2 · pp. 5–6, Figure 7 · p. 6
  • Across LLMs and environments, 6–41% of ExpertPromptShell’s relevant tasks used incorrect commands, which can block host compromise and downstream vulnerability discovery.Manual review found recurring problems implementing exploits and network scans, including a failed Apache Struts exploit attempt.Source: §3.2 · p. 6
  • PentestGPT and CAI could discover external services with scanning tools but struggled to find remote-code-execution vulnerabilities, sometimes running unrelated exploits or giving up.In one case, CAI used nine commands to find web servers before running three unrelated exploits.Source: §3.2 · p. 6
  • ExpertPromptShell used exploits to run commands on compromised hosts rather than establishing command-and-control agents, a brittle approach that grows less reliable across longer chains.The authors observed this only for ExpertPromptShell because the other systems made little progress; SSH and reverse shells also failed under common firewall configurations outside one chain challenge.Source: §3.2 · p. 6
  • Prior LLM systems accumulated low-level outputs in context; one ExpertPromptShell run reached 54K tokens after 108 commands, likely hindering high-level planning.A single command produced more than 30K characters of file paths.Source: §3.2 · p. 6

4. Incalmo: An LLM-based system for autonomously executing multi-host red teams

Incalmo separates LLM planning from task execution, using declarative red-team tasks, domain-specific agents, and auxiliary services to manage context and assets. Its Equifax-inspired case study follows a multi-host attack path to exfiltrate data from 48 databases.

4.1. High-level idea

  • Incalmo separates LLM planning from execution: the planner selects high-level declarative tasks, while expert agents carry them out and auxiliary services offload accumulated context.This contrasts with prior systems that use the LLM for both planning and execution.Source: §4.1 · p. 7

4.2. Detailed design

  • Declarative tasks model cyber-kill-chain stages, and task agents translate them into low-level commands using security best practices.The task interface is extensible to new tasks and execution agents.Source: §4.2 · pp. 7–8
  • The environment-state service maintains queryable, updated network knowledge; the dynamic attack-graph service uses it to recommend paths based on current vulnerabilities and reachability.The C&C service executes commands on infected hosts and supports downloading and executing malware to infect additional hosts.Source: §4.2 · p. 8

4.3. Illustrative case study

  • In the Equifax-inspired case, Sonnet 4 found SSH credentials on a second web server, then used attack and environment services to exfiltrate data through a web server.The workflow was repeated to infect and exfiltrate data from all 48 databases; an earlier web-server infection was a dead end.Source: §4.3 · p. 9

5. Implementation

The implementation provides agents for high-level red-team tasks and a benchmark of 40 emulated multi-host environments. MHBench varies network structure, vulnerabilities, and attack complexity.

5. Implementation

  • Incalmo is an approximately 8K-line Python framework with a custom C&C server, Caldera malware capabilities, and parsers that update its environment knowledge base.It implements non-LLM and LLM-based agents for five high-level tasks; non-LLM lateral-movement and privilege-escalation agents use known vulnerabilities and corresponding exploits.Source: §5 · p. 9
  • The orchestration loop extracts LLM-generated Python functions into tasks or context queries, then runs attacks until the LLM signals completion or the time limit is reached.The implementation uses LangChain for iterative prompting.Source: §5 · p. 9
  • MHBench contains 40 OpenStack-hosted environments: 10 target data exfiltration and 30 target root access to key hosts.The environments range from 22 to 50 hosts and include both algorithmically generated and manually designed topologies.Source: §5 · p. 9
  • MHBench varies topology, vulnerability types, and red-team complexity, with environments spanning 2–48 critical assets and 5–104 tasks.Included vulnerabilities range from plain-text credentials to remote-code-execution and privilege-escalation vulnerabilities.Source: §5 · pp. 9–10

6. Evaluation

Across MHBench, Incalmo substantially outperforms ExpertPromptShell, and its success holds across different LLMs. Ablations show contributions from high-level tasks and auxiliary services; successful attacks take minutes and cost at most about $15.

6.1. Red team success evaluation

  • 37 of 40 environments: Incalmo with Sonnet 4 succeeded, versus 3 of 40 for ExpertPromptShell with Sonnet 4.Incalmo achieved perfect reliability in 28 environments; ExpertPromptShell was perfect in none.Source: §6.1 · p. 10
  • In 21 of 40 environments, Incalmo acquired at least 50% of critical assets; ExpertPromptShell never exceeded 24% in any environment.Incalmo acquired all critical assets in 9 environments.Source: §6.1 · pp. 10–11, Figure 10 · p. 10

6.2. Factor analysis

  • Across 10 LLMs, Incalmo succeeded in 6–9 of 10 representative environments, while smaller-model Incalmo outperformed larger-model ExpertPromptShell on Success in 9 of 10.In the Equifax environment, Haiku 3.5 with Incalmo exfiltrated all 25 databases, while ExpertPromptShell with Sonnet 4 exfiltrated one file.Source: §6.2 · p. 11, Figure 12 · p. 11
  • Incalmo-WHT failed across all 10 environments and all 10 LLMs; with services, Incalmo succeeded in 1–5 more environments than Incalmo-WS.The Incalmo-WS variant made environment and attack-graph services inaccessible to the LLM.Source: §6.2 · pp. 11–12, Figure 13 · p. 12

6.3. Cost and speed

  • 12–18 minutes: Incalmo gained root access to all 15 critical hosts in Enterprise C; it exfiltrated data from all 48 Equifax-inspired databases in 54 minutes.The most expensive experiment cost around $15 in LLM credits; Gemini 2 Flash usage was within the free tier.Source: §6.3 · p. 12, Figure 14 · p. 12

6.4. Extensibility case study

  • Sonnet 3.5 task agents showed promise on individual tasks, but using them for all tasks failed in all three test environments; replacing only the lateral-movement agent succeeded in all three.The study also identifies steps prior systems struggled with and suggests a roadmap toward 0-day capabilities via novel agents.Source: Figure 15 · p. 12, §6.4 · p. 13

7. Discussion and limitations

The discussion identifies limits in asset coverage and failure scenarios, then outlines scope boundaries and directions for expanding Incalmo and MHBench. These include known-vulnerability exercises, benchmark realism, and evaluations without defenders.

7. Discussion and limitations

  • In some trials, the planner stopped after acquiring one critical asset despite additional attack-graph paths it could have explored.The authors speculate that limited training data on using attack graphs for multi-host red teaming may contribute.Source: §7 · p. 13
  • Three unsuccessful environments required both external and internal scans, and the authors hypothesize that reasoning about network segments and access control could improve Success and TotalAcquisition.The proposed attack-graph service extension would account for constraints such as same-subnet access and inter-segment firewalls.Source: §7 · p. 13
  • The evaluation is scoped to exercises with known vulnerabilities; future versions could add specialized agents for 0-day exploits.The authors describe the known-vulnerability setting as representative of many real-world red teams that chain known techniques.Source: §7 · p. 13
  • Because enterprise network details are sensitive and public information is limited, MHBench uses public sources and prior reports to design realistic environments.Future work could extend the benchmark and evaluate Incalmo across a broader range of real, possibly proprietary, enterprise settings.Source: §7 · p. 13
  • Incalmo was evaluated without defenders; future work could test realistic or autonomous defenses and add features to evade detection.Source: §7 · p. 13
  • Making MHBench available creates a potential memorization concern, so the authors envision an evolving benchmark with future holdout tests.They note that providers may incorporate the benchmark into training data after its release.Source: §7 · p. 13

8. Other related work

Related work includes LLM security benchmarks and other LLM-based security applications, but these differ from the paper’s focus on multi-host red teaming.

8. Other related work

  • Existing LLM security benchmarks often cover CTF challenges and single-host attacks, while other benchmarks assess general security knowledge.Source: §8 · p. 13
  • Other LLM security applications include vulnerable-code discovery, security-log summarization, anomaly detection, and social engineering such as phishing.Source: §8 · p. 13
  • The paper distinguishes these related areas from its focus on multi-host red teams.Source: §8 · p. 13

9. Conclusions

The paper concludes that Incalmo demonstrates the feasibility of LLM-assisted red teaming in complex multi-host settings. Across most MHBench environments, it carries out attack stages from service discovery through data exfiltration.

9. Conclusions

  • Across a majority of diverse MHBench environments, Incalmo autonomously carried out attack stages from finding vulnerable services to exfiltrating data.The reported stages include exploiting services, gaining network access, finding configurations and vulnerabilities for lateral movement, and escalating privileges.Source: §9 · p. 14
  • By raising abstraction and decoupling planning from execution with domain-specific task agents, Incalmo demonstrates the feasibility of LLM-assisted red teaming in complex multi-host settings.Source: §9 · p. 14
  • The authors believe lowering the cost, time, and effort of red-team exercises could help defenders proactively protect networks against future human- and AI-based attacks.Source: §9 · p. 14

Ethics considerations

The authors frame Incalmo as dual-use: defenders can use it to assess security gaps, while attackers can use it against networks. They proceed because they judge the benefits of autonomous red teaming to outweigh potential harms.

Ethics considerations

  • Incalmo is dual-use: defenders can proactively test networks for security gaps, but real attackers can also use it to attack networks.The authors describe its risks as similar to those of other LLM-based and non-LLM attack systems.Source: Ethics considerations · p. 14
  • Understanding the limits of AI-assisted autonomous attacks can benefit red teams and help defenders proactively assess vulnerabilities and security gaps.The authors also say this understanding is valuable for advancing AI-meets-security research among stakeholders.Source: Ethics considerations · p. 14
  • LLM providers could profit from Incalmo-like security tools, but misuse could harm their reputation or prompt regulation that affects profits.Source: Ethics considerations · p. 14
  • Companies could use autonomous attack tools to find security gaps, while bad actors could use them to attack companies; vendors' customers may benefit from lower red-teaming costs, time, and effort.Source: Ethics considerations · p. 14
  • Incalmo and MHBench can help policymakers measure LLM red-teaming capability and inform policy discussions about AI technology.Source: Ethics considerations · p. 14
  • The authors proceeded because they judged benefits to outweigh potential harms, notified LLM providers about guardrails, and plan to make Incalmo, MHBench, and reproduction tools open source.They note autonomous attack tools can both help prevent security risks and lower the bar for malicious attacks.Source: Ethics considerations · p. 14

LLM usage considerations

The authors report that meaningful results were observed only with closed-source LLMs, while noting reproducibility concerns and steps taken to mitigate them. Their appendix uses attack graphs and reference solutions to analyze task execution and failures in multi-host challenges.

LLM usage considerations

  • Closed-source LLMs produced the only meaningful results observed, which may make them harder to reproduce; the authors mitigate this by releasing MHBench, prompts, model numbers, and Incalmo’s code.They also report performance across diverse LLMs and envision capable open-source models replacing closed-source ones.Source: LLM usage considerations · p. 15
  • The authors could not calculate exact experiment carbon footprints, but report spending around $3,000 in LLM credits and designing and debugging Incalmo on smaller models to reduce environmental impact.They conclude that potential benefits of cheaper red teaming outweigh the research’s environmental impact.Source: LLM usage considerations · p. 15

Appendix A. Attack Graph Formalism and Log analysis

  • An attack graph represents attacker states, actions between states, initial states, and goal states; a successful path reaches every goal.Each action can contain commands, represented by their host, command name, parameters, and output.Source: Appendix A · p. 17
  • For each environment, the authors manually create a reference attack graph and a minimal command sequence, along with tasks and commands needed to reach all critical assets.The Equifax-inspired environment requires 246 tasks to obtain all critical data.Source: Appendix A · p. 17
  • Task achievement is assessed by matching generated-command outputs against reference-solution outputs, while the authors acknowledge that alternative solutions may lack the matched keywords.They manually review logs to check for such alternatives.Source: Appendix A · p. 17
  • Because detailed failure analysis requires substantial manual effort, the authors focus it on the Equifax-inspired and 4-Layer Chain environments, where ExpertPromptShell performed best.Source: Appendix A · p. 17
  • Commands are classified as irrelevant when their command name and host do not occur in the reference solution; potentially relevant commands are correct only if their parameters are correct and they have no syntax errors.The authors manually inspect irrelevant commands for possible alternate solutions.Source: Appendix A · p. 17

Appendix B. Environments

MHBench includes algorithmically generated small-enterprise networks and manually constructed environments based on real attacks and topologies. Its generated networks have validated paths from the attacker host to each goal.

Appendix B. Environments

  • Thirty MHBench environments are algorithmically generated with 2–4 subnets, 7–15 hosts per subnet, and goals assigned to 30% of hosts on non-external subnets.One subnet is designated external, and subnet connections are randomly assigned with bidirectional, all-traffic connectivity.Source: Appendix B · p. 17
  • Generated attack paths assign lateral-movement vulnerabilities or misconfigurations and privilege-escalation vulnerabilities, then a verifier checks for a valid path to every goal.Source: Appendix B · p. 17
  • The task-achievement heuristic can miss alternative ways to reach a state when they do not contain the matched keywords, so the authors manually review logs.Source: Appendix B · p. 17
  • Ten MHBench environments were manually created from real attacks and network topologies.The overview is identified as Table 4, whose caption does not provide environment-level values in the supplied text.Source: Appendix B · p. 18, Table 4 · p. 18
  • The Equifax-inspired environment has two web servers running vulnerable Apache Struts (CVE-2017-5638) and a separate network of 48 database hosts with fake critical consumer data.A random web server has a plaintext SSH configuration file containing credentials to all databases.Source: Appendix B · p. 18

Appendix C. Token usage

Incalmo’s reported planning-LMM token usage varies substantially across environments, while autonomous red-team costs remain below $15 per attack.

Appendix C. Token usage

  • Planning LLMs used 3.5K–5897.1K input tokens across Incalmo runs.Source: Appendix C · p. 18
  • Planning LLMs used 0.2K–60.1K output tokens across Incalmo runs.Source: Appendix C · p. 18
  • Autonomous red teams cost $0–$15, which the authors describe as significantly cheaper than a human-led red team.Source: Appendix C · p. 18

Related papers

Loading 2501.16466v4…