Source-linked AI summary
Towards Hierarchical Cyber Defense with Large Language Models: From Planning to Execution
Harshith Doppalapudi, Nathaniel D. Bastian, Ankit Shah
TL;DR
Learned cyber defenders can require retraining when network configurations change, and hierarchical RL does not eliminate that dependence. This study evaluates frozen LLMs in a planner-executor hierarchy and finds that capable models controlling both levels perform strongly across network scales, whereas planner-only integration offers limited gains.
Problem
Learned defenders are typically tied to their training network configurations, so changes in network size or topology can require costly retraining and limit transfer across environments.
Source: §1 · pp. 1–2Method
A controller-agnostic planner-executor hierarchy allows frozen LLMs to be used at either level while preserving the same environment, reward, observation, and action interfaces.
Source: §1 · p. 2Results
Takeaways & Limitations
Across the evaluated network scales, effective LLM-based control depends on extending capability to tactical execution, and cybersecurity specialization alone does not guarantee scale generalization.
Source: §6 · p. 8The evaluation leaves generalization across multiple independent seeds, unseen topologies, attacker strategies, and multi-agent enterprise environments for future work.
Source: §6 · p. 8
Abstract
from arXiv · showhide
An autonomous cyber defender trained with reinforcement learning (RL) is typically tied to the network on which it was trained, limiting its ability to generalize as network scale changes. Hierarchical RL reduces decision complexity by separating strategic targeting from tactical execution, but it does not eliminate this retraining dependence. We investigate whether frozen, zero-shot large language models (LLMs) can provide retraining-free control in hierarchical cyber defense and how performance changes as LLM control is extended from planning to execution. We formulate a controller-agnostic planner-executor hierarchy in which the planner selects a subnet to defend over a fixed horizon and the executor selects defensive actions within that subnet. Using the high fidelity Cyberwheel environment, with its built-in automated red team agent mapped to the MITRE ATT&CK framework, we compare RL+RL, LLM+RL, and LLM+LLM configurations using six models ranging from 3B to 70B parameters, including two cybersecurity-specialized models, across small, medium, and large networks. Replacing only the planner with an LLM yields limited gains as network size increases. In contrast, extending LLM control to execution produces notable improvements for sufficiently capable models. For instance, a frozen general purpose 70B model holds successful lateral movement to approximately 1% of steps and attacker impact near zero across all three network scales using the same model weights, while the RL baseline is retrained for each scale. Our results show that sufficiently capable frozen LLMs can maintain strong defensive performance across the evaluated network scales without task-specific retraining, while also indicating that strong tactical execution is important to realizing the benefits of LLM-based control.
1 Introduction
Because learned defenders and hierarchical reinforcement learning remain tied to training environments, this study tests whether frozen LLMs can control planning and execution across network scales without retraining.
1 Introduction
- Hierarchical RL separates network-region targeting from defensive action selection, reducing joint decision-space complexity but not removing dependence on training environments.Changes to network size or topology can still require retraining, limiting transfer across evolving or heterogeneous networks.Source: §1 · pp. 1–2
- Prior work had not tested frozen LLM control in a two-level hierarchy across network scales or how performance changes when LLM control extends from strategic planning to tactical execution.The study also tests the assumption that an LLM planner paired with an RL executor would capture much of the generalization benefit.Source: §1 · p. 2
- Capable frozen LLMs controlling both planning and execution achieve stronger cross-scale performance than planner-only LLM integration, while smaller and some specialized models do not maintain robustness as networks grow.Llama-3.3-70B maintains compromise rate at approximately 1% across all three network sizes and drives attacker impact to nearly zero using the same weights without retraining.Source: §1 · p. 2
- The controller-agnostic planner-executor framework compares RL+RL, LLM+RL, and LLM+LLM using six 3B–70B models across a 67-fold increase in network size.The framework allows frozen LLMs to be interchanged at either level while preserving environment, reward, observation, and action interfaces.Source: §1 · p. 2
2 Related Work
Prior work provides cyber-defense environments, hierarchical RL approaches, and LLM applications, but does not establish frozen LLM control across network scales within a planner-executor hierarchy.
Autonomous cyber defense environments
- Cyberwheel is a configurable simulation and emulation platform for controlled evaluation of autonomous cyber defense across network scales.Its settings include network, attacker, observation, action, and reward conditions.Source: §2 · p. 3
Hierarchical reinforcement learning for cyber defense
- Hierarchical PPO cyber defense uses a master policy and trained sub-policies to manage large policy spaces and improve performance and convergence across multiple adversaries.Prior work also reports transfer and fine-tuning of learned sub-policies when network or adversary conditions change.Source: §2 · p. 3
Large language models for cyber defense
- Prior LLM cyber-defense studies examined mixed LLM/RL teams or LLM-generated rewards, but not frozen LLM control across network scales in a two-level planner-executor hierarchy.This study compares planner-only LLM integration with LLM control extending to tactical execution under common hierarchical interfaces.Source: §2 · p. 3
3 A Controller-Agnostic Hierarchical Control Architecture
The architecture separates subnet-level strategic targeting from stepwise tactical actions while allowing RL policies or frozen LLMs to occupy either controller role. A common decision process and reward support comparisons across controller configurations.
3.1 Defense as a Partially Observable Decision Process
- The defender is modeled as a POMDP with hidden network state and noisy detector-derived observations, using current observations as a practical approximation to full history or belief-state policies.The state includes network topology, host compromise, attacker progress, and defender modifications.Source: §3.1 · p. 3
3.2 Planner-Executor Decomposition
- The planner selects a subnet as the strategic goal, while the executor chooses local defensive actions; the goal is refreshed every k steps and held between decisions.The executor acts at every environment step using the current observation and selected goal.Source: §3.2 · p. 4
3.3 Reward Structure
- A shared additive reward combines attacker progress, productive defense, defensive-action costs, and penalties for unnecessary disruption across all controller configurations.Attacker progress is weighted by kill-chain severity, while the same operational objective applies to RL+RL, LLM+RL, and LLM+LLM.Source: §3.3 · pp. 4–5
3.4 Controller Realizations
- The planner and executor expose common decision interfaces, allowing either role to use a trained RL policy or a frozen, zero-shot LLM without changing the hierarchy or reward.Frozen LLMs receive no task-specific parameter updates.Source: §3.2 · p. 4, §3.4 · p. 5
3.5 Experimental Instantiations
- RL+RL uses PPO-trained planner and executor policies; LLM+RL replaces only the planner, while LLM+LLM also replaces the executor.The latter comparison tests the extension of frozen LLM control from strategic targeting to tactical action selection.Source: §3.5 · p. 5
- The controller-agnostic hierarchy progresses from trained RL at both levels, to an LLM planner with an RL executor, to LLM control of both planning and execution.The configurations retain the same decision process, reward, goal set, planner horizon, and environment interfaces.Source: §3.5 · p. 5
4 Experimental Setup
The experiments use Cyberwheel networks at three scales and a scripted ATT&CK-mapped attacker, comparing frozen LLM controllers with PPO baselines under shared interfaces. A matched-seed protocol evaluates scale changes and the extension of LLM control to execution, with compromise, impact, and defense rates as outcome measures.
4.1 Cyberwheel Environment and Threat Model
- Cyberwheel tests small, medium, and large networks against a scripted ATT&CK-mapped attack sequence, using detector-derived host and subnet observations and three primitive defensive actions.The networks contain 15, 100, and 1,010 hosts; the planner holds each subnet goal for five environment steps.Source: §4.1 · pp. 5–6
4.2 Controller Instantiation and LLM Models
- Six LLMs spanning 3B–70B parameters, including two cybersecurity-specialized models, are compared with PPO policies retrained for each network scale under shared interfaces.The comparison targets deployment behavior, not matched training data, compute, or sample efficiency.Source: §4.2 · p. 6
4.3 Evaluation Protocol
- Matched-seed evaluations use 20 episodes of 100 steps to test scale changes and extending LLM control to execution, but do not characterize variability across independent seeds.All generated LLM decisions parse successfully, with no heuristic fallback required.Source: §4.3 · p. 6
4.4 Evaluation Metrics
- Compromise rate tracks successful lateral movement, impact rate tracks successful terminal attacks, and defense rate counts failed or diverted red actions as a supporting rather than primary measure.Impact is the primary security outcome; defense rate can include intrinsic attacker failure rather than an explicit defender block.Source: §4.4 · p. 7
5 Results
Across network scales, the results distinguish limited gains from LLM planning alone from stronger performance when capable LLMs also control execution. The results also show that LLM control does not improve performance universally.
5.1 RL Baseline Performance Across Network Scale
- RL+RL compromise rises from 5.32% on small networks to 14.88% on medium and 17.95% on large networks, despite separate training at each scale.Impact likewise increases from 1.22% to 5.10% and 15.20%.Source: §5.1 · p. 7
5.2 LLM Planning Alone Provides Limited Gains
- On large networks, LLM+RL configurations remain at 10.75%–13.05% compromise and 8.40%–10.85% impact for all models except the 3B model.Larger planner models provide relatively limited additional benefit while the fixed PPO executor performs tactical actions.Source: §5.2 · p. 7
5.3 Extending LLM Control to Execution Enables Strong Cross-Scale Performance
- Compromise is 1.15%, 1.40%, and 1.00%, while impact is 0.10%, 0.05%, and 0.00% for Llama-3.3-70B across small, medium, and large networks.The same frozen weights achieve these results across all three scales without task-specific retraining.Source: §5.3 · p. 7, Table 1 · p. 8
- LLM control does not improve performance universally: Llama-3.2-3B underperforms RL at several scales, while Trendyol-Cybersecurity-70B degrades on large networks.Foundation-Sec-8B outperforms general-purpose Qwen3-8B at the largest scale; specialization can help at matched scale but does not guarantee cross-scale robustness.Source: §5.3 · p. 7
6 Conclusion and Future Work
The conclusion finds that capable frozen LLMs controlling both planning and execution achieve strong performance across the evaluated Cyberwheel scales, while planner-only integration provides limited gains. It also identifies evaluation gaps, including unseen topologies and attacker strategies.
6 Conclusion and Future Work
- Across Cyberwheel networks, capable LLMs controlling both levels achieve strong cross-scale performance, while planner-only integration provides limited gains at larger scales.Smaller and some cybersecurity-specialized models degrade as networks grow.Source: §6 · p. 8
- Adding pretrained reasoning only at the hierarchy’s top may be insufficient; effective tactical execution is important for translating strategic decisions into defensive performance.Source: §6 · p. 8
- Future work should evaluate multiple independent seeds and test generalization to unseen topologies, attacker strategies, and multi-agent enterprise environments.The authors also call for analyzing decision traces and evaluating the complementary RL+LLM configuration.Source: §6 · p. 8