Deep hierarchical reinforcement agents for automated penetration testing

1 Introduction

• AI-based penetration testing is presented as a way to simulate attackers with strategies rather than test every possible value.The motivation is that existing automatic tools cannot learn new strategies in response to simple defensive measures.Source: §1 · p. 1
• The paper frames reinforcement learning as suitable for automating penetration testing because agents learn policies by exploring and exploiting environmental dynamics.Source: §1 · p. 1
• Conventional deep Q-learning can become difficult and unstable in penetration testing because the discrete action space can grow to thousands even in relatively small scenarios.The action space scales exponentially with the number of hosts across sub-networks.Source: §1 · p. 1
• HA-DRL decomposes the large action space into smaller subsets and trains individual reinforcement-learning agents on those subsets.The introduction describes the approach as inspired by multi-agent reinforcement learning and intended to support different network configurations.Source: §1 · p. 1
• The authors use the CybORG autonomous cybersecurity simulator to validate HA-DRL’s learning and convergence properties.The introduction says the paper reports empirical results for the architecture.Source: §1 · p. 1

2 Related Work

• Prior RL and DRL penetration-testing studies showed feasibility on small-scale networks or capture-the-flag challenges, while extending DRL to larger action spaces remained a challenge.The described PT formulations use a partially observable Markov decision process in which agents observe compromised hosts or subnets rather than the whole network.Source: §2 · p. 2
• Wolpertinger embeds actions using prior domain knowledge and nearest-neighbor clustering, but may be unstable in sparse-reward environments [Dulac-Arnold et al., 2015].Source: §2 · p. 2
• Action elimination uses rule-based negative signals to remove irrelevant choices, and is best suited to settings where expert rules are readily available [Zahavy et al., 2018].Source: §2 · p. 2
• The section contrasts PT’s entirely discrete actions and varied effects with approaches based on parameterized actions or correlated action groups [Tavakoli et al., 2018; de Wiele et al., 2020; Dulac-Arnold et al., 2015].It cites branches, learned representations, and action distributions as strategies for reducing large action sets.Source: §2 · p. 2
• HA-DRL uses hierarchically grouped agents trained on action subsets with a shared external reward, aiming to address large discrete PT action spaces with minimal domain knowledge.The authors also describe the method as model-free and potentially applicable where actions can be flattened into a large discrete set.Source: §2 · p. 2

3 Problem description

• In this formulation, the agent observes the compromised portion of the network, acts, receives a reward, and seeks to maximize cumulative reward in an MDP.The task is formalized by the tuple ⟨S, A, P, R, γ⟩.Source: §3 · p. 2
• The discount factor γ determines the importance assigned to long-term rewards.Source: §3 · p. 2
• The action-value function represents the expected utility of an agent’s action in a given state, and recursive optimization yields the optimal action value used in DQN.Source: §3 · p. 2
• The authors formulate PT as an MDP rather than a POMDP to demonstrate the architecture’s effect without compounding factors, while stating it can also serve as a feature in a POMDP solution.Source: §3 · p. 2
• DQN uses action values to choose actions, but a large action space can make it unstable when many actions have similar Q-values.The section notes that DQN’s performance and action selection are hindered as the action space increases.Source: §3 · pp. 2–3
• The proposed framework decomposes a flat action space into smaller sets and combines them using predefined linear functions.The approach targets large flat discrete spaces and flattened parameterized spaces.Source: §3 · p. 3
• The algebraic action representation builds larger action identifiers from smaller integer values through successive levels.The passage describes the sets of action representations as growing with each level.Source: §3 · p. 3

4.1 Action decomposition scheme

• HA-DRL decomposes a large discrete action space into smaller sets, whose primitive action outputs are linearly combined into the final action.Each agent can train its own network using the external reward signal.Source: §4.1 · p. 3

4.2 Hierarchical Agents Deep Reinforcement Learning (HA-DRL)

• Separate DQN agents learn action subsets from a shared reward; sufficient exploration can make their composed policy optimal, though sparse rewards or enormous spaces may make this impractical.The agents use smaller networks, and their training and inference can run in parallel while action composition remains sequential.Source: §4.2 · p. 4
• Current empirical results show faster training and better convergence than a single DQN with a combined neural network.The paper cautions that sparse rewards or extremely large action spaces may make the required exploration impractical.Source: §4.2 · p. 4

4.3 Interpretability of action decomposition

• Linear range-shifting chains one agent per level, requiring only a handful even for millions of actions, with the number of levels growing about logarithmically with action-space size.The paper likens the decomposition to traversing a tree of action-selection nodes.Source: §4.3 · pp. 4–5

4.4 Optimality of action value function

• Global optimality is not guaranteed when one agent’s action affects another’s reward or exploration is limited, although the architecture handles medium-sized, frequently rewarded scenarios better than an individual DRL agent.Source: §4.4 · p. 5

5.1 CybORG simulator

• CybORG models an agent interacting with a network through observed-state vectors, where actions reveal information or penetrate hosts to capture flags.Scenarios vary in network topology, flag locations, host count, operating systems, and services.Source: §5.1 · p. 5
• The 24-host example offers 550 actions, while the action-space formulas scale with host and subnet counts across three action categories.Host-to-host actions scale as m × p × (p − 1), host-to-subnet actions as n × p × q, and on-host actions as o × p.Source: §5.1 · pp. 5–6
• Actions can fail when the agent lacks remote access or because exploits are probabilistic, even when the target is accessible.The observed state includes information such as host operating systems and listening services, with unknown values where the agent lacks knowledge.Source: §5.1 · p. 6
• The simulator is deliberately sparse-reward: flags are hidden from the agent, successful attacks earn small rewards, and capturing a flag earns a large reward.The flag location is not inferable from the state vector, unlike objectives visible in typical RL environments.Source: §5.1 · p. 6

5.2 Neural network architecture

• Each agent uses a three-layer Dueling DQN; HA-DRL limits individual agents to action spaces of 8–10 actions and uses at most four agents for scenarios up to 5,000 actions.The experiments use at most 100 hosts and 4,646 actions because of CybORG’s complexity.Source: §5.2 · p. 6

6.1 Policy training

• Across tested scenarios of 6–100 hosts, HA-DRL convergence was similar to or better than DDQN as action spaces grew from 49 to 4,646.The comparison varied with scenario complexity and action-space size.Source: §6.1 · p. 6
• In the 100-host scenario, DDQN could not explore the action space or learn a policy; HA-DRL began converging toward optimality after approximately 4,000 episodes and learned minimum-action policies in each tested scenario.Source: §6.1 · p. 6
• The paper describes this as the first demonstration of deep reinforcement learning in automatic penetration testing that handles such a large action space.Source: §6.1 · p. 6
• HA-DRL converged faster and more stably than DDQN when both learned the optimal policy; DDQN succeeded in only 1 of 4 runs at 60 and 70 hosts, and made no progress at 100 hosts.HA-DRL maintained performance at 100 hosts.Source: §6.1 · p. 6, Figure 6 · p. 7

6.2 Discussion

• In the 50-host scenario, learned state representations formed nine clusters matching private subnets, although subnet information was not observable to agents.The 1,326-action space mapped to subsets of 10–15 actions per agent, with 2–3 actions in each subset needed to optimally capture hidden assets.Source: §6.2 · p. 7

• HA-DRL showed superior performance to a single DDQN agent in all tested scenarios.Source: §7 · p. 7
• The architecture scales to larger action spaces with sub-linear growth in network computational complexity.Source: §7 · p. 7
• Separate networks are trained and used for inference, with their outcomes combined sequentially to compute the final action.Source: §7 · p. 7