PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomous Penetration Testing using Reinforcement Learning

Jonathon Schwartz, Hanna Kurniawati

arXiv:1905.05965v1cs.CRcs.AIcs.LG

TL;DR

Automated pentesting could help address a shortage of skilled cybersecurity professionals, while model-based approaches require transition models that are difficult to keep current. This study built a simulator and tested model-free reinforcement learning, finding that it could identify optimal attack paths in the simulated networks, with applicability limited to relatively small networks and numbers of exploits.

  • Problem

    Automated pentesting could help address a shortage of skilled cybersecurity professionals, while existing approaches require transition models that vary across systems and change as exploits evolve.

    Source: §2.3 · p. 22, §5.1 · p. 73

  • Method

    The study built a fast, easy-to-use network attack simulator and used it to test Q-learning, a model-free method that learns action values from experience.

    Source: §4.1 · p. 48, §5.1 · p. 73

  • Results

    In simulation, RL found optimal attack paths across tested network topologies and configurations using network topology and available scans and exploits, without prior exploit-outcome models.

    Source: §5.1 · p. 73

  • Takeaways & Limitations

    The learned policies could help identify network components for patching, and the same RL algorithm applied across different topologies and configurations in simulation.

    Source: §4.7 · p. 71

    The implemented algorithms were limited to relatively small networks and numbers of exploits; experiments covered networks of up to 16 machines and five exploitable services.

    Source: §4.6 · p. 69, §5.1 · p. 73

Abstract

from arXiv · show

Penetration testing (pentesting) involves performing a controlled attack on a computer system in order to assess it's security. Although an effective method for testing security, pentesting requires highly skilled practitioners and currently there is a growing shortage of skilled cyber security professionals. One avenue for alleviating this problem is automate the pentesting process using artificial intelligence techniques. Current approaches to automated pentesting have relied on model-based planning, however the cyber security landscape is rapidly changing making maintaining up-to-date models of exploits a challenge. This project investigated the application of model-free Reinforcement Learning (RL) to automated pentesting. Model-free RL has the key advantage over model-based planning of not requiring a model of the environment, instead learning the best policy through interaction with the environment. We first designed and built a fast, low compute simulator for training and testing autonomous pentesting agents. We did this by framing pentesting as a Markov Decision Process with the known configuration of the network as states, the available scans and exploits as actions, the reward determined by the value of machines on the network. We then used this simulator to investigate the application of model-free RL to pentesting. We tested the standard Q-learning algorithm using both tabular and neural network based implementations. We found that within the simulated environment both tabular and neural network implementations were able to find optimal attack paths for a range of different network topologies and sizes without having a model of action behaviour. However, the implemented algorithms were only practical for smaller networks and numbers of actions. Further work is needed in developing scalable RL algorithms and testing these algorithms in larger and higher fidelity environments.

Introduction

Pentesting can improve system security, but its expertise and time costs strain a shortage of security professionals. This section motivates automation and introduces reinforcement learning (RL) and a simulator for investigating it.

Introduction

  • Pentesting is an authorized controlled attack used to identify exploitable security vulnerabilities, but its time and skill requirements contribute to unmet demand for security professionals.The introduction describes cybersecurity as increasingly important as organizations and individuals rely on connected systems.Source: Introduction · p. 15

2.2 Tools used for penetration testing

  • Pentesting tools such as scanners and Metasploit improve efficiency, but assessments still require trained users and become harder to perform systematically as systems grow more complex.Scanners gather network and vulnerability information; Metasploit lets users search for and launch exploits.Source: Introduction · p. 16, §2.2 · p. 20

Introduction

  • RL is investigated for pentesting because action outcomes are difficult to model as cyber networks and exploits change, while learning requires substantial simulation data.The study develops a simulator for training and testing pentesting agents, then investigates RL policies in simulation.Source: Introduction · p. 17

2.1 Penetration testing

  • Pentesting gathers information to find vulnerabilities, exploits them, and repeats the process until the desired target is reached.The sequence can vary between attacks, but the general steps remain the same across systems and networks.Source: §2.1 · p. 19

2.2 Automated pentesting: attack graphs

Prior approaches automate pentesting through attack graphs, MDPs, and POMDPs, each representing aspects of attacks and uncertainty. Their practical constraints include complete network knowledge, required outcome probabilities, or poor scalability.

2.2 Automated pentesting: attack graphs

  • Attack graphs represent system states as nodes and known exploits as edges, enabling searches for attack paths but requiring complete network and machine-configuration knowledge.They also require manual construction for each system being assessed.Source: §2.2 · p. 21

2.3 Automated pentesting: using MDP

  • An MDP frames pentesting with network configurations as states, scans and exploits as actions, and rewards based on action cost and the value of compromised systems.One approach models uncertainty through action success probabilities, but assumes prior knowledge of those probabilities and treats machines as identical.Source: §2.3 · pp. 21–22

2.4 Automated pentesting: modelling uncertainty with POMDP

  • A POMDP represents uncertain system configurations as a distribution over possible states and models observations from scans or exploits.In pentesting, observations can include open ports or exploit success or failure; the approach can model incomplete attacker knowledge but scales poorly as the state space grows.Source: §2.4 · pp. 22–23

2.5 Automated pentesting: when the world model is unknown

The section motivates model-free reinforcement learning for pentesting because maintaining accurate exploit models is difficult and existing training environments are lacking. It presents a lightweight network attack simulator that frames pentesting as an MDP, with states for attacker knowledge, scan and exploit actions, and rewards tied to machine value and action cost.

2.5 Automated pentesting: when the world model is unknown

  • Maintaining accurate exploit-effect models is difficult as attacks and systems evolve, while reinforcement learning can learn policies through interaction without a transition model.Learning requires many environment interactions, making rapid simulation useful.Source: §2.5 · pp. 23–24

3.1 Introduction

  • A lightweight, open-source network attack simulator addresses the lack of a freely available benchmark for developing and comparing automated pentesting algorithms.It is designed to run fast on a personal computer and support rapid prototyping at a higher level of abstraction.Source: §3.1 · pp. 25–26

3.2 Design

  • The simulator separates a network model of machines, services, topology, and firewalls from an environment that tracks attacker knowledge and position as an MDP.Actions are scans and exploits; rewards reflect compromised-machine value minus action cost.Source: §3.2 · p. 26, §3.2.2 · p. 31, MDP overview · p. 31

3.2.1 The network model

  • The network model abstracts away low-level connection details so agents can choose which scans or exploits to use against which machines and in what order.Subnets group communicating machines, while topology and firewall settings constrain inter-subnet communication.Source: §3.2.1 · p. 26, Subnetworks · p. 27, Topology · p. 27

Machines

  • Machines carry user-assigned values and service configurations, with services representing exploitable vulnerabilities and exploits characterized by success probability and cost.The model permits machine-specific service configurations.Source: Machines · p. 28, Services · p. 29

State space

  • The environment state records compromise, reachability, and known, absent, or unknown services; its size grows exponentially with the number of machines and exploitable services.The stated state-space size is O(3^(|E||M|)).Source: State space · pp. 31–32

Action space

  • Agents can scan to learn service presence or use service-specific exploits whose outcomes depend on reachability, service presence, firewall access, and exploit success probability.Exploit actions may be deterministic or non-deterministic, and actions can carry configurable costs.Source: Action space · p. 33

Reward

  • Transition rewards value newly compromised machines and subtract action costs, while successful actions update the attacker’s network knowledge and position.The transition outcome depends on action success, connectivity, firewalls, target service, and—in non-deterministic exploits—success probability.Source: Reward · pp. 33–34, Transition function · p. 35

3.3 Implementation

The simulator provides an agent-facing environment interface and tools for viewing networks and attack episodes. Users can load custom configurations or generate benchmark networks with specified sizes and modeled service, exploit, and firewall properties.

3.3 Implementation

  • The Environment module provides load, reset, step, and render functions, with step returning the next state, reward, and whether the goal was reached.A training episode repeats action selection and stepping until the goal is reached or the time limit expires.Source: §3.3 · pp. 35–36

Configuring a new environment

Generating an environment

  • Generated benchmark networks use a DMZ, sensitive subnet, and user subnets arranged as a binary tree, with correlated machine configurations sampled using a Nested Dirichlet Process.Default exploit success probabilities are 0.2, 0.5, or 0.8 for low, medium, or high attack complexity; firewall permissiveness is configurable.Source: Generating an environment · pp. 40–42

3.4 Results

The simulator’s load time rose with network size, while its action throughput remained high across the tested configurations. Even the largest tested network loaded in seconds and processed actions much faster than the VM comparisons.

3.4 Results

  • 0.0007 ± 0.00007 sec at 10 machines and 10 services, versus 3.557 ± 0.06 sec at 1000 machines and 1000 services.Mean load time increased linearly with the number of machines and services.Source: §3.4 · p. 43, Figure 3.4.1 · p. 43
  • 0.249 ± 0.023 sec to load one headless VM, over 300 times the NAS load time for 10 machines and 10 services.The VM measurement was averaged over 10 runs.Source: §3.4 · p. 43
  • 17329 ± 1907 actions per second at 480 machines and 10 services, versus 126383 ± 3843 actions per second at 10 machines and 30 services.The tested range was 10–480 machines and services; results were averaged over runs.Source: §3.4 · pp. 43–44, Figure 3.4.2 · p. 44
  • A one-port Nmap scan averaged 0.253 ± 0.03 sec, while launching the tested exploit until a shell opened took 3–5 seconds.The exploit timing was estimated externally rather than measured exactly.Source: §3.4 · pp. 44–45

3.5 Discussion

The NAS was designed as an easy-to-deploy, fast testbed for automated pentesting. Its speed supports interactive agent training, while its abstraction trades real-world fidelity for versatility.

3.5 Discussion

  • At 1000 machines and 1000 services, NAS load time was approximately 3.5 sec, compared with roughly 0.25 seconds to launch a single VM.The authors state the worst-case NAS load time is far below the time an agent would spend training, and that rapid loading can speed training by enabling more data collection.Source: §3.5 · p. 45
  • The simulator processed around 17,000 actions per second in its worst tested case, versus roughly 4 Nmap scans and 0.3 exploit actions per second for a two-VM network.Performance decreased with more machines; increasing services could improve throughput when networks had more machines because fewer exploit actions succeeded and required state updates.Source: §3.5 · p. 45
  • NAS offers arbitrary services and exploits and networks of any size, while higher-fidelity lightweight virtualization requires more configuration information.The authors frame this as a trade-off between fidelity and versatility, with versatility useful during early interactive-agent development.Source: §3.5 · p. 46

3.5 Conclusion

The NAS is an easy-to-deploy, fast, flexible testbed for developing automated pentesting agents. Its abstraction supports rapid prototyping and algorithm studies, but higher-fidelity systems are needed before real-world use.

3.5 Conclusion

  • NAS provides an easy-to-deploy, fast, flexible penetration-testing testbed for developing automated agents.The conclusion identifies rapid agent-design prototyping and studying algorithm performance as use cases.Source: §3.5 · p. 46
  • The NAS abstracts away real-environment complexity in exploits, services, machine configurations, and network dynamics.This limits its fidelity to real environments.Source: §3.5 · p. 46
  • The authors identify higher-fidelity systems as a necessary next step before autonomous agents can be used in real-world applications.Source: §3.5 · p. 46

4.1 Introduction

Automated pentesting could reduce reliance on skilled practitioners, but model-based approaches face the challenge of keeping exploit-outcome models current. The study investigates model-free reinforcement learning in a simulated environment, using Q-learning to learn attack policies through interaction.

4.1 Introduction

  • Model-free reinforcement learning learns a policy through environment interaction without requiring a transition model.The paper motivates this approach by the difficulty of maintaining accurate, up-to-date models of exploit outcomes as systems and exploits change.Source: §4.1 · pp. 47–48
  • The study uses Q-learning, which learns state-action Q-values from experience and selects the action with the highest Q-value to derive a policy.Given enough time and exploration, the algorithm is stated to converge on optimal Q-values.Source: §4.1 · p. 48
  • The study frames pentesting as an MDP with an unknown transition model and uses RL in simulation to generate attack policies.Source: §4.1 · p. 48
  • The study investigates whether RL can find attack paths, their optimality, scaling performance, and generality using the NAS.Source: §4.1 · p. 48

4.2 Background

Reinforcement learning updates action-value estimates through interaction with an environment. The section introduces Q-learning and contrasts tabular and function-approximation value functions.

Reinforcement learning algorithms

  • RL agents iteratively update Q(s, a) from action, environment, and reward experience, beginning from an initial policy.Algorithms differ in action selection, value updates, and the form of the value function.Source: §4.2 · p. 49
  • ε-greedy selects a random action with probability ε and the current best action otherwise, while UCB adds exploration for less-visited actions.The study uses both strategies to compare action selection in automated pentesting.Source: §4.2 · pp. 49–50
  • Q-learning is an off-policy temporal-difference method whose update uses step size α and discount factor γ to weight immediate versus future rewards.Its state-action values converge to optimal values as visits to each state-action pair approach infinity.Source: §4.2 · p. 50
  • Tabular methods store values for each state-action pair, whereas function approximation estimates them with a learned function.Tabular methods are limited by state size and do not generalize across similar states; function approximation can generalize to unseen states but depends strongly on its representation.Source: §4.2 · pp. 50–51

4.3 Penetration testing using Reinforcement learning

The study frames pentesting as a model-free MDP with known network topology and compares tabular and neural-network Q-learning. Experiments vary network design, size, and available exploits under a fixed training and evaluation procedure.

4.3 Penetration testing using Reinforcement learning

  • Pentesting is represented as an MDP with machine configuration knowledge as states, scans and exploits as actions, and compromised-machine value minus action cost as reward.The transition model is left unknown, while the agent is assumed to know network topology.Source: §4.3 · pp. 51–52

Reinforcement learning algorithms

  • The study compares tabular Q-learning with ε-greedy and UCB action selection against single-layer neural-network Q-learning with ε-greedy selection.The DQL implementation uses experience replay and a separate target network.Source: Reinforcement learning algorithms · pp. 52–54

4.4 Experiments

  • Experiments ask whether RL finds attack paths, how reward-optimal they are, how performance scales with machines and exploits, whether it generalizes across configurations, and how algorithms compare.The experimental procedure trains agents on chosen scenarios and evaluates their final policies.Source: §4.4 · pp. 55–56

Experiment scenarios

Training details

  • Each agent is trained for two minutes per scenario, with ε-decay used for tabular ε-greedy and DQL agents.The initial and final ε values are 1.0 and 0.05, respectively.Source: Training details · pp. 59–60

Evaluation procedure

  • Trained policies are evaluated 10 or 30 times using ε-greedy action selection with ε = 0.05, alongside a uniformly random policy where applicable.Generated scenarios use 10 configurations, each evaluated 10 times, because performance varied across random configurations.Source: Evaluation procedure · p. 61

Experiment setup

  • Experiments ran on a single-core laptop with an Intel Core i7 7th Gen CPU at 2.7 GHz and 16GB RAM, using CPU-only neural-network computation.The implementations used Python, with Keras running on TensorFlow for DQL.Source: Experiment setup · p. 61

4.5 Results

In fixed network scenarios, all three RL algorithms reached approximately optimal reward, while their learning speed and evaluation consistency differed. Scaling tests showed different limits for tabular and DQL methods as network size and exploit count increased.

Custom Scenarios

  • All three algorithms converged to approximately optimal reward in the standard, single-site, and multi-site scenarios.For multi-site networks, DQL converged in about 100 episodes, compared with more than 1,000 for the tabular methods; in elapsed time it was still slower.Source: Figure 4.5.1 · p. 62, Custom Scenarios · pp. 62–63, Figure 4.5.2 · p. 63
  • Tabular methods completed up to 50 times more training episodes than DQL within the two-minute limit, despite DQL tending to learn faster per episode.Tabular ε-greedy completed 50 times more episodes than DQL in the worst case, and Tabular UCB completed 37 times more.Source: Custom Scenarios · pp. 62, 64, Table 4.5.1 · p. 64
  • Only DQL solved 100% of evaluation runs across all fixed scenarios, and its performance was the most consistent across scenarios.All algorithms achieved approximately optimal maximum reward, while random policies used more actions.Source: Custom Scenarios · pp. 64–65, Figure 4.5.3 · p. 65

Scaling

  • Tabular ε-greedy maintained near-optimal performance across tested exploit counts, while DQL performed comparably with random; network-size performance declined beyond algorithm-specific thresholds.For more than 18 machines, DQL and Tabular UCB declined rapidly; Tabular ε-greedy remained consistently high through 33 machines.Source: Scaling · pp. 67–68, Figure 4.5.4 · p. 67, Figure 4.5.5 · p. 68

4.6 Discussion

The study finds that RL can produce reward-optimal attack paths across tested configurations, but scaling is limited by network size and exploit count. Its results are bounded by small scenarios, limited topologies, and a fixed training budget.

Discussion

  • In the simulated environment, all tested RL algorithms found attack policies that compromised sensitive machines while minimizing actions and outperforming random policies in maximum reward.The authors conclude that RL found valid, reward-optimal paths across the tested network configurations.Source: §4.6 · p. 69
  • The experiments covered networks with 16 machines and five exploitable services and only three topologies, limiting how broadly the results apply.The authors note that real-world commercial networks may have hundreds of machines.Source: §4.6 · p. 69
  • Performance dropped rapidly beyond 18 machines for Tabular UCB and DQL, and beyond 38 for Tabular ε-greedy.The state space grows exponentially with machine count while training remained fixed at two minutes.Source: §4.6 · pp. 69–70
  • The authors suggest that more training time, more efficient function approximation, or decomposing topology and machine-level decisions could help address scaling.They describe hierarchical RL as a possible decomposition approach, while noting it remains an active research area.Source: §4.6 · pp. 70–71
  • Tabular methods handled more exploits relatively well, but DQL performed little better than random as exploit counts increased.The authors attribute DQL's decline to slower computation and exploration in a larger state space.Source: §4.6 · p. 70
  • The authors judge the tested approach unlikely to scale well to large networks and many exploits, with tabular methods limited by machines and DQL by actions.They identify ε-greedy as the stronger tabular action-selection strategy across most tested scenarios.Source: §4.6 · p. 71

4.7 Conclusion

The conclusion presents RL as a preliminary approach that found optimal attack policies in simulation without prior exploit-transition models, while noting scalability hurdles.

4.7 Conclusion

  • In simulation, RL generated policies capable of successfully and optimally exploiting target networks.The conclusion describes this as a preliminary investigation.Source: §4.7 · p. 71
  • The same algorithm can apply across network topologies, configurations, and exploit sets without prior knowledge of exploit and network transition models.The stated prior knowledge is limited to the network and its configuration in the conclusion's account of the approach.Source: §4.7 · p. 71
  • Scaling with network machines and available exploits remains a hurdle for automated pentesting.The conclusion points to more efficient algorithms or decomposing the problem as possible directions for addressing it.Source: §4.7 · p. 71

5.1 Summary and conclusions

The work built a Network Attack Simulator (NAS) and used it to study RL for automated pentesting. RL found optimal attack paths in simulated networks, but practical use requires better scalability and higher-fidelity evaluation.

5.1 Summary and conclusions

  • The project had two stages: building a fast, easy-to-use Network Attack Simulator (NAS), then using it to study RL algorithms for automated pentesting.The project was motivated in part by a shortage of skilled cybersecurity professionals.Source: §5.1 · p. 73
  • In simulation, RL found optimal attack paths across network topologies and configurations using only network topology and available scans and exploits as prior knowledge.The authors contrast this with approaches that require accurate models of exploit outcomes.Source: §5.1 · p. 73

5.2 Possible future work

  • Practical use requires more scalable RL algorithms for modern networks and hundreds or thousands of exploits, followed by evaluation in realistic VM networks informed by organizational networks.The authors describe the study as preliminary and say these steps are needed before commercial use may be practical.Source: §5.2 · p. 74

Appendix A

Appendix A lists project code and describes its modules for the Network Attack Simulator, agents, and experiments.

Program listing

  • The project source code and results are available in a GitHub repository, and the code is written in Python 3.The project is divided into Environment, Agents, and Experiments modules.Source: Program listing · p. 75

A.1 Environment

  • The Environment module contains the Network Attack Simulator code, with Table A1 listing its files and their roles.The program architecture is shown in Figure 3.3.1 in Chapter 3.Source: §A.1 · p. 76, Table A1 · p. 76

A.2 Agents

A.3 Experiments

  • The Experiments module contains scripts for running experiments on the RL agents and NAS, with Table A3 describing its files.The appendix also names agent_perf_eval_plotter.py.Source: §A.3 · p. 78, Table A3 · p. 78, File · p. 78

Related papers

Loading 1905.05965v1…