Source-linked AI summary
CybORG: A Gym for the Development of Autonomous Cyber Agents
Maxwell Standen, Martin Lucas, David Bowman, Toby J. Richer, Junae Kim, Damian Marriott
TL;DR
ACO requires autonomous agents to act in adversarial cyber scenarios, where changing actions make static datasets unsuitable. CybORG provides an interface spanning simulation and emulation for developing agents; initial emulator tests report a 66% success rate and demonstrate the approach’s feasibility.
Problem
ACO requires agents to make autonomous decisions in adversarial scenarios, where varying adversary actions in complex environments make static datasets unsuitable for learning behavior.
Source: §1 · p. 1Method
CybORG is a work-in-progress ACO gym with a common interface for agent interaction across simulation and emulation, supporting simulation training and testing on real-world systems.
Source: §1 · p. 1, §4.2 · p. 3Results
139 successful emulator tests yielded a 66% success rate, and the authors report that the results demonstrate the approach’s feasibility.
Source: §5.2 · p. 5Takeaways & Limitations
CybORG could provide an easy-to-use toolkit, benchmark problems, and a pathway toward validating RL techniques on virtual infrastructure.
Source: §6 · p. 5The scenario had a significant number of failures transferring trained agents from simulation to emulation, sometimes associated with simulator-model or emulator-interface deficiencies.
Source: §5.2 · p. 5
Abstract
from arXiv · showhide
Autonomous Cyber Operations (ACO) involves the development of blue team (defender) and red team (attacker) decision-making agents in adversarial scenarios. To support the application of machine learning algorithms to solve this problem, and to encourage researchers in this field to attend to problems in the ACO setting, we introduce CybORG, a work-in-progress gym for ACO research. CybORG features a simulation and emulation environment with a common interface to facilitate the rapid training of autonomous agents that can then be tested on real-world systems. Initial testing demonstrates the feasibility of this approach.
1 Background
Autonomous Cyber Operations requires agents to defend systems through autonomous decisions, but adversarial behavior, complex environments, and rapidly changing actions complicate AI development. The paper presents CybORG as a gym for developing agents through simulation and real-world testing.
1 Background
- ACO is needed where security experts cannot cover every network and location or defenders cannot reliably access systems.The cited access barriers include unreliable communication channels and adversary action.Source: §1 · p. 1
- ACO combines adversarial decision-making, simulation’s reality gap, and rapidly changing environments and action sets.These features make AI approaches for ACO challenging to develop.Source: §1 · p. 1
- Changing adversary actions in complex environments make static datasets unsuitable for learning ACO behavior.AI Gyms instead support reinforcement learning through interaction with a simulated problem.Source: §1 · p. 1
- CybORG is designed to develop ACO agents through a two-stage process combining simulation with testing in real-world environments.The authors describe validated, refined AI Gyms as a promising tool for training cyber agents.Source: §1 · p. 1
2 Related Work
Existing cyber-security environments meet some ACO research requirements, but the authors report that none described provides all of them in one environment. Their requirements include realistic red and blue agents, rapid simulation training, and testing in emulation.
2 Related Work
- DETERlab emulates hosts and networks but relies on local hardware, limiting network size and making resets or reconfiguration time-consuming.VINE, SmallWorld, and BRAWL instead use cloud IaaS and virtualization to emulate larger enterprise-like networks; BRAWL can use CALDERA as a red-team agent.Source: §2 · p. 1
- GALAXY supports evolutionary-algorithm research on red agents blending into normal traffic, but is restricted to one VM per physical machine.It uses VM snapshotting for fast environment resets.Source: §2 · p. 1
- Insight simulates hundreds of hosts on one computer for offensive-agent training, but does not support defensive-agent training.Source: §2 · pp. 1–2
- CyberBattleSim models red-agent post-breach lateral movement, while FARLAND supports simulation training and emulation testing with network-traffic rather than host-based exploitation and defense.CyberBattleSim does not train defenders and is not connected to emulation; FARLAND is described as a possible complement for network-focused scenarios.Source: §2 · p. 2
- The reviewed environments collectively lack a single platform with realistic red and blue agents, rapid simulation training, and emulation-based testing and validation.The authors state that, to their knowledge, no described environment combines all these features.Source: §2 · p. 2
3 Overview of CybORG
CybORG is a modular environment for scenarios at differing levels of fidelity. It runs agents through repeated action-and-observation steps, with scenarios and actions defined for both simulation and emulation.
3 Overview of CybORG
- CybORG supports scenarios at different fidelity levels, either modeled as a finite state machine in simulation or implemented in virtual infrastructure in emulation.Source: §3 · p. 2
- CybORG runs a scenario in discrete steps: agents select available actions, receive observations of updated states, and continue until a step limit or an agent goal ends the run.Agents receive final state information at termination, and scenarios can be reset for further training or testing.Source: §3 · p. 2
- Each scenario and action has definitions for both modes: simulation actions are state transitions, while emulation actions are executable commands with parameters.Scenario definitions include information needed to simulate and system images needed to emulate.Source: §3 · p. 2
4 Design
CybORG defines scenarios, actions, observations, and simulation and emulation implementations to support agent interaction across levels of fidelity. Its design allows agents trained in simulation to be tested on virtual infrastructure.
4.1 Scenarios
- CybORG scenarios specify agents, actions, initial information, rewards, host configurations, and network connections.Scenario descriptions use YAML, with separate files for host images and actions.Source: §4.1 · pp. 2–3
4.2 Actions
- Actions are based on those available to cybersecurity professionals; simulation models their effects, while emulation implements them where possible.Agents interact with CybORG through the OpenAI interface [Brockman et al., 2016].Source: §4.2 · p. 3
4.3 Observations
- Observations include information about the updated system state, limited to what an agent could conceivably observe with its reconnaissance tools.The observation dictionary reports prior-action success as successful, unsuccessful, or unknown, and organizes other observations by host and state category.Source: §4.3 · p. 3
4.4 Simulator
- The simulator models scenarios as finite state machines and updates state according to action preconditions and effects.The state includes file and network-connection details to reduce divergence from emulator behavior, particularly actions succeeding in simulation when they would not on real systems.Source: §4.4 · p. 3
4.5 Emulator
- The emulator deploys virtual networks in AWS from scenario descriptions and uses actuator objects to execute actions through SSH or specialized tool handlers.It supports pre-deployed or snapshot-started VMs; action results are filtered and merged into a single observation, and multiple adversarial agents can act simultaneously.Source: §4.5 · p. 3
4.6 Summary
- CybORG’s stated capability is to train and test the same agent, using the same body of code, at different fidelity levels.Agents can be trained in simulation and validated on virtual infrastructure.Source: §4.6 · pp. 3–4
5 Experimentation
CybORG’s red agent was trained in simulation and tested in emulation, using a three-host penetration-testing scenario. The emulator results show successful transfers alongside failures that sometimes pointed to simulator or interface deficiencies for refinement.
5.1 Scenario
- The three-host, two-subnet scenario required a red agent to gain System access on the Windows Internal host through reconnaissance, exploitation, sessions, and pivoting.The hosts included an Ubuntu Gateway with open SSH and a Windows 2008 Internal server with SSH and SMB; the scenario provided actions including port scanning, SSH brute force, and MS17-010 exploitation.Source: §5.1 · p. 4
Agent Training Methodology
- The agent was trained in simulation with a Deep Q-Network and LSTM memory, using wrappers to convert observations and actions to formats suitable for Deep Q-learning.The action wrapper masks parameters the agent has not encountered, which the paper says improves realism and convergence rate.Source: Agent Training Methodology · pp. 4–5
5.2 Results
- Training ran for up to 2,500 iterations of at most 20 steps, stopping on System access; selected trained agents were then tested on virtual hosts in emulation.A run counted as successful if System access was achieved within 10 steps; the emulator evaluation tested each of 21 agents 10 times.Source: §5.2 · p. 5
- Across 21 trained agents evaluated 10 times each, emulator testing yielded 139 successful tests (66% success).Figure 2 shows the distribution of successful runs across the trained agents.Source: §5.2 · p. 5, Figure 2 · p. 5
- Almost half of the trained agents succeeded on every emulator run, while four never succeeded; some transfer failures indicated simulator or interface deficiencies.One likely cause was overfitting to a simulation observation artifact, which led an agent not to use autoroute around the external firewall.Source: §5.2 · p. 5
6 Conclusion
CybORG is a work-in-progress gym integrating simulation and emulation for autonomous cyber operations research. Its simulation-trained agents can be tested in emulation, with results used to assess effectiveness and guide refinement.
6 Conclusion
- CybORG is a work-in-progress gym for Autonomous Cyber Operations research.Source: §6 · p. 5
- Its integrated simulation and emulation modes support large-scale reinforcement learning across diverse scenarios, followed by testing in emulation.The paper reports progress toward spawning and playing games in either mode with cloud infrastructure.Source: §6 · p. 5
- Testing can validate agent effectiveness and guide refinement of the simulator model or emulator interface; gyms like CybORG could also provide tools, benchmarks, and a validation pathway.Source: §6 · p. 5
7 Future Work
Future work focuses on extending red-agent training to blue agents and developing a more complex scenario with potential for blue-team deception.
7 Future Work
- The reported experiments train red agents; future work would implement the actions, sensors, and actuators needed for the same training with a blue agent.Source: §7 · p. 5
- Another future-work priority is a more complex scenario that allows blue agents to employ deception.Source: §7 · p. 5
- The paper’s initial results concern red-agent training, not blue-agent training.Source: §7 · p. 5