PaperlayerView exact revision on arXiv

Source-linked AI summary

A Cyber Range Evaluation of Autonomous Network Incident Response Agents

Jakob Nyberg, Teodor Sommestad, Andrei Buhaiu, Joakim Loxdal, Pontus Johnson, Mathias Ekstedt

arXiv:2609.16541v1cs.CRcs.AI

TL;DR

The paper asks whether automated reinforcement-learning defenders can mitigate network intrusions more efficiently than rule-based agents while limiting availability costs. It trains policies with a MAL-based simulator and evaluates them in a cyber range containing adversarial and user agents. RL agents, especially the noise-augmented policy, achieve lower overall cost than the heuristic baseline, but results depend strongly on attacker policy and user presence and remain insufficient for practical deployment.

  • Problem

    The paper asks whether reinforcement learning can perform automated network defense more efficiently than a manually crafted rule-based agent.

  • Method

    The study trains RL policies in a MAL-based simulator and evaluates them against Lore on a 37-virtual-machine cyber-range network with simulated users.

  • Results

    The noise-augmented RL agent achieves the lowest overall average cost, while returns depend strongly on Lore’s policy and, to a lesser extent, simulated users.

  • Takeaways & Limitations

    RL shows promise for automated cyber defense and can outperform the heuristic policy on joint attack and defense cost.

Abstract

from arXiv · show

We test the performance of agents for automated network intrusion response in a cyber range intended for human operator training. The range implements an emulated networking environment with a variable network topology, red-team emulation and simulated user agents. The goal of the defensive agents is to prevent hosts in the network from being accessed by the red-team agent, while minimizing the availability costs induced from defensive measures. Alerts are generated using a SIEM platform and mapped to a data modeling language used by the agents. We test a combination of heuristic agents and policies learned using reinforcement learning. The learned policies are optimized to minimize the combined cost using a cyber attack simulator modeling the network. We found that the reinforcement learning agents were overall more efficient at defending the system than the heuristic policy, and that the performance depends highly on the policy of the adversary in combination with the simulated users.

I. INTRODUCTION

The paper evaluates whether reinforcement-learning defender agents can mitigate network attacks more efficiently than manually crafted rule-based agents. It uses a sim-to-real workflow and a cyber-range network with adversarial and simulated user agents.

  • The study tests whether reinforcement learning can defend a network more efficiently than a manually crafted rule-based agent.
  • The target system is ADS-24, a 37-virtual-machine network attacked by Lore while simulated users perform additional tasks.
  • RL agents are trained in simulation from a MAL model of the target system, then evaluated on the target network to test sim-to-real performance.
  • RL agents consistently outperform the heuristic agent on joint cost, while the heuristic blocks Lore but incurs high availability costs by acting on all alerts.
  • The experiments produced public artifacts including the MAL model, simulator interoperability library, event-mapping interface, and network data.

II. MOTIVATION & RELATED WORK

The paper addresses the sim-to-real challenge in autonomous cyber defense by combining generalizable system modeling with configurable simulation and target-network evaluation. Its design emphasizes testing across variations in topology and adversary behavior.

  • The design requires agents to generalize across problem variations, motivating choices in modeling, simulation, and evaluation.
  • MAL and CoreLang provide a general network-and-attack modeling framework with existing simulation tooling for constructing the system model.
  • The simulator must encode both the defense task and system model while allowing variation in network topology and attacker policy for in- and out-of-distribution tests.
  • The evaluation flow trains a policy in simulation and transfers it through an interface that maps network events and actions between the target system and agent model.

C. Agent Architectures for Automated Cyber Defense

The paper situates defender-agent architectures within cyber-range evaluation, emphasizing graph-based representations and environments that balance fidelity, scalability, control, and execution cost. It uses a cyber range previously designed for human training.

  • C. Agent Architectures for Automated Cyber Defense: Graph neural networks are selected because they encode topology, scale linearly, and support factored state spaces through the Vejde library.
  • D. Defender Agent Evaluation: Real-network testing offers higher fidelity but suffers from limited control, high costs, and difficulty establishing threat progress ground truth.
  • D. Defender Agent Evaluation: Cyber ranges provide representative responses through full software stacks, but typically prioritize scale and fidelity over control and can become complex.
  • D. Defender Agent Evaluation: The evaluation uses Crate, a cyber range for human cyber-defense training, rather than a generic network simulator.
  • D. Defender Agent Evaluation: Cyber-range exercises combine adversarial activity from human or automated tools with benign events and simulated user behavior.

A. The Meta Attack Language

The paper models networks and attack behavior with MAL and formulates defense as a partially observable sequential decision problem. Its relational, factored representation supports policies that can generalize across network instances.

  • A. The Meta Attack Language: MAL represents network systems with abstract asset types, relations, attack steps, causal dependencies, and defensive steps.
  • B. Markov Decision Processes: The defense task is modeled as an episodic POMDP in which actions minimize expected cost under partially observed states.
  • B. Markov Decision Processes: The formulation treats changing network configurations and adversary behaviors as a collection of related decision problems rather than one fixed MDP.
  • C. Vejde: Vejde converts first-order-logic facts into bipartite factor graphs, uses message passing for latent representations, and computes action probabilities from those factors.
  • C. Vejde: Vejde observes entity types rather than identifiers, representing hosts such as ap01 and ap02 as identical Application assets.
  • D. The Cyber Range Crate: Crate instantiates virtual-machine networks from descriptions and supports snapshots for restoring deployed network states.

IV. NETWORK & DEFENSE TASK MODELING

ADS-24 models a 37-machine network defense task in which agents protect an ongoing intrusion while minimizing operational costs. Wazuh provides event detection and active-response commands for shutting down or isolating hosts.

  • Defense task: The task runs for two hours, with defenders protecting the network against intrusion while minimizing operational costs.The interval is divided into discrete timesteps at 1/30-second frequency.
  • Network scenario: ADS-24 contains four connected subnets—CLIENT, DMZ, SRV, and SOC—and 37 machines based on an expert-defined maintenance-management scenario.
  • Alerting: Wazuh logs host events and issues commands when configured rules match entries from monitored log sources.Each host except “flightlogs” runs a Wazuh agent service sending events to the SOC database.
  • Active responses: The defensive interface provides commands to power off a machine or block traffic between a host and other subnets.Traffic blocking adds firewall rules on host “fw1”; SOC interactions remain available for Wazuh reporting and command delivery.

B. Network Activity

Network activity combines scheduled simulated-user behavior with Lore-generated adversarial activity, while CadsLang models network state, attack steps, defenses, and costs. The evaluation records attack and defense events through Wazuh-to-model mappings.

  • User activity: Simulated users exchange emails, access internal web interfaces, use remote desktops, open files, and occasionally execute administrative commands.
  • Attacker activity: Lore enters through “flightlogs”, which lacks a Wazuh agent, so defenders cannot fully expel the attacker from the network.
  • Attacker strategies: Guided Lore follows the fastest route toward “payroll”, whereas Exploratory Lore can pursue broader actions and usually compromises CLIENT machines.Guided attacks remain directed toward the DMZ; Exploratory paths are more dynamic.
  • Data modeling: CadsLang defines nine asset types, 49 attack steps, and two defense steps corresponding to Wazuh’s shutdown and connection-restriction commands.“notPresent” blocks both modeled attack vectors, while “restricted” blocks external access but permits internal-subnet traffic.
  • Data modeling: Osquery data constructs MAL instance models for simulation and observations, while pattern rules map Wazuh events to 11 attack steps using shared asset identifiers.
  • Cost function: Episode return is the discounted sum of costs, with attack and defense steps represented through mappings used to calculate the combined cost.The evaluation defines observed data access and defense activation from Lore and Wazuh interactions.

V. AGENT EVALUATION IN ADS-24

The ADS-24 evaluation tests defender agents across attack, topology, and observation conditions using simulator-trained policies and repeated two-hour episodes. Observation databases are built from Osquery and updated from Wazuh alerts before actions are executed through the range interface.

  • Evaluation design: The evaluation measures defensive-agent performance against Lore while varying simulated users, attacker strategies, and network topology.
  • Training: RL policies are trained in the MAL simulator on 2 million transitions spanning attacker policies, entry points, and topology variations.
  • Evaluation design: Each episode lasts two hours and begins from a restored Crate snapshot after a settling period; evaluated agents include Vejde, noisy Vejde, and Heuristic.The noisy policy is trained with 1% false-positive and false-negative rates per attack step.
  • Observation pipeline: At episode start, Osquery data retrieved through Wazuh initializes a new observation database using the network instance-model construction procedure.
  • Observation pipeline: Every 30 seconds, new Wazuh events are mapped to MAL attack steps and appended to the observation database, merging repeated instances.
  • Action execution: Selected defenses are added to the observation model, mapped to active responses, and sent to Wazuh through its REST API; defended assets are removed from subsequent observations.If a removed asset later appears in an alert, it is temporarily reintroduced.

B. Scenario Variations

The evaluation varied simulated users, attacker strategy, and network topology, revealing that attacker policy and user activity materially shaped defender returns. Across 134 episodes, the noise-trained RL agent achieved the highest average return, while cost trade-offs depended on how aggressively agents acted.

  • Scenario results: The “Vejde w/ Noise” agent received the highest overall average return, largely because it incurred lower defense and availability costs.It disabled fewer assets than the other evaluated agents.
  • Scenario results: The conservative “Vejde w/ Noise” strategy sometimes let Lore access the client net, whereas “Vejde” minimized attack cost but incurred the highest defense cost.The policies waited 94%, 85%, and 88% of the time for “Vejde w/ Noise”, “Vejde”, and “Heuristic”, respectively.
  • Attacker Strategy: Lore’s policy was the most significant factor affecting returns, with all agents performing better against “Guided” than “Exploratory”.The “Guided” configuration has Lore explore fewer machines.
  • Simulated Users: Simulated users generally reduced average returns, although their effect was absent for the (“Heuristic”, “Exploratory”) and (“Vejde w/ Noise”, “Exploratory”) combinations.Under “Exploratory”, Lore generates client-net alerts even when it cannot capture machines, resembling the alert conditions created by simulated users.
  • Network Topology: Removing hosts from the tested topology did not significantly affect any agent’s returns.The removed machines were “rootca”, “timereporter”, and “print”.
  • Cost trade-offs: When availability receives no weight, “Vejde” is best for most configurations; when availability dominates, “NoOp” is always best among the agents.The evaluation varies Rα = (1 − α)Ra + αRd across α ∈ [0, 1].

VI. DISCUSSION

The noisy RL defender performed best on joint attack and defense cost, but the paper emphasizes interpretability and operational limitations before deployment.

  • VI. DISCUSSION: The Vejde w/ Noise RL agent minimized joint attack and defense cost better than the heuristic agent against Lore.The heuristic blocked Lore but incurred higher availability cost by acting against all observed alerts.
  • VI. DISCUSSION: The learned RL policies cannot be directly interpreted in terms of the behaviors encoded by their policy functions.A decision-tree surrogate was attempted, but grounded inputs prevented it from fully representing the GNN policy.
  • VI. DISCUSSION: The return distributions compare defender behavior against Lore with and without simulated users, with higher returns indicating better behavior.Dashed lines mark median returns.

B. Modeling Improvements

The paper identifies mismatches between the cyber-range model and the target network, especially in false-alert behavior, observability, and reproducibility.

  • B. Modeling Improvements: ADS-24 false-positive rates averaged 1.7% without users and 3.3% with users, while the simulator used a uniform 1% rate.Observed rates also varied substantially by attack step.
  • B. Modeling Improvements: The MAL Simulator allows attack steps and false positives to persist only once or for one observation step, unlike repeated observations possible in the real network.The paper suggests counting or grouping repeated instances as alternatives.
  • B. Modeling Improvements: ADS-24 and Lore cannot be shared publicly, but evaluation data and source code for several components were released to support transparency.The offline interface has limited functionality when using saved Wazuh data.
  • B. Modeling Improvements: The evaluation found the RL agent with added measurement noise had the lowest overall average cost, while cost depended strongly on attacker policy and somewhat on user agents.The authors still describe practical deployment as requiring further work.

APPENDIX

The appendix documents how ADS-24 data were converted into MAL network models and how Wazuh alerts were mapped to MAL attack steps for inference.

  • APPENDIX: MAL network instances were generated from Osquery tables covering users, system information, operating-system versions, interfaces, and addresses.Missing tables were treated as missing model information.
  • APPENDIX: Network, connectivity, application, and identity assets were instantiated from CIDR ranges, secondary connectivity data, interfaces, and user-account records.These assets represented host and network relationships in the MAL model.
  • APPENDIX: Because ADS-24 exposes only Wazuh logs, the evaluation required a mapping from Wazuh rules to MAL attack steps.The simulator instead directly maps attacker steps to defender observations.
  • APPENDIX: Wazuh and Sigma rules were manually classified using alerts collected from Lore and simulated users, introducing potential mapping bias.The full rule mapping was made available in the monitor repository.

C. Experiment Anecdotes

Experiment anecdotes expose operational and modeling failures that affected observability, attacker behavior, defensive actions, and network restoration during the cyber-range evaluation.

  • C. Experiment Anecdotes: Defender actions were assumed to succeed even though Wazuh queue saturation could cause active responses to fail without confirmation.Some actions powered machines off, making confirmation difficult.
  • C. Experiment Anecdotes: A defanged Lore instance initially made defender performance appear strong, so NOP-agent calibration rounds were added to verify attacker behavior.The incident demonstrates the value of simple baselines for debugging task difficulty.
  • C. Experiment Anecdotes: Simulated users could turn machines back on after defenders shut them down, while network loss required snapshots and manual restoration.These events illustrate interactions between user behavior and defensive actions, plus the need for recoverable system definitions.
  • C. Experiment Anecdotes: Background events occurred without agents and were often generated by Windows Defender or Microsoft Exchange processes.Such activity added benign event traffic to the range.
  • C. Experiment Anecdotes: A missing Wazuh agent on flightlogs made Lore’s activity there invisible and prevented fully removing Lore from the network.Lore could continue actions such as ping scans from that machine.
  • C. Experiment Anecdotes: The attack step ConnectionRule.attemptAccessNetworks matched no evaluation events, functionally producing a false-negative rate of 1.0.The authors had not otherwise estimated false-negative rates because an additional Lore-to-MAL mapping was required.

D. Additional Tables

This section collects additional tables and figures covering training settings, network asset priorities and host classes, alongside agent returns, attack modeling and simulator-state representations.

  • The supplementary tables document Vejde training hyperparameters, confidentiality, integrity and availability priorities, and network host classes.
  • A further figure compares defender returns against Lore with and without simulated user agents, where higher returns are better.
  • Additional figures show agent returns under varying attack-versus-defense cost priorities, with α = 0.5 denoting the current balance.
  • Other figures depict the MAL CadsLang attack-step graph, Lore compromise probabilities under Guided and Exploratory strategies, and initial simulator-state representations.
Loading 2609.16541v1…