PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Intrusion Response Strategies for OT Systems

Duc Huy Le, Rolf Stadler

arXiv:2609.10298v1cs.CRcs.AI

TL;DR

OT cyberattacks threaten industrial processes and essential services, motivating automated intrusion response. The paper formulates an OT use case as a POMDP with traffic-based partial observability and develops PPO-based learning methods. In an emulated OT system, the learned strategies are effective for the studied MITRE attack types, with BF-PPO approaching a full-observability baseline.

  • Problem

    OT cyberattacks can disrupt industrial processes and essential services, while existing response approaches motivate automated strategies that handle partial observability.

  • Method

    The paper models the OT intrusion-response use case as a POMDP with traffic-based observations and proposes k-Obs-PPO and BF-PPO to learn defender strategies.

  • Results

    BF-PPO produces effective defender strategies in emulation, performing close to a full-observability strategy across three attack types.

  • Takeaways & Limitations

    Learning-based response strategies can be computed at reasonable cost and can approach full-observability performance for the studied OT use case.

Abstract

from arXiv · show

Cyberattacks against Operational Technology (OT) systems, which monitor and control industrial processes, pose an increasing threat to essential societal services. For this reason, developing automated intrusion response strategies is highly important. In this paper, we present a formal model of an OT intrusion response use case using the POMDP framework. It includes a realistic model of partial observability that is based on traffic measurements. This approach allows us to develop tractable, learning-based solution methods for automated intrusion response, which are based on PPO. We evaluate the obtained response strategies on an emulated OT system and find that they are effective against several types of MITRE attacks for the studied use case.

I. INTRODUCTION

The paper addresses rising OT cyber threats by modeling automated intrusion response as a partially observable decision problem and learning defender strategies with PPO-based methods. The methods use traffic-based observations and are evaluated in an emulated OT environment, where BF-PPO performs close to a full-observability strategy for three attack types.

  • Motivation: OT systems face increasing cyber threats whose consequences can disrupt essential services and affect communities beyond targeted organizations.The paper notes that OT attacks grew by more than 90% annually between 2019 and 2023, while ransomware attacks against industrial organizations increased by 87% in 2024.
  • Motivation: Existing OT guidance emphasizes proactive protection and monitoring-based controls, while earlier automated responses largely relied on expert-defined firewall or SDN rules.
  • Approach: The paper models an OT intrusion-response scenario as a POMDP, representing attacker and defender actions while accounting for partial observability from continuous network-traffic observations.The attacker seeks to disrupt industrial processes; the defender seeks to maintain normal operation and mitigate attacks.
  • Evaluation: BF-PPO performs close to a full-observability strategy on the emulation system for three different attack types and converges in about one hour on an Apple M3 Pro processor.
  • Approach: The proposed k-Obs-PPO and BF-PPO methods learn automated defender strategies rather than relying on rules or expert configuration.Both methods are based on PPO and are designed to address the large state and belief spaces in the formal model.
  • Evaluation: The POMDP uses a realistic partial-observability model based on online traffic measurements, with measurements collected from an emulation system to estimate its observation function.The obtained strategies are evaluated on that emulation system.
  • Limitations: The model assumes simultaneous discrete-time actions with immediate effects, does not establish generalization beyond the studied use case, and currently omits operational safety constraints.More realistic continuous-time dynamics and higher-granularity state modeling would increase computational complexity.

II. RELATED WORK

Prior OT intrusion-response research includes rule-based automation, heuristic reinforcement learning, and formal decision models, but often assumes full observability or leaves real-system observation modeling unspecified. This paper combines a formal OT model, learning-based response methods, realistic partial observability, and emulation-based evaluation.

  • Existing approaches: Rule-based OT response methods automate firewall or software-defined-networking actions, but human experts define and maintain the response strategies.
  • Existing approaches: Heuristic reinforcement-learning methods learn from measurements without formal system models, limiting assessment of how closely their strategies approach an optimal strategy.
  • Formal models: Formal intrusion-response studies use MDPs, POMDPs, or stochastic games, but many assume that defenders can observe attacker actions and progress.The paper identifies full observability as unrealistic for the studied setting.
  • Formal models: Some prior studies use partial observability but rely on abstract observation models without explaining how those models are obtained for real systems.
  • Positioning: The paper combines a formal OT intrusion-response model, learning-based automated-response methods, realistic partial observability, and evaluation in an emulated OT system.
  • Use case: The studied infrastructure follows the Purdue architecture, partitioning enterprise, supervisory, control, and physical subnets.The supervisory subnet contains HMIs and engineering workstations, while PLCs in the control subnet manage water tanks in the physical subnet.
  • Use case: The attacker starts with an enterprise-subnet foothold and seeks to disrupt tank processes through reconnaissance, exploitation, and inspection of supervisory hosts before interacting with PLCs.
  • Use case: The defender receives IDS network measurements rather than direct attacker-progress information and can reset supervisory hosts, industrial processes, or both supervisory and control hosts.Traffic measurements also reflect normal process operations, complicating their interpretation.

B. A POMDP Model of the Intrusion Response Use Case

The POMDP models attacker progression, industrial-process integrity, defender actions, and IDS-derived observations in a discrete-time OT intrusion-response scenario. Its cost function balances process safety, attacker progression, and disruption from defensive resets.

  • System state: The system state combines supervisory-host states, industrial-process states, and the attacker action from the previous time step.The state space is defined as S = Sh3 × Sp2 × A, with an initially intrusion-free state distribution.
  • Host and process states: Host states U, D, S, E, and I represent progression from undiscovered through discovery, scanning, exploitation, and PLC-control inspection.These stages are motivated by MITRE ATT&CK and describe the attacker’s progression on each supervisory host.
  • Host and process states: Process states W and C indicate whether each water tank is operating normally or has been corrupted by the attacker.Process tampering requires an inspected host and is prevented when the defender resets the process at the same time.
  • Defender actions: The defender’s action space includes doing nothing, resetting a supervisory host, resetting a process, or resetting the entire system.These actions are represented by target host and process components in the defender action tuple.
  • Observations and objective: The defender cannot observe the system state directly and instead receives IDS-derived packet-count observations between enterprise, supervisory, and control subnets.The observation function gives the conditional probability of each next observation given the next state and defender action.
  • Observations and objective: The cost function prioritizes physical-process safety, penalizes attacker progression, and discourages unnecessary resets.Process corruption has cost σp(C) = 10, while action costs penalize disruptive defensive responses.

8) The defender’s optimization problem:

The defender’s optimization problem is to find a strategy that minimizes expected cumulative cost over the finite-horizon POMDP. An optimal strategy exists because the model has finite state, action, and observation spaces.

  • Optimization problem: The optimal defender strategy is defined as the strategy that minimizes expected cumulative cost J.The strategy may map histories or belief states to defender actions.
  • Optimization problem: An optimal defender strategy exists because the POMDP has finite state, action, and observation spaces over a finite horizon.The existence claim follows from the stated finite-horizon POMDP conditions.

V. LEARNING DEFENDER STRATEGIES WITH REINFORCEMENT LEARNING

Dynamic programming can solve the defender’s optimization problem in principle, but the large state and belief spaces make direct PPO application and exact computation infeasible. The paper therefore proposes PPO-based solution methods to address this scale.

  • Motivation for PPO: Dynamic programming can solve the optimization problem, but the large state space makes such methods computationally intractable for this use case.The paper therefore parameterizes the defender strategy and learns an almost optimal strategy with PPO.
  • Motivation for PPO: A direct application of PPO is infeasible because both the state space and belief space are too large.The paper proposes two PPO-based methods that use different techniques to address this issue.

A. PPO with k latest observations: k-Obs-PPO

k-Obs-PPO learns a defender strategy directly from a fixed window of recent IDS observations instead of computing the belief state. The window length k trades temporal information against input size and learning complexity.

  • Observation-history input: k-Obs-PPO uses the k most recent IDS observations as input to the defender policy.The observation history is given by (Ot, . . . , Ot−k+1).
  • Observation-history input: The method trains from observation histories without explicitly computing the belief state.This provides a direct policy input based on recent measurements.
  • Choosing k: A larger k provides more temporal information but increases input dimension and learning complexity, whereas a smaller k simplifies learning.The choice of k also affects computational complexity and sampling efficiency.

B. Belief Filter Proximal Policy Optimization: BF-PPO

BF-PPO approximates the POMDP belief state with particles and uses factorized marginal beliefs to provide a compact input for learning the defender strategy.

  • BF-PPO represents the belief at time t with M sampled system states from a particle filter.The particle set is P_t = {s^(1)_t, ..., s^(M)_t}.
  • The approximate belief assigns each system state a relative frequency among the sampled particles.This replaces direct Bayes-filter computation with an empirical distribution over particles.
  • Because the full belief vector is high dimensional, BF-PPO computes marginal beliefs for each factor of the system state.The factorized representation addresses the learning complexity caused by the large state space.
  • Concatenated marginal beliefs provide the defender strategy with a compact input while preserving information about each system-state component.

A SIMULATION ENVIRONMENT TO TRAIN AND EVALUATE DEFENDER STRATEGIES

The authors build a containerized OT emulation environment to collect traffic measurements for the POMDP observation model and to train and evaluate defender strategies under realistic conditions.

  • The emulation environment supports both IDS-measurement collection for the POMDP observation model and realistic evaluation of defender strategies.A separate simulation environment generates POMDP episodes efficiently for training.
  • The target infrastructure is emulated with Docker containers orchestrated by ContainerLab, including enterprise, supervisory, PLC, process, attacker, defender, and IDS components.
  • VLANs enforce logical network segmentation, while router-container rules provide inter-subnet routing and firewalling; a mirrored switch port feeds the IDS.
  • Attacker commands vary targets, ports, probes, arguments, and payload sizes to capture diverse traffic patterns for the same action.
  • Each 30-second period combines an attacker action, a defender action, and IDS network statistics used as the observation data.The observation is represented as o = (g, o_h0, o_h1, o_h2, o_p0, o_p1).
  • 40000 periods of measurements are simplified into an observation model conditioned on attacker action because estimating the full state- and action-conditioned model would require approximately 10^8 observations.The resulting model is Ẑ(o, a) = Pr(O_t+1 = o | A_t = a), and Fig. 3 presents selected empirical action distributions.

C. Running the POMDP simulation

The study trains PPO-based defender strategies in simulation and evaluates them in emulation against opportunistic, explorative, and adaptive attackers, alongside threshold and idealized full-observation baselines.

  • Simulation episodes begin from a specified initial state, update through attacker and defender actions, sample observations from Ẑ, accrue costs, and terminate after T steps.
  • The evaluated learned strategies include BF-PPO, 1-Obs-PPO, and 4-Obs-PPO, trained in simulation before emulation testing.
  • The comparison includes a threshold-based defender and MDP-PPO, an idealized PPO baseline with full observation of system states and attacker actions.MDP-PPO represents a cost lower bound for the proposed PPO-based methods.
  • The attacker strategies are opportunistic, explorative, and adaptive, differing respectively in speed toward tampering, information gathering, and responses to defender resets.
  • Training uses four random-seed runs with 500 iterations, 100 episodes per iteration, and a time horizon of 100.
  • Each learned defender strategy is evaluated against each attacker strategy for 20 emulation episodes with time horizon 100.One defender–attacker evaluation takes about 17 hours on an Apple M3 Pro processor.

B. Evaluating the methods in the simulation environment

BF-PPO is the strongest proposed method in simulation, converging faster and at lower cumulative cost than k-Obs-PPO variants across all attacker strategies. Its performance remains close to the full-observability baseline, although a gap appears when evaluated in the more realistic emulation environment.

  • Simulation results: 1-Obs-PPO and 4-Obs-PPO converge to higher cumulative costs than BF-PPO and exhibit larger variation across attacker strategies.1-Obs-PPO tends to perform better than 4-Obs-PPO, suggesting that longer observation history adds insufficient information in this setting.
  • Simulation results: BF-PPO converges faster and to lower cumulative costs than the k-Obs-PPO variants for all attacker strategies.Its learning curves also outperform the Threshold strategy and remain close to MDP-PPO, the full-observability baseline.
  • Emulation results: In emulation, BF-PPO outperforms the k-Obs-PPO variants for all attacker strategies and has cumulative costs close to MDP-PPO.This agrees with the simulation results, where MDP-PPO represents full observability.
  • Emulation results: BF-PPO shows a performance gap between emulation and simulation because its strategy was optimized for the formal simulation model, which approximates the emulated system.The authors expect performance to be lower in emulation than in simulation.
  • Conclusion: Overall, BF-PPO is the strongest proposed method for the studied use case and considered attacker strategies.The methods learn effective defender strategies at reasonable cost and with low sample complexity, while BF-PPO approaches the full-observability baseline.

APPENDIX

The appendix identifies the emulated physical-component configuration, attacker-action implementation, training hyperparameters, and a belief-state-based cost-score formulation.

  • Table II documents the configuration of the emulated physical components.
  • Table III documents how attacker actions are implemented in the emulation.
  • Table IV lists the hyperparameters used to train defender strategies.
  • Equation (11) defines the cost score for each host and process at time t using belief state b_t.
Loading 2609.10298v1…