PaperlayerView exact revision on arXiv

Source-linked AI summary

ROAM: Reinforcement Learning Optimized Adaptive Moving Target Defense for Smart Grid

Mohammad Zakaria Haider, Prabin Mali, Muhammad Nadeem, Sumit Paudyal, Fei Ding, Mohammad Ashiqur Rahman

arXiv:2610.12730v1eess.SYcs.SY

TL;DR

Stealthy false-data-injection attacks can evade conventional bad-data detection, while existing sensor-remapping defenses do not adapt to live threat evidence and face different key-security and latency constraints. ROAM uses a DQN to select sensor remappings and recovery modes at the aggregation layer. Across held-out test systems, stealthy attacks remained below 3% with detection above 93%.

  • Problem

    Existing sensor-remapping defenses use fixed or pre-scheduled randomization, and seed-based and ML-based approaches have different key-security and latency constraints.

    Source: §I · p. 2

  • Method

    ROAM uses a DQN to choose deception coverage, sensors to remap, and the seed-based versus ML-based recovery split at the PDC aggregation layer.

    Source: §I · p. 2

  • Results

    Across held-out IEEE 14-, 57-, and 118-bus scenarios, stealthy attacks stayed below 3% with detection above 93%.

    Source: §V · p. 12

  • Takeaways & Limitations

    The hybrid pathway confines the impact of a seed compromise to the seed-routed subset.

    Source: §V · p. 12

Abstract

from arXiv · show

Smart grids rely on distributed phasor measurement units (PMUs) and wide-area communication networks whose data-acquisition pipelines are vulnerable to stealthy false data injection (FDI) attacks that evade conventional bad-data detection (BDD). Moving target defense (MTD) counters this exposure by continuously remapping sensor identities, but existing schemes apply static or pre-scheduled randomization and are each constrained by a single failure mode: seed-based methods are fast yet fragile to key compromise, while machine learning (ML) based prediction methods are resilient yet incur prohibitive latency. Neither adapts its randomization to live threat evidence. This paper presents ROAM, a deep reinforcement learning (DRL) driven MTD framework in which a deep Q-network (DQN) agent continuously re-optimizes the sensor-to-ID mapping at the aggregation layer. At each decision step, the agent selects the deception coverage, flags the specific sensors to remap, and sets the split of the remapped sensors between seed-based and ML-prediction-based recovery, without modifying PMU hardware and while preserving IEEE C37.118 compatibility. On held-out BDD-compliant FDI scenarios across the IEEE 14-, 57-, and 118- bus systems, ROAM keeps the percentage of stealthy attacks below 3% while sustaining detection rates above 93%. Against an adaptive soft actor-critic (SAC) attacker that observes the defender's configuration, ROAM keeps stealthy attacks below 2.6% with detection above 92%. Hardware-in-the-loop testing on an OPAL-RT platform confirms real-time feasibility, with sub-18ms average shuffle execution and an average reshuffle blind window of 2.6% of the 3-s decision cycle.

I. INTRODUCTION

ROAM addresses the lack of threat-responsive sensor-ID remapping by using system feedback to choose coverage, sensors, and recovery pathways at the PDC aggregation layer. It combines seed-based and ML-based recovery and evaluates the defense against an adaptive attacker and on hardware.

I. INTRODUCTION

  • BDD-evading false-data-injection attacks can corrupt automatic generation control, while existing remapping schemes do not adapt their strategy to live threat evidence.The unresolved choices include how many sensors to remap, which to prioritize, and when to use prediction-based rather than lightweight shuffling.Source: §I · pp. 1–2
  • Seed-based methods achieve sub-2 ms recovery but depend on shared keys, whereas ML-based matching raises latency by roughly 60×.Both approaches use fixed or pre-scheduled randomization rather than adjusting the remapping strategy to live threats.Source: §I · p. 2
  • At each decision epoch, a DQN selects deception coverage, sensors to remap, and the split between seed-based and ML-based recovery at the PDC.ROAM requires no PMU hardware modification and preserves IEEE C37.118 compatibility.Source: §I · p. 2
  • The hybrid architecture assigns remapping modes per sensor, so a seed compromise affects only the seed-routed subset.The DQN’s factored action grows linearly with PMU count, and its reward reflects detection, deception cost, latency, and attacker accuracy.Source: §I · p. 2
  • Sub-18 ms average shuffle execution and a 2.6% average reshuffle blind window were demonstrated in OPAL-RT hardware-in-the-loop testing.The test used sim-to-real transfer of the trained policy without retraining.Source: §I · p. 2

II. TECHNICAL OVERVIEW

  • ROAM is a reinforcement-learning hybrid MTD technique at the PMU data-aggregation layer, where a threat model guides deception strategies.The system links PMUs to PDCs, which synchronize measurements before forwarding them to the EMS for state estimation, bad-data detection, and AGC.Source: §II · p. 2, A. Hierarchical Control Architecture · p. 2

A. Hierarchical Control Architecture

  • A sliding-window reconciliation protocol retransmits an unacknowledged seed after 50 ms and activates ML recovery for affected sensors until synchronization returns.The seed-based pathway relies on a time-varying seed shared between PDC and EMS over an authenticated, hardware-backed channel.Source: A. Hierarchical Control Architecture · p. 3
  • Under partial seed compromise, ML-protected sensors continue producing accurate measurements, while low-deception compromise can raise stealthy attacks above the no-deception baseline.The hybrid pathway limits this exposure to the seed-routed subset.Source: A. Hierarchical Control Architecture · p. 3

B. State Estimation

ROAM uses an extended Kalman filter for recursive state estimation and evaluates it under a threat model with an informed, persistent attacker. The model assumes gradual, rate-limited injections and places explicit boundaries on the attacker’s access and mapping-recovery capabilities.

B. State Estimation

  • ROAM uses an extended Kalman filter for recursive state estimation, uncertainty propagation, and real-time anomaly detection across 14-, 57-, and 118-bus systems.The state dimension is twice the bus count: 28, 114, and 236, respectively.Source: B. State Estimation · p. 3
  • The measurement model relates synchronized PDC measurements to bus states through the network topology matrix, and the filter uses the previous estimate as its prior state.The identity process model is justified by small frame-to-frame state changes at the PMU sampling rate.Source: B. State Estimation · p. 3
  • State changes remain below 0.8% under normal conditions and 3.1% during simulated N−1 contingencies; a physics-informed process model is left for future work.The proposed linearized swing-equation alternative would require real-time generator-inertia estimation, introducing a secondary attack surface.Source: B. State Estimation · p. 3

C. Threat Model

  • The threat model assumes an informed, persistent attacker who seeks to corrupt AGC while evading BDD, with knowledge of BDD constraints and generator parameters.The attacker may access PMU and IED measurements and inject continuously or selectively, subject to system-level operating limits.Source: C. Threat Model · p. 3, Adversary Knowledge · p. 3, Adversary Capability · p. 3
  • Rate-limited injections must steer the system gradually, creating a detection window that ROAM can exploit through timely reshuffling.The formal attack model represents measurements as true values plus an injected perturbation bounded by system-level rate limits.Source: Stealth Constraints · p. 3, Formal Attack Model · p. 4
  • The baseline attacker cannot compromise the secure seed channel, and an attacker unable to recover the full mapping within one PMU frame falls outside the threat model.The remapped subsets change each epoch; the attacker must match all measurement streams within 16.7–33.3 ms, while even reduced-subset EMS matching takes 17–92 ms.Source: Trust Boundary · p. 4

III. ROAM FRAMEWORK

ROAM performs MTD at the PDC aggregation layer, preserving PMU hardware and IEEE C37.118 compatibility. It combines fixed and randomized sensor roles, with the randomized pathway using time-varying seed-based ID shuffling and EMS recovery.

III. ROAM FRAMEWORK

  • ROAM operates at the PDC aggregation layer without modifying PMU hardware and preserves IEEE C37.118 compatibility.Source: §III · p. 4
  • The DRL agent updates randomized-sensor assignments each decision epoch, choosing the randomized fraction and split between remapping pathways.Fixed sensors pass measurements unchanged; randomized sensors have their logical IDs shuffled before transmission.Source: §III · p. 4

1) Seed-Based ID Shuffling:

  • A time-varying seed drives a deterministic, bijective ID shuffle, and the EMS applies its inverse to recover authentic sensor identities.The shuffled stream preserves measurement values and timestamps; the seed is distributed to the PDC and EMS over a pre-established secure channel.Source: 1) Seed-Based ID Shuffling · p. 4
  • The changing seed is intended to prevent adversaries from inferring future mappings from intercepted historical traffic.The recovered measurements form the seed-remapped vector.Source: 1) Seed-Based ID Shuffling · p. 4

2) ML-Based State Prediction and Remapping:

ROAM's ML pathway predicts expected measurements from historical state trajectories, then uses assignment matching to restore the ordering of shuffled sensor measurements. Tree-based and sequential predictors are trained offline, and the lowest-test-MSE model is selected for deployment.

2) ML-Based State Prediction and Remapping:

  • ROAM predicts the next system state and expected measurement vector from historical state trajectories, then reorders shuffled measurements through combinatorial assignment.Source: 2) ML-Based State Prediction and… · p. 4
  • Tree-based models use hand-crafted features, whereas LSTM and GRU models consume the raw scaled lookback sequence.The tree-based candidates are XGBoost and Random Forest; the sequential candidates are LSTM and GRU.Source: 2) ML-Based State Prediction and… · p. 4
  • The four predictors are trained independently on an 80/20 chronological split, and the model with the lowest held-out test MSE is selected once for deployment.Source: 2) ML-Based State Prediction and… · p. 4
  • Assignment matching minimizes differences between predicted and shuffled measurements under one-to-one constraints, then feeds the recovered ordering to the EKF and subsequent BDD stage.Infeasible pairs above a proximity threshold are suppressed before the optimization is solved with OR-Tools CBC.Source: 2) ML-Based State Prediction and… · pp. 4–5

3) Hybrid Remapping:

ROAM combines seed-based and ML-based recovery for selected sensors, merging both recovered groups with unshuffled measurements before SE-BDD. Its DQN adapts coverage, sensor selection, and recovery mix using system and detection feedback, with a reward balancing detection against operational costs and attacker accuracy.

3) Hybrid Remapping:

  • ROAM partitions selected sensors between seed-based and ML-based recovery, then merges the recovered groups with fixed sensors into one measurement vector.Source: 3) Hybrid Remapping · p. 5
  • If the remapped vector preserves observability, the EMS forwards it to SE; otherwise, it blends remapped and predicted measurements to maintain SE continuity.The blend weight is set to 1 when observability is satisfied and reduced toward 0 in proportion to flagged sensors.Source: 3) Hybrid Remapping · p. 5

4) DQN-Directed Adaptive Control of the Hybrid MTD:

  • At each step, the DQN observes EKF state and BDD outcome and selects shuffle amount, per-sensor flags, and the seed-recovery share.The seed share ρt ranges from mostly ML recovery at small values to mostly seed-based recovery at large values.Source: 4) DQN-Directed Adaptive Control of… · p. 5

A. Markov Decision Process Formulation

  • The MDP state is a temporal window of normalized EKF-estimated system states, with example lookback length T = 10 and 13 monitored features for IEEE 14-bus.Source: A. Markov Decision Process Formulation · pp. 5–6
  • The reward favors BDD detection while penalizing deception cost, latency, and measured attacker accuracy.The reward weights were selected by grid search on the IEEE 14-bus system and held fixed across topologies.Source: A. Markov Decision Process Formulation · p. 6
  • Mean PSA ranged from 1.8% at λ3 = λ4 = 4.0 to 5.2% at λ3 = λ4 = 0.5 across 50 held-out episodes, with predictable degradation when detection incentives were underweighted.The 16 configurations varied λ3 and λ4 independently on the IEEE 14-bus system.Source: A. Markov Decision Process Formulation · p. 6
  • The factored action uses per-sensor flags rather than enumerating subsets, so its representation grows linearly with the number of PMUs.Jointly enumerating sensor subsets would require about 5.2 × 10^5 choices for N = 19.Source: A. Markov Decision Process Formulation · p. 6

B. Primary DRL Model

ROAM uses a DQN to select among discrete deception and recovery actions with binary sensor flags. Its value-learning setup uses a target network and replay buffer, while a four-layer MLP estimates action values.

B. Primary DRL Model

  • DQN fits ROAM's discrete action space of integer-percentage deception and seed ratios plus binary sensor flags, using Q-values to select shuffle configurations.Source: B. Primary DRL Model · p. 6
  • A periodically synchronized target network decouples the Bellman target from the online network to reduce oscillations, and training uses temporal-difference loss with a replay buffer.The replay buffer capacity is 10^5 transitions.Source: B. Primary DRL Model · p. 6
  • The Q-network is a four-layer MLP with ReLU activations and a symmetric expansion–contraction structure for producing action-value estimates.Source: B. Primary DRL Model · pp. 6–7

C. Comparative Baseline

The section introduces PPO as a parallel defender solving the same MDP as DQN, enabling a controlled comparison of on-policy and off-policy methods.

C. Comparative Baseline

  • PPO solves the same MDP as DQN through an on-policy policy-gradient approach, providing a controlled comparison with the off-policy value-based method.Both defenders share the observation, action space, and reward.Source: C. Comparative Baseline · p. 7
  • PPO optimizes a clipped surrogate objective to limit destructively large policy updates.The objective uses a probability ratio, GAE, and εclip = 0.2 as a trust-region constraint.Source: C. Comparative Baseline · p. 7
  • The full PPO objective adds a value-function loss and entropy bonus, with GAE using λGAE = 0.95.The entropy coefficient encourages policy diversity.Source: C. Comparative Baseline · p. 7

D. Adversarial Multi-Agent Framework

The adversarial evaluation casts ROAM as a competitive Markov game, pairing trained defenders with a SAC attacker that conditions its strategy on the defender’s latest configuration.

D. Adversarial Multi-Agent Framework

  • The framework stress-tests trained defenders against an adaptive attacker in a competitive Markov game, with attacker rewards opposing defender detection and state-accuracy objectives.The game is intended to probe worst-case conditions beyond the static scenario pool.Source: D. Adversarial Multi-Agent Framework · p. 7
  • The SAC attacker observes the power-system state sequence augmented with the defender’s latest deception amount and seed ratio, allowing configuration-conditioned attacks.Its continuous action encodes sensor-selection logits, per-sensor injection magnitudes, and a global intensity scalar, then maps to a physical FDI attack matrix.Source: D. Adversarial Multi-Agent Framework · p. 7
  • The attacker’s multi-objective reward weighs state damage, detection penalty, targeting accuracy, and sparsity cost to favor targeted, stealthy attacks.The detection and damage terms oppose the defender’s objectives.Source: D. Adversarial Multi-Agent Framework · p. 7
  • Training alternates between freezing and updating the defenders and SAC attacker so each can adapt to the other’s learned strategies.The protocol first trains SAC against a frozen DQN, then fine-tunes defenders against the attacker, and retrains SAC against the updated DQN.Source: D. Adversarial Multi-Agent Framework · pp. 7–8
  • The evaluation compares DQN and PPO over 50 deterministic episodes against the same fully trained SAC attacker, logging six defense and attack metrics.The metrics are PSA, PEA, CSER, EDR, attack accuracy, and cumulative reward.Source: D. Adversarial Multi-Agent Framework · p. 8

IV. NUMERICAL CASE STUDIES

The numerical studies evaluate ROAM on held-out BDD-compliant FDI scenarios, compare defense policies across three grid sizes, and test adversarial and real-time performance.

IV. NUMERICAL CASE STUDIES

  • Models train on 1,000 and evaluate on 500 held-out BDD-compliant FDI scenarios, with real-time shuffling tested using virtual and physical PMUs over IEEE C37.118.The study uses Python 3.11 and Stable Baselines3.Source: §IV · p. 8
  • The studies ask whether hybrid remapping outperforms seed-only and ML-only strategies, and whether per-sensor assignment lets DQN adapt beyond fixed single-mode defense.These questions assess attack displacement, detection, and state-estimation accuracy.Source: §IV · p. 8
  • DQN and PPO are compared for training efficiency, converged behavior, and defense effectiveness across IEEE 14-, 57-, and 118-bus systems.The evaluation also asks whether their relative performance persists against an adaptive SAC attacker.Source: §IV · p. 8
  • The study separately tests defense performance under held-out deterministic evaluation and against an adaptive SAC attacker, and asks whether ROAM is feasible in real time.The adversarial comparison checks whether either algorithm consistently dominates across topologies.Source: §IV · p. 8

A. Evaluation of Hybrid MTD in Dynamic Attack Conditions

Under cross-level shuffling, the hybrid strategy balances attack displacement, recovery accuracy, and latency; seed compromise exposes a distinct low-deception risk.

A. Evaluation of Hybrid MTD in Dynamic Attack Conditions

  • With a secure key, seed recovery achieves PSA 0.94% and sub-2 ms recovery; at full deception under compromise, CSER is 8.45% and EDR is 100%.Incorrect de-shuffling at low deception can push PSA above the no-deception baseline.Source: A. Evaluation of Hybrid MTD in Dynamic… · p. 8
  • Under seed compromise, PSA reaches 46.19% at 20% deception versus the 30.63% no-deception baseline, as incorrect de-shuffling realigns falsified data.At 40% deception, PSA is 28.97%; at full deception, displacement leaves only 6.01% of injections on target.Source: A. Evaluation of Hybrid MTD in Dynamic… · p. 8
  • ML recovery has the lowest EDR at full deception and full resilience to seed compromise, but roughly 60× the latency of seed-based recovery.The hybrid’s reduced MIP sensor subset yields a 5× latency reduction over pure ML at full deception.Source: A. Evaluation of Hybrid MTD in Dynamic… · p. 8
  • At full deception, hybrid remapping yields PSA 1.27% versus pure ML’s 1.96%, with latency 17.99 versus 90.86 ms.The hybrid uses 60% seed and 40% ML recovery, and is within 0.7 percentage points of uncompromised seed-only PSA at 60–100% deception.Source: A. Evaluation of Hybrid MTD in Dynamic… · p. 8, Table IV · p. 9
  • In IEEE 118-bus training, DQN converges near 88% seed ratio and 75% deception coverage, with detection above 90%, EDR below 10%, and attack accuracy below 30%.These are the reported converged DQN values in hybrid MTD mode.Source: Figure 3 · p. 9
  • End-to-end hybrid accuracy under seed compromise remains unquantified and is left to future work.The stated limitation concerns the hybrid’s accuracy under compromise.Source: A. Evaluation of Hybrid MTD in Dynamic… · p. 9
  • The hybrid configuration delivers operational-grade latency with partial resilience and is recommended as ROAM’s deployment baseline.Seed-based recovery is fastest but brittle, while ML-based recovery is robust but computationally expensive.Source: A. Evaluation of Hybrid MTD in Dynamic… · pp. 9–10

B. Performance of DQN-MTD in Hybrid Operating Mode

The hybrid DQN learns a seed-dominant remapping policy, while DQN–PPO comparisons show topology-dependent trade-offs rather than a universal winner.

B. Performance of DQN-MTD in Hybrid Operating Mode

  • In hybrid operation, DQN converges to remapping roughly 75% of sensors per step, with detection above 90%, state deviation below 10%, and PSA falling from about 3.5% to 1.8%.It routes about 88% of remapped sensors through seed-based recovery and chooses sensor flags and the seed ratio at every decision step.Source: B. Performance of DQN-MTD in Hybrid… · p. 10, Figure 4 · p. 10

C. Comparative Analysis of DQN and PPO MTD Agents

  • On 118-bus, DQN trains 4.5× faster than PPO and adopts 64–74% deception with ρ ≈69–88%, versus PPO’s roughly 53% deception and 50% seed ratio.PPO’s action choices remain near the midpoints of the action ranges during training.Source: C. Comparative Analysis of DQN and PPO… · p. 10
  • During training, DQN achieves lower PSA, higher detection, and better state estimation across systems, with 4.1× lower EDR than PPO on 118-bus.The PSA and detection gaps narrow on 57-bus, where DQN uses comparatively lower deception.Source: C. Comparative Analysis of DQN and PPO… · p. 10

D. Post-Training Evaluation of DQN and PPO MTD Agents

  • On held-out topologies, neither agent uniformly dominates: DQN leads CSER and EDR on 14- and 118-bus systems and PSA on 14- and 57-bus systems, while PPO reaches 1.05% PSA versus DQN’s 1.70% on 118-bus.DQN also outperforms PPO across all metrics on 14-bus; PPO reverses the EDR comparison on 57-bus by a factor of 1.6.Source: D. Post-Training Evaluation of DQN and… · p. 10

E. Evaluation with Adversarial Multi-Agent Framework

Across held-out systems, ROAM maintains low stealthy-attack rates and high detection, including under an adaptive SAC attacker. OPAL-RT testing demonstrates real-time operation, while adversarial evaluation reveals topology-dependent recovery trade-offs.

E. Evaluation with Adversarial Multi-Agent Framework

  • Under SAC attack, DQN has lower PSA on 14- and 57-bus systems, while PPO reaches 95.25% CSER versus DQN’s 93.77% on 57-bus.On 118-bus, the defenders’ PSAs are near-identical.Source: E. Evaluation with Adversarial… · p. 11, Table VII · p. 11
  • On 57-bus, DQN’s adversarial EDR reaches 57.79%, nearly double its standard-evaluation value, while PPO’s rises to 38.54%.The reported comparison indicates that stronger attack displacement does not ensure better state recovery against an adaptive adversary.Source: E. Evaluation with Adversarial… · pp. 11–12

F. Real-Time System Integration

  • On the OPAL-RT testbed, shuffle execution averaged 17.97 ms for 19 PMUs and the average reshuffle blind window was 2.6% of the 3-second cycle.Detection rose from 0% to 83.2% during the session, and 84.8% of attacked frames were nullified.Source: F. Real-Time System Integration · p. 12, Table VIII · p. 12

V. CONCLUSIONS

  • Across held-out systems, ROAM keeps stealthy attacks below 3% and detection above 93%; against SAC, stealthy attacks stay below 2.6%, while state-estimation deviation rises by up to 2.6×.The conclusion identifies end-to-end testing of the hybrid pathway under seed compromise as future work.Source: Table VII · p. 11, §V · p. 12
Loading 2610.12730v1…