PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Cyber-Attack Detection in Smart Grid: A Reinforcement Learning Approach

Mehmet Necip Kurt, Oyetunji Ogundijo, Chong Li, Xiaodong Wang

arXiv:1809.05258v1cs.LGcs.CRstat.ML

TL;DR

Because smart-grid failures or anomalies can cause substantial damage, the paper formulates online cyber-attack detection as a POMDP and develops a model-free RL solution. Numerical studies report that the scheme significantly outperforms a benchmark in almost all cases and supports fast, reliable detection.

  • Problem

    Smart-grid failures or anomalies can cause substantial damage, motivating timely online cyber-attack detection.

  • Method

    The paper formulates online cyber-attack detection as a POMDP and proposes a universal online algorithm using model-free RL for POMDPs without requiring attack models.

  • Results

    Numerical studies show that the proposed RL-based detection scheme significantly outperforms the benchmark in almost all cases and enables fast, reliable detection.

  • Takeaways & Limitations

    The model-free, attack-model-independent scheme can detect new unknown attack types and is widely applicable.

Abstract

from arXiv · show

Early detection of cyber-attacks is crucial for a safe and reliable operation of the smart grid. In the literature, outlier detection schemes making sample-by-sample decisions and online detection schemes requiring perfect attack models have been proposed. In this paper, we formulate the online attack/anomaly detection problem as a partially observable Markov decision process (POMDP) problem and propose a universal robust online detection algorithm using the framework of model-free reinforcement learning (RL) for POMDPs. Numerical studies illustrate the effectiveness of the proposed RL-based algorithm in timely and accurate detection of cyber-attacks targeting the smart grid.

I. INTRODUCTION

Smart-grid cyber-attacks can cause extensive damage, making early online detection important. Existing detectors face a trade-off between detection speed and false alarms, while unknown attack models and sample-by-sample decisions limit reliability; the paper frames detection as a POMDP and uses model-free RL.

  • Threat and motivation: Cyber-attacks exploit smart-grid control and communication infrastructure through mechanisms including FDI, jamming, and topology manipulation.These attacks can corrupt or block meter measurements and may damage or mislead state estimation.
  • Threat and motivation: 81?
  • Detection objective: Online detection must balance detecting changes quickly against maintaining a minimal false-alarm level.Decision makers can declare a change or wait for additional sequential measurements; higher desired accuracy generally reduces detection speed.
  • Limits of existing approaches: CUSUM-based detectors require sufficiently accurate models of both pre-change and post-change measurement distributions.Unknown attacker capabilities and strategies can make post-change behavior difficult to model, motivating universal detectors without attack models.
  • Proposed perspective: The paper formulates quickest attack detection as a POMDP with hidden pre- and post-attack states and proposes a model-free RL solution.Model-free RL avoids learning an approximate POMDP model in a separate step and is suited to uncertain environments.
  • Limits of existing approaches: Outlier detectors use measurement dissimilarity thresholds but ignore temporal relations and make sample-by-sample decisions.Consequently, they cannot distinguish instantaneous high-level system noise from persistent anomalies.

B. Contributions

The paper proposes a universal robust online cyber-attack detection algorithm based on model-free reinforcement learning for POMDPs. It learns observation-to-action decisions without attack models and trains against low-magnitude, difficult-to-detect attacks.

  • The proposed algorithm uses model-free reinforcement learning for online cyber-attack detection formulated as a POMDP problem.
  • The universal detector does not require attack models, enabling detection of new unknown attack types.
  • The defender learns a direct mapping from observations to stop-or-continue actions through trial-and-error.
  • Because real attack data are difficult to obtain, training uses low-magnitude attacks corresponding to worst-case detection scenarios.
  • Training on difficult low-magnitude attacks makes the defender sensitive to slight deviations from normal meter measurements and limits the attacker’s effective action space.
  • The work is presented as the first attempt to use RL techniques for online cyber-attack detection in the smart grid.

C. Organization and Notation

The paper introduces its system model, problem formulation, proposed solution, simulations, and conclusion in successive sections. It also defines notation conventions and summarizes common symbols and parameters in Table I.

  • Section II introduces the system model and state estimation mechanism, while Section III formulates the detection problem.
  • Section IV presents the proposed solution approach, and Section V illustrates its performance through extensive simulations.
  • Section VI concludes the paper.
  • Boldface denotes vectors and matrices, vectors are column vectors, and oT denotes the transpose of o.
  • P and E denote probability and expectation operators, respectively, while Table I summarizes common symbols and parameters.

A. System Model

The smart-grid model represents bus phase-angle states and meter measurements with a linear dynamic state-space model. It extends the normal measurement model to several cyber-attack types, including false data injection, jamming, denial of service, and topology attacks.

  • The grid has N state variables for a system with N + 1 buses and K meters, with one bus serving as a reference.
  • The state vector contains bus phase angles, and the measurement vector contains readings from the K meters.
  • Under normal operation, the smart grid follows linear state-space equations with transition matrix A, measurement matrix H, process noise, and measurement noise.
  • The process and measurement noises are assumed to be independent additive white Gaussian random processes, and the system is assumed observable.
  • False data injection adds malicious data to meter measurements after attack time τ.
  • The model also represents jamming noise, hybrid FDI/jamming, partially unavailable measurements in DoS attacks, topology changes, and mixed topology-hybrid attacks.
  • For DoS attacks, a diagonal availability matrix records whether each meter measurement is available, with D_t = I_K before τ.
  • Network topology attacks are modeled through changes in the measurement matrix.

B. State Estimation

State estimation is central to smart-grid operation because regulation relies on estimated states. The paper uses a dynamic Kalman filter and shows that cyber-attacks increasingly distort estimates as attack magnitude grows.

  • State estimation is fundamental because the smart grid is regulated using estimated system states.
  • The dynamic grid’s time-varying load, generation, and cyber-attacks motivate dynamic modeling and real-time state estimation.
  • For discrete-time linear systems with Gaussian noise, the Kalman filter is optimal for minimizing mean squared state estimation error.
  • The online Kalman filter alternates prediction and measurement-update steps, producing state estimates and covariance updates using the Kalman gain.
  • In the illustrative experiment, random FDI attacks begin at τ = 100 after normal operation, with three attack magnitude levels tested.
  • Attack Level 2 uses b_k,t ∼ U[−0.07, 0.07], while Level 3 uses b_k,t ∼ U[−0.1, 0.1].
  • Under cyber-attacks, state estimates deviate from actual states, and the deviation increases with attack magnitude.

III. PROBLEM FORMULATION

The paper formulates online smart-grid attack detection as a POMDP with hidden pre-attack and post-attack states, unknown attack behavior, and a stopping decision balancing false alarms against detection delay.

  • POMDP formulation: An agent in the POMDP maps observations to actions while the environment transitions between hidden states and supplies rewards or costs.The defender minimizes expected total discounted cost when costs replace rewards.
  • Online detection problem: An attack begins at an unknown time τ, creating hidden pre-attack and post-attack states that the defender must distinguish from measurements.The defender can stop to declare an attack or continue collecting measurements.
  • Unknown model: The post-attack observation distribution and transition probability are unknown because attacker strategies and the launch time are unknown, so the exact POMDP model is unavailable.The paper therefore uses model-free reinforcement learning to learn a direct observation-to-action mapping.
  • Cost objective: The model assigns cost 1 to false alarms and cost c to each post-attack continuation step, with stopping entering a terminal state and ending further costs.The resulting objective minimizes expected total cost and yields a detection-delay/false-alarm tradeoff as c varies.
  • Observation information: Single observations can be identical before and after attacks, so finite observation history is needed to reduce state ambiguity and support detection of persistent anomalies.The paper distinguishes this sequential setting from sample-by-sample outlier detection.

IV. SOLUTION APPROACH

The solution transforms measurement deviations into quantized sliding-window observations and learns stopping decisions with a SARSA Q-table. The resulting detector selects the action with the lowest expected future cost during online operation.

  • Observation construction: The observation signal η_t is derived from measurement likelihoods relative to a baseline model, with persistent high values indicating possible attacks or anomalies.Likelihoods are expected to decrease when the system deviates from normal operation.
  • Observation construction: Continuous observations are quantized into I intervals using thresholds β_i, and M consecutive quantized values form a sliding observation window.This produces I^M possible observation windows for the agent.
  • RL policy: For every observation-action pair, SARSA learns Q(o, a) values in a Q-table of size I^M × 2, and the policy chooses the action with minimum Q value.Increasing I or M may improve learning but enlarges the table and increases training requirements.
  • Online phase: During online detection, the defender repeatedly collects measurements, forms the current observation window, and selects the lowest-cost action until stop declares an attack.After recovery to normal operation, the trained detector can be restarted without further training.
  • Learning phase: During learning, a simulation environment supplies observations and costs after each action, allowing the defender to update the Q-table repeatedly across episodes.The ε-greedy policy selects the minimum-Q action with probability 1−ε and explores the other action with probability ε.
  • Complexity: The learning procedure has time complexity O(TE) and memory complexity M + 2I^M, with complexity determined by episode length, episode count, and window size.The memory requirement is fixed over time and stores the observation window and Q-table.

A. Simulation Setup and Parameters

The simulations use an IEEE-14 bus smart-grid model with random FDI and hybrid FDI/jamming attacks, training the detector under multiple launch times and cost settings to examine delay–false-alarm tradeoffs.

  • System model: The simulation uses an IEEE-14 bus power system with 14 buses and 23 smart meters.Initial phase-angle state variables are obtained using the DC optimal power flow algorithm in MATPOWER.
  • Detector parameters: The detector uses four quantization levels, thresholds β1 = 0.95 × 10^-2, β2 = 1.05 × 10^-2, and β3 = 1.15 × 10^-2, and a four-entry observation window.The learning rate and exploration rate are both set to 0.1.
  • Training setup: The defender is trained for 4 × 10^5 episodes with τ = 100 and another 4 × 10^5 episodes with τ = 1.The second training stage increases exploration under normal operating conditions and attack conditions.
  • Attack generation: The learning episodes include low-magnitude attacks to train robustness against slight measurement deviations from normal operation.Random FDI and hybrid FDI/jamming attacks are generated with specified bounded FDI magnitudes and Gaussian jamming noise.

B. Performance Evaluation

The proposed RL-based detector is evaluated against benchmark detectors across multiple FDI, jamming, DoS, topology, and hybrid attack scenarios. It generally outperforms the benchmarks, while trading false alarms against detection delay through its cost parameter.

  • Results: As the relative false-alarm cost 1/c increases, the proposed detector’s false alarm rate decreases.The reported operating-point parameters include c = 0.2 and c = 0.02, with average false alarm periods of 9.4696 × 10^5 and 7.9210 × 10^6, respectively.
  • Evaluation setup: The evaluation compares the proposed RL-based detector with Euclidean and cosine-similarity benchmark detectors using Monte Carlo simulations over 10000 trials.Thresholds are varied for the benchmarks, while c is varied for the proposed algorithm.
  • Results: For almost all cases, the proposed RL-based detection scheme significantly outperforms the benchmark tests.The proposed scheme is evaluated using false alarm probability, average detection delay, precision, recall, and F-score.
  • Results: The trained defender reduces false alarms by distinguishing persistent attacks from instantaneous high-level system noise realizations.The benchmark sample-by-sample detectors are described as more vulnerable to false alarms because they cannot make this distinction.
  • Results: All detectors detect DoS attacks with almost zero average detection delays because meter measurements become partially unavailable.The DoS setting makes each meter measurement unavailable with probability 0.2 after the attack begins.

VI. CONCLUDING REMARKS

The paper formulates online cyber-attack detection as a POMDP and proposes a universal model-free RL solution that does not require attack models. Numerical studies show fast and reliable detection, while the approach remains open to improvements in memory, value approximation, and deep RL.

  • Contribution: The paper formulates online cyber-attack detection as a POMDP and proposes a solution based on model-free RL for POMDPs.The hidden pre-change and post-change states arise from the unknown attack launch time.
  • Conclusion: Numerical studies illustrate the advantages of the proposed scheme in fast and reliable detection of cyber-attacks targeting the smart grid.The results also demonstrate the potential of RL algorithms for complex cybersecurity problems.
  • Future directions: Future work includes more sophisticated memory, function approximation for Q values, and deep RL algorithms to improve performance.The current discussion contrasts these directions with finite-size sliding windows and tabular Q-value computation.
  • Scope: The proposed online detection method is universal and does not require attack models, supporting application when normal operation is modeled but post-change behavior is unknown.The paper describes this setting as common in practical anomaly and attack detection.
  • Future directions: Real post-change attack or anomaly data could be enhanced with simulated data for training, potentially improving detection performance.This possibility is stated conditionally for applications where real post-change data can be obtained.
Loading 1809.05258v1…