PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Cyber-Resilience in Integrated Energy System Scheduling with Demand Response Using Deep Reinforcement Learning

Yang Li, Wenjie Ma, Yuanzheng Li, Sen Li, Zhe Chen, Mohammad Shahidehpor

arXiv:2311.17941v2eess.SYcs.LGcs.SY

TL;DR

IES scheduling must contend with renewable-generation and load uncertainty as well as cyberattacks. The paper combines integrated demand response with state-adversarial DRL scheduling. Simulations report higher post-attack rewards than SAC and resilient scheduling with stable energy supply.

  • Problem

    IES scheduling must manage RES and load uncertainty while facing cyberattacks that can undermine reliable, economical operation.

    Source: §I · p. 2

  • Method

    The paper combines an IDR-enabled scheduling method, an SA-MDP that models attacks as state-observation uncertainty, and an SA-SAC algorithm with adversarial defense training.

    Source: §I · p. 2

  • Results

    The proposed method achieves higher rewards post-attack than SAC and is reported to outperform SAC in resilience and performance under attack and normal conditions.

    Source: B. Analysis of Algorithm Robustness · p. 8, §VI · p. 10

  • Takeaways & Limitations

    The simulations support resilient IES scheduling that manages multiple uncertainties and maintains stable energy supply across scenarios.

    Source: §VI · p. 10

Abstract

from arXiv · show

Optimally scheduling multi-energy flow is an effective method to utilize renewable energy sources (RES) and improve the stability and economy of integrated energy systems (IES). However, the stable demand-supply of IES faces challenges from uncertainties that arise from RES and loads, as well as the increasing impact of cyber-attacks with advanced information and communication technologies adoption. To address these challenges, this paper proposes an innovative model-free resilience scheduling method based on state-adversarial deep reinforcement learning (DRL) for integrated demand response (IDR)-enabled IES. The proposed method designs an IDR program to explore the interaction ability of electricity-gas-heat flexible loads. Additionally, the state-adversarial Markov decision process (SA-MDP) model characterizes the energy scheduling problem of IES under cyber-attack, incorporating cyber-attacks as adversaries directly into the scheduling process. The state-adversarial soft actor-critic (SA-SAC) algorithm is proposed to mitigate the impact of cyber-attacks on the scheduling strategy, integrating adversarial training into the learning process to against cyber-attacks. Simulation results demonstrate that our method is capable of adequately addressing the uncertainties resulting from RES and loads, mitigating the impact of cyber-attacks on the scheduling strategy, and ensuring a stable demand supply for various energy sources. Moreover, the proposed method demonstrates resilience against cyber-attacks. Compared to the original soft actor-critic (SAC) algorithm, it achieves a 10% improvement in economic performance under cyber-attack scenarios.

I. INTRODUCTION

The introduction frames resilient IES scheduling as a joint challenge of source/load uncertainty and cyberattacks, identifying underused real-time IDR and limited DRL-based resilience research. It presents an IDR-enabled, state-adversarial DRL scheduling approach.

I. INTRODUCTION

  • IES scheduling must handle RES/load uncertainty and cyberattacks, while prior studies often omit real-time IDR and few use DRL for resilient IES scheduling.Stealthy false-data attacks can evade detection and lead to uneconomical operation.Source: §I · pp. 1–2
  • The proposed method uses DRL for pricing that combines time-of-use and real-time prices, considers predicted-average-voting thermal comfort, and treats attacks as state-observation uncertainty in an SA-MDP.The SA-MDP handles continuous state and action spaces.Source: §I · p. 2
  • SA-SAC adds a policy-network regularizer reflecting cyber-attack hazard and uses certified defense training to seek a policy resilient to adversarial state observations.The introduction reports testing the algorithm when cyber-attacks act as state-observation adversaries.Source: §I · p. 2

A. Structure Modeling of IES

  • The IES integrates electricity, heat, and natural gas through power, district-heating, and gas networks, with generation, conversion, storage, and customer-load components.Listed components include MT, WT, ESD, HSD, EB, and P2G.Source: A. Structure Modeling of IES · p. 2

1) Modeling of Load

  • The load models include time-shiftable electrical and gas demand and heat demand that can be reduced while maintaining users’ preferred thermal comfort.The study limits nighttime PMV to [-0.9, 0.9] and daytime PMV to [-0.5, 0.5].Source: 1) Modeling of Load · pp. 2–3

2) Energy Storage Device Model

The energy-storage model covers both electrical and heat storage devices, including their operating powers, efficiencies, capacities, and state of charge.

2) Energy Storage Device Model

  • The model represents ESD and HSD charging and discharging, their respective efficiencies and capacities, and storage state of charge.The supplied text identifies maximum capacities for both devices.Source: 2) Energy Storage Device Model · p. 4

3) Electric Boiler Model

The electric-boiler model converts absorbed electricity into heat, while the power-to-gas model converts electricity into natural gas.

3) Electric Boiler Model

  • The electric boiler’s heat output equals absorbed electrical energy multiplied by its conversion efficiency.The model identifies absorbed electrical energy, heat output, and conversion efficiency for each period.Source: 3) Electric Boiler Model · p. 4

4) Power to Gas Model

  • Power-to-gas production converts absorbed electricity into natural gas, providing flexibility for managing renewable intermittency.Natural gas production is correlated with electricity consumption through the P2G conversion model.Source: 4) Power to Gas Model · p. 4

5) Micro-Gas Turbine Model

This section describes micro-gas-turbine outputs, bad-data detection in IES state estimation, and indoor-temperature manipulation through a heat-load redistribution attack. The scheduling model also states an operator-profit objective.

5) Micro-Gas Turbine Model

  • The micro-gas turbine model formulates heat and electricity outputs in relation to its natural-gas consumption.The passage identifies the turbine’s heat output, electrical power output, gas consumption, and efficiency coefficients.Source: 5) Micro-Gas Turbine Model · p. 4

B. Modeling of LR Attack

  • Load redistribution attacks alter load data, producing incorrect meter readings that can mislead scheduling decisions.The passage identifies LR attack as a type of false-data injection attack.Source: B. Modeling of LR Attack · p. 4

1) Bad Data Detection in IES State Estimation

  • Bad-data detection compares the Euclidean norm of measurement residuals with a threshold to identify invalid state estimates.A residual at or below the threshold indicates a valid estimate; a larger residual indicates bad data.Source: 1) Bad Data Detection in IES State… · p. 4

2) HLR Attack Modeling

  • The indoor temperature delay scaling attack manipulates indoor-temperature data, changing heat load and potentially affecting interconnected subsystems.The heating system’s less mature state estimation is identified as a vulnerability; the attack is described as stealthy and economically impactful.Source: 2) HLR Attack Modeling · p. 4

III. FORMULATION OF IES SCHEDULING MODEL

  • The model formulates IES dispatch to maximize the operator’s net profit.Its scheduling formulation also proposes dynamic pricing that combines time-of-use and real-time prices to exploit demand-response potential.Source: §III · p. 4

A. Dynamic Pricing Mechanism

The dynamic pricing mechanism adjusts electricity, gas, and heat prices in real time while bounding them relative to time-of-use prices. The scheduling model maximizes operator profit and includes energy-balance and storage operating constraints.

A. Dynamic Pricing Mechanism

  • Dynamic pricing combines predictable time-of-use benchmarks with real-time prices for electricity, gas, and heat, bounded around time-of-use peaks and valleys.Real-time adjustments use grid feedback and IES device status to guide demand and increase operator profits.Source: A. Dynamic Pricing Mechanism · p. 5

B. Objective Function

  • The operator-profit objective accounts for revenue and operating costs, including compensation for shifted electric and gas loads and cut heat load.The passage also identifies electricity and natural-gas purchases and sales as operator revenue terms.Source: B. Objective Function · p. 5, T SQ · p. 5

1) Energy Balance Constraints

  • The electricity balance equates load with grid purchases, wind and storage output, and microturbine generation, net of boiler and power-to-gas consumption.The passage identifies wind-turbine output and electricity-storage charging or discharging as balance components.Source: 1) Energy Balance Constraints · p. 5

2) ESD Operating Constraints

  • The ESD operating constraints bound its charging and discharging power and keep its capacity within limits, with starting and ending capacities specified.The passage notes that analogous heat-storage operating constraints are not restated.Source: 2) ESD Operating Constraints · p. 5

3) Electric Boiler Constraint

The constraints bound electric-boiler output, power-to-gas and microturbine inputs, and energy exchanges with external electricity and gas grids.

3) Electric Boiler Constraint

  • Electric-boiler heat output is constrained between zero and its maximum heat power.The maximum heat power is the boiler’s specified output limit.Source: 3) Electric Boiler Constraint · p. 5

4) P2G Operating Constraints

  • Power-to-gas input constraints specify minimum and maximum power and upper and lower ramping limits.The passage identifies these as limits on P2G input power and climbing ability.Source: 4) P2G Operating Constraints · p. 5

5) MT Operating Constraints

  • Microturbine input constraints specify minimum and maximum gas consumption and upper and lower ramping limits.The passage identifies these as limits on gas consumption volume and climbing ability.Source: 5) MT Operating Constraints · p. 5

6) External Energy Transmission Power Constraints

  • The model constrains electricity and natural-gas trading between the IES and their respective external grids.The passage identifies upper and lower limits for both electricity and gas interactions.Source: 6) External Energy Transmission Power… · p. 6

IV. MODEL SOLVING

The SA-MDP formulates IES scheduling with load and renewable-energy uncertainty while representing cyber-attacks as disturbances to the agent’s state observation. Its objective balances operator profit and supply-demand balance.

A. SA-MDP Model

  • The SA-MDP models IES scheduling under load and renewable-energy uncertainty, with cyber-attacks represented as adversarial disturbances to state observations.The adversary changes the state perceived by the agent, while the environment transitions from the underlying state.Source: A. SA-MDP Model · p. 6

3) Adversary

  • The adversarial action can make the policy’s chosen action sub-optimal and reduce its reward.The model constrains the adversary’s perturbation to a set around the state and uses HLR attack as its adversary.Source: A. SA-MDP Model · p. 6, 3) Adversary · p. 6
  • Theorem 1 bounds the attacked-versus-unattacked performance difference when policy distributions remain sufficiently close across allowed perturbed states.The method uses this result to motivate a state-adversarial regularizer for a resilient scheduling policy.Source: 3) Adversary · p. 6

4) Reward Function

  • The reward maximizes operator net profit while adding penalties for action-rate overruns and supply-demand imbalance.The action-rate penalty applies when action changes exceed upper or lower limits.Source: 4) Reward Function · p. 6

B. SA-SAC based Solution Process

The solution incorporates state-adversarial regularization and certified defense training into a maximum-entropy SAC framework. Its policy objective combines the SAC objective with an upper bound on policy divergence under state perturbations.

B. SA-SAC based Solution Process

  • SA-SAC retains SAC’s maximum-entropy objective, balancing cumulative reward with an entropy term that encourages exploration.The policy evaluation step uses a modified Bellman equation.Source: B. SA-SAC based Solution Process · pp. 6–7
  • The state-adversarial regularizer limits policy divergence between the observed state and perturbed states within the attack set.The method uses KL divergence to establish an upper bound on total variation distance, then adds the distance bound to the policy-gradient loss.Source: B. SA-SAC based Solution Process · pp. 7–8
  • CROWN-IBP combines interval-bound propagation with CROWN convex relaxation to derive policy bounds for cyber-attack perturbations.The bounds are used in defense training, and the paper states that this combination aims to accelerate training while avoiding excessive regularization.Source: B. SA-SAC based Solution Process · p. 7
  • The training loop updates twin Q-functions, bounds the policy with CROWN-IBP, applies the regularized policy loss, and updates temperature and target networks.The paper describes the resulting policy network as the resilient IES scheduling model.Source: B. SA-SAC based Solution Process · p. 8

V. CASE STUDY

The case study evaluates the proposed scheduling approach through simulation. The authors describe the software and hardware used and specify the training setup.

V. CASE STUDY

  • The case study evaluates the approach using an IES simulation implemented in Python 3.8 and PyTorch 1.6.0.The simulations ran on an AMD R7-5800H CPU with 16 GB of RAM.Source: §V · p. 8
  • The actor and critic use ReLU hidden layers and Adam, with learning rates of 0.0005 and 0.002, respectively.Training lasts 1,000 epochs with batch size 256 and discount factor 0.95.Source: §V · p. 8
  • Training delays actor regularization for 500 episodes, then increases attack perturbation over 300 episodes before holding the target perturbation for 200 episodes.The target-network soft update rate is 0.005.Source: §V · p. 8

A. Settings in Test Case

The case study compares SA-SAC with SAC under attack and examines scheduling across electricity, gas, and heat. The reported results include higher post-attack rewards for SA-SAC, demand-response cost reductions, and subsystem dispatch patterns.

B. Analysis of Algorithm Robustness

1) Scheduling Schemes Analysis for Each Device

  • The SA-SAC dispatch strategy meets each subsystem’s needs, prioritizing wind output, using P2G mainly off-peak, and shifting heat supply between MT and EB.The paper reports grid electricity use mainly from 0:00–7:00, MT operation during peak hours, and nighttime EB use when temperatures and electricity prices are lower.Source: 1) Scheduling Schemes Analysis for… · p. 9

2) Flexible Demand Response Analysis

  • Demand response shifts electricity and gas loads across time, reducing procurement costs and increasing renewable-energy utilization.The paper also reports peak-shaving and valley-filling patterns in electricity and gas loads, and a nighttime thermal-load decrease associated with reduced temperature perception and wider thermal comfort ranges.Source: 2) Flexible Demand Response Analysis · p. 9, Figure 10 · p. 9

D. Impact Analysis of Cyber-Attack on Scheduling

1) Scheduling Strategy Analysis under Attack Conditions

The scheduling comparisons show that cyber-attacks can sharply alter energy dispatch and reduce scheduling quality, while the proposed method is reported to improve profits and resilience across operating scenarios.

1) Scheduling Strategy Analysis under Attack Conditions

2) Economic Analysis

  • The method reports higher profits across operating scenarios, with defensive training reducing volatility and HSD participation improving economic efficiency and resilience under attack.Profit calculations include energy production and consumption costs as well as potential penalties and losses from network attacks.Source: Table III · p. 10, 2) Economic Analysis · p. 10

VI. CONCLUSION

  • The method increases renewable use through P2G and EB, guides user demand response with pricing, and uses HSD devices to mitigate HLR damages.Source: §VI · p. 10
  • The SA-MDP represents scheduling challenges during attacks, and SA-SAC with an action-space regularizer outperforms SAC in resilience and performance under both attacked and normal conditions.Source: §VI · p. 10
  • Test results describe the scheduling strategy as economically superior, resilient to multiple uncertainties, and suitable for real-time scheduling across scenarios.Source: §VI · p. 10
Loading 2311.17941v2…