PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Power System Emergency Control using Deep Reinforcement Learning

Qiuhua Huang, Renke Huang, Weituo Hao, Jie Tan, Rui Fan, Zhenyu Huang

arXiv:1903.03712v2cs.LGeess.SYstat.ML

TL;DR

Existing emergency controls are often designed offline from typical scenarios, creating an adaptiveness and robustness gap under modern-grid uncertainty. The paper develops DRL-based generator braking and UVLS schemes within the open-source RLGC platform, and case studies report adaptive, robust performance across new scenarios, parameter uncertainty, and noisy observations.

  • Problem

    Existing emergency controls rely mainly on offline settings from worst-case or typical scenarios, while modern-grid uncertainties require greater adaptiveness and robustness.

  • Method

    The paper develops DRL-based generator dynamic braking and UVLS schemes and introduces RLGC for developing and benchmarking power-system control algorithms.

  • Results

    The proposed DRL emergency-control schemes demonstrate adaptiveness and robustness to new scenarios, model-parameter uncertainty, and observation noise, with comparisons against Q-learning, MPC, and relay control.

  • Takeaways & Limitations

    RLGC provides an open benchmark for future DRL research, while the studied schemes support adaptive emergency control for complex power systems within the tested cases.

Abstract

from arXiv · show

Power system emergency control is generally regarded as the last safety net for grid security and resiliency. Existing emergency control schemes are usually designed off-line based on either the conceived "worst" case scenario or a few typical operation scenarios. These schemes are facing significant adaptiveness and robustness issues as increasing uncertainties and variations occur in modern electrical grids. To address these challenges, for the first time, this paper developed novel adaptive emergency control schemes using deep reinforcement learning (DRL), by leveraging the high-dimensional feature extraction and non-linear generalization capabilities of DRL for complex power systems. Furthermore, an open-source platform named RLGC has been designed for the first time to assist the development and benchmarking of DRL algorithms for power system control. Details of the platform and DRL-based emergency control schemes for generator dynamic braking and under-voltage load shedding are presented. Extensive case studies performed in both two-area four-machine system and IEEE 39-Bus system have demonstrated the excellent performance and robustness of the proposed schemes.

I. INTRODUCTION

Power-system emergency control must respond adaptively to uncertain real-time conditions, but conventional schemes rely mainly on fixed, offline settings. The paper addresses this gap with DRL-based emergency controls and the open-source RLGC platform.

  • Emergency control is vital for minimizing power outages and widespread blackouts, using actions such as redispatch, tripping, load shedding, islanding, and dynamic braking.
  • Existing emergency controls generally use fixed settings determined offline from typical scenarios and operate in a “set-and-forget” mode rather than adapting to real-time conditions.
  • Power-system emergency control is dynamic sequential decision-making under uncertainty, challenging static SC-ACOPF, poorly scalable optimal control, and conventional methods lacking adaptiveness.
  • DRL combines deep learning with reinforcement learning to provide high-dimensional feature extraction, nonlinear generalization, and improved scalability for complex control problems.
  • RLGC is introduced as an open platform for developing, training, and benchmarking DRL algorithms for power-system control, addressing reproducibility and comparison challenges.
  • The paper applies DRL to generator dynamic braking and under-voltage load shedding while investigating adaptation, robustness, and comparisons with conventional Q-learning and optimal-control methods.

A. Reinforcement Learning

Reinforcement learning models emergency control as sequential interaction between an agent and an environment. The agent observes states, selects actions, and seeks to maximize discounted future rewards.

  • An RL agent learns decisions through exploration and exploitation while interacting with an environment modeled as a partially observable Markov decision process.
  • The MDP defines state space S, action space A, and discount factor γ, with states and actions allowed to be continuous or discrete.
  • At each time step, the agent observes state st, receives reward rt, and selects action at to change the environment.
  • The objective is to maximize discounted future return Rt by choosing optimal actions given the current state.
  • The Q function evaluates state-action choices, while Q*(s, a) represents the maximum discounted future return after taking action a in state s.
  • Tabular Q-learning requires discretizing observations, which can cause memory explosion and slow convergence in large-scale problems and struggles with noisy or incomplete states.

B. Deep Reinforcement Learning

DRL combines reinforcement learning with deep learning so agents can learn from raw or high-dimensional state representations. DQN improves value learning through neural approximation, target networks, and experience replay.

  • DRL uses deep learning for direct raw-state inputs, high-dimensional feature extraction, and nonlinear generalization in complex systems and tasks.
  • DQN estimates Q-values with a neural network and improves efficiency and stability using a target network and experience replay.
  • DQN training computes target Q-values with the target network and updates neural-network weights through stochastic gradient descent.
  • The DQN algorithm samples transitions from replay memory, selecting exploratory random actions or greedy Q-value actions during interaction.
  • DQN periodically resets the target network and decays exploration from 1.0 toward a small minimum value, shifting from exploration toward exploitation.
  • Unlike discretized tabular methods, DQN avoids memory explosion and can capture patterns from noisy observations.

C. Grid Emergency Control

The paper formulates large-scale power-system emergency control as a nonlinear, non-convex decision problem with dynamic and algebraic grid states, disturbances, controls, constraints, and costs. It then maps the problem to an MDP for RL.

  • Large-scale emergency control is formulated as a highly nonlinear, non-convex optimal decision-making problem.
  • The grid dynamics are represented by a differential equation linking dynamic states, algebraic states, disturbances, and emergency-control actions.
  • Dynamic states include generator rotor angles and speeds, while algebraic states typically represent bus voltages.
  • Emergency-control variables include generator tripping and load shedding, and disturbances represent contingencies that may occur in the grid.
  • The RL formulation observes only a subset of grid state variables, defines action bounds through the action space, and models transitions through differential-algebraic equations.
  • The reward-related function incorporates action costs and penalties for violations of system constraints.

III. AN OPEN PLATFORM FOR DEVELOPING AND BENCHMARKING RL ALGORITHMS FOR GRID CONTROL

RLGC is an open-source platform for developing, training, and benchmarking reinforcement-learning algorithms for power-system control. Its modular architecture combines an RL module with a power-system simulation and control module, supporting extensive and flexible experimentation.

  • RLGC provides an open-source benchmark for developing, training, and comparing RL algorithms for power-system control.The platform is intended to address reproducibility and benchmarking challenges in power-system RL research.
  • The platform separates an OpenAI Gym-based RL module from an InterPSS-based power-system simulation and control module, connected through Py4J.The two modules are programmed in Python and Java, respectively, and communicate through a decoupled interface.
  • OpenAI Gym integration allows users to apply open-source algorithms such as DQN for discrete actions and PPO for continuous actions.The paper uses OpenAI Baselines’ DQN implementation for its two discrete-action emergency-control problems.
  • RLGC supports diverse RL algorithms, power-system dynamics and measurements, emergency-control actions, and transmission-distribution system simulations.Its extensive framework is designed to capture varied RL and power-system configurations.
  • Users can configure observations, actions, and rewards with at least two configuration files or by programming new functions.This flexibility enables customized environments for training and testing power-system control algorithms.

B. Implementation Details and Usage

RLGC implements the agent-environment interaction through wrapper functions and supports a two-stage workflow for training and testing neural-network control policies. The power-system simulator advances on a finer time step than the DRL agent to maintain numerical stability.

  • Implementation Details: The simulation wrapper exposes functions for initialization, action execution, state and reward retrieval, and terminal-state checking.These functions connect the learning agent with the dynamic simulation environment.
  • Usage: Platform use consists of a training stage that learns and saves a policy, followed by a testing stage that validates the trained neural network.Training uses exploration and exploitation over many steps and automatically saves the best-performing parameters.
  • Usage: During training, the agent receives states and rewards, selects actions, and sends them to the power-system simulation for dynamic execution.The updated states and rewards are then returned for continued neural-network learning.
  • Usage: The dynamic simulation uses a smaller time step than the DRL interaction loop, ranging from 1 ms to half a cycle, to ensure numerical stability.This separates numerical integration from the agent-environment interaction time scale.
  • Usage: The platform is used to develop DRL schemes for generator dynamic braking and under-voltage load shedding.The paper discusses neural networks, observations, actions, and rewards for both emergency-control applications.

A. Neural Network Architecture

The emergency-control agents use neural networks with two hidden layers and direct system measurements, while historical observations help represent dynamics. The control designs use task-specific rewards for generator braking and coordinated voltage-support load shedding.

  • A. Neural Network Architecture: The neural network architecture uses input and output layers separated by two hidden layers followed by ReLU units.The layer sizes are denoted by Ni, No, Nh1, and Nh2; the paper notes that greater depth is not required for effective DRL.
  • B. Generator Dynamic Brake: Generator dynamic braking aims to prevent loss of synchronism and damp large electromechanical oscillations under limited brake-on time.The brake is therefore intended for emergency use because of energy losses and operating constraints.
  • B. Generator Dynamic Brake: The generator-braking reward uses average generator speed and angle, penalizes brake activation, and assigns -1000 after synchronism is lost.Synchronism loss is defined here as |δ| > π rad.
  • B. Generator Dynamic Brake: The proposed generator-braking agent directly observes monitored-generator rotor angles and speeds rather than hand-crafted pseudo-states.Stacking recent observations provides information about system dynamics and trends beyond the current state.
  • C. Under-voltage Load Shedding: Under-voltage load shedding addresses FIDVR, in which voltage remains significantly reduced for several seconds after fault clearance because residential air-conditioner motors stall.The scheme targets voltage-recovery requirements evaluated by the transient voltage recovery criterion.
  • C. Under-voltage Load Shedding: The DRL load-shedding scheme coordinates multiple substations using voltage magnitudes, remaining-load percentages, stacked history, and binary 20%-shedding actions.Its reward combines voltage deviations, total load shedding, and invalid-action penalties, with -1000 assigned when any bus remains below 0.95 p.u. 4 s after clearance.

V. TEST RESULTS

The proposed DRL controller was evaluated for generator dynamic braking on the two-area, four-machine system across varied faults and operating conditions. It maintained stability across 220 tested scenarios and outperformed conventional Q-learning under noisy observations.

  • Generator Dynamic Brake: The generator dynamic-brake controller was tested on the two-area, four-machine system with a 400 MW resistance brake at bus 6.The brake size was g = 4.0 p.u. mhos on a 100 MVA base.
  • Training: Training used randomized fault durations around the 0.583 s critical clearing time, exposing the agent to stable and unstable postfault conditions.Faults were applied at bus 3 after 1.0 s, with durations from 0.581 s to 0.585 s.
  • Results: Without dynamic braking, the system withstands faults only up to 0.583 s, whereas the DRL-trained brake maintained stability across the tested scenarios.The comparison establishes the baseline fault-duration limit before applying the trained control law.
  • Robustness Testing: The trained DRL policy remained stable in 220 scenarios spanning different power flows, fault locations, and fault durations, despite 1% Gaussian observation noise.The tests covered all 10 buses and fault durations from 0.3 s to 0.7 s.
  • Comparison: The DRL controller outperformed conventional Q-learning in all noisy testing scenarios by using fewer brake-operation time steps and obtaining higher rewards.Both controllers sustained stability in the illustrated fault cases, but DRL produced better control actions.

B. Under Voltage Load Shedding

The paper applies DRL to coordinated under-voltage load shedding for FIDVR on a modified IEEE 39-bus system. Across diverse test scenarios, DRL generally outperformed UVLS relays and was more robust than MPC under modeling uncertainty.

  • System Model: The coordinated UVLS scheme was tested on a modified IEEE 39-bus system with step-down transformers and air-conditioner motor loads at buses 4, 7, and 18.Each affected load was modeled as 50% single-phase air-conditioner motors and 50% constant-impedance loads.
  • Controller Design: The DQN controller used voltage and load-related observations and selected among eight combinations of 20% shedding actions at buses 4, 7, and 18.The last 10 observation states were stacked, producing 110 input nodes.
  • Test Design: The robustness study covered 960 scenarios varying load level, air-conditioner motor parameters, fault location, and fault duration.The scenarios included 80%, 90%, 110%, and 120% load levels, 30 fault locations, and a 10% parameter increase condition.
  • DRL Versus UVLS: 92.22% of the 462 scenarios requiring load shedding favored DRL over the UVLS relay.These scenarios were those that could lead to FIDVR without corrective action.
  • DRL Versus MPC: DRL outperformed MPC in 57.22% of Test Set A and 90.56% of Test Set B scenarios.Test Set B introduced a 10% increase in Tstall and Vstall to represent modeling gaps or uncertainties.
  • Representative Case: In a 120% load test, total rewards were -1271.61 for DRL, -1548.14 for MPC, and -3778.80 for UVLS.The test included a 0.1 s fault, a 10% increase in Tstall and Vstall, and 1% observation noise.
  • Execution: The well-trained DRL model executed emergency control in 0.13 s on average, supporting real-time emergency-control operation.The reported action time applies during an 8-second simulation event.

VI. DISCUSSIONS

The paper highlights important considerations for applying DRL to power-system emergency control.

  • Discussion: Applying DRL to power-system emergency control requires attention to considerations specific to both DRL and power-system operation.The discussion introduces these considerations without specifying them in the supplied passage.

1) Applicability to general emergency control problems:

Applying DRL to general power-system emergency control depends heavily on formulating the problem properly as an MDP. Because automated formulation remains immature, domain expertise and DRL expertise must work together.

  • Successful DRL application depends heavily on defining appropriate states, actions, and rewards for the emergency-control MDP.
  • Automating the formulation of general emergency-control problems remains at an early research stage.
  • The paper recommends close collaboration between power-domain and DRL experts when designing these MDP formulations.

2) Parameter selection:

The paper identifies manual parameter tuning, simulation-to-reality gaps, and penalty-based safety handling as important boundaries for DRL emergency control. It also points to broader future extensions for RLGC and larger, more uncertain control problems.

  • 2) Parameter selection:: The proposed algorithms manually tune penalty factors and reward-function weights, making parameter selection a recognized DRL challenge.The authors plan to automate this process in future work.
  • 2) Parameter selection:: Training mission-critical power-grid controllers in simulation leaves reality gaps between the models and real-world systems.The authors plan to adapt technologies developed in robotics to address this issue.
  • 2) Parameter selection:: Safety and operating constraints are handled through violation penalties in the reward functions.The paper identifies constrained policy optimization and safe exploration as alternative directions for constrained reinforcement learning.
  • 2) Parameter selection:: The RLGC platform is intended to support future extensions, including other simulators, larger-scale systems, continuous actions, safe exploration, and deep meta-reinforcement learning.These directions target control challenges associated with increased power-system uncertainties.
Loading 1903.03712v2…