PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning based Model-free On-line Dynamic Multi-Microgrid Formation to Enhance Resilience

Jin Zhao, Fangxing Li, Srijib Mukherjee, Christopher Sticht

arXiv:2203.03030v1eess.SYcs.SY

TL;DR

Changing conditions can weaken initially formed microgrids, making adaptive MMGF important for resilience. This paper develops a model-free deep reinforcement learning scheme that reduces the topology-action search burden and improves learning. In case studies, it preserves all loads in the 7-bus comparison and returns feasible strategies for the IEEE 123-bus system in about 0.1 seconds.

Abstract

from arXiv · show

Multi-microgrid formation (MMGF) is a promising solution to enhance power system resilience. This paper proposes a new deep reinforcement learning (RL) based model-free on-line dynamic multi-MG formation (MMGF) scheme. The dynamic MMGF problem is formulated as a Markov decision process, and a complete deep RL framework is specially designed for the topology-transformable micro-grids. In order to reduce the large action space caused by flexible switch operations, a topology transformation method is proposed and an action-decoupling Q-value is applied. Then, a CNN based multi-buffer double deep Q-network (CM-DDQN) is developed to further improve the learning ability of original DQN method. The proposed deep RL method provides real-time computing to support on-line dynamic MMGF scheme, and the scheme handles a long-term resilience enhancement problem using adaptive on-line MMGF to defend changeable conditions. The effectiveness of the proposed method is validated using a 7-bus system and the IEEE 123-bus system. The results show strong learning ability, timely response for varying system conditions and convincing resilience enhancement.

I. INTRODUCTION

Extreme events and uncertain operating conditions motivate adaptive multi-microgrid formation to sustain distribution-system resilience. The paper addresses the challenges of dynamic formation with a model-free deep reinforcement learning scheme designed to reduce the action space and improve learning.

I. INTRODUCTION

  • Microgrids can operate grid-connected or islanded, and self-supported MGs are presented as a way to improve restoration capability and critical-load survival.The motivation links this potential to distributed energy resources and supporting metering, communication, and automatic-control infrastructure.Source: §I · p. 1
  • Prior MMGF approaches include mathematical programming and heuristic search, with work addressing outage-area sectionalization, post-outage topology, and large-system computation.The introduction also notes studies of resilient MG formation, resource allocation, self-healing, and islanded-MG control.Source: §I · p. 1
  • Uncertain renewable generation and unexpected grid damage can undermine initially formed microgrids, motivating adaptive formation under changing conditions.The introduction describes existing formation strategies as focused mainly on observable conditions and short-term considerations.Source: §I · p. 1
  • DRL-based MMGF is challenging because feasible radial topologies are difficult to ensure and the action space grows exponentially with the number of switches.The paper identifies these issues as obstacles to learning ability.Source: §I · p. 2
  • The proposed framework combines an MDP formulation, topology transformation, CNN-based action-decoupling Q-values, and CM-DDQN to support adaptive on-line MMGF.The authors describe CM-DDQN as improving learning ability and computational performance for real-time strategies based on updated system information.Source: §I · p. 2

II. MMGF PROBLEM FORMATION USING DRL FRAMEWORK

The paper formulates dynamic MMGF as a sequential MDP in which a distribution-system operator selects topology-changing actions from observed conditions. The formulation includes network and supply constraints in its reward and accounts for cumulative discounted return.

II. MMGF PROBLEM FORMATION USING DRL FRAMEWORK

  • Dynamic MMGF is a sequential process that adaptively reconfigures and splits a distribution system into islanded microgrids based on current conditions.The objective is to maintain load supply over time under changing environments.Source: §II · p. 2
  • The MDP represents the operator's state-based action, resulting reward, and transition to the next state until termination or the final step.Its four elements are state, action, transition probability, and reward.Source: §II · p. 2
  • The state includes switch configuration, DG active and reactive outputs, load active and reactive amounts, and time.The state is defined from the current environment at each time step.Source: §II · p. 2
  • The action space comprises topology configurations and has 2^w combinations, creating a large search space for DRL.The passage notes that this scale can cause non-convergence; topology handling is deferred to Section III.Source: §II · p. 2
  • The reward combines a topology-formation reward with penalties for AC power-flow results, frequent switch operations, and insufficient power supply.The return accumulates current reward and discounted future rewards to represent long-term feedback.Source: §II · p. 3

B. Dynamic MMGF using deep Q-learning

The paper uses deep Q-learning to select actions in the dynamic MMGF MDP, estimating action values from rewards and future value. A neural network approximates these values, and its loss trains the network toward Q-value targets.

B. Dynamic MMGF using deep Q-learning

  • Deep Q-learning fits MMGF's discrete action space and continuous state space by iteratively updating an action-value function.The method combines a deep neural network with Q-learning.Source: B. Dynamic MMGF using deep Q-learning · p. 3
  • The Q-value represents expected discounted reward for an action and state under a policy, while an optimal policy selects the highest-valued action.The paper describes the value as guiding MMGF decisions.Source: B. Dynamic MMGF using deep Q-learning · p. 3
  • The Q-value is approximated by a neural network because continuous states make a reasonable Q-table impractical for dynamic MMGF.This approximation transforms Q-learning into DQN.Source: B. Dynamic MMGF using deep Q-learning · p. 3
  • The Q-network is trained by minimizing mean-squared temporal-difference error between its current estimate and a target combining immediate reward with future value.The passage describes the target terms as measuring the current action's value.Source: B. Dynamic MMGF using deep Q-learning · p. 3

III. ACTION GENERATION OF DYNAMIC MMGF PROCESS

The action-generation method reduces the search burden of dynamic MMGF by transforming the topology problem and restricting switch combinations while maintaining radial networks.

A. Search space reduction of spinning forest

  • The original action space has 2^w switch combinations, including many infeasible configurations that violate the radial-network requirement.These infeasible configurations also make power-flow-based environment interaction difficult.Source: A. Search space reduction of spinning… · p. 3
  • Multi-MG formation can be represented as reconfiguring the original radial distribution system and then splitting it by opening closed switches.The process seeks spanning forests, which are intractable to trace directly.Source: A. Search space reduction of spinning… · pp. 3–4, Figure 2 · p. 4
  • The topology transformation reduces the search to selecting a subset of remote-controlled switches, yielding significantly fewer combinations than the original 2^w space.The transformed network includes virtual nodes and lines; the stated reduction is relative to the original action space.Source: A. Search space reduction of spinning… · p. 4

B. CNN-based action-decoupling Q-value

The CNN-based action-decoupling method represents switch values in the network output and derives an action-level Q-value from selected switch values.

B. CNN-based action-decoupling Q-value

  • The Q-value reflects expected discounted rewards, while the MMGF reward depends on AC power flow and neighboring branches have stronger topology relationships.These properties motivate organizing the neural network with a CNN for feature learning.Source: B. CNN-based action-decoupling Q-value · p. 4
  • The CNN outputs values V for individual switches; the method selects the top n-ng values and averages them to form the Q-value.The selected switch indices are closed, and the remaining switches are opened to form the new switch status.Source: Figure 5 · p. 4, B. CNN-based action-decoupling Q-value · p. 5
  • Input and output data are min-max normalized to [0, 1], while switching-operation frequencies are normalized by the total step number.The passage states that this normalization helps create a more regular search region for faster convergence.Source: B. CNN-based action-decoupling Q-value · p. 5

IV. LEARNING AND APPLICATION OF CM-DDQN

The CM-DDQN learning and application design combines replay-based training, exploration, and a fixed target network to improve learning for dynamic MMGF.

A. Techniques for better learning

  • Multiple buffers preserve high-reward experiences with feasible topologies, and replay samples from the buffers to form training minibatches.The replay mechanism allows experiences to be reused, while the paper states that replay can eliminate bias from correlations between training samples.Source: A. Techniques for better learning · p. 5
  • Epsilon-greedy exploration uses an annealed ε value to guide the choice between Q-based action selection and random actions.The passage describes ε as gradually decreasing after DQN training begins.Source: A. Techniques for better learning · p. 5
  • DDQN separates action selection by the original Q network from action evaluation by a fixed target Q network, addressing overestimation in Q-learning updates.The paper states that the fixed-network feature enhances learning efficiency and stability.Source: A. Techniques for better learning · pp. 5–6
  • The paper presents experience replay, exploration, and DDQN as additions to the original deep Q-learning structure for the CM-DDQN method.These designs are described as addressing overestimation and providing a better learning process.Source: A. Techniques for better learning · p. 6

B. CM-DDQN learning process

The learning process uses a reward function to train the Q-network, with penalties tied to topology, power flow, power balance, and switch operation.

B. CM-DDQN learning process

  • The detailed reward function is used to determine the Q-network for the dynamic MMGF problem.The paper identifies reward-function design as an indispensable part of the DRL learning process.Source: B. CM-DDQN learning process · p. 6
  • An infeasible topology receives a penalty and triggers the MDP game-over signal, while a feasible topology receives the topology reward.The topology signal indicates whether the switch action forms multi-MGs with radial networks.Source: B. CM-DDQN learning process · p. 6
  • The reward penalizes voltage violations, power loss, branch overflow, excessive switch operations, and power-deficient microgrids.Switch-operation penalties apply when a switch exceeds its allowed operation count over the dynamic MMGF process.Source: B. CM-DDQN learning process · p. 6

1 DG,…, q

The learning process treats feasible topology as a hard constraint and other operating limits as soft constraints that shape rewards.

1 DG,…, q

  • The algorithm records state-action-reward transitions, adds qualifying experiences to separate replay memories, and updates the Q-network when replay conditions are met.Source: §1 · p. 6
  • A topology violation ends the Markov process, making feasible radial topology the hard constraint.The switch actions must preserve a radial network for each microgrid.Source: §1 · p. 7
  • Voltage, branch flow, switching number, and power balance violations incur reward penalties rather than immediately ending an episode.Source: §1 · p. 7

C. Deep RL based dynamic model-free MMGF scheme

The scheme uses a model-free, online DRL agent to adapt microgrid topology as system conditions change. In the 7-bus case, it retained all load where the conventional formation shed 18 MW.

C. Deep RL based dynamic model-free MMGF scheme

  • The pre-trained Q-network proposes switch actions online, while a topology check blocks infeasible actions and preserves the last feasible radial network.New experience can further update the network when training conditions are triggered.Source: C. Deep RL based dynamic model-free… · p. 7
  • The model-free agent computes switch decisions without the actual power-system model or power-flow equations, supporting real-time online application.Source: C. Deep RL based dynamic model-free… · p. 7

A. DRL based dynamic MMGF process using 7-bus system

  • After almost 350 episodes, the proposed method learned to form feasible topologies through switch decisions over the dynamic multistep process.A 20-step feasible topology constitutes a successful process; infeasibility ends an episode.Source: A. DRL based dynamic MMGF process… · p. 8
  • 18 MW of load was shed by conventional MMGF, while Dy_MMGF retained all system load under the same DG-output conditions.The dynamic scheme changed the MG formation as DG1 output fell to 45–49 MW in steps 17–20.Source: A. DRL based dynamic MMGF process… · p. 8, Figure 11 · p. 8

B. Performance of CM-DQN in IEEE 123-bus system

On the modified IEEE 123-bus system, CM-DDQN learned feasible topologies more stably than DDQN and CNN-based DDQN without multi-buffering. Its online application produced feasible strategies in about 0.1 seconds, with some soft-constraint violations in testing.

B. Performance of CM-DQN in IEEE 123-bus system

  • With an improved CNN structure and multi-buffer design, CM-DDQN captured the feasible-topology feature after about 1000 episodes; without multi-buffer, C-DDQN sometimes reached 20 successful MMGF steps but quickly lost the features.Source: B. Performance of CM-DQN in IEEE… · p. 9
  • CM-DDQN return became relatively stable, whereas DDQN and C-DDQN failed to evaluate actions properly.The authors attribute stability to CNN feature learning and the multi-buffer design; random exploration caused a large fluctuation during steps 1000–1200.Source: B. Performance of CM-DQN in IEEE… · p. 9, Figure 13 · p. 9
  • In 100 MDP runs, the large-scale system had voltage violations at 5 buses over 4 steps, all within 0.005 p.u.; three runs exceeded the switch-operation limit.The small-scale system had no soft-constraint violations, and neither system had hard-constraint violations.Source: B. Performance of CM-DQN in IEEE… · p. 9, Table I · p. 10
  • The online strategy took about 0.1 seconds even for the 123-bus system.The reported computation time increased only slightly from the 7-bus to the 123-bus system.Source: B. Performance of CM-DQN in IEEE… · pp. 9–10

VI. Conclusions

The proposed online dynamic MMGF scheme uses a DRL framework to adapt microgrid formation under changing conditions. Case studies report resilience enhancement through feasible topology adjustment that holds all loads.

VI. Conclusions

  • The scheme enhances resilience by holding all loads through feasible topology adjustment.The case study results also report a satisfactory security guarantee.Source: §VI · p. 10
  • A DRL-based MDP supports long-term dynamic MMGF, while topology transformation and action-decoupled CNN Q-values address large action spaces.Source: §VI · p. 10
  • The proposed CM-DDQN is reported to have strong learning ability, real-time computation speed, and a satisfactory security guarantee.Source: §VI · p. 10

Related papers

Loading 2203.03030v1…