PaperlayerView exact revision on arXiv

Source-linked AI summary

Two-stage Deep Reinforcement Learning for Inverter-based Volt-VAR Control in Active Distribution Networks

Haotian Liu, Wenchuan Wu

arXiv:2005.11142v1eess.SYcs.LGcs.SY

TL;DR

Model-based Volt/VAR control relies on network parameters that may differ from actual operating conditions, creating a challenge for active distribution networks. The paper trains an offline agent with adversarial reinforcement learning and transfers it for online learning and control; simulations report improved online safety and efficiency over benchmark methods.

  • Problem

    Network parameters in active distribution networks are not onsite identified, so model-based VVC may use inaccurate models and face significant mismatch.

    Source: §I · p. 1

  • Method

    A two-stage DRL method trains an offline agent with adversarial MDP-based JASAC and transfers it as an online agent for continued learning and control.

    Source: §I · p. 2, §V · pp. 7–8

  • Results

    On IEEE 33-bus and 69-bus test cases, the two-stage method outperforms benchmark methods in online safety and efficiency, while JASAC improves offline convergence and efficiency against the benchmark adversarial algorithm.

    Source: §V · p. 8

  • Takeaways & Limitations

    The reported test cases indicate that offline adversarial training and transfer can support safer and more efficient online VVC despite inaccurate model parameters.

    Source: §V · pp. 7–8

Abstract

from arXiv · show

Model-based Vol/VAR optimization method is widely used to eliminate voltage violations and reduce network losses. However, the parameters of active distribution networks(ADNs) are not onsite identified, so significant errors may be involved in the model and make the model-based method infeasible. To cope with this critical issue, we propose a novel two-stage deep reinforcement learning (DRL) method to improve the voltage profile by regulating inverter-based energy resources, which consists of offline stage and online stage. In the offline stage, a highly efficient adversarial reinforcement learning algorithm is developed to train an offline agent robust to the model mismatch. In the sequential online stage, we transfer the offline agent safely as the online agent to perform continuous learning and controlling online with significantly improved safety and efficiency. Numerical simulations on IEEE test cases not only demonstrate that the proposed adversarial reinforcement learning algorithm outperforms the state-of-art algorithm, but also show that our proposed two-stage method achieves much better performance than the existing DRL based methods in the online application.

I. INTRODUCTION

Volt/VAR control can reduce voltage violations and network losses, but model-based control depends on network parameters that may differ from actual operating conditions. The paper proposes offline adversarial training followed by online transfer and learning to address model mismatch and improve online safety and efficiency.

I. INTRODUCTION

  • Increasing distributed generation, especially with reversed active-power flow, aggravates voltage violations and network losses, motivating Volt/VAR support from inverter-based resources.The passage notes that many such resources produce less active power than their rated capacity.Source: §I · p. 1
  • ADN parameters are theoretical rather than onsite-identified, so model-based VVC can involve significant errors and be hindered in real-world application.Offline simulation is needed when historical data is unsuitable, but inaccurate simulation models create a transfer gap.Source: §I · p. 1
  • The proposed two-stage DRL method transfers an offline-trained agent to online learning and application to improve online safety and efficiency.The offline stage uses approximate models; the online stage learns and controls with the transferred agent.Source: §I · pp. 1–2
  • The offline stage formulates an adversarial MDP and trains an adversary to expose model errors, making the offline agent robust to mismatch.Joint adversarial soft actor-critic shares information between agents to improve training efficiency and convergence.Source: §I · p. 2

II. PRELIMINARIES

  • Section II introduces MDP and adversarial MDP preliminaries before formulating the VVC problem for active distribution networks.The passage describes the section’s organization.Source: §II · p. 2

A. Markov Decision Process and Reinforcement Learning

The paper describes online reinforcement learning as agent-environment interaction under an MDP, then extends that setup to an adversarial MDP for offline training. In the extension, a protagonist and adversary respectively maximize and minimize the protagonist’s expected return.

A. Markov Decision Process and Reinforcement Learning

  • An online agent interacts with an environment through an MDP, where the next state depends on the current state and action.The paper uses an observation space because the exact real-world state may be unavailable, treating unobserved states as noise.Source: A. Markov Decision Process and… · p. 2
  • The MDP uses continuous state and action spaces, a transition probability, a reward for each transition, and a discount factor trading off current and future rewards.The state-value and action-value functions represent expected discounted rewards under a policy.Source: A. Markov Decision Process and… · p. 2
  • The policy is a stochastic distribution over actions given the current state, and the learning goal is to maximize expected discounted reward.The passage expresses the objective as maximizing J(π).Source: A. Markov Decision Process and… · p. 2

B. Adversarial Markov Decision Process

  • For offline training, the adversarial MDP models the protagonist and an adversary that mocks the transfer gap as a two-player discounted zero-sum Markov game.The adversary’s action space represents modeling errors, such as parameter errors.Source: B. Adversarial Markov Decision Process · p. 2
  • In the adversarial MDP, the protagonist maximizes expected return while the adversary minimizes it through its policy.The expanded transition and reward functions depend on the protagonist’s and adversary’s actions.Source: B. Adversarial Markov Decision Process · pp. 2–3

C. VVC Problem Formulation

The VVC problem is formulated as an online MDP and an offline adversarial MDP, using observations of network injections and voltages. The protagonist controls reactive-power resources, while the offline adversary perturbs line parameters within specified deviation ranges.

C. VVC Problem Formulation

  • The VVC problem is formulated as an MDP online and an adversarial MDP offline.State, action, and reward definitions are specified for both formulations.Source: C. VVC Problem Formulation · p. 3

1) State Space:

  • The state is the observation vector of nodal active and reactive power injections, voltage magnitudes, and episode time.The paper assumes the observation space equals the state space because the full ADN state is too complex to perceive.Source: 1) State Space · p. 3

2) Action Space:

  • The protagonist’s action space covers reactive-power outputs from inverter-based energy resources and SVCs.These devices provide high-speed reactive-power support; slower shunt capacitors are generally scheduled offline because daily switching is limited.Source: 2) Action Space · p. 3
  • The protagonist action is defined by the reactive-power generation vectors of inverter-based resources and SVCs.The passage denotes the vectors as QG and QC.Source: 2) Action Space · p. 3
  • The offline adversary’s actions are deviations in branch resistance and reactance, used to represent modeling errors.The simulation parameters are calculated using these deviations and Equation (2).Source: 2) Action Space · p. 3
  • The deviation ranges determine adversarial force; an excessively strong adversary can make the offline agent overly conservative and training unstable.The passage notes that real line-parameter deviations are limited by factors such as temperature, shape, and manufacturing tolerance.Source: 2) Action Space · p. 3

3) Reward Function:

The reward combines penalties for active power loss and voltage violations, balancing the two VVC objectives with a weight. Its voltage-violation term remains useful when available regulation cannot eliminate every violation.

3) Reward Function:

  • The reward is the weighted sum of an active-power-loss penalty and a voltage-violation penalty, with Cv setting their weight ratio.The stated form is rt = RP(t) + CvRV(t).Source: 3) Reward Function · p. 3
  • The active-power-loss penalty RP is calculated from active-power injections P(t).The supplied equation gives RP(t) = −Σ_i Pi(t).Source: 3) Reward Function · p. 3
  • The voltage-violation penalty RV uses the 2-norm of voltage-magnitude violations and is zero only when all voltage magnitudes meet constraints.The paper calls this quantity the voltage violation rate (VR).Source: 3) Reward Function · p. 3
  • When regulation capacity cannot eliminate every violation, RV can still mitigate the remaining voltage violations.Source: 3) Reward Function · p. 3

III. METHODS

The method trains a transferable offline agent against model mismatch, then uses it for online VVC with continuous learning. Its offline adversarial training shares information between agents, while SAC supplies the off-policy online learning framework.

A. Overview of the Two-stage DRL-based VVC

  • The offline agent is adversarially trained against modeling errors for robustness, then transferred as the online agent for continuous learning and VVC.The online stage uses SAC because direct online training can create severe security problems or prohibitive start-up cost.Source: A. Overview of the Two-stage DRL-based… · pp. 3–4
  • JASAC shares value-function information between adversary and protagonist, promoting training efficiency and convergence, particularly with large state and action spaces.The adversary perturbs model parameters to expose difficult scenarios for the protagonist.Source: A. Overview of the Two-stage DRL-based… · p. 4

B. Soft Actor-Critic

  • SAC is an off-policy maximum-entropy method, unlike on-policy PPO and A3C, whose sample efficiency is described as often unsatisfactory.Source: B. Soft Actor-Critic · p. 4
  • SAC reparameterizes squashed-Gaussian actions using noise samples and updates its Q-network from replay-buffer samples with an entropy-regularized Bellman target.The paper notes practical stabilizing techniques including automatic α optimization, frozen targets, and two Q-functions.Source: B. Soft Actor-Critic · p. 4

C. Jointly Adversarial Soft Actor-Critic

JASAC is an off-policy, maximum-entropy adversarial RL algorithm for the AMDP. It alternately optimizes both agents while sharing a state-action value function, addressing efficiency and stability concerns associated with separate on-policy adversarial training.

C. Jointly Adversarial Soft Actor-Critic

1) Learning Q-Functions:

JASAC uses a joint state-action value function for the protagonist and adversary because they participate in the same zero-sum game. Its shared estimate incorporates both current policies during value approximation.

1) Learning Q-Functions:

  • JASAC replaces separate value approximators with a joint Q-function because both agents in the zero-sum game share the same equilibrium state value.The joint function takes state and both agents' actions as inputs.Source: 1) Learning Q-Functions · p. 5
  • The shared Q-function approximates value while accounting for both current policies, rather than updating their value estimates separately.Source: 1) Learning Q-Functions · p. 5
  • Delayed target Q-networks supply the Bellman target used to update the learned Q-functions by minimizing mean-squared Bellman error.Source: 1) Learning Q-Functions · p. 5

2) Learning the Policy:

Policy learning uses stochastic actions and joint value estimates that account for both agents. The protagonist is then transferred to online SAC, where no adversary is used.

2) Learning the Policy:

  • The policy value is evaluated over both action spaces, with Gaussian noise passed through tanh to reparameterize each agent’s stochastic action.The policy parameters are independent in the entropy terms, and the policies are learned alternately by descent.Source: 2) Learning the Policy · p. 5
  • Joint Q-functions estimate state value using both the agent’s own action and its opponent’s action, so policy updates account for the opponent’s stochastic behavior.The passage says this corrects the search direction.Source: 2) Learning the Policy · p. 5
  • The protagonist policy transfers directly as the SAC policy for online learning; the adversary is absent online.The passage also notes that the adversary-dependent Q* is not carried into the online stage.Source: 2) Learning the Policy · p. 5

IV. NUMERICAL STUDY

The numerical study evaluates the proposed approach on IEEE 33-bus and 69-bus distribution test cases against DRL and optimization-based benchmarks.

IV. NUMERICAL STUDY

  • The evaluation uses IEEE 33-bus and 69-bus distribution cases to compare the proposed method with DRL and optimization-based benchmarks.The environments follow the Gym toolkit scheme.Source: §IV · pp. 5–6
  • The JASAC procedure initializes experience and function approximators, collects environment transitions, and updates the value function and both agents’ policies.Its output includes a transferable protagonist state-action value.Source: §IV · p. 6

A. Proposed and Baseline Algorithms Setup

The study compares the offline algorithms and online transfer strategy across IEEE distribution cases. Online, JASAC and preSAC outperform SAC; offline, JASAC improves adversarial-training efficiency and convergence relative to ASAC.

A. Proposed and Baseline Algorithms Setup

  • The offline benchmarks include JASAC, ASAC, SAC, and model-based VVO variants; online comparisons also include preSAC, with five independent random seeds per experiment group.The optimization-based variants use oracle or fake parameters, and the VVO benchmark uses SOCP relaxation.Source: A. Proposed and Baseline Algorithms… · p. 6

B. Offline Convergence and Efficiency

  • JASAC improves training efficiency and convergence rate over ASAC while achieving efficiency and convergence similar to SAC, with shared value information cited as the mechanism.The adversarial methods converge above SAC, reflecting a training-performance trade-off for protagonist robustness to model mismatch.Source: B. Offline Convergence and Efficiency · pp. 6–7, Figure 2 · p. 6

C. Online Safety and Efficiency

V. CONCLUSION

The paper proposes two-stage DRL for inverter-based ADN reactive-power control without accurate model parameters. IEEE 33-bus and 69-bus studies report better online safety and efficiency than benchmarks, alongside improved offline JASAC convergence and efficiency over benchmark adversarial learning.

V. CONCLUSION

  • The proposed two-stage DRL method targets reactive-power optimization in inverter-based ADNs without accurate model parameters.Its central novelty is transfer of knowledge from offline training to online control.Source: §V · pp. 7–8
  • Offline JASAC formulates an AMDP to make the offline agent robust to the transfer gap, then transfers that agent for continuous online learning and control using SAC.Source: §V · p. 8
  • IEEE 33-bus and 69-bus studies indicate better online safety and efficiency than benchmark methods and better offline JASAC convergence and efficiency than benchmark adversarial learning.Applying JASAC to more complex control problems is identified as future work.Source: §V · p. 8

Related papers

Loading 2005.11142v1…