PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Multi-agent Reinforcement Learning for Decentralized Inverter-based Volt-VAR Control

Haotian Liu, Wenchuan Wu

arXiv:2006.12841v2eess.SYcs.LGcs.MA

TL;DR

The paper addresses VVC under incomplete ADN models, difficult-to-maintain P2P communication, and safety challenges in online exploration. It proposes OLDC with MACSAC, combining online multi-agent learning, decentralized local decisions, stochastic policies, and explicit constraints. Simulations report more stable and efficient performance than comparison learning methods and support the framework’s online applicability.

  • Problem

    Existing VVC methods rely on accurate ADN models or difficult-to-maintain P2P communication, while online exploration can deteriorate control performance and constraints are often not explicit.

  • Method

    OLDC trains MACSAC online using maximum-entropy stochastic policies and explicitly modeled voltage constraints, then performs decentralized control from local measurements.

  • Results

    MACSAC achieves more stable and efficient training and application than MADDPG, with better power loss and voltage-violation performance in reported simulations.

  • Takeaways & Limitations

    The framework supports continuous learning for incomplete models while retaining fast local decisions and reducing redundant online exploration costs.

Abstract

from arXiv · show

The distributed Volt/Var control (VVC) methods have been widely studied for active distribution networks(ADNs), which is based on perfect model and real-time P2P communication. However, the model is always incomplete with significant parameter errors and such P2P communication system is hard to maintain. In this paper, we propose an online multi-agent reinforcement learning and decentralized control framework (OLDC) for VVC. In this framework, the VVC problem is formulated as a constrained Markov game and we propose a novel multi-agent constrained soft actor-critic (MACSAC) reinforcement learning algorithm. MACSAC is used to train the control agents online, so the accurate ADN model is no longer needed. Then, the trained agents can realize decentralized optimal control using local measurements without real-time P2P communication. The OLDC with MACSAC has shown extraordinary flexibility, efficiency and robustness to various computing and communication conditions. Numerical simulations on IEEE test cases not only demonstrate that the proposed MACSAC outperforms the state-of-art learning algorithms, but also support the superiority of our OLDC framework in the online application.

I. INTRODUCTION

VVC in active distribution networks must address voltage violations and losses despite incomplete models and burdensome communication. The paper proposes OLDC with MACSAC to combine online learning, decentralized local control, explicit constraints, and asynchronous operation.

  • Motivation: High distributed-generation penetration increases voltage-violation and network-loss concerns, while inverter-based resources can provide fast Volt/VAR support.VVC is used to optimize voltage profiles and reduce losses using inverter free capacity.
  • Limitations of existing control: Centralized VVC faces single-point failure, computation and communication burdens, while distributed methods depend on P2P communication that is difficult to maintain.Communication-dependent delays become more problematic as inverter-based resources and network size increase.
  • Model uncertainty: Most VVC algorithms require accurate ADN models, which are impractical and expensive for complex systems with numerous buses.Reinforcement learning is introduced as a model-free route for incomplete-model challenges.
  • Research gap: Existing multi-agent approaches either train offline using simulation models or learn synchronously online with heavy communication and computation burdens.The paper identifies a need for online learning without heavy communication or local computation requirements.
  • Proposed approach: OLDC coordinates MACSAC learning in a control center while sending trained policies to local controllers for decentralized control.Its sampling, learning, and application processes are asynchronous, and MACSAC uses stochastic policies with explicit voltage constraints.

B. Markov Games and Reinforcement Learning

The paper formulates multi-agent VVC as a constrained Markov game in which agents act from local observations, receive rewards, and satisfy auxiliary cost constraints. Reinforcement learning then seeks policies from sequential data without requiring transition probabilities.

  • Constrained Markov games: A constrained Markov game extends a Markov game with local observations, actions, rewards, common state transitions, and auxiliary cost constraints.The framework models multiple agents interacting with a shared environment while imposing safety-related expectations.
  • Sequential decision process: At each time step, every agent observes locally, samples an action from a stochastic policy, and receives a reward after the environment transitions.The transition probability is treated as unknown in the reinforcement-learning setting.
  • Optimization objective: The learning objective is to maximize each agent’s expected return while satisfying its corresponding auxiliary-cost constraint.The policy-learning formulation explicitly separates reward maximization from constrained costs.
  • Model-free learning: Reinforcement learning can optimize agents from sequential decision data without knowledge of transition probability distributions, enabling model-free optimization.This property addresses settings where accurate system models are unavailable.

C. Actor-Critic and Multi-agent Actor-Critic

Actor-critic methods use learned actors and critics, while multi-agent variants combine local actors with critics that can access global information during training. The paper applies this structure to online, asynchronous, decentralized VVC learning.

  • Actor-critic methods: Actor-critic algorithms alternate training an actor that generates actions from observations and a critic that evaluates the policy.The paper identifies sample efficiency and stability as motivations for this family of methods.
  • Multi-agent actor-critic: Multi-agent actor-critic architectures use a local actor and critic for each agent, with critics allowed to use global information during training.This design addresses the non-stationary environment faced by individual agents.
  • Online learning: OLDC replaces offline training with online learning and control through totally asynchronous sampling, training, and application.The framework is designed to adaptively operate active distribution networks during the online stage.
  • Decentralized control: In OLDC, agents perform decentralized control without direct communication while data is asynchronously processed in a control center for policy updates.The asynchronous centralized learning does not affect the control process.

A. VVC Formulation in Constrained Markov Game

The paper formulates VVC as a constrained Markov game using local measurements, controllable reactive-power actions, coordinator-evaluated rewards, and explicit action, entropy, and voltage constraints. MACSAC addresses these constraints with stochastic policies and dynamically updated multipliers.

  • A. VVC Formulation in Constrained Markov Game: VVC is formulated as a constrained Markov game with state, local observations, actions, rewards, and constraints for multiple agents.The formulation uses the constrained Markov-game setting to represent VVC decisions across areas.
  • 2) Observation Spaces:: Each agent’s local observation is based on local measurements, while its action space contains controllable reactive-power resources such as PV inverters and SVCs.The action space includes reactive-power outputs for devices in the agent’s area.
  • 4) Reward and Cost Functions:: Agent rewards are computed at the coordinator from available observations to minimize active-power loss and voltage violations, with βi weighting global welfare.The coordinator evaluates the reward and cost functions using system-wide observations.
  • 4) Reward and Cost Functions:: The voltage-violation rate is a nonnegative 2-norm measure that equals zero exactly when all voltage magnitudes satisfy their constraints.VVR provides a smoother index than counting violated nodes when violations cannot all be eliminated.
  • B. Multi-agent Constrained Soft Actor-Critic: MACSAC reformulates each agent’s problem with an original objective, bounded actions, an entropy constraint, and a state constraint based on discounted voltage-violation rate.The action constraint is encoded in the action space, while entropy and state constraints are handled in the optimization formulation.
  • B. Multi-agent Constrained Soft Actor-Critic: Action bounds are built into the action spaces, whereas Lagrange relaxation handles entropy and state constraints through multipliers αi and λi.The two multiplier pairs correspond to entropy and state-constraint terms.
  • B. Multi-agent Constrained Soft Actor-Critic: Dynamic updates of λi replace a fixed voltage-penalty hyperparameter and are intended to guarantee the safety of the proposed algorithm.The paper contrasts this with directly penalizing voltage constraints using an inappropriate fixed penalty.
  • B. Multi-agent Constrained Soft Actor-Critic: MACSAC uses stochastic policies rather than MADDPG’s deterministic policies to reduce overfitting and instability while improving training robustness.The stochastic policy design is paired with the constrained optimization formulation.

1) Preparation:

The actor policy is sampled stochastically by transforming Gaussian neural-network outputs through a tanh function. The resulting state-action values represent expected discounted rewards used in learning.

  • 1) Preparation:: Each policy samples actions by applying tanh to a neural-network mean plus standard deviation multiplied by Gaussian noise.The noise is drawn from a standard multivariate normal distribution.
  • 1) Preparation:: The policy’s mean and standard deviations are approximated by neural networks.These parameters define the stochastic action distribution.
  • 1) Preparation:: The state-action value function represents the expected discounted reward after taking an action under policy π.The trajectory is generated by applying π.
  • 1) Preparation:: Experience tuples containing observations, actions, rewards, and next observations are stored in a replay buffer for alternating critic and actor learning.The stored transition is represented as {x, a, r, x′}t.

2) Learning the critics:

MACSAC learns centralized critics and stochastic actors from replayed experience, using delayed target networks and multiplier updates. Its off-policy design permits locally executed sampling policies to lag behind the latest trained policies.

  • 2) Learning the critics:: Centralized critics use all agents’ observations and actions to address the non-stationarity seen from an individual agent’s perspective.Because the agents are cooperative, other agents’ policies are available when training a critic.
  • 2) Learning the critics:: Bellman updates approximate current state-action values from expected next states and actions under the policy.The critic update is based on the Bellman equation.
  • 2) Learning the critics:: Delayed target parameters are updated by a weighted combination of current and previous parameters during critic learning.The delayed parameters are denoted by ˆφi and updated using η.
  • 3) Learning the actors:: Actor optimization replaces the difficult return objective with an approximation using learned reward critics, cost critics, and entropy terms.The transformed objective is optimized with respect to the actor parameters.
  • 3) Learning the actors:: MACSAC combines stochastic maximum-entropy policies with explicit per-agent constraints, contrasting with MADDPG’s deterministic unconstrained formulation.The paper associates the stochastic design with higher sample efficiency and stability and the constraints with explicit voltage safety.
  • 2) Learning the critics:: The algorithm initializes replay and function approximators, samples locally, updates critics and actors in parallel, and then updates multipliers and target networks.The procedure uses off-policy replayed transitions and parallel per-agent updates.
  • 2) Learning the critics:: Because MACSAC is off-policy, locally executed sampling policies need not be the latest policies, enabling the OLDC framework’s asynchronous operation.Sampling, training, and control can therefore proceed without requiring synchronized current policies.

C. Online Centralized Training and Decentralized Execution Framework

OLDC separates online learning from decentralized control through asynchronous sampling, learning, and application processes. The framework carries out MACSAC online with high efficiency.

  • OLDC carries out MACSAC online through totally asynchronous sampling, learning, and application processes.

1) Timing:

OLDC performs local control continuously while uploading samples and updating policies asynchronously. This separates application from slower communication and centralized computation, improving robustness to practical resource conditions.

  • Application: Each agent obtains local measurements, samples actions from its local policy, and sends them to controlled devices without centralized communication.The application interval depends on measurement, policy-computation, and device speeds.
  • Sampling: Samples are uploaded asynchronously at intervals longer than the local control interval, so communication does not delay application.Lost samples can be safely ignored, and the upload quantity can be adapted to communication conditions.
  • Learning: Agents are trained asynchronously from replay-buffer batches, and updated policies are sent back without delaying application or sampling.Training occurs on a centralized server with relatively greater communication and computation requirements.
  • Resource allocation: Local controllers evaluate small neural networks with little computation, while most MACSAC computation runs on a resource-rich centralized server.This design requires no communication among local controllers or with the upper control center during application.
  • Exploration and exploitation: OLDC balances exploration and exploitation by uploading only selected samples and can reduce or stop online learning by changing m and Ts.Setting m = 0 stops learning, while properly tuned m and Ts can improve online application efficiency.
  • Special case: The framework remains applicable to single-agent actor-critic reinforcement learning, with asynchronous sampling, training, and execution when needed.
  • Evaluation: OLDC is designed to remain robust under varied communication and computation conditions.The experiments use independent random seeds and report mean values with error bounds.
  • Evaluation: Experiments adapt MACSAC, MADDPG, CSAC, and optimization-based VVO benchmarks to 33-bus and 141-bus active distribution networks.The study uses steady-state power-system environments and balanced power-flow simulations.

B. Algorithm Convergence and Efficiency with Ideal Simulation

Under ideal and online conditions, MACSAC is compared with centralized and multi-agent reinforcement-learning baselines and model-based VVO. It achieves lower losses and voltage-violation measures than competing learning methods while approaching oracle performance in ideal tests.

  • Ideal simulation: In ideal tests, CSAC and MACSAC converge to lower active power loss than approximate-model optimization, while DRL methods approach oracle VVO after iterations.Oracle VVO achieves the theoretical minimum when given true parameters.
  • Ideal simulation: MACSAC achieves performance similar to centralized CSAC using only local measurements during application.CSAC uses global measurements, giving it an advantage in the ideal centralized comparison.
  • Ideal simulation: MACSAC outperforms MADDPG on active power loss and VVR within limited training steps.The paper attributes the improvement to maximum-entropy regularized stochastic policies rather than deterministic policies.
  • Online application: Online tests impose slower communication, real-system exploration, and eight-step training and sampling intervals for learning methods.MACSAC and MADDPG use OLDC with Tu = Ts = 8 and m = 1, while VVO remains an ideal lower-bound reference.
  • Online application: In online application, MACSAC achieves smaller active power loss and VVR than CSAC.The comparison is reported for real-world simulations of 33-bus and 141-bus cases.
  • Online application: Under OLDC, MACSAC converges to better power loss and VVR than MADDPG with more stable performance.MACSAC does not reach the exact oracle optimum but outperforms the other reinforcement-learning methods in online application.

D. Unbalanced Active Distribution Network

The proposed OLDC framework remains applicable to unbalanced distribution networks using input/output data and modest observation and reward modifications. In the IEEE 37-bus simulation, MACSAC achieved near-optimal, more stable control than comparison methods while reducing losses and voltage violation rate.

  • Method: The method extends to unbalanced ADNs without requiring the controlled network’s detailed internal model.Only the observation spaces and reward functions require modification.
  • Experimental setup: For the IEEE 37-bus feeder, the experiment used three PV inverters, one SVC, three stations, and OpenDSS simulation.The case evaluates online application under the same setting as the preceding section.
  • Results: MACSAC achieved performance similar to the optimal point in the unbalanced distribution network.This supports applicability beyond balanced-network cases.
  • Results: Compared with centralized CSAC, MACSAC with OLDC achieved smaller active power loss and voltage violation rate.The comparison concerns the online application in the unbalanced-network case.
  • Results: Compared with MADDPG, MACSAC with OLDC converged to a better operation point with more stable performance.The result is reported for the IEEE 37-bus unbalanced-network simulation.
  • Overall comparison: Across the summarized comparisons, OLDC with MACSAC learns near-optimal VVC without accurate models and supports decentralized control with local measurements.The framework also incorporates stochastic policies, explicit voltage constraints, timing design, and adjustable sampling ratios.

APPENDIX A HYPERPARAMETERS

The appendix reports the hyperparameters used for MACSAC, MADDPG, and CSAC across the evaluated test cases.

  • Hyperparameters: Table IV lists the algorithm hyperparameters for MACSAC, MADDPG, and CSAC.These settings are the parameters used in the paper’s experiments.
  • Scope: The appendix provides implementation settings rather than additional performance results.The supplied appendix passages identify parameter tables and case-dependent settings only.
  • Case-specific settings: When parameters differ among the 33-bus, 141-bus, and IEEE 37-bus cases, the case-specific values are listed separately.The passage indicates a set-based listing for differing parameters.
Loading 2006.12841v2…