Source-linked AI summary
Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks
Yuanqi Gao, Wei Wang, Nanpeng Yu
TL;DR
VVC needs to reduce losses and improve voltage profiles without depending on inaccurate network models or vulnerable centralized control. The paper formulates VVC as a networked multi-agent MDP and solves it with maximum-entropy RL and communication-efficient consensus. Across IEEE test feeders, the proposed method matches the single-agent RL benchmark while demonstrating communication efficiency and resilience.
Problem
VVC must reduce network losses and improve voltage profiles despite inaccurate or unavailable network models and vulnerability to centralized-controller failure.
Method
The paper extends off-policy maximum-entropy RL to a networked multi-agent MDP and uses consensus so agents learn distributed policies from local rewards.
Results
The proposed C-MARL algorithm matches the centralized or single-agent RL benchmark and is more communication-efficient than ADMM-based consensus.
Takeaways & Limitations
The resulting decentralized, fully data-driven controller supports VVC without a central controller or network-topology and parameter information, while retaining resilience to agent and communication-link failures.
Abstract
from arXiv · showhide
Volt-VAR control (VVC) is a critical application in active distribution network management system to reduce network losses and improve voltage profile. To remove dependency on inaccurate and incomplete network models and enhance resiliency against communication or controller failure, we propose consensus multi-agent deep reinforcement learning algorithm to solve the VVC problem. The VVC problem is formulated as a networked multi-agent Markov decision process, which is solved using the maximum entropy reinforcement learning framework and a novel communication-efficient consensus strategy. The proposed algorithm allows individual agents to learn a group control policy using local rewards. Numerical studies on IEEE distribution test feeders show that our proposed algorithm matches the performance of single-agent reinforcement learning benchmark. In addition, the proposed algorithm is shown to be communication efficient and resilient.
I. INTRODUCTION
VVC must manage increasingly complex active distribution networks while traditional model-based approaches depend on accurate network models and centralized coordination. The paper proposes a consensus multi-agent RL framework that learns distributed control policies and reports benchmark-level performance, communication efficiency, and resilience.
- Motivation: Rapid distributed energy resource growth makes voltage-profile management increasingly difficult for active distribution networks.
- Limitations of existing methods: Model-based VVC methods require complete and accurate network models, while centralized data-driven methods remain vulnerable to central-controller failure.
- Limitations of existing methods: Existing data-driven decentralized VVC methods either cannot handle large state spaces or do not coordinate individual agents.
- Proposed approach: The proposed C-MARL algorithm learns distributed Volt-VAR control policies from historical operational data without relying on accurate network models.
- Proposed approach: C-MARL uses networked agents with local policies and value functions trained through local rewards and consensus over neighboring agents’ information.
- Results: The algorithm matches the single-agent RL benchmark in operational costs, reduces consensus communication requirements, and remains resilient to controller and communication-link failures.
B. Formulate VVC as a MAMDP
The paper formulates distributed VVC as a networked multi-agent problem in which device-specific agents act using local measurements and rewards while optimizing long-term operational costs and voltage violations.
- The framework models a radial distribution network with a substation reference node and nodal voltage, real-power, and reactive-power states.
- Each VVC device is assigned an agent that selects its local control action, including discrete tap positions for voltage regulators and switching states for capacitor banks.
- Local rewards combine metered branch power-loss costs, device-switching costs, and penalties for voltage-constraint violations.
- The global state contains network power injections, previous VVC-device statuses, and the discrete time step, while the discount factor is less than one.
- Agents minimize long-term discounted operating cost and constraint violations because switching costs couple decisions across adjacent time steps.
III. TECHNICAL METHODS
The proposed algorithm is developed in three stages: centralized off-policy maximum-entropy reinforcement learning, distributed multi-agent reformulation, and communication-efficient consensus learning for VVC.
- The method first reviews centralized off-policy maximum-entropy reinforcement learning before reformulating it for distributed agents.
- The final stage derives a communication-efficient consensus multi-agent algorithm to solve the VVC problem.
A. Off-policy Maximum Entropy RL
The method uses maximum-entropy reinforcement learning to combine return maximization with policy entropy, supporting exploration and off-policy learning when the physical network model is inaccurate or unavailable.
- Maximum-entropy reinforcement learning maximizes both expected return and policy entropy.
- The off-policy formulation enables learning from past experiences and can therefore use fewer samples collected from the distribution grid.
- Entropy regularization balances exploration and exploitation by maintaining stochasticity in the policy.
- The optimality equation is difficult to solve directly because the VVC problem has high-dimensional, continuous state spaces and is initially expressed centrally.
B. Distributed Optimization
The paper reformulates local policy and value-function learning as a distributed consensus optimization problem, with agents using local rewards and neighbor communication to align their function approximators efficiently.
- The distributed objective constrains local decision variables to agree while minimizing agent-specific objective functions.
- Each agent maintains local copies of the global value and policy functions, which must reach consensus at optimality.
- Each agent receives a local reward and takes a local action, with consensus achieved through neighbor-to-neighbor communication.
- Deep neural networks parameterize the local value and policy functions, with a separate target network used to evaluate successor states.
- Communicating complete neural-network parameter sets is costly, so the proposed stochastic-approximation approach retains the consensus constraints while improving communication efficiency.
C. Communication-Efficient Multi-Agent Policy Consensus
The method approximates consensus constraints stochastically and updates local policy and value models through neighbor communication using sampled data. Agents alternate learning, consensus updates, and control actions while communicating only selected information.
- Stochastic consensus approximation: The approach randomizes enforcement of the infinite consensus constraints to obtain a stochastic programming representation.The approximation converts the original constraint set into a form handled through sampled data.
- Theoretical guarantees: Under continuity and full-support assumptions, the stochastic representation is equivalent to the original constraints.Without continuity, feasible points satisfy most original constraints except a measure-zero subset.
- Theoretical guarantees: In practice, the state-action distribution is approximated using an experience replay dataset.The paper explicitly notes that the theoretical measure is replaced by the data distribution in implementation.
- Optimization: A quadratic non-consensus penalty yields a tractable sample gradient that agents optimize with established deep-learning routines.The implementation uses fixed multiplier parameters with empirically good performance.
- Algorithm procedure: Each iteration samples an agent and minibatch, updates its parameters, exchanges neighbor evaluations, and applies another consensus update.The transmitted information includes sampled state, action, and neighboring network evaluations.
- Algorithm procedure: Agents communicate and update C times on average between adjacent control steps, then jointly actuate VVC devices and store transitions.The stored transitions support subsequent experience-replay updates.
D. Algorithm Implementation
The implementation uses multilayer perceptrons for value estimation and a device-decoupled policy network with ordinal tap-position encoding. Hour-of-week information is encoded periodically.
- Network architecture: Value networks take the global state as input and output its estimated value.They use standard multilayer perceptron architectures.
- Network architecture: The policy network separates outputs into device-specific groups corresponding to each local action space.Shared hidden layers support all groups, while ordinal encoding represents tap-position order.
- Time encoding: Hour-of-week is encoded with cosine and sine coordinates over a 168-hour period.This representation captures the periodic nature of the global time step.
IV. NUMERICAL STUDIES
The numerical studies evaluate the proposed C-MARL algorithm on three distribution test feeders, with validation focused on sample efficiency, communication efficiency, and resiliency.
- Experimental scope: Three test feeders are used for the numerical studies.The experimental setup is provided separately for the three feeders.
- Evaluation criteria: The evaluation validates sample efficiency, communication efficiency, and resiliency of the proposed algorithm.These properties are examined in the numerical-results section.
A. Numerical Setup
The study uses IEEE feeders, smart-meter-derived load data, fixed communication networks, and tuned algorithm benchmarks. Results compare reward, constraint violations, and communication behavior across test cases.
- Test systems: The experiments use IEEE 4-bus, 34-bus, and 123-bus distribution feeders with voltage-regulation devices.Voltage regulators have 21 tap positions spanning ratios from 0.95 to 1.05.
- Load data: Hourly loads are derived from one year of half-hourly measurements from approximately 5,000 London smart-meter customers.The measurements are aggregated and scaled to the feeders while preserving standard-case spatial distributions and power factors.
- Reward and constraints: The experiments use specified electricity, switching, and constraint-violation costs together with voltage bounds of 0.95–1.05 p.u.These parameters define the local reward and operational-violation settings.
- Benchmarks: The proposed method is compared with single-agent SAC and multi-agent off-policy reinforcement learning using linearized ADMM consensus.SAC serves as a stability baseline, while ADMM provides a consensus comparison.
- Benchmark setup: For single-agent SAC, the reward averages local rewards and the action combines local actions.The ADMM implementation maintains local neural-network parameters and dual variables.
- Benchmark setup: Algorithm hyperparameters are tuned individually, with shared settings used across numerical studies unless otherwise specified.The three network-specific parameter values are ordered 4-bus, 34-bus, and 123-bus.
- Results: All algorithms stabilize after training, while C-MARL matches the single-agent benchmark on hourly reward and constraint violation.C-MARL also improves communication efficiency over ADMM because it transmits sample data rather than full network weights.
D. Resiliency against Agent and Communication Link Failure
The proposed algorithm is evaluated under random agent and communication-link failures, with replacement states used when disconnection makes the network partially observable. Simulations show negligible degradation when failures are cleared promptly, especially over the long term.
- Failure scenarios: The resiliency study evaluates internal agent errors and temporary communication-link failures as two component-failure scenarios.An agent freezes its device and stops training while continuing communication; disconnected links trigger replacement states based partly on historical averages.
- Agent failure: When an agent fails internally, it freezes its device and stops training while the other agents continue control and training.
- Communication-link failure: When a communication failure disconnects the graph, agents construct replacement states and act without storing those experiences in replay memory.If the graph remains connected, agents operate normally with the altered connectivity.
- Failure model: Random failures are modeled with Poisson occurrences and geometrically distributed durations, with equal failure probability across agents in the first experiment.The occurrence and duration assumptions define the simulated failure process.
- Results: The proposed algorithm remains resilient to random agent and communication-link failures, with negligible degradation when failures are cleared quickly.Long-term performance is affected much less than short-term performance.
- Conclusion: The conclusion reports similar performance to the centralized reinforcement-learning benchmark and resilience to communication-link and agent failures.
APPENDIX PROOF OF PROPOSITIONS
The appendix proves propositions by identifying abstract metric-space sets with neural-network constraint sets and showing their equality under continuity and measure-support assumptions. A second lemma extends the equality argument when continuity and full support are absent.
- Lemma 1: Lemma 1 establishes equality of two sets for a nonnegative function continuous in one argument under a full-support measure.The proof uses continuity to extend positivity from one point to a neighborhood, whose measure is positive under full support.
- Notation: The appendix notes that equalities and inequalities are interpreted elementwise.
- Proposition 1: The proof of Proposition 1 maps neural-network weights to the metric space X and state-action pairs to Y, applying Lemma 1 to the constraint function.
- Lemma 2: Lemma 2 removes continuity and full-support assumptions and proves set equality by showing the positive-value set has measure zero.Its argument establishes both set inclusions through the sets N1(x) and N2(x).
- Proposition 2: The proof of Proposition 2 selects a full-support state-action measure, covering almost all state-action pairs.