PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks

Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, Tim C. Green

arXiv:2110.14300v5cs.LGcs.MA

TL;DR

High-PV distribution networks create a challenging active-voltage-control problem for MARL because many distributed devices must coordinate under partial information and voltage constraints. The paper formulates the task as a Dec-POMDP, releases an open-source environment, and evaluates MARL with varied barrier functions. It finds that barrier-function design affects performance, MARL may scale to larger networks at low control rates, and learned behaviours raise interpretability and robustness concerns.

  • Problem

    High PV penetration can cause voltage fluctuations beyond grid standards, motivating coordinated control of distributed apparatuses under limited local information.

  • Method

    The paper formulates active voltage control as a Dec-POMDP and provides an open-source Python environment with scenarios spanning 6 to 38 agents.

  • Results

    Voltage barrier-function design affects MARL performance, and MARL may scale with agent count and network complexity while requiring only a low control rate.

  • Takeaways & Limitations

    The environment supports research on distributed active voltage control, while barrier design and low control rates are important considerations for MARL evaluation.

  • Takeaways & Limitations

    Learned behaviours can improve or deteriorate performance and raise concerns about interpretability and robustness for industrial applications.

Abstract

from arXiv · show

This paper presents a problem in power networks that creates an exciting and yet challenging real-world scenario for application of multi-agent reinforcement learning (MARL). The emerging trend of decarbonisation is placing excessive stress on power distribution networks. Active voltage control is seen as a promising solution to relieve power congestion and improve voltage quality without extra hardware investment, taking advantage of the controllable apparatuses in the network, such as roof-top photovoltaics (PVs) and static var compensators (SVCs). These controllable apparatuses appear in a vast number and are distributed in a wide geographic area, making MARL a natural candidate. This paper formulates the active voltage control problem in the framework of Dec-POMDP and establishes an open-source environment. It aims to bridge the gap between the power community and the MARL community and be a drive force towards real-world applications of MARL algorithms. Finally, we analyse the special characteristics of the active voltage control problems that cause challenges (e.g. interpretability) for state-of-the-art MARL approaches, and summarise the potential directions.

1 Introduction

Active voltage control is a promising, relatively lower-risk real-world MARL problem, but existing approaches lack validated scalability, robustness, and a common benchmark. The paper addresses this gap with an open-source Dec-POMDP environment and broad algorithmic experiments.

  • Motivation: Increasing rooftop-PV penetration can cause voltage fluctuations beyond grid-standard thresholds through excessive active-power injection.PV inverter flexibility can help relieve these fluctuations.
  • Motivation: Active voltage control coordinates distributed PVs, SVCs, and OLTCs using limited local information to regulate network-wide voltage.Voltage at each node depends on other nodes’ real and reactive powers, with influence receding over distance.
  • Problem characteristics: Active voltage control combines local and global interactions, constrained optimisation, uncertain radial-network structure, and relatively tolerant failure consequences.These properties make it challenging for conventional methods while being less reliability-sensitive than some other power-network problems.
  • Research gap: Prior MARL studies showed promising case-specific performance, but scalability, robustness across penetration and load scenarios, and fair comparison remained unclear.The field lacked a commonly accepted benchmark.
  • Contributions: The paper introduces a Dec-POMDP formulation, an open-source environment, three real-data scenarios spanning 6 to 38 agents, and experiments with seven state-of-the-art MARL algorithms.It also studies voltage barrier functions and discusses challenges such as interpretability.

2 Related Work

Prior active-voltage-control research relies mainly on OPF, droop control, or early MARL applications, while available environments differ in scope and centralisation. The paper positions its environment as a decentralised distribution-network benchmark focused on active voltage control.

  • Traditional methods: Traditional strategies include reactive-power dispatch through OPF and droop control based on local voltage and power measurements.Centralised and distributed OPF address voltage constraints while minimising power loss, whereas droop control is decentralised.
  • Traditional methods: OPF requires an accurate system model, while distributed optimisation replaces the centralised solver with methods such as ADMM.The cited limitation concerns model accuracy.
  • MARL approaches: Existing MARL studies apply algorithms including MADDPG, MATD3, and MASAC with inverter or SVC reactive-power actions and related adaptations.Some approaches instead use reference-voltage actions or combine reactive-power control with active-power curtailment.
  • Concurrent environments: Compared with L2RPN and Gym-ANM, this environment targets decentralised or distributed active voltage control in power distribution networks with more complicated scenarios.L2RPN focuses on centralised transmission control, while Gym-ANM addresses centralised distribution-network management.

3 Background

The paper models a PV-rich distribution system as a radial graph with voltage, load, and generation variables, then frames control against voltage and power-loss objectives. It contrasts distributed droop control with model-based OPF and MARL.

  • Power distribution network: The distribution network is represented as a tree graph whose nodes are buses and whose edges are branches, with bus 0 connected to the main grid.The model covers medium- and low-voltage networks with high PV penetration.
  • Voltage deviation: PV generation can create reverse current flow that raises bus voltages beyond the nominal range.The stated safe range is 0.95–1.05 p.u. around a 1.0 p.u. reference.
  • Optimal power flow: OPF minimises total power loss subject to power-balance, reactive-power, and bus-voltage constraints, using full system information as a benchmark.Its performance depends on system-model accuracy.
  • Droop control: Droop control is fully decentralised and maps local voltage deviation to PV reactive-power generation, but ignores total voltage deviations and power loss.This creates a contrast with network-wide optimisation objectives.
  • MARL background: MARL extends reinforcement learning to multiple agents and is conventionally formulated as a Dec-POMDP for cooperative tasks.The formulation uses joint actions and observations across agents.

4 Distributed Active Voltage Control Problem

The paper formulates distributed active voltage control as a Dec-POMDP in which PV-inverter agents use regional observations and continuous reactive-power actions. Its reward penalises voltage violations and reactive-power generation through a selected barrier function.

  • Problem formulation: PV-inverter agents are assigned to network nodes and operate across disjoint regions managed by distribution-network owners.Regional structure defines how agents and observations are organised.
  • State and observations: The Dec-POMDP state contains load powers, PV active powers, preceding inverter reactive powers, and network voltages, while each agent observes measures from its region.The shared safety objective and partial information motivate the Dec-POMDP formulation.
  • Action space: Each agent uses a continuous action representing the ratio of maximum reactive power generated by its PV inverter.Positive actions inject reactive power and negative actions absorb it, subject to inverter capacity and operational limits.
  • Dynamics and observations: State transitions combine power-flow solutions after actions with stochastic load changes, while observations include isotropic Gaussian sensor-measurement noise.The control interval is normalised to Δt = 1 and power-flow evolution occurs over a much shorter interval τ.
  • Reward design: The reward combines a voltage barrier penalty with reactive-power generation loss, transforming the constrained objective through a Lagrangian multiplier.The barrier function is difficult to choose because it affects the reactive-power term.
  • Voltage barrier function: The proposed Bowl-shape barrier combines L1- and L2-shape baselines by using a steep gradient outside the safety range and a gentler gradient near the reference voltage.The design targets vref = 1 p.u. and the 0.95–1.05 p.u. safety range.

5 Experiments

Experiments evaluate seven MARL algorithms across three network scales, voltage-barrier designs, and real-data scenarios. Results show strong algorithm- and reward-dependence, with persistent gaps in scalability, robustness, interpretability, and performance relative to centralized control.

  • Experimental Settings: The test bench spans modified 33-bus, 141-bus, and 322-bus networks with 6, 22, and 38 agents, respectively.It uses real Portuguese electricity-consumption and Belgian solar data, interpolated at 3-minute resolution.
  • Experimental Settings: Performance is evaluated using controllable rate (CR), the fraction of time steps with all bus voltages controlled, and power loss (PL), averaged per time step.The objective is high CR and low PL.
  • Main Results: MADDPG and MATD3 generally perform well, whereas COMA and SQDDPG degrade as the number of agents increases; MAPPO and IPPO perform poorly on these power-network scenarios.COMA and SQDDPG also exhibit high power loss in some settings, while IDDPG remains intermediate.
  • Main Results: Bowl-shaped barriers preserve high CR with low PL on 33-bus and 141-bus networks, while L2-shaped barriers perform best on the 322-bus network.The reported preferred barriers are L1-shape for 33-bus, Bowl-shape for 141-bus, and L2-shape for 322-bus networks.
  • Main Results: Reward design strongly affects algorithm performance: the same algorithm can behave differently under reward functions with different voltage-barrier shapes.This dependence is shown using median CR and PL surfaces over algorithm and reward types.
  • Discussion: Across tested comparisons, MARL keeps voltages within the safety range, has lower power loss than droop control, and higher power loss than OPF; substantial room for improvement remains.The paper identifies remaining gaps in performance, robustness, and interpretability and calls for systematic barrier-function design and domain-knowledge integration.

6 Conclusion

The paper frames active voltage control as a real-world MARL challenge, studies it through a Dec-POMDP formulation, and releases an accessible environment for further research.

  • The paper investigates MARL for active voltage control in power distribution networks.
  • It formulates active voltage control as a Dec-POMDP and studies MARL behavior under different voltage barrier functions.
  • The study compares MARL behavior with droop control and OPF, finding that MARL can produce inexplicable behaviors.
  • The authors argue that interpretable and trustworthy MARL algorithms are highly desirable for industrial applications.
  • The environment is open-sourced and designed to support contributions from the machine learning community.

A.2 Voltage Deviation and Control

The section develops the voltage-control setting from a two-bus network, highlighting the tension between voltage safety and power-loss minimization and introducing conventional and learning-based control formulations.

  • Voltage Deviation and Control: A two-bus example relates voltage deviation to PV reactive power and illustrates how inverters participate in voltage control.
  • Traditional Voltage Control Methods: Traditional regulators may not respond effectively to intermittent solar-driven voltage fluctuations or increasing PV penetration.
  • Voltage Deviation and Control: Voltage control and minimum power loss conflict even in the two-bus network.
  • Voltage Deviation and Control: The two-bus demonstration is linear, whereas real power systems are nonlinear and more complex but exhibit the same contradiction.
  • Optimal Power Flow: The OPF objective is equivalent to minimizing overall power loss, but voltage constraints may require slack variables when high PV penetration makes the problem infeasible.
  • Traditional Voltage Control Methods: Droop control maps PV-bus voltage measurements to reactive power through a piecewise-linear function with saturation regions and a dead band.

C Experimental Settings

The experimental source code is intended to support straightforward reproduction and further study.

  • Experimental Settings: The authors state that the experimentation source code will be released for easy reproduction and further studies.

C.1 Algorithm Settings and Training Details

The experimental setup specifies algorithm, training, barrier-function, simulation, and network configurations for evaluating MARL on three distribution-network scenarios.

  • Algorithm Settings and Training Details: The experiments report settings for COMA, MATD3, SQDDPG, IPPO, and MAPPO, while IDDPG and MADDPG require no extra hyperparameters.
  • Algorithm Settings and Training Details: Training uses online updates, shared agent parameters, GRU-based recurrent policies, MLP critics, batch size 32, and a 5,000-entry off-policy replay buffer.
  • Algorithm Settings and Training Details: COMA and SQDDPG use sample size M = 10, while MATD3 clips exploration noise at c = 1.
  • Algorithm Settings and Training Details: IPPO and MAPPO use GAE with λ = 0.95, value-loss coefficient 2, clipping parameter 0.4, normalized advantages, and 10 update epochs.
  • Simulation and Barrier Functions: The environment simulates 480 time steps per day by combining PV and load profiles with power-flow-computed voltage status.
  • Simulation and Barrier Functions: Three voltage barrier functions—L1-shape, L2-shape, and Bowl-shape—are evaluated, with the Bowl-shape parameterized by a, b, c, and d.
  • Network Scenarios: The benchmark contains three distribution-network models with distinct topologies, parameters, load profiles, and PV profiles.

D.2 Data Descriptions.

The data descriptions assemble real-world load and PV profiles across three network scales, seasonal periods, and geographically distinct regions. The figures show aggregate and bus-level power behavior alongside power factors for the test systems.

  • Load Profiles: Load profiles use Portuguese electricity-consumption data from 232 consumers over three years at 15-minute resolution.The source dataset covered residential and industrial clients from 2011 to 2014; locations with more than 20 missing data points were removed.
  • PV Profiles: PV profiles represent ten regions in Belgium, the Netherlands, and Luxembourg with distinct zonal solar-radiation levels.The PV data were collected from Elia, a Belgian power-network operator.
  • PV Profiles: PVs within the same control region share profiles, and Gaussian noise is added to PV active power in real time.This produces geographically correlated profiles while introducing real-time variation.
  • Network-Scale Profiles: Figures 13–18 summarize total and bus-level PV generation and load consumption for 33-, 141-, and 322-bus networks in January and July.The bus-level figures compare selected buses across winter and summer months, while the total-power figures cover representative days and months.
  • Power Factors: Figure 19 reports power factors for the three systems, with values above 0.9 typically representing residential consumers and below 0.5 potentially representing industrial consumers.The passage presents these thresholds as typical interpretations rather than strict classifications.

E.1 Extra Results during Training

The extra training analysis adds voltage-violation and reactive-power metrics to control rate and power loss. Results compare reward functions built from different voltage barrier functions.

  • Metrics: Two additional training metrics are introduced: voltage out of control ratio and Q loss.They complement the previously introduced control rate and power loss metrics.
  • Metrics: Voltage out of control ratio averages the per-time-step fraction of voltages outside the 0.95–1.05 p.u. safety range.The metric evaluates voltage violations over an episode.
  • Metrics: Q loss averages agents’ mean reactive-power generation per time step during an episode.It measures reactive-power use rather than voltage deviation directly.
  • Evaluation: Training results are reported as median metrics across five random seeds, with each test using ten randomly selected episodes.This evaluation procedure is applied to the reported performance measures.
  • Barrier-Function Comparisons: Figures 20 and 21 compare voltage out-of-control ratios and reactive-power losses across reward functions with distinct voltage barrier functions.Sub-captions identify the barrier scenario, and BL abbreviates Bowl.

E.2 Extra Results for Case Studies

The case studies compare MARL with OPF and droop control on representative buses and across all buses in 141- and 322-bus networks. Performance varies by season, network size, and PV penetration.

  • Experimental Setup: MATD3 with a Bowl-shaped voltage barrier represents MARL for the 141-bus case, while MADDPG with an L2-shaped barrier represents it for the 322-bus case.These trained agents are compared with traditional OPF and droop-control methods.
  • 141-Bus Network: In the 141-bus summer case, MARL fails to keep voltage within the safety range from 20:00 to 22:00, although its power loss is lower than droop control.Most of the time, all methods maintain the voltage within the safety range.
  • 322-Bus Network: In the 322-bus summer case, only droop control maintains voltage within the safety range when PV active power is extremely high.MARL and OPF fail during 10:00–14:00, and the text attributes OPF’s poor performance possibly to computational difficulty in the large network.
  • All-Bus Analysis: Across all buses, traditional methods outperform MARL in summer for the 33- and 141-bus networks, while droop control is far better than OPF and MARL for the 322-bus network.In winter, all methods control all buses within the safety range; the text links some summer MARL failures to relative overgeneralisation under frequent high-PV situations.

E.3 Extra Results during Testing

The testing analysis defines voltage-control and deviation metrics, then reports final-model evaluations and visual comparisons across network sizes. The figures and tables organize performance by methods, seasons, buses, and network scale.

  • Testing Procedure: Final MARL models are tested on ten randomly selected initial states, with means and standard deviations reported for continuous metrics.Ratio metrics are reported by their means because they do not satisfy the stated normality hypothesis.
  • Testing Metrics: The reported metrics include voltage-out-of-control ratios, threshold-specific voltage violations, control rate, voltage deviation, maximum rise or drop deviation, and power loss.These metrics quantify both safety compliance and control-side effects.
  • Case-Study Comparisons: Figure 23 compares MARL with traditional methods on a typical 322-bus-network bus for summer and winter, including voltage, PV active power, and reactive power.The figure distinguishes uncontrolled voltage and the safety range, and labels active and reactive power as P and Q.
  • All-Bus Results: Figures 24–26 show all-bus voltage trajectories against the safety boundary for the 33-, 141-, and 322-bus networks.Green lines represent bus-voltage variation, red dashed lines represent the safety boundary, and captions identify method-season combinations.
  • Tabulated Results: Tables 3–5 report mean test results for the 33-, 141-, and 322-bus networks, using ten episodes for MARL and 100 episodes for traditional control methods.The table captions state that results are recorded as mean ± standard deviation.
Loading 2110.14300v5…