Source-linked AI summary
Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems
Ying Zhang, Xinan Wang, Jianhui Wang, Yingchen Zhang
TL;DR
VVO in unbalanced distribution systems must regulate voltage and reduce losses under changing operating conditions without relying on a specific optimization model. The paper proposes a model-free MADRL/DQN framework with coordinated control of capacitors, regulators, and smart inverters, using forward-backward-sweep power flow as its environment. Tests on IEEE 13-bus and 123-bus systems report high voltage-regulation success and substantial power-loss reduction.
Problem
Model-based VVO depends on specific optimization formulations, while large joint action spaces make single-agent DQN control computationally challenging for scalable, time-varying operation.
Method
A model-free MADRL framework assigns capacitor, AVR, and smart-inverter controls to multiple DQN agents interacting with a forward-backward-sweep distribution-system environment.
Results
The method achieves voltage-regulation success rates of 99.80% on the 13-bus system and up to 99.975% on the 123-bus system, while reducing average losses by 14.78% and 36.09%, respectively.
Takeaways & Limitations
The proposed DQN-based method realizes voltage regulation and power-loss reduction simultaneously in the tested unbalanced distribution systems.
Abstract
from arXiv · showhide
This paper develops a model-free volt-VAR optimization (VVO) algorithm via multi-agent deep reinforcement learning (MADRL) in unbalanced distribution systems. This method is novel since we cast the VVO problem in unbalanced distribution networks to an intelligent deep Q-network (DQN) framework, which avoids solving a specific optimization model directly when facing time-varying operating conditions of the systems. We consider statuses/ratios of switchable capacitors, voltage regulators, and smart inverters installed at distributed generators as the action variables of the DQN agents. A delicately designed reward function guides these agents to interact with the distribution system, in the direction of reinforcing voltage regulation and power loss reduction simultaneously. The forward-backward sweep method for radial three-phase distribution systems provides accurate power flow results within a few iterations to the DQN environment. Finally, the proposed multi-objective MADRL method realizes the dual goals for VVO. We test this algorithm on the unbalanced IEEE 13-bus and 123-bus systems. Numerical simulations validate the excellent performance of this method in voltage regulation and power loss reduction.
I. INTRODUCTION
The paper motivates a model-free MADRL approach to VVO because conventional model-based methods are computationally burdensome, approximation-dependent, and poorly adaptive to time-varying conditions. It combines multi-agent DQN control with accurate forward-backward-sweep power flow for unbalanced distribution systems.
- VVO limitations: Traditional VVO is formulated as a nonconvex, NP-hard MINLP problem, while heuristic solutions do not guarantee optimality.The formulation includes integer variables and nonlinear voltage-dependent load models.
- VVO limitations: Decentralized and hierarchical methods can require thousands of iterations and hundreds of seconds, while power-flow approximations may reduce accuracy and produce suboptimal controls.These methods also rely on specific optimization models and have limited ability to adapt rapidly to time-varying loads.
- Proposed approach: The proposed data-driven, model-free MADRL method casts multi-objective VVO as a DQN problem for auto-adaptive voltage control under time-varying operating conditions.Agents interact with the distribution system environment to determine actions.
- Proposed approach: DQN agents control capacitor statuses or ratios, AVR settings, and smart-inverter actions to pursue voltage regulation and power-loss reduction simultaneously.A customized reward function guides the agents toward both objectives.
- System modeling: The method integrates voltage-dependent ZIP loads, distributed-generation penetration, and multiple regulating devices into forward-backward-sweep power flow for unbalanced systems.The paper describes this power-flow method as efficient and accurate without approximation for radial distribution systems.
- Multi-agent design: Multiple DQN agents share observations to distribute global control variables and improve computational efficiency and scalability in larger three-phase systems.The multi-agent design addresses the explosively growing joint action space faced by a single agent.
III. PROPOSED VVO ALGORITHM
The section integrates models of switchable capacitor banks, AVRs, and smart inverters into power-flow calculations, using their status changes as control actions in a multi-agent DQN-based VVO method.
- Switchable capacitor banks, AVRs, and smart inverters are integrated into power-flow calculations to evaluate how regulating-device status changes affect distribution-system operation.
- The proposed method uses the status changes of regulating devices as control actions for a multi-agent DQN-based VVO algorithm.
- The method targets effective voltage regulation and power loss reduction.
A. Voltage Regulation Devices
The system model represents capacitor banks, voltage regulators, and smart inverters in three-phase power-flow calculations, with discretized control actions for reinforcement learning.
- Capacitor Bank: Capacitor banks provide phase-specific reactive power according to their connection status and bus-voltage-dependent susceptance.The capacitor status is binary, and connected capacitors provide reactive power.
- Voltage Regulator: Voltage regulators use 32 steps across a ±10% regulating range, with step values from 0.9 to 1.1 p.u.Their series and shunt impedances are neglected because they are regarded as extremely small.
- Voltage Regulator: Regulator installation is modeled by introducing an additional bus and replacing the original voltage and current variables during forward-backward sweeps.The regulator transforms the voltage and current quantities used in the power-flow calculation.
- Smart Inverters: Smart inverters at DG units provide or absorb per-phase reactive power subject to their apparent-power and active-power limits.The reactive-power range is relatively narrow because high power factor, such as 0.95, is preferable during DG operation.
- Smart Inverters: Smart-inverter actions are discretized for reinforcement learning, while their control values range between −1 and 1 at a 0.1 step.This discretization supports flexible Q-learning or DQN applications under assumed interval-specific operator strategies.
- Integrated Model: The updated power-flow process integrates device models and voltage-dependent loads, providing the DQN environment under time-varying operating conditions.The integrated calculations evaluate the effects of capacitor, inverter, and AVR setting changes.
B. Multi-agent DQN-based Method
The multi-agent DQN distributes VVO control across agents that share observations and interact with a power-flow environment, using rewards that combine voltage compliance with loss reduction.
- Motivation: A multi-agent DQN addresses the explosively growing joint action space caused by many controllable three-phase devices.Agents coordinate through shared observations and jointly selected actions to improve computational efficiency and scalability.
- State and Action: Each agent contributes to a global action vector containing capacitor, regulator, or smart-inverter control actions.The action vector is formed by concatenating the action selected by every agent.
- State and Action: The environment returns three-phase bus voltages as states, while the VVO objective keeps voltages within 0.95 to 1.05 p.u. and minimizes active power loss.The state contains voltage vectors for all buses in the distribution system.
- Reward: The reward interpreter embeds voltage constraints into the objective, penalizing violations and rewarding reductions in active power loss.A significant penalty is imposed when voltage constraints are violated, and greater violations produce more negative rewards.
- Training and Testing: During training, agents apply ε-greedy exploration, store transitions in replay memory, and update their Q functions using sampled experience.After training, the learned policies produce actions for new operating conditions, which are combined and sent to the environment for online VVO control.
IV. CASE STUDY
The proposed algorithm is tested on unbalanced IEEE 13-bus and 123-bus distribution systems, including a modified 13-bus system with added photovoltaic generation.
- Test Systems: The experiments use the three-phase unbalanced IEEE 13-bus and 123-bus distribution systems.The 13-bus system is modified by adding single-phase and three-phase PV units at specified buses.
- Test Systems: The modified 13-bus case includes two single-phase PV units at buses 675 and 684 and one three-phase PV unit at bus 680.
A, B, C
The IEEE test systems include phase-specific regulators, capacitors, and distributed-generator smart inverters, creating a high-dimensional control space for training and testing.
- The control devices include phase-specific regulators and capacitors together with smart inverters at distributed generators.
- Considering action variables for all controllable devices produces a high-dimensional joint decision space.
- The study randomly generates 9,000 operating conditions for offline training and 13,000 for online testing.
- The DQN agents use three fully connected layers, ReLU hidden-layer activations, sigmoid output activations, and a learning rate of 0.0001.
A. Learning Performance
On the IEEE 123-bus system, multi-agent DQN training increasingly produces positive rewards, and trained agents maintain effective VVO behavior on new operating conditions.
- Early training produces negative rewards because limited positive experiences and unoptimized policies fail to maintain voltages within 0.95~1.05 p.u. consistently.
- Positive rewards indicate no voltage violation and further power-loss reduction after the agents take an action.
- The average training reward continuously increases, indicating improved realization of multi-objective VVO.
- 4,000 new online-test cases receive positive rewards, and the trained agents implement effective control policies under new operating conditions.
B. VVO Performance
The proposed model-free MADRL method evaluates online VVO under random operating conditions using voltage-regulation success rates and controlled three-phase voltage profiles.
- Voltage-regulation success rate S_v is defined as the fraction of initially violating test cases that avoid voltage violations after control.
- 99.80% success is achieved on the IEEE 13-bus system, while the IEEE 123-bus system reaches 99.975%.
- In a selected 13-bus case, uncontrolled A- and C-phase voltages violate limits at buses 611, 652, 671, 675, and 684, whereas controlled voltages enter the normal range.
- On the 123-bus system, the method avoids voltage violations by jointly dispatching regulators, capacitors, and smart inverters.
2) Power Loss Reduction
The proposed control reduces active power losses across the evaluated test cases while coordinating voltage-regulation devices under stated baseline settings.
- The loss-reduction calculation compares active power losses with and without control for each test case.
- Before control, no capacitor bank is connected, AVR steps are set to 1, and smart-inverter power factors use the default value 1.
- 34.12 kW, or 14.78%, is the average power-loss reduction in the 13-bus system.
- 109.09 kW, or 36.09%, is the average power-loss reduction in the 123-bus system.
- The proposed method enables power-loss reduction in all test cases and realizes power-loss reduction and voltage regulation simultaneously.
C. Computation Time
The proposed MADRL method is evaluated for computational efficiency and power-loss reduction in unbalanced 13-bus and 123-bus systems. Its online execution times are reported as promising for real-time implementation, while the method remains competitive in high-dimensional action spaces.
- C. Computation Time: 21.7 and 39.2 milliseconds are the average online execution times for the 13-bus and 123-bus systems, respectively.These averages are reported across all test cases and are described as promising for real-time implementation.
- C. Computation Time: The proposed MADRL method retains high computational efficiency in the unbalanced 123-bus system.
- C. Computation Time: The method is competitive when handling the high-dimensional action space of three-phase distribution systems.
V. CONCLUSION
The paper proposes a real-time, DQN-based VVO algorithm for unbalanced distribution systems. By combining efficient power flow calculation with multi-agent interaction, it adaptively selects control actions for voltage regulation and power loss reduction, with promising performance demonstrated on two systems.
- V. CONCLUSION: The paper proposes a novel and real-time DQN-based VVO algorithm for unbalanced distribution systems.
- V. CONCLUSION: The method integrates voltage-dependent loads, DG penetration, and three voltage-regulating device types into the DQN environment.
- V. CONCLUSION: Multiple agents interact with the distribution-system environment to adaptively choose actions that enable voltage regulation and power loss reduction.
- V. CONCLUSION: The algorithm achieves promising VVO performance in two unbalanced distribution systems.