Source-linked AI summary
Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems
Ying Zhang, Xinan Wang, Jianhui Wang, Yingchen Zhang
TL;DR
VVO in unbalanced distribution systems is studied under changing operating conditions, where a single-agent DQN faces an extremely high-dimensional joint action space. This paper applies multi-agent deep Q-learning to coordinate device controls and reports voltage regulation and power-loss reduction on the IEEE 13-bus and 123-bus systems.
Problem
A single-agent DQN faces an extremely high-dimensional joint action space for VVO in distribution systems with changing operating conditions.
Source: §I · p. 2, B. Multi-agent DQN-based Method · p. 4Method
The paper applies model-free multi-agent deep Q-learning to VVO, using capacitor, regulator, and smart-inverter settings as agent actions.
Source: §I · p. 2Results
99.975% success rate in voltage regulation was reached in the 123-bus test, and the method reduced power losses across both tested systems.
Source: B. VVO Performance · p. 7, 2) Power Loss Reduction · p. 7Takeaways & Limitations
The findings support simultaneous voltage regulation and power-loss reduction as VVO objectives in the tested unbalanced distribution systems.
Source: 2) Power Loss Reduction · p. 7, §V · p. 8
Abstract
from arXiv · showhide
This paper develops a model-free volt-VAR optimization (VVO) algorithm via multi-agent deep reinforcement learning (MADRL) in unbalanced distribution systems. This method is novel since we cast the VVO problem in unbalanced distribution networks to an intelligent deep Q-network (DQN) framework, which avoids solving a specific optimization model directly when facing time-varying operating conditions of the systems. We consider statuses/ratios of switchable capacitors, voltage regulators, and smart inverters installed at distributed generators as the action variables of the DQN agents. A delicately designed reward function guides these agents to interact with the distribution system, in the direction of reinforcing voltage regulation and power loss reduction simultaneously. The forward-backward sweep method for radial three-phase distribution systems provides accurate power flow results within a few iterations to the DQN environment. Finally, the proposed multi-objective MADRL method realizes the dual goals for VVO. We test this algorithm on the unbalanced IEEE 13-bus and 123-bus systems. Numerical simulations validate the excellent performance of this method in voltage regulation and power loss reduction.
I. INTRODUCTION
The introduction motivates model-free multi-agent deep reinforcement learning for coordinated volt-VAR optimization in unbalanced distribution systems, targeting voltage regulation and loss reduction under time-varying conditions. It presents a DQN framework that coordinates capacitors, voltage regulators, and smart inverters using accurate power-flow calculations and a tailored reward.
I. INTRODUCTION
- VVO aims to keep bus voltages within a normal range while reducing losses, and DG-driven bidirectional power flow increases voltage-violation risk; smart inverters can provide or absorb reactive power.The introduction gives 0.95–1.05 p.u. as an example operating range under ANSI C84.1.Source: §I · p. 1
- Prior VVO formulations are nonconvex and NP-hard; centralized device coordination creates large search spaces and slow computation, while approximations can reduce accuracy and adapt poorly to changing loads,.The cited decentralized method required up to thousands of iterations and hundreds of seconds.Source: §I · pp. 1–2
- Existing RL voltage-control methods focus on voltage profiles, use single agents that learn more slowly at scale, and omit voltage-dependent loads, smart inverters, and coordinated control across unbalanced three-phase systems –[21].Deep RL is presented as promising for nonlinear, high-dimensional search, but these gaps leave further work needed for distribution-system VVO.Source: §I · p. 2
- The proposed MADRL method casts multi-objective VVO as a DQN problem, coordinating capacitor, AVR, and smart-inverter actions with backward-forward sweep power flow and a reward for voltage regulation and loss reduction.The approach targets auto-adaptive control under time-varying operating conditions.Source: §I · p. 2
- The method integrates regulating devices and load models into forward-backward sweep power flow, which the paper describes as highly efficient for unbalanced systems compared with DC and linearized AC methods.The sweep method supplies accurate power-flow results in a few iterations.Source: §I · p. 2
- Unlike single-objective RL approaches,, the proposed MADRL method incorporates ZIP loads and smart inverters for multi-objective VVO within milliseconds, supporting online implementation.The introduction identifies voltage regulation and power-loss reduction as the simultaneous objectives.Source: §I · p. 2
- For scalability in larger three-phase systems, the method distributes global control variables among multiple DQN agents and uses observation sharing.This design is intended to address the scalability issue.Source: §I · p. 2
II. SYSTEM MODEL IN UNBALANCED DISTRIBUTION NETWORKS
The system model uses voltage-dependent ZIP loads and a forward-backward sweep (FBS) power-flow method for radial, unbalanced distribution networks. FBS supports accurate power-flow calculation, while the resulting nodal voltages are used to calculate total active-power loss.
System model
- Radial distribution networks with high line r/x ratios can make Newton-Raphson fail to converge, so the model uses FBS for accurate results without approximation.The three-phase line model represents each branch with impedance and shunt-capacitance matrices.Source: §II · p. 2
- The model adopts voltage-dependent ZIP loads for active and reactive power at each phase, a formulation widely used in related research,, and.The phase-specific ZIP coefficients for active and reactive power each sum to one.Source: §II · p. 2
- FBS initializes three-phase bus voltages at nominal values, calculates current injections using ZIP-modeled load consumption or DG production, then sweeps backward and forward through the feeder.The backward sweep starts at end buses and uses Kirchhoff's current law; the forward sweep starts at the root and calculates downstream nodal voltages.Source: §II · pp. 2–3
- Total active-power loss is calculated from nodal voltages across the system, using branch voltage drops and currents through line impedances.The branch voltage drop is the difference between the voltages at its two buses.Source: §II · p. 3
III. PROPOSED VVO ALGORITHM
The proposed multi-agent DQN-based VVO method uses regulating-device status changes as control actions to achieve voltage regulation and power-loss reduction.
III. PROPOSED VVO ALGORITHM
- The algorithm integrates models of switchable capacitor banks, AVRs, and smart inverters into power-flow calculations to evaluate how device-status changes affect distribution-system operation.These changes serve as control actions for the multi-agent DQN-based VVO method.Source: §III · p. 3
A. Voltage Regulation Devices
The section models capacitor banks, voltage regulators, and DG smart inverters as controllable devices, then integrates their settings into the power-flow environment used by DQN agents.
A. Voltage Regulation Devices
- Capacitor banks provide phase-specific reactive power when switched on, modeled as a nonlinear function of phase voltage and capacitor susceptance.The control variable is binary, indicating whether each phase’s capacitor is on or off.Source: A. Voltage Regulation Devices · p. 3
- Voltage regulators use 32 steps across a ±10% range, with 0.00625 p.u. increments; their small impedances are neglected.For power flow, an intermediate bus is inserted at each regulator location, and the regulator-modified voltages and currents are used in the forward and backward sweeps.Source: A. Voltage Regulation Devices · pp. 3–4
- DG smart inverters follow per-phase reactive power control, with their action space discretized for the DQN framework.Reactive output is bounded by inverter capacity and active output; the passage notes a preference for high power factor (e.g., 0.95) and an action range from −1 to 1 in 0.1 steps.Source: A. Voltage Regulation Devices · p. 4
- The power-flow model integrates capacitor, inverter, and regulator settings with voltage-dependent loads, serving as the DQN agents’ environment under time-varying operating conditions.The updated process combines device models (9)–(13) with the load models (1) and (2).Source: A. Voltage Regulation Devices · p. 4
B. Multi-agent DQN-based Method
The method uses multiple DQN agents to coordinate VVO actions, addressing the scaling challenge of a single agent’s high-dimensional joint action space. Agents observe three-phase bus voltages and use rewards that promote voltage compliance and lower active power loss.
Multi-agent DQN-based Method
- Multiple DQN agents distribute control actions to improve computational efficiency and scalability as the number of controllable devices grows.The paper identifies the single-agent joint action space as extremely high-dimensional [21].Source: B. Multi-agent DQN-based Method · p. 4
State and Action
- Each agent selects actions for regulating devices, and their joint action determines the system control while three-phase bus voltages form the shared state.Actions originate from smart inverters, regulators, or capacitors; agents coordinate through shared state observations and exchange their current actions.Source: B. Multi-agent DQN-based Method · pp. 4–5
Reward
- The reward favors active-power-loss reductions when voltages remain within limits and imposes a significant penalty for voltage violations.The VVO objective is to keep nodal voltages in a normal range, such as 0.95 to 1.05 p.u., while concurrently minimizing total active power loss,; greater voltage deviation receives a more negative reward.Source: B. Multi-agent DQN-based Method · pp. 4–5
Offline Training and Online Test
- During offline training, agents apply joint actions to the power-flow environment, receive rewards and new states, and update their Q functions; trained policies then control VVO online.Training uses epsilon-greedy action selection and experience replay of transitions to improve agent generality across diverse states.Source: B. Multi-agent DQN-based Method · p. 5
IV. CASE STUDY
The case study evaluates the proposed algorithm on unbalanced IEEE 13-bus and 123-bus systems using randomized operating conditions and a specified DQN setup.
IV. CASE STUDY
- The tests use three-phase unbalanced IEEE 13-bus and 123-bus distribution systems, with added PV units in the modified 13-bus system.The 13-bus system adds single-phase PV units at buses 675 and 684 and a three-phase unit at bus 680.Source: §IV · p. 5
- 9,000 and 13,000 operating-condition episodes are generated for offline training and online testing, respectively, with base-load fluctuations of 80% to 120%.The test phase uses new operating conditions as test cases.Source: §IV · p. 6
- Considering control actions for the listed devices creates a high-dimensional joint decision space.The control-device specifications include smart inverters installed at DG units in both test systems.Source: §IV · p. 6
- The DQN uses three fully connected NN layers, a learning rate of 0.0001, ReLU hidden-layer activations, and sigmoid output-layer activations.Table II also summarizes DQN hyperparameters for the two test systems.Source: §IV · p. 6
A. Learning Performance
On the IEEE 123-bus system, DQN agents’ average training reward increases, and trained agents earn positive rewards in 4,000 new online test cases.
A. Learning Performance
- Early in training, agents received negative rewards because limited positive learning experiences and unoptimized policies failed to keep voltages within 0.95~1.05 p.u. at all times.As training progressed, positive rewards occurred more frequently.Source: A. Learning Performance · p. 6
- The average training reward continuously increased, indicating the DQN’s ability to realize multi-objective VVO.A positive reward means no voltage violation and further power-loss reduction after the action, compared with before it was taken.Source: A. Learning Performance · p. 6
- Positive rewards occurred in 4,000 new online test cases, demonstrating effective VVO control under new operating conditions.The cases were evaluated in episodes 9,000–13,000 of Fig. 5.Source: A. Learning Performance · p. 6, Figure 5 · p. 6
B. VVO Performance
The proposed MADRL method performs online voltage regulation and power-loss reduction simultaneously under random operating conditions.
1) Voltage Regulation
- The success rate Sv is the fraction of initially violating test cases that avoid voltage violations after control; higher Sv indicates better regulation.Sv = Nv/Mv, where Mv is the number of initially violating cases and Nv is the number resolved by the method.Source: B. VVO Performance · p. 6
- Voltage-regulation success reaches 99.80% across 1,000 test cases in the 13-bus system and up to 99.975% in the 123-bus system.Table III summarizes the voltage-regulation performance statistics.Source: B. VVO Performance · p. 7
- In a 13-bus test case, control brings previously violating A- and C-phase voltages at buses 611, 652, 671, 675, and 684 within normal limits.Fig. 6 compares three-phase voltage magnitudes before and after control.Source: Figure 6 · p. 7, B. VVO Performance · p. 7
- In a 123-bus test trial, jointly dispatched regulating devices avoid voltage violations, with smart inverters supplying reactive power for voltage lift.The paper reports device taps/statuses and compares voltage-profile minimum, maximum, and average values for the trial.Source: B. VVO Performance · p. 7
2) Power Loss Reduction
The proposed method reduces power losses across all test cases in both the IEEE 13-bus and 123-bus systems while supporting the paper’s dual goals of loss reduction and voltage regulation.
Power Loss Reduction
- Fig. 7 compares power losses with and without control across 50 randomly selected 13-bus test cases and shows effective loss reduction with the proposed method.Source: 2) Power Loss Reduction · p. 7
- The evaluation defines ∆P_i as the active-power-loss difference between uncontrolled and proposed-control operation, assuming no capacitor banks and default AVR steps and inverter power factors of 1.Source: 2) Power Loss Reduction · p. 7
- Average power-loss reduction was 34.12 kW (14.78% of uncontrolled losses) in the 13-bus system and 109.09 kW (36.09% loss reduction) in the 123-bus system.The reported averages show reductions in all test cases.Source: 2) Power Loss Reduction · p. 7
C. Computation Time
The proposed MADRL method executes quickly in online tests on both benchmark systems and remains computationally efficient for the high-dimensional action spaces of three-phase distribution systems.
C. Computation Time
- Average executive time was 21.7 milliseconds on 13-bus and 39.2 milliseconds on 123-bus test cases, promising real-time implementation.Tests used MATLAB on a 2.5 GHz Intel Core i5 CPU with 8 GB of RAM.Source: C. Computation Time · p. 7
- The proposed MADRL method remains computationally efficient on the unbalanced 123-bus system and is competitive in high-dimensional three-phase distribution-system action spaces.Source: C. Computation Time · p. 8
V. CONCLUSION
The paper proposes a real-time DQN-based VVO method that uses distribution-system power flow as the learning environment to adaptively regulate voltage and reduce losses in unbalanced systems.
V. CONCLUSION
- The proposed VVO method adaptively selects control actions to regulate voltage and reduce power losses, with promising performance in two unbalanced distribution systems.The method interacts with an efficient power-flow environment through multiple agents.Source: §V · p. 8
- The DQN environment integrates voltage-dependent loads, DG penetration, and three types of voltage-regulating devices into distribution-system operation.The paper proposes the algorithm as a real-time method for unbalanced distribution systems.Source: §V · p. 8
- Future work will apply MADRL to load-frequency control using continuous action variables in three-phase unbalanced distribution systems.Source: §V · p. 8