Source-linked AI summary
Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning
Qiuling Yang, Gang Wang, Alireza Sadeghi, Georgios B. Giannakis, Jian Sun
TL;DR
Rapid renewable- and load-driven voltage fluctuations create a need to coordinate slow capacitor commitments with fast inverter controls. The paper combines deep reinforcement learning for slow-timescale capacitor configuration with physics-based optimization for fast inverter setpoints, and numerical tests report effective voltage regulation on real distribution-system data.
Problem
Distribution-grid voltage regulation must coordinate utility-owned devices with discrete, slow controls and smart inverters with fast, continuously valued controls.
Method
The scheme models capacitor configuration as an MDP solved by deep reinforcement learning and optimizes inverter reactive-power setpoints on a faster timescale using exact or approximate grid models.
Results
Extensive numerical tests on real-world distribution systems using real solar and consumption data found the two-timescale scheme effective in smoothing voltage fluctuations and attaining lower cost after learning.
Takeaways & Limitations
Coupling learned slow-timescale capacitor control with fast inverter optimization provides a practical two-timescale approach to voltage regulation.
Abstract
from arXiv · showhide
Modern distribution grids are currently being challenged by frequent and sizable voltage fluctuations, due mainly to the increasing deployment of electric vehicles and renewable generators. Existing approaches to maintaining bus voltage magnitudes within the desired region can cope with either traditional utility-owned devices (e.g., shunt capacitors), or contemporary smart inverters that come with distributed generation units (e.g., photovoltaic plants). The discrete on-off commitment of capacitor units is often configured on an hourly or daily basis, yet smart inverters can be controlled within milliseconds, thus challenging joint control of these two types of assets. In this context, a novel two-timescale voltage regulation scheme is developed for distribution grids by judiciously coupling data-driven with physicsbased optimization. On a faster timescale, say every second, the optimal setpoints of smart inverters are obtained by minimizing instantaneous bus voltage deviations from their nominal values, based on either the exact alternating current power flow model or a linear approximant of it; whereas, on the slower timescale (e.g., every hour), shunt capacitors are configured to minimize the longterm discounted voltage deviations using a deep reinforcement learning algorithm. Extensive numerical tests on a real-world 47- bus distribution network as well as the IEEE 123-bus test feeder using real data corroborate the effectiveness of the novel scheme.
I. INTRODUCTION
Modern distribution grids face increasingly frequent voltage fluctuations that expose the mismatch between slow utility-owned devices and fast smart inverters. The paper proposes a two-timescale scheme combining data-driven learning for capacitors with physics-based optimization for inverters.
- Motivation: Growing electric-vehicle, demand-response, and renewable deployment produces frequent voltage fluctuations and challenging load peaks.Solar generation can cause feeder voltage excursions, while reverse power flows strain substation transformers.
- Motivation: Traditional voltage regulation relies on utility-owned devices whose mixed-integer configuration problems are generally NP-hard.Their limited life cycle also leads to daily or monthly control, which is poorly matched to rapidly fluctuating distributed generation.
- Motivation: Smart inverters provide continuously valued reactive-power control within seconds, but their optimal setpoints require solving a non-convex optimal power-flow problem.Prior work therefore explored stochastic, online, decentralized, and localized inverter-control schemes.
- Proposed approach: The proposed framework jointly controls shunt capacitors and smart inverters across slow and fast timescales using data-driven and physics-driven methods.Capacitor commitments are learned with deep reinforcement learning, while inverter setpoints minimize instantaneous voltage deviations using exact or approximate grid models.
- Proposed approach: A hyper deep Q-network addresses the action-space dimensionality created by the large number of capacitor configurations.With Na capacitors, the action space contains 2^Na possible on-off configurations.
A. System model
The system model separates discrete shunt-capacitor decisions from continuously adjusted smart-inverter reactive-power control. Capacitor commitments persist over slow intervals, while inverter outputs respond on a faster timescale within capability limits.
- A. System model: The model distinguishes generation and consumption components for active and reactive power and permits their values to be predicted from markets or forecasting algorithms.The relevant predicted quantities are collected as p^c, q^c, p^g, and q^g.
- A. System model: The grid includes N_a shunt capacitors and remaining buses equipped with inverters, with zero reactive-power bounds representing buses without installed inverters.Capacitor and inverter buses are treated as disjoint under the stated modeling assumption.
- A. System model: Shunt capacitors provide discrete on-off reactive-power injections that remain fixed throughout each slow interval.Each capacitor commitment is binary, and an active unit injects its nameplate reactive power; otherwise, it injects zero.
- A. System model: Smart-inverter reactive-power outputs are adjusted every fast-timescale slot and constrained by inverter power capability.The inverter capability depends on its nameplate apparent-power rating and instantaneous active-power generation.
- A. System model: At renewable-generation maximum output, conventional inverter sizing leaves no reactive-power capacity, motivating oversized inverter nameplates.Oversized inverters satisfy the advocated condition s̄_i > p̄^g_i, preserving reactive-power headroom.
B. Two-timescale voltage regulation formulation
Voltage regulation is formulated as a stochastic, multi-stage optimization that configures capacitors across slow intervals and adjusts inverter outputs in real time. The objective is to minimize long-term voltage deviation under uncertain load and generation processes, but the resulting problem is difficult because of temporal coupling and binary decisions.
- B. Two-timescale voltage regulation formulation: The control objective is to configure capacitors and adjust inverter outputs so that long-term average voltage deviation is minimized.Bus voltages are represented as functions of reactive-power control variables.
- B. Two-timescale voltage regulation formulation: The stochastic formulation takes expectations over joint load-consumption and solar-generation trajectories across intervals and slots.A discount factor γ ∈ (0, 1) weights the long-term objective.
- B. Two-timescale voltage regulation formulation: The optimization is challenging because inverter and capacitor decisions are coupled across time, while binary capacitor variables make the problem nonconvex and generally NP-hard.It is also multi-stage and must account for real-time power variability.
- B. Two-timescale voltage regulation formulation: The proposed solution sequentially observes available predictions and solves the formulation near-optimally using both grid physics and real-time data.The approach assumes realizations can be made available online through accurate forecasting methods.
III. FAST-TIMESCALE OPTIMIZATION OF INVERTERS
Fast-timescale inverter optimization can use either an exact AC grid model or a linearized approximation. The exact formulation is handled through convex relaxation, while the linearized formulation offers lower computational cost.
- III. FAST-TIMESCALE OPTIMIZATION OF INVERTERS: The exact AC model yields a second-order cone program, whereas the linearized model yields a linearly constrained quadratic program.The linearized alternative trades modeling accuracy for computational affordability.
A. Branch flow model
The branch-flow formulation models radial distribution networks using parent-child bus relationships, line power flows, and squared current magnitudes. Relaxing its quadratic equalities produces a convex SOCP that can support fast-timescale voltage regulation, although large systems may make SOCP solution computationally demanding.
- A. Branch flow model: Each nonroot bus has one unique parent connected through its corresponding distribution line.The parent-child structure follows from the radial topology.
- A. Branch flow model: The model represents line flows using complex power at the line’s front end and squared current magnitude for each distribution line.Power-flow variables are indexed by interval and time slot, with interval notation sometimes omitted for brevity.
- A. Branch flow model: Quadratic branch-flow equalities are relaxed into inequalities, which can be rewritten as second-order cone constraints.This relaxation addresses the nonconvexity caused by quadratic terms in active and reactive line flows.
- A. Branch flow model: The relaxed equations define a convex feasible set, and under certain conditions the SOCP relaxation is exact at the optimum.Exactness means the relaxed inequalities hold with equality at the optimum.
- A. Branch flow model: The fast-timescale voltage-control problem with a fixed prior capacitor configuration is formulated as a convex SOCP for computing inverter setpoints.For relatively large distribution grids, solving SOCPs may be computationally demanding, motivating an approximate model.
B. Linearized power flow model
The linearized distribution flow model neglects relatively small line-current terms, expressing squared voltage magnitudes as linear functions of inverter reactive-power outputs. With fixed capacitor decisions, inverter setpoints are obtained each fast-timescale slot through a convex quadratic program.
- Neglecting relatively small line-current terms yields the linearized distribution flow model.
- The model expresses all squared voltage magnitudes v(t) as linear functions of qg(t).
- Given the latest capacitor configuration, inverter setpoints are optimized per slot t within each slow-timescale interval.
- Because constraints are linear and cost is quadratic, the inverter problem is a standard convex quadratic program solvable efficiently by optimization algorithms or software.
IV. SLOW-TIMESCALE CAPACITOR RECONFIGURATION
Slow-timescale capacitor reconfiguration is treated as a data-driven alternative to heuristic or semidefinite-relaxation methods for integer-valued control. The approach is intended to learn, track, and adapt to unknown generation and consumption dynamics.
- Capacitor reconfiguration determines the on-off status of shunt capacitors for the ensuing slow-timescale interval.
- Earlier heuristic and semidefinite-relaxation approaches do not guarantee optimality and incur high computational and storage complexities.
A. A data-driven solution
The slow-timescale capacitor problem is formulated as an MDP whose state includes load and capacitor information, while actions encode binary capacitor commitments. Because transition probabilities are impractical to obtain and the state space is high-dimensional and continuous, function-approximation-based RL is used.
- Capacitor configuration is formulated as an MDP by modeling load demand and renewable generation as Markovian processes.
- The MDP is defined by states, actions, transition matrices, costs, and a discount factor γ.
- Each action is a binary on-off commitment of all capacitors, producing an action space of cardinality |A| = 2^Na.
- Each state combines average active power at non-substation buses with the current capacitor configuration.
- The policy selects actions from the most recent state, while capacitor decisions and inverter setpoints influence one another across timescales.
- Exact optimal-policy derivation requires transition probabilities, but obtaining them is impractical in distribution systems.
- Q-learning is unsuitable for the problem's high-dimensional continuous state space, motivating function-approximation-based Q-learning schemes.
B. A deep reinforcement learning approach
The deep reinforcement learning implementation uses a DQN to select slow-timescale capacitor configurations while fast-timescale inverter optimization operates within each interval. Target networks and experience replay are incorporated for stable online learning.
- The DQN approximates the Q-function and outputs one Q-value for every possible capacitor action.
- The operator selects the capacitor configuration with the smallest predicted Q-value for the next interval.
- DQN weights are updated online after an action, state transition, and observed instantaneous cost are available.
- The target value combines observed instantaneous cost with a discounted smallest predicted future cost.
- The DQN parameters are tuned by minimizing an ℓ2-norm loss using stochastic-gradient updates.
- Target-network and experience-replay modifications are adopted because the basic DQN update can be unstable in practice.
- Algorithm 1 alternates action selection, cost evaluation, fast inverter computations, state updates, replay-buffer sampling, and DQN training.
V. NUMERICAL TESTS
Numerical tests evaluate the proposed DRL-based voltage-control scheme on real-world 47-bus and IEEE 123-bus feeders using real consumption and solar-generation data. Across the experiments, DRLCap learns stable control policies and reduces voltage fluctuations relative to the evaluated alternatives.
- Experimental setup: Tests use a real-world 47-bus feeder and the IEEE 123-bus benchmark with real consumption and solar-generation data.The reported numerical results use the linearized distribution-flow model because it closely approximates the exact AC model.
- Experimental setup: The 47-bus experiment includes three controlled shunt capacitors and five smart inverters embedded with photovoltaic plants.The feeder has four capacitors overall, but the substation capacitor is excluded because the substation voltage is regulated at 1.
- 47-bus results: DRLCap attains lower time-averaged instantaneous cost than FixCap, RandCap, and Real-time after a short learning period.The comparison covers the first 1 ≤ τ ≤ 2,000 intervals; the Real-time policy can underperform because its convexified problem differs from the original nonconvex problem.
- 47-bus results: After approximately 4,500 slots of training, DRLCap learns a stable and near-optimal policy that smooths voltage fluctuations under large solar generation and heavy load demand.Voltage profiles are compared at buses 10 and 33 near the end of the 10,000-slot simulation and across all buses at slot 9,900.
- 123-bus scalability: A hyper deep Q-network scales capacitor control by splitting 2^N_a Q-value predictions among K parallel smaller DQNs.For the IEEE 123-bus experiment, eight controlled capacitors yield 2^8 = 256 configurations, represented using K = 64 DQNs.
VI. CONCLUSIONS
The paper proposes joint reactive-power control of utility-owned capacitors and smart inverters through a two-timescale combination of physics-based optimization and deep reinforcement learning. Tests on real distribution systems using real solar and consumption data found the scheme efficient and practical to implement.
- Contribution: The scheme coordinates traditional capacitor banks and smart inverters to minimize bus-voltage deviations despite their different response times.Smart-inverter setpoints are optimized on a fast timescale, while capacitor configurations are selected on a slower timescale.
- Control strategy: Smart-inverter setpoints minimize instantaneous voltage deviations, whereas capacitor banks minimize long-term expected deviations using deep reinforcement learning.Load consumption and active-power generation dynamics are modeled as Markov decision processes.
- Conclusion: Extensive tests on real-world distribution systems with real solar and consumption data found the developed scheme efficient and easy to implement in practice.The conclusion reports this outcome without restricting it to a single feeder or experiment.