PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal Tap Setting of Voltage Regulation Transformers Using Batch Reinforcement Learning

Hanchen Xu, Alejandro D. Domínguez-García, Peter W. Sauer

arXiv:1807.10997v2math.OCcs.LG

TL;DR

Distribution-system tap optimization is difficult when line parameters and power injections are unavailable and load conditions change. The paper uses offline batch reinforcement learning with a virtual transition generator and sequential learning for individual LTCs. Simulations on two IEEE feeders report higher daily mean reward than the conventional scheme, with results similar to exhaustive search on the 13-bus feeder.

  • Problem

    Distribution-system OPF approaches rely on line parameters and power injections that may be unavailable, while snapshot-based optimization must be repeated as conditions change.

    Source: §I · p. 1

  • Method

    The paper formulates tap setting as an MDP and combines a linearized-model virtual transition generator with batch RL and sequential LSPI learning for each LTC.

    Source: §I · p. 1, §V · p. 4, B. Virtual Transition Generator · p. 5

  • Results

    On both IEEE feeders, batch RL achieved higher daily mean reward than the conventional scheme; on the 13-bus feeder, its voltage profiles were similar to exhaustive search.

    Source: B. Case Study on the IEEE 13-bus Test… · p. 8, C. Case Study on the IEEE 123-bus Test… · p. 8

  • Takeaways & Limitations

    The proposed policy determines LTC taps using voltage magnitudes and topology information, without requiring power injections or line parameters.

    Source: §VII · p. 8

Abstract

from arXiv · show

In this paper, we address the problem of setting the tap positions of load tap changers (LTCs) for voltage regulation in radial power distribution systems under uncertain load dynamics. The objective is to find a policy to determine the tap positions that only uses measurements of voltage magnitudes and topology information so as to minimize the voltage deviation across the system. We formulate this problem as a Markov decision process (MDP), and propose a batch reinforcement learning (RL) algorithm to solve it. By taking advantage of a linearized power flow model, we propose an effective algorithm to estimate the voltage magnitudes under different tap settings, which allows the RL algorithm to explore the state and action spaces freely offline without impacting the system operation. To circumvent the "curse of dimensionality" resulted from the large state and action spaces, we propose a sequential learning algorithm to learn an action-value function for each LTC, based on which the optimal tap positions can be directly determined. The effectiveness of the proposed algorithm is validated via numerical simulations on the IEEE 13-bus and 123-bus distribution test feeders.

I. INTRODUCTION

The introduction motivates tap setting under limited distribution-system information and presents an offline batch-RL approach. The paper also introduces a radial-system power-flow model that accounts for LTC tap positions.

I. INTRODUCTION

  • Conventional local voltage control may cause frequent LTC actions and wear while failing to minimize voltage deviation.OPF-based alternatives require system information that may be unavailable in distribution systems and typically solve for individual snapshots.Source: §I · p. 1
  • Distribution-system OPF methods require line parameters and power injections that may be unknown or unavailable in real time, and must be resolved for each snapshot.The cited motivation notes that loads are assumed constant between consecutive snapshots.Source: §I · p. 1
  • The proposed MDP/RL policy uses voltage magnitudes and topology, while a linearized power-flow model estimates voltages for offline exploration under alternative tap settings.The virtual exploration avoids changing controls on the real system.Source: §I · p. 1
  • An LSPI-based sequential learner estimates an action-value function for each LTC to address exponential growth of the joint state and action spaces.The policy is computed offline, with minimal computation needed to select tap positions online.Source: §I · p. 1

A. Power Distribution System Model

  • LTC tap ratios are modeled with discrete positions, and feasible actions are tap-ratio changes on LTC-equipped lines.The described tap range has 33 positions from 0.9 to 1.1 in increments of 0.00625 p.u.Source: A. Power Distribution System Model · p. 2
  • The system model extends LinDistFlow to include LTCs, relating squared voltage magnitudes to active and reactive injections through network and tap-dependent matrices.For radial systems, the model assumes a connected network; the paper reports validation on IEEE 13-bus and 123-bus feeders.Source: A. Power Distribution System Model · p. 2, §I · p. 2

B. Optimal Tap Setting Problem

The optimal tap-setting problem seeks a policy that chooses LTC tap changes to reduce voltage deviation using current tap ratios and voltage-magnitude measurements. The formulation assumes known topology but does not require known line parameters or measured power injections.

B. Optimal Tap Setting Problem

III. MARKOV DECISION PROCESS AND BATCH REINFORCEMENT LEARNING

The paper frames sequential decision-making with an MDP and describes how policies are evaluated through discounted rewards. When transition models are unknown, reinforcement learning can use sampled transitions to estimate an optimal action-value function.

A. Markov Decision Process

  • An MDP is defined by states, actions, transition probabilities, rewards, and a discount factor.A transition records a state, action, reward, and resulting state.Source: A. Markov Decision Process · p. 3
  • A policy maps states to actions, and its action-value function is the expected discounted return after taking an action and following that policy.The optimal action-value function is the maximum over policies, and choosing its maximizing action gives an optimal policy.Source: A. Markov Decision Process · p. 3
  • When transition probabilities are unknown but transition samples are available, reinforcement-learning algorithms can solve the MDP.With a known transition model, policy iteration and value iteration are cited as alternatives.Source: A. Markov Decision Process · p. 3

B. Batch Reinforcement Learning

The section motivates function approximation and batch LSPI for large or continuous state spaces. It explains how repeated least-squares updates use transition samples and temporal-difference targets to estimate action values.

B. Batch Reinforcement Learning

  • Small discretization steps can improve state-space approximation but worsen dimensionality, while large steps reduce computation at possible performance cost.Source: B. Batch Reinforcement Learning · p. 3
  • Each LSPI update fits the approximate action value to a temporal-difference target over transition samples and repeats until the parameter change falls below a threshold.The target uses the reward plus the discounted value of the next state and a greedy next action.Source: B. Batch Reinforcement Learning · p. 4
  • LSPI approximates action values with a linear function and reuses batches of transitions across iterations, with parameter updates obtained through least squares.The paper characterizes this approach as data-efficient and stable.Source: B. Batch Reinforcement Learning · pp. 3–4

IV. OPTIMAL TAP SETTING PROBLEM AS AN MDP

The tap-setting problem is formulated as an MDP with tap ratios and squared voltage magnitudes as state, feasible tap changes as actions, and voltage deviation as the regulation objective.

IV. OPTIMAL TAP SETTING PROBLEM AS AN MDP

  • The MDP state combines LTC tap ratios with squared voltage magnitudes at buses other than the substation.The state space includes both discrete tap ratios and continuous voltage variables.Source: 1) State space · p. 4

2) Action space:

  • Actions are discrete LTC tap-ratio changes, and the action-space size grows exponentially with the number of LTCs.The action space contains all feasible tap-ratio changes.Source: 2) Action space · p. 4

3) Reward function:

  • The regulation objective is to minimize voltage deviation measured by the L2 norm, which determines the transition reward.Source: 3) Reward function · p. 4

4) Transition model:

Because the transition model is unknown under the paper’s assumptions, the method generates virtual transitions and learns from them offline. Its framework uses those samples to fit an action-value function and choose greedy tap actions.

4) Transition model:

  • Under the stated assumptions, line parameters and power-injection distributions are unknown, so the transition model is not known a priori.The formulation therefore resorts to reinforcement learning algorithms that do not require an explicit transition model.Source: 4) Transition model · p. 4

V. OPTIMAL TAP SETTING ALGORITHM

  • An LSPI-based algorithm combines transition generation with sequential action-value learning to solve the tap-setting MDP.Source: §V · p. 4

A. Overview

  • The learning agent fits an approximate action-value function from transition samples; the acting agent uses it to select a greedy action for the current state.The learning agent can explore offline through the virtual generator without directly interacting with the power system.Source: A. Overview · pp. 4–5

B. Virtual Transition Generator

  • The virtual transition generator reuses an observed injection transition to estimate voltages under exploratory tap changes, without injection or line-parameter knowledge.It generates samples for offline exploration rather than requiring random exploration on the operating system.Source: B. Virtual Transition Generator · p. 5

C. LSPI-based Sequential Action-Value Function Learning

To avoid the exponential parameter burden of learning over the full action vector, the method learns action-value functions one LTC at a time, using LSPI and RBF features.

C. LSPI-based Sequential Action-Value Function Learning

  • For each LTC, the method learns a separate approximate action-value function while other LTCs act greedily; the target LTC explores tap changes uniformly.The process repeats in round-robin fashion across LTCs for J iterations.Source: C. LSPI-based Sequential Action-Value… · p. 5
  • The sequential method reduces unknowns to L_t|ΔT|, rather than |ΔT^L_t| for an action-value function over the full action vector.The full-action parameter count grows exponentially with the number of LTCs.Source: C. LSPI-based Sequential Action-Value… · pp. 5–6
  • Each LTC’s learning iteration uses a newly generated transition set, and its initial parameter vector can use the previous iteration’s estimate.Source: C. LSPI-based Sequential Action-Value… · pp. 5–6
  • RBF features represent each state-action pair in segments corresponding to tap changes, with only the selected change’s segment nonzero.The features compare estimated squared voltages with prespecified RBF centers.Source: C. LSPI-based Sequential Action-Value… · p. 6

D. Tap Setting Algorithm

The algorithm updates its learned action-value function periodically, then selects a greedy tap action for the current state. It holds tap positions when the greedy action’s value gain does not exceed a threshold.

D. Tap Setting Algorithm

  • Every K time instants, the learning agent updates w(l) with LSPI-based sequential learning, and the acting agent selects a greedy action for the current state.The action is sent to the LTCs.Source: D. Tap Setting Algorithm · p. 6
  • At each step, the algorithm observes the new state and the reward following the previous action, adding observations to history H at update times.Source: D. Tap Setting Algorithm · p. 6
  • For each LTC, the algorithm generates virtual transitions using uniform exploration for that LTC and greedy exploration for the others, then runs LSPI-based sequential learning.Source: D. Tap Setting Algorithm · p. 6
  • Tap positions remain unchanged unless the greedy action’s action-value exceeds the current value by more than threshold ǫ.This threshold is used to reduce LTC wear.Source: D. Tap Setting Algorithm · p. 6

VI. NUMERICAL SIMULATION

The proposed methodology is applied to the IEEE 13-bus and 123-bus test feeders.

VI. NUMERICAL SIMULATION

  • The proposed methodology is applied to the IEEE 13-bus and 123-bus distribution test feeders.Source: §VI · p. 6

A. Simulation Setup

The simulations use constructed load profiles and compare batch RL with exhaustive search and conventional tap setting on the IEEE 13-bus feeder. Batch RL’s daily mean reward is close to exhaustive search and higher than the conventional scheme’s.

A. Simulation Setup

  • The feeder loads use one year of historical hourly residential-building data, scaled to 6.15 MW maximum system load and interpolated to five-minute intervals.Each interval is multiplied by a normally distributed variable with mean 1 and standard deviation 0.02; bus loads are pseudorandomly redistributed with constant power factor 0.95.Source: A. Simulation Setup · pp. 6–7
  • The virtual transition generator’s maximum absolute voltage-magnitude estimation error was below 0.001 p.u. against full AC power-flow results.Source: A. Simulation Setup · p. 7

B. Case Study on the IEEE 13-bus Test Feeder

  • For the one-LTC feeder, the simulation used 21 RBF centers, 6,000 virtual transitions, and policy updates every two hours.The time interval was five minutes; no sequential updating was needed because the feeder had only one LTC.Source: B. Case Study on the IEEE 13-bus Test… · p. 7
  • The benchmark exhaustive search assumes perfect system parameters and active and reactive injections, while conventional control adjusts taps only when voltage leaves a desired range such as [0.9, 1.1] p.u.Exhaustive search is described as infeasible in practice because it requires this information and has high computational burden.Source: B. Case Study on the IEEE 13-bus Test… · p. 7
  • Daily mean reward was ρ = −4.279 × 10−3 for batch RL and ρ = −4.156 × 10−3 for exhaustive search, versus ρ = −19.26×10−3 for the conventional scheme.Batch RL and exhaustive-search tap positions were aligned during most of the day; their voltage profiles were also quite similar, and both had higher daily mean reward than the conventional scheme.Source: B. Case Study on the IEEE 13-bus Test… · pp. 7–8, Figure 4 · p. 7, Figure 5 · p. 7

C. Case Study on the IEEE 123-bus Test Feeder

On the IEEE 123-bus feeder, batch RL’s daily mean reward was closer to exhaustive search than to conventional control. The paper concludes that its MDP-based batch RL approach was effective on both IEEE feeders.

C. Case Study on the IEEE 123-bus Test Feeder

  • Daily mean reward was ρ = −1.646 × 10−3 for batch RL and ρ = −1.402 × 10−3 for exhaustive search, versus ρ = −7.513 × 10−3 for the conventional scheme.Tap positions were restricted to 9 options for two LTCs and 5 options for the other two after excluding settings estimated to exceed the desired voltage range.Source: C. Case Study on the IEEE 123-bus Test… · p. 8

VII. CONCLUDING REMARKS

  • The proposed method uses voltage magnitude measurements and network topology to determine tap positions, and simulations on the IEEE 13-bus and 123-bus feeders validated its effectiveness.It combines an MDP formulation, a virtual transition generator, and LSPI-based sequential learning for an action-value function for each LTC.Source: §VII · p. 8
Loading 1807.10997v2…