PaperlayerView exact revision on arXiv

Source-linked AI summary

Batch-Constrained Reinforcement Learning for Dynamic Distribution Network Reconfiguration

Yuanqi Gao, Wei Wang, Jie Shi, Nanpeng Yu

arXiv:2006.12749v1eess.SYcs.SY

TL;DR

Dynamic distribution network reconfiguration is challenging for model-based control because it depends on accurate network parameters and can be difficult to scale. This paper develops batch-constrained soft actor-critic to learn from historical operational data without interacting with the network. The reported tests show improved performance over off-policy RL baselines and favorable comparison with the behavior policy, alongside lower computation time than model-based controllers.

  • Problem

    Dynamic DNR model-based control depends on accurate network parameters and can be difficult to scale as the problem grows.

    Source: §I · p. 1

  • Method

    BCSAC learns a reconfiguration policy from historical operational data while constraining it toward the behavior policy, without direct network interaction.

    Source: §I · p. 2, C. The Learning Setup · p. 3

  • Results

    Tests report that BCSAC outperforms state-of-the-art off-policy RL algorithms and matches or exceeds the behavior policy, with much lower computation time than model-based controllers.

    Source: §V · p. 11

  • Takeaways & Limitations

    The authors report that BCSAC can learn effective dynamic reconfiguration policies without network-parameter information and is scalable for distribution-network control.

    Source: §I · p. 2, §V · p. 11

    The study’s historical switch-configuration data were simulated because real-world switch-configuration data were unavailable.

    Source: 3) Network Configuration Data · p. 8

Abstract

from arXiv · show

Dynamic distribution network reconfiguration (DNR) algorithms perform hourly status changes of remotely controllable switches to improve distribution system performance. The problem is typically solved by physical model-based control algorithms, which not only rely on accurate network parameters but also lack scalability. To address these limitations, this paper develops a data-driven batch-constrained reinforcement learning (RL) algorithm for the dynamic DNR problem. The proposed RL algorithm learns the network reconfiguration control policy from a finite historical operational dataset without interacting with the distribution network. The numerical study results on three distribution networks show that the proposed algorithm not only outperforms state-of-the-art RL algorithms but also improves the behavior control policy, which generated the historical operational data. The proposed algorithm is also very scalable and can find a desirable network reconfiguration solution in real-time.

I. INTRODUCTION

Dynamic DNR sequences switch configurations over time to improve network operation, but model-based methods can be unreliable with inaccurate parameters and computationally difficult to scale. The paper introduces batch-constrained reinforcement learning to learn reconfiguration policies from historical data, with reported improvements over the behavior policy and off-policy RL baselines.

I. INTRODUCTION

  • Dynamic DNR must choose a sequence of switch configurations over time, and its MIP formulation is larger and harder to handle under load and generation uncertainty than static DNR.DNR changes switch status while satisfying operating constraints such as voltage limits and network radiality.Source: §I · p. 1
  • Model-based control can be unreliable with incomplete or inaccurate network parameters, while computation grows exponentially with switches, DERs, and the operation horizon.These limitations make real-time reconfiguration difficult for model-based algorithms.Source: §I · p. 1
  • Historical-data learning is attractive because physical interaction is costly and reliable simulation is difficult with inaccurate parameters, but finite data leaves many feasible actions unseen and creates extrapolation risk.Directly applying state-of-the-art RL can underperform the policy that generated the data.Source: §I · p. 2
  • The proposed BCSAC learns from finite historical operational data and improves the behavior policy while outperforming state-of-the-art off-policy RL algorithms.The method is intended to avoid direct interaction with the distribution network.Source: §I · p. 2
  • The paper reports a convergence proof for off-policy batch-constrained policy iteration and claims lower operational costs than off-policy RL methods and shorter computation time than model-based controllers.The introduction describes the test networks as having billions of feasible configurations and says the method needs no network-parameter information or direct real-system interaction.Source: §I · p. 2

B. Formulate Dynamic DNR as an MDP

The dynamic DNR problem is formulated as an MDP whose state tracks injections, switch configuration, and time, while actions exchange one branch pair subject to radiality. The learning setup uses off-policy batch RL with historical measurements, although the available switch-configuration data in this study are simulated.

B. Formulate Dynamic DNR as an MDP

  • The state includes nodal real and reactive injections, switch statuses, and time; the continuing-task formulation uses a finite action space and continuous state variables.A switch status of 1 denotes a closed switch.Source: B. Formulate Dynamic DNR as an MDP · pp. 2–3
  • Each action exchanges one pair of branch statuses—closing one switch and opening another—while keeping the resulting network radial.Only a subset of switch pairs is feasible from a given configuration; exchanging a switch with itself leaves the configuration unchanged without switching cost.Source: B. Formulate Dynamic DNR as an MDP · p. 3
  • Power-injection transitions follow a random process independent of the action, whereas the switch configuration changes according to the chosen exchange and time advances by one step.Together, these rules specify the transition from the current state to the next.Source: B. Formulate Dynamic DNR as an MDP · p. 3
  • The reward accounts for line-loss cost, switch-opening or closing cost, and penalties for operating-limit violations, including voltage violations.The voltage penalty uses measured-node voltage bounds and a penalty factor chosen based on operational considerations and empirical performance.Source: B. Formulate Dynamic DNR as an MDP · p. 3

C. The Learning Setup

  • Off-policy batch RL learns a policy different from the one that generated the historical data, rather than evaluating and improving a policy through environment interaction.Batch RL here means learning from a given historical dataset containing relevant state, action, and reward information.Source: C. The Learning Setup · p. 3
  • The intended historical measurements include nodal injections, substation power, voltages at a subset of nodes, and remotely controllable switch statuses.These measurements are used to construct historical MDP states, actions, and rewards.Source: C. The Learning Setup · p. 3
  • Batch learning is constrained by a high-dimensional state space that grows exponentially with network size and by the limited information in finite historical data.The state-space growth makes neural-network value-function or policy approximation nontrivial.Source: C. The Learning Setup · p. 3

III. TECHNICAL METHODS

The methods build on policy-gradient reinforcement learning and soft actor-critic, then describe how actor and critic functions are represented and updated. SAC adds policy entropy to the reward, with its influence controlled by a temperature parameter.

A. Basics of Reinforcement Learning

  • Policy-gradient methods learn a parameterized policy that selects actions directly, an alternative to estimating action values in problems with very large state and action spaces.SAC is presented as a maximum-entropy RL method intended to improve policy-gradient sample efficiency and robustness.Source: A. Basics of Reinforcement Learning · pp. 3–4

1) Soft Actor-Critic:

  • Soft actor-critic adds τH(π(·|s)) to the reward, with temperature τ controlling the entropy contribution.The method’s value functions and policy can be approximated by neural networks.Source: 1) Soft Actor-Critic · p. 4
  • SAC iteratively updates value-function and policy parameters using gradients of the state-value squared residual, soft Bellman residual, and policy objective.The passage also defines DKL(p||q) as the KL divergence between distributions p and q.Source: 1) Soft Actor-Critic · p. 4

B. Batch-constrained Reinforcement Learning

Batch reinforcement learning uses finite historical data, but ordinary off-policy learning can misestimate action values and overestimate actions poorly represented in that data. The section constrains learned policies toward the behavior policy and establishes convergence for the resulting policy iteration, while noting the theorem is not directly implementable in the dynamic DNR setting.

B. Batch-constrained Reinforcement Learning

  • With finite batch data, ordinary off-policy algorithms can misestimate action values and erroneously assign high values to actions.The section identifies this as extrapolation error.Source: B. Batch-constrained Reinforcement… · p. 4
  • A batch-constrained policy tends to visit state-action pairs contained in the batch, where value estimation can be accurate.The stated zero-error condition is equality between empirical and true transition probabilities for all state-action pairs visited by the policy.Source: B. Batch-constrained Reinforcement… · p. 4

C. KL-Divergence Regularization and the Bellman Equation

  • KL-divergence regularization penalizes dissimilarity between the learned policy and the behavior policy that generated historical data.The paper describes minimizing the KL divergence as maximizing target-policy entropy coupled with minimizing cross entropy.Source: C. KL-Divergence Regularization and… · p. 4
  • Under bounded policy-to-behavior KL divergence, alternating batch-constrained soft policy evaluation and improvement converges, but the theorem cannot be directly implemented for dynamic DNR.The stated obstacles are the infinite state space and finite training data; the paper proceeds to derive an approximate practical algorithm.Source: C. KL-Divergence Regularization and… · pp. 4–5

D. Overview of the Proposed Framework

The framework trains BCSAC offline from a utility's historical operational dataset. Its trained policy network takes network configuration and injection patterns as input and outputs a reconfiguration action.

D. Overview of the Proposed Framework

  • Historical operational data is used for offline training of the proposed BCSAC algorithm.The framework includes a conditional generative model and three groups of neural networks.Source: D. Overview of the Proposed Framework · p. 5
  • The conditional generative model is trained independently, while the three neural-network groups are trained simultaneously.The framework diagram uses red for the generative model and green for the neural-network groups.Source: D. Overview of the Proposed Framework · p. 5
  • After training, the policy network maps network configuration and injection patterns to a reconfiguration action.It is trained to approximate the optimal batch-constrained soft policy.Source: D. Overview of the Proposed Framework · p. 5

E. Batch-Constrained Soft Actor-Critic

BCSAC approximates batch-constrained policy iteration with an actor-critic trained from finite historical operational data. The critic and actor use neural networks, with an actor update designed to be estimable from historical data.

E. Batch-Constrained Soft Actor-Critic

  • BCSAC uses an actor-critic algorithm to approximate policy iteration and learn a batch-constrained policy from finite historical operational data.The critic approximates the regularized action-value function, while the actor approximates the policy.Source: E. Batch-Constrained Soft Actor-Critic · p. 5

1) The Critic:

  • The critic uses Q and value neural networks trained on historical transition batches, with target-value and clipped-double-Q methods for stabilization.The batches contain state, action, reward, and next-state data; sampled actions from the current policy are also used.Source: 1) The Critic · pp. 5–6

2) The Actor:

  • The actor is a neural network whose input is state and whose output is a conditional probability distribution over actions.The actor parameters are updated using gradient ascent.Source: 2) The Actor · p. 6
  • The actor update omits the Q-function gradient, allowing an off-policy policy gradient to be estimated from historical data while retaining monotonic policy-improvement approximation.The paper says including that derivative would yield an on-policy gradient that cannot be estimated from the historical dataset.Source: 2) The Actor · p. 6

F. Representing the Batch Distribution as a Parametric Model

The paper models the historical behavior policy with a conditional variational autoencoder (CVAE), a parametric generative model suited to high-dimensional state-action data. Its decoder represents the behavior policy after training on historical state-action pairs.

F. Representing the Batch Distribution as a Parametric Model

  • A conditional variational autoencoder represents the behavior policy when high-dimensional state and action spaces make estimation difficult.The paper cites scalability, handling of mixed discrete and continuous variables, and empirical performance as reasons for its choice.Source: F. Representing the Batch Distribution… · p. 6
  • The CVAE encoder maps state-action pairs to a latent representation, and its decoder produces action probabilities conditioned on state and latent input.The trained decoder is used to represent the historical behavior policy.Source: F. Representing the Batch Distribution… · p. 6

G. Summary of BCSAC Algorithm

BCSAC trains a control policy from a fixed historical dataset, combining batch-constrained soft policy iteration with a pretrained conditional generative model. Its DNR implementation represents configurations and feasible switching actions for neural networks.

G. Summary of BCSAC Algorithm

  • Each BCSAC iteration samples historical experiences, evaluates Q- and V-networks, then updates the policy; the pretrained CVAE remains fixed because the dataset does not change.The policy update uses a finite-action-space gradient.Source: G. Summary of BCSAC Algorithm · pp. 6–7
  • BCSAC differs from existing actor-critic frameworks through batch-constrained soft policy iteration, finite-action policy gradients, and a pretrained conditional generative model.The policy gradient replaces the reparameterization trick used in [28].Source: G. Summary of BCSAC Algorithm · pp. 6–7
  • The DNR dataset encodes nodal injections, SCADA measurements, voltages, and switch statuses as state-action-reward-next-state tuples.The passage says these tuples are prepared before training.Source: G. Summary of BCSAC Algorithm · p. 7

H. Algorithm Implementation

  • A binary line-status vector represents each configuration, while a masked softmax assigns probability only to feasible switching pairs.The same masked-softmax output design is used for the policy and the parametric generative model; the Q-network takes the state and a closeable-switch encoding, while the V-network takes the state.Source: H. Algorithm Implementation · p. 7

IV. NUMERICAL STUDIES

The numerical studies evaluate BCSAC and benchmark reinforcement-learning algorithms on four distribution networks using historical operational datasets. They also assess whether a CVAE approximates the behavior policy represented in those datasets.

1) Distribution Networks:

  • The study uses 16-, 33-, 70-, and 119-bus networks, and feasible configurations grow exponentially with remotely controllable switches.Each line segment is assumed to have a remotely controllable switch; the configuration count is calculated using the matrix-tree theorem.Source: 1) Distribution Networks · p. 7, Table I · p. 8

2) Nodal Power Data:

  • Load data come from Irish smart-meter measurements, while solar-generation data come from southern California; the first 52 weeks train and the following week tests each case.Customer loads are aggregated to network nodes, with a constant lagging power factor assumed.Source: 2) Nodal Power Data · pp. 7–8

3) Network Configuration Data:

  • Because real-world switch-configuration data were unavailable, historical configurations were simulated using model-based, fixed, or random changes, with scenario probabilities varied across datasets.The model-based scenario uses line parameters deviating 10% from their true values; the fixed scenario leaves configurations unchanged, and the random scenario selects another constraint-satisfying topology.Source: 3) Network Configuration Data · p. 8

B. Algorithm Setup

  • BCSAC, DQN, and SAC hyperparameters were tuned individually, and the study also compares the algorithms with the mixed historical operational strategy.The historical strategy mixes model-based, passive, and random control scenarios.Source: B. Algorithm Setup · p. 8, Table II · p. 9

C. Approximating Behavior Policy by CVAE

  • On the 16-bus sample, the CVAE approximates behavior-policy actions at state S18 accurately despite only one observed action there; the figure reports TV-distance = 0.19.Figure cells represent probabilities of closing and opening switch pairs; infeasible pairs are white.Source: Figure 3 · p. 9, C. Approximating Behavior Policy by CVAE · p. 9
  • The CVAE generalizes well across states and data distributions, with average total-variation distance reported across dataset states.The table reports the average distance between the behavior policy and its CVAE approximation.Source: Table III · p. 9, C. Approximating Behavior Policy by CVAE · p. 9

D. Optimality and Scalability

BCSAC lowers weekly operational cost against RL benchmarks and historical control, and its advantage extends across larger feeders and varied historical datasets. Its performance is stable across a moderate temperature range but degrades when the temperature is very large.

D. Optimality and Scalability

  • BCSAC yields lower weekly operational cost than DQN, SAC, and the historical strategy on the 16-bus feeder, with little or no observed overfitting.The comparison covers the testing week; training with a small neural network and early stopping helped limit overfitting.Source: Figure 4 · p. 9, D. Optimality and Scalability · pp. 9–10
  • BCSAC’s performance is nearly identical for τ from 0.001 to 0.1, but a very large τ degrades performance by shifting learning toward the behavior policy.The authors suggest choosing τ so the numerical ranges of r(s, a) and |A(s)|/τ are roughly the same order; both quantities can be calculated from historical data.Source: D. Optimality and Scalability · p. 10, Figure 5 · p. 10
  • Across six additional historical datasets and four feeders, BCSAC outmatches DQN and SAC in most experiments and improves the behavior policy on most datasets.It consistently outperforms the RL benchmarks on the 33-, 70-, and 119-bus feeders.Source: D. Optimality and Scalability · p. 10, Table V · p. 10
  • On the 70- and 119-bus feeders, with more than 3.8 quadrillion feasible configurations, BCSAC outperforms DQN and SAC and the existing behavior policy.DQN and SAC fail to learn a dynamic DNR strategy for the highly resistive 70-bus feeder; the historical data cover only a tiny subset of feasible state-action pairs.Source: D. Optimality and Scalability · p. 10, Table IV · p. 10

E. Behavior of BCSAC in Response to Unforeseen States

BCSAC retains performance against unfamiliar configurations and extreme power injections, and RL-based control is substantially faster to test than model-based control. The conclusion reports strong performance across four networks, scalability, and lower computation time than model-based controllers.

E. Behavior of BCSAC in Response to Unforeseen States

  • For unseen network configurations and extreme injection patterns, BCSAC yields lower or nearly the same operational cost as the behavior policy.The extreme patterns disconnect solar generation and add large loads; their deviation from the training-set average is 1.3 times the largest training-set deviation for a typical hour.Source: E. Behavior of BCSAC in Response to… · pp. 10–11, Figure 6 · p. 11

F. Computation Speed

  • RL-based algorithms’ testing time is at least two orders of magnitude shorter than model-based control, and the advantage grows with network size.With a five-hour optimization horizon, the model-based solver fails to converge within one hour.Source: F. Computation Speed · p. 11, Table VIII · p. 11

V. CONCLUSION

  • Across four distribution networks, BCSAC outperforms state-of-the-art off-policy RL and matches or outperforms the behavior policy without network-parameter information.The conclusion also describes the algorithm as scalable and faster than model-based controllers.Source: §V · p. 11

APPENDIX A PROOFS

The appendix establishes convergence properties for the policy evaluation and improvement operators used in the paper’s batch-constrained framework. It shows that policy improvement is non-decreasing and that the converged policy is optimal.

A. Proof of Lemma 1

  • For bounded KL divergence and γ < 1, the policy evaluation operator is a contraction and has a unique fixed point to which iteration converges.The augmented reward subtracts a τ-weighted KL divergence between the evaluated policy and the behavior policy.Source: A. Proof of Lemma 1 · pp. 11–12

B. Proof of Lemma 2

  • The policy update maximizes expected value penalized by KL divergence from the behavior policy, yielding an objective at least as high as that of the current policy.The proof uses this inequality to establish the policy-improvement step.Source: B. Proof of Lemma 2 · p. 12

C. Proof of Theorem 1

  • Repeated policy evaluation and improvement produce a non-decreasing, bounded sequence, and the converged policy is optimal.At convergence, the policy maximizes the value-minus-KL objective for every state.Source: C. Proof of Theorem 1 · p. 12
Loading 2006.12749v1…