PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics-Informed Graph-Based Safe RL with Constraint-Specific Critics for Community Battery Coordination in Distribution Network

Chenghao Huang, Yue Yang, Hao Wang

arXiv:2610.33654v1cs.AIcs.SYeess.SYmath.OC

TL;DR

High DER penetration and EV charging make safe, low-cost distribution-network operation challenging, while coordinated control must account for coupled network constraints. The paper combines physics-informed graph representations with constraint-specific critics to coordinate community batteries. In the tested cases, the approach reduces voltage-violation degree by approximately 49.0%–80.5% versus baselines and achieves the lowest empirical constraint violations with competitive operating cost.

  • Problem

    Coordinating multiple community batteries under high behind-the-meter DER penetration is challenging because network-level voltage and line constraints are coupled across buses and interact with economic optimization.

    Source: B. Objective · pp. 4–5

  • Method

    The framework combines physics-informed graph propagation and physical-consistency losses with LDCPPO, which estimates heterogeneous constraint returns separately and combines advantages during the Lagrangian policy update.

    Source: 2) PI Adjacency Matrix · p. 6, 3) Spatial Processing · p. 6, §V · p. 14

  • Results

    PISTG-LDCPPO reduces voltage-violation degree by approximately 49.0%–80.5% versus all baselines in Table III and achieves the lowest empirical constraint violations with competitive operating cost.

    Source: 1) Overall Performance · p. 11

  • Takeaways & Limitations

    The reported gains arise from complementary physics-informed graph propagation and constraint-specific value estimation for CBESS coordination.

    Source: 1) Overall Performance · p. 11, §V · p. 14

    Future work targets three-phase unbalanced and reconfigurable networks, local-observation multi-agent coordination, shorter intervals, higher-resolution measurements, and noisy, delayed, or incomplete measurements.

    Source: §V · p. 14

Abstract

from arXiv · show

High penetrations of distributed energy resources (DERs) and fast-varying electric vehicle (EV) charging intensify voltage and line-limit violations in distribution networks (DNs), making safe, low-cost operation increasingly challenging. This paper proposes a physics-informed safe deep reinforcement learning (DRL) framework for coordinating community battery energy storage systems to mitigate constraint violations while minimizing operating cost. To represent nonlocal spatiotemporal interactions, we develop a dual-level physics-informed graph representation in which line impedance and capacity shape spatial information propagation, while power-balance, voltage-profile, and flow-reconstruction losses regularize the latent state toward control-relevant physical consistency. Building on this representation, we develop Lagrangian Dedicated-Critic PPO (LDCPPO), which implements constraint-specific value decomposition: independent reward and cost critics preserve heterogeneous voltage- and line-loading-related learning signals and combine their advantages only during the Lagrangian policy update. Case studies on DNs with high rooftop photovoltaic penetration and EV charging demonstrate that the proposed method reduces the key voltage-violation degree by up to 80.5% compared with the tested baselines, while also achieving the lowest voltage-violation count and line-overload degree with competitive operating cost.

I. INTRODUCTION

High DER penetration and EV charging heighten distribution-network safety challenges, while existing approaches leave gaps in representing network physics and learning heterogeneous constraint signals. The paper combines a physics-informed spatiotemporal graph representation with constraint-specific value decomposition for CBESS coordination.

A. Background and Motivation

  • High DER adoption creates voltage and line-capacity violations, while household-scale resources provide limited, fragmented regulation and traditional devices have responsiveness or deployment-cost constraints.Community batteries are presented as shared DSO-managed assets that can provide voltage and power-flow regulation.Source: A. Background and Motivation · p. 1

B. Literature Review and Challenges

  • The identified challenges include coupled spatial and temporal effects, insufficient physical awareness in data-driven policies, and aggregated constraint feedback that obscures individual violations.The literature review also notes that existing multi-critic approaches do not explicitly isolate long-term learning signals for heterogeneous voltage and line-loading constraints in a unified Lagrangian framework.Source: B. Literature Review and Challenges · pp. 2–3

C. Main Work and Contributions

  • The PISTG encoder uses line resistance, reactance, and capacity in graph propagation, then regularizes latent representations with KCL, voltage-drop, and voltage/branch-flow reconstruction consistency.This is intended to represent nonlocal network effects and their physical consequences for coordinated control.Source: C. Main Work and Contributions · p. 3
  • LDCPPO estimates economic return and voltage-violation degree, voltage-violation count, and line-overload degree with separate critics, combining advantages only in the Lagrangian actor update.The design aims to reduce interference and credit-assignment ambiguity among heterogeneous constraint signals.Source: C. Main Work and Contributions · p. 3
  • Compared with all tested baselines, PISTG-LDCPPO reduces voltage-violation degree by approximately 50% and achieves the lowest voltage-violation count and line-overload degree with competitive operating cost.The paper also reports real-time responsiveness in simulations on IEEE 33- and 69-bus systems.Source: C. Main Work and Contributions · p. 3

II. SYSTEM MODELING

The system model considers a DSO-managed radial distribution network with rooftop PV, nighttime EV charging, and multiple strategically placed community batteries. It specifies battery operating dynamics and constraints for the coordination setting.

II. SYSTEM MODELING

  • The modeled radial network places several CBESS units at nodes in DER-dense regions, where daytime PV and nighttime EV demand can challenge voltage compliance, energy efficiency, and line loading.The network is managed by a DSO over intra-day periods.Source: §II · p. 3, A. System Description · p. 4

1) CBESS:

  • Each CBESS is characterized by charging and discharging efficiencies, power limits, state-of-charge bounds, active and reactive power, stored energy, and capacity.The model defines battery dynamics, SOC and charging-rate boundaries, charging/discharging exclusivity, loading limits, and daily storage initialization.Source: 1) CBESS · p. 4
  • The CBESS equations specify how stored energy evolves and constrain its operating range, charge/discharge rates, exclusivity, inverter loading, and starting storage level.The initial stored-energy level is generally set to 50% of SOC.Source: 1) CBESS · p. 4

2) High DER-Penetration Buses:

The model identifies DER-dense buses, represents PV inverter control and distribution-network power flows, and sets economic and network-safety objectives for CBESS coordination.

2) High DER-Penetration Buses:

  • High-DER buses are characterized by significant rooftop PV and residential EV charging, with time-varying load and available PV profiles; other buses use standard loads without PV or EV charging.The model concentrates these buses around CBESS-connected locations to form locally intensive DER regions.Source: 2) High DER-Penetration Buses · p. 4

3) PV Inverters:

  • PV inverters can curtail rooftop output and provide reactive-power support, while the radial network is represented with Baran-Wu DistFlow power-flow relationships.The network model includes bus power balances and branch-flow and voltage relations; voltage magnitudes and branch loads must remain within their limits.Source: 3) PV Inverters · p. 4
  • The physical network constraints require bus voltages to remain within admissible bounds and branch apparent loading to remain within capacity limits.Line resistance and reactance enter the voltage and flow relationships.Source: 3) PV Inverters · p. 4

B. Objective

  • The DSO seeks a causal CBESS policy that minimizes expected daily economic cost while keeping expected voltage-violation degree, voltage-violation count, and line-overload degree within specified tolerances.The objective accounts for energy and battery-degradation costs, with network-level constraints coupled across buses through power flows.Source: B. Objective · p. 5

III. METHODOLOGY

The framework combines a physics-informed spatiotemporal graph representation with constraint-specific LDCPPO for DSO-coordinated community battery control. Its graph encoder uses windowed bus features and prices to represent distribution-network dynamics.

III. METHODOLOGY

  • LDCPPO uses independent reward and constraint-cost critics, combining their advantages during a Lagrangian policy update with adaptive multipliers.The CMDP represents voltage and line-loading violations as separate operational-constraint costs.Source: §III · p. 5

A. Dual-Level Physics-Informed Spatiotemporal Representation

  • PISTG integrates line-coupling-aware graph propagation with physics-regularized latent learning to represent distribution-network spatiotemporal dependencies.Its physics-informed adjacency shapes electrical message passing, while KCL- and voltage-drop-related losses regularize learned representations.Source: A. Dual-Level Physics-Informed… · p. 5

1) Data Structure Construction:

  • The graph input stacks a window of bus features, including active/reactive load, PV generation, voltage, and CBESS energy, with prices.Bus-level variables are assumed to come from pseudo-measurements, while feeder topology and line parameters come from the DSO network model.Source: 1) Data Structure Construction · pp. 5–6

2) PI Adjacency Matrix:

The encoder uses line impedance and capacity to weight electrical coupling, then processes bus-level information spatially and combines network and price sequences temporally. This yields a joint embedding of network state and market signals.

2) PI Adjacency Matrix:

  • PI adjacency weights encode relative electrical coupling strength using a physically derived score based on line impedance and capacity.The score increases for lower-impedance and/or higher-capacity lines, and a sigmoid maps it to a bounded edge weight.Source: 2) PI Adjacency Matrix · p. 6

3) Spatial Processing:

  • GCN layers propagate bus signals through PI-weighted electrical neighbors, and attention pooling gives more critical buses greater influence in the graph summary.The learned attention weights determine which buses contribute more to the summarized spatial embedding.Source: 3) Spatial Processing · p. 6

4) Temporal Processing:

  • TCNs process the graph-level sequence and a separate price sequence; their outputs form a joint embedding of network state and market signals.Causal and dilated convolutions capture short- and long-term temporal correlations without recurrence.Source: 4) Temporal Processing · p. 7

5) Physics-Regularized Latent Representation:

A reconstruction head and physics-consistency losses regularize PISTG embeddings toward power-system consistency. These training-time priors and losses are not a hard power-flow feasibility layer during policy execution.

5) Physics-Regularized Latent Representation:

  • The physics-informed auto-encoder adds a reconstruction head that predicts branch active/reactive flows and bus voltages from the encoder.Its reconstruction loss penalizes differences between predicted and true voltages and branch flows.Source: 5) Physics-Regularized Latent… · p. 7
  • The active-power balance loss guides the encoder to follow KCL by aligning reconstructed flows with nodal injections and consumptions.The line-level voltage-consistency term constrains voltage differences across connected buses in line with Ohm’s law.Source: 5) Physics-Regularized Latent… · p. 7
  • The PI adjacency and physics-informed losses provide structural priors and soft training regularization, not hard power-flow feasibility during policy execution.Source: 5) Physics-Regularized Latent… · p. 7

B. CMDP Modeling for Problem

The paper formulates DSO-coordinated CBESS operation as a CMDP that balances operating cost against separate voltage- and line-loading-related constraint costs. The state, actions, reward, and costs represent network/device conditions, control decisions, economics, and operational violations.

B. CMDP Modeling for Problem

1) CMDP Elements:

  • The state encodes network and device variables, while actions include CBESS active/reactive power, PV curtailment, and inverter reactive power.The state includes demands, PV generation, voltages, state of charge, and market prices.Source: 1) CMDP Elements · p. 8
  • The reward combines wholesale-price energy cost and battery degradation cost based on charging and discharging throughput.Source: 1) CMDP Elements · p. 8
  • Separate costs measure voltage-violation degree, voltage-violation count, and line-overload degree.Voltage degree measures aggregated deviation magnitude, while voltage count measures violating buses; line-overload cost measures apparent-power exceedance.Source: 1) CMDP Elements · p. 8

2) CMDP Formulation:

The CMDP formulation treats voltage and line-loading limits as expected discounted-cost budgets. LDCPPO keeps reward and constraint-value estimates separate, combining their advantages for policy updates.

2) CMDP Formulation:

  • The formulation distinguishes economic reward from operational-constraint costs and discounts their returns over time.The stated notation defines reward as economic-oriented and costs as operational constraints.Source: 2) CMDP Formulation · p. 8
  • Expected discounted-cost budgets regulate each constraint’s Lagrangian multiplier and penalize budget exceedance; safe RL here does not guarantee pointwise feasibility.The paper defines safe RL as CMDP-based learning under expected discounted-cost budgets.Source: 2) CMDP Formulation · p. 8

1) Independent Reward and Cost Critics:

2) Actor Network:

The actor uses a PPO clipped-surrogate update with a Lagrangian advantage. Parameter updates combine gradient descent for the actor and critics with projected dual ascent, in a five-minute battery-control simulation.

2) Actor Network:

  • The actor update maximizes the PPO clipped surrogate using a Lagrangian advantage that combines reward and cost generalized advantages with Lagrange multipliers.The resulting policy update accounts for the reward and constraint costs.Source: 2) Actor Network · p. 9

3) Parameter Update:

  • Critics and actor use gradient descent, while projected dual ascent adapts nonnegative multipliers according to deviations from expected constraint-cost budgets.The minibatch mean discounted cost return is used in the multiplier update.Source: 3) Parameter Update · p. 9

1) Simulator Setup:

  • The IEEE 33-bus simulation uses four 1-MWh batteries and 288 five-minute steps per day, with faster variations left to local Volt–VAR control and protection.The units are at buses 6, 11, 16, and 31; PV and load profiles use interpolated Ausgrid data with nighttime EV charging from 8 PM to 6 AM.Source: 1) Simulator Setup · p. 9
  • Training spans 2,000 daily episodes, yielding about 5.76 × 10^5 transitions and 5.54 × 10^5 graph-temporal windows.The encoder uses a rolling window length of 12.Source: 1) Simulator Setup · p. 10

2) Comparison Baselines & Computational details:

The evaluation compares the proposed method with classical DRL, constrained DRL, and physics-informed baselines. On the IEEE 33-bus system, PISTG-LDCPPO reports lower violations and operating cost than all tested baselines, with tighter test-state distributions and a more grid-aware battery dispatch.

2) Comparison Baselines & Computational details:

  • The comparison includes seven DRL baselines; PPO, TD3, and SAC aggregate constraint costs as reward penalties, while several alternatives use Lagrangian constraint handling.The baselines include LDCPPO, STG-LDCPPO, PISTG-CPO, and PI-GR-STG-LDCPPO.Source: 2) Comparison Baselines &… · p. 10

1) Overall Performance:

  • Against all baselines, PISTG-LDCPPO reduces voltage-violation degree 49.0%–80.5%, voltage-violation count 28.8%–57.5%, line-overload degree 0.9%–26.6%, and operating cost 2.1%–4.9%.The test comparison covers 100 episodes for five runs.Source: Table III · p. 10, 1) Overall Performance · p. 11

2) Analysis of Voltage and Load:

  • PISTG-LDCPPO yields the most compact voltage and line-loading distributions, with fewer extreme values and reduced variability across buses and lines.The voltage and loading plots mark operational limits and highlight violating ranges.Source: Figure 7 · p. 10, 2) Analysis of Voltage and Load · p. 11, Figure 8 · p. 11
  • Across 100 test episodes, PISTG-LDCPPO keeps voltage extremes closer to the admissible band and reduces maximum line-loading violations and fluctuations.These system-level trajectories track minimum and maximum bus voltage and maximum line loading.Source: 2) Analysis of Voltage and Load · p. 11, Figure 9 · p. 11
  • Compared with PPO, PISTG-LDCPPO charges more moderately during daytime PV peaks and discharges more proactively during evening peaks, with a slight economic-performance sacrifice.The SOC trajectories cover four CBESS units and are shown alongside PV and load profiles.Source: 2) Analysis of Voltage and Load · p. 11, Figure 10 · p. 11

3) Scalability Analysis on the IEEE 69-Bus System:

The framework is evaluated on a larger IEEE 69-bus system, on PV curtailment, and through ablations of its graph encoder and critic design. Results report maintained constraint-aware performance at larger scale and show complementary benefits from the physics-informed representation and separate critics.

3) Scalability Analysis on the IEEE 69-Bus System:

  • PISTG-LDCPPO maintains lower empirical network-constraint violations than baseline DRL and safe-DRL methods on the IEEE 69-bus system.The test comparison is reported for the larger system in Table IV.Source: Table IV · p. 12
  • Parameter count increases from 0.47 M to 0.53 M on the 69-bus system; the authors characterize this increase as negligible relative to the 5-min control interval.Source: 3) Scalability Analysis on the IEEE… · p. 12

4) PV Curtailment Analysis:

  • PISTG-LDCPPO maintains low PV curtailment on both test systems while achieving the lowest voltage- and line-related violations, balancing PV use and network security.Some baselines have slightly lower curtailment in individual cases but substantially higher network-constraint violations.Source: 4) PV Curtailment Analysis · p. 12, Table VI · p. 12

5) Ablation of PISTG and Constraint-Specific Critics:

  • The full PISTG encoder has the best reconstruction accuracy and physics consistency; removing PI graph propagation, KCL loss, or voltage-drop loss degrades corresponding properties.Compared with STG-LDCPPO, PISTG-LDCPPO reduces voltage-violation degree by 52.8% and count by 41.3%.Source: 5) Ablation of PISTG and… · pp. 12–13, Table VIII · p. 13
  • With the encoder, actor, Lagrangian update, and protocol held constant, separate constraint critics reduce empirical violations—especially voltage degree and count—relative to one aggregated cost critic.The comparison uses PISTG-LDCPPO and PISTG-LSCPPO on both systems, with comparable operating costs.Source: 5) Ablation of PISTG and… · p. 13, Figure 13 · p. 13

6) Sensitivity Analysis of PIAM:

The full physics-informed adjacency performs best among the tested edge-weight constructions, and PIAM remains robust across tested capacity-weight coefficients. The learned embedding is more strongly correlated with voltage-related stress than the ordinary STG embedding, while future work targets broader deployment settings.

6) Sensitivity Analysis of PIAM:

7) Physical Correlation Analysis of Learned Embeddings:

  • The physical-correlation analysis probes the encoder's latent vector against voltage deviation, voltage magnitude, and line overload on testing trajectories.The linear probes are interpretive and are not involved in policy training or control execution.Source: 7) Physical Correlation Analysis of… · pp. 13–14
  • The PISTG embedding correlates more strongly with voltage-related physical stress than the ordinary STG embedding.The results also suggest better alignment with power-flow-related stress indicators, especially voltage stress.Source: Table X · p. 13, 7) Physical Correlation Analysis of… · p. 14

V. CONCLUSION AND FUTURE WORK

  • Experiments substantially reduce empirical voltage-limit and line-loading violations while maintaining competitive operating cost against representative DRL and constrained-RL baselines in high-DER scenarios.Source: §V · p. 14
  • Future work targets three-phase unbalanced and reconfigurable networks, limited-communication multi-agent coordination, shorter decision intervals, and noisy, delayed, or incomplete measurements.Source: §V · p. 14
Loading 2610.33654v1…