PaperlayerView exact revision on arXiv

Source-linked AI summary

Converter-Grid Interaction Stability Guaranteed Safe Deep Reinforcement Learning for Energy Storage Systems in Grid Frequency Support

Fei Liu, Mengfan Zhang, Zhipeng Li, Frede Blaabjerg, Qianwen Xu

arXiv:2609.16817v1eess.SYcs.SY

TL;DR

Converter-interfaced resources raise converter-grid interaction stability concerns, while ESS can provide frequency support. The paper combines a learned stability region with Safe-TD3 action projection to keep commands in the stable set before execution. Case studies report improved frequency regulation while preventing unstable operating points.

  • Problem

    Converter-interfaced resources bring fast control interactions with grid impedance and network dynamics that can create converter-grid interaction stability issues.

    Source: §I · p. 1

  • Method

    CIS-DRL uses a double-DNN stability-region evaluation and projects unsafe Safe-TD3 actions into the stable set before execution.

    Source: §I · p. 2, C. Converter-grid interaction… · p. 4

  • Results

    Case studies report improved frequency regulation while preventing unstable operating points, with ESS current and power trajectories remaining within feasible limits.

    Source: §V · pp. 9–10

  • Takeaways & Limitations

    The results indicate potential for practical ESS frequency support in converter-dominated grids under uncertainty and varying operating conditions.

    Source: §V · p. 10

Abstract

from arXiv · show

The growing integration of converter interfaced renewable energy resources (RESs) intensifies stability challenges. Energy storage system (ESS) can provide fast and flexible frequency support to mitigate frequency deviations. However, the interface converter of ESS may encounter converter-grid interaction stability issues. This paper proposes a converter-grid interaction stability guaranteed safe DRL (CIS-DRL) method for ESS integrated power systems to achieve frequency regulation. We first obtain a double DNN-based stability region to identify the guaranteed converter-grid interaction stability. Next, a novel converter-grid interaction stability Safe-TD3 (CIS-STD3) algorithm is designed that integrates a stability feasibility projection layer to map unsafe actions into stable action set before execution, enforcing converter-grid interaction stability as a hard constraint throughout learning process. The proposed approach enables ESS for grid frequency support with 100% converter-grid interaction stability without violations. Experimental results show that the proposed CIS-DRL method achieves improved frequency regulation performance while preventing unstable operating points, demonstrating its practical applicability for real time ESS frequency support.

I. INTRODUCTION

Converter-interfaced resources introduce fast control interactions with grid impedance and network dynamics, while existing frequency-regulation methods rarely model or enforce converter–grid interaction stability. The paper proposes stability-constrained ESS frequency regulation, addressing a nonlinear stability boundary that conventional online optimization and soft penalties cannot reliably enforce.

I. INTRODUCTION

  • Fast converter controls can couple with grid impedance and network dynamics, creating converter–grid interaction stability issues across wide frequency ranges.The paper situates these issues alongside the growing integration of renewable resources and energy storage and reduced synchronous inertia.Source: §I · p. 1
  • Prior ESS frequency-control studies focus on frequency performance and operational boundaries, while converter–grid interaction stability is rarely modeled and enforced during operation.Source: §I · p. 1
  • CIS-DRL combines a double-DNN stability-region estimate with a CIS projection layer in Safe-TD3 to map unstable actions into the stable set before execution.The method is intended to support real-time monitoring without detailed converter inner-loop information and to maintain stability across wide operating-point variations.Source: §I · p. 2

II. PROBLEM FORMULATION

  • ESS frequency regulation is formulated with converter operational limits, but those limits omit converter–grid interaction stability and may permit actions that enter unstable regions.The stated limits include charging and discharging power, SOC, converter current, and voltage bounds.Source: §II · p. 2
  • The stability boundary is nonlinear and nonconvex, making iterative online feasibility checks difficult, while a soft-penalty relaxation cannot guarantee stability.Source: §II · pp. 2–3

III. CIS-DRL BASED FREQUENCY REGULATION WITH CONVERTER-GRID INTERACTION STABILITY REGION INTEGRATED

The section presents CIS-DRL as a frequency-support method that evaluates a converter-grid stability region and embeds projection into Safe-TD3. Unsafe actions are projected into the stable action set before execution.

III. CIS-DRL BASED FREQUENCY REGULATION WITH CONVERTER-GRID INTERACTION STABILITY REGION INTEGRATED

  • CIS-DRL is designed to provide ESS frequency support with guaranteed converter-grid interaction stability.Source: §III · p. 3
  • The method evaluates a stability region to determine whether candidate ESS operating points are stable.Source: §III · p. 3
  • A CIS projection layer is integrated into Safe-TD3 to project stability-violating actions into the stable action set before execution.Source: §III · p. 3

A. Converter-grid interaction stability region generation

The stability-region method represents the converter operating point in a synchronous dq frame and characterizes stability with an operating-point-dependent margin. It uses that margin to define a stability-admissible region and corresponding action set, despite the unavailable analytical margin expression.

A. Converter-grid interaction stability region generation

  • The PCC operating point is represented in the synchronous dq frame as steady-state quantities for stability assessment.The description identifies voltage and current components in that frame; with PLL alignment, vq is regulated toward zero.Source: A. Converter-grid interaction… · p. 3
  • The stability margin defines a boundary at zero, with a larger positive margin indicating greater distance from that boundary.Source: A. Converter-grid interaction… · p. 3
  • The operating-point stability-admissible region induces a stability action set in the power-command space.Source: A. Converter-grid interaction… · p. 3
  • The stability action set is operating-point dependent and generally nonlinear and nonconvex because it combines interaction dynamics and time-varying PCC voltage.Source: A. Converter-grid interaction… · p. 3
  • The analytical stability-margin map is typically unavailable because converter internal control details are limited, so the method treats it as a black-box map built from measurement-based assessments.The admissible decision set combines this assessment with the operational feasibility set.Source: A. Converter-grid interaction… · p. 3

B. Environment formulation for converter-grid interaction stability guaranteed frequency regulation

The environment models frequency regulation as a constrained Markov decision process in which ESS commands must satisfy hard constraints at every step. Because the stability-margin evaluation is black-box and uncertain, the controller is learned model-free through simulator interaction.

B. Environment formulation for converter-grid interaction stability guaranteed frequency regulation

  • Frequency regulation is formulated as a constrained Markov decision process, with the controller observing a state, issuing an ESS command, and receiving a next state and scalar reward.Source: B. Environment formulation for… · p. 3
  • ESS power commands must satisfy hard constraints at every time step.Source: B. Environment formulation for… · p. 3
  • Each time step represents a load-disturbance scenario, with the short-term frequency response simulated to calculate the reward and next state.Source: B. Environment formulation for… · p. 3
  • The transition probability is unavailable in closed form because of modeling uncertainty and the black-box stability-margin evaluation, motivating a model-free approach.The policy is learned through interactions with a simulator-based environment.Source: B. Environment formulation for… · p. 3

1) State space:

  • The state includes variables relevant to frequency dynamics, ESS operation, and converter-grid interaction stability assessment.Source: 1) State space · p. 3
  • The state includes power imbalance and the estimated stability margin at the current operating point.Source: 1) State space · p. 3
  • The state supports coordination of active power for frequency regulation and reactive power for voltage support while accounting for proximity to the stability boundary.Source: 1) State space · p. 3

2) Action space:

  • The agent's action is the ESS's active and reactive power commands, bounded by converter rating and power limits, including reactive-power capability.Source: 2) Action space · p. 4

3) Reward function:

The section describes a stability region learned from impedance and stability assessments, a projection-based CIS constraint, and a reward based on frequency deviation.

3) Reward function:

  • The reward uses next-step frequency deviation, and maximizing cumulative discounted reward drives the agent to suppress frequency deviation over the control horizon.Source: 3) Reward function · p. 4

C. Converter-grid interaction stability constraint

  • The CIS-DRL controller integrates a projection layer into Safe-TD3 to correct tentative actions into the stability-feasible set and target 100% interaction stability without violations.Source: C. Converter-grid interaction… · p. 4

1) Double DNN-based stability region evaluation:

  • DNN1 approximates dq-axis impedance from multi-operating-point measurements, while DNN2 learns stable classifications assigned using the Nyquist criterion and defines a confidence-thresholded safe set.The safe set describes stability in (vd, id, iq) space and supplies the stability constraint.Source: 1) Double DNN-based stability region… · p. 4

2) CIS projection layer generation :

The CIS projection layer seeks a nearby stable operating point when an action would lead outside the learned safe set. It combines an outer radius search with an inner feasibility search.

2) CIS projection layer generation :

  • The projection seeks the closest stable operating point in Sτ to an unstable tentative point, but the black-box DNN2 classifier prevents an analytical solution.Source: 2) CIS projection layer generation · p. 5
  • An outer bisection search tests midpoint radii, updating its bounds according to whether the inner search finds a stable point at that distance.The search stops when the radius interval is below a chosen tolerance.Source: 2) CIS projection layer generation · p. 5
  • The inner search tests candidate points along directions toward known stable points and random unit directions, then redirects unstable tentative actions into Sτ.Source: 2) CIS projection layer generation · p. 5

D. Proposed CIS-DRL method for frequency regulation

The method combines a simulator-based ESS frequency-control environment with a learned converter-grid stability region and an online projection layer. Safe-TD3 then learns from transitions generated using admissible actions.

D. Proposed CIS-DRL method for frequency regulation

  • The simulator returns state, reward, and next state after each admissible action, and the actor–critic learner follows TD3 with two critics and one actor.The target networks are softly updated; the soft-update rate is reduced later in training for more stable targets.Source: D. Proposed CIS-DRL method for… · pp. 5–6
  • The stability region is trained from impedance measurements and frequency-domain Nyquist assessments, then used as an online feasibility query for the controller.Source: D. Proposed CIS-DRL method for… · p. 5
  • The CIS projection corrects noisy tentative actions before execution, so each executed command satisfies stability feasibility and operational limits.Source: D. Proposed CIS-DRL method for… · p. 6

IV. CASE STUDY AND VALIDATIONS

The case study evaluates a VSC-based ESS integrated grid through stability-region analysis, online feasibility enforcement, frequency-regulation comparisons, and hardware experiments.

IV. CASE STUDY AND VALIDATIONS

  • The tested VSC connects to the grid through converter-side and grid-side inductors, with the PCC between them for voltage and current measurements.Source: §IV · p. 6
  • The validation compares frequency regulation with a baseline, no ESS, and ESS without stability consideration, tracking frequency deviation, ESS power, SOC, and converter current.Source: §IV · p. 6
  • Hardware experiments use an imperix controller, a programmable DC source, and a programmable grid simulator.Source: §IV · p. 7
  • The learned stability region maps stable and unstable operating points in (vd, id, iq) space, with orange dots marking stable points and blue dots unstable points.Source: Figure 4 · p. 7

A. Converter-grid interaction stability region evaluation and projection results

The learned stability region separates stable from unstable converter operating points, and the projection redirects inadmissible points into that region. Voltage-slice results show the projection moving points toward the boundary along the shortest direction in the current plane.

A. Converter-grid interaction stability region evaluation and projection results

  • The learned region distinguishes stable from unstable operating points in the (vd, id, iq) space.Fig. 4 uses orange dots for stable points and blue dots for unstable points.Source: A. Converter-grid interaction… · p. 7
  • Inadmissible tentative points are redirected into the learned stability region before execution, and the double-DNN model provides reliable stability-region evaluation.Fig. 5 shows original and projected operating points alongside the three-dimensional region.Source: A. Converter-grid interaction… · p. 7, Figure 5 · p. 7
  • In four voltage slices, projection moves points along the shortest direction toward the boundary in the (id, iq) plane.The slices correspond to vd = 159.4 V, 111.6 V, 171.5 V, and 151.5 V.Source: A. Converter-grid interaction… · p. 7, Figure 6 · p. 7
  • 99.64% final validation accuracy was reached using 18,081 labelled samples to train and validate the two DNNs.Fig. 7 presents DNN2 validation ROC and PR curves; the trained networks are used online for fast stability evaluation.Source: A. Converter-grid interaction… · p. 8, Figure 7 · p. 8
  • Fig. 8 tracks maximum and mean frequency deviation during the training process.Source: Figure 8 · p. 8

B. Converter-grid interaction stability guaranteed frequency regulation

The section evaluates CIS-projected ESS frequency support against unprojected ESS and a no-ESS baseline. Projection slightly sacrifices maximum frequency-deviation performance relative to unprojected ESS while keeping actions stable and converter currents within a tighter envelope.

B. Converter-grid interaction stability guaranteed frequency regulation

  • CIS-STD3 applies its projection layer to every tentative action before environment execution, using one actor and two critics in a twin-delayed structure.The actor is a three-layer fully connected network with 400 and 300 hidden neurons, layer normalization, ReLU activation, and a tanh output.Source: B. Converter-grid interaction… · p. 8
  • During training, maximum and mean frequency deviations trend downward, while cumulative reward rises from strongly negative values toward a plateau close to zero.The paper interprets the deviation trend as progressively improved damping of load-disturbance frequency excursions.Source: B. Converter-grid interaction… · p. 8, Figure 9 · p. 8
  • Maximum frequency deviation was 0.7338 Hz without ESS, 0.4017 Hz with unprojected ESS, and 0.4371 Hz with CIS projection.Fig. 10 compares the three cases over 50 frequency-regulation steps.Source: B. Converter-grid interaction… · p. 8, Figure 10 · p. 9
  • Unprojected ESS produced the smallest maximum deviation but entered the unstable region; CIS projection slightly sacrifices frequency performance to keep every ESS action stable.The paper identifies the unprojected policy as impractical for real deployments.Source: B. Converter-grid interaction… · p. 9
  • With CIS projection, charging and discharging are moderated, SOC evolves more smoothly, and d- and q-axis currents remain within a tighter envelope.The current profiles remain smoother while still providing effective power support.Source: B. Converter-grid interaction… · p. 9, Figure 12 · p. 9

V. CONCLUSION

The conclusion presents a safe DRL frequency-regulation method that projects unsafe ESS commands into a learned converter-grid interaction stability region. Case studies and experiments report frequency-regulation improvement while preventing unstable operating points and keeping ESS current and power trajectories within feasible limits.

V. CONCLUSION

  • CIS-DRL uses a learned stability region and projection to correct infeasible commands before execution, targeting stepwise stability while minimizing frequency deviation.The stated scope includes ESS-integrated systems with incomplete converter information and wide operating-point variations.Source: §V · p. 9
  • Case studies report improved frequency regulation and prevention of unstable operating points, with ESS current and power trajectories remaining within feasible limits.The conclusion frames experimental results as indicating potential for practical deployment under uncertainty and varying operating conditions.Source: §V · pp. 9–10
  • Fig. 13 compares conventional and proposed methods at four measured grid-voltage values: 159.4, 111.6, 171.5, and 151.5 V.The figure caption identifies paired conventional and proposed-method panels for each value.Source: Figure 13 · p. 10
Loading 2609.16817v1…