PaperlayerView exact revision on arXiv

Source-linked AI summary

Risk and Anomaly Identification for Distribution Network Optimal Operation Based on Reinforcement Learning and Uncertainty Quantification

Ziqi Zhang

arXiv:2609.03308v1cs.LGcs.MAcs.SYeess.SY

TL;DR

Distribution-network operators need to distinguish inherent stochastic operating risk from anomalies associated with out-of-distribution conditions. The paper proposes second-order uncertainty quantification within distributional reinforcement learning, using epistemic estimates for exploration and fallback control and aleatoric estimates for intrinsic risk. Simulations report improved learning-based reward and constraint-cost performance, as well as substantial gap removal with epistemic-triggered fallback.

  • Problem

    Distribution-network operation must distinguish inherent stochastic risk from OOD behavior, while limited training data and high-dimensional scenarios complicate anomaly identification.

    Source: §I · p. 1

  • Method

    The framework applies second-order UQ to return distributions, decomposes uncertainty into aleatoric and epistemic components, and uses epistemic estimates to gate MISOCP fallback control.

    Source: §I · p. 2

  • Results

    The proposed method achieved the best final reward and lowest long-term constraint cost among the learning-based baselines; at ϵmiss = 0.10, fallback removed 94.0% of the critical constraint-cost gap and 90.7% of the reward gap.

    Source: 2) EU Evaluation via Training… · p. 9, 4) Closed-Loop Evaluation With… · p. 12

  • Takeaways & Limitations

    The simulations support using decomposed uncertainty for characterizing operational risk and triggering fallback control to improve constraint compliance across operating conditions.

    Source: §V · p. 12

Abstract

from arXiv · show

Reliable operation of modern distribution networks requires timely identification of operational risks and anomalous events under pervasive uncertainty. In practice, operators must identify risks that are inherent in stochastic yet in-distribution conditions, and anomalies that correspond to out-of-distribution behaviors such as unusual load patterns, extreme weather or cyber-physical attacks. This paper addresses this joint risk and anomaly identification problem for optimal distribution network operation and proposes a deep reinforcement learning framework that is explicitly uncertainty aware. We integrate distributional and Bayesian deep reinforcement learning to realize a second- order uncertainty quantification scheme that decomposes total uncertainty into aleatoric and epistemic components, which are respectively used to characterize inherent risk and out-of- distribution anomalies. The resulting epistemic estimates drive both exploration during training and out-of-distribution detec- tion with fallback control during deployment, whereas aleatoric estimates are used to characterize intrinsic operational risk. Simulation results demonstrate the performance of our DRL agent and the effectiveness of the uncertainty quantification.

I. INTRODUCTION

The paper frames distribution-network risk and anomaly identification as uncertainty quantification that separates inherent stochasticity from uncertainty about the model or operating regime. It proposes a distributional, second-order UQ framework with epistemic-triggered fallback and places operation in a constrained Markov decision process.

I. INTRODUCTION

  • Distribution-network operation combines irreducible stochasticity with OOD shifts, while neural DRL agents may generalize poorly and prior approaches often aggregate uncertainty or address only part of the control problem.The cited settings include atypical loads, unusual renewable behavior, infrastructure deterioration, limited training data, and adversarial perturbations.Source: §I · pp. 1–2
  • The framework casts risk and anomaly identification as return-law uncertainty quantification, separating aleatoric and epistemic uncertainty through second-order UQ.It develops critic-side distance-based estimates using one-dimensional quantile representations and a Wasserstein barycenter, with equivalence and Monte Carlo consistency established for the tractable forms.Source: §I · p. 2
  • The framework uses epistemic unreliability as an online signal to switch from fast DRL inference to a conservative MISOCP-based OPF controller.The paper presents this as an operational fallback under distribution shift, rather than using epistemic uncertainty only as an anomaly score.Source: §I · p. 2

II. OPTIMAL DN OPERATION FORMULATION

  • The paper first formulates distribution-network operation as a CMDP and specifies the physical model and operating constraints.Its preliminaries describe a discounted CMDP with states, actions, transitions, reward, constraint costs, and a discount factor.Source: §II · p. 2, A. Preliminaries on CMDP · p. 2

A. Preliminaries on CMDP

  • Lagrangian relaxation aggregates operational limits into a scalar cost and uses a multiplier to define the reward for the distributional return model.The return variable follows the distributional Bellman equation under the Lagrangian reward.Source: A. Preliminaries on CMDP · pp. 2–3

B. Physical model and operational constraints

The physical model instantiates the CMDP for a radial distribution network by defining its states, actions, operating objective, and device representation. The stage reward is the negative total operating cost.

B. Physical model and operational constraints

  • The CMDP is instantiated with a radial distribution-network model that specifies state and action spaces, system transitions, operating cost, and constraints.The state includes bus voltages and angles, generator outputs, storage charge, tap and capacitor status, and load and renewable forecasts; actions aggregate device setpoints and switching or tap commands.Source: B. Physical model and operational… · p. 3, 1) States and actions · p. 3

2) Operating cost and CMDP reward:

  • The stage reward is the negative total operating cost, including upstream-grid energy purchases, local generation costs, and network losses.The loss term is converted into monetary units using a loss-weighting coefficient.Source: 2) Operating cost and CMDP reward · p. 3

3) Operational constraints:

Operational constraints define device feasibility and network security in the CMDP. The network model also uses AC power-flow equations to relate nodal injections.

3) Operational constraints:

  • Voltage and line thermal limits are represented as operational-security inequalities, alongside device ratings and operating constraints.The passage also identifies storage state-of-charge and charging or discharging limits, generator bounds, capacitor steps, and discrete tap positions.Source: 3) Operational constraints · p. 3

4) Network constraints and AC power flow:

  • Nodal injections satisfy AC power-flow equations, with the slack-bus injection including imported grid power.Bus-indexed device variables are treated as aggregate injections and set to zero when a bus hosts no such device.Source: 4) Network constraints and AC power flow · p. 3

5) Scalar constraint cost:

The framework combines a distributional actor–critic, second-order uncertainty quantification, and uncertainty-aware training and deployment. Its scalar constraint cost aggregates residual network-security violations for constrained operation.

5) Scalar constraint cost:

  • The scalar constraint cost combines voltage violations, thermal-loading violations, and AC power-flow non-convergence using penalty weights.Voltage violations are measured in p.u., thermal-loading violations in percentage points, and non-convergence is indicated by a binary term.Source: 5) Scalar constraint cost · pp. 3–4

A. Framework Overview

  • The framework uses an IQN-based distributional critic to estimate Lagrangian return distributions while handling CMDP performance and constraints.Its three components are a distributional actor–critic backbone, a second-order UQ module, and uncertainty-aware training and deployment.Source: A. Framework Overview · p. 4
  • Critic uncertainty is decomposed into aleatoric and epistemic components; epistemic uncertainty guides exploration and can trigger fallback control at deployment.Aleatoric uncertainty represents irreducible stochasticity, while epistemic uncertainty reflects limited data or distributional shift.Source: A. Framework Overview · p. 4

B. Distributional Actor–Critic Backbone

  • The deterministic actor selects actions while an IQN critic estimates quantiles of discounted Lagrangian returns for state–action pairs.The backbone operates on the Lagrangian MDP induced by the CMDP.Source: B. Distributional Actor–Critic Backbone · p. 4

C. Bayesian Distributional Critic

The Bayesian distributional critic generates multiple stochastic return-quantile predictions using dropout and ensembles. These predictions provide return-law samples and are regularized to preserve epistemic spread.

C. Bayesian Distributional Critic

  • Monte Carlo dropout perturbs one critic around one approximate posterior mode, whereas independently trained deep ensembles approximate a multimodal posterior.The critic networks are treated as approximate Bayesian neural networks.Source: C. Bayesian Distributional Critic · p. 5
  • Each critic maps a state–action pair and quantile fraction to a stochastic approximation of the Lagrangian return quantile function.Dropout masks generate stochastic predictions for each critic member.Source: C. Bayesian Distributional Critic · p. 5
  • A repulsive diversity regularizer discourages ensemble collapse, and the module flattens its MK stochastic quantile functions into return-law samples.The regularizer uses critic members’ mean return estimates.Source: C. Bayesian Distributional Critic · p. 5

D. Distance-based Second-order UQ

The distance-based second-order UQ framework represents uncertainty about predictive distributions and quantifies it by distances to reference families. Its reference families encode states with no uncertainty of a specified type.

D. Distance-based Second-order UQ

  • A second-order distribution is a distribution over first-order predictive distributions, and the framework compares these using a Wasserstein–p metric induced by a metric on first-order distributions.The construction uses p ≥ 1.Source: 1) General distance-based second-order… · p. 5
  • The distance-based construction quantifies total, aleatoric, and epistemic uncertainty by distance to reference families representing least-uncertain states of each type.The Wasserstein distance is the minimal transport cost between second-order distributions, with first-order distance defining the cost of moving mass.Source: 1) General distance-based second-order… · p. 5
  • The total-certainty reference family consists of second-order Dirac masses at first-order Dirac measures, representing certainty about both outcome and predictive distribution.The zero-aleatoric reference family is supported on deterministic predictors.Source: 1) General distance-based second-order… · p. 5
  • The zero-aleatoric family contains second-order distributions supported on deterministic predictors, while the zero-epistemic family uses second-order Dirac measures on arbitrary first-order distributions.Distance from the zero-aleatoric family measures how far a predictive distribution is from having deterministic first-order predictors.Source: 1) General distance-based second-order… · p. 5

E. Uncertainty-aware Exploration

Epistemic uncertainty supports exploration during training and threshold-triggered fallback control at deployment. The study also tests penalty-weight sensitivity and evaluates OOD scenarios spanning observation attacks, profile shifts, frequency-domain changes, and external data.

E. Uncertainty-aware Exploration

  • Exploration scores feasible candidate actions using critic mean return and epistemic uncertainty, with an adaptive bonus that shifts from broader exploration toward exploitation during training.The bonus fraction is gradually reduced as training proceeds.Source: E. Uncertainty-aware Exploration · p. 6

F. Uncertainty-guided fallback control

  • At deployment, the agent hands control to a single-step MISOCP-based OPF fallback when the actor’s epistemic uncertainty exceeds its calibrated threshold.The fallback minimizes instantaneous operating cost while enforcing modeled single-step constraints.Source: F. Uncertainty-guided fallback control · p. 7

1) Test Cases and Network Adaptation:

  • Across tested penalty ratios, both violation terms remain small; larger voltage penalties reduce voltage violations, while larger loading penalties reduce thermal violations and slightly raise reward.The study varies the relative voltage/loading penalty ratio while keeping their overall scale fixed.Source: Table I · p. 7, 1) Test Cases and Network Adaptation · p. 7

2) OOD Scenario Construction:

  • OOD tests cover observation noise and FDIA-style attacks, synthetic profile shifts, frequency-domain perturbations, and hold-out or external real-world trajectories.The observation attacks include white-box and black-box variants; data-driven cases include unseen load, PV, and weather records.Source: 2) OOD Scenario Construction · pp. 7–8

B. Test Case I: Qualitative Demonstration on the Original System

The evaluations test whether AU tracks intrinsic stochasticity and whether EU responds to distribution shifts and supports effective exploration. They compare AU estimation methods and assess learning-based baselines and exploration coverage.

1) AU Evaluation:

2) EU Evaluation:

  • EU is higher than the InD baseline across OOD settings and generally rises over the 24-hour horizon as FDIA attack steps increase, especially for white-box attacks.Black-box attacks also produce increases at several timesteps, with a different temporal pattern.Source: 2) EU Evaluation · p. 8, Figure 2 · p. 9, Figure 3 · p. 9

1) AU Evaluation:

  • In Test Case II, both AU estimators increase monotonically with environmental noise, while the proposed estimator has lower variance across repeated rollouts than the IQN-style baseline.The proposed estimator combines ensemble prediction with the second-order AU definition, retaining sensitivity to environmental stochasticity while suppressing member-specific fluctuation.Source: 1) AU Evaluation · p. 9, Figure 4 · p. 9, Figure 5 · p. 9

2) EU Evaluation via Training Performance:

  • Among learning-based baselines, the proposed method achieves the best final reward and lowest long-term constraint cost in Test Case II.The figure also reports sensitivity tests for the exploration bonus coefficient, ensemble size, and dropout rate.Source: 2) EU Evaluation via Training… · p. 9, Figure 6 · p. 10
  • EU-guided exploration achieves more unique reachable operating-region bins and a higher useful-bin ratio than fixed Gaussian exploration noise.A bin counts as useful only when its transition is feasible, with converged AC power flow and no voltage or loading violation.Source: 2) EU Evaluation via Training… · p. 10

3) EU Evaluation via OOD Detection and Degradation Correlation:

EU is associated with more severe operational degradation and can guide fallback toward higher-risk episodes. The section also evaluates exploration, concludes that the framework couples uncertainty decomposition with fallback control, and develops the one-dimensional Wasserstein results used in its UQ constructions.

3) EU Evaluation via OOD Detection and Degradation Correlation:

  • The proposed UQ exploration achieves the largest number of unique bins and the highest useful-bin ratio, where useful transitions are feasible and satisfy power-flow, voltage, and loading criteria.Fig. 7 plots unique-bin coverage and useful-bin ratio during training.Source: Figure 7 · p. 10
  • As ensemble size decreases, EU-based OOD detection degrades mildly but remains competitive with feature-space baselines; Table II reports AUROC and FPR95, with higher AUROC and lower FPR95 indicating better separation.The compared baselines include KDE, KNN, GMM, and Mahalanobis-distance detectors.Source: 3) EU Evaluation via OOD Detection and… · p. 10, Table II · p. 11
  • Higher EU is associated with larger reward and constraint-cost gaps, and within-model EU percentiles shift upward across increasing degradation-severity bins.The severity analysis normalizes EU to within-model percentiles before pooling models.Source: 3) EU Evaluation via OOD Detection and… · p. 11, Figure 8 · p. 11, Figure 9 · p. 11

4) Closed-Loop Evaluation With EU-Triggered Fallback:

  • At ϵmiss = 0.10, fallback covers 75.8% of held-out episodes, removing 94.0% of the critical constraint-cost gap and 90.7% of the reward gap; at 0.20, coverage falls to 61.5% while cost-gap removal remains 88.0%.Fig. 10 summarizes the fallback-rate/gap-removal tradeoff.Source: Figure 10 · p. 11, 4) Closed-Loop Evaluation With… · p. 12
  • As fallback coverage increases, EU-ranked episodes show lower cost-exceedance probability and retained 95th-percentile constraint cost, indicating that EU prioritizes episodes with greater empirical safety risk.The threshold-free evaluation assigns the highest-EU fraction to MISOCP and leaves the rest under RL control.Source: Figure 11 · p. 12, 4) Closed-Loop Evaluation With… · p. 12

V. CONCLUSION:

  • The conclusion presents second-order UQ as decomposing predictive uncertainty into aleatoric and epistemic components, with EU-triggered fallback improving constraint compliance across diverse operating conditions.The framework couples distributional RL with uncertainty quantification.Source: §V · p. 12

APPENDIX A PROOFS FOR SECOND-ORDER UQ CONSTRUCTIONS:

  • Appendix A uses the first-order 2-Wasserstein distance on return distributions and its induced second-order distance on distributions over those distributions.It derives critic indices from the general distance-based definitions using one-dimensional Wasserstein results.Source: Appendix A · p. 13

A. Auxiliary Results on Wasserstein Distance in One Dimension:

  • The auxiliary one-dimensional results include quantile representations, a Wasserstein barycenter characterization, and a projection-to-Dirac variance identity.These results provide the mathematical tools used in the return-based critic derivations.Source: A. Auxiliary Results on Wasserstein… · p. 13

B. Return-based Critic

The return-based critic derives its epistemic index from a Wasserstein Fréchet variance over return distributions. For the empirical second-order distribution, the resulting quantity is EUcrit.

B. Return-based Critic:

1) Epistemic component as Wasserstein Fréchet variance:

  • The critic's epistemic index is the Wasserstein Fréchet variance of the return-distribution ensemble, with its square preserving the ordering of epistemic uncertainty levels.The empirical barycenter has a one-dimensional closed form through quantile averaging, yielding EUcrit(s, a).Source: 1) Epistemic component as Wasserstein… · pp. 13–14

2) Aleatoric component as expected intrinsic variance:

The aleatoric index reduces to the expected intrinsic variance of the first-order return distributions, providing a closed-form quantity for the critic.

2) Aleatoric component as expected intrinsic variance:

  • The aleatoric index equals the expected intrinsic variance of first-order distributions, obtained by projecting each distribution onto a Dirac predictor.Applying this expression to the empirical critic distribution gives AUcrit(s, a) as the average variance across its draws.Source: 2) Aleatoric component as expected… · p. 14

APPENDIX B CONSISTENCY OF SECOND-ORDER UQ ESTIMATORS

The appendix establishes almost-sure consistency of the Monte Carlo estimators for critic-side epistemic and aleatoric uncertainty under its stated moment conditions.

APPENDIX B CONSISTENCY OF SECOND-ORDER UQ ESTIMATORS

  • The Monte Carlo EU and AU estimators converge almost surely to their population quantities under the stated finite-second-moment assumptions.The proof uses empirical Wasserstein-barycenter consistency for EU and the strong law for AU.Source: Appendix B · pp. 14–15
Loading 2609.03308v1…