PaperlayerView exact revision on arXiv

Source-linked AI summary

RestoreBench: Can AI Agents Restore Power Flow Convergence?

Riccardo Mansutti, Andrea Pomarico, Robert Jakob, Qian Zhang, Alberto Berizzi, Kevin O'Sullivan

arXiv:2609.00384v1cs.AIeess.SY

TL;DR

Non-convergent power-flow cases require iterative diagnosis, simulator interaction, and engineering judgment, but dedicated agentic AI benchmarks have been lacking. The paper introduces RestoreBench to evaluate seven LLMs across chatbot, single-agent, and multi-agent architectures on this task. Results show higher observed success for tool-augmented agents, while multi-agent coordination often raises cost and runtime without consistently improving over a single agent.

  • Problem

    Dedicated agentic AI frameworks or benchmarks for diagnosing and resolving non-convergent power-flow cases are lacking, despite the task requiring iterative inspection, residual analysis, and corrective-action testing.

  • Method

    RestoreBench evaluates seven LLMs across chatbot, single-agent, and multi-agent architectures using defined simulator interactions, action spaces, constraints, and metrics.

  • Results

    Tool-augmented agents generally achieve higher observed success rates than the chatbot baseline, while multi-agent coordination often increases cost and runtime without consistently improving over a single agent.

  • Takeaways & Limitations

    The benchmark provides a reproducible basis for developing agentic AI systems for power-system planning and operation while exposing performance–efficiency trade-offs.

  • Takeaways & Limitations

    Each model–architecture–scenario configuration was executed once, so reported results are single realizations rather than expected performance over repeated trials.

Abstract

from arXiv · show

Large Language Model (LLM) agents increasingly automate multi-step engineering workflows through tool use, interpretation of intermediate results, and iterative planning. Diagnosing and resolving non-convergent power flow cases is a promising yet largely unexplored application, as it requires engineering judgment, experimentation, and decision-making within constrained action spaces. We introduce a benchmark that evaluates these capabilities across multiple LLMs and three architectures: \emph{chatbot}, \emph{single agent}, and \emph{multi-agent} systems. The evaluation covers two power grids and 46 cases per grid, each requiring one or more corrective actions to restore convergence. The benchmark defines the simulation environment, observation and action spaces, and evaluation metrics, providing a reproducible foundation for developing agentic AI systems for power system planning and operation. The code is available at https://github.com/Mansutti081/RestoreBench

1 Introduction

Power-system AI research has largely focused on completed simulations, leaving non-convergent power-flow diagnosis and recovery insufficiently addressed. RestoreBench targets this gap by evaluating LLMs that iteratively interpret simulator feedback and choose corrective actions.

  • Research gap: Existing power-system agent studies mainly automate analyses that complete successfully, not the diagnosis and resolution of non-convergent power-flow cases.Resolving such cases requires network inspection, residual analysis, issue identification, and corrective-action testing.
  • Contribution: The paper introduces a benchmark for evaluating agentic AI systems on diagnosing and resolving non-convergent power-flow cases.
  • Contribution: The benchmark defines observation and action spaces, simulator feedback, operational constraints, and evaluation metrics.
  • Contribution: The evaluation covers seven LLMs and compares chatbot, single-agent, and multi-agent interaction architectures.

2 Benchmark Framework

The benchmark evaluates whether LLM-based systems can restore convergence in non-convergent power-flow scenarios through constrained, feedback-driven corrective actions. It compares chatbot, single-agent, and multi-agent interaction designs using defined metrics for success, cost, runtime, and voltage quality.

  • Task definition: The task focuses on non-convergence caused by insufficient reactive power support, which can result in severe voltage collapse.Agents must identify the cause and restore a feasible operating condition using permitted control actions.
  • Interaction architectures: All three architectures receive a non-convergent grid snapshot called a Scenario Card and return a sequence of corrective actions.
  • Interaction architectures: The chatbot follows a fixed action sequence without intermediate feedback, while single- and multi-agent systems iteratively use diagnostic tools and updated simulator feedback.The multi-agent design distributes the iterative process across an Analyst, Executor, and Orchestrator.
  • Evaluation protocol: Success Rate measures restored convergence with all contractual electrical constraints satisfied, while Cost and Runtime measure average model cost and computation time per case.
  • Evaluation protocol: In-band Buses measures the percentage of buses within [0.95, 1.05] p.u. among successfully solved cases, separately from terminal success.

3 Numerical Results

Experiments evaluate seven LLMs and three architectures on IEEE 118-bus cases, with PEGASE 89-bus results assessing generalization. Agent-based systems generally outperform the chatbot, while multi-agent coordination adds cost and runtime without consistent performance gains.

  • Experimental setup: The experiments evaluate seven LLMs across chatbot, single-agent, and multi-agent architectures on the IEEE 118-bus system, with PEGASE 89-bus results provided separately.
  • Success and cost: The chatbot generally achieves the lowest Success Rate, whereas agent-based architectures perform better.
  • Success and cost: Multi-agent architecture significantly increases cost without systematically improving performance, making the single-agent architecture more cost-effective.
  • Scope boundary: Post-convergence sensitivity tools for refining out-of-band voltages are left for future work.
  • Failure modes: Failure modes include persistent non-convergence, structured-output failures, invalid actions, and timeouts.DeepSeek V3.2 frequently fails to issue the structured terminal output required by tool-based architectures.
  • Voltage quality and runtime: Voltage quality varies only modestly across LLMs and architectures, while runtime increases with interaction complexity.Chatbot configurations are generally faster, whereas multi-agent systems require longer execution times because of additional model calls and simulator interactions.

4 Limitations and Future Work

The study reports one execution per model–architecture–scenario configuration, so its results are single stochastic realizations rather than expected performance over repeated trials. Future work will quantify variability through repeated runs and extend post-convergence voltage refinement.

  • Limitations: Each model–architecture–scenario configuration was executed once, limiting the results to one realization despite deterministic benchmark and simulator behavior.LLM outputs are stochastic, so repeated trials are needed to estimate expected performance and confidence intervals.
  • Future work: Future work will add repeated independent runs and post-convergence sensitivity tools to quantify variability and refine bus voltages toward their admissible range.

5 Conclusion

The paper introduces RestoreBench for diagnosing and resolving non-convergent power-flow cases across two grids, seven LLMs, and three architectures. Tool-augmented agents generally outperform the chatbot baseline, while multi-agent coordination often adds cost and runtime without consistent gains over single agents.

  • The benchmark evaluates seven LLMs across chatbot, single-agent, and multi-agent architectures on two power grids.
  • Tool-augmented agents generally achieve higher observed success rates than the chatbot baseline.
  • Additional multi-agent coordination often increases cost and runtime without consistently improving performance over a single agent.
  • The benchmark provides a reproducible basis for developing agentic AI systems for power-system planning and operation.

A Benchmark Environment

The benchmark environment deterministically evaluates constrained corrective actions using a locked Newton–Raphson power-flow policy and a terminal success contract. Agents can adjust generator voltage setpoints, switch shunts, or move transformer taps through atomic, validated maneuvers.

  • Every electrical result and terminal verdict is produced by a deterministic simulator under a locked power-flow policy.The same policy is used for curation, agent-facing tools, and terminal evaluation.
  • The solver uses Newton–Raphson iterations with PV-to-PQ switching, DC initialization, 30 iterations, and tolerances of 10^-8 MVA and 10^-6 MVA for recovery.
  • SUCCESS requires convergence plus satisfaction of active/reactive limits and connectivity; voltage quality is reported separately through In-band Buses.A converged state violating another non-voltage constraint is classified as SOLVED_INFEASIBLE.
  • Agents may perform only atomic generator-voltage, shunt-switching, and transformer-tap maneuvers within declared operating bounds.Topology switching, load modification, redispatch, and component changes are excluded.
  • Episodes end deterministically as SUCCESS, BUDGET_EXHAUSTED, or TIMEOUT, with at most 10 maneuver slots and 2400 seconds for tool-bearing agents.

B Dataset

The dataset is generated deterministically from feasible base networks by varying initialization profiles, localized electrical load pockets, and stress trajectories until controlled non-convergence is found. Cases are admitted only when they are reactive-limit related and recoverable through the benchmark’s constrained action space.

  • B.1 Base cases: The pipeline starts from convergent, electrically feasible networks and applies grid-specific augmentation before using common scenario-generation logic.Augmentation defines missing bounds without changing topology, component availability, or controllable-device inventory.
  • B.2 Initialization Profiles and Electrical Pockets: Initialization profiles vary existing shunt states and transformer taps, retaining only unstressed configurations that remain convergent and feasible.
  • B.2 Initialization Profiles and Electrical Pockets: Electrical load pockets are localized around anchor buses using impedance-weighted network distance, with τ controlling their spatial extent.Multiple τ values create stress regions with different spatial extents.
  • B.3 Convergence Boundary Search: Stress trajectories progressively scale active and reactive demand inside each pocket while preserving the original power factor and compensating active demand through bounded internal redispatch.Redispatch is not exposed as an agent action.
  • B.4 Reactive-Deficit Admission: Candidates are admitted only if they robustly fail, require exceeding a generator reactive-power limit when Q limits are disabled, and exhibit Q–V deterioration.Generator Q limits remain enforced in the benchmark.
  • B.5 Recoverability Certification: An offline search retains only cases solvable within 10 available maneuvers, classifying them as direct or sequential according to corrective-sequence length.Private ground-truth sequences certify recoverability but are not exposed to agents.
  • B.6 Final Dataset Selection: The final evaluation selects 46 scenarios per network to diversify initialization, stress location, severity, resolution regime, and ground-truth length.Near-duplicate stress families are grouped to avoid an electrically redundant corpus.
  • B.7 Agent Interface: Agents access four tools for topology inspection, action validation, single-maneuver AC evaluation, and limited candidate ranking.Applicability checks reject illegal or exhausted maneuvers before power-flow evaluation.

C.4 Single-maneuver power flow evaluation

Single-maneuver evaluation previews an atomic action on an isolated grid copy, runs AC power flow, and returns convergence or diagnostic progress without altering the current reasoning state. Candidate ranking reduces the legal action space to a bounded, evaluated shortlist, while multi-agent decisions preserve the same iterative maneuver protocol.

  • Single-maneuver evaluation: An optional atomic maneuver is applied to an isolated copy of the current state before AC power flow evaluation.Previewing does not modify the state on which the agent is reasoning.
  • Single-maneuver evaluation: If the candidate converges, the tool returns solver results and whether the resulting state satisfies the success contract.Numerical convergence alone does not necessarily imply successful resolution.
  • Single-maneuver evaluation: Non-convergent candidates are compared with the unchanged reference state and classified as improved, unchanged, worsened, or unknown.The comparison includes overstress and retreat-point diagnostics, using a reference diagnostic reused across candidates from the same state.
  • Single-maneuver evaluation: A divergent maneuver without measurable improvement is blocked from identical repetition, while an improving maneuver remains available for multi-maneuver solutions.This preserves potentially useful intermediate steps while preventing repeated failed attempts.
  • Candidate-maneuver ranking: The ranking tool evaluates at most M = 12 legal candidates per call and orders them by convergence, feasibility, and measured progress.Candidates are evaluated independently on copies using the locked solver and diagnostic procedure.
  • Multi-Agent Architecture: The multi-agent architecture retains one committed maneuver per iteration while decomposing decisions among Analyst, Executor, and Orchestrator roles.Each role receives role-specific context and produces schema-validated structured output.

E Failure Mode Analysis

Failure analysis separates unsuccessful episodes by termination mode, budget-consuming events, progress quality, and timeout behavior. The results show architecture- and model-dependent failure patterns, while improved maneuvers can still fail to restore convergence within the available budget.

  • Failure-mode analysis: Unsuccessful episodes terminate as either BUDGET_EXHAUSTED or TIMEOUT, which are analyzed separately.Aggregate success rates do not distinguish qualitatively different failure mechanisms.
  • Budget-exhausted episodes: BUDGET_EXHAUSTED episodes contain budget-consuming events including still diverged, structured-output failure, invalid action, and solved infeasible outcomes.A single episode may contain several event types across successive iterations, with up to 10 budget-consuming events.
  • Budget-exhausted episodes: Chatbot failures are dominated by valid maneuvers that leave the system non-convergent, whereas structured-output failures occupy a larger share in tool-bearing architectures.Invalid actions and solved-infeasible outcomes represent smaller fractions of overall budget consumption.
  • Budget-exhausted episodes: A high share of improved maneuvers in unsuccessful episodes indicates that agents may progressively reduce overstress without restoring convergence before budget exhaustion.Progress composition classifies determinate committed maneuvers as improved, unchanged, or worsened.
  • Timeout episodes: TIMEOUT episodes may end before the maneuver budget is fully consumed, so their action histories are treated as truncated trajectories.They are excluded from the budget-composition analysis.
  • Timeout episodes: Lower remaining-maneuver counts indicate timeout after most available opportunities were consumed, whereas larger counts indicate earlier interruption.Figure 8 reports the share of TIMEOUT episodes for each number of remaining maneuvers.
  • Timeout episodes: A high LLM-calls-per-committed-maneuver tail identifies timeout episodes with substantial interaction overhead and comparatively few committed maneuvers.These quantities characterize computational behavior before timeout, not its causal mechanism.
  • Model-dependent failures: DeepSeek V3.2 shows model-dependent structured-output failures concentrated in tool-bearing architectures and associated with empty terminal payloads.These events occur across a large fraction of the evaluated cases.

F Additional Experimental Results

On PEGASE 89, tool-bearing architectures generally outperform the chatbot, while single agents provide the strongest reliability–efficiency trade-off. Multi-agent coordination often adds cost and runtime without improving success, and unsuccessful episodes reflect both protocol failures and incomplete electrical progress.

  • Performance across architectures: PEGASE 89 reproduces the pattern that tool-bearing architectures generally achieve higher Success Rate than chatbot configurations.GPT-5.6 Sol rises from 82.6% to 100%, while Claude Sonnet 5 and Claude Opus 5 rise from 50.0% and 47.8% to 100% with either tool-bearing architecture.
  • Performance across architectures: Increasing complexity beyond a single agent does not systematically improve Success Rate.Claude Haiku 4.5 falls from 97.8% with a single agent to 87.0% with multi-agent coordination.
  • Performance across architectures: DeepSeek V3.2 rises from 30.4% with the chatbot to 63.0% with a single agent, then falls to 15.2% with the multi-agent architecture.The reported behavior is consistent with structured-output failures when the required interaction protocol is not reliably followed.
  • Cost and efficiency: For GPT-5.6 Sol, Claude Sonnet 5, and Claude Opus 5, multi-agent coordination raises average cost without improving the 100% Success Rate.Costs increase from $0.472 to $0.915, $0.315 to $0.720, and $0.725 to $1.600, respectively; Kimi K2.5 reaches 100% Success Rate at $0.097 per case with a single agent.
  • Quality and runtime: Voltage quality varies only moderately across architectures, whereas runtime increases with iterative tool use and additional multi-agent coordination.For GPT-5.6 Sol, median runtime increases from 58.8 s for chatbot to 92.5 s for single agent and 147.8 s for multi-agent; voltage distributions include only successfully solved cases.
  • Unsuccessful episodes: BUDGET_EXHAUSTED episodes differ by architecture: chatbot budgets mostly contain non-convergent valid maneuvers and invalid actions, while tool-bearing budgets are dominated by structured-output failures.The failure analysis therefore distinguishes electrical-action quality from failures to follow required formats or interaction steps.
  • Unsuccessful episodes: Successful tool-bearing episodes are dominated by improved maneuvers, while some budget-exhausted episodes improve the grid without restoring convergence within ten maneuvers.PEGASE 89 also shows architecture-dependent differences in how maneuver budgets are consumed before BUDGET_EXHAUSTED termination.
  • Unsuccessful episodes: In TIMEOUT episodes, only one to three maneuver opportunities typically remain, and all observed cases require 10–20 LLM calls per committed maneuver.Thus, the reported timeouts occur after most maneuver opportunities have been used rather than from exceptionally many calls for individual maneuvers.

G Experimental Cost Breakdown

The benchmark comprises 1,932 evaluation episodes and incurred $977.86 in LLM API costs, with costs varying across models, networks, and interaction architectures.

  • Cost accounting: Each model is evaluated on 46 scenarios across three architectures per network, totaling 138 evaluation episodes per model-network combination.Per-episode costs use stored token counts repriced at providers’ published list rates at each campaign’s time.
  • Total cost: 1,932 evaluation episodes incurred $977.86 in total LLM API costs.IEEE 118-bus experiments account for $637.24 and PEGASE 89 experiments for $340.62.
  • Cost accounting: More complex agent configurations require additional LLM calls, producing substantial variation in cost across models and interaction architectures.The resulting cost motivated broader evaluation across models, architectures, networks, and scenarios rather than repeated evaluation of a narrower configuration.

H Reproducibility

The implementation releases the exact prompts and runtime substitutions used for all chatbot, single-agent, and multi-agent experiments.

  • Reproducibility: The public repository releases complete prompt templates, tool descriptions, and runtime substitutions for reproducibility.It includes the exact prompts used by the chatbot, single-agent, and multi-agent configurations.
Loading 2609.00384v1…