PowerDAG: Supervisory Agentic AI System for Automating Distribution Grid Analysis

I. INTRODUCTION

• Grid studies span diverse tasks and require correctly ordered, stateful workflows, while utilities often lack the engineering capacity to conduct them at scale.Rule-based automation relies on fixed templates and requires continual manual curation to accommodate real-world queries.Source: §I · p. 1
• LLMs can select incorrect tools, sequence calls wrongly, or bind invalid arguments, and general manuals rarely explain cross-tool handoffs, ordering, and state updates.Fine-tuning requires domain-specific data and retraining when tools, models, or APIs change; in-context learning avoids changing model weights.Source: §I · p. 1
• Fixed top-k exemplar retrieval can omit needed steps for complex queries or add misleading, irrelevant workflows that increase token costs.Similarity based on query phrasing may select an exemplar whose tool sequence differs from the procedure the current query requires.Source: §I · p. 1
• PowerDAG adaptively selects exemplars using query and workflow relevance, filtering expert-verified traces for matching tool sequences and prerequisites.Its two stages replace a global k with a score-decay cutoff, then filter candidates by procedural structure.Source: §I · p. 2
• Just-in-Time supervision checks prerequisites before each tool call, blocking invalid actions and returning an advisory without changing environment state.The paper also introduces a benchmark of 200 expert-verified records across ten distribution-grid task families.Source: §I · p. 2

II. RELATED WORKS

• Prior agentic grid systems run iterative tool-use loops, but none explicitly enforce prerequisite dependencies across calls.Prior exemplar-conditioned systems leave open how to select the right exemplars, while fixed top-k retrieval can omit relevant workflows or inject irrelevant context.Source: A. LLMs for Tool Orchestration · p. 2, B. Exemplar-Conditioned Orchestration · p. 2, C. Retrieval of Relevant Exemplars · p. 2

D. Runtime Supervision of Tool Execution

• Prior prompting and post-hoc correction cannot guarantee prerequisite compliance or catch silent errors when tools run successfully on stale state.Existing deterministic checks are described as lacking enforcement of cross-tool prerequisites across long-horizon workflows.Source: D. Runtime Supervision of Tool Execution · pp. 2–3

A. Agentic AI System

• An agentic system maps interaction history and relevant context to a proposed tool call, which the environment executes and returns as a new state and observation.The closed-loop interaction continues as observations inform subsequent actions.Source: A. Agentic AI System · p. 3

B. State Transition and Tool Classification

• Read tools query current objects without changing state, whereas Write tools modify persistent environment objects.A Read tool can execute correctly only after the required Write tools have established the state it reads.Source: B. State Transition and Tool… · p. 3

C. Workflow as a Directed Acyclic Graph (DAG)

• A workflow DAG represents tool invocations as nodes and prerequisite relationships as directed edges, requiring predecessors to finish before dependent calls execute.Calls without a dependency path may run in either order; repeated calls to the same tool are distinct nodes.Source: C. Workflow as a Directed Acyclic… · p. 3, Figure 1 · p. 3

D. Annotated Query-Workflow Exemplars (Wav)

• Annotated query-workflow exemplars pair natural-language queries with expert-verified tool-call sequences that produce correct outputs.The archive is denoted Wav.Source: D. Annotated Query-Workflow Exemplars… · p. 3

A. PowerDAG System Architecture

• PowerDAG prompts its LLM policy with the query, selected exemplars, interaction history, and tool schemas, then runs a closed-loop interaction with the environment.The architecture places the JIT supervisor between the agent and executor; invalid calls are blocked before environment mutation.Source: Figure 2 · p. 3, A. PowerDAG System Architecture · p. 4

B. Adaptive Retrieval of Annotated Exemplars

• Adaptive retrieval first applies a cutoff to ranked query similarities, then filters candidates by procedural relevance to produce the final exemplar subset.The two-stage selection is illustrated in Fig. 3.Source: B. Adaptive Retrieval of Annotated… · p. 4, Figure 3 · p. 4
• A two-segment least-squares fit to the ranked similarity profile selects the adaptive cutoff at the breakpoint minimizing weighted fit error.Stage 1 ranks query-to-query cosine similarities; it excludes workflow traces because tool names, arguments, and execution artifacts can distort semantic similarity.Source: B. Adaptive Retrieval of Annotated… · p. 4
• An LLM filter uses candidate queries and expert-verified traces to retain workflows whose tool sequences match the unseen query’s requirements.The filter selects among existing traces rather than synthesizing a new sequence, and operates on the small candidate set.Source: B. Adaptive Retrieval of Annotated… · p. 4

C. Just-in-Time Supervision

• The JIT supervisor checks each proposed call against prerequisite rules before environment mutation, blocking violations and returning an advisory while preserving state.Rules are derived from repeated patterns in archived workflows and verified by an expert against tool documentation; tools without a rule remain executable.Source: C. Just-in-Time Supervision · pp. 4–5
• For a covered tool, the supervisor checks whether all required prerequisite Write tools have already executed; calls without a rule are not blocked.The rule library specifies required Write tools and an advisory message for each covered tool.Source: C. Just-in-Time Supervision · p. 5
• The supervisor permits only DAGs satisfying prerequisite constraints, preventing prerequisite-invalid execution orders from reaching the executor.The analysis compares supervised counts with the corresponding unsupervised counts under its stated workflow-counting assumptions.Source: C. Just-in-Time Supervision · p. 5
• Each prerequisite rule weakly reduces the workflow DAGs that can reach the executor; requiring A before B reduces the two-tool example from six workflows to three.With no bounds on tool selection or repetition, the possible workflow space is unbounded; the baseline count assumes each tool is called at most once and ignores arguments.Source: C. Just-in-Time Supervision · pp. 5–6

• The evaluation covers benchmark queries, an exemplar archive, agentic systems, LLMs, and metrics for correctness and efficiency.Source: §V · p. 6

• The benchmark contains 200 expert-verified query-workflow records across ten distribution-grid task families, with workflows ranging from 2 to 21 tool calls.Source: A. Queries and Study Scope · p. 6
• The evaluation uses 150 held-out records, while 50 records—five per task family—form an exemplar archive with no query overlap.Source: A. Queries and Study Scope · p. 6
• Held-out queries vary from exemplars in feeder choice, component parameters, time windows, solver settings, physical limits, and required artifacts.The study therefore evaluates adaptation within the studied task families.Source: A. Queries and Study Scope · p. 6

B. Curation of Annotated Query-Workflow Exemplars

• The 50-record exemplar archive provides five verified natural-language query and tool-workflow pairs per task family as in-context examples.Source: B. Curation of Annotated… · p. 6

C. Agentic Systems, Tools, and Evaluation Setup

• Six configurations compare three unsupervised, no-exemplar baselines, two single-component variants, and full PowerDAG using the same environment, data pipelines, and 108 tools.Table I summarizes the agentic systems and enabled components.Source: C. Agentic Systems, Tools, and… · p. 6, Table I · p. 6

D. LLMs and Compute Platforms

• The evaluation spans five proprietary and five open-weight LLMs, with open-weight models served on two 80-GB NVIDIA H100 GPUs.Source: D. LLMs and Compute Platforms · p. 6
• Stage 1 uses GEMINI-EMBEDDING-2 for retrieval, while Stage 2 filters candidates with the evaluated model; Gemma-4 E2B rises from 5.3% without exemplars to 16.7% with them.The same Stage 1 embedder is used across models.Source: D. LLMs and Compute Platforms · p. 6

E. Benchmark Evaluation and Metrics

• The study runs each of 150 queries once for each of ten LLMs and six configurations, totaling 9,000 independently scored runs.The evaluator checks tool coverage, argument compatibility, dependency order, and required validation or export calls; the protocol follows τ-bench [39].Source: E. Benchmark Evaluation and Metrics · p. 6

1) Pass@k (P@k):

• Pass@k estimates the probability that at least one of k generated workflows succeeds [40]; this study reports k=1 because each query is run once.Pass@1 is the task-success metric: required calls, compatible arguments, and valid dependency order must all be present.Source: E. Benchmark Evaluation and Metrics · p. 6, 1) Pass@k (P@k) · p. 7

2) Precision (Pr):

• Precision measures trace equivalence between agent and expert workflows after DAG normalization, treating valid reorderings with nonconflicting read/write sets equally.Source: 2) Precision (Pr) · p. 7

3) Token use (Tk):

• Token use is the mean prompt-plus-completion tokens across model calls, calculated over successful runs and reported in thousands.Source: 3) Token use (Tk) · p. 7

VI. RESULTS

• The results section reports Pass@1, Precision, and token use for six agentic configurations across ten LLMs on the 150-query benchmark.Pass@1 and Precision are reported as percentages.Source: §VI · p. 7

VII. CONCLUSION

• For GPT-5.5, successful PowerDAG runs average 73.95k tokens, below ReAct at 84.04k, LangChain at 92.50k, and CrewAI at 90.14k.PowerDAG uses more tokens per run overall because exemplars and advisories add prompt context; the conclusion says fewer error-driven calls may explain the strongest models’ lower successful-run use.Source: B. Precision, Token Use, and Advisories · p. 7, §VII · p. 8

C. Effect of Combining Retrieval and Supervision

• Combining adaptive retrieval and prerequisite checking yields higher Pass@1 than either ablation on every model, with gains reaching 23.33% over PowerChain-AR and 20.66% over the JIT-supervised agent.The components address distinct failure modes: exemplars improve tool selection and call ordering, while supervision blocks calls with unmet prerequisites.Source: C. Effect of Combining Retrieval and… · p. 7, §VII · p. 8

B. Precision, Token Use, and Advisories

• PowerDAG achieves best or tied-best Precision on six of ten models; other configurations with higher Precision have lower Pass@1.Across all 9,000 runs, average token use is 113.4k per PowerDAG run, versus 86.5k for PowerChain-AR, 84.8k for ReAct, 85.0k for LangChain, and 79.0k for CrewAI; 78.8% of PowerDAG runs receive an advisory.Source: C. Effect of Combining Retrieval and… · p. 7, B. Precision, Token Use, and Advisories · p. 7

A. Pass@1 across Model Families

• 98.00% Pass@1 on GPT-5.5, with PowerDAG highest on every evaluated LLM and improving 2.00% to 20.66% over the best non-PowerDAG configuration.Fig. 4 compares Pass@1 across the benchmark’s 150 queries and ten task families.Source: A. Pass@1 across Model Families · p. 7, §VII · p. 7, Figure 4 · p. 8

VII. CONCLUSION

• QWEN3.6-27B, GEMMA-4 31B, and GPT-OSS 120B reach 92.67%, 89.33%, and 88.00% Pass@1, while GEMMA-4 E2B and LLAMA 3.2-3B reach 16.67% and 12.67%.The authors say strong open-weight results support local deployment for utilities unable to share grid data with commercial APIs.Source: §VII · p. 8
• The released benchmark contains 200 expert-verified query-workflow records across ten distribution-grid task families, alongside an exemplar archive, execution environment, and evaluator.The conclusion presents these resources as a reproducible benchmark for agentic workflow correctness.Source: §VII · p. 8

APPENDIX A BENCHMARK TASK SET

• The benchmark contains 200 expert-verified query-workflow records across ten task families, with 50 exemplars and 150 held-out evaluation queries.Each record pairs a natural-language query with a workflow; experts verify tool calls, argument bindings, and outputs.Source: Appendix A · p. 8, Table III · p. 8

APPENDIX B ANNOTATED WORKFLOW EXEMPLARS

• The 50-record archive contains five validated query-workflow exemplars per task family, and no held-out evaluation query appears in it.The agent retrieves relevant records from the archive at inference time.Source: Appendix B · p. 8

APPENDIX C RETRIEVAL DIAGNOSTICS

• Google GEMINI-EMBEDDING-2 with cosine similarity leads three of four reported retrieval metrics.Table IV reports the scorer comparison.Source: Appendix C · p. 9, Table IV · p. 9
• The adaptive two-segment elbow cutoff achieves the highest RCS, 0.635, balancing candidate precision with recovery of same-family exemplars.The comparison uses Google GEMINI-EMBEDDING-2 and contrasts fixed top-k retrieval with five adaptive cutoff policies.Source: Appendix C · p. 9, Table V · p. 9

APPENDIX D DERIVATION OF ADAPTIVE TWO-SEGMENT ELBOW CUTOFF

• The derivation assumes similarity scores are sorted in non-increasing order and considers candidate breakpoints from 2 through N−2.For each breakpoint, it defines left and right index sets spanning the sorted scores.Source: Appendix D · p. 9
• For an index interval, Φ(r,s) is the RMSE of the least-squares affine fit to the corresponding index-score pairs.Source: Appendix D · p. 9
• The weighted two-segment objective combines the two segment RMSE values for each candidate breakpoint.Source: Appendix D · p. 9

APPENDIX E WORKFLOW-FILTER PROMPT

• At inference, the workflow filter receives the unseen query and JSON-formatted Stage 1 candidates.Source: Appendix E · p. 9
• The prompt asks the model to keep workflows that help solve the query and exclude only clearly unrelated candidates, returning unique indices as JSON.Source: Appendix E · p. 9

APPENDIX F SUPERVISOR ADVISORY AND DIAGNOSTICS

• Before each tool call, the supervisor checks prerequisites and blocks execution if required environment states are missing, leaving state unchanged.Source: Appendix F · p. 9
• The advisory identifies the blocked tool and violation, directs the agent to call the missing prerequisite first, and says not to restart or repeat successful setup tools.Source: Appendix F · pp. 9–10
• Prerequisite rules derive from expert-verified workflows and tool-level state contracts; checks cover 94 of 108 tools and 1,097 of 1,330 held-out workflow calls.Table VI reports rule coverage.Source: Appendix F · p. 10, Table VI · p. 10
• Across ten task families, the supervisor issues advisories in 78.8% of full PowerDAG runs and blocks 6.57 proposed calls per run on average.The JIT-only ablation issues advisories in 87.5% of runs and blocks 4.64 calls per run.Source: Appendix F · p. 10
• The rule-level audit finds 0.0% false positives on 1,034 admissible calls and 0.0% false negatives on 1,393 constructed prerequisite violations.Eight of 1,178 advisory runs fail afterward, all on smaller models, due to agent-compliance errors.Source: Appendix F · p. 10