Exploring the Capabilities and Limitations of Large Language Models in the Electric Energy Sector

Subir Majumder; Lin Dong; Fatemeh Doudi; Yuting Cai; Chao Tian; Dileep Kalathi; Kevin Ding; Anupam A. Thatte; Na Li; Le Xie

Source-linked AI summary

Exploring the Capabilities and Limitations of Large Language Models in the Electric Energy Sector

Subir Majumder, Lin Dong, Fatemeh Doudi, Yuting Cai, Chao Tian, Dileep Kalathi, Kevin Ding, Anupam A. Thatte, Na Li, Le Xie

arXiv:2403.09125v5Mar 14, 2024eess.SY

TL;DR

The paper investigates whether LLMs can improve electric-energy-sector operations amid growing system complexity and data demands. It examines GPT-based power-engineering tasks and identifies capabilities, limitations, and research directions involving domain data, embedded tools, and RAG. The study finds that LLMs can support diagnostics, forecasting, document analysis, and safety-related assessment, but their usefulness is constrained by domain-data scarcity, physical-principle challenges, and safety limitations.

Problem
The paper addresses the need to understand LLM capabilities and limitations for improving electric energy operations as system complexity and data volume increase.
Method
The study evaluates GPT models across power-flow, forecasting, image and pattern recognition, document analysis, and safety-related power-engineering tasks.
Results
LLMs demonstrated support for power-system diagnostics, forecasting, document question answering, multimodal defect assessment, and hazard recognition, with performance varying by task and prompting.
Takeaways & Limitations
Future work should develop domain-specific fine-tuning data, embed power-system tools, and use RAG-based knowledge resources for stronger responses and safety-critical applications.
Takeaways & Limitations
LLM use is constrained by scarce domain-specific data, limited context windows for long-range power-system signals, difficulty handling physical principles, and safety-related restrictions.

Abstract

from arXiv · show

Large Language Models (LLMs) as chatbots have drawn remarkable attention thanks to their versatile capability in natural language processing as well as in a wide range of tasks. While there has been great enthusiasm towards adopting such foundational model-based artificial intelligence tools in all sectors possible, the capabilities and limitations of such LLMs in improving the operation of the electric energy sector need to be explored, and this article identifies fruitful directions in this regard. Key future research directions include data collection systems for fine-tuning LLMs, embedding power system-specific tools in the LLMs, and retrieval augmented generation (RAG)-based knowledge pool to improve the quality of LLM responses and LLMs in safety-critical use cases.

1 Introduction

The paper examines whether LLMs can help manage the electric energy sector as system complexity, data volume, and operational demands grow, while emphasizing the need to test their capabilities and limitations.

1 Introduction: Prior work reports applications including customized code generation, RAG-based technical question answering, power-network data synthesis, and deep-reinforcement-learning power-flow solutions.Related concerns include data ownership, privacy, and safety guarantees.
1 Introduction: The electric energy sector faces expanding devices, data, renewable resources, and technologies that challenge operators without proportional workforce growth.The accumulating knowledge and instantaneous data exceed unaided human processing capacity.
1 Introduction: LLMs may help interpret human prompts, reduce sensory overload, and provide near real-time guidance for extreme-weather and uncertainty-related risks.The study therefore investigates their readiness as an interface between humans and electric energy systems.
1 Introduction: The paper uses production-grade GPT models to explore LLM capabilities in realistic power-engineering tasks, both independently and through add-on capabilities.The investigation is framed as a comprehensive analysis of capabilities and limitations.

2 Capabilities of LLMs to Fill in the Gap

The paper presents LLM capabilities relevant to power engineering, including prompt-based reasoning, tool delegation, in-context learning, and multimodal processing. These capabilities support tasks such as diagnostics, forecasting, document analysis, and image-based equipment assessment.

2 Capabilities of LLMs to Fill in the Gap: LLMs can connect power-system data, software, tools, and cross-domain datasets to support real-time diagnostics, on-demand analysis, and control-center operations.The paper identifies these implications for improving operational efficiency and decision support.
2 Capabilities of LLMs to Fill in the Gap: Prompt engineering improves responses on difficult power-engineering tasks, whereas naïve prompts often fail to produce desirable outputs.RAG and chain-of-thought prompting are identified as prominent techniques.
2 Capabilities of LLMs to Fill in the Gap: In-context learning lets LLMs infer patterns or rules from a few examples without model updates, which is useful when power-system datasets are protected.Custom domain-specific knowledge can reduce variability in generated responses.
2 Capabilities of LLMs to Fill in the Gap: Tool embedding allows LLMs to delegate tasks, execute code, and infer results from embedded tools for complex power-system problems.API calling could connect LLMs with tools such as PSS/E, PSCAD, PowerWorld, and CyME.
2 Capabilities of LLMs to Fill in the Gap: Multimodal processing helps LLMs contextualize time series, images, and videos, including diagnosing defects in insulator images alongside in-context learning.The paper also examines forecasting and document-based applications across power-system tasks.

3 Limitations of LLMs for Applications in the Electric Energy Sector

LLMs face domain-data, context-window, physical-modeling, safety, explainability, cybersecurity, and privacy limitations in electric-energy applications. These constraints motivate expert oversight, specialized tools, governance, and carefully prepared training data.

Domain-specific pre-training data are scarce because privacy concerns and regulations restrict access, while curated fine-tuning datasets must address CEII and privacy leakage.
Long-range power-system time series may require customized embeddings, while limited context windows can miss long-range signal dependence.
Probabilistic outputs, absent uncertainty estimates, prompt sensitivity, unsafe responses, and insufficient guardrails complicate safety-critical power-system operation.
Because LLM risks differ from traditional software risks, safe deployment in power infrastructure requires a governance framework.
LLMs struggle with tasks governed by physical principles and compounded human, load, and market behavior, so specialized physics-based tools remain indispensable.
General-purpose LLMs can assist engineers through tool embedding, but cybersecurity threats include privilege escalation, backdoors, sensitive-data extraction, and exposed specialized prompts.
Training data require anonymization and sanitization, yet context-dependent personal or group information remains difficult to remove reliably.

4 Future Prospects

The paper identifies data, tool-integration, and retrieval-based knowledge resources as directions for extending LLM capabilities in power-system research and applications. These directions target better task performance and more effective use of power-system expertise.

The study reports that LLMs are strong at detecting correlations among text, images, and data but remain weak on physics-intensive problems involving complex mathematics.
The paper proposes curated data collection for fine-tuning foundational LLMs, requiring power-system expertise to identify sources and construct high-quality datasets.
Power-system-specific tool embeddings could connect LLMs to diverse existing tools, but naïve embeddings may lose efficiency or create conflicts.
A power-system knowledge base for retrieval augmentation is proposed because generic knowledge-base approaches may not fully exploit physical constraints.

arXiv:2403.09125v5 [eess.SY] 20 Jun 2024

The supplemental material covers correlation analysis, recognition, document analysis, retrieval-augmented knowledge pools, forecasting, and related power-system topics.

The supplemental information includes correlation analysis, wildfire-risk and equipment-damage recognition, hazard recognition, document analysis, retrieval-augmented knowledge-pool analysis, and power-system forecasting.

SI.1 Correlation Analysis for the Power Systems

The exploratory correlation-analysis experiments show that GPT-4 can interpret power-system data and generate useful analyses, but its reliability depends on contextual information, prompting, and human oversight.

SI.1.1 Correlation Analysis with Power Flow Data: GPT-4 may misidentify power-flow CSV files unless prompted to load the dictionaries first, and that prompt appears to alleviate the problem.The study reports this as an observed behavior rather than a comprehensive performance evaluation.
SI.1.1 Correlation Analysis with Power Flow Data: GPT-4 produced exploratory correlation analyses, including visualizations distinguishing positive correlations in red from negative correlations in blue.Changing the request from exploratory data analysis to general insights produced substantially different responses.
SI.1.1 Correlation Analysis with Power Flow Data: GPT-4 correctly compared maximum absolute branch flows with flow limits when asked which lines were approaching their limits.The comparison used flow-limit information from the supplied network dictionary.
SI.1.1 Correlation Analysis with Power Flow Data: With explicit guidance to use absolute branch-flow values, GPT-4 more accurately identified renewable-generation correlations with line flows and could estimate overloads for an unknown scenario.Without that guidance, direction-specific line flows led to misleading interpretations of renewable contributions.
SI.1.2 Correlation Analysis with Demand and Prices Data: For demand-and-price data, GPT-4 interpreted domain-specific column headers and, when asked to use its best judgment, performed time-series visualization, correlation, and distribution analyses.It also recommended constructing an LSTM load-forecasting model, but the generated training script encountered platform-related errors.
SI.1 Correlation Analysis for the Power Systems: The experiments indicate that LLMs require contextual information for time-series analysis, may not address data-distribution issues without prompting, and still require human oversight.The authors note that generated models can be erroneous unless users check their accuracy.

SI.2 Wildfire Risks Recognition on the Power Lines

The wildfire experiments show that GPT-4 can extract wildfire regions and overlay them on transmission-line maps for situational awareness, while future-spread prediction remained unavailable.

SI.2 Wildfire Risks Recognition on the Power Lines: GPT-4 could use uploaded-file metadata to identify and label wildfire maps before overlaying multiple monthly wildfire areas on a transmission-line map.The task involved August, September, and October wildfire maps plus a corresponding transmission-line map.
SI.2 Wildfire Risks Recognition on the Power Lines: The generated overlay code varied across trials, and close inspection found failures primarily associated with mistakes in filter applications.The study used additional experiments and AST comparisons to examine variability in generated Python code.
SI.2 Wildfire Risks Recognition on the Power Lines: GPT-4 iteratively extracted wildfire-affected areas and overlaid them on transmission-line maps to visualize potential wildfire impacts on power lines.The authors frame this capability as supporting visualization and situational awareness rather than predicting wildfire behavior.
SI.2 Wildfire Risks Recognition on the Power Lines: GPT-4 declined to predict which vegetated areas would catch fire the following month because it lacked real-time data and the ability to run such models.The authors describe this as a bottleneck for the tested system and note that localized LLMs might face different constraints.
SI.2 Wildfire Risks Recognition on the Power Lines: Prompt engineering can divide wildfire-processing tasks into manageable steps that GPT-4 performs with fewer errors and may improve credibility with power-systems engineers.The paper presents this as an observed direction for improving generated results, not as a comprehensive guarantee.

SI.3 Equipment Damage Detection in Power Grids

GPT-4’s few-shot prompting improved recognition of faulty versus intact insulators, but performance remained vulnerable to unfamiliar defects and low-quality images.

Few-shot fault detection: Few-shot prompting markedly improved GPT-4’s accuracy in distinguishing faulty from intact insulators.The method supplied labeled examples and descriptions of intact and defective insulators.
Failure modes: GPT-4 misclassified defective insulators under unfamiliar failure conditions and sometimes mistook shadows for chips.
Quantitative evaluation: Using 40 evenly split insulators, the study reported accuracy below a prior result exceeding 90%, with a smaller dataset and no synthetic training images.
Robustness: Responses stayed consistent across prompt styles, but GPT-4 failed to identify faulty insulators when presented with low-quality images.
Scope: The insulator-focused methodology could be adapted to other power-system equipment, although further research is needed to improve performance and robustness.

SI.4 On-site Hazards Recognition

GPT-4 identified several hazards in power-line work and generated repeated audit scores, but numerical scoring and some safety conditions remained constrained by image-based uncertainty.

Hazard recognition: GPT-4 identified hazards including proximity to power lines, PPE, crane stability, fall protection, and safe working radius.
Limitation: GPT-4 could not provide a precise numerical safety score directly, despite recognizing multiple site hazards.
Score interpretation: GPT-4 often assigned 5 when protocols were indeterminate from one image, while assigning lower scores where it judged procedures were not followed.
Uncertainty: Weather scores ranged from 6 to 9 and peaked at 8 because a single image could not establish complete weather conditions.
Application: The experiments suggest image streams, and potentially video, could support real-world situational surveillance around power infrastructure.

SI.5 Document analysis for power systems

GPT-4 handled broad technical-document summarization well, while RAG improved direct domain-specific answers but became less reliable and comprehensive for nuanced questions.

Document processing: GPT-4’s web interface comprehended and discussed all sections of a 170-page Department of Energy smart-grid report without additional context.
Practical implications: Customized prompts improved GPT-4’s ability to generate specialized technical content, while effective power-system deployment still requires refinement and adaptation.
RAG performance: RAG produced more accurate and coherent answers to direct questions than GPT-4’s web interface and aligned more precisely with ERCOT protocol excerpts.
RAG limitations: For complex queries, RAG sometimes failed to answer or produced variable, non-comprehensive responses, even at lower temperatures.
Comparison: Uploading nodal protocols to GPT-4’s web interface produced similar response fluctuations, but it partially answered a question that RAG could not answer.

SI.6 Forecasting in Power Systems: Load and Price Forecasts

The section evaluates three LLM-based approaches for ERCOT load and day-ahead price forecasting, finding stronger load than price-forecasting performance. Fine-tuning improves load forecasts, while longer-term and winter-peak predictions remain challenging.

Methods: The study compares GPT-4 linear regression, text-embedded numerical data, and fine-tuned GPT-3.5 using ERCOT weather, load, and Houston price data.The fine-tuned model uses one year of historical hourly data to generate next-day forecasts.
Results: Fine-tuned GPT achieves promising short-term load forecasts, whereas all tested approaches perform notably worse on price forecasts.The authors attribute the price difficulty to intricate interdependencies with variables not included in the exercise.
Longer-term forecasting: Weekly load forecasting reaches 2% to 4% MAPE during the summer peak, but winter-peak forecasting is very challenging.The authors link winter difficulty to the reversed temperature-load relationship and note that text embedding worsened during winter peaks.
Generalization: The fine-tuned GPT model is evaluated on the GEF14 dataset using 2012 data for day-ahead forecasts in 2013 and 2014, with pinball loss compared against deep-learning benchmarks.The supplied table caption identifies the benchmark comparison but does not provide cell values.

SI.7 Power Flow-related Problems

The section tests LLMs on power-flow, optimal-power-flow, and related computational tasks, showing useful code-generation and tool-use capabilities but unreliable interpretation and solution consistency. Prompt engineering and expert feedback improve consistency without eliminating the need for verification.

Code generation: GPT-4 generates diverse DC power-flow and DC optimal-power-flow code, including code that calls different solvers across repeated prompts.The study compares generated Python programs using Abstract Syntax Trees and similarity scores.
Computational capability: LLMs can solve some linear-equation tasks by generating Python code, while GPT-3.5 self-consistency sometimes produces erroneous responses.GPT-4 can also use embedded tools to generate a solution.
Power flow: GPT-generated power-flow procedures recognize several DC power-flow assumptions but still contain discrepancies in the resulting methodology.Recognized elements include a slack bus, fixed reference angle, negligible resistance, and small phase-angle differences.
Prompt engineering: Embedding matrix details and concise code-only instructions can reduce improper Y-bus reproduction and verbose, fragmented outputs.The experiments vary whether the power-flow method is specified and whether the prompt explicitly requests one code block.
Optimal power flow: GPT sometimes fails to interpret network connectivity, generator limits, or power-flow equations correctly in optimal-power-flow tasks.Custom prompts alleviate some discrepancies, but repeated experiments show inconsistent use of the required equations.
Key insights: LLMs rely heavily on code generation and remote execution for complex power-engineering tasks, making consistently correct generated code and expert feedback essential.The authors identify this reliance as a central insight from the experiments.

SI.8 Ensuring Safe Power Systems Operation

The section examines whether LLM safety guardrails protect electric-power operations from harmful assistance. Guardrails block direct blackout requests, but prompt variation and irrelevant appended text can elicit dangerous procedural detail or generic guidance for prohibited work.

Guardrails: Direct requests to trigger a blackout receive refusal responses from GPT-3.5 and GPT-4, indicating safety guardrails can block straightforward harmful prompts.The paper frames these guardrails as intended to prevent harm to power systems and people.
Prompt variation: Masquerading as a security researcher and adding technical procedural detail can produce responses exposing procedures for accessing substation Modbus systems.The responses become semantically diverse as the prompts become more technical.
Domain-specific safety: For energized capacitor-bank repair, GPT warns of electric shock, arcing, or explosion but still supplies generic guidelines for work prohibited by a laboratory safety manual.This observation motivates domain-specific rather than solely general safety guardrails.
Guardrail circumvention: Appending irrelevant information can sometimes negate blackout-related safety behavior, with responses ranging from refusal to detailed cyber-intrusion guidance.The authors identify this as a significant vulnerability in the safety mechanism.
Implication: The results indicate that applying safety guardrails in domain-specific contexts requires further investigation.The paper distinguishes general guardrails from comprehensive adaptation to electric-energy safety requirements.