PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of AIOps in the Era of Large Language Models

Lingzhe Zhang, Tong Jia, Mengxi Jia, Yifan Wu, Aiwei Liu, Yong Yang, Zhonghai Wu, Xuming Hu, Philip S. Yu, Ying Li

arXiv:2507.12472v1cs.SEcs.CL

TL;DR

LLM-based AIOps lacks a comprehensive account of its impact, potential, and limitations despite rapidly growing research and operational importance. This survey analyzes the field across four research questions and 183 papers, finding expanded data use and substantial promise alongside unresolved challenges and limited evidence for fully automatic remediation.

  • Problem

    Comprehensive studies of LLM-based AIOps are lacking, although understanding its approaches, trends, and limitations matters for software-system reliability and availability.

  • Method

    The survey systematically examines LLM-based AIOps through four questions covering data sources, task evolution, methods, and evaluation.

  • Results

    163 studies were selected through the reported review workflow, while findings show expanded human-generated data use, new preprocessing directions, and emerging LLM-enabled tasks.

  • Takeaways & Limitations

    LLM-powered AIOps is advancing dynamically and may support broader real-world implementations as the field addresses its identified challenges.

  • Takeaways & Limitations

    Automatic execution of LLM-generated remediation scripts remains limited in related work, and its practical effectiveness remains to be verified.

Abstract

from arXiv · show

As large language models (LLMs) grow increasingly sophisticated and pervasive, their application to various Artificial Intelligence for IT Operations (AIOps) tasks has garnered significant attention. However, a comprehensive understanding of the impact, potential, and limitations of LLMs in AIOps remains in its infancy. To address this gap, we conducted a detailed survey of LLM4AIOps, focusing on how LLMs can optimize processes and improve outcomes in this domain. We analyzed 183 research papers published between January 2020 and December 2024 to answer four key research questions (RQs). In RQ1, we examine the diverse failure data sources utilized, including advanced LLM-based processing techniques for legacy data and the incorporation of new data sources enabled by LLMs. RQ2 explores the evolution of AIOps tasks, highlighting the emergence of novel tasks and the publication trends across these tasks. RQ3 investigates the various LLM-based methods applied to address AIOps challenges. Finally, RQ4 reviews evaluation methodologies tailored to assess LLM-integrated AIOps approaches. Based on our findings, we discuss the state-of-the-art advancements and trends, identify gaps in existing research, and propose promising directions for future exploration.

1 INTRODUCTION

Modern software systems are difficult to operate reliably at scale, while traditional AIOps approaches face feature-engineering, generalizability, and automation challenges. This survey addresses the growing LLM-based AIOps literature by organizing its analysis around data, tasks, methods, and evaluation.

  • Motivation: Large-scale software systems require continuous availability, yet their complexity makes failures difficult to detect, localize, diagnose, and remediate.Effective AIOps therefore targets rapid failure detection, efficient diagnosis, accurate root-cause identification, and timely remediation.
  • Motivation: Traditional AIOps methods require complex feature extraction and often have limited cross-platform generality when systems or log-generation rules change.These limitations are especially relevant for unstructured data such as logs and traces.
  • Motivation: Existing deep-learning AIOps methods provide restricted automation, with many remediation efforts stopping at incident triage or solution recommendation.Significant manual intervention and configuration remain necessary.
  • Why a Survey of AIOps in the Era of LLMs?: LLM-based AIOps research has grown rapidly, particularly after ChatGPT, and the pace of growth is expected to accelerate.The survey is motivated by this expanding research landscape and the need to analyze its evolution comprehensively.
  • Why a Survey of AIOps in the Era of LLMs?: Prior AIOps reviews either focus on traditional machine-learning or deep-learning approaches or omit a systematic treatment of the full AIOps task process.The authors position their survey as covering the entire LLM-based AIOps process.
  • Research questions: The survey organizes its analysis around four questions covering data and preprocessing, task evolution, LLM-based methods, and evaluation methodologies.The questions are presented as connected stages, from data foundations through tasks and methods to evaluation.

2 SYSTEMATIC REVIEW PROCESS

The survey uses a systematic review protocol with defined search, inclusion, and exclusion procedures across five literature databases. From 761 retrieved records, 163 studies were ultimately selected for analysis.

  • Review protocol: The review protocol defines the search strategy and scope, inclusion and exclusion criteria, and an overview of selected publications to support a clear, rigorous, and reproducible study.The protocol is established before conducting the systematic review.
  • Search strategy: Five databases were searched: Scopus, Web of Science, IEEE Xplore, the ACM Digital Library, and arXiv.The databases were chosen for indexing coverage, relevance to applied research, and prominence in computer science preprints.
  • Search strategy: The search used broad keywords related to LLM-based AIOps and focused on works published after 2020, followed by manual filtering using exclusion criteria.Broader terms included IT Operations and Pre-trained Model.
  • Selection criteria: Papers were included when they introduced data sources, LLM-based methods, changes in AIOps tasks, or evaluation metrics or datasets tailored to LLM-based AIOps.At least one of four inclusion criteria had to be satisfied.
  • Selection criteria: Studies were excluded for using models smaller than 1 billion parameters, lacking software-system or AIOps relevance, being retracted or unavailable, or presenting only conceptual ideas without experiments.Judgments were primarily based on titles and abstracts, with full-paper review for uncertain cases.
  • Selected publications: 163 studies were selected after retrieving 761 papers, removing duplicates to 614, excluding 333 during screening, and rejecting 222 after detailed reading.The final set satisfied the predefined inclusion criteria.

3 RQ1: TRANSFORMATIONS IN DATA WITH LLM INTEGRATION

LLMs expand AIOps beyond traditional system-generated metrics, logs, and traces by improving preprocessing and incorporating human-generated software information, QA data, and incident reports. Log parsing is especially prominent, while trace synthesis and semantic analysis support downstream failure detection and root-cause analysis.

  • Data sources: The survey examines new techniques for processing traditional data sources and novel data sources introduced for AIOps in the LLM era.Traditional runtime data is complemented by human-generated information.
  • Traditional data sources: Traditional AIOps primarily relies on system-generated metrics, logs, and traces containing intrinsic and extrinsic information about software operation.Metrics, logs, and traces form the three main categories of traditional runtime data.
  • Trace preprocessing: LLM-based trace generation remains limited, but fine-tuning has been used to synthesize realistic workload traces in the form of microservice call graphs.This direction addresses missing or incomplete traces caused by monitoring-infrastructure failures.
  • Log preprocessing: Log preprocessing receives the most attention because logs are unstructured, information-dense, and important for capturing system behavior.LLMs commonly convert raw log messages into structured representations for downstream failure detection and root-cause analysis.
  • Log preprocessing: Log parsing transforms semi-structured messages into event templates with constant fields and variable parameters, after which events are grouped into windows for failure perception and root-cause analysis.The workflow uses session, sliding, or fixed windows to organize parsed events.
  • LLM-based preprocessing methods: Prompt-based methods use in-context learning, demonstrations, caching, and few-shot tuning to guide LLMs in log parsing.LILAC uses hierarchical candidate sampling and an adaptive parsing cache, while LLMParser evaluates four LLMs across 16 open-source systems.
  • LLM-based preprocessing methods: Fine-tuning methods adapt pretrained LLMs for log parsing using supervised, mixture-adapter, or instruction-based tuning.OWL uses the OWL-Instruct dataset with LLaMA, while LogLM uses LLaMA2-7B and seven Loghub datasets.
  • Emerging data sources: LLM-based AIOps incorporates human-generated software information, QA data, and incident reports alongside system-generated data.These sources provide semantic context and auxiliary knowledge for failure management.

4 RQ2: EVOLVING TASKS IN AIOPS WITH LLMS

LLMs are reshaping AIOps from a three-stage workflow of failure perception, root cause analysis, and assisted remediation. Traditional subtasks persist, while new capabilities include root cause report generation and increasingly automated remediation.

  • AIOps comprises three sequential stages: failure perception detects anomalies, root cause analysis identifies their location and nature, and assisted remediation supports mitigation.
  • LLM integration has introduced new subtasks while significantly transforming many traditional AIOps subtasks.
  • Failure Perception: Failure perception includes failure prevention, failure prediction, and anomaly detection, with recent work extending beyond logs and metrics to software-related information.
  • Failure Perception: Anomaly detection remains the most prominent failure-perception research area, with LLMs supporting generalization, smaller-model enhancement, and training-free prediction.
  • Root Cause Analysis: Root cause analysis now includes root cause report generation, which combines failure location and category information with detailed reasoning about underlying causes.
  • Root Cause Analysis: LLM-era root cause analysis increasingly starts from incident reports and supplements them with documentation and code, while some approaches directly generate reports.
  • Root Cause Analysis: Failure category classification is expanding beyond predefined categories through prompting, external knowledge, and foundation models that support adding categories with minimal fine-tuning.
  • Assisted Remediation: Assisted remediation is organized by increasing automation as assisted questioning, solution generation, command recommendation, script generation, and automatic execution.

5 RQ3: LLM-BASED METHODS FOR AIOPS

The survey organizes LLM-based AIOps methods into five groups and examines how foundation models, fine-tuning, prompting, embeddings, and external knowledge support different tasks. Applications span failure detection, forecasting, diagnosis, questioning, and remediation, with model architecture and tuning choices matched to task requirements.

  • LLM-based approach taxonomy: Five approach groups are identified: foundation models, fine-tuning, embedding-based, prompt-based, and knowledge-based approaches.Knowledge-based methods supply external context or specialized tools, while prompt-based methods guide responses through natural-language instructions.
  • Foundation models: Encoder-only models remain useful for detailed and efficient input analysis, including log-based failure diagnosis and metrics-based anomaly detection.LoFI targets log diagnosis, while ART integrates anomaly detection, failure triage, and root cause localization.
  • Foundation models: Decoder-only models support generative and sequential AIOps tasks such as probabilistic forecasting, long-term sequence modeling, and shell-script automation.Examples include Lag-Llama, TimesFM, ShellGPT, and Timer.
  • Fine-tuning: Direct foundation-model application is often suboptimal, motivating domain-specific fine-tuning using metrics, logs, question-answer data, and source code.Fine-tuning includes full and parameter-efficient variants.
  • Fine-tuning: Parameter-efficient tuning reduces computational demands by updating only subsets of model parameters through freezing, adapters, or task-conditional methods.Studies apply these methods to auto-remediation, root cause analysis, assisted questioning, log anomaly detection, and forecasting-based anomaly detection.
  • Prompt-based methods: Prompt-based methods have been used for zero-shot log anomaly detection and instruction-based metrics anomaly detection, but early zero-shot studies reported limited performance without specialization.SigLLM and TabLLM convert time series to text and prompt LLMs to identify anomalies.

6 RQ4: EVALUATING LLM-BASED AIOPS

Evaluation of LLM-based AIOps expands beyond traditional classification and regression measures to generation, execution, and human-centered assessment. The survey also reviews datasets designed specifically for LLM-era AIOps, which are concentrated largely on assisted remediation.

  • Evaluation overview: LLM-based AIOps evaluation uses four metric categories: classification, generation, execution, and manual evaluation.Generation, execution, and manual evaluation emerged as additions in the LLM era.
  • Classification task metrics: Classification metrics include precision, recall, F1-score, accuracy, AUC-ROC, Acc@, and grouping accuracy for classification, localization, anomaly detection, and log parsing.Regression-oriented error metrics include MAE, MSE, RMSE, and MAPE for time-series tasks.
  • Generation task metrics: Generation metrics evaluate textual outputs through lexical similarity measures such as BLEU, ROUGE, and METEOR, or semantic measures such as BERTScore, BLEURT, and NUBIA.Lexical metrics suit standardized reports, while semantic metrics assess meaningfulness and contextual relevance in explanations or mitigation suggestions.
  • Execution task metrics: Execution metrics assess generated scripts at task level through Functional Correctness and Average Correctness, and at workflow level through Execution Success Rate and Correct Refactorings.These measures target assisted remediation, including script generation and automatic execution.
  • Manual evaluation: Manual evaluation is needed when generated outputs lack fixed labels or have flexible ground truth, especially for root cause reports and mitigation solutions.It includes qualitative grading and human judgments of practical utility, relevance, and actionability.
  • Datasets: LLM-specific dataset development is concentrated on assisted remediation, while LogEval evaluates multiple log-analysis tasks using 4,000 public log entries and 15 prompts per task.Only a few datasets target failure prediction and root cause analysis.

7 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies domain-specific barriers involving computational cost, incomplete data-source coverage, uncertain generalization, and weak integration with existing AIOps toolchains. It proposes efficiency improvements, broader validation, system-generated data flows, and modular hybrid architectures as future directions.

  • Challenges: LLM-based AIOps faces domain-specific challenges in efficiency, diverse failure data, software-evolution adaptability, and integration with existing toolchains.General LLM issues such as hallucinations and context limitations are outside the paper’s scope.
  • Time-efficiency and cost-effectiveness: High training, deployment, and inference costs may make smaller models combined with human expertise more practical in resource-constrained settings.The survey recommends balancing computational cost against performance and exploring hybrid approaches.
  • Time-efficiency and cost-effectiveness: Failure perception is especially constrained by LLM computational overhead because it must operate continuously with high real-time performance.With a 10-second detection window, inference must finish within 1 second, and no LLM-based work has adequately addressed this requirement.
  • Failure data sources: Traces remain underused because their high volume, complexity, and nested event sequences are difficult to represent for LLM processing.Metrics and logs have received substantially more attention than traces.
  • Failure data sources: Incident-report-centered root cause analysis can disrupt AIOps automation because root cause analysis should follow system-triggered failure perception.The proposed direction is to connect system-generated detection, incident-report generation, and root cause analysis.
  • Generalizability and adaptability: Empirical validation of LLM generalizability and adaptability across software systems and software evolution remains limited.Future studies should test changing environments and resilience through incremental fine-tuning or continuous learning.
  • Integration with existing AIOps toolchains: The survey argues that LLMs should integrate with smaller models, legacy systems, and specialized tools rather than replace existing AIOps methods.Modular integration is presented as a way to reduce computational burden and operational risk while improving scalability and adoption.

8 CONCLUSION

LLMs are advancing AIOps by reshaping tasks and methodologies, while broader real-world implementation still depends on addressing efficiency, adaptability, data, and integration challenges.

  • LLMs are reshaping AIOps tasks and methodologies through their generative capabilities.The survey examines LLMs with more than 1 billion parameters and their utilization in AIOps.
  • Broader real-world implementation requires improving the time-efficiency and cost-effectiveness of LLM-based AIOps solutions.
  • Future work should deepen the use of diverse data sources and improve generalizability and model adaptability during software evolution.
  • Integration with existing AIOps toolchains remains an important area for further exploration.
  • Addressing these challenges could support more resilient, efficient, intelligent, adaptive, and future-ready software-failure management.
Loading 2507.12472v1…