Exploring LLM-based Agents for Root Cause Analysis

Devjeet Roy; Xuchao Zhang; Rashi Bhave; Chetan Bansal; Pedro Las-Casas; Rodrigo Fonseca; Saravan Rajmohan

Source-linked AI summary

Exploring LLM-based Agents for Root Cause Analysis

Devjeet Roy, Xuchao Zhang, Rashi Bhave, Chetan Bansal, Pedro Las-Casas, Rodrigo Fonseca, Saravan Rajmohan

arXiv:2403.04123v1Mar 7, 2024cs.SEcs.CLcs.LG

TL;DR

Cloud RCA is difficult because prior LLM approaches cannot dynamically collect incident diagnostics, despite RCA’s importance and labor demands. This paper evaluates a zero-shot ReAct agent with retrieval, discussion-based augmentation, and team-specific diagnostic tools, finding competitive performance with stronger factual accuracy, limited benefit from discussions, and practical potential in a Microsoft case study.

Problem
Prior LLM-based RCA approaches cannot dynamically collect additional diagnostic information such as logs, metrics, or database data, although RCA requires substantial domain knowledge and experience.
Method
The paper empirically evaluates a zero-shot ReAct agent with retrieval tools, augments historical incidents with discussions, and conducts a case study using team-specific diagnostic services.
Results
ReAct performs competitively with retrieval and reasoning baselines while substantially reducing factual inaccuracies, whereas discussion comments do not clearly improve RCA performance.
Takeaways & Limitations
Agents can overcome prior RCA limitations when given access to external diagnostic services, while static discussion augmentation provides little consistent benefit.
Takeaways & Limitations
The evaluation uses an internal Microsoft dataset, and a random n=500 test-set sample may not reflect performance on the larger test set.

Abstract

from arXiv · show

The growing complexity of cloud based software systems has resulted in incident management becoming an integral part of the software development lifecycle. Root cause analysis (RCA), a critical part of the incident management process, is a demanding task for on-call engineers, requiring deep domain knowledge and extensive experience with a team's specific services. Automation of RCA can result in significant savings of time, and ease the burden of incident management on on-call engineers. Recently, researchers have utilized Large Language Models (LLMs) to perform RCA, and have demonstrated promising results. However, these approaches are not able to dynamically collect additional diagnostic information such as incident related logs, metrics or databases, severely restricting their ability to diagnose root causes. In this work, we explore the use of LLM based agents for RCA to address this limitation. We present a thorough empirical evaluation of a ReAct agent equipped with retrieval tools, on an out-of-distribution dataset of production incidents collected at Microsoft. Results show that ReAct performs competitively with strong retrieval and reasoning baselines, but with highly increased factual accuracy. We then extend this evaluation by incorporating discussions associated with incident reports as additional inputs for the models, which surprisingly does not yield significant performance improvements. Lastly, we conduct a case study with a team at Microsoft to equip the ReAct agent with tools that give it access to external diagnostic services that are used by the team for manual RCA. Our results show how agents can overcome the limitations of prior work, and practical considerations for implementing such a system in practice.

1 INTRODUCTION

Cloud-system complexity makes incidents costly and RCA especially demanding, while prior LLM approaches cannot dynamically collect diagnostic information. This paper evaluates LLM-based agents, beginning with ReAct and retrieval tools, to address that limitation and study practical deployment questions.

Cloud-system complexity makes production incidents costly and increases the importance of prompt mitigation and resolution.
RCA is labor- and skill-intensive because engineers need deep familiarity with a team’s services to identify causes and ensure comprehensive fixes.
Prior LLM approaches cannot dynamically query real-time diagnostics or collect novel incident data, limiting their ability to identify specific root causes.
Adapting agents to RCA is challenging because incident data is confidential, out of distribution, and requires specialized queries over logs, traces, and monitoring data.
The paper evaluates zero-shot ReAct with retrieval tools, augments retrieval with historical discussions, and studies practical considerations through progressively richer settings.

2 BACKGROUND AND RELATED WORK

Incident management addresses costly service disruptions, with RCA requiring experienced engineers to collect and analyze diverse diagnostic data. Prior automation spans statistical, machine-learning, deep-learning, anomaly-detection, knowledge-mining, and LLM-based approaches, but does not solve dynamic diagnostic collection.

Production incidents disrupt customers, making incident management central to software reliability and reducing time from occurrence to resolution.
RCA requires domain expertise to select, collect, and analyze supplementary data spanning structured tables, unstructured logs, and customer reports.
Prior automated approaches cannot collect supplementary diagnostic data or analyze it to produce a root cause.
Automated RCA research includes pattern learning, anomaly detection, monitoring hierarchies, event correlation, and structured knowledge mining from incident artifacts.
Fine-tuned LLMs address confidential incident data through domain adaptation, while augmented language models add retrieval, tools, reasoning, and planning capabilities.

3 LLM-BASED AGENTS FOR RCA

The paper uses ReAct agents that interleave reasoning and tool use for RCA, matching the task’s sequential troubleshooting and knowledge-intensive diagnosis. The implementation combines iterative planner actions with incident and historical-incident retrieval tools in a zero-shot setting.

3 LLM-BASED AGENTS FOR RCA: ReAct interleaves reasoning, planning, and environment feedback, supporting sequential troubleshooting and knowledge-intensive root-cause assessment.
3.1 Overview: A ReAct trajectory repeatedly maps a planner’s thought to a tool action and returned observation until the task is completed.
The study uses a zero-shot prompt because constructing grounded few-shot reasoning traces was difficult when only incident titles and descriptions were available.
Evaluation initially restricts agents to generalized tools because realistic external-diagnostic evaluation lacks a reliable simulated environment and complete records of engineers’ diagnostic steps.
3.4 Tools: The agent can query raw incident descriptions and retrieve historical incidents, with a second variant adding question answering over retrieved documents.

4 RESEARCH QUESTIONS

The research questions assess whether LLM agents can perform RCA without fine-tuning and what practical considerations arise in real-world use. The evaluation covers generalized tools and an in-person test of assistance to on-call engineers.

RQ1 asks how effectively LLM agents find root causes out of distribution using only generalized, team-independent tools.
The study equips the agent with generalized retrieval over historical incidents and question-answering capabilities.
The researchers also conduct an in-person experiment to assess whether the ReAct agent can assist on-call engineers with a small set of incidents.

5 METHODOLOGY

The methodology evaluates untuned language models and a ReAct agent for RCA using retrieved historical incidents, automatic metrics, and qualitative annotation. It combines a production-incident dataset, retrieval-based baselines, and manual coding to assess both similarity and factual performance.

Dataset: 107,000 incidents were split into train, evaluation, and test sets, with 100 evaluation and 500 test incidents sampled for cost reduction.The training split primarily served as the retrieval corpus.
Retrieval: Retrieval uses dense Sentence-BERT with MMR and sparse BM25 methods over historical incident reports.The retrieval corpus encompasses the entire training split.
Baseline Models: The evaluation compares retrieval, chain-of-thought, interleaved retrieval-and-reasoning, and ReAct approaches without fine-tuning.Baselines use historical incident retrieval with a retrieval budget of k=10.
Evaluation Metrics: Models are evaluated with BLEU, METEOR, ROUGE-L, and BERTScore to measure lexical and semantic similarity.BLEU is reported at both corpus and segment levels.
Qualitative Analysis: Because automatic metrics may reward generic or repeated incident descriptions, authors manually code predictions for correctness and factual accuracy.The coding distinguishes precise, imprecise, and factually contaminated correct predictions among other outcomes.

6 RQ1 AND RQ2 RESULTS

RQ1 shows that ReAct achieves competitive correctness with substantially fewer hallucinations than retrieval and chain-of-thought baselines, while discussions added to retrieved incidents do not clearly improve RCA performance.

RQ1: Generalized toolkit: ReAct correctly filters semantically different historical incidents that share lexical similarity with the target, although it also makes reasoning errors and often reports insufficient information.ReAct correctly predicts 4 cases missed by both CoT and RB (k=10), while CoT and RB correctly predict 8 and 9 cases missed by ReAct.
RQ1: Generalized toolkit: 35% correctness for ReAct S+Q BM25 versus 39% for retrieval and chain-of-thought baselines, with hallucinations at 4% versus 12% for CoT and 40% for RB (k=10).ReAct’s higher factual accuracy comes with slightly lower overall accuracy than the strongest baselines.
RQ2: Discussion comments: Discussion comments may have limited value because many report diagnostic outcomes that models cannot reproduce without access to the same diagnostic services and resources.The authors also note that length-based filtering can leave low-quality comments in the corpus.
RQ2: Discussion comments: Adding discussion comments to historical incidents does not clearly improve RCA performance, with lexical metrics improving or degrading depending on the model and metric while semantic metrics remain largely unchanged.For ReAct S+Q BM25, METEOR improves by +0.24, whereas CoT declines on all reported lexical metrics.

7 PRACTICAL IMPLEMENTATION OF RCA AGENT: A CASE STUDY

The case study equips ReAct with team-specific diagnostic tools and shows how it performs on both straightforward and complex incidents. The findings highlight the importance of KBAs, human interaction, and scalable execution for practical RCA.

7 PRACTICAL IMPLEMENTATION OF RCA AGENT: A CASE STUDY: The case study was conducted with Azure Fundamental Team over four weeks to identify diagnostic services and iteratively develop tools for the agent.The work began with understanding the team’s RCA needs and continued through tool development informed by team discussions and historical incidents.
7.3 Agent Development: Team-specific tools replaced generalized tools and provided access to diagnostic data, including database querying and human interaction.The database tool required constructing queries and knowing the relevant cluster and database; human interaction helped supply missing information or correct errors.
7.4.2 Real world RCA using ReAct.: ReAct autonomously completed a straightforward RCA by using a KBA, executing its database query, and assessing the resulting table.It consistently identified the correct case despite lacking prior domain or query-language knowledge, and corrected an initial query failure using tool error messages.
7.4.2 Real world RCA using ReAct.: Complex incidents exceeded ReAct’s practical execution capacity because required information was distributed across multiple KBAs and diagnostic queries.The agent usually completed only one or two steps before the 20-iteration limit, motivating scalable multi-trial frameworks that reuse experience across trials.
Practical considerations: KBAs are critical because they provide specialized procedures and environmental facts, such as database addresses and example queries, needed for RCA.Incident reports generally contain diagnostic outcomes rather than the operational knowledge required to perform the steps.

8 THREATS TO VALIDITY

The evaluation has limited generalizability because it uses Microsoft’s internal data and a budget-constrained sample, while qualitative analysis relies on manual annotations.

8 THREATS TO VALIDITY: The evaluation may not generalize beyond Microsoft because the dataset is internal, and the n=500 test sample may not reflect the larger test set.The sample was randomly selected to reduce costs, but sampling remains a threat to validity.
8 THREATS TO VALIDITY: Manual annotations of model predictions introduce another validity threat, although the authors used prior criteria and repeated discussions to address ambiguity.Two authors conducted multiple rounds of discussion to converge on difficult cases.

9 CONCLUSION & FUTURE WORK

The paper evaluates ReAct for RCA, finding competitive performance with strong baselines and fewer factual inaccuracies, while showing that discussion comments add little on static data. A Microsoft case study demonstrates potential for tool-enabled real-world RCA and motivates simulated environments for future work.

Conclusion: ReAct performed competitively with retrieval-augmented generation and chain-of-thought baselines while producing substantially fewer factual inaccuracies.This result was obtained in an out-of-domain, zero-shot setting.
Conclusion & Future Work: Adding incident-report discussion comments did not significantly affect performance, revealing limitations of conducting RCA on a static dataset.The paper identifies simulated RCA environments as a promising direction for overcoming this limitation.
Conclusion: The case study showed the potential for LLM agents to autonomously perform real-world RCA when given access to appropriate tools.The authors present this as a first step toward practical LLM-based RCA systems.