Source-linked AI summary
Automatic Root Cause Analysis via Large Language Models for Cloud Incidents
Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, Jun Zeng, Supriyo Ghosh, Xuchao Zhang, Chaoyun Zhang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Tianyin Xu
TL;DR
Cloud incident RCA is difficult because manual investigation across heterogeneous data sources is laborious and error-prone. RCACopilot automates incident-specific data collection and uses an LLM to predict and explain root-cause categories. It reaches Micro-F1 0.766 and has had diagnostic collection in Microsoft use for over four years.
Problem
Manual cloud incident RCA requires substantial human effort to collect and interpret complex diagnostic data, while troubleshooting guides can be incomplete or outdated.
Method
RCACopilot matches incidents to alert-specific handlers, collects multi-source diagnostics, and uses an LLM to predict root-cause categories with explanations.
Results
Micro-F1 0.766 and Macro-F1 0.533 were achieved for root-cause category prediction, outperforming all baselines with 4.205 seconds of overhead.
Takeaways & Limitations
RCACopilot demonstrates automated root-cause prediction for unseen incidents and practical deployment of its diagnostic collection component at Microsoft.
Takeaways & Limitations
Troubleshooting guides can remain outdated and lack sufficient detail or coverage, limiting their support for incident diagnosis.
Abstract
from arXiv · showhide
Ensuring the reliability and availability of cloud services necessitates efficient root cause analysis (RCA) for cloud incidents. Traditional RCA methods, which rely on manual investigations of data sources such as logs and traces, are often laborious, error-prone, and challenging for on-call engineers. In this paper, we introduce RCACopilot, an innovative on-call system empowered by the large language model for automating RCA of cloud incidents. RCACopilot matches incoming incidents to corresponding incident handlers based on their alert types, aggregates the critical runtime diagnostic information, predicts the incident's root cause category, and provides an explanatory narrative. We evaluate RCACopilot using a real-world dataset consisting of a year's worth of incidents from Microsoft. Our evaluation demonstrates that RCACopilot achieves RCA accuracy up to 0.766. Furthermore, the diagnostic information collection component of RCACopilot has been successfully in use at Microsoft for over four years.
1 Introduction
Cloud incident RCA is difficult because manual, multi-source investigation is laborious, error-prone, and vulnerable to information overload or scarcity. RCACopilot automates incident-specific diagnostic collection and LLM-based root-cause prediction, with reported Microsoft deployment and evaluation results.
- Manual RCA requires collecting and analyzing logs, metrics, traces, and incident tickets, creating a laborious and error-prone process.
- The information spectrum ranges from too little data to information overload, and both extremes hinder efficient incident resolution.
- Traditional troubleshooting guides are difficult to search, become outdated, and may lack incident-specific coverage as cloud systems evolve.
- RCACopilot lets on-call engineers construct alert-specific incident handlers that collect focused diagnostic information from multiple sources.
- Its LLM analyzes collected diagnostics to predict root-cause categories and generate corresponding explanations.
- The diagnostic information collection component has been used at Microsoft for over four years, while the prediction component was deployed for several months.
2 Background and Motivation
Cloud incident RCA depends on integrating complex, heterogeneous data under time pressure, while traditional troubleshooting guides remain manual, static, and incomplete. The paper motivates an automated pipeline in which experienced engineers define workflows and LLMs assist with diagnosis.
- 2.1 Incident Root Cause Analysis: Cloud incident RCA includes data collection, data analysis, and hypothesis verification to identify underlying causes of service disruption.
- 2.1 Incident Root Cause Analysis: RCA is challenging because cloud systems are dynamic, data-intensive, and require substantial expertise and time.
- 2.2 The Opportunities and Challenges of Multi-Source Data in Incident Management: Traces, logs, and metrics provide complementary views of system state, but their volume, inconsistency, and noise complicate analysis.
- 2.2.3 Limitations of TSGs.: Traditional troubleshooting guides require manual data integration, which is time-consuming, error-prone, and stressful for on-call engineers.
- 2.2.3 Limitations of TSGs.: Troubleshooting guides can become outdated and provide insufficient detail or coverage, forcing additional research and potentially incomplete resolutions.
- The paper proposes an automated diagnostic process using LLMs, with experienced engineers constructing pipelines rather than relying solely on existing guides.
3 Insights from Incidents
The incident study identifies recurring root causes, frequent novel root-cause categories, and the difficulty of diagnosing incidents from limited or outdated guidance.
- Diagnostic challenges: Manual diagnosis can be difficult because a single monitor alert may not reveal the underlying cause of a complex incident.The DNS-resolution example involved exhausted UDP hub ports, which were not apparent from the monitor alert alone.
- Recurring incidents: 93.80% of recurring incidents reappeared within 20 days, indicating that similar root causes often recur quickly.Examples include 11 recurrences in 15 days for invalid customer configuration incidents and 22 DispatcherTaskCancelled incidents within a week.
- New root causes: 24.96% of incidents, or 163 among 653, belonged to new root-cause categories.New causes may lack troubleshooting guides, increasing the challenge of incident analysis.
4 RCACopilot
RCACopilot automates cloud-incident RCA through diagnostic information collection followed by root-cause prediction with explanations for OCE review.
- System overview: RCACopilot has two stages: diagnostic information collection and root-cause prediction.The stages form the system’s end-to-end RCA workflow.
- Diagnostic information collection: The collection stage parses each incident, matches it to a predefined handler for its alert type, and gathers relevant data from multiple sources.Handlers are tailored to specific alert types.
- Root-cause prediction: The prediction stage determines the likely root-cause category, generates an explanatory narrative, and presents the category label to experienced OCEs for review.The output is intended to supplement categorization with the reasoning behind the prediction.
4.1 Diagnostic Information Collection Stage
The diagnostic collection stage uses expert-configured, reusable handler actions to gather focused multi-source evidence, adjust investigation scope, query data, and suggest mitigation.
- Handler design: Incident handlers are alert-type workflows built from reusable actions that collect diagnostic information from diverse sources.OCEs can construct and modify handlers based on their expertise.
- Handler design: Handlers can be updated dynamically when system changes or newly discovered root causes require new diagnostic actions.For example, a new metric can be incorporated by adding an action to the relevant handler.
- Handler actions: Scope switching narrows or widens data collection, enabling fine-grained or holistic investigation according to incident needs.The example moves from forest-level alerts to machine-level analysis and can reverse that direction when broader context is needed.
- Handler actions: Query actions retrieve data or execute predefined scripts, returning results that can determine subsequent workflow branches and team engagement.Queries can identify known issues, aggregate common stack traces, or route incidents based on top error messages.
- Handler actions: Mitigation actions suggest steps such as restarting a service or engaging other teams, but uncertain handlers may provide diagnostic information without mitigation.Predefined handlers may not cover every incident, including difficult code-regression cases.
- Multi-source diagnostics: Combining error logs, exception stack traces, and socket metrics can expose a root cause that individual sources do not reveal.In the example, socket metrics helped identify UDP-socket exhaustion.
- Multi-source diagnostics: Predefined collection actions focus diagnostic output on relevant information, helping OCEs avoid unnecessary data while investigating new incidents.Common checks include provisioning status and thread-stack analysis.
4.2 LLMs for Incident Explanation
RCACopilot uses summarized, semantically and temporally selected incident information to prompt LLM-based root-cause reasoning and explanation. Its approach combines incident embeddings, nearest-neighbor demonstrations, and few-shot chain-of-thought prompting.
- RCACopilot identifies incident root-cause categories as its primary downstream task because manual categorization is time-consuming for on-call engineers.
- 4.2.1 Embedding model: FastText maps diagnostic information into dense embedding vectors whose Euclidean distances represent incident-semantic similarity.
- 4.2.2 Nearest neighbor search: Nearest-neighbor search combines embedding distance with temporal distance and selects the top K incidents from different categories as LLM demonstrations.
- 4.2.3 Diagnostic information summary: Diagnostic information exceeding 2000 tokens is summarized into 120–140 words before diagnosis to reduce noise and improve readability.
- Few-shot chain-of-thought prompting uses summarized diagnostic information and labeled root-cause categories as demonstrations, asking the LLM to select a matching cause and explain it.
4.3 Implementation
RCACopilot was implemented as a substantial software system with a web application for constructing and modifying incident handlers. The interface lets on-call engineers add handlers for new alert types using reusable action nodes.
- RCACopilot comprises 58,286 lines of code, including 56,129 lines of C# and 2,157 lines of Python.
- Its handler-construction web application lets on-call engineers add handlers for new alert types according to their expertise.
- Constructed handlers are stored in a database and can be modified by adding or deleting action nodes.
5 Evaluation
RCACopilot was evaluated on Microsoft’s Transport incidents for prediction effectiveness, component contributions, deployment, and stability. It achieved strong classification performance, supported unseen-category descriptions, and showed practical use across Microsoft teams.
- Effectiveness and Efficiency: RCACopilot outperformed baselines while incurring a low running overhead of 4.205 seconds.The authors attribute the performance to domain-specific knowledge learned from minimal cases through the LLM-based design.
- Effectiveness and Efficiency: 0.766 Micro-F1 and 0.533 Macro-F1 were achieved on the testing dataset for root cause category prediction.The evaluation used 653 incidents from Microsoft’s Transport service, split into training and testing sets.
- Effectiveness and Efficiency: RCACopilot generated a new “I/O Bottleneck” category for an unseen incident whose underlying aspects closely aligned with the later human-derived “DiskFull” label.The system also generated an explanation for this categorization.
- Comparison Analysis: Diagnostic information alone achieved 0.689 Micro-F1 and 0.510 Macro-F1, while adding other information did not improve prediction.The results indicate that excess information can negatively affect LLM prediction performance.
- Comparison Analysis: Summarized diagnostic information improved Micro-F1 by 0.077 and Macro-F1 by 0.023 over non-summarized diagnostic information.The authors report that summarization condenses incident data for more efficient and accurate processing.
- Deployment Status and Scale: The diagnostic collection module was deployed across more than 30 Microsoft teams for over four years, while the prediction module was rolled out in Transport.Average incident-handler runtime ranged from 15 to 841 seconds across the ten teams using the most handlers.
6 Discussion
The discussion identifies deployment and validity boundaries for RCACopilot. Its effectiveness depends on the selected LLM, manually labeled categories, monitor detection, handler availability, and potentially variable LLM behavior.
- Discussion: RCACopilot currently integrates only OpenAI GPT models, so performance may vary with the strengths and weaknesses of the employed LLM.The effectiveness of other available LLMs has not yet been explored.
- Discussion: The prediction evaluation uses manually labeled root cause categories from the Transport service, making label quality a factor influencing effectiveness.The authors identify evaluation across different services as future work for assessing generalizability and adaptability.
- Discussion: RCACopilot is constrained when monitors fail to detect an incident or no corresponding handler exists for its incident type.The handler is activated with 100% accuracy when a designated handler exists for a monitor alert type.
- Discussion: Occasional LLM instability may cause effectiveness to vary across experimental rounds, creating a potential threat to internal validity.The authors conducted three experiment rounds and checked implementations to mitigate this risk.
7 Related Work
Related work frames cloud RCA as a multi-source problem and situates RCACopilot among approaches using individual data sources and LLM-based techniques for software engineering tasks.
- Root Cause Analysis: Cloud RCA research seeks failure and performance root causes from data sources including metrics, logs, and traces.Prior studies proposed approaches relying on one of these data sources.
- Large Language Models: LLM research has applied fine-tuned T5, prompting, and GPT-x models to tasks such as code repair, logging generation, code summarization, and root-cause recommendation.These examples connect RCACopilot’s LLM use to broader software engineering automation work.
8 Conclusion
RCACopilot combines predefined incident handlers for multi-source diagnostic collection with an LLM that analyzes the collected information for cloud incident RCA.
- Conclusion: RCACopilot uses OCE-constructed incident handlers to systematically gather multi-source diagnostic information before subsequent analysis.The diagnostic collection stage is presented as the foundation for the system’s RCA process.
- Conclusion: The system facilitates efficient RCA for on-call engineers by integrating automated collection with large language model analysis.Its approach is positioned as a tool for cloud incident management.