PaperlayerView exact revision on arXiv

Source-linked AI summary

CloudIR-Bench: A Benchmark for Root Cause Analysis and Mitigation of Cloud Incidents

Hoang Tran, Huong Ha, Tri Dang, Jeffrey Chan, Hongyu Zhang

arXiv:2610.29185v1cs.SE

TL;DR

Incident-report RCA and mitigation prediction lack public datasets despite the promise of LLM approaches. CloudIR-Bench supplies a dataset and evaluation framework from five cloud systems; evaluations show differing model strengths across RCA and mitigation tasks.

  • Problem

    Incident-report RCA and mitigation prediction lack public datasets, limiting reproducibility and transparency in a field where existing datasets are private.

    Source: §I · p. 1

  • Method

    CloudIR-Bench curates incident reports from five cloud systems and provides an open-source evaluation framework for RCA and mitigation prediction.

    Source: §I · p. 2

  • Results

    GPT-4.1 shows the strongest and most consistent RCA performance, while Qwen3-30B-A3B leads BERTScore across all five mitigation datasets.

    Source: 1) RQ1: Performance of LLMs on RCA · p. 7, 2) RQ2: Performance of LLMs on… · p. 7

  • Takeaways & Limitations

    CloudIR-Bench enables reproducible research through a public incident-report dataset and a comprehensive evaluation environment for recent LLM methods.

    Source: §VI · p. 11

    Public post-incident reports vary in completeness and detail, so ground-truth labels may reflect documented information and omit causal factors.

    Source: §V · p. 11

Abstract

from arXiv · show

Root cause analysis (RCA) and mitigation prediction are essential for incident management, enabling the identification of underlying causes of failures and ensuring the reliability of complex cloud-based systems. With rising incident volume and system complexity, incident reports have become a valuable resource for these tasks. Recent studies have shown promising results using Large Language Models (LLMs) to predict root causes and mitigation actions from textual descriptions in incident reports. Despite these advances, further progress is hindered by the lack of publicly available datasets; existing work relies on private data, which limits transparency, reproducibility, and generalization. To address this gap, we introduce CloudIR-Bench, the first publicly available benchmark providing a dataset and evaluation framework for incident report-based RCA and mitigation prediction. Curated from incident reports across five leading cloud-based systems, CloudIR-Bench captures a diverse range of real-world production scenarios representing authentic failures and engineers' reasoning. We also release an open-source evaluation framework that includes eight recent LLM-based approaches. Our comprehensive evaluation includes zero-shot and in-context learning experiments, along with a qualitative analysis to identify where and why LLMs fail. Benchmarking results reveal the capabilities and limitations of existing LLMs in this domain. The benchmark, including the dataset, framework and supplementary materials, is available at https://github.com/hoangtnm/CloudIR-Bench.

I. INTRODUCTION

Cloud incidents remain costly, and incident reports offer contextual information for RCA and mitigation prediction. CloudIR-Bench addresses the lack of public data with a benchmark, evaluation framework, and LLM evaluation.

I. INTRODUCTION

  • Incidents can harm profitability and customer satisfaction, making rapid root-cause identification and effective mitigation important for limiting operational and financial impact.Incident reports capture service interactions, human reasoning, and customer impacts often invisible in raw telemetry.Source: §I · p. 1
  • Existing incident-report RCA research relies on private data, and no public dataset or benchmark supports these tasks.This lack of access limits reproducibility, standardized evaluation, and broader research progress.Source: §I · pp. 1–2
  • CloudIR-Bench introduces the first publicly available benchmark for incident-report RCA and mitigation prediction, curated from five cloud-based systems.It includes long-context incident information and metadata, and represents diverse incident characteristics.Source: §I · p. 2
  • The benchmark pairs an open-source framework using standard LLM inference APIs with an evaluation of recent models’ capabilities and limitations.The framework supports comparisons of LLM baselines for incident-report RCA and mitigation prediction.Source: §I · p. 2

II. BACKGROUND

Cloud incident management moves from detection and triage to diagnosis and mitigation. Incident-report RCA uses contextual text rather than telemetry alone, while LLM research has explored fine-tuning and prompting; the commercial/open-weight performance gap remains unknown for these tasks.

A. Incident Management

B. RCA and Mitigation Prediction for Incidents

  • Incident-report RCA generates root causes and mitigation recommendations from free-form text that captures context, service dependencies, and human observations absent from telemetry alone.Traditional telemetry approaches can miss full incident context needed to understand complex failures.Source: B. RCA and Mitigation Prediction for… · p. 2

C. LLMs for RCA and Mitigation Prediction

  • LLM research has used fine-tuning, in-context learning, and chain-of-thought prompting for incident-report RCA and mitigation prediction, but the commercial/open-weight performance gap on these tasks remains unknown.Earlier work fine-tuned GPT-3.x variants on a private Microsoft incident dataset; later studies explored prompting strategies.Source: C. LLMs for RCA and Mitigation… · p. 3

D. Datasets for RCA and Mitigation Prediction

Incident-report RCA lacks public datasets, and telemetry benchmarks do not contain the textual context needed for this task. CloudIR-Bench targets that gap with an open-source benchmark built from five selected cloud systems.

D. Datasets for RCA and Mitigation Prediction

  • Private incident-report datasets limit reproducibility and transparency, while telemetry benchmarks lack the unstructured context, human reasoning, and cross-service insights needed for incident-report RCA.The two benchmark types address fundamentally different problems.Source: D. Datasets for RCA and Mitigation… · p. 3
  • CloudIR-Bench combines authentic production incident reports from five cloud systems with an open-source framework for fair LLM baseline comparisons.Its analysis examines current LLM capabilities and limitations for RCA and mitigation prediction.Source: D. Datasets for RCA and Mitigation… · p. 3

A. Objectives

  • Dataset objectives prioritize real-world relevance, incident diversity, reproducibility, and automation, with construction code open-sourced for replication and extension.The workflow selects suitable systems, then extracts report information through automated extraction and manual review.Source: A. Objectives · p. 3

B. Selecting Cloud-based Systems

  • The authors selected Azure, Google Cloud, Google Workspace, Atlassian, and GitHub after reviewing candidates for report detail and excluding systems with insufficient detail or too few reports.AWS was excluded because it supplied only 17 reports; CircleCI, Notion, and Alibaba Cloud lacked sufficient root-cause and mitigation detail in most reports.Source: B. Selecting Cloud-based Systems · pp. 3–4

C. Data Collection

The construction process scrapes reports into standardized JSON, removes duplicates and low-information entries, then screens reports for useful root-cause and mitigation content.

C. Data Collection

  • Custom Selenium and Beautiful Soup crawlers collect reports from sources without official APIs and unify extracted information in standardized JSON.Separate scripts accommodate each platform’s HTML layout.Source: C. Data Collection · p. 4

D. Screening and Quality Enhancement

  • The screening process removes duplicates and reports lacking substantive information, including root causes or mitigation steps, and uses system-specific extraction strategies.Reports missing either root-cause or mitigation information are excluded.Source: D. Screening and Quality Enhancement · p. 4

1) De-duplication:

  • System-specific de-duplication removes repeated reports so each incident is represented only once.When reports share an incident ID within a system, the earliest report is retained.Source: 1) De-duplication · p. 4

2) Screening Incident Reports:

  • Keyword-based filtering reduces the report set before manual review verifies relevance and helps avoid excluding valuable reports.The filtering targets reports unlikely to contain relevant root-cause or mitigation information.Source: 2) Screening Incident Reports · p. 4

3) Label Annotation and Consensus Verification:

The section describes annotation and consensus procedures, the final dataset’s scale and coverage, and the open-source framework for evaluating LLMs on incident-report RCA and mitigation prediction.

3) Label Annotation and Consensus Verification:

  • Cohen’s kappa is 0.89–0.91 for Summary, Root Cause, and Mitigation on Azure, Google Cloud, and Google Workspace, and 0.63–0.64 on GitHub and Atlassian.Temporal fields achieve κ of 0.71–1.00 across all datasets; disagreements are resolved through discussion.Source: 3) Label Annotation and Consensus… · p. 5

E. Statistics of the Final Dataset

  • CloudIR-Bench contains 950 production incident reports from five cloud systems, including 911 root causes and 832 mitigation strategies.Atlassian contributes 429 reports, followed by Google Cloud (176), GitHub (144), Azure (120), and Google Workspace (81).Source: E. Statistics of the Final Dataset · p. 5
  • The dataset spans multiple organizations and failure types, including configuration errors, service dependencies, cascading failures, resource exhaustion, software bugs, and infrastructure issues.The multi-organizational coverage is presented as broader than single-organization proprietary datasets while remaining publicly accessible.Source: E. Statistics of the Final Dataset · p. 5

IV. EVALUATION FRAMEWORK

  • The open-source evaluation framework includes eight LLM baselines and supports standard-API inference, preprocessing, and lexical and semantic benchmarking.Its documentation includes instructions for integrating additional baselines and reproducible workflows.Source: §IV · p. 5

A. Evaluation Baselines

The evaluation compares recent closed-source and open-weight LLMs for incident-report RCA and mitigation prediction, using a summarization step before model prediction.

A. Evaluation Baselines

  • The benchmark compares closed-source GPT-4.1 and GPT-5.2, direct successors to GPT-3.5 and GPT-4 variants used in prior work,, with open-weight SWE-bench models selected across sizes and architectures.Examples include DeepSeek-R1-Distill-Qwen-32B, DeepSeek-R1-Distill-Llama-8B, Qwen3-32B, Qwen3-30B-A3B, and Llama-3.1-8B-Instruct.Source: A. Evaluation Baselines · pp. 5–6
  • The evaluation uses the incident title and a GPT-4.1-generated concise summary as inputs for predicting root causes and mitigation.The authors remove leaked sentences before summarization and state that this does not expose information beyond what annotation already filtered.Source: A. Evaluation Baselines · p. 6

B. Evaluation Metrics

The evaluation combines lexical overlap measures with semantic assessment to compare model predictions against ground-truth root causes and mitigation actions.

B. Evaluation Metrics

  • The evaluation uses lexical and semantic metrics because RCA and mitigation prediction require semantic reasoning, while lexical overlap captures matching technical terms.Lexical overlap can reflect whether service names and failure types are identified; semantic similarity captures meaning despite different phrasing.Source: B. Evaluation Metrics · p. 6

1) Lexical Metrics:

  • BLEU-4 measures n-gram overlap through 4-grams, ROUGE-L measures longest-common-subsequence similarity, and METEOR combines unigram precision and recall with stemming and synonym matching.Higher lexical scores indicate more shared technical entities with the label, such as affected services and configuration parameters.Source: 1) Lexical Metrics · pp. 6–7

2) Semantic Metric:

The evaluation uses semantic similarity alongside lexical metrics and reports both model-level findings and limitations of these measures. Across the reported tasks, leading models vary by metric, while mitigation prediction remains difficult.

2) Semantic Metric:

  • BERTScore uses contextualized BERT embeddings and token-level cosine similarity to measure semantic similarity when wording differs.The authors note that lexical and semantic similarity metrics do not always capture causal correctness.Source: 2) Semantic Metric · p. 7

3) Top1 and Top5 Scores:

  • Top1 averages the metric score of each incident’s highest-confidence prediction, whereas Top5 averages the best metric score among its five highest-ranked predictions.Top5 is an upper bound for a downstream selector that can reliably identify the best candidate from that shortlist.Source: 3) Top1 and Top5 Scores · p. 7

1) RQ1: Performance of LLMs on RCA:

  • GPT-4.1 has the strongest and most consistent zero-shot RCA results, while Qwen3-32B is the strongest open-weight model across the reported comparisons.Qwen3-32B leads on several ROUGE-L@Top1 results and BERTScore@Top5 on Google Cloud and GitHub.Source: 1) RQ1: Performance of LLMs on RCA · p. 7
  • DeepSeek-R1-Distill-Llama-8B leads METEOR@Top1 on Azure, Google Cloud, GitHub, and Atlassian, while GitHub is especially challenging with BLEU-4@Top1 below 1.3 for every model.The authors also report modest absolute RCA scores across datasets.Source: 1) RQ1: Performance of LLMs on RCA · p. 7

2) RQ2: Performance of LLMs on Mitigation Prediction:

  • GPT-4.1 leads mitigation METEOR@Top1 on all five datasets, while Qwen3-30B-A3B leads BERTScore across all five and ROUGE-L@Top1 on four.Qwen3-30B-A3B leads ROUGE-L@Top1 on Google Cloud, Google Workspace, GitHub, and Atlassian; GPT-4.1 leads it on Azure.Source: 2) RQ2: Performance of LLMs on… · p. 7
  • Low BLEU-4@Top1 scores reflect mitigation’s difficulty: actions require system-specific reasoning not available from incident symptoms, and models often produce generic rather than context-specific steps.The authors identify a capability gap in predicting mitigation accurately.Source: 2) RQ2: Performance of LLMs on… · p. 7

3) RQ3: Performance of In-Context Learning with LLMs:

In-context learning (ICL) consistently improves semantic similarity across datasets, but its effect on lexical overlap varies. Gains are more consistent for mitigation prediction than for root cause analysis (RCA).

RQ3: Performance of In-Context Learning with LLMs

  • The comparison uses GPT-4.1 and GPT-5.2 with ICL examples drawn from 70% of each dataset and evaluates on the remaining 30%.For each validation incident, the five most relevant incidents were retrieved by semantic similarity; the supplied Table IV caption covers GPT-4.1 Top1 scores.Source: 3) RQ3: Performance of In-Context… · pp. 7–8, Table IV · p. 9

RCA Insights

  • ICL strengthens semantic alignment across datasets, but its effect on lexical overlap is inconsistent and dataset-dependent for RCA.For GPT-4.1, Azure shows the clearest gains; Google Cloud and GitHub have marginal BERTScore@Top1 improvements but BLEU-4@Top1 drops to 0.Source: 3) RQ3: Performance of In-Context… · p. 8

Mitigation Prediction Insights

  • ICL benefits mitigation more consistently than RCA, with GitHub showing the biggest improvement overall and Azure also improving on reported metrics.The authors suggest mitigation’s structured, action-oriented nature makes it more receptive to ICL than RCA’s varied, context-specific descriptions.Source: 3) RQ3: Performance of In-Context… · pp. 8–9

Key Findings

  • ICL yields more consistent gains for mitigation prediction than for RCA, with GitHub showing the biggest overall mitigation improvement.For RCA, Google Cloud and GitHub show mixed results: semantic scores improve slightly while lexical scores decline.Source: 3) RQ3: Performance of In-Context… · p. 9

4) RQ4: Qualitative Analysis of LLM Failure Modes:

The qualitative analysis examines sampled RCA and mitigation predictions, classifying their alignment and error types. It finds modest correctness, with hallucination and reasoning failures prominent, and shows that high textual-overlap scores can coexist with an incorrect root cause.

Qualitative Analysis

  • The analysis samples 25 incidents from each of five cloud systems and manually classifies predictions from three top baselines.Annotators first assessed semantic alignment, then classified correct predictions as precise, imprecise, or hallucinated, and incorrect ones by error type.Source: 4) RQ4: Qualitative Analysis of LLM… · pp. 9–10

RCA Insights

  • RCA correctness is 26.4%; errors are dominated by hallucinations and faulty reasoning, while correct predictions often lack technical precision.Qwen3-32B has the highest reasoning-error rate, GPT-4.1 leans toward hallucinations, and DeepSeek-R1 balances the two error types.Source: 4) RQ4: Qualitative Analysis of LLM… · p. 10

Mitigation Prediction Insights

  • Mitigation correctness is 25.87%, with hallucination the dominant failure mode, often triggered by insufficient summaries that prompt unrelated actions.Qwen3-32B has the highest hallucination rate; GPT-4.1 has fewer hallucinations but more reasoning errors, while DeepSeek-R1 falls between them.Source: 4) RQ4: Qualitative Analysis of LLM… · p. 10

Key Findings

  • High lexical and semantic scores can mask an incorrect root cause because overlap metrics reward shared wording and models may reproduce consequences rather than underlying causes.The illustrated prediction names the affected file-upload component but attributes the cause to a generic backend issue instead of the Global Service Load Balancer configuration change.Source: 4) RQ4: Qualitative Analysis of LLM… · p. 10, Figure 4 · p. 10

V. THREATS TO VALIDITY

The evaluation’s validity is bounded by uncertainty in report-based labels, imperfect textual metrics, possible pretraining contamination, and a fixed model set. The conclusion presents CloudIR-Bench as a reproducibility resource and identifies several directions for future work.

V. THREATS TO VALIDITY

  • Because annotators lacked direct engagement with system owners, some labels may reflect subjective interpretations or misalign with incident intent.Two independent annotators and consensus discussions were used to resolve discrepancies; the evaluation also generated 10 predictions per incident and assessed methods with four metrics and qualitative analysis.Source: §V · p. 11
  • Public incident reports vary in detail and may omit causal factors, so ground-truth labels reflect what engineers document under operational and disclosure constraints.The authors characterize authentic reports as a practical choice that preserves real-world variability, ambiguity, and incompleteness.Source: §V · p. 11
  • Lexical and semantic metrics capture textual overlap and similarity but do not necessarily measure whether RCA and mitigation predictions are correct.The study complements these metrics with qualitative analysis and identifies LLM-as-a-judge as a future evaluation approach.Source: §V · p. 11
  • Because reports are public and pretraining data are undisclosed, the authors cannot entirely rule out evaluated LLMs having encountered similar reports during training.They note that the models’ relatively low performance suggests predictions are unlikely to be primarily driven by memorization of the original reports.Source: §V · p. 11
  • The evaluation covers a fixed set of models selected from widely adopted, top-performing SWE-bench models at the time of the study.The authors plan to extend evaluation to stronger models through standard inference APIs.Source: §V · p. 11

VI. CONCLUSIONS

  • The conclusion presents CloudIR-Bench as a public benchmark for reproducible incident-report RCA and mitigation research, with fine-tuning, RAG, agent-based reasoning, and LLM-as-a-judge planned for future exploration.The benchmark includes reports from five cloud-based systems, metadata, and ground-truth labels for root causes and mitigation strategies.Source: §VI · p. 11

Related papers

Loading 2610.29185v1…