PaperlayerView exact revision on arXiv

Source-linked AI summary

Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models

Toufique Ahmed, Supriyo Ghosh, Chetan Bansal, Thomas Zimmermann, Xuchao Zhang, Saravan Rajmohan

arXiv:2301.03797v2cs.SEcs.LG

TL;DR

Cloud incident diagnosis and mitigation require manual effort and service-specific knowledge. This study evaluates GPT-3.x on more than 40,000 Microsoft incidents and reports gains from fine-tuning and stronger performance than encoder-decoder models, alongside incident-owner validation.

  • Problem

    On-call engineers still spend substantial manual effort and need domain knowledge to root-cause and mitigate production incidents.

    Source: §I · p. 1, C. Root-causing and mitigating incidents · p. 2

  • Method

    The authors evaluate GPT-3.x and baseline models on more than 40,000 incidents using semantic and lexical metrics, with actual incident owners assessing recommendations.

    Source: §I · p. 1, A. Methodology · p. 9

  • Results

    GPT-3 and GPT-3.5 significantly outperform encoder-decoder models, fine-tuning significantly improves effectiveness on incident data, and owner evaluation supports the approach’s efficacy.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The authors conclude that GPT-3 and GPT-3.5 are effective for helping identify incident root causes and mitigation steps in their Microsoft study.

    Source: §VIII · p. 11

    The models were trained and evaluated on services within Microsoft, so their behavior may differ on services at other organizations.

    Source: D. Threats to Validity · p. 10

Abstract

from arXiv · show

Incident management for cloud services is a complex process involving several steps and has a huge impact on both service health and developer productivity. On-call engineers require significant amount of domain knowledge and manual effort for root causing and mitigation of production incidents. Recent advances in artificial intelligence has resulted in state-of-the-art large language models like GPT-3.x (both GPT-3.0 and GPT-3.5), which have been used to solve a variety of problems ranging from question answering to text summarization. In this work, we do the first large-scale study to evaluate the effectiveness of these models for helping engineers root cause and mitigate production incidents. We do a rigorous study at Microsoft, on more than 40,000 incidents and compare several large language models in zero-shot, fine-tuned and multi-task setting using semantic and lexical metrics. Lastly, our human evaluation with actual incident owners show the efficacy and future potential of using artificial intelligence for resolving cloud incidents.

I. INTRODUCTION

The paper studies whether large language models can help with incident root-cause analysis and mitigation, tasks that still demand substantial manual effort. It evaluates these models at scale and reports stronger performance from fine-tuning and GPT-3.x than from encoder-decoder models.

  • On-call engineers still spend substantial manual effort identifying incident root causes and mitigation steps despite prior automation work.The paper frames these two tasks as its research focus.Source: §I · p. 1
  • The study evaluates models on more than 40,000 incidents from over 1,000 cloud services and uses six semantic and lexical metrics.The authors also validate recommendations with actual incident owners.Source: §I · p. 1
  • GPT-3 and GPT-3.5 significantly outperform encoder-decoder models in the experiments.The authors also report that fine-tuning significantly improves effectiveness on incident data.Source: §I · p. 1
  • The authors report that a human study with actual incident owners supports the efficacy and future potential of AI for resolving cloud incidents.Source: §I · p. 2

II. OVERVIEW

The overview situates incident diagnosis and mitigation within a costly, multi-stage cloud incident lifecycle. It introduces GPT-3.x as a potential aid and evaluates recommendations using information available when an incident is created.

A. Incident management

  • Diagnosis requires iterative engineering communication, while mitigation follows root-cause identification to restore service health and limit user impact.AIOps tools automate parts of the lifecycle, but root-cause identification and mitigation remain effort-intensive.Source: A. Incident management · p. 2

B. The promise of LLMs/GPT-3.x models

  • GPT-3.x models are presented as large language models with broad language-task capabilities and text-generation use cases.The overview describes their scale and prior performance across NLP tasks.Source: B. The promise of LLMs/GPT-3.x models · p. 2

C. Root-causing and mitigating incidents

  • Root causes and mitigations are difficult to identify because incidents have varied causes and remedies, and the work requires service knowledge and experience.The evaluation generates recommendations from the incident title and summary at creation, before later investigation details become available.Source: C. Root-causing and mitigating incidents · pp. 2–3

D. Research questions

The research questions examine GPT-3.x effectiveness for root-cause and mitigation generation, then test how training choices and incident information affect performance. They also ask whether performance varies by incident type.

  • RQ1 and RQ2 ask whether fine-tuned GPT-3.x models can identify incident root causes and recommend mitigation plans.The models are fine-tuned because the OpenAI models were not trained on incident-management data.Source: D. Research questions · p. 3
  • RQ3 and RQ4 compare fine-tuning with zero-shot learning and single-task with multi-task learning.Few-shot learning is excluded because long sequences were truncated when inferring one sequence after fine-tuning; the multi-task experiment combines training data for both tasks.Source: D. Research questions · p. 3
  • RQ5 tests whether supplying a root cause helps GPT-3.x propose mitigation plans, which depend on the specific root cause.Source: D. Research questions · p. 3
  • RQ6 asks whether mitigation plans differ in performance for machine-detected and human-detected incidents.The paper notes that the two incident types have different characteristics.Source: D. Research questions · p. 3

E. Human validation

The paper uses incident-owner judgments to complement automatic evaluation and builds its dataset from historical incident records. It also describes the pretrained model families used as baselines and notes constraints on the incident data.

E. Human validation

  • Because root causes and mitigation steps are open-ended, the authors ask incident owners to assess model outputs for correctness and readability.Automatic metrics compare suggestions with a single reference, which may differ from another correct, relevant output.Source: E. Human validation · p. 3

A. Dataset Preparation

  • The dataset uses incident summaries from creation time to avoid input-output data leakage, but the free-form records may include tables and images that were discarded.Discarding those non-text elements may remove critical information.Source: A. Dataset Preparation · p. 3
  • After deduplication and filtering, the authors retained 35,820 root causes and 5,455 mitigation plans for training, then manually selected useful test examples.The study covers incidents created from January 2018 through July 15, 2022.Source: A. Dataset Preparation · p. 4

B. OpenAI models and baselines

  • The incident dataset cannot be shared because it may contain confidential and private information.Sharing it would violate the terms of service.Source: B. OpenAI models and baselines · p. 4
  • The baseline discussion describes pretrained NLP and code models as Transformer-based models trained with different pretraining objectives.The paper notes that encoder-decoder models encode incident input before generating a root cause or mitigation.Source: B. OpenAI models and baselines · p. 4

1) Baselines encoder-decoder models:

The study compares natural-language and code models as incident-management baselines, then configures GPT-3.x models for root-cause and mitigation generation. It uses parameter-efficient fine-tuning and tunes generation temperature to balance creative recommendations against divergence.

1) Baselines encoder-decoder models:

  • RoBERTa and CodeBERT provide architecturally matched 125M-parameter NLP and code baselines, respectively.The authors select them partly because CodeBERT is a widely used baseline for CodeXGLUE software-engineering tasks.Source: 1) Baselines encoder-decoder models · p. 4
  • Four OpenAI models—Curie, Codex-cushman, Davinci, and Code-davinci-002—span GPT-3.x and GPT-3.5, with natural-language and code pretraining represented.The code-model choice reflects overlap between incident descriptions and software identifiers; the study does not compare every available encoder-decoder model.Source: 2) OpenAI generative models · p. 5

C. Model configuration

  • Encoder-decoder models generate root causes or mitigations from encoded inputs, but their 512-token limit is shorter than the GPT models’ 2,048–4,000-token contexts.The authors report that some test samples are truncated in GPT models and infer that traditional encoder-decoder models may lack capacity for these sequences.Source: 1) Baselines encoder-decoder models · p. 4, C. Model configuration · p. 5
  • LoRA fine-tuning freezes pretrained weights and adds trainable rank-decomposition matrices; GPT-3 and GPT-3.5 models were trained for 2,000 steps, or four epochs.The authors use one V100 GPU for Curie and Codex, four V100 GPUs for Davinci, and four A100 GPUs for Code-davinci-002.Source: C. Model configuration · pp. 5–6
  • Temperature was selected by grid search to limit divergence across five generated samples: 0.7 for Curie, Codex, and Davinci, and 0.5 for Code-davinci-002.Higher temperature encourages risk-taking, while lower values approach argmax sampling.Source: C. Model configuration · p. 6

D. Evaluation Metrics

The evaluation combines lexical overlap measures with semantic measures because lexical metrics can disregard meaning. Semantic scoring is computationally expensive, so it is limited to the first two research questions.

1) Lexical Metrics:

  • BLEU-4, ROUGE-L, and METEOR provide lexical evaluation through n-gram overlap, longest common subsequence structure, and unigram precision-recall with stemming and synonym matching, respectively.ROUGE-L uses longest common subsequence statistics.Source: 1) Lexical Metrics · p. 6

2) Semantic Metrics:

  • BERTScore, BLEURT, and NUBIA assess semantic similarity or meaning-related qualities that lexical metrics may miss.BERTScore matches contextual word embeddings by cosine similarity; BLEURT assesses fluency and conveyed meaning, while NUBIA combines semantic similarity, logical inference, and legibility.Source: 2) Semantic Metrics · p. 6

D. Evaluation Metrics

  • Semantic metric computation took two days on one GPU for Tables I and II, so the authors report semantic metrics only for RQ1 and RQ2.The remaining research questions use less computationally expensive lexical metrics.Source: 2) Semantic Metrics · p. 6

IV. RESULT

The results distinguish automatic scores from incident-owner judgments: GPT-3.5 leads the fine-tuned models on root-cause metrics, while encoder-decoder models can score well despite generic outputs. Zero-shot performance is lower than fine-tuned mitigation performance, and owner assessments do not always align with automatic metrics.

A. How effective are fine-tuned GPT-3.x models in generating incidents’ root cause recommendation? (RQ1)

  • Encoder-decoder models perform well on all six automatic metrics, but generic repeated outputs help their scores; RoBERTa and CodeBERT produce unique candidates in only 6.10% and 16.67% of cases.The paper notes that the common phrase “There is a bug in the code” can inflate overlap-based scores, despite the models’ generic recommendations.Source: A. How effective are fine-tuned… · p. 6
  • GPT-3.5 Code-davinci-002 significantly outperforms all GPT-3 models and other baselines on all six automated root-cause metrics.Table I reports fine-tuned GPT-3.x effectiveness for root-cause recommendation.Source: A. How effective are fine-tuned… · p. 7, Table I · p. 7
  • Automatic metrics can miss weaknesses in generic recommendations and may undervalue useful open-ended answers that differ from the single reference.Incident-owner interviews found that suggestions with lower lexical scores could still be helpful.Source: A. How effective are fine-tuned… · p. 7

B. How effective are fine-tuned GPT-3.x models in recommending mitigation plans for an incident? (RQ2)

  • Mitigation BLEU-4 scores of 4.44–6.76 exceed root-cause scores of 3.38–4.24, yet incident owners rate Davinci and Code-davinci-002 as the best-performing models.Fine-tuned Davinci underperforms other baselines on automatic metrics for both tasks, illustrating the difference between metric rankings and owner assessments.Source: B. How effective are fine-tuned… · p. 7

C. How much fine-tuning improves over zero-shot learning performance of GPT-3.x models? (RQ3)

  • Zero-shot top-candidate BLEU-4 is 0.80–2.18 for mitigation, compared with 5.47–6.76 after fine-tuning; Davinci outperforms other OpenAI models in zero-shot root-cause and mitigation settings.The authors attribute Davinci’s zero-shot advantage to its larger parameter count and greater training-data volume.Source: C. How much fine-tuning improves over… · p. 7

D. Does multi-task learning improve the performance of GPT-

The additional experiments test whether combining tasks, supplying root causes, or separating incidents by detection source changes mitigation performance. Multi-task training generally does not improve results, whereas root-cause context and machine-detected incident patterns are associated with better scores in the reported comparisons.

3.x models at finding root causes and mitigation plans? (RQ4)

  • Multi-task training does not significantly outperform single-task training; mitigation scores decline by an average of 4.1% across the four models.Curie and Codex also decline across reported lexical metrics, Davinci is marginally better on BLEU-4, and Code-davinci-002 is almost always lower in the multi-task setting.Source: 3.x models at finding root causes and… · p. 7, Table V · p. 8

E. Do GPT-3.x models get better at proposing mitigation plans if the root cause is given? (RQ5)

  • With root causes supplied, mitigation performance improves across all four models: 9.8% for Curie, 8.3% for Codex, 5.4% for Davinci, and 26% for Code-davinci-002.The mitigation training set shrinks from 5,455 to 2,973 because some incidents lack root causes, and the test set contains 166 samples; Code-davinci-002’s Top-5 gain is modest relative to its Top-1 gain.Source: E. Do GPT-3.x models get better at… · pp. 7–8, Table VI · p. 8

F. Do the models better propose mitigation plans for machine-detected incidents than human-detected ones? (RQ6)

  • For Code-davinci-002 Top-1 mitigation recommendations, machine-detected incidents outperform human-detected incidents by 9.5% on BLEU-4, 20% on ROUGE-L, and 23% on METEOR.The authors report 592 machine-recognized and 1,188 human-detected test incidents and note that machine-detected incidents tend to follow patterns easier for models to recognize.Source: F. Do the models better propose… · p. 8

V. LOOKING THROUGH THE INCIDENT OWNERS’ EYES

Incident owners evaluated model outputs for correctness, readability, and usefulness, finding stronger performance from some models alongside shortcomings tied to missing incident information. Their feedback suggests promise while underscoring that outputs are not consistently helpful.

A. Methodology

  • Owners evaluated 50 recent incidents, with 25 owners responding to interviews lasting 20–30 minutes.Owners assessed candidate correctness and readability on 1–5 scales; correctness concerned helpfulness and relevance, while readability concerned ease of understanding.Source: A. Methodology · pp. 8–9

B. Results

  • Davinci and Code-davinci-002 scored higher in mean correctness than Curie and Codex for root causes (2.88 and 2.56) and mitigation plans (3.04 and 3.16).Incident owners assigned the correctness and readability scores summarized in Table VIII.Source: B. Results · p. 9
  • Owners said Davinci and Code-davinci-002 were more correct than Curie and Codex, while readability scores exceeded correctness scores across models.OpenAI models’ correctness scores ranged from 2.28 to 3.16; owners also noted generic recommendations from encoder-decoder models.Source: B. Results · p. 9
  • Owners described cases where models found hard-to-detect root causes and said suggestions could guide on-call engineers even when not always correct.They also recommended improving readability, including by avoiding acronyms and adding more specific comments.Source: B. Results · p. 9
  • More than 70% of incident owners gave the model recommendations an overall usefulness score of three or above.The authors found at least one model effective for most incidents.Source: B. Results · p. 9
  • Some outputs were unhelpful when incident records lacked discussion or other valuable signals, and owners warned that incorrect suggestions could prolong investigation.Owners also cited unrecorded or classified incident data as a concern.Source: B. Results · p. 9

C. Two illustrative examples

The section presents two examples comparing owner-written incident texts with generated recommendations. It notes that textual differences help explain why owners found the suggestions promising while automatic metrics did not.

C. Two illustrative examples

  • Two examples compare on-call engineers’ incident texts with fine-tuned Code-davinci-002 outputs for root causes and mitigation plans.The authors describe the generated suggestions as relevant and valuable despite wording differences.Source: C. Two illustrative examples · p. 9
  • Table VIII concerns owner-assigned correctness and readability, while Table IX concerns perceived usefulness for incident resolution.Source: Table VIII · p. 10, Table IX · p. 10
  • Textual dissimilarities between owner-written and generated texts help explain why owners found GPT-3.x promising while automatic metrics did not.Source: C. Two illustrative examples · pp. 9–10

VI. DISCUSSION & THREATS

The discussion questions whether lexical metrics reflect incident owners’ judgments, compares natural-language and code-model families, and identifies ways to improve model performance. The authors caution that more cases are needed for firm conclusions about metric alignment.

A. Do automatic metrics reflect human perception?

  • Pearson correlations between BLEU-4, ROUGE-L, and METEOR and owners’ correctness and readability scores ranged from -0.42 to +0.62, with no specific pattern; more cases are needed for a concrete conclusion.The authors note challenges in human evaluation, including experts’ difficulty evaluating incidents they did not resolve and owners’ limited recall of incidents from two months earlier.Source: A. Do automatic metrics reflect human… · p. 10

better for incident management?

  • The study found no winning model family, though Davinci and Code-davinci-002 produced more correct and readable suggestions than other models; both have 175 billion parameters.Fine-tuning larger code models or pre-training on incident data was left for future research.Source: better for incident management? · p. 10

C. How the models’ performance can be improved?

  • Owners recommended incorporating on-call discussions, while the study excluded later-updated summaries to prevent data leakage and lost information by removing images and tables.The authors say incorporating these sources after addressing leakage and data cleaning may improve performance.Source: C. How the models’ performance can be… · p. 10

D. Threats to Validity

The study reports GPT-3 and GPT-3.5 effectiveness for incident root-cause identification and mitigation, while noting limits in evaluation models, organizational scope, rare incidents, and human-study sample size.

D. Threats to Validity

  • Semantic evaluation used default pretrained models; organizational service coverage constrains transfer, owners raised rare-incident concerns, and small human-study samples make statistical significance difficult.The authors note that incident-management-pretrained models could change evaluation performance, while scaling the human study is challenging.Source: D. Threats to Validity · p. 10

A. Incident management

  • Prior incident-management research analyzed issue- or service-specific incidents – and automated triage –, diagnosis –, and mitigation; this is the first effort using state-of-the-art language models for incident resolution.The authors hope to motivate merging task-specific discriminative models with LLMs for end-to-end incident automation.Source: A. Incident management · pp. 10–11

B. LLMs in Software Engineering

  • Unlike code-generation work with lexical and structural information and extensive training data, the study examines incident resolution with LLMs, which had not previously been explored.Prior LLM work in software engineering included code generation [8], docstring generation [8],, code repair [62], [63], and few-shot tasks [64].Source: B. LLMs in Software Engineering · p. 11

VIII. CONCLUSION

  • GPT-3 and GPT-3.5 were effective for identifying root causes and mitigation steps in a Microsoft study of over 40,000 incidents that involved actual incident owners.The authors plan production deployment and exploration of incident summarization as next steps.Source: §VIII · p. 11
Loading 2301.03797v2…