PaperlayerView exact revision on arXiv

Source-linked AI summary

Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems

Yilun Wang, Guangba Yu, Haiyu Huang, Yujie Huang, Zirui Wang, Pengfei Chen, Michael R. Lyu

arXiv:2603.00468v2cs.SE

TL;DR

Cloud RCA benchmarks must evaluate both diagnostic correctness and the evidence acquired through interactive investigation, but existing static and live paradigms leave this combination incomplete. Cloud-OpsBench uses runtime-verified state snapshots, replayed diagnostic interfaces, and evidence graphs to support matched process-and-outcome evaluation. Across ten agents, final-answer accuracy substantially exceeds evidence closure, revealing a persistent outcome–process gap.

  • Problem

    Existing benchmarks separately trade off reproducible inputs, system-facing interaction, controlled comparison, and process-level evidence assessment.

  • Method

    Cloud-OpsBench combines runtime-verified fault cases, snapshot-backed diagnostic replay, outcome labels, and milestone-based evidence graphs for interactive RCA evaluation.

  • Results

    The strongest JRA reaches 0.76 on OnlineBoutique and 0.68 on TrainTicket, while corresponding ECR values are 0.38 and 0.15 across ten agents.

  • Takeaways & Limitations

    Outcome-only correctness substantially overestimates agents’ ability to perform evidence-grounded diagnosis.

  • Takeaways & Limitations

    Cloud-OpsBench focuses on Kubernetes-based cloud-native systems, so broader validation requires additional workloads, deployment configurations, and real-world incidents.

Abstract

from arXiv · show

LLM agents are increasingly explored for automating root cause analysis (RCA) in cloud-native systems, creating a need to evaluate both diagnostic correctness and the quality of the supporting investigation. Existing static benchmarks offer repeatable inputs but limited system-facing interaction, while live testbeds expose realistic tools but hinder controlled comparison because incident evidence varies across runs; both paradigms focus primarily on final answers. To address these limitations, we present Cloud-OpsBench, an evaluation infrastructure for interactive and evidence-grounded cloud RCA. It comprises 754 runtime-verified cases across 57 fault types on two microservice workloads spanning application services and Kubernetes platform layers. Each fault is captured as a state snapshot and replayed through standard diagnostic interfaces, with outcome labels and diagnostic evidence graphs that enable matched comparisons and process-level analysis. Across ten LLM agents, the strongest Joint RCA Accuracy (JRA) reaches 0.76 on OnlineBoutique and 0.68 on TrainTicket, while the corresponding Evidence Closure Rates (ECR) are only 0.38 and 0.15. This outcome--process gap shows that final-answer correctness alone substantially overestimates agents' ability to perform evidence-grounded diagnosis.

1 Introduction

Cloud-OpsBench addresses limitations in existing cloud RCA benchmarks by combining reproducible, interactive investigation with outcome and evidence-grounded process evaluation. Across ten LLM agents, results reveal a substantial gap between correct final diagnoses and complete evidential support.

  • Motivation: Existing benchmarks lack a joint evaluation of adaptive system-facing investigation, controlled incident conditions, and evidence-grounded diagnosis.Static benchmarks provide reproducible inputs without interactive tool selection, whereas live environments vary across runs and primarily score outcomes.
  • Benchmark: Cloud-OpsBench contains 754 runtime-verified cases spanning 57 fault types across application services and Kubernetes platform layers.The benchmark uses OnlineBoutique and TrainTicket workloads and retains cases after live verification.
  • Benchmark: Each verified incident is captured as a state snapshot and replayed through diagnostic interfaces, enabling agents to choose tools and targets under matched observations.Cases pair root-cause labels with milestone-based evidence graphs for process-level scoring.
  • Results: 0.76 JRA on OnlineBoutique and 0.68 JRA on TrainTicket were the strongest results, while corresponding ECR values were only 0.38 and 0.15.The outcome–process divergence indicates that correct final answers do not necessarily reflect complete evidence-grounded diagnosis.
  • Results: Trajectory-as-ICL raises DeepSeek-V4-Flash from 0.76 to 0.83 JRA and from 0.38 to 0.57 ECR on OnlineBoutique, while TrainTicket effects are smaller and mixed.SOP prompting yields similar gains, and failed trajectories cluster into four recurring categories.

2 Motivation

Agentic RCA requires diagnosing failures through adaptive tool-mediated evidence collection, but existing benchmarks separately trade off interaction, reproducibility, and process measurement. Cloud-OpsBench motivates matched snapshot-based evaluation that complements final-answer accuracy with investigation quality.

  • Agentic RCA: Cloud-native RCA spans application code, deployment specifications, runtime configurations, service dependencies, and infrastructure states.A symptom at one service may originate in another component or system layer.
  • Agentic RCA: Agents select system objects, logs, configurations, and dependencies to inspect, so identical root-cause labels can differ in evidential support.Operational knowledge may assist interpretation, but accepted diagnoses must remain grounded in case-specific tool observations.
  • Evaluation gaps: Static benchmarks provide reproducible artifacts but omit adaptive system-facing evidence collection.Agents reason over supplied files rather than selecting tools, targets, and subsequent queries from returned observations.
  • Evaluation gaps: Live benchmarks preserve interaction, but runtime manifestations vary in severity, duration, and affected scope, preventing matched pairwise comparison across separate runs.Repeated trials estimate averages and variance but cannot ensure identical fault conditions.
  • Evaluation gaps: Outcome-only metrics give equal credit to supported diagnoses and unsupported guesses while obscuring whether an investigation establishes a causal explanation.Process measures should assess completeness, causal coherence, and efficiency over observable tool-mediated actions.
  • Requirements: Cloud-OpsBench therefore requires system-facing evidence acquisition and matched, reusable incident evidence through deterministic diagnostic interfaces.Its scope is interactive investigation over preserved, runtime-verified evidence rather than live remediation or open-ended environment mutation.

3 Cloud-OpsBench

Cloud-OpsBench combines runtime-verified fault cases with snapshot-backed interactive evaluation and evidence-graph process labels. It spans application and Kubernetes layers while supporting reproducible comparisons without prescribing one diagnostic workflow.

  • Benchmark scope: 754 fault cases span 57 fault types across application services and Kubernetes platform layers.The benchmark uses OnlineBoutique and TrainTicket workloads in a Kubernetes testbed.
  • Snapshot-backed evaluation: Cloud-OpsBench separates live fault realization from agent evaluation by capturing verified incident evidence as state snapshots.Snapshots preserve tool-observable context from real fault realizations while enabling controlled replay.
  • Snapshot-backed evaluation: Agents investigate through diagnostic interfaces that replay snapshot-backed observations while the harness records their tool-use trajectories and final diagnoses.The snapshot is hidden from agents, who receive a user query and issue concrete diagnostic tool calls.
  • Fault construction: The benchmark constructs fault categories from Kubernetes documentation, operational communities, and academic literature, then validates case-specific instantiations at runtime.Candidate specifications receive consistency checks and practitioner review before runtime validation.
  • Evidence-grounded annotation: Diagnostic evidence graphs score established facts and dependencies rather than a canonical command sequence, allowing alternative evidence sources and investigation orders.Evidence bundles may contain a single sufficient response or complementary responses jointly establishing a fact.

4 Evaluation

Cloud-OpsBench shows that agents can achieve strong RCA outcome scores while remaining incomplete, shallow, or inefficient in evidence-grounded investigation. Performance varies by workload, fault difficulty, model capability, and experience-reuse method.

  • RQ1: RCA Outcome Effectiveness: Agents often localize affected Kubernetes components without identifying the underlying fault mechanism.On OnlineBoutique, DeepSeek-V4-Flash achieves 0.88 CA, 0.77 FA, and 0.76 JRA; GPT-5 achieves 0.86 CA, 0.69 FA, and 0.68 JRA.
  • RQ1: RCA Outcome Effectiveness: TrainTicket produces less consistent relationships between component and fault-type accuracy because homogeneous services, gateways, and databases obscure either service responsibility or configuration-level causes.GPT-5 obtains 0.70 CA and 0.71 FA, while Qwen3.5-Plus obtains 0.67 CA and 0.79 FA.
  • RQ1: RCA Outcome Effectiveness: Hard faults require telemetry attribution and source analysis beyond observing abnormal signals, yielding lower JRA despite nonzero ECR.The benchmark groups Startup, Scheduling, and Runtime faults as Easy; Admission Control, Service Routing, and Infrastructure faults as Medium; and Performance and Application Code Defect faults as Hard.
  • RQ2: RCA Process Alignment: 0.38 ECR versus 0.76 JRA on OnlineBoutique and 0.15 ECR versus 0.68 JRA on TrainTicket show that correct final labels can lack complete evidence chains.Agents may infer the correct component-fault pair from salient local signals without establishing the full diagnostic chain.
  • RQ2: RCA Process Alignment: Agents explore relevant evidence broadly but shallowly, with moderate milestone coverage failing to become complete diagnostic closure.For example, Qwen3.5-27B reaches 0.75 MC and 0.43 ECR on OnlineBoutique, while GPT-5 reaches 0.64 MC and 0.21 ECR.
  • RQ2: RCA Process Alignment: Dependency-aware evidence scores are only slightly below milestone coverage, whereas the larger MC-to-ECR gap indicates difficulty converting scattered evidence into closure.On OnlineBoutique, Qwen3.5-27B drops from 0.75 MC to 0.70 EOC, while GPT-5 drops from 0.64 to 0.55.
  • RQ2: RCA Process Alignment: RCA synthesis, process completeness, and diagnostic efficiency are distinct capabilities rather than a single general ability.GPT-5 reaches 0.64 EE in 5.56 steps on OnlineBoutique, whereas Qwen3-8B uses 16.01 steps, reaches 0.37 EE, and has 0.46 RAR.

5 Discussion

Cloud-OpsBench makes cloud RCA investigations controlled and measurable by replaying verified incidents, aligning evidence, and supporting process-level analysis. It also exposes scope boundaries, including Kubernetes-focused coverage and individually verified faults.

  • Cloud-OpsBench evaluates both final diagnosis and the supporting investigation, rather than treating RCA correctness as the sole outcome.
  • Snapshot replay holds incident evidence fixed, enabling controlled comparisons of agent components, policies, and diagnostic outcomes.Researchers can vary planners, memory, retrieval, topology, or tool-selection policies under the same user query and incident state.
  • Diagnostic evidence graphs align investigations by causal facts, revealing failures in dependency traversal, mechanism confirmation, or premature investigation.
  • Aligned trajectories make successful and failed strategies comparable across models, fault types, and application or platform layers.The same alignment also supports studying transfer of diagnostic experience across incidents.
  • Future work can study state-adaptive evidence acquisition and transferable diagnostic skills based on milestones, unresolved alternatives, and expected information gain.
  • The benchmark focuses on individually verified faults and Kubernetes-based cloud-native systems, limiting coverage of concurrent faults, changing states, and other architectures.Broader validation requires additional workloads and deployment environments.

6 Related Work

Agentic RCA extends traditional and generative approaches by interactively selecting tools, interpreting observations, and revising hypotheses. Existing benchmarks separately provide reproducibility or live interaction, leaving the combined evaluation need only partially addressed.

  • Traditional AIOps RCA localizes faulty components from metrics, logs, and traces, while LLM systems generate analyses from incident context and retrieved operational documents.
  • Recent agentic workflows add interactive tool selection, observation interpretation, and hypothesis revision to the RCA process.
  • Static benchmarks provide deterministic artifacts but do not let agents investigate through operational interfaces, whereas live benchmarks restore tool use with variable fault manifestations.
  • OpenRCA improves inspectability through code-based queries over supplied files but does not expose system-facing SRE tools.

7 Conclusion

Cloud-OpsBench provides a controlled infrastructure for interactive, evidence-grounded cloud RCA. Its matched replay, evidence graphs, and process metrics support both agent comparison and analysis of investigation failures.

  • 754 cases spanning 57 fault types cover application and Kubernetes platform layers for controlled RCA-agent comparison.
  • Runtime-verified fault realization and snapshot-backed replay give agents matched incident observations while preserving independent tool and target selection.
  • Outcome labels and diagnostic evidence graphs measure root-cause correctness, milestone coverage, evidence-order consistency, and investigation efficiency without prescribing commands.
  • The benchmark supports analysis of failures in evidence acquisition, dependency traversal, mechanism confirmation, and final diagnosis.
Loading 2603.00468v2…