RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models

1 INTRODUCTION

• Cloud anomalies can severely affect customer experience and potentially violate service-level agreements, motivating RCA improvements.AIOps reduces mean time to resolve but faces poor data quality, shifting distributions, costly annotation, and limited model generalization.Source: §1 · p. 1
• Existing LLM-based RCA methods rely on supervised adaptation or few-shot retrieval, use GPT-family models, and do not exploit autonomous information collection or environmental interaction.These limitations leave privacy concerns and production-level interactive RCA insufficiently addressed.Source: §1 · pp. 1–2
• RCAgent introduces a privacy-aware, tool-augmented LLM agent for practical cloud RCA, using locally deployed models instead of external APIs.The framework combines prompting, tool-setting, stabilization, and aggregation methods for complex cloud environments.Source: §1 · p. 2
• RCAgent analysis is used in Alibaba Cloud’s Real-time Compute Platform for Apache Flink to diagnose anomalous stream-processing jobs and discover PaaS and IaaS issues.The deployment includes a feedback mechanism providing insights to development teams.Source: §1 · p. 2
• The paper demonstrates RCAgent through real-world experiments on Alibaba Cloud computing jobs.This practical evaluation is presented as evidence for privacy-aware cloud RCA with locally deployed models.Source: §1 · p. 2

2 CHALLENGE

• RCAgent’s action cycle records thoughts, actions, and environmental observations in the prompt while using stored observations to support lengthy-data analysis.The cycle enables interaction with the environment rather than fixed manual workflows.Source: Figure 1 · p. 2, §2 · p. 2
• Autonomous agents can reduce reliance on laboriously annotated training data and support handling of unseen anomalies, but cloud AIOps remains a difficult setting.The paper identifies context length and action validity as central agent challenges.Source: §2 · p. 2
• Cloud RCA agents must handle privacy risks, enormous heterogeneous context, and invalid open-ended actions when operating on production data.These constraints are intensified when stronger external models cannot be used and cloud data are noisy.Source: §2 · p. 2

3 METHODOLOGY

• The controller uses framework rules, task requirements, and tool documentation, emits JSON actions, and can terminate through a freely chosen finalize action.Few-shot examples are discarded to limit context usage.Source: §3 · p. 3
• RCAgent enhances ReAct with observation management, domain-specific tools, action stabilization, and aggregation for locally hosted LLMs.The controller coordinates actions, while expert agents provide specialized functionality.Source: §3 · pp. 2–3

3.1 Observation Snapshot Key

• OBSK exposes only an observation head and a snapshot key to the controller, then retrieves the full observation from a key-value store when requested.This controls prompt length while keeping necessary information available for decision-making.Source: §3.1 · p. 3

3.2 Tool Preparation

• Information-gathering tools accept simple entity parameters instead of exposing SQL or log-query interfaces, reducing invalid exploration in large data warehouses.Similar information is deduplicated and messages below WARNING level are excluded.Source: §3.2.1 · p. 3, §3.2 · p. 3

3.2.1 Information-gathering Tools.

• Expert agents extend the controller with domain knowledge through zero-shot CoT analysis and answer extraction, using separate code and log analysis tools.These tools generate complementary analytical observations for RCAgent.Source: §3.2.2 · p. 3

3.2.2 Analytical Tools.

• The code analysis tool recursively follows recommended classes, queues pending files, stops at exhausted or external dependencies, and summarizes analyzed code for the controller.Its recursive process is illustrated by Figure 2.Source: §3.2.2 · p. 3, Figure 2 · p. 3
• The log analysis tool builds a distance-decayed similarity graph over log lines, clusters it with Louvain detection, and feeds resulting chunks to a RAG-based expert agent.The agent must copy supporting log evidence, and analyses without fuzzy-matched evidence are discarded.Source: §3.2.2 · pp. 3–4

3.3 Stabilization

• These stabilizations address action-validity degradation caused by noisy data and less capable local LLMs.Source: §3.3 · p. 4, §3.3.2 · p. 4

3.3.1 JSON Repairing.

• JsonRegen repairs malformed JSON by cleaning sensitive characters and regenerating through YAML until valid structure is parsed.The process preserves the same structure and content across regeneration rounds.Source: §3.3.1 · p. 4

3.3.2 Error Handling.

• Figure 3 represents each RCAgent step as a sequential thought, action, and observation procedure.Source: Figure 3 · p. 4
• RCAgent uses predefined criteria to identify duplicate tool calls, trivial expert-agent inputs, and premature finalization.The controller receives error messages and suggestions intended to reduce meaningless actions.Source: §3.3.2 · p. 4

3.4 Self-Consistency Aggregation

• Self-Consistency is applied to free-form agent generation, an underexplored extension beyond its established use in closed-ended NLP tasks.Source: §3.4 · p. 4

3.4.1 Self-Consistency for Text Data.

• For text outputs, RCAgent either selects the embedding-nearest candidate to the semantic majority or prompts an LLM to aggregate candidates.The embedding method generalizes unweighted majority voting, while LLM aggregation accommodates diverse content and similar output lengths.Source: §3.4.1 · p. 4
• Naively sampling complete ReAct trajectories is expensive and can produce flooding errors such as repeated calls to nonexistent tools.The cost is especially high when actions activate resource-intensive expert agents.Source: §3.4.2 · p. 4

3.4.2 Self-Consistency for Tool Using Trajectories.

• Trajectory-level Self-Consistency starts sampling only when the controller reaches finalization, sharing preliminary steps across trajectory samples.This reduces unnecessary computation and uses greedy action history as implicit exemplification without extra few-shot context.Source: §3.4.2 · pp. 4–5

4 EXPERIMENT

• RCAgent is developed on Alibaba Cloud’s Real-time Compute Platform, an Apache Flink system reaching 100 million records per second at peak.Source: §4 · p. 5

4.1 Model Configuration

• The implementation uses Vicuna-13B-V1.5-16K on one NVIDIA A100 GPU, with greedy decoding for reproducibility and stability.GTE-LARGE supplies embeddings, and self-consistency uses Vicuna’s default random-sampling configuration.Source: §4.1 · p. 5
• Self-consistency uses 10 output samples by default, with step-wise sampling restricted to trajectories that finalize synchronously after greedy decoding.Source: §4.1 · p. 5

4.2 Dataset Preparation

• The dataset begins with 15,616 anomalous jobs, is filtered to about 5,000 non-trivial cases, and yields 161 class-balanced jobs for offline evaluation.Annotations cover root cause, solution, evidence, and responsibility, with targets summarized from Flink Advisor and proofread by SREs.Source: §4.2.1 · p. 5

4.2.1 Anomaly Selection.

• The experiment avoids uninformative annotation templates that could make semantic evaluation scores unreliable.Source: §4.2.1 · p. 5

4.2.2 Data Sources.

• Analysis tools expose platform, runtime, and infrastructure logs, advisor-service history, and code repositories while restricting retrieval to data available before anomaly detection.The expert-agent database excludes content used for labeling.Source: §4.2.2 · p. 5

4.3 Evaluation Metrics

• Evaluation combines METEOR, NUBIA, BLEURT, BARTScore, EmbScore, and GPT-4 judgments of root-cause and solution correctness and helpfulness.GPT-4 scores correctness and helpfulness on a 0–10 scale.Source: §4.3 · p. 5

5.1 Effectiveness

• RCAgent outperforms ReAct across root cause, solution, and evidence prediction, including +8.71 and +6.52 METEOR in root cause and solution prediction.Source: §5.1 · p. 5
• TSC with LLM summarization further improves solution prediction by +3.51 METEOR, +4.50 BLEURT, and +2.28% G-Helpfulness through broader solution sampling.Source: §5.1 · p. 5

5.2 Ablation Study

• Removing RCAgent enhancements causes a drastic metric decline, reducing root-cause METEOR improvement from +8.71 to +3.16 and leaving only marginal gains over ReAct.Source: §5.2.1 · p. 6

5.2.1 w/o LLM Expert Agents.

• Without LLM expert agents, malformed outputs and erroneous decisions substantially reduce performance, while removing JsonRegen causes invalid data interchange and instability.Source: §5.2.2 · p. 6, §5.2.3 · p. 6

5.2.3 w/o OBSK.

• OBSK’s snapshot mechanism provides a major benefit: removing the observation head and showing only a snapshot causes a −1.10 BLEURT decline, while summarizing the original observation causes a −0.06 G-Correctness degradation.Source: §5.2.3 · p. 6

5.3 Stability

• With all enhancements, RCAgent reaches a 99.38% Pass Rate and 7.93% Invalid Rate within 15 steps, whereas nucleus sampling yields 70.19% Pass Rate and 44.80% Invalid Rate.Source: §5.3 · p. 6
• Replacing semantically minimalist tools with SQL and SLS execution tools reduces Pass Rate by 33.54%, largely because Invalid Rate rises to 70.95%.Source: §5.3 · p. 6

5.4 Self-Consistency

• Self-consistency improves BARTScore and NUBIA, plateaus around 20 samples, and favors TSC and LLM aggregation as the candidate pool grows.Source: §5.4 · p. 6, Figure 4 · p. 6

6 DEPLOYMENT

• RCAgent analyzes OoD jobs beyond existing automatic SRE rules and forwards platform-responsibility cases with RCA results to Apache Flink SREs for diagnosis.Source: §6 · p. 6

6.1 Performance on OoD Jobs

• The OoD evaluation used samples beyond current rules, labeled by experienced SREs, with human assessment on a 0–5 H-Helpfulness scale.Source: §6.1 · p. 6
• On deployed OoD jobs, RCAgent achieved 2.92 H-Helpfulness, outperformed ReAct with and without TSC, and reached 82.06% responsibility precision with TSC and LLM aggregation.Source: §6.1 · p. 7, Table 5 · p. 7

6.2 Computational Scalability

• As data increased, RCAgent and TSC performed better than ReAct while resource consumption scaled nearly linearly without substantial performance degradation.Source: §6.2 · p. 7, Figure 5 · p. 7

7.1 LLM as Autonomous Agents

• Autonomous LLM agents explore self-directedly without human intervention and adapt actions and outputs according to environmental feedback.Source: §7.1 · p. 7
• RCAgent is presented as the first work introducing autonomous LLM agents to realistic cloud RCA tasks.Source: §7.1 · p. 7

7.2 LLM Augmented by Tools

• Tool-using LLMs invoke functions or APIs to extend their knowledge and capabilities, and RCAgent applies this paradigm to real-world cloud RCA.Source: §7.2 · p. 7

7.3 Cloud RCA with LLMs

• Existing cloud-RCA LLM methods use fine-tuning or in-context learning but do not model the RCA workflow, leaving them as analytical tools rather than agents.Source: §7.3 · p. 7

• RCAgent uses internally deployed models for secure cloud-system usage and combines Self-Consistency for action trajectories with prompting, expert-agent, and stabilization methods.Source: §8 · p. 7
• RCAgent has been applied in Alibaba Cloud’s Real-time Compute Platform for Apache Flink, demonstrating a real-world deployment of LLM agents for cloud RCA.Source: §8 · p. 7