Source-linked AI summary
CloudRCA: A Root Cause Analysis Framework for Cloud Computing Platforms
Yingying Zhang, Zhengxiong Guan, Huajie Qian, Leili Xu, Hengbo Liu, Qingsong Wen, Liang Sun, Junwei Jiang, Lunting Fan, Min Ke
TL;DR
Cloud platforms make root cause analysis difficult, while existing methods may rely on KPIs or logs separately and return results that do not directly identify root-cause types. CloudRCA combines metrics, logs, and topology in a hierarchical Bayesian network; it outperforms existing approaches across cloud systems and has been deployed in Alibaba platforms.
Problem
Existing RCA methods may use KPIs or logs separately and return KPI nodes or representative log sequences rather than direct root-cause types that guide recovery.
Source: §1 · pp. 1–2Method
CloudRCA engineers features from monitoring metrics and log patterns, then combines them with CMDB dependencies in KHBN to infer root causes.
Source: §3.2 · p. 3, §3.3 · p. 4Results
Takeaways & Limitations
Experiments identify multi-source data, predefined knowledge, rich training data, and cross-platform transfer learning as useful for improving RCA accuracy.
Source: §4.9 · p. 9
Abstract
from arXiv · showhide
As business of Alibaba expands across the world among various industries, higher standards are imposed on the service quality and reliability of big data cloud computing platforms which constitute the infrastructure of Alibaba Cloud. However, root cause analysis in these platforms is non-trivial due to the complicated system architecture. In this paper, we propose a root cause analysis framework called CloudRCA which makes use of heterogeneous multi-source data including Key Performance Indicators (KPIs), logs, as well as topology, and extracts important features via state-of-the-art anomaly detection and log analysis techniques. The engineered features are then utilized in a Knowledge-informed Hierarchical Bayesian Network (KHBN) model to infer root causes with high accuracy and efficiency. Ablation study and comprehensive experimental comparisons demonstrate that, compared to existing frameworks, CloudRCA 1) consistently outperforms existing approaches in f1-score across different cloud systems; 2) can handle novel types of root causes thanks to the hierarchical structure of KHBN; 3) performs more robustly with respect to algorithmic configurations; and 4) scales more favorably in the data and feature sizes. Experiments also show that a cross-platform transfer learning mechanism can be adopted to further improve the accuracy by more than 10\%. CloudRCA has been integrated into the diagnosis system of Alibaba Cloud and employed in three typical cloud computing platforms including MaxCompute, Realtime Compute and Hologres. It saves Site Reliability Engineers (SREs) more than $20\%$ in the time spent on resolving failures in the past twelve months and improves service reliability significantly.
1 INTRODUCTION
Alibaba’s growing, data-intensive cloud platforms require stable service and fast fault recovery, but their scale, time-sensitive workloads, scarce fault samples, and varied architectures complicate root cause analysis. CloudRCA combines monitoring metrics, logs, and expert knowledge to address gaps in existing methods and support actionable root-cause identification.
1 INTRODUCTION
- Alibaba’s platforms process PB-level data for growing e-commerce activity, making stability and fast fault recovery essential.The paper notes demand peaks during the annual Double 11 shopping festival.Source: §1 · p. 1
- RCA is difficult because platforms and data are increasingly complex, diagnosis is time-sensitive, faults are rare, and cloud architectures vary.The paper identifies MTTR as a measure of diagnosis efficiency and notes that rare faults constrain training samples.Source: §1 · p. 1
- Existing methods use KPIs or logs separately and return KPI nodes or representative log sequences rather than direct root-cause types that guide recovery.The paper names CloudRanger, O&M Graph, and LogCluster as examples.Source: §1 · pp. 1–2
- CloudRCA integrates monitoring metrics, system logs, and expert knowledge, and the paper reports higher accuracy than methods using only KPIs or logs.The authors also describe improvements to log-template pruning, NLP-based log clustering, and hierarchical Bayesian networks as contributing to accuracy and efficiency.Source: §1 · p. 2
2 RELATED WORK
Prior RCA work spans log mining, event-and-metric correlation, and dependency or causality graphs. The paper frames CloudRCA against limitations in feature coverage, interpretability, and use of expert knowledge and multiple data sources.
2 RELATED WORK
- RCA research on complex, large-scale systems is organized into multiple method categories.The section introduces log-based, correlation-based, and DAG-based approaches.Source: §2 · p. 2, §2.0.1 · p. 2, §2.0.2 · p. 2
2.0.1 Log-based methods.
- Log-based RCA mines system logs to diagnose and locate anomalies, but limited log-feature coverage can constrain accuracy.The paper cites for a classic distributed-systems method and on log mining for cloud applications.Source: §2.0.1 · p. 2
2.0.2 Correlation analysis-based methods.
- Correlation methods combine events, metrics, and alerts, but omitting infrastructure topology and domain knowledge can make results difficult for SREs to interpret.Examples include a noisy-time-lag model and statistical inference over anomalous events.Source: §2.0.2 · p. 2
2.0.3 DAG-based methods.
- For cloud applications, CloudRanger and O&M Graph build dependency or causality graphs from historical KPIs to infer root causes online.The paper notes that these works do not fully use expert knowledge and multiple sources such as logs.Source: §2.0.3 · p. 2
3 PROPOSED CLOUDRCA FRAMEWORK
CloudRCA converts monitoring and logging data into feature matrices, then combines those features with CMDB module dependencies in a hierarchical Bayesian network to infer root causes. Its workflow includes decomposition-based anomaly detection, adaptive log-template extraction and semantic clustering, and KHBN construction and inference.
3 PROPOSED CLOUDRCA FRAMEWORK
- CloudRCA converts monitoring metrics, logs, and CMDB module dependencies into unified features and uses KHBN to infer root causes in real time.The workflow first engineers features from multiple sources, then combines processed signals with dependency knowledge in the network.Source: §3 · p. 2
3.1 Metrics Anomaly Detection
- The metric detector decomposes time series and uses statistical tests to identify anomalies in different components, with robust statistics and incremental online updates.RobustSTL estimates trend using sparse-regularized least absolute deviation and seasonality using non-local seasonal filtering; the tests use robust statistics to hedge against outliers and noise.Source: §3.1 · pp. 2–3
3.2 Log Templates Extraction and Clustering
- AFT-tree extracts log templates, while Word2vec representations and hierarchical clustering group templates by semantic similarity into log patterns.AFT-tree uses dictionary-based child storage and leaf-count pruning; the authors report it is 27 times faster than FT-tree on their datasets.Source: §3.2 · p. 3
- For online log clustering, CloudRCA selects each cluster’s most central log as its representative and assigns new logs by similarity or starts a new cluster.The representative is the log with the minimum average distance to other cluster logs; the threshold is based on average representative similarity across clusters.Source: §3.2 · p. 3
3.3 Knowledge-informed Hierarchical Bayesian Network (KHBN)
- KHBN represents selected log and monitoring features in a first layer, with module-level and root-cause-type nodes above them, and uses CMDB topology to initialize edges.The network learns directed causal edges from normal and abnormal system metrics, estimates conditional probabilities, and selects the highest-probability root-cause type from observed features.Source: §3.3 · pp. 3–4
4 REAL-WORLD DEPLOYMENTS
CloudRCA is evaluated across three Alibaba platforms with different purposes and scales but some shared infrastructure. The evaluation covers workflow components, datasets, and platform comparisons.
4 REAL-WORLD DEPLOYMENTS
- The evaluation examines workflow components, stage-specific algorithms, cross-platform performance, novel root causes, knowledge removal, transfer learning, sensitivity, overhead, and real-world cases.The workflow is deployed on MaxCompute, Realtime Compute, and Hologres.Source: §4 · p. 4
4.1.1 MaxCompute.
- MaxCompute supports large-scale data warehousing and analysis, with more than 100 thousand servers and over one million jobs processed daily.Its stored data had reached EB-level volume, and its architecture includes Pangu storage and decentralized multi-scheduler resource management.Source: §4.1.1 · p. 4
4.1.2 Realtime Compute.
- Realtime Compute is a Flink-based stream-processing platform supporting more than 35,000 jobs across over 1.5 million CPUs.During Double 11, peak processing reached 4 billion records per second.Source: §4.1.2 · p. 4
4.1.3 Hologres.
- Hologres is a cloud-native hybrid serving and analytical system supporting highly concurrent writes and queries at up to 100 million transactions per second.Its compute and storage layers are decoupled, with computation deployed on Kubernetes and Pangu as the default storage layer.Source: §4.1.3 · p. 4
- The three platforms differ in purpose, scale, compute engines, schedulers, and storage, while sharing infrastructure such as hosts and networks.The paper notes a trend toward cloud-native services that may increase shared storage and scheduler infrastructure.Source: §4.1.3 · p. 4
4.2 Dataset and Setup
- The dataset combines five years of metrics, logs, and CMDB tree data; normal samples train the model, while fault samples are split 60:40.Evaluation uses precision, cover rate, and F1-score; a type is covered when at least 60% of its test samples are correctly predicted.Source: §4.2 · pp. 4–5, Table 4 · p. 5
4.3 Ablation Study
The ablation study tests feature-engineering components and alternative stage algorithms, compares KHBN with existing RCA methods, and examines the effect of predefined topology knowledge.
4.3.1 Validation of Feature Engineering.
- Skipping anomaly detection, log-template extraction, or log clustering compromises performance on all three platforms.The authors say these steps extract relevant information from complex raw data, improving KHBN graph construction and root-cause inference efficiency.Source: §4.3.1 · pp. 5–6
4.3.2 Comparison of Different Techniques in Feature Engineering.
- Robust STL improves performance and efficiency over traditional STL, proposed anomaly detection beats iForest on all platform metrics, and AFT-tree is more computationally efficient than FT-tree.DBSCAN and hierarchical clustering perform similarly, so other decent clustering methods may serve as substitutes.Source: §4.3.2 · p. 6
4.3.3 Evaluation of KHBN.
- KHBN beats LogCluster on all three platforms, exceeds CloudRanger and OM Graph by at least 0.09 in f1 score, reaches 0.78 f1 score on MaxCompute, and is fastest at model building and inference.The authors attribute KHBN’s speed to its parallel computation design.Source: §4.3.3 · p. 6
4.3.4 The Impact of Predefined Knowledge.
- Removing predefined topology knowledge slightly lowers KHBN f1 score in Realtime Compute and Hologres, while both KHBN and OM Graph remain robust on data-rich MaxCompute.The authors conclude that sufficient training data can partially offset performance loss from removing predefined knowledge.Source: §4.3.4 · p. 6
4.4 Localization of Novel Root Causes
CloudRCA is tested on root-cause types withheld from training to assess localization of novel faults. KHBN has the highest f1 score across the three products, using its hierarchical layer to locate the associated module when a new type is absent from the graph.
4.4 Localization of Novel Root Causes
- KHBN achieves the highest f1 score, above 0.5 on all three products, when tested on root-cause types excluded from training.The evaluation tests whether models identify novel causes at the metric or module level.Source: §4.4 · p. 6
- For an unseen root-cause type absent from the graph, KHBN can locate the second-layer node indicating its associated module.This behavior follows from KHBN’s hierarchical root-cause layer.Source: §4.4 · p. 6
- CloudRanger and OM Graph lose accuracy on specific metric nodes, while LogCluster performs similarly for novel and existing causes.The authors link the graph-based methods’ loss to strong correlations among metrics.Source: §4.4 · p. 6
4.5 Sensitivity Analysis
KHBN maintains strong F1-score performance across almost all tested parameter configurations. The sensitivity experiments vary individual workflow parameters on MaxCompute.
4.5 Sensitivity Analysis
- The tested parameters are anomaly-detection significance level α, log-template extraction max_num_of_leaf, and log-clustering distance_threshold θ.Source: §4.5 · p. 6
- The sensitivity experiments vary one parameter at a time while keeping the others constant, using MaxCompute data.The results are presented in Figure 3, but that figure is not among the supplied passages for this summary.Source: §4.5 · p. 7
- KHBN exceeds 0.6 F1-score and outperforms other RCA methods across almost all tested parameter configurations.The authors also report that KHBN appears less sensitive to parameter changes, supporting more stable performance in practice.Source: §4.5 · p. 8
4.6 Computational Scalability
KHBN runtime grows linearly with training-data volume and feature size, and it runs faster than two graph-based competitors in the reported scalability evaluation.
4.6 Computational Scalability
- KHBN consistently runs over 50% faster than the two graph-based competitors in the synthetic-data scalability evaluation.The authors attribute its linear runtime growth to parallel computation.Source: §4.6 · p. 8
- With 2,000 samples and 1,000 feature nodes, CloudRCA completes model construction in 3.5 minutes.Source: §4.6 · p. 8
- KHBN runtime increases linearly with training-data volume and feature size.The evaluation varies training-data size from 50 to 1,000 and feature-node counts of 50, 100, 300, 500, and 1,000.Source: §4.6 · p. 8
4.7 Case Studies
CloudRCA case studies show its use in MaxCompute, Realtime Compute, and Hologres, while cross-platform transfer learning combines same-module samples to enrich training data. Common modules benefit from transfer learning, although gains are limited for some modules.
4.7 Case Studies
- CloudRCA is deployed on MaxCompute, Realtime Compute, and Hologres, and has helped SREs reduce failure-resolution time by more than 20%.The platforms support daily operations and Double 11 with more than 100,000 servers.Source: §4.7 · p. 8
- In MaxCompute, CloudRCA localized an unseen resource-scheduler bug to the resource management module, reducing debugging time from up to two hours to a few minutes.Source: §4.7 · p. 8
- In Realtime Compute, CloudRCA identified a hotspot as the root cause amid simultaneous KPI anomalies across modules, reducing troubleshooting time by nearly 50%.The hotspot also affected the job scheduler.Source: §4.7 · p. 8
- In early-stage Hologres, CloudRCA detected an anomaly from increasing error-log patterns and identified the root cause when monitoring KPIs did not alarm.Source: §4.7 · p. 8
4.8 Cross-Platform Transfer Learning
- The transfer-learning mechanism combines samples from the same modules across three Alibaba cloud platforms to enrich the training set.The authors motivate this approach by noting that newer Hologres models underperform MaxCompute models, in part because Hologres has limited historical training data.Source: §4.8 · p. 8
- Host, network, and storage modules benefit remarkably from transfer learning, while improvement is limited for resource scheduler modules and storage in Realtime Compute.Table 12 reports module-level KHBN performance before and after transfer learning.Source: §4.8 · p. 8, Table 12 · p. 9
- The authors describe transfer learning as a promising approach for improving root-cause analysis in relatively new cloud products as cloud-native techniques spread.Source: §4.8 · p. 9
4.9 Key Insights
The experiments identify feature engineering, multi-source data, predefined knowledge, and transfer learning as contributors to CloudRCA’s performance. Transfer learning is especially relevant when newer cloud platforms have limited training samples.
4.9 Key Insights
- Feature engineering extracts relevant information and reduces data complexity, boosting accuracy and efficiency in root-cause inference.Source: §4.9 · p. 9
- Combining multiple data sources gives RCA models more comprehensive information for precisely locating root causes.Source: §4.9 · p. 9
- Predefined knowledge and richer training data help improve RCA accuracy significantly.Source: §4.9 · p. 9
- Transfer learning across platforms can improve accuracy when relatively new cloud platforms have limited training samples.The authors identify it as a promising direction for cloud-native RCA.Source: §4.9 · p. 9
5 CONCLUSION AND FUTURE WORKS
CloudRCA combines anomaly detection, log clustering, and Bayesian network inference, with experiments reporting superiority, scalability, and robustness. Deployment across three Alibaba cloud platforms reduced failure-resolution time by more than 20%, while cross-platform transfer learning improved accuracy by more than 10%.
5 CONCLUSION AND FUTURE WORKS
- More than 20% reduction in failure-resolution time was reported after CloudRCA deployment on MaxCompute, Realtime Compute, and Hologres.The reported reduction covers the past 12 months.Source: §5 · p. 9
- More than 10% accuracy improvement was reported for transfer learning across different platforms.The conclusion describes this as promising for Alibaba’s big data ecosystem.Source: §5 · p. 9
- CloudRCA combines anomaly detection, log clustering, and Bayesian network inference to detect anomalies and identify root causes.The conclusion reports comprehensive experiments demonstrating superiority, scalability, and robustness against well-studied RCA frameworks.Source: §5 · p. 9
- The authors plan to try GANs with active learning to reduce manual fault-injection and labeling costs.Source: §5 · p. 9