PaperlayerView exact revision on arXiv

Source-linked AI summary

CausalRCA: Causal Inference based Precise Fine-grained Root Cause Localization for Microservice Applications

Ruyue Xin, Peng Chen, Zhiming Zhao

arXiv:2209.02500v2cs.DC

TL;DR

Fine-grained root cause localization matters for performance-anomaly recovery, but prior work has mainly focused on identifying faulty services, and existing causal-inference methods have limitations. CausalRCA uses gradient-based causal-structure learning and root-cause inference on monitoring data. Its experiments report higher localization accuracy than baseline methods, including an average AC@3 of 0.719 for fine-grained metric localization in the faulty service.

  • Problem

    Prior metric-based localization has focused mainly on faulty services, while fine-grained localization must pinpoint root causes among diverse anomalous metrics and complex service dependencies.

    Source: §I · pp. 1–2

  • Method

    CausalRCA learns weighted causal graphs from monitoring data using a gradient-based method, then applies root-cause inference to rank metrics.

    Source: §I · p. 2, B. Causal structure learning · p. 5, C. Root cause inference · p. 6

  • Results

    Average AC@3 for fine-grained root cause metric localization in the faulty service is 0.719, a 10% improvement over baseline methods.

    Source: §I · p. 2

  • Takeaways & Limitations

    CausalRCA’s coarse- and fine-grained experiments report better localization accuracy than baseline methods.

    Source: §V · p. 12, §VI · p. 12

    Evaluation uses one specific microservice configuration, which may limit generality, and localization performance heavily relies on input data.

    Source: C. Threats to validity · p. 12

Abstract

from arXiv · show

Effectively localizing root causes of performance anomalies is crucial to enabling the rapid recovery and loss mitigation of microservice applications in the cloud. Depending on the granularity of the causes that can be localized, a service operator may take different actions, e.g., restarting or migrating services if only faulty services can be localized (namely, coarse-grained) or scaling resources if specific indicative metrics on the faulty service can be localized (namely, fine-grained). Prior research mainly focuses on coarse-grained faulty service localization, and there is now a growing interest in fine-grained root cause localization to identify faulty services and metrics. Causal inference (CI) based methods have gained popularity recently for root cause localization, but currently used CI methods have limitations, such as the linear causal relations assumption and strict data distribution requirements. To tackle these challenges, we propose a framework named CausalRCA to implement fine-grained, automated, and real-time root cause localization. The CausalRCA uses a gradient-based causal structure learning method to generate weighted causal graphs and a root cause inference method to localize root cause metrics. We conduct coarse- and fine-grained root cause localization to evaluate the localization performance of CausalRCA. Experimental results show that CausalRCA has significantly outperformed baseline methods in localization accuracy, e.g., the average AC@3 of the fine-grained root cause metric localization in the faulty service is 0.719, and the average increase is 10% compared with baseline methods. In addition, the average Avg@5 has improved by 9.43%.

I. INTRODUCTION

Fine-grained root cause localization can identify both faulty services and indicative metrics, but existing metric-based research has largely focused on faulty services. CausalRCA applies causal inference to monitoring data, and its experiments report improved localization accuracy over baseline methods.

I. INTRODUCTION

  • Coarse-grained localization identifies only faulty services, and restarting or migrating a service may fail to resolve anomalies while risking effects on other services and longer recovery.Fine-grained localization instead pinpoints indicative metrics on the faulty service.Source: §I · p. 1
  • Fine-grained localization identifies the faulty service and underlying resource metrics, enabling more accurate actions such as scaling when resources are insufficient.The paper describes service scale-out as having a positive effect and shorter recovery time than service restart in that case.Source: §I · p. 1
  • Most metric-based root cause research is coarse-grained, while existing causal-inference methods can impose strict assumptions on input data and causal relations.The paper identifies fine-grained localization using monitoring data as its research focus.Source: §I · p. 2
  • CausalRCA is an automated, real-time framework for localizing faulty services and system resources, using gradient-based learning to capture linear and nonlinear relations between monitoring metrics.Its experiments evaluate coarse- and fine-grained localization.Source: §I · p. 2
  • Average AC@3 for fine-grained root cause metric localization in the faulty service is 0.719, a 10% improvement over baseline methods.The paper reports experiments spanning coarse- and fine-grained localization.Source: §I · p. 2

II. RELATED WORKS

Related work spans log-, trace-, and metric-based localization, with metric-based research supporting automated, real-time analysis from monitoring data. Within that category, prior work includes both coarse service localization and emerging fine-grained metric localization.

A. Metric-based root cause localization research

  • Metric-based research uses monitoring data for automated, real-time localization, whereas log- and trace-based approaches face real-time processing, information-extraction, or source-code analysis burdens.Statistical correlation methods cannot pinpoint root causes, and topology-graph methods are limited to coarse-grained localization.Source: §II · p. 3, A. Metric-based root cause… · p. 3
  • Coarse-grained metric-based methods use service-level metrics and graph algorithms to identify faulty services, but do not support accurate recovery actions.Microscope uses PC and breadth-first search; CloudRanger, MS-Rank, and AutoMAP use PC and random-walk algorithms.Source: A. Metric-based root cause… · p. 3
  • Fine-grained methods localize resource metrics using approaches including CauseInfer,, MicroCause, and MicroDiag, helping operators take actions such as resource scaling.These methods combine causal-graph construction with graph traversal, random walk, or PageRank inference.Source: A. Metric-based root cause… · p. 3
  • Prior research has mostly focused on coarse-grained localization, and existing localization accuracy can be improved; the paper therefore motivates advanced causal-inference methods for fine-grained analysis.The paper cites root-cause identification success rates below 20% in some studies,.Source: A. Metric-based root cause… · p. 4

B. Causal inference methods

Traditional causal-structure learning methods have limitations, including ambiguous relations, high search costs, or restrictive assumptions. The paper motivates gradient-based learning and introduces CausalRCA as a monitoring-data pipeline that builds weighted causal graphs and ranks metrics.

B. Causal inference methods

  • Traditional causal-structure learning methods have trade-offs: PC can leave causal relations ambiguous, GES can be slow, and LiNGAM assumes linear, non-Gaussian relations.These approaches respectively include constrained-based, score-based, and function-based methods.Source: B. Causal inference methods · p. 4
  • Gradient-based methods can model linear and nonlinear relations, automatically generate weighted DAGs, and had not previously been applied to microservice root cause localization, to the authors’ knowledge.The paper identifies this as the motivation for exploring gradient-based causal graphs in this setting.Source: B. Causal inference methods · p. 4

III. ROOT CAUSE LOCALIZATION FRAMEWORK

  • CausalRCA is presented as a framework comprising causal structure learning and root cause inference for localizing microservice anomalies.The paper states that its code and data are available in the CausalRCA repository.Source: §III · p. 4

A. Framework overview

  • The framework collects service-latency and resource time series, learns a weighted DAG of metric relations, then applies PageRank to rank metrics as likely root causes.With service-latency input it supports coarse-grained faulty-service localization; with resource metrics it supports fine-grained root-cause metric localization.Source: A. Framework overview · p. 4, Figure 1 · p. 5

B. Causal structure learning

CausalRCA uses gradient-based DAG-GNN structure learning to model linear and nonlinear metric relations without strict input-data limitations. It produces a weighted DAG whose edges encode the direction and degree of metric impact.

B. Causal structure learning

  • DAG-GNN uses a gradient-based variational autoencoder with a structural constraint to learn weighted DAGs from monitoring data and capture linear or nonlinear metric relations.The authors state that this method has no limitation on input data, unlike traditional causal structure learning methods.Source: B. Causal structure learning · p. 5
  • The method represents metric observations as X and uses a generative model to reconstruct X from latent representation Z.The paper relates the decoder to a linear SEM abstraction and develops the generative model using a VAE.Source: B. Causal structure learning · p. 5
  • The augmented-Lagrangian update increases penalty parameter c when constraint violation grows, balancing feasibility and optimality through tuning parameters η and γ.The paper notes that high γ slows convergence, while high η can make results oscillate.Source: B. Causal structure learning · p. 6
  • In the learned DAG, nodes represent metrics and weighted edges encode cause-effect direction and impact magnitude, with positive or negative weights indicating increase or decrease.The weighted DAG is then used by the root cause inference method to pinpoint a root-cause metric.Source: B. Causal structure learning · p. 6

C. Root cause inference

Root cause inference ranks metrics in a weighted DAG using PageRank. It reverses graph edges and takes absolute edge weights before ranking the resulting scores.

C. Root cause inference

  • PageRank scores reflect incoming edges and the probability of anomaly spread, using transition probabilities between nodes.The teleportation probability α controls continuation of the random walk versus a jump to a random node.Source: C. Root cause inference · p. 6
  • Root cause inference reverses the weighted DAG’s edges and uses absolute edge weights before ranking metrics by PageRank score.A higher-ranked metric is considered more likely to be the root cause.Source: C. Root cause inference · p. 6

IV. EXPERIMENTS AND RESULTS

The evaluation covers coarse-grained faulty-service localization and fine-grained metric localization within the faulty service or across all services.

IV. EXPERIMENTS AND RESULTS

  • CausalRCA is evaluated on faulty-service localization and on root-cause metric localization within the faulty service and across all services.The all-service experiment addresses limited understanding of an application’s services and underlying infrastructure.Source: §IV · p. 6

A. Experimental settings

The experiments use sock-shop deployed on Kubernetes VMs, inject three anomalies, and collect service- and resource-level monitoring data.

1) Testbed:

2) Anomaly injection:

  • The evaluation injects CPU hog, memory leak, and network delay anomalies for five minutes per service, with ten minutes to cool down between injections.Pumba is used to inject the anomalies; the method is stated to apply to anomalies that manifest as increased microservice response time.Source: 2) Anomaly injection · p. 7

3) Data collection:

  • Monitoring data includes service latency and container CPU, memory, disk read/write, and network receive/transmit metrics.Table II is identified as the collected monitoring metrics table.Source: 2) Anomaly injection · p. 7, Table II · p. 7

4) Baseline methods:

CausalRCA is compared with PageRank combined with three causal-structure-learning methods. Localization is evaluated with rank-based accuracy metrics at different top-k cutoffs.

4) Baseline methods:

  • The baselines combine PageRank with PC, GES, or LiNGAM, representing constraint-, score-, and function-based causal structure learning.The baseline structure-learning methods use their default causal-learn settings.Source: 4) Baseline methods · p. 7

5) Evaluation metrics:

  • AC@k measures whether the top k ranked results include the real root cause; higher values at small k indicate more accurate localization.The root cause is represented by the root cause set Vrc.Source: 5) Evaluation metrics · p. 7
  • The experiments use AC@1, AC@3, and Avg@5 to assess top-result accuracy, top-three candidate coverage, and average localization ability.AC@1 is the most restrictive metric; AC@3 helps reduce the root-cause candidate set.Source: 5) Evaluation metrics · pp. 7–8

6) Statistical testing:

CausalRCA is evaluated against baseline RCA methods, with statistical tests used to compare performance. In coarse-grained localization, it significantly outperforms the baselines overall, though results vary by anomaly and parameter setting.

B. Experimental results

  • The evaluation covers faulty-service localization from service latency, metric localization within the faulty service, and metric localization across all services.These are the three experiments reported for CausalRCA.Source: B. Experimental results · p. 8

6) Statistical testing:

1) Coarse-grained faulty service localization:

  • Up to 10%, CausalRCA improved coarse-grained localization accuracy across anomalies; average AC@3 was 0.5749 and average Avg@5 was 0.5815, a 6.72% increase.For the three anomalies, average AC@1 was 0.2.Source: Table III · p. 8, 1) Coarse-grained faulty service… · p. 8
  • The default γ = 0.25, η = 10 setting performed best on average and for CPU hog and network latency, while γ = 0.75, η = 10 performed best for memory leak.Higher γ can slow convergence but take more time; high η produced different performance across CPU hog and memory leak.Source: 1) Coarse-grained faulty service… · p. 8, Figure 5 · p. 9
  • Localization varied by service and anomaly: payment performed poorly for CPU hog but well for network delay, while front-end, user, and catalogue performed worse for memory leak.The paper attributes weak memory-leak localization in those services to limited effects on service latency, and strong payment network-delay localization to its network dependence.Source: 1) Coarse-grained faulty service… · pp. 8–9, Figure 6 · p. 9

2) Fine-grained root cause metric localization in the faulty service:

CausalRCA localizes root-cause metrics within faulty services more accurately than baseline methods across the tested anomalies. Its average top-three accuracy is 0.719, while parameter choices and service-specific anomaly patterns affect performance.

  • CausalRCA improved localization accuracy by up to 14.29%; it led on all three metrics for network delay but had metric-specific exceptions for CPU hog and memory leak.For CPU hog, AC@3 and Avg@5 were best but AC@1 trailed PC-based methods; for memory leak, AC@3 and Avg@5 were best but AC@1 trailed LiNGAM.Source: 2) Fine-grained root cause metric… · p. 9
  • 0.719 average AC@3, a 10% average improvement over baselines, and 0.6681 average Avg@5, a 9.43% increase, were reported across three anomalies.Average AC@1 was 0.2476; the table covers localization accuracy in faulty services for different anomalies.Source: Table IV · p. 9, 2) Fine-grained root cause metric… · p. 9
  • ANOVA found a significant difference among four RCA methods (p-value 0.0013), and t-tests found CausalRCA significantly different from baselines.GES showed no significant difference from PC-based and LiNGAM-based methods.Source: 2) Fine-grained root cause metric… · pp. 9–10, Figure 7 · p. 10
  • γ = 0.25 and η = 10 had the highest average localization performance, though anomaly-specific parameter settings performed best for some cases.γ = 0.25 and η = 100 worked best for memory leak and network latency; increasing γ and η could potentially improve accuracy.Source: Figure 8 · p. 10, 2) Fine-grained root cause metric… · p. 10
  • Localization varied by service: CPU-hog accuracy was low for payment and shipping, memory-leak results were strong across services, and network-delay results were best for payment and shipping.The paper notes variance in Avg@5 results, potentially due to cloud dynamics or resource fluctuations.Source: Figure 9 · p. 10, 2) Fine-grained root cause metric… · p. 10

3) Fine-grained root cause metric localization with all monitoring metrics:

With all monitoring metrics, CausalRCA ranks root-cause metrics better than LiNGAM, but identifying a metric across the full set remains difficult. Results depend on parameter choices and are bounded by the tested application configuration, anomaly types, and data collection setting.

3) Fine-grained root cause metric localization with all monitoring metrics:

  • CausalRCA's average root-cause metric rank was about 13, lower than LiNGAM's, with a significant difference (t-test p-value 0.0335).The authors conclude that pinpointing root causes among all observable metrics remains challenging.Source: 3) Fine-grained root cause metric… · pp. 10–11, Figure 10 · p. 11
  • On average, γ = 0.25 and η = 10 performed best; for memory leak, γ = 0.75 and η = 10 was most effective.Higher γ and η improved memory-leak localization, while parameter adjustment could improve accuracy for that anomaly.Source: Figure 11 · p. 11, 3) Fine-grained root cause metric… · p. 11
  • The authors recommend drill-down localization: identify the faulty service first, then locate its root-cause metric.Anomaly propagation differs among services, such as payment being sensitive to network delay but not CPU hog.Source: 3) Fine-grained root cause metric… · p. 11

C. Threats to validity

  • The validity assessment identifies hyperparameters and evaluation metrics as construct threats; careful tuning may improve localization accuracy.The study used three common, frequent anomaly types, while microservices can exhibit a broader variety of performance anomalies.Source: C. Threats to validity · p. 11
  • Generality may be limited by the specific application configuration and data collection strategy; evaluation used fixed-time-range injected anomalies.The framework's localization performance depends heavily on input data.Source: C. Threats to validity · p. 12

V. DISCUSSION

CausalRCA performs well at localizing faulty services and root-cause metrics within faulty services, but localization across all monitoring metrics remains difficult. The paper recommends drill-down localization and identifies time-lag modeling, richer data sources, and efficiency as areas for further work.

V. DISCUSSION

  • CausalRCA performs well on faulty-service and within-service metric localization, but the average rank across all monitoring metrics is above ten.Feature reduction is proposed as a way to reduce training time and improve localization accuracy.Source: §V · p. 12
  • The gradient-based structure learner may ignore time lags in monitoring data, which the authors suggest could help improve causal structure learning.Personalized PageRank is also proposed to incorporate node preferences in root-cause inference.Source: §V · p. 12
  • Trace and log data could calibrate monitoring-data causal graphs using deployment and interaction information, and may improve localization accuracy.The paper identifies combining data sources as future work.Source: §V · p. 12
  • For hundreds or thousands of services, causal-graph construction may take longer than service migration; the study roughly estimates tens of seconds on sock-shop.The experiments mainly use data from five minutes after anomaly detection; scalability, exact time cost, and other time ranges remain future work.Source: §V · p. 12

VI. CONCLUSION

  • Across coarse- and fine-grained experiments, CausalRCA had better localization accuracy than baseline methods; anomaly propagation also differed between services.The authors say these patterns give operators a better understanding of microservice applications, while all-metric root-cause localization remains difficult.Source: §VI · p. 12
  • Drill-down localization—finding the faulty service before its root-cause metric—is presented as a practical approach when all-metric localization is difficult.The difficulty is linked to anomalous metrics manifesting diverse symptoms in different services.Source: §VI · p. 12
  • The conclusion identifies further work in hyperparameter tuning, time-lag modeling, node-preference inference, trace/log calibration, and localization efficiency.These are proposed directions for improving performance and supporting fast recovery.Source: §VI · p. 13

Related papers

Loading 2209.02500v2…