PaperlayerView exact revision on arXiv

Source-linked AI summary

Groot: An Event-graph-based Approach for Root Cause Analysis in Industrial Settings

Hanzhang Wang, Zhengkai Wu, Huai Jiang, Yichao Huang, Jiamu Wang, Selcuk Kopru, Tao Xie

arXiv:2108.00344v2cs.SE

TL;DR

Industrial microservices complicate RCA through operational, scale, and monitoring challenges. GROOT constructs a customizable event-causality graph from diverse monitoring events and domain rules. On 952 real incidents, it achieved 95% top-3 and 78% top-1 accuracy.

  • Problem

    Industrial microservices make RCA difficult because of operational knowledge gaps, system scale, and diverse monitoring data.

    Source: §I · p. 2

  • Method

    GROOT builds a real-time causality graph from monitoring events and supports user-defined events and domain-specific rules.

    Source: §I · p. 2

  • Results

    78% top-1 accuracy versus 3% for the naive approach and 33% for the non-adaptive approach, and 95% top-3 accuracy versus 6% and 38%, respectively.

    Source: 2) RQ2 · p. 8

  • Takeaways & Limitations

    GROOT is deployed in eBay's production system, and surveyed SRE users found it helpful and easy to use during triage.

    Source: §I · p. 2, §VII · p. 10

Abstract

from arXiv · show

For large-scale distributed systems, it's crucial to efficiently diagnose the root causes of incidents to maintain high system availability. The recent development of microservice architecture brings three major challenges (i.e., operation, system scale, and monitoring complexities) to root cause analysis (RCA) in industrial settings. To tackle these challenges, in this paper, we present Groot, an event-graph-based approach for RCA. Groot constructs a real-time causality graph based on events that summarize various types of metrics, logs, and activities in the system under analysis. Moreover, to incorporate domain knowledge from site reliability engineering (SRE) engineers, Groot can be customized with user-defined events and domain-specific rules. Currently, Groot supports RCA among 5,000 real production services and is actively used by the SRE teamin a global e-commerce system serving more than 185 million active buyers per year. Over 15 months, we collect a data setcontaining labeled root causes of 952 real production incidents for evaluation. The evaluation results show that Groot is able to achieve 95% top-3 accuracy and 78% top-1 accuracy. To share our experience in deploying and adopting RCA in industrial settings, we conduct survey to show that users of Grootfindit helpful and easy to use. We also share the lessons learnedfrom deploying and adopting Grootto solve RCA problems inproduction environments.

I. INTRODUCTION

GROOT addresses operational, scale, and monitoring complexities in industrial microservices with a customizable event-graph approach to root cause analysis. Its evaluation and deployment experience report strong accuracy, production use, and positive user feedback.

I. INTRODUCTION

  • Industrial RCA must bridge centered and domain SRE knowledge gaps, scale across thousands of interdependent services, and process diverse, unstructured monitoring data.Potential root causes may be distributed several steps from the initially anomalous service, while services produce differing logs and metrics and may undergo code or configuration changes.Source: §I · pp. 1–2
  • GROOT builds a causality graph from detailed monitoring events and domain-informed rules, supporting transparent, interpretable root-cause rankings.Events include performance-metric deviations, status changes, and developer activities; visualized causal links and event details support investigation and targeted rule improvements.Source: §I · p. 2
  • Event-level nodes, dynamic dependencies, and contextual event details address monitoring, scale, and operational complexities.GROOT supports metrics, status logs, and developer activities, while SRE engineers can add event types and implement domain-specific causality rules through a grammar.Source: §I · p. 2
  • 95% top-3 accuracy and 78% top-1 accuracy were achieved on 952 real production incidents collected over 15 months.GROOT was also used daily by SRE teams, with 73% top-1 accuracy in action and execution under five seconds per incident.Source: §I · p. 2
  • A survey found GROOT helpful and easy to use during triage, while developers valued its adaptability to changes and new requirements.The paper also reports lessons from production adoption, including technology transfer and adoption.Source: §I · p. 2

II. RELATED WORK

Prior RCA work spans anomaly detection, machine learning, and graph-based methods, with limitations in data, error coverage, granularity, or customization. The paper positions GROOT's event graph and adaptable link rules against these gaps and evaluates it at production scale.

II. RELATED WORK

  • Anomaly-detection research includes historical batch analysis, machine-learning methods, and adaptive concept-drift approaches.Source: §II · p. 2
  • Traditional multidimensional RCA methods work effectively on discrete static data but become less effective with continuous time-series information.Source: §II · pp. 2–3
  • Machine-learning RCA approaches face limited training data, while some target QoS violations or rely on difficult-to-simulate systems for meaningful data.The related work describes Seer as using historical tracking data but lacking support for other kinds of errors.Source: §II · p. 3
  • Graph-based methods model dependencies at service, server, VM, or metric levels, and some lack customization or are difficult to extend.The cited approaches include service-level metric graphs and models that track propagation across services and machines.Source: §II · p. 3
  • Prior graph-based RCA approaches vary in granularity and extensibility; GROOT uses event graphs and customizable link rules to improve causal-graph detail and accommodate changing requirements.The cited work includes service- or host-level graphs, while other approaches have limited information use or are difficult to update.Source: §II · p. 3
  • Existing RCA evaluations were relatively small; GROOT was evaluated on 5,000 production services and 952 real incidents, while seven of eight listed approaches used simulated faults.The average service sub-dependency graph in GROOT's data set contained 77.5 services.Source: §II · p. 3

III. MOTIVATING EXAMPLES

GROOT’s event causality graph can trace an incident through event-level links to a likely root cause, where a service-dependency view alone leaves several candidates. SRE engineers can adapt event types and rules to encode domain knowledge.

Event causality graph

  • The Checkout latency incident’s event chain traces through Service-A and Service-C to a likely code deployment in Service-E, unlike a service-only view that leaves several plausible causes.In the motivating example, severity alone would favor Service-B, while the event graph excludes it because no causal rule links its events to Service-A’s.Source: §III · pp. 3–4

Event type addition

  • SRE engineers can add event types by specifying data sources, properties, and causal-link rules, as in the proposed ML Model Deployment event.The example describes fetching the event from the ML model management system and setting a detection time range.Source: §III · p. 4, Figure 2 · p. 4

Event and rule update

  • Conditional rules link latency spikes only when events satisfy specified conditions, such as occurring in the same data center.GROOT can distinguish data centers by adding a DataCenter property to the Latency Spike event.Source: §III · p. 4, Figure 3 · p. 4

IV. APPROACH

GROOT responds to alerts or SRE suspicions by building service and event causality graphs, then ranking likely root causes. Its interactive UI presents causal context and details for human investigation.

Workflow

  • GROOT’s three main steps are constructing the service dependency graph, constructing the event causality graph, and ranking root causes by likelihood.Triggers are usually automated anomaly-detection alerts, but can also be an SRE engineer’s suspicion.Source: §IV · p. 5
  • The interactive UI presents service dependencies, events, causal links, and details such as raw metrics or deployment contacts for investigation.Offline, GROOT collects and labels incidents, validates them, and summarizes knowledge for daily improvement.Source: §IV · p. 5

A. Constructing Service Dependency Graph

GROOT builds a service-dependency subgraph around alerted or suspected services, then collects supported events and constructs causal links. The implementation supports multiple event categories and event-specific properties.

A. Constructing Service Dependency Graph

  • The service-dependency subgraph is built by traversing from alerted or suspected services within a radius of two in the global graph, which is updated daily.The radius can be extended dynamically as event details reveal longer issue chains or additional dependencies.Source: A. Constructing Service Dependency Graph · p. 5

B. Constructing Event Causality Graph

  • GROOT collects supported events for each service in the subgraph and constructs causal links between those events.The approach first establishes the service-dependency subgraph, then uses its services as the scope for event collection.Source: B. Constructing Event Causality Graph · p. 5

1) Collecting Events:

  • Events include performance metrics, status logs, and developer activities, with each event carrying its service, type, start time, and properties.Event types specify a name, lookback period, and property types; the listed implementation examples are in Table II.Source: 1) Collecting Events · p. 5, Table II · p. 5

2) Constructing Causal Link:

GROOT builds causal links between monitoring events using SRE-authored rules, then ranks probable root causes with GrootRank. It handles ranking ties using service access distance and historical root-cause frequency.

2) Constructing Causal Link:

  • GROOT uses SRE knowledge encoded in rules to create causal links, where a source event may be caused by a target event.The graph's nodes are events, and the rules determine which event pairs are linked.Source: 2) Constructing Causal Link · p. 6
  • Each causal-link rule specifies event types, link direction, target service, rule type, and an optional condition.Static rules can target the same, upstream, or downstream services.Source: 2) Constructing Causal Link · p. 6
  • Dynamic rules create services and events for dependencies that are not readily available, including third-party services and database issues.For example, a database issue can produce a new service, event, and causal link in the graph.Source: 2) Constructing Causal Link · p. 6
  • Conditional rules create a causal link only when a specified boolean predicate is true, and they override a basic rule for the same event pair.The example condition requires latency spikes to occur in the same data center.Source: 2) Constructing Causal Link · p. 6
  • Graph construction applies dynamic rules first, then recursively creates links from initial events until no new links can be added.Each newly reached target event becomes an origin for further link construction.Source: 2) Constructing Causal Link · p. 6

C. Root Cause Ranking

  • GrootRank customizes PageRank to rank root causes, using weighted causal links and a personalization vector that favors dangling nodes.The default edge weight is 1 and is reduced for alerts with high false-positive rates; dangling nodes receive score 1 while other nodes receive f_n < 1.Source: C. Root Cause Ranking · p. 6
  • For tied joint events, GROOT ranks the event with the shorter summed access distance from initial anomaly services higher.An unreachable access is assigned a distance of d_m + 1.Source: C. Root Cause Ranking · p. 6
  • If access distances remain tied, GROOT ranks event types with higher historical root-cause frequency under the same trigger conditions higher.The frequency information comes from the manually labeled dataset.Source: C. Root Cause Ranking · p. 7

D. Rule Customization Management

GROOT manages rule relationships in separate graphs for within-service and cross-service links, warning users about possible conflicts when rules change. The evaluation frames accuracy, efficiency, baseline comparison, and end-to-end performance as research questions.

D. Rule Customization Management

  • GROOT represents within-service and cross-service rule relationships in separate graphs, tracking absent, basic, and conditional rules between event types.Conditional rules overwrite basic rules, and the within-service graph disallows self-loops.Source: D. Rule Customization Management · p. 7
  • When users add or update a rule, GROOT checks for an existing edge and warns about possible overwrites; removing a rule removes its graph edge.After changes, edges are converted back into rules, tested against labeled data, and reviewed through validation reports before going online.Source: D. Rule Customization Management · p. 7

V. EVALUATION

  • The evaluation assesses GROOT's accuracy and efficiency on the collected dataset and asks how accuracy compares with baselines.The evaluation also asks about accuracy and efficiency in an end-to-end scenario.Source: §V · p. 7

A. Evaluation Setup

GROOT is evaluated in eBay's production microservice ecosystem, which spans over 5,000 services and three data centers. Its labeled dataset contains 952 incidents collected over 15 months, with SRE teams determining the root cause for each case.

A. Evaluation Setup

  • The production ecosystem contains over 5,000 services across three data centers, uses varied languages and protocols, and generates 147B distributed traces per day on average.The system serves more than 159 million active buyers.Source: A. Evaluation Setup · p. 7

1) Data Set:

  • The labeled dataset contains 952 incidents collected from January 2020 through April 2021, grouped as business-domain or service-based incidents.It includes 782 business-domain incidents and 170 service-based incidents.Source: 1) Data Set · p. 7
  • SRE teams manually label and validate each root cause; when causes interact, they label only the most actionable or influential event.These labels and incident contexts serve as evaluation ground truth.Source: 1) Data Set · p. 8

2) GROOT Setup:

On the collected dataset, GROOT outperforms both implemented baselines in top-1 and top-3 accuracy. The evaluation also reports runtime costs, end-to-end performance changes, and cases where multiple co-existing causes complicate single-root labeling.

2) GROOT Setup:

  • The deployed GROOT production system consists of three microservices federated across three data centers, with nine 8-core CPU, 20GB RAM pods each on Kubernetes.Source: 2) GROOT Setup · p. 8

3) Baseline Approaches:

  • The naive baseline ranks service-level anomaly scores with PageRank, while the non-adaptive baseline replaces GROOT's special rules with basic rules.The non-adaptive approach is intended to approximate graph-based methods that lack context-aware capabilities.Source: 3) Baseline Approaches · p. 8

B. Evaluation Results

  • GROOT achieves 96% top-3 accuracy for business-domain incidents, while unsuccessful top-three rankings are mostly attributed to missing events.Adding necessary events and rules addressed more than one-third of these unsuccessful cases over time.Source: 1) RQ1 · p. 8
  • GROOT may rank the labeled root cause below top one when other causes coexist, although SRE feedback indicates it still facilitates RCA in such cases.The dataset labels only one root cause per case.Source: 1) RQ1 · p. 8
  • Average runtime is 1.06s for service-based incidents and 0.98s for business-domain incidents; end-to-end accuracy drops by up to 9 percentage points.End-to-end runtime increases by up to nearly 3 seconds, partly due to fetching data from multiple sources.Source: 1) RQ1 · p. 8, 3) RQ3 · p. 8, Table IV · p. 8
  • 78% top-1 accuracy and 95% top-3 accuracy for GROOT compare with 3% and 6% for naive, and 33% and 38% for non-adaptive, respectively.The comparison is reported for the collected dataset.Source: 2) RQ2 · p. 8
  • The naive approach performs worst because it propagates scores at the service level, while dynamic and conditional rules help GROOT form causal links that exclude irrelevant events.The authors suggest that soft errors may be ranked highly by the non-adaptive approach because they resemble injected faults.Source: 2) RQ2 · p. 8
  • Some end-to-end failures arise from production issues such as missing data and service or storage failures.The end-to-end scenario also requires fetching events from different data sources.Source: 3) RQ3 · p. 8

VI. EXPERIENCE

GROOT’s live interface presents event dependencies and incident context for SRE investigation. In surveys, users reported helpful triage support and developers generally found the system convenient to customize.

VI. EXPERIENCE

  • In a checkout incident, GROOT identified an external provider’s ErrorSpike as the root cause and ignored an irrelevant deployment event.The interface displays causal links and incident context, with event enrichment exposing raw metrics and other details on demand.Source: §VI · pp. 8–9

A. Feedback from GROOT Users and Developers

  • Users rated triage help 3.79 on average, with 11 of 14 finding GROOT very helpful; estimated triage-time savings were 25–50%.Even when GROOT does not locate the root cause correctly, users rated its graph helpful for further investigation at 3.43 on average.Source: A. Feedback from GROOT Users and… · p. 9
  • Most developers found it convenient to change and customize GROOT’s events, rules, and domains.The developer survey included six maintainers.Source: A. Feedback from GROOT Users and… · p. 9

B. Lessons learned

The deployment lessons emphasize embedding development in SRE practice, tailoring automation to operational needs, maintaining data reliability, and introducing machine learning cautiously.

B. Lessons learned

  • Embedding R&D in daily SRE work and incorporating user feedback helped make GROOT more practical and well adopted.Daily incident reviews and feedback led to UI changes including alert enrichment, interactivity, and user guides.Source: B. Lessons learned · p. 10
  • Vertical enhancements specialized GROOT for grouped cross-service alerts and critical business issues, and supported 18 business domains and sub-domains within 18 months of R&D.The UI averaged more than 50 active internal users and the service sent thousands of results monthly.Source: B. Lessons learned · p. 10
  • GROOT’s unsuccessful cases can stem from imperfect data, flawed algorithms, or code defects, and a missing critical event may lead to a very different inferred root cause.The team estimates alert accuracy should exceed 0.6 to be useful and traces requests to diagnose failures.Source: B. Lessons learned · p. 10
  • GROOT selects anomaly-detection models by balancing accuracy, scalability, efficiency, effort, and robustness against event-specific acceptance levels.The team tests models in staging; some alerts use thresholds, while stochastic service-client errors use fast statistical models operating on the fly.Source: B. Lessons learned · p. 10
  • GROOT prioritizes knowledge engineering and interpretability because auto-ML anomaly detection and unsupervised RCA for large service ecosystems are not yet ready in this context.The authors also note challenges labeling supervised training data and end users’ need to understand results when correctness is not guaranteed.Source: B. Lessons learned · p. 10

VII. CONCLUSION

GROOT applies an adaptable event-graph approach to industrial RCA and was evaluated on real production incidents. The authors report high accuracy and efficiency across scenarios, outperforming baseline graph-based approaches.

VII. CONCLUSION

  • GROOT constructs a real-time causality graph to address operational, scale, and monitoring complexities, with customization for diverse anomalies and activities.The approach is integrated into a distributed system containing more than 5,000 microservices.Source: §VII · p. 10
  • Evaluation on 952 real production incidents showed high accuracy and efficiency across scenarios and that GROOT largely outperformed baseline graph-based approaches.Source: §VII · p. 10
  • The paper also shares lessons from deploying and adopting GROOT in production environments.Source: §VII · p. 10
Loading 2108.00344v2…