PaperlayerView exact revision on arXiv

Source-linked AI summary

MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems

Dewei Liu, Chuan He, Xin Peng, Fan Lin, Chenxi Zhang, Shengfang Gong, Ziang Li, Jiayu Ou, Zheshun Wu

arXiv:2103.01782v1cs.SE

TL;DR

Availability issues in large-scale microservice systems arise when service anomalies propagate through dependencies, while existing localization methods face anomaly-detection and traversal limitations. MicroHECL dynamically builds service call graphs, analyzes anomaly propagation chains, detects multiple anomaly types, prunes irrelevant calls, and ranks causes by correlation. It outperforms two baselines in accuracy and efficiency and, in Alibaba deployment, reaches a 68% top-3 hit ratio while reducing localization time from 30 minutes to 5 minutes.

  • Problem

    Large-scale microservice availability issues require rapid root-cause localization, but existing approaches have inaccurate anomaly detection and inefficient dependency-graph traversal.

  • Method

    MicroHECL dynamically constructs service call graphs, analyzes anomaly propagation chains, uses customized models for three anomaly types, prunes irrelevant calls, and ranks causes by correlation.

  • Results

    MicroHECL significantly outperforms MonitorRank and Microscope in accuracy and efficiency; Alibaba deployment achieves a 68% top-3 hit ratio and reduces localization time from 30 minutes to 5 minutes.

  • Takeaways & Limitations

    MicroHECL provides an accurate and efficient root-cause localization approach validated by experiments and practical use in Alibaba.

  • Takeaways & Limitations

    All studies use Alibaba e-commerce data, so reported accuracy and efficiency may not apply to microservice systems in other companies or domains.

Abstract

from arXiv · show

Availability issues of industrial microservice systems (e.g., drop of successfully placed orders and processed transactions) directly affect the running of the business. These issues are usually caused by various types of service anomalies which propagate along service dependencies. Accurate and high-efficient root cause localization is thus a critical challenge for large-scale industrial microservice systems. Existing approaches use service dependency graph based analysis techniques to automatically locate root causes. However, these approaches are limited due to their inaccurate detection of service anomalies and inefficient traversing of service dependency graph. In this paper, we propose a high-efficient root cause localization approach for availability issues of microservice systems, called MicroHECL. Based on a dynamically constructed service call graph, MicroHECL analyzes possible anomaly propagation chains, and ranks candidate root causes based on correlation analysis. We combine machine learning and statistical methods and design customized models for the detection of different types of service anomalies (i.e., performance, reliability, traffic). To improve the efficiency, we adopt a pruning strategy to eliminate irrelevant service calls in anomaly propagation chain analysis. Experimental studies show that MicroHECL significantly outperforms two state-of-the-art baseline approaches in terms of both accuracy and efficiency. MicroHECL has been used in Alibaba and achieves a top-3 hit ratio of 68% with root cause localization time reduced from 30 minutes to 5 minutes.

I. INTRODUCTION

Availability issues in large-scale microservice systems can arise when service anomalies propagate through complex, dynamic call chains, making rapid root-cause localization difficult. MicroHECL addresses this challenge with dynamic graph analysis, customized anomaly detection, pruning, and correlation-based ranking, outperforming two baselines in accuracy and efficiency.

  • Challenge: Large-scale microservice systems are dynamic and complex, with anomalies propagating along service-call chains to cause business-level availability issues.Services may scale across many instances, and executions may involve numerous asynchronous interactions.
  • Existing limitations: Existing trace- and dependency-graph-based approaches are limited by expensive trace processing, inaccurate anomaly detection, or inefficient graph traversal.Industry developers often rely on visualization tools to inspect logs, traces, and propagation chains manually.
  • Approach: MicroHECL dynamically constructs a service call graph, analyzes anomalous propagation chains, detects performance, reliability, and traffic anomalies, prunes irrelevant calls, and ranks root causes by correlation.The approach targets both root-cause services and anomaly types associated with an availability issue.
  • Evaluation: MicroHECL significantly outperforms MonitorRank and Microscope in both localization accuracy and efficiency in Alibaba e-commerce studies.The evaluation uses availability-issue data collected from Alibaba’s e-commerce system.
  • Practical application: 68% top-3 hit ratio and localization time reduced from 30 minutes to 5 minutes after deployment in Alibaba.MicroHECL handled more than 600 availability issues during more than five months of deployment.

II. BACKGROUND

MicroHECL addresses business-level availability issues caused by performance, reliability, or traffic anomalies in large-scale microservice systems. It builds a current service call graph, analyzes propagation chains, and ranks candidate causes using recent metric correlations.

  • System context: Alibaba’s e-commerce system contains more than 30,000 services and uses large-scale monitoring infrastructure to support high availability.The system serves more than 846 million monthly active users.
  • Availability issues: Availability issues include declines in successful orders or transaction success rates and may result from anomalies propagating along service calls.The paper focuses on three anomaly types that cause most Alibaba availability issues.
  • Anomaly types: Performance, reliability, and traffic anomalies are indicated by anomalous response time, error count, and queries per second, respectively.Traffic anomalies may involve increases or decreases in request volume.
  • Graph construction: MicroHECL dynamically constructs a service call graph from monitored calls and metrics, recording service dependencies and quality metrics such as RT, EC, and QPS.The graph aggregates service instances and their calls over recent time windows.
  • Localization: The system traverses anomalous service-call edges to identify propagation chains and ranks candidate root causes using correlations between business and quality metrics.Ranking uses metric values from a recent time window and the absolute Pearson correlation coefficient.

IV. ANOMALY PROPAGATION CHAIN ANALYSIS

Anomaly propagation chain analysis is a staged process that identifies candidate root causes by following anomaly-specific propagation paths through the service call graph.

  • Process overview: The analysis process consists of three steps: entry node analysis, anomaly propagation chain extension, and candidate root-cause output.The steps are applied to the service call graph.

A. Analysis Process

MicroHECL uses anomaly-specific metrics and propagation directions to traverse service call graphs, extend anomalous chains, and report their terminal services as candidate root causes.

  • Analysis Process: Performance, reliability, and traffic anomalies use RT, EC, and QPS, respectively, with traffic propagating upstream-to-downstream and the other two downstream-to-upstream.The selected metric and direction depend on anomaly type.
  • Analysis Process: Entry node analysis treats the initially anomalous service as the entry node and starts chains from neighboring anomalous nodes whose relationships match the detected propagation direction.In the example, S4 starts a traffic chain and S7 starts a performance chain from entry node S5.
  • Analysis Process: Chain extension iteratively backtracks along the anomaly propagation direction, detecting neighboring anomalies and adding matching nodes until no further nodes can be added.The process extends each chain according to its anomaly type.
  • Analysis Process: When all chains terminate, MicroHECL reports the services at their endpoints as candidate root causes.The illustrated process reports S1, S9, and S10.

B. Service Anomaly Detection

MicroHECL continuously detects service anomalies from quality-metric histories, using models tailored to performance anomalies and their periodic fluctuations. Performance detection uses OC-SVM with features comparing recent response times against historical periods.

  • Detection process: MicroHECL detects possible anomalies by analyzing historical RT, EC, and QPS data for service calls during propagation-chain analysis.Different anomaly types use different analysis models based on the characteristics of their quality metrics.
  • Performance Anomaly: Periodic response-time fluctuations across daily and weekly periods make intuitive rules such as 3-sigma prone to false performance-anomaly detections.The detection task is to distinguish anomalous fluctuations from normal periodic variation.
  • Performance Anomaly: OC-SVM learns normal RT fluctuations and identifies other samples as anomalous outliers.The model uses only target-class information and is described as interpretable with strong generalization ability.
  • Performance Anomaly: The model uses four feature types: over-maximum counts, maximum-value deltas, over-average counts, and average-value ratios.These features compare the current detection window with selected historical periods.
  • Performance Anomaly: The current detection window is 10 minutes, compared with the preceding hour, the same hour on the previous day, and the same hour in the previous week.These settings produce 12 features for the OC-SVM model.
  • Evaluation: 100,000 training cases and 600 verification cases were collected from Alibaba’s monitoring infrastructure; the model achieved very high accuracy with 70% of the training set.The verification cases were distinct from training cases and had a 1:1 positive-to-negative ratio.

2) Reliability Anomaly:

Reliability anomalies concern anomalous increases in error counts, whose sparse and sometimes transient nature requires a model different from performance-anomaly detection. MicroHECL uses Random Forest with EC-related and correlated RT/QPS features.

  • Reliability Anomaly: Reliability anomalies are anomalous increases in error counts, but transient errors may not affect business outcomes or may disappear after load decreases.Circuit-breaker behavior is given as an example of errors that can recover quickly.
  • Reliability Anomaly: Random Forest trains the reliability-anomaly model by combining multiple decision trees and features to reduce overfitting.Some features combine EC with RT and QPS because anomalous EC increases often correlate with those metrics.
  • Reliability Anomaly: The five-feature design includes previous-day and previous-minute EC-delta outliers, RT-over-threshold status, maximum error rate, and EC–RT correlation.The RT threshold is exemplified as 50ms, while the correlation is measured with Pearson’s coefficient.

3) Traffic Anomaly:

Traffic anomalies are detected from anomalous QPS fluctuations. MicroHECL applies a 3-sigma rule to recent QPS values and then checks correlation with the initial service’s business metrics to reduce false positives.

  • Traffic Anomaly: Traffic anomalies are defined by anomalous fluctuations in queries per second, whose short- and long-term values follow a normal distribution.This distributional characteristic motivates statistical outlier detection.
  • Traffic Anomaly: MicroHECL uses the 3-sigma rule to detect anomalous QPS fluctuations.The rule is applied to outliers in the current detection window using the previous hour of QPS values.
  • Traffic Anomaly: A Pearson correlation check between QPS and the initial anomalous service’s business metrics further filters traffic-anomaly detections.Only detections exceeding a predefined correlation threshold are retained.

C. Pruning Strategy

MicroHECL prunes anomaly-propagation branches by comparing metric-trend correlations between adjacent service calls, targeting irrelevant edges while preserving analysis efficiency. The experiments evaluate localization accuracy, efficiency, pruning effects, and setup conditions against two baselines.

  • C. Pruning Strategy: Anomaly-propagation branches can grow exponentially, and some anomalous services or calls may be irrelevant to the reported availability issue.The pruning strategy addresses this growth during chain extension.
  • C. Pruning Strategy: Pearson correlation measures similarity between quality-metric change trends on successive service calls using recent metric histories.The graph records RT, EC, or QPS values per minute, with an example using the latest 60 minutes.
  • C. Pruning Strategy: Edges are pruned when their change-trend correlation with an adjacent upstream or downstream edge falls below a threshold such as 0.7.The check occurs before adding an anomalous node to the current propagation chain.
  • Experimental Study: The study asks whether MicroHECL localizes root causes accurately, operates efficiently at scale, and benefits from pruning across similarity thresholds.The evaluation includes localization accuracy, localization efficiency, and pruning-effect research questions.
  • Experimental Setup: The dataset contains 75 Alibaba availability issues from 28 subsystems, averaging 265 services per subsystem and spanning February to June 2020.Operation engineers annotated anomaly types and root causes; an issue may have multiple anomaly types, each with one root cause.
  • Experimental Setup: MicroHECL is compared with MonitorRank and Microscope, two state-of-the-art approaches for ranked root-cause localization.MonitorRank uses service metrics and a service call graph, while Microscope uses a service-instance causality graph.
  • Metrics: HR@k measures whether the root cause appears in the top-k results, whereas MRR is the inverse rank of the first correct answer.If the correct answer is absent, its rank for MRR is treated as positive infinity.
  • Experimental Setup: Experiments run on 15 Alibaba Cloud virtual machines with default thresholds of 50ms for RT reliability detection, 0.9 for traffic correlation, and 0.7 for pruning.The machines use 4 Intel Xeon 2.50GHz CPUs, 8 GB RAM, and 60 GB disk each.

B. Localization Accuracy (RQ1)

MicroHECL outperforms MonitorRank and Microscope in overall root-cause localization accuracy and across all three evaluated anomaly types. Its advantage is strongest for performance anomalies and least significant for traffic anomalies.

  • MicroHECL achieves top-1, top-3, and top-5 hit ratios of 0.48, 0.67, and 0.72, respectively, with an MRR of 0.58.
  • MicroHECL significantly outperforms MonitorRank and Microscope across all reported overall accuracy metrics.
  • MicroHECL outperforms both baseline approaches for performance, reliability, and traffic anomalies.
  • The accuracy advantage is most significant for performance anomalies and least significant for traffic anomalies.
  • MicroHECL uses successive-service-call correlations for pruning, whereas MonitorRank emphasizes front-end/back-end correlations and Microscope uses the 3-sigma rule for anomaly detection.

C. Localization Efficiency (RQ2)

MicroHECL is more efficient than both baselines, with advantages increasing in larger systems while execution time scales linearly with service number. Its pruning strategy reduces analysis time while preserving accuracy.

  • MicroHECL’s execution time is 22.3% lower than Microscope’s and 31.7% lower than MonitorRank’s.
  • When systems exceed 250 services, MicroHECL’s efficiency advantage increases as service number grows.
  • MicroHECL’s execution time increases linearly with service number, indicating good scalability.
  • MicroHECL prunes likely irrelevant branches and extends propagation chains in only one direction for each anomaly type.

D. Effect of Pruning (RQ3)

MicroHECL’s pruning strategy is evaluated by varying the correlation-coefficient threshold across 75 availability issues. Increasing the threshold reduced localization time while preserving accuracy through a threshold of 0.7.

  • The evaluation analyzed 75 availability issues under different correlation-threshold settings.MicroHECL’s accuracy and localization time were measured for each threshold setting.
  • 0.67 HR@3 remained unchanged as the correlation threshold increased from 0 to 0.7.The evaluation used top-3 hit ratio (HR@3) as the accuracy indicator.
  • Localization time decreased from 75 seconds to 46 seconds as the threshold increased from 0 to 0.7.Higher thresholds prune more services and service-call edges, so fewer services are reached and considered.
  • The pruning strategy significantly improved efficiency while keeping accuracy for these availability issues.The best threshold for these availability issues was 0.7.

VII. RELATED WORK

Related work includes log-, trace-, and service-call-graph-based approaches for anomaly detection and root cause localization. The paper positions MicroHECL as combining dynamically constructed call graphs, anomaly propagation analysis, customized anomaly models, and pruning, with accuracy and efficiency confirmed experimentally and in Alibaba.

  • Log analysis: Log-based approaches detect anomalies from distributed-system, cloud-application, or Kubernetes logs.Examples include unstructured log analysis, LogDC, and automatic log-based anomaly detection.
  • Visualization and analysis: Visualization-based practice uses logs, traces, topology, alarms, and application events to support fault analysis and root cause analysis.The cited systems provide views of service or component relationships and operational evidence.
  • Trace analysis: Trace-based automatic localization uses trace similarity or learned prediction models but often requires many traces for pattern extraction or training.These approaches are described as inefficient for large-scale microservice systems.
  • Graph-based localization: Service-call-graph and causality-graph approaches analyze anomaly or fault subgraphs to locate root causes in service-based systems.Examples include graph representations of architectures and MicroRCA-style performance-root-cause localization.
  • MicroHECL: MicroHECL dynamically constructs service call graphs, analyzes anomaly propagation chains, detects three anomaly types with customized models, and prunes irrelevant calls.Its accuracy and efficiency were confirmed by experiments and practical application in Alibaba.
Loading 2103.01782v1…