PaperlayerView exact revision on arXiv

Source-linked AI summary

Anomaly Detection and Failure Root Cause Analysis in (Micro)Service-Based Cloud Applications: A Survey

Jacopo Soldani, Antonio Brogi

arXiv:2105.12378v1cs.SE

TL;DR

Hundreds of interacting services make failure detection and root-cause identification difficult, motivating a survey of techniques for both tasks. The survey structures their methods, evidence, accuracy, granularity, setup costs, applicability, and integration, and discusses open challenges. It concludes that the analysis can support pipeline design, while runtime behaviour changes and correlation-based analysis constrain applicability and accuracy.

  • Problem

    Hundreds of interacting services make failures harder to detect and their possible root causes harder to identify, although both are important for recovery and repair.

  • Method

    The paper surveys and qualitatively analyzes anomaly-detection and root-cause-analysis techniques, comparing their artifacts, integration, granularity, setup costs, accuracy, and applicability.

  • Results

    The survey provides a structured basis for selecting and integrating anomaly-detection and root-cause-analysis techniques into application pipelines.

  • Takeaways & Limitations

    Application operators can use the survey to identify techniques and combinations suited to detecting and analyzing failures in multi-service applications.

  • Takeaways & Limitations

    Many surveyed techniques assume runtime behaviour resembles failure-free training runs, while root-cause methods commonly rely on correlation, which does not ensure causation.

Abstract

from arXiv · show

The momentum gained by microservices and cloud-native software architecture pushed nowadays enterprise IT towards multi-service applications. The proliferation of services and service interactions within applications, often consisting of hundreds of interacting services, makes it harder to detect failures and to identify their possible root causes, which is on the other hand crucial to promptly recover and fix applications. Various techniques have been proposed to promptly detect failures based on their symptoms, viz., observing anomalous behaviour in one or more application services, as well as to analyse logs or monitored performance of such services to determine the possible root causes for observed anomalies. The objective of this survey is to provide a structured overview and a qualitative analysis of currently available techniques for anomaly detection and root cause analysis in modern multi-service applications. Some open challenges and research directions stemming out from the analysis are also discussed.

1 INTRODUCTION

Multi-service applications improve cloud-native delivery through loosely coupled, independently deployable and scalable services, but their proliferation makes failures harder to detect and explain. The survey organizes existing detection and root-cause techniques to support pipeline selection and identifies open challenges.

  • Motivation: Hundreds of interacting services make it difficult to determine whether failures are isolated or cascading across service dependencies.Operators regard detecting and understanding such failures as a concrete operational pain.
  • Existing approaches: Existing approaches detect anomalous service behaviour as a possible failure symptom, then analyze runtime evidence to identify possible root causes.The approaches distinguish failures originating within a service from failures associated with underperforming interacting services.
  • Survey objective: The literature is scattered across anomaly detection and root cause analysis, hampering operators who need an end-to-end analysis pipeline.The survey addresses this by covering both technique families and their integration requirements.
  • Survey objective: The survey examines required input artifacts and whether techniques already combine anomaly detection with root cause analysis.It also considers whether anomalies observable by one technique can be explained by another.
  • Contribution: The survey aims to help practitioners and researchers select suitable techniques while highlighting open challenges and possible research directions.Its organization covers terminology, anomaly detection, root cause analysis, related work, and future directions.

2 TERMINOLOGY

Failures are inability to perform service functions, whereas anomalies are observable symptoms that may indicate failures at application or service level. Root cause analysis uses runtime evidence to identify possible reasons for observed anomalies and differs from debugging because it analyzes what happened during execution.

  • Core terminology: Anomalies are observable symptoms such as slower response time, reduced throughput, or logged errors, while failures are inability to perform service functions.Detection therefore seeks anomalies that can possibly correspond to service failures.
  • Core terminology: Anomalies may be detected at application level from frontend symptoms or at service level by focusing on specific services.The distinction concerns whether the application is considered as a whole or individual services are examined.
  • Root cause analysis: Root cause analysis identifies possible reasons for an application- or service-level anomaly using information about what happened during application execution.Its goal is to provide possible reasons for the corresponding failure.
  • Runtime evidence: Runtime analysis relies on KPIs such as response time, availability, and resource consumption, together with events logged by application services.Each logged event provides information about something that happened to the service producing it.
  • Runtime evidence: Distributed traces track events generated by the same user request across involved services using a shared request identifier.Instrumentation records service invocations and internal operations, including timing information.

3 ANOMALY DETECTION

The survey organizes anomaly-detection techniques according to the runtime information they process and the instrumentation they require. It distinguishes direct log processing from approaches based on distributed tracing or additional application instrumentation.

  • Technique classes: The survey separates techniques that directly process application-service logs from those requiring distributed tracing instrumentation.This organization is based on how anomaly-detection evidence is collected and processed.
  • Technique classes: Distributed-tracing techniques require applications to be instrumented so that traces can be generated for analysis.Tracing is treated as a distinct basis for anomaly detection from direct log processing.
  • Technique classes: The classification also includes techniques requiring other forms of application instrumentation beyond direct log processing and distributed tracing.The passage presents instrumentation requirements as a defining organizing dimension.

3.1 Log-based Anomaly Detection Techniques

Log-based anomaly detection typically learns service logging behaviour from failure-free runs and compares newly observed events against that baseline. The surveyed methods use control-flow structure and timing information to identify functional or performance deviations.

  • Unsupervised learning: Unsupervised machine-learning methods process service logs online to detect anomalies in application services.They learn normal logging behaviour from failure-free runs before monitoring new logs.
  • Baseline modeling: Baseline models represent expected service events and their order, then flag newly logged behaviour that diverges from the learned pattern.OASIS mines log templates and builds a control-flow graph from failure-free training logs.
  • Baseline modeling: Jia et al. and LogSed model each service with time-weighted control-flow graphs derived from normal runs.Arc weights represent the average time between logging the source and target events.
  • Anomaly types: Functional anomalies include missing expected events or unexpected logs, while performance anomalies arise when observed timing behaviour deviates from the baseline.The baseline combines service event graphs with the application topology graph.

3.2 Distributed Tracing-based Anomaly Detection Techniques

Distributed tracing-based anomaly detection uses service traces, often combined with machine learning or trace comparison, to identify anomalous application behavior. Techniques differ in whether they learn normal or failure-specific patterns and whether they compare traces against historical structures.

  • Distributed tracing supplies the traces used by supervised or unsupervised learning and by trace-comparison techniques for online anomaly detection.
  • Unsupervised Learning: Unsupervised approaches learn normal application behavior from failure-free traces using neural networks that process production traces for anomalies.TraceAnomaly uses a deep Bayesian neural network, while Nedelkoski et al. use a multi-modal long-short term memory network incorporating logged events and service response times.
  • Unsupervised Learning: Principal component analysis can identify services involved in anomalous trace interactions before unsupervised algorithms assess their runtime performance metrics.The evaluated algorithms include isolation forest, one-class support vector machine, local outlier factor, and 3σ.
  • Supervised Learning: Supervised approaches train neural networks on labelled traces distinguishing normal runs from runs with known anomalies affecting specific services.Seer uses convolutional and long-short term memory layers, with service KPIs and resource consumption as inputs and affected services as outputs.
  • Trace Comparison: Trace comparison detects anomalies by comparing newly collected traces with representative traces or call trees observed during application testing.Meng et al. and Wang et al. reduce traces to call trees and use tree-edit distance, whereas Chen et al. use matrix sketching for response-time anomalies.
  • Trace Comparison: Chen et al.’s matrix sketch represents historically non-anomalous service response times with orthogonal basis vectors and flags new values outside a tolerance threshold.Values within or sufficiently close to the represented space update the sketch; other values indicate a service performance anomaly.

3.3 Monitoring-based Anomaly Detection Techniques

Monitoring-based anomaly detection processes service or frontend KPIs, using SLO comparisons, learned baselines, supervised classifiers, or architectural and statistical models. These techniques vary in KPI scope, training assumptions, and deployment instrumentation.

  • Frontend monitoring can detect application-level performance anomalies by comparing monitored response time or KPIs with application SLOs.CauseInfer, Microscope, and ε-diagnosis use frontend monitoring, with ε-diagnosis focusing on tail latency in small time windows.
  • Unsupervised Learning: Unsupervised monitoring techniques learn baseline service behavior from failure-free runs and flag newly monitored KPIs that diverge from the baseline.Gulenko et al. and LOUD support KPI-agnostic monitoring, while MicroRCA and Wu et al. focus on response time and resource consumption.
  • Unsupervised Learning: LOUD builds an application-wide KPI baseline and causality graph, whose weighted edges represent probabilistic causal relationships between service KPIs.The relationships are computed with the Granger causality test and used during online anomaly detection.
  • Unsupervised Learning: DLA models response-time fluctuations associated with concurrent user transactions to distinguish expected variation from service performance anomalies.It uses Spearman’s rank correlation coefficient and monitoring agents installed in virtual machines hosting containerized services.
  • Offline baseline methods assume application execution conditions remain stable or that training observations adequately represent future behavior.Frontend response-time anomalies can then be detected when monitored response time differs from the expected value beyond a threshold.
  • Hora combines architectural models with statistical analysis to preemptively determine performance anomalies in multi-service applications.
  • Supervised Learning: Supervised monitoring methods inject specified failures during training, label KPI observations, and classify known anomalous service behavior at runtime.ADS uses CPU, memory, and network consumption with classifiers including nearest neighbour, random forest, naïve Bayes, and support vector machines; it assumes no more than one simultaneous anomaly.
  • Supervised Learning: PreMiSE separately models normal behavior and failure signatures by injecting failures such as packet loss, network latency, memory leaks, and CPU hogs.It treats monitored KPIs as time series and trains classifiers to detect anomalies corresponding to the injected failures.

3.4 Discussion

The survey compares anomaly-detection techniques by their classes, methods, anomaly types and granularity, while highlighting trade-offs among setup cost, accuracy, explainability and countermeasures.

  • Table 1 classifies anomaly-detection techniques as log-based, distributed tracing-based, or monitoring-based, alongside their methods, anomaly types, granularity, and required inputs.
  • Most techniques use training-run logs, traces, or KPIs to build baseline models, except SLO-checking and heartbeating approaches.Baseline models commonly use unsupervised or supervised machine learning.
  • Trace comparison can be time consuming: two of three surveyed techniques are limited to offline detection, while Chen et al. supports online detection at large scale.
  • Finer-grained anomaly detection generally requires training runs, instrumentation, deployment adaptation, or additional artifacts, creating a trade-off with setup cost.
  • Runtime conditions and application composition can change after training, challenging techniques that assume behavior remains similar to training runs.Changing resource contention, workloads, and service sets can alter performance, events, interactions, and possible failures.
  • False positives and negatives motivate baseline adaptation and explainability, while anomaly information may support countermeasures such as circuit breakers or bulkheads.Suggested adaptations include larger training sets, periodic retraining, and continuous learning.

4 ROOT CAUSE ANALYSIS

The survey reviews techniques for determining possible root causes of anomalies in multi-service applications, organizing them by the data and instrumentation they require.

  • Root cause analysis techniques are organized into log-based, distributed tracing-based, and monitoring-agent-based approaches.

4.1 Log-based Root Cause Analysis Techniques

Log-based root cause analysis uses application logs without requiring additional instrumentation, deriving causal relationships among services to explain observed anomalies.

  • Log-based analysis can operate on natively produced service logs without requiring distributed tracing or monitoring agents.
  • The approach derives a causality graph whose vertices represent services and whose directed arcs represent possible anomaly propagation between services.
  • Aggarwal et al. analyze multivariate time series from the frontend and services logging errors, applying Granger causality tests to determine causal dependencies.

4.2 Distributed Tracing-based Root Cause Analysis Techniques

Distributed-tracing-based root cause analysis ranges from manual trace comparison to automated analysis of anomalous services, interaction topologies, response times, and propagation chains.

  • Visualization-based Analysis: Trace-based root cause analysis can support manual visual comparison of anomalous traces with traces from successful executions.
  • Automated approaches either analyze traces for anomalous services or infer application topology graphs to guide root cause analysis.
  • Visualization-based Analysis: GMTA abstracts traces into ordered service-interaction paths, groups them by operator-defined business flows, and compares successful and anomalous executions.
  • Visualization-based Analysis: Trace comparison can identify root-cause events and their services, while nearby service events may help operators determine why the anomaly occurred.
  • Response-Time Analysis: CloudDiag and TraceAnomaly analyze response times in service interactions to identify anomalies that may have caused frontend anomalies.
  • Topology-Based Analysis: Topology-based methods use service interactions, timing, performance metrics, and request identifiers to analyze performance-anomaly propagation.
  • Topology-Based Analysis: MonitorRank ranks possible causes using PageRank over time-period topology graphs, with transition relevance based on correlations between service and frontend metrics.
  • Topology-Based Analysis: MicroHECL constructs a time-window topology ending at the anomaly and extends propagation chains from the affected service.

4.3 Monitoring-based Root Cause Analysis Techniques

Monitoring-based techniques use service KPIs, topology information, or causality graphs to identify possible root causes of observed anomalies. They vary in how they reconstruct application structure and search it.

  • KPI-based analysis: Monitoring agents collect service KPIs that can be compared across services to identify anomalies occurring alongside an application-level anomaly.The approaches include direct KPI comparison and offline anomaly detection using external monitoring tools.
  • KPI-based analysis: PAL and FChain analyze system-level KPIs such as CPU usage, free memory, and network traffic to explain frontend performance anomalies.Their assumption is that application performance anomalies also appear as observable changes in one or more system-level KPIs.
  • Topology-graph analysis: Topology-based techniques reconstruct or receive application graphs and use them to constrain root-cause searches to connected services, hosts, or interactions.MicroRCA derives a graph from Kubernetes deployment status, while DLA relies on an operator-provided topology.
  • Topology-graph analysis: MicroRCA uses topology-driven search, and Wu et al. extend it by applying an autoencoder to KPIs from potentially root-causing services.The refinement targets service KPIs that may have caused detected anomalies.
  • Other graph methods: Other techniques reconstruct topologies from network system calls, match anomalous subgraphs to operator-provided patterns, or transform operator-provided topologies into hierarchical hidden Markov models.These methods use dimensionality reduction, pattern matching, or probabilistic modeling to support root-cause identification.
  • Causality-graph analysis: Causality-graph techniques model dependencies among services or KPIs and commonly analyze the resulting graph with breadth-first search or random walks.The graphs are typically built from monitored time series using the PC-algorithm, while CauseInfer and Microscope apply monitoring agents to collect service interactions and response times.

4.4 Discussion

The survey compares root-cause techniques by data source, analysis method, anomaly scope, integration, and required inputs. It highlights trade-offs among diagnostic detail, setup cost, accuracy, explainability, and countermeasure support.

  • Integration and search: Root-cause analysis is often automatically triggered after anomaly detection, although some techniques rely on externally detected anomalies.Integrated pipelines start analysis when an application or service anomaly is detected.
  • Integration and search: Most techniques represent the application with topology or causality graphs, then search from the frontend or the anomalous service or KPI.Causality graphs commonly represent service or KPI dependencies and are usually derived with the PC-algorithm.
  • Comparison framework: Table 2 classifies techniques by log, distributed-tracing, or monitoring input; analysis method; pipeline integration; anomaly type and granularity; identified causes; and required artifacts.It distinguishes functional versus performance anomalies and application-level versus service-level explanations.
  • Integration and search: Random walks rank candidate causes using performance correlation, while BFS explores graph paths that could explain the anomaly.Other methods include topology-driven pairwise comparisons, pattern matching, and whole-graph processing.
  • Trade-offs: Trace-based and monitoring-based techniques provide deeper anomaly and root-cause detail than the surveyed log-based technique.Techniques differ in anomaly type, granularity, identified causes, and applicability to changing applications and runtime environments.
  • Trade-offs: Selecting a technique requires trading off the detail and ranking of identified causes against the cost of setting it up for the target application.The survey’s classification is intended to support operators in choosing a suitable technique.
  • Accuracy: Multiple candidate causes reduce false-negative risk but often introduce false positives that require operators to troubleshoot irrelevant services.False positives occur when returned services or KPIs are unrelated to the observed anomaly.
  • Accuracy: Correlation-based analysis can miss actual causes because correlation does not ensure causation, including in correlation-guided topology walks.Spurious correlations particularly affect techniques that infer causalities or choose among co-occurring anomalies using correlation alone.

5 RELATED WORK

This survey addresses a gap in prior studies by jointly reviewing anomaly detection and root-cause analysis for modern multi-service applications. It also compares their applicability and discusses open challenges.

  • Scope of prior work: Earlier surveys generally addressed either anomaly detection or root-cause analysis rather than both problems together in multi-service applications.The surveyed paper positions its contribution as a combined treatment of the two tasks.
  • Scope of prior work: Unlike generic anomaly-detection surveys, this study focuses on online detection in logs, traces, and KPIs from application services.This scope supports more detailed discussion of techniques for multi-service applications.
  • Scope of prior work: Prior root-cause surveys covered computer networks, single-program source code, or broader software systems rather than this specific application setting.The paper distinguishes its focus from these domains.
  • Related evaluation: Arya et al. quantitatively evaluated Granger-causality root-cause techniques on a common multi-service application-log dataset, complementing this survey’s qualitative comparison.Their study provides an evaluation dimension not supplied by the survey’s qualitative synthesis.
  • Contribution: The authors describe this work as the first survey jointly covering anomaly detection and possible root-cause analysis for failures in multi-service applications.It also discusses application- versus service-level analysis, instrumentation needs, and behavior under changing services and runtime environments.

6 CONCLUSIONS

The survey structures anomaly detection and root cause analysis techniques for modern multi-service applications, assessing their practical characteristics and supporting pipeline design. It identifies quantitative comparability, continuous application change, explainability, and countermeasure recommendation as important directions for future work.

  • The survey evaluates techniques by anomaly type and granularity, setup costs, accuracy, applicability to changing applications, and open challenges.It considers instrumentation and additional artifacts required by the techniques.
  • It provides a baseline for integrating independent anomaly detection and root cause analysis techniques when they address the same anomaly type and granularity.The survey also identifies which techniques already form integrated pipelines.
  • Quantitative comparison remains difficult because techniques are evaluated on different reference applications and runtime environments.The survey identifies standardized comparative evaluation as future work.
  • Continuous releases and runtime migrations motivate techniques that remain effective as multi-service applications change over time.The authors suggest adapting existing methods or using continual learning solutions.
  • Explainable-by-design techniques could help operators assess all possible root causes and exclude false positives using reasons for anomaly and cause assignments.The survey links explainability to operator decisions about which root causes to troubleshoot.
  • Automatically recommending countermeasures could reduce recurrence or propagation of failures, including through circuit breakers or bulkheads.Root cause analysis data and error logs could be further processed to suggest ways to prevent future error propagation.
Loading 2105.12378v1…