Robust Failure Diagnosis of Microservice System through Multimodal Data

1 INTRODUCTION

• Single-modal diagnosis may miss patterns spanning multiple aspects of a service instance or fail when a failure is not reflected in the chosen modality.An empirical study of hundreds of service-instance failures concludes that combining traces, logs, and metrics is crucial for accurate diagnosis.Source: §1 · pp. 1–2
• The approach targets two challenges: differing formats across metrics, logs, and traces, and imbalanced failure types, where some failures are much rarer than others.Microservice failures can propagate across instances, making timely and accurate diagnosis important.Source: §1 · pp. 1–2
• DiagFusion maps trace, log, and metric data into a shared vector space, augments data to learn from limited labeled failures, and trains a GNN on historical patterns.It combines trace and deployment data into a dependency graph to capture possible propagation paths, then localizes root-cause instances and determines failure types.Source: §1 · p. 2
• Avg@5 of 0.75 and 0.76, and F1-score of 0.84 and 0.80, accompany reported improvements of 20.9% to 368% in localization and 11.0% to 169% in failure type determination.The two datasets are from an open-source platform and a real-world microservice system; training used 160 and 80 cases, respectively.Source: §1 · p. 2

2.1 Microservice Systems and Multimodal Data

• Traces record request execution paths, logs capture service-instance events, and metrics track performance over time; each offers different diagnostic information.Traces can expose possibly faulty services, logs can reveal causes, and metrics can identify anomalous resources or bottlenecks.Source: §2.1 · pp. 2–3
• Deployment data records relationships among system assets and helps operators understand failure-propagation paths and characteristics.Source: §2.1 · p. 3

2.2 Preliminaries

• The paper uses fastText to learn unified event representations from multimodal data; its output is a real-valued vector representation.GNNs learn from node information and neighbors through message passing, and the paper uses them to learn historical failure-propagation patterns.Source: §2.2 · p. 3

2.3 Problem Statement

• Root cause instance localization is a ranking task, while failure type determination classifies a failure into a predefined set of types.Source: §2.3 · p. 3
• DiagFusion is trained on failure cases labeled with their root-cause instance and failure type; chaos engineering can generate additional cases and failure types.Source: §2.3 · p. 3

• No single modality distinguishes the patterns of GAIA’s five failure types, although traces, logs, and metrics display different anomalous patterns.The dataset contains 10 microservices, two database services, and five host machines.Source: §3 · p. 4
• 67 occurrences of Process crash, File missing, and Access denied together equal only 12% of Login failure’s 527 occurrences.Source: §3 · p. 4
• Microsoft’s completed incidents from August 2021 to August 2022 cover code, data, network, hardware, and external failures, with imbalanced category counts.Some incident details are withheld due to company policy.Source: §3 · p. 4, Figure 2 · p. 4

4.1 Design Overview

• DiagFusion extracts and timestamp-serializes events from traces, logs, and metrics, then trains event embeddings and a GNN using historical failures.Source: §4.1 · p. 4, Figure 3 · p. 4
• Traces and deployment data form a dependency graph, which the GNN combines with event representations to learn system failure propagation.Source: §4.1 · p. 4
• When an alert signals a new failure, DiagFusion performs real-time diagnosis and returns results to operators.The trained event embedding model and GNN are saved after training.Source: §4.1 · p. 4

4.2 Unified Event Representation

• Trace groups are anomalous when a field is anomalous, using 3-sigma for numerical fields and occurrence-count increases for categorical fields.Extracted trace events contain a timestamp, caller instance, and callee instance.Source: §4.2 · pp. 4–5
• Drain parses logs into templates and variable fields, then hashes each template into an event-template ID.Log events are represented as <timestamp, instance-id, event-template-id>.Source: §4.2 · p. 5
• Metric events are detected with 3-sigma thresholds, and their records include the instance, metric name, and anomaly direction.Source: §4.2 · p. 5
• Events share timestamp and instance-id fields, enabling grouping by instance and serialization in timestamp order.Figure 4 illustrates extraction and serialization for one instance.Source: §4.2 · p. 5, Figure 4 · p. 5
• Operators’ post-failure labels are assigned instance-wise: the root-cause instance receives the failure type and other instances receive “non-root-cause.”With p historical failures and q instances, relabeling yields N = p×q event sequences.Source: §4.2 · p. 5
• FastText is trained on relabeled event sequences to produce vector representations for events from all three modalities.Failure types serve as document labels in the adapted FastText task.Source: §4.2 · p. 5

4.3 Graph Neural Network

• A GNN provides a system-wide view by learning failure propagation between instances and integrating information across the system.Source: §4.3 · p. 5
• Instances are graph nodes represented by the average of their events, while graph edges encode call relationships and co-deployment resource sharing.Call-graph pairs receive directed edges in both directions; co-deployed instances are also connected.Source: §4.3 · p. 5
• Topology-adaptive graph convolution updates instance representations through message passing, followed by MaxPooling and a fully connected prediction layer.The output neurons represent service groups with possible root-cause instances or failure types.Source: §4.3 · p. 6

4.4.1 Training of Event Embedding Model

• Augmentation expands each failure type’s event sequences by replacing an event with its nearest neighbor, balancing the training set before the event embedding model is retrained.The final unified representations are generated by the retrained model on the expanded data.Source: §4.4.1 · p. 6

4.4.2 Training of Graph Neural Network

• DiagFusion trains task #1 on service groups, then ranks instances within a candidate group by event-sequence length to accommodate changing deployments.Instances with more anomaly events rank higher and are considered more likely to be the root cause.Source: §4.4.2 · p. 6
• Joint learning lets the GNN use shared information between the two diagnosis tasks, with one loss covering service-group and failure-type labels and predictions.Training two models separately would require twice the training time.Source: §4.4.2 · p. 6

4.5 Real-time failure diagnosis

• After training, DiagFusion saves the event embedding model and GNN for use when a new failure is alerted.The real-time diagnosis process is shown in Figure 5.Source: §4.5 · pp. 6–7

4.5.1 Running Example

• Figure 7 aligns B1’s serialized multimodal event sequence, its original data, and part of the failure’s dependency graph.The example presents the data and graph alongside the diagnosis workflow.Source: Figure 7 · p. 7, §4.5.1 · p. 7
• For an access-denied failure, the GNN predicts service group B and the failure type, then within-group ranking identifies B1 as the Top1 instance in under 10 seconds.The example’s event sequence combines failure-indicative events from different modalities over time.Source: §4.5.1 · p. 7

5 EVALUATION

• The study compares performance, component contributions, computational efficiency, and hyperparameter effects, splitting each dataset chronologically to avoid data leakage and averaging five experiment runs.The experiments use a CPU server without GPUs.Source: §5 · p. 7, §5.1.1 · p. 7, §5.1.4 · p. 8

5.1.1 Dataset

• The evaluation uses two datasets from systems with different business backgrounds and architectures; D2 comes from a commercial bank and covers five operator-classified failure types.D2’s studied system has 14 instances and its records span January through June 2021.Source: §5.1.1 · p. 7

5.1.2 Baseline Methods

• Baselines include six single-modal and two multimodal methods, divided by which diagnosis task each can address.Cloud19, LogCluster, and CloudRCA cannot address instance localization; five other baselines cannot determine failure type.Source: §5.1.2 · p. 7

5.2 Overall Performance (RQ1)

• Avg@5 exceeds 0.75 on both datasets; Task #2 F1-score is 0.80 on D2, at least 0.195 higher than baselines.On D1, Task #2 precision, recall, and F1-score are all over 0.80; Task #1 Avg@5 is at least 0.13 higher than single-modal baselines and at least 0.18 higher than PDiagnose.Source: Figure 8 · p. 8, §5.2 · p. 8, Table 3 · p. 8

5.1.3 Evaluation Metrics

• Task #1 uses Top-k accuracy and Avg@5, while Task #2 uses weighted average precision, recall, and F1-score.Top-k accuracy measures whether the output’s top-k instances contain the root cause; Avg@5 reflects localization capability across the top five.Source: §5.1.3 · p. 8

5.3 Ablation Study (RQ2)

• DiagFusion outperforms all five ablation variants on both datasets; removing augmentation, replacing fastText, or replacing the GNN output layer degrades performance.The authors attribute the augmentation effect to reduced majority-class bias, fastText’s use of operator labels and event co-occurrence, and the GNN’s capture of graph interactions and fault propagation.Source: §5.3 · pp. 8–9, Table 4 · p. 9

5.4 Efficiency (RQ3)

• DiagFusion diagnoses one failure in 12 seconds on average online, below the datasets’ minimum 30-second data-collection interval.The authors conclude that it meets online diagnosis needs despite lacking apparent advantages among methods in the runtime comparison.Source: Table 5 · p. 9, §5.4 · p. 9

5.5 Hyperparameter Sensitivity (RQ4)

• Sensitivity tests find dataset-dependent embedding dimensions, possible degradation from excessive augmented samples, better performance with fewer than three GNN layers, and little time-window impact.The selected embedding dimension is 100; excessive augmentation can introduce noise, while deeper GNNs require extra training samples.Source: §5.5 · p. 9

6.1 Why Learning-Based Methods?

• Learning-based methods are chosen for accuracy, generalization across systems, and handling complex, high-dimensional monitoring data without manually configured rules.DiagFusion uses fastText for unified event representation and a GNN; the authors say its training sets contain 160 and 80 samples for D1 and D2, respectively.Source: §6.1 · pp. 9–10
• FastText sequences traces, logs, and metrics by shared timestamps, and the authors report better performance than word2vec and GloVe.They consider ELMo, BERT, and GPT unsuitable because there are too few failure cases to train those large models.Source: §6.1 · p. 9
• GNNs learn domain knowledge from graph data and are described as more generalizable than Random Walk approaches across systems.The graph structure represents service instances and their relationships.Source: §6.1 · pp. 9–10

6.2 Robustness

• On 62 D1 failure cases lacking metrics, DiagFusion remains better than PDiagnose and other Task #1 baselines despite performance degradation.The authors attribute this robustness to a unified representation learned from complete modalities during training; PDiagnose treats modalities independently.Source: Table 6 · p. 10, §6.2 · p. 10

6.3 Concerns about Deployment and Validity

• DiagFusion can accommodate instance creation or destruction through service groups, but requires retraining when a new service group is created.The authors describe new service-group creation as very rare; the system can work when any two of the three modalities are available.Source: §6.3 · p. 10
• The evaluation is bounded by relatively small datasets and failure cases that include simpler-than-industrial examples and only a limited range of failure types.The authors identify these as threats to validity while expressing optimism about application to larger industrial systems and more complex failures.Source: §6.3 · pp. 10–11

• Metric-based methods build dependency graphs or extract failure features, but some depend heavily on parameter settings and others are affected by the scarcity of failure cases.The cited examples include Microscope, MS-Rank, AutoMAP, Graph-RCA, and iSQUAD.Source: §7 · p. 11
• Trace-based methods such as TraceRCA, MEPFL, MicroHECL, and MicroRank often emphasize global system features rather than local service-instance features.Source: §7 · p. 11
• Log-based methods use clustering, embeddings, contrast analysis, or anomaly detection, but often ignore microservice topology.Examples include LogCluster, Cloud19, Onion, DeepLog, and LogFlash.Source: §7 · p. 11
• CloudRCA and PDiagnose combine modalities but ignore system topology, while Groot relies on numerous predefined rules.CloudRCA combines metrics and logs; PDiagnose combines metrics, logs, and traces.Source: §7 · p. 11
• DiagFusion considers metrics, logs, and traces and is among the first multimodal methods to represent different modalities in a unified manner.The authors characterize it as more robust and accurate than existing multimodal-based methods.Source: §7 · p. 11

• DiagFusion combines trace and deployment data into a dependency graph and integrates that graph with event embeddings through a GNN.Source: §8 · p. 11
• The method extracts events from traces, metrics, and logs, uses fastText to unify their representations, and applies data augmentation to address imbalanced failures.Source: §8 · p. 11
• Evaluations on two real-world datasets confirm DiagFusion's effectiveness and efficiency in reporting root-cause instances and failure types.Source: §8 · p. 11