PaperlayerView exact revision on arXiv

Source-linked AI summary

Autoregressive Multimodal Anomaly Detection with Large Language Models for Microservice Systems

Yixiang Tang, Fagui Liu, Guoxiang Zhong, Xi Yao, C. L. Philip Chen

arXiv:2610.31722v1cs.LGcs.AIcs.SE

TL;DR

Existing multimodal anomaly detection can neglect modality-specific cues, and reconstruction-based methods can suffer identity shortcuts. LLMMAD processes multimodal embeddings with an autoregressive LLM while retaining modality-wise reconstruction, and it reports higher F1-scores than the previous state-of-the-art method on three datasets.

Abstract

from arXiv · show

Microservice systems have become a dominant architecture for large-scale cloud applications, enabling rapid deployment and elastic scaling. However, ensuring the reliability and availability of microservice systems critically relies on effective anomaly detection, as failures and performance anomalies can quickly propagate across services and severely impact end-to-end system behavior. Existing multimodal anomaly detection approaches for microservice systems often struggle with two fundamental limitations: (i) the neglect of modality-specific characteristics in favor of cross-modality fusion, and (ii) the "identity-shortcut" problem that affects the feature reconstruction process. To address these issues, we propose LLMMAD, an autoregressive multimodal anomaly detection framework for microservice systems based on Large Language Models (LLMs). Initially, to reconcile cross-modality fusion with modality-specific preservation, LLMMAD embeds and recombines multimodal data into a unified sequence for LLM processing to model inter-modality relationships, while employing modality-wise reconstruction heads to retain modality-specific information. Subsequently, to eliminate the identity-shortcut, we implement a prediction-driven autoregressive objective, under which the model predicts future states rather than copies current inputs. Finally, we conduct extensive experiments on three real-world datasets. The results demonstrate that LLMMAD consistently outperforms state-of-the-art methods, achieving F1-score improvements of 3.1%, 1.4%, and 0.9%, respectively, thereby providing a more reliable solution for microservice monitoring.

I. INTRODUCTION

Microservice anomaly detection must account for signals spread across metrics, logs, and traces, yet existing multimodal methods can underrepresent modality-specific information and reconstruction models can fall into identity shortcuts. LLMMAD uses an LLM-based autoregressive design to model cross-modal relationships while preserving modality-specific cues.

I. INTRODUCTION

  • Multimodal methods may neglect modality-specific information, while reconstruction models can copy inputs through an identity shortcut, weakening anomaly representations.Single-modality methods also miss signals associated with some faults because each modality provides an incomplete view of system state.Source: §I · pp. 1–2
  • LLMMAD combines unified LLM processing of chronologically ordered modality embeddings with modality-wise reconstruction and future-state prediction.The framework also unifies modality decisions with OR-logic aggregation for fine-grained detection.Source: §I · p. 2
  • The paper reports comprehensive experiments on three real-world microservice datasets and says LLMMAD outperforms existing anomaly detection approaches.The supplied introduction gives no quantitative results for this comparison.Source: §I · p. 2

A. Cross-Modality Knowledge Transfer via LLMs

  • The paper identifies mapping heterogeneous microservice modalities into a shared language representation as an unexplored cross-modal transfer direction.It argues this shared sequential representation can support modeling complex cross-modal dependencies.Source: A. Cross-Modality Knowledge Transfer… · p. 2

B. “Identity Shortcut” in Anomaly Detection

The section reviews why reconstruction-based anomaly detection can fail through identity shortcuts, and why single- and multimodal approaches may miss cross-modal or modality-specific signals. It then defines the paper’s multimodal, per-timestep detection task.

B. “Identity Shortcut” in Anomaly Detection

  • Reconstruction models can converge to near-identity mappings that reconstruct anomalous as well as normal samples, undermining reconstruction-based detection.The cited analysis reports that MLPs, CNNs, and Transformers are prone to trivial identity solutions under reconstruction objectives.Source: B. “Identity Shortcut” in Anomaly… · p. 3

1) Single-Modality Approaches:

  • Single-modality approaches detect only anomalies expressed in their chosen modality and can miss cross-modal correlations and cascading behaviors.The section gives examples of log-, metric-, and trace-based approaches.Source: 1) Single-Modality Approaches · p. 3

2) Multi-Modality Approaches:

  • Multimodal methods may prioritize a holistic system representation, leaving modality-specific signatures insufficiently represented in fused features.This can limit detection robustness for anomalies concentrated in one modality.Source: 2) Multi-Modality Approaches · p. 3

A. Symbol Definition

  • Table II summarizes symbols for multimodal data, with other symbols explained where they first appear.The section notes that the three modalities require many symbols.Source: A. Symbol Definition · p. 3

B. Problem Formulation

  • The task is to learn a function that jointly uses metrics, logs, and traces to identify anomalies at each timestep in a sliding window.The window’s timesteps are equally spaced, and the output labels distinguish anomalous from normal states.Source: B. Problem Formulation · p. 4

IV. METHOD

LLMMAD processes metric, log, and trace data through unified input matrices to produce anomaly scores and a threshold. Its time-alignment step addresses the differing temporal structures of these modalities.

IV. METHOD

  • Metrics, logs, and traces are collected and fed as unified input matrices to the LLM-based model, which produces anomaly scores and a threshold for alerting.An alert is triggered at a timestep when its anomaly score exceeds the threshold.Source: §IV · p. 4

A. Time Alignment

  • Event-driven logs and variable-count traces must be aligned with the fixed temporal structure of regularly sampled metric data for unified processing.The paper presents Fig. 2 to illustrate the alignment process.Source: A. Time Alignment · p. 4, Figure 2 · p. 4

1) Metric:

The section describes metric aggregation and log alignment as ways to represent operational data on a common temporal structure. Log features combine event occurrence with template semantics.

1) Metric:

  • Metric vectors from all instances are concatenated along the metric dimension to form an extended metric time series.The resulting vector at each timestep contains the metrics of all instances.Source: 1) Metric · p. 4

2) Log:

  • Log messages are parsed into templates, whose binary occurrence over time is multiplied by their Word2Vec-based semantic embeddings to form aligned log features.Each timestep receives a D-dimensional feature vector independent of the template-set size and its expansion.Source: 2) Log · pp. 4–5

3) Trace:

The section describes how trace spans are aggregated over time and how multimodal features are prepared and embedded for LLMMAD. It also addresses metric scale heterogeneity through staged normalization.

3) Trace:

  • Trace spans are aggregated by caller-callee-operation group into a temporal matrix, with duration statistics such as P99, P95, mean, median, or standard deviation selectable.The passage also describes time-aligned multimodal representations.Source: 3) Trace · p. 5, B. Normalization · p. 5

B. Normalization

  • Metric normalization addresses scale heterogeneity with logarithmic compression for excessively large values, rescaling for extremely small values, and variable-wise robust normalization across modalities.The robust formulation uses the median and interquartile range, which alleviates outlier and heavy-tailed distribution effects.Source: B. Normalization · p. 5

1) Embedding Layers:

  • LLMMAD’s embedding layers smooth temporal features with a 1D CNN, project them into the LLM embedding space, and apply layer normalization.Fig. 3 colors intermediate feature matrices by modality: metrics are light blue, logs light yellow, and traces light red.Source: 1) Embedding Layers · p. 5, Figure 3 · p. 5

2) LLM w/ LoRA:

LLMMAD uses a frozen LLM backbone with LoRA modules, then recombines modality embeddings for processing and separates its outputs back into modality streams. Its one-step-ahead predictions align outputs with future timesteps.

2) LLM w/ LoRA:

  • The LLM backbone retains position embeddings and transformer layers while removing its tokenizer, token embeddings, and prediction head; frozen parameters are adapted with LoRA.The authors state this reduces training costs while maintaining generalization capabilities.Source: 2) LLM w/ LoRA · p. 6

3) Recombination and Inverse Recombination:

  • Recombination interleaves modality embeddings in temporal order, while inverse recombination partitions the LLM output into modality-specific streams.Together, the operations support multimodal fusion and separation and make the data compatible with autoregressive LLMs.Source: 3) Recombination and Inverse… · p. 6
  • During training, every timestep except the first is predicted; during detection, only a fixed proportion of the sliding window’s latter part is predicted.The detection window uses its first half for historical context and its second half for anomaly detection.Source: Figure 4 · p. 6
  • Because each output predicts the next timestep, inverse recombination shifts the modality order by one position relative to the recombined input.The stated order changes from metric, log, trace in the input to log, trace, metric in the output.Source: 3) Recombination and Inverse… · p. 6

4) Modality-Wise Reconstruction:

LLMMAD independently reconstructs each modality while training its outputs to predict the next timestep. Causal context supports this prediction objective and discourages copying the current input.

4) Modality-Wise Reconstruction:

  • Each modality is reconstructed independently through its own linear layer, preserving modality-specific features for loss calculation and anomaly detection.The outputs are obtained after inverse recombination.Source: 4) Modality-Wise Reconstruction · pp. 6–7

1) Prediction-Driven Autoregressive Learning:

  • The model uses one-step-ahead prediction: the reconstructed output at timestep t predicts that modality’s input at timestep t + 1.The first sliding-window timestep is excluded from the loss, which uses one-step-ahead MSE.Source: 1) Prediction-Driven Autoregressive… · p. 7
  • Causal masking limits prediction of timestep t + 1 to preceding context, preventing the model from simply copying input embeddings.The prediction objective is also aligned with decoder-only LLM autoregressive training, reducing fine-tuning difficulty and overhead.Source: 1) Prediction-Driven Autoregressive… · p. 7

2) Anomaly Detection with OR-Logic Aggregation:

For detection, LLMMAD uses a history portion of each sliding window to predict the detection portion, then combines modality-specific anomaly decisions with OR logic. The experiments are framed around detection accuracy, component contributions, efficiency, identity-shortcut mitigation, and robustness.

2) Anomaly Detection with OR-Logic Aggregation:

  • At detection time, the sliding window is split into historical context and a detection portion, with anomaly scores calculated only on the latter.The history provides additional context for predictions on the detection window; the history-to-window proportion is tunable.Source: 2) Anomaly Detection with OR-Logic… · p. 7
  • Each modality’s score is thresholded into a binary judgment, and the final timestep label is anomalous if any modality is anomalous.OR logic combines the three modality decisions.Source: 2) Anomaly Detection with OR-Logic… · p. 7

A. Datasets

The evaluation uses three publicly available real-world microservice datasets with varied systems, faults, and data scales. LLMMAD achieves the best F1-score on all three and improves over FUSION by 3.1%, 1.4%, and 0.9%, respectively.

A. Datasets

  • The SN and TT datasets use SocialNetwork and TrainTicket benchmarks with manually injected CPU overload, network delay, and packet-loss faults.MSDS comes from an OpenStack-based microservice cloud system and uses concurrent-request data with container restart, service API, and network agent disruption faults.Source: A. Datasets · pp. 7–8
  • The datasets vary in duration, instance count, and log and span volumes, supporting evaluation across different system scales and temporal characteristics.Experimental settings use normal periods for training on SN and TT; MSDS is split 6:1:3, with anomalous samples removed from training.Source: A. Datasets · p. 8, B. Experimental Settings · p. 8

C. RQ1: How accurate is LLMMAD in detecting microservice anomalies?

  • 3.1%, 1.4%, and 0.9% F1-score improvements over FUSION are reported for the three datasets, where LLMMAD achieves the best F1-score on each.The comparison covers precision, recall, and F1-score; the authors report a favorable precision-recall balance.Source: C. RQ1: How accurate is LLMMAD in… · p. 8
  • Across the datasets, three-modality methods outperform two-modality methods, which in turn outperform single-modality approaches.The reported hierarchy includes LLMMAD, FUSION, and Triple among three-modality methods; SCWarn and DeepTralog among two-modality methods; and LogAnomaly, TraceAnomaly, and AnomalyTransformer among single-modality methods.Source: C. RQ1: How accurate is LLMMAD in… · p. 8

D. RQ2: How does each component and modality contribute

Ablations report contributions from prediction-driven learning, LoRA, modality inputs, OR aggregation, and robust embeddings. The efficiency analysis reports a small trainable parameter count and training and inference times compared with multimodal baselines.

to the effectiveness of LLMMAD?

  • Removing any single modality degrades detection, removing two degrades it further, and excluding metrics causes the largest performance drop.The authors describe the modalities as complementary and metrics as dominant in anomaly analysis.Source: to the effectiveness of LLMMAD? · p. 8
  • Removing robust embedding layers slightly degrades performance across all datasets.These layers are introduced to improve robustness and generalization.Source: to the effectiveness of LLMMAD? · p. 9
  • Removing prediction-driven learning degrades performance, while disabling LoRA leaves non-trivial detection capability but consistently higher precision than recall.The authors interpret the prediction result as support for temporal-regularity learning over identity mappings, and the LoRA result as evidence that trainable parameters adapt the LLM to multimodal detection.Source: to the effectiveness of LLMMAD? · p. 9
  • Replacing OR aggregation with majority voting sharply lowers recall and significantly lowers F1-score, despite a slight precision improvement.The authors report that this conservative scheme misses anomalies and suggest failures may appear in only one modality.Source: to the effectiveness of LLMMAD? · p. 9

scale and time overhead?

  • LLMMAD updates approximately 260K parameters, less than 0.5% of total model parameters, with statistics reported using SN as a representative dataset.Embedding and output layer parameter counts vary slightly across datasets because their input dimensionalities differ.Source: scale and time overhead? · p. 9, Table VI · p. 10
  • Training time is comparable to Triple and substantially faster than FUSION; inference is only slightly slower than both and remains sufficiently low for practical online anomaly detection.The reported training-inference time gap is small, which the authors attribute mainly to the small number of trainable parameters.Source: scale and time overhead? · pp. 9–10, Figure 5 · p. 10

F. RQ4: How effectively does LLMMAD mitigate the “identity shortcut” problem?

LLMMAD’s prediction-driven objective yields higher anomaly scores for anomalous samples than its ablated variant, while results across tested LLM backbones and settings show limited sensitivity in several respects. The conclusion presents LLMMAD as an unsupervised multimodal framework that combines unified LLM processing with modality-wise reconstruction.

F. RQ4: How effectively does LLMMAD mitigate the “identity shortcut” problem?

  • LLMMAD assigns anomalous samples higher anomaly scores than LLMMAD w/o prediction, while scores for normal samples are generally similar across three datasets.The comparison uses overall anomaly scores averaged across the three modalities; LLMMAD also has higher medians, upper quartiles, and maxima for anomalous samples.Source: F. RQ4: How effectively does LLMMAD… · p. 10, Figure 7 · p. 11

G. RQ5: How robust is LLMMAD to different LLM backbones and hyperparameter settings?

  • Performance differences among tested LLM backbones are marginal; increasing model size generally raises Recall while slightly reducing Precision.The Precision change is attributed as likely related to higher-dimensional embeddings and overfitting, and is less evident on the TT dataset, which has the most training data.Source: G. RQ5: How robust is LLMMAD to… · p. 10, Table VII · p. 11
  • Window lengths of at least 20 yield relatively stable performance across datasets, while shorter windows degrade performance, most at length 10.The text attributes this to short windows truncating long-range dependencies needed to capture sufficiently complete temporal patterns.Source: G. RQ5: How robust is LLMMAD to… · p. 10, Figure 8 · p. 11
  • F1 scores improve as partition ratio γ increases, with marginal gains above 50%; selecting γ balances detection performance against practical efficiency.Larger γ incorporates more history per detected timestep, whereas smaller γ permits more timesteps to be reconstructed in parallel.Source: G. RQ5: How robust is LLMMAD to… · p. 10, Figure 8 · p. 11
  • Across six modality input permutations, performance variations are negligible on all datasets, indicating insensitivity to input order.The text links this observation to the recombination mechanism integrating multimodal information without relying on a specific order.Source: G. RQ5: How robust is LLMMAD to… · pp. 10–11, Figure 8 · p. 11

VI. CONCLUSION

  • LLMMAD interleaves multimodal embeddings for unified LLM processing and uses modality-wise reconstruction heads to preserve modality-specific anomaly cues.Its prediction-driven autoregressive objective infers future states from historical context rather than copying inputs.Source: §VI · p. 11
Loading 2610.31722v1…