PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Graph Structures with Transformer for Multivariate Time Series Anomaly Detection in IoT

Zekai Chen, Dingshuo Chen, Xiao Zhang, Zixuan Yuan, Xiuzhen Cheng

arXiv:2104.03466v3cs.LGcs.CReess.SY

TL;DR

IoT anomaly detection must handle large multivariate time series whose sensor dependencies are complex, nonlinear, and initially unknown. GTA learns a directed sensor graph, models influence propagation and temporal dependencies with graph convolutions and a Transformer architecture, and replaces standard attention with multi-branch attention. On four real-world datasets, GTA outperformed state-of-the-art approaches in prediction accuracy, including best F1-scores of 0.91 on SWaT and 0.84 on WADI.

  • Problem

    IoT anomaly detection involves large multivariate time series with complex, nonlinear sensor dependencies that are initially unknown, alongside temporal dependency and stochasticity.

  • Method

    GTA learns a bi-directed sensor graph with Gumbel-Softmax, uses Influence Propagation convolution for anomaly information flow, and models temporal context with a Transformer using multi-branch attention.

  • Results

    GTA outperformed state-of-the-art approaches across four real-world datasets, achieving best F1-scores of 0.91 on SWaT and 0.84 on WADI.

  • Takeaways & Limitations

    The framework supports anomaly detection by jointly learning sensor dependencies, modeling influence flow, and improving Transformer inference speed without sacrificing model performance.

Abstract

from arXiv · show

Many real-world IoT systems, which include a variety of internet-connected sensory devices, produce substantial amounts of multivariate time series data. Meanwhile, vital IoT infrastructures like smart power grids and water distribution networks are frequently targeted by cyber-attacks, making anomaly detection an important study topic. Modeling such relatedness is, nevertheless, unavoidable for any efficient and effective anomaly detection system, given the intricate topological and nonlinear connections that are originally unknown among sensors. Furthermore, detecting anomalies in multivariate time series is difficult due to their temporal dependency and stochasticity. This paper presented GTA, a new framework for multivariate time series anomaly detection that involves automatically learning a graph structure, graph convolution, and modeling temporal dependency using a Transformer-based architecture. The connection learning policy, which is based on the Gumbel-softmax sampling approach to learn bi-directed links among sensors directly, is at the heart of learning graph structure. To describe the anomaly information flow between network nodes, we introduced a new graph convolution called Influence Propagation convolution. In addition, to tackle the quadratic complexity barrier, we suggested a multi-branch attention mechanism to replace the original multi-head self-attention method. Extensive experiments on four publicly available anomaly detection benchmarks further demonstrate the superiority of our approach over alternative state-of-the-arts. Codes are available at https://github.com/ZEKAICHEN/GTA.

I. INTRODUCTION

GTA addresses multivariate IoT anomaly detection by learning hidden sensor dependencies, modeling influence propagation, and improving Transformer efficiency. The framework combines graph learning, graph convolution, temporal context encoding, and multi-branch attention, with experiments on anomaly-detection benchmarks.

  • Motivation: IoT infrastructures generate large multivariate time series and require anomaly detection because cyberattacks increasingly affect critical cyber-physical systems.Examples include smart power grids, water networks, transportation, and autonomous cars.
  • Motivation: Sensor measurements exhibit complex topological and nonlinear correlations, but their dependencies are initially hidden and costly to access.The paper motivates entity-level detection because overall system status is generally more important and less expensive to monitor than individual sensors.
  • Related limitations: Existing reconstruction- and forecasting-based methods improve anomaly detection but generally do not explicitly learn topological structure among sensors.This leaves high-dimensional sensor interrelationships insufficiently modeled.
  • GTA framework: GTA learns a global bi-directed sensor graph with a differentiable Gumbel-Softmax connection policy that discovers hidden associations during training.The policy determines information-flow directions while restricting neighborhood scope for inference efficiency.
  • GTA framework: GTA introduces Information Propagation convolution and combines graph convolution with multi-scale dilated convolution for hierarchical temporal-context encoding.The graph operation models anomaly influence flow between sensors.
  • Efficiency and evaluation: A multi-branch attention mechanism addresses the quadratic-complexity challenge of conventional multi-head attention, and extensive benchmark experiments evaluate GTA against state-of-the-art methods.The paper frames the attention design as an efficiency-oriented contribution and reports broad empirical validation.

II. RELATED WORK

Prior multivariate anomaly-detection work models variables independently or exploits correlations, while graph methods extend convolution to irregular sensor relationships. GTA's formulation uses normal data for forecasting-based detection and represents sensor dependencies with a directed graph.

  • Anomaly detection in multivariate time series: Time-series anomaly detection has used independent-variable modeling or multivariate correlation modeling to improve detection performance.The related-work literature is organized around these two broad approaches.
  • Anomaly detection in multivariate time series: Traditional anomaly-detection frameworks estimate values and compare them with observations using dynamically adjusted thresholds.This estimation-then-detection structure is described for univariate time series.
  • Anomaly detection in multivariate time series: Multivariate methods exploit relationships among variables through statistical features, nonlinear functions, and temporal-correlation models such as LSTMs.These methods seek improved anomaly-detection accuracy by modeling cross-variable dependence.
  • GTA architecture: GTA's architecture combines dilated convolution and graph convolution across three encoder layers and one decoder layer.Input time series are split into train sequences for the encoder and label sequences for the decoder.
  • Graph-based modeling: Graph neural networks generalize convolution from regular grids to irregular, non-Euclidean structures and capture correlations between nodes.This makes graph-based modeling relevant to sensor systems with non-grid relationships.
  • Problem statement: The problem formulation treats each timestamp as an M-dimensional sensor vector and constructs sequence modeling on normal data before testing on data containing anomalies.M denotes the number of sensors or measuring nodes within an entity.
  • Problem statement: Given n historical multivariate observations, GTA produces an anomaly score for each testing timestamp and obtains binary anomaly decisions through threshold selection.The formulation uses forecasting-based single-step prediction for anomaly detection.
  • Graph-based modeling: A directed graph represents sensors as nodes and edges as one-way information flow, with each node's neighborhood containing its outgoing neighbors.The paper uses this structure to formulate sensor dependencies.

IV. METHODOLOGY

GTA learns a directed graph over sensors using differentiable Gumbel-Softmax sampling, avoiding top-K distance-based connectivity and reducing graph-learning complexity. The learned structure supports global associations while remaining trainable with gradient descent.

  • A. Gumbel-Softmax Sampling: GTA learns directed sensor connections automatically through a graph structure learning policy based on Gumbel-Softmax sampling.The policy determines whether information should flow between node pairs during training.
  • A. Gumbel-Softmax Sampling: Binary connection variables represent whether information flows from node i to node j.The connection-control variable is sampled for each pair of candidate nodes.
  • A. Gumbel-Softmax Sampling: The Gumbel-Softmax trick replaces nondifferentiable arg max sampling with a Softmax reparameterization for optimization.Gumbel noise is sampled independently of the policy logits, enabling standard gradient descent.
  • A. Gumbel-Softmax Sampling: As the temperature τ approaches 0, the Gumbel-Softmax distribution becomes identical to a one-hot categorical distribution.This provides a smooth training relaxation that approaches discrete connection selection.
  • A. Gumbel-Softmax Sampling: The proposed graph learning reduces computation complexity from O(M^2) to O(1) and avoids restricting neighborhoods to top-K nearest sensors.It learns global topological connections without dot products among high-dimensional node embeddings.

B. Influence Propagation via Graph Convolution

Influence Propagation convolution updates each sensor representation by aggregating neighbor differences over learned graph edges. The design targets anomaly propagation while regularization and initialization improve graph compactness and convergence.

  • B. Influence Propagation via Graph Convolution: IP convolution updates each node representation by aggregating information from neighbors connected through the learned topology.The operation models influence propagation over edges emanating from each node.
  • B. Influence Propagation via Graph Convolution: IP convolution applies a symmetric aggregation operation to differences between nodes associated with outgoing edges.Summation is used as the aggregation operation, with a neural network hΘ processing the combined representations.
  • B. Influence Propagation via Graph Convolution: The difference xj − xi explicitly models influence propagation delay from node j to node i through timestamp-wise embedding differences.The additional xi + xj term provides a scale benchmark for generalized anomaly impact.
  • B. Influence Propagation via Graph Convolution: The framework combines hierarchical dilated convolution with graph convolution to represent temporal and topological context.Figure 3 visualizes this combined progression.
  • B. Influence Propagation via Graph Convolution: A sparsity regularization encourages compact subgraphs by minimizing the likelihood of establishing connections, while complete-graph warm-up supports convergence.The compact structure is intended to omit redundant connections without deteriorating forecasting accuracy.

C. Hierarchical Dilated Convolution

Hierarchical dilated convolution extracts temporal patterns across multiple sequence lengths and receptive fields, while graph convolution captures sensor relationships. Their combined outputs provide temporal-context embeddings for Transformer forecasting.

  • C. Hierarchical Dilated Convolution: Dilated convolution captures high-level temporal context by extracting sequential patterns with different dilation rates.Different dilation levels expose temporal patterns across various ranges and support long-sequence modeling.
  • C. Hierarchical Dilated Convolution: Multi-scale dilation sizes explore temporal contexts with different sequence lengths and receptive fields.The hierarchical design applies repeated dilated convolutions with increasing dilation rates and filter width two.
  • C. Hierarchical Dilated Convolution: Hierarchical dilated convolution and graph convolution jointly encode long-term temporal dependencies and topological relationships among sensors.The resulting representations are passed as inputs to the next Transformer-based forecasting procedure.

D. More Efficient Multi-branch Transformer

The section replaces vanilla multi-head self-attention with efficient global, local, and pairwise branches while addressing self-attention’s quadratic sequence-length complexity.

  • Self-attention in Transformers: Vanilla multi-head self-attention captures long-distance dependencies but incurs O(n^2) complexity from pairwise token interactions.The mechanism projects token representations into queries, keys, and values across multiple heads before computing scaled dot-product attention.
  • Branch-wise Mixing: The multi-branch architecture splits embeddings into channels for global-fixed attention, vanilla multi-head attention, and neighborhood convolution.These branches combine global context, long-distance interactions, and restricted-neighborhood patterns.
  • Global-learned Attention: Global-learned attention replaces pairwise dot-products with a globally learned alignment matrix and computes weighted sums over projected values.Its attention weights are not conditioned on individual input tokens.
  • Computation Analysis: The proposed mixing strategy combines pairwise interactions with global-learned attention to trade computation complexity against model size and performance.The architecture retains a local pattern extractor using lightConv or dynamicConv while changing the global-context branch.
  • Computation Analysis: When m ≤ 2/hd, the global attention module is more computationally efficient than the other variants.The computation analysis compares memory, parameter count, and Mult-Adds across attention variants.

E. Anomaly Scoring

GTA forecasts multivariate sequences and scores anomalies by comparing predicted and observed values, then labels timestamps using a threshold.

  • Forecasting: The decoder receives encoder and label sequences, pads target elements with zeros, and generates output elements through weighted attention composition.The model predicts outputs in a generative style.
  • Loss and anomaly score: Mean Square Error measures the difference between predicted outputs ˆY and observations Y as the training loss.
  • Loss and anomaly score: The anomaly score compares the expected value at time t with the observed value using their deviation level.
  • Thresholding: A timestamp is labeled anomalous when ˆy(t) exceeds a fixed threshold selected through grid search for the best F1-score and Recall.Different thresholds can produce different prediction performance.

A. Datasets

The evaluation uses four real-world anomaly-detection datasets from water-treatment, water-distribution, and spacecraft-related systems, with preprocessing applied before training.

  • SWaT and WADI: SWaT contains 51 sensors and actuators, 11 days of operation, and 41 attacks launched during the final four collection days.The dataset records network traffic and all sensing values at each timestamp.
  • SWaT and WADI: WADI contains 123 sensors and actuators across 16 days, including 14 regular-operation days and 2 attack-scenario days.
  • Dataset statistics: Tables II and III summarize the four datasets, covering SWaT, WADI, SMAP, and MSL.
  • Preprocessing: SWaT and WADI samples are downsampled to one measurement every 10 seconds using median values for fair comparison.
  • Preprocessing: The authors standardize both training and testing data before training to improve model robustness.The preprocessing uses training-set maximum and minimum values.

2) Evaluation metrics:

The evaluation reports Precision, Recall, and F1, emphasizing Recall and F1 because anomaly detection may prioritize finding attacks despite false alarms; metrics use point adjustment.

  • Evaluation metrics: Precision, Recall, and F1 score are the standard metrics used to evaluate anomaly-detection performance.The metric definitions use true positives, false positives, true negatives, and false negatives.
  • Experimental results: GTA achieves F1 0.9041 and Recall 91.76 for the reported GTA** result.
  • Evaluation metrics: Recall and overall F1 receive greater emphasis than Precision because detecting all real attacks may justify tolerating some false alarms.
  • Evaluation metrics: The evaluation reports best Recall and F1 results across all datasets because different anomaly-score thresholds produce different metric values.
  • Point adjustment: Point adjustment treats any detected observation within a contiguous ground-truth anomaly segment as detection of the whole window.Observations outside the ground-truth segment are treated normally.

3) Baselines:

GTA is evaluated against reconstruction- and forecasting-based anomaly-detection baselines on four datasets, with particularly strong gains on the cyber-attack datasets SWaT and WADI.

  • Baselines: The comparison includes reconstruction-based models such as PCA, AE, KitNet, DAGMM, GAN-Li, OmniAnomaly, LSTM-VAE, and MAD-GAN.
  • Baselines: Forecasting-based baselines include KNN, FB, MTAD-GAT, and GDN.
  • Experimental setup: GTA uses a 60-frame historical window, predicts the next timestamp with a 30-frame label series, and uses three dilated convolution levels.
  • Experimental comparison: GTA achieves best F1-scores of 0.91 on SWaT and 0.84 on WADI, outperforming all compared approaches.Against GDN, the gains are 12.35% on SWaT and 47.47% on WADI.
  • Baseline observations: Deep learning methods generally outperform conventional unsupervised approaches by modeling long sequences and temporal context.
  • Dataset-dependent results: GTA’s improvements are less pronounced on SMAP and MSL, where sensor dependencies are weaker than in SWaT and WADI.

D. Ablation Studies

Ablations indicate that graph learning, the proposed learning policy, and Transformer-based sequence modeling each contribute to GTA’s anomaly-detection performance. A WADI case study further connects the learned graph to attack propagation and sensor localization.

  • D. Ablation Studies: Removing graph learning creates a considerable performance gap, supporting topological-structure modeling for multivariate anomaly detection.
  • D. Ablation Studies: Replacing the learning policy with a static complete graph shows that learned links help filter noise and capture proper information flow.
  • D. Ablation Studies: Replacing the Transformer with a GRU reduces performance, indicating the importance of self-attention for sequence modeling.
  • E. Graph Learning and Case Study: In WADI, abruptly turning on valve 1 MV 001 STATUS caused an overflow in the primary tank during a 25.16-minute attack.
  • E. Graph Learning and Case Study: The learned partial graph depicts sensor interactions, while predicted curves exceeding ground truth increase anomaly scores for affected downstream sensors.
  • VI. CONCLUSION: GTA combines learned sensor dependencies, Influence Propagation convolution, and multi-branch attention in a Transformer-based anomaly-detection framework.
Loading 2104.03466v3…