PaperlayerView exact revision on arXiv

Source-linked AI summary

GT-PSSM: Unified Probabilistic Framework for Stochastic Dynamics Modeling and Dependency Learning in Multivariate Time Series Anomaly Detection

Wonmo Koo, Jaeyeong Lee, Taeseong Yoon, Heeyoung Kim

arXiv:2609.34161v1stat.MLcs.LG

TL;DR

Multivariate time series anomaly detection must handle stochastic observations, while existing PSSM-based methods can struggle with long-range temporal and inter-variable dependencies. GT-PSSM combines probabilistic state-space modeling with Graph Transformer dependency learning. Experiments on real-world benchmark datasets report state-of-the-art MTAD performance.

  • Problem

    Reliable MTAD must address stochastic data, and existing PSSM-based methods may struggle to capture long-range temporal and inter-variable dependencies.

    Source: §1 · pp. 1–2

  • Method

    GT-PSSM integrates probabilistic state-space modeling with Graph Transformer representations to model stochastic dynamics and learn temporal and inter-variable dependencies.

    Source: §1 · p. 2

  • Results

    GT-PSSM achieves state-of-the-art performance on real-world MTAD benchmark datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GT-PSSM provides a PSSM framework that explicitly models measurement noise and latent stochastic dynamics while learning temporal and inter-variable dependencies.

    Source: §1 · p. 2

    The low-rank inductive bias cannot be isolated because removing the LRMF also affects the GConv layer through parameter sharing.

    Source: §5.5 · p. 8

Abstract

from arXiv · show

Multivariate time series anomaly detection (MTAD) is crucial for ensuring the safe and reliable operation of complex systems. Many existing methods learn normal patterns by training reconstruction or forecasting models on predominantly normal data. However, a large portion of these approaches rely on deterministic models and their associated point-wise output errors for anomaly scoring. Since real-world multivariate time series are inherently stochastic due to measurement noise and intrinsic system randomness, purely error-based scores can be unreliable, as large errors may arise from benign fluctuations rather than true anomalies. Probabilistic approaches address this limitation by quantifying uncertainty in model outputs. In particular, probabilistic state-space models (PSSMs) provide a principled framework by modeling stochastic system dynamics through latent state transitions and measurement noise via emission models. Despite this advantage, existing PSSM-based MTAD methods often struggle to capture long-range temporal dependencies and inter-variable dependencies, as they typically rely on noise-sensitive recurrent architectures and lack explicit cross-variable structure modeling. To address these limitations, we propose Graph-Transformer-Enhanced Probabilistic State-Space Model (GT-PSSM), a novel PSSM-based MTAD method that tightly integrates PSSM-based probabilistic modeling of stochastic dynamics with Graph Transformer-based learning of temporal and inter-variable dependencies. By jointly modeling stochasticity, long-range temporal dependence, and variable interactions within a unified probabilistic framework, GT-PSSM enables more robust anomaly detection.

1 Introduction

Multivariate time series anomaly detection must contend with complex dependencies and intrinsic stochasticity, which can make deterministic point-error scores unreliable. GT-PSSM combines probabilistic state-space modeling with Graph Transformer dependency learning to address gaps in existing PSSM-based methods.

1 Introduction

  • Deterministic point-wise reconstruction or forecasting errors can mistake benign random fluctuations for anomalies in inherently stochastic series.Probabilistic scoring accounts for uncertainty in model outputs and expected observation variability.Source: §1 · p. 2
  • Existing PSSM-based methods may miss long-range temporal and inter-variable dependencies because they rely on noise-sensitive RNNs and often lack explicit cross-variable modeling.Latent stochastic states alone may not preserve the historical information needed for long-range dependencies.Source: §1 · p. 2
  • GT-PSSM conditions state-transition and emission distributions on Graph Transformer representations combining temporal self-attention with graph convolution over a learnable weighted graph.The model uses amortized inference for stochastic latent states; 𝛽-NLL stabilizes emission-mean learning, and emission-mean dependencies are aligned with learned variable interactions.Source: §1 · p. 2
  • GT-PSSM achieves state-of-the-art MTAD performance in extensive experiments on real-world benchmark datasets.It jointly captures stochasticity, temporal dependencies, and inter-variable dependencies.Source: §1 · p. 2

2 Related Work

Prior MTAD work includes deterministic, probabilistic, and hybrid reconstruction or forecasting approaches. GT-PSSM's related-work context emphasizes the challenge of combining stochastic-dynamics modeling with effective temporal and inter-variable dependency learning.

2 Related Work

  • Deterministic methods use point-wise errors, while probabilistic methods use likelihood-based scores; deterministic work increasingly uses Transformers and graph networks but does not account for stochasticity.Hybrid methods are categorized as probabilistic when at least one component uses likelihood-based scoring.Source: §2 · pp. 2–3
  • Predictive-distribution models can account for stochasticity, but approaches without latent system dynamics may not explicitly model the underlying stochastic process.PSSMs separately model measurement noise through emissions and latent stochastic dynamics through transitions.Source: §2 · p. 3
  • Most PSSM-based MTAD methods condition distributions on RNN hidden states, leaving long-range and inter-variable dependencies challenging; Chen et al. add graph convolution to aggregate cross-variable information.Tian et al. use a Graph Transformer encoder for probabilistic reconstruction but do not explicitly model dependencies among latent variables.Source: §2 · p. 3

3 Preliminaries

The preliminaries define an unsupervised anomaly-detection setting and summarize how PSSMs represent observations through stochastic latent states. They also establish the predictive-distribution scoring and variational reconstruction framework used in the paper.

3.1 Problem Statement

  • In the unsupervised setting, training data contain only normal samples, while test data may contain normal and anomalous samples; GT-PSSM scores sliding-window reconstructions and flags scores above a predefined threshold.The model outputs a predictive distribution, and the anomaly score is an upper-bound surrogate of the observation's negative log-likelihood.Source: §3.1 · p. 3

3.2 Probabilistic State-Space Models

  • A PSSM links each observation to a per-time-step latent state through an emission model and models state evolution with a transition model.Deep PSSMs approximate the latent posterior with an inference model and optimize an evidence lower bound; treating inference as encoder and emission as decoder enables probabilistic reconstruction scored by NLL or its upper bound.Source: §3.2 · p. 3

4 Methodology

GT-PSSM combines probabilistic state-space modeling with a Graph Transformer to model stochastic dynamics and temporal and inter-variable dependencies. Its methodology specifies the model components, training objective, and probabilistic anomaly score.

4.1 Stochastic Dynamics Modeling via PSSM

  • Both PSSM transition and emission distributions condition on Graph Transformer representations of past observations to model stochastic dynamics and dependencies.The PSSM represents observations through latent states and uses transition and emission models.Source: §4.1 · p. 4
  • Learnable variable-wise embeddings factor the emission weight matrices at low rank, coupling variable-specific emission mappings through shared factors.The factorization uses an embedding dimension much smaller than the number of variables.Source: §4.1 · p. 4
  • Diagonal emission and transition covariances allow time- and variable-dependent variances, while latent states can still capture cross-variable covariance.The diagonal parameterization is adopted for computational efficiency.Source: §4.1 · p. 4

4.2 Dependency Modeling via Graph Transformer

  • The Graph Transformer combines temporal multi-head self-attention, graph convolution, and a position-wise feed-forward layer to produce hidden representations of past observations.Each hidden representation summarizes observations up to its corresponding time step.Source: §4.2 · p. 4, §4.2.3 · p. 5

4.2.1 MHSA Layer.

  • Temporal self-attention lets each time step attend to itself and preceding steps; positional embeddings encode signed offsets, while values avoid mixing information across variables.A causal mask excludes future time steps from attention.Source: §4.2.1 · p. 4

4.2.2 GConv Layer.

  • Graph convolution learns a fully connected weighted graph over variables from variable-wise embeddings, with shared embeddings encouraging consistent dependency structures..Source: §4.2.2 · pp. 4–5

4.3 Model Training

  • Because standard negative-ELBO training may underweight mean estimation at large predicted variances, GT-PSSM uses β-NLL and reweights KL-divergence terms.The final loss approximates expectations with one Monte Carlo sample from the inference distribution.Source: §4.3 · p. 5

4.4 Anomaly Scoring

  • Anomaly scoring uses an upper bound on reconstruction NLL, normalizes variable contributions by validation-set medians and IQRs, and estimates uncertainty with many Monte Carlo samples.The median and IQR are computed for each variable over all validation time steps.Source: §4.4 · pp. 5–6

5 Experiments

Experiments compare GT-PSSM with 12 baselines on four real-world datasets and evaluate its benchmark performance, component contributions, and training behavior. Results support the value of uncertainty-aware scoring and explicit temporal and inter-variable dependency modeling, while showing sensitivity to β.

5.1 Dataset Description; 5.2 Baselines; 5.3 Evaluation Metrics

  • Evaluation used WADI, PSM, SMD, and SWaT, compared GT-PSSM with 12 deterministic and probabilistic baselines, and reported F1 as the primary metric alongside F1PA.The reported scores use grid-searched thresholds; F1PA counts all time steps in a segment as detected when at least one is identified.Source: §5.1 · p. 6, §5.2 · p. 6, §5.3 · p. 6

5.5 Ablation Study

  • Removing uncertainty-aware scoring reduced F1 on WADI, PSM, and SMD, while removing the state-transition model reduced F1 on those same datasets; both variants were comparable on SWaT.The uncertainty-free variant uses squared error between observations and mean reconstructions, with variable-wise Med-IQR normalization.Source: §5.5 · p. 6, Table 2 · p. 7

5.4 Comparison with Baselines

  • GT-PSSM achieved the best F1 on WADI, PSM, and SMD, with gains of +5.94%, +2.96%, and +5.68% over each dataset’s best baseline; it was second-best on SWaT (0.8466 vs. 0.8530).Results are averaged over 10 repeated experiments; F1PA was best on WADI, competitive on SMD and SWaT, and relatively lower on PSM.Source: §5.4 · p. 6, Table 1 · p. 7

5.5 Ablation Study

  • Replacing temporal self-attention, removing graph convolution, or disabling low-rank weight factorization each significantly degraded performance, supporting temporal and inter-variable dependency modeling.The larger drop without low-rank factorization does not isolate its inductive-bias effect, because removing it also affects GConv through parameter sharing.Source: §5.5 · pp. 7–8
  • Removing β-NLL degraded performance on WADI, PSM, and SWaT but was comparable on SMD; the authors report that each component contributes and their integration yields the best performance.The authors suggest β-NLL stabilizes emission-mean learning and thereby contributes to Graph Transformer-based dependency learning.Source: §5.5 · p. 8

5.6.1 Effectiveness of Uncertainty-Aware Anomaly Scoring.

  • On WADI, uncertainty-aware scoring detected true anomalies missed by the error-based variant and correctly classified nearly all highlighted normal points during abrupt water-flow fluctuations.GT-PSSM made one false alarm among the highlighted normal points; the authors attribute the fluctuation-related benefit to explicit stochasticity modeling.Source: Figure 2 · p. 7, §5.6.1 · p. 8

5.6.2 Importance of Inter-Variable Dependency Modeling.

  • On PSM, GT-PSSM’s learned variable relationships helped expose a long anomaly in which Variables 1 and 3 diverged, producing reconstruction deviations and elevated anomaly scores.Figure 3’s adjacency heat map encodes target variables by row and neighboring variables by column; lighter colors indicate stronger dependencies.Source: §5.6.2 · p. 8, Figure 3 · p. 8

5.7 Training Dynamics Analysis; 5.8 Hyperparameter Sensitivity Analysis

  • Across four datasets, ELBO, F1, and F1PA generally rose during training except for PSM F1PA, while WADI performance was stable across w, d_h, and d_z but increased with β.The authors fixed β at 1.0 across datasets to avoid dataset-specific tuning.Source: §5.7 · p. 9, §5.8 · p. 9, Figure 4 · p. 9

6 Conclusion

GT-PSSM integrates a Graph Transformer with a probabilistic state-space model to capture stochastic dynamics and temporal and inter-variable dependencies. Experiments on four real-world benchmarks show state-of-the-art anomaly detection performance against deterministic and probabilistic baselines.

6 Conclusion

  • Across four real-world benchmarks, GT-PSSM achieves state-of-the-art anomaly detection performance and consistently outperforms strong deterministic and probabilistic baselines.The conclusion also proposes extending its learned associative graph toward causal structures for interpretable root-cause analysis.Source: §6 · p. 9
  • GT-PSSM parameterizes transition and emission distributions through Graph Transformer representations and aligns variable-wise emission means with learned inter-variable structure.The paper also uses β-NLL to stabilize emission-mean learning and, consequently, Graph Transformer training.Source: §6 · p. 9
  • The framework combines probabilistic state-space modeling with Graph Transformer learning to capture stochastic dynamics, long-range temporal dependencies, and inter-variable structures.The conclusion presents this integration as a broader direction for multivariate time-series learning.Source: §6 · p. 9

A Dataset Statistics

The appendix reports dataset-specific preprocessing choices for WADI, SMD, and SWaT, alongside a table of dataset statistics.

A Dataset Statistics

  • WADI and SWaT each discard the first 21,600 training samples and downsample every 10 time steps; WADI also uses 112 selected variables.The WADI settings follow [12], as do the SWaT discard and downsampling choices.Source: A Dataset Statistics · pp. 9–10
  • For SMD, experiments use machine-1-1, selected because it appears in subsets used by [53] [62] amid prior exclusions for concept drift or trivial anomalies.The original SMD comprises 28 independent server machines.Source: A Dataset Statistics · p. 10
  • For SWaT, AIT201 and P201 were excluded because their training-data distributions differed noticeably from test data under normal operating conditions, following [31]; all datasets used Min-Max normalization and a 0.2 validation ratio.Source: A Dataset Statistics · p. 10
  • Table 3 summarizes each dataset’s variable count, training and test time steps, and test-data anomaly ratio.Source: Table 3 · p. 10

B Implementation Details

The implementation uses Adam with a 10^-3 learning rate, early stopping, and dataset-specific batch sizes and MLP widths.

B Implementation Details

D Evaluation on the Full SMD

On the full 28-machine SMD dataset, GT-PSSM achieved the highest average F1 among the selected baselines.

D Evaluation on the Full SMD

  • GT-PSSM achieved the highest average F1 across all 28 SMD machines in the full-dataset evaluation.Table 5 reports mean and standard deviation of machine-averaged F1 across 10 repeated experiments; the supplied text gives no score values.Source: D Evaluation on the Full SMD · p. 10, Table 5 · p. 10
  • The evaluation compares GT-PSSM with GDN, NPSR, and SensitiveHUE, selected as strong representatives based on results across SMD, PSM, SWaT, and WADI.Source: D Evaluation on the Full SMD · p. 10
  • The full-SMD evaluation addresses robustness beyond the main experiments, which used only machine-1-1.Source: D Evaluation on the Full SMD · p. 10

E Computational Cost Analysis

GT-PSSM incurs relatively higher computational costs, yet its forward pass takes less than 0.1 seconds per batch on both WADI and SMD, which the authors consider feasible for online inference.

  • Under 0.1 seconds per batch on both WADI and SMD, GT-PSSM remains feasible for online inference despite relatively higher computational costs.The reported inference time is per batch.Source: E Computational Cost Analysis · p. 10
  • Table 6 compares training time per epoch, forward-pass time, and memory usage for GT-PSSM and selected baselines on WADI and SMD.Source: E Computational Cost Analysis · p. 10, Table 6 · p. 10
  • In Table 6, bold marks the highest computational costs and underlining marks the second-highest.Source: Table 6 · p. 10

Related papers

Loading 2609.34161v1…