PaperlayerView exact revision on arXiv

Source-linked AI summary

Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encoder

Longyuan Li, Junchi Yan, Haiyang Wang, Yaohui Jin

arXiv:2102.01331v1cs.LGcs.AI

TL;DR

Multidimensional time-series anomaly detection requires robust modeling of temporal dependencies and changing noise in contaminated observations. SISVAE addresses this with a sequential VAE, smoothness-inducing prior, and SGVB training, and experiments report strong performance across synthetic and real-world benchmarks.

  • Problem

    Robust unsupervised point-level detection must handle correlated time series, anomalies, and non-stationary noise without relying on constant-noise assumptions.

  • Method

    SISVAE combines a recurrent sequential VAE with a variational smoothness regularizer, per-timestamp mean and variance parameterization, and SGVB training.

  • Results

    Experiments on synthetic and real-world datasets show SISVAE outperforms state-of-the-art unsupervised anomaly-detection models and supports reconstruction probability and error criteria.

  • Takeaways & Limitations

    Smoothness regularization provides a robust estimation mechanism for detecting anomalies in multidimensional correlated time series.

Abstract

from arXiv · show

Deep generative models have demonstrated their effectiveness in learning latent representation and modeling complex dependencies of time series. In this paper, we present a Smoothness-Inducing Sequential Variational Auto-Encoder (SISVAE) model for robust estimation and anomaly detection of multi-dimensional time series. Our model is based on Variational Auto-Encoder (VAE), and its backbone is fulfilled by a Recurrent Neural Network to capture latent temporal structures of time series for both generative model and inference model. Specifically, our model parameterizes mean and variance for each time-stamp with flexible neural networks, resulting in a non-stationary model that can work without the assumption of constant noise as commonly made by existing Markov models. However, such a flexibility may cause the model fragile to anomalies. To achieve robust density estimation which can also benefit detection tasks, we propose a smoothness-inducing prior over possible estimations. The proposed prior works as a regularizer that places penalty at non-smooth reconstructions. Our model is learned efficiently with a novel stochastic gradient variational Bayes estimator. In particular, we study two decision criteria for anomaly detection: reconstruction probability and reconstruction error. We show the effectiveness of our model on both synthetic datasets and public real-world benchmarks.

I. INTRODUCTION

The paper addresses unsupervised point-level anomaly detection in correlated multivariate time series, where temporal continuity, changing noise, and contamination complicate robust estimation. SISVAE combines sequential variational modeling with a smoothness prior and evaluates reconstruction probability and error on synthetic and real-world data.

  • Motivation: Multidimensional time-series anomaly detection is challenging because temporal continuity and cross-stream correlations must be modeled while detecting unusual changes.The paper targets correlated streams and emphasizes the need for robust, flexible models.
  • Problem scope: Unsupervised point-level detection avoids anomaly labels and can adapt more flexibly when anomaly patterns drift, whereas sequence-level detection is left for future work.The paper focuses on individual measurements in multidimensional time series.
  • Related work: Existing deterministic methods regularize temporal smoothness but often pay less attention to stochastic behavior, while probabilistic models may be limited by simple distributions and constant-noise assumptions.Sequential VAE variants capture temporal structure but generally do not account for anomalies in their objectives.
  • Proposed approach: SISVAE introduces a Bayesian variational smoothness regularizer that penalizes non-smooth mean and variance transitions and can be optimized with standard backpropagation.The design combines smoothness regularization with deep generative uncertainty modeling.
  • Proposed approach: Per-timestamp neural parameterization of mean and variance enables dynamic anomaly-detection thresholds without assuming constant noise.This is intended to handle time-varying data variance.
  • Evaluation: Experiments on synthetic and real-world benchmarks report that SISVAE consistently outperforms state-of-the-art competing models using reconstruction probability and reconstruction error.Both decision criteria are studied for anomaly detection.

III. PRELIMINARIES

The paper formulates contaminated multivariate time series as observations combining a true signal, non-stationary noise, and anomalous components. Its objective is to recover the true signal and identify anomalies, motivating smoothness-based estimation.

  • Problem formulation: Observed time series are modeled as a true signal plus non-stationary observation noise and an anomaly vector selected by a binary indicator matrix.The indicator marks whether each stream-time observation contains an anomaly.
  • Problem formulation: The central goal is to recover the true signal f(t) and detect the anomaly indicator I from contaminated observations.This formulation explicitly separates signal recovery from anomaly identification.
  • Problem formulation: Figure 1 depicts the pipeline from a four-dimensional input series through time-varying density estimation to threshold-based anomaly detection.Shaded regions represent variance levels, and red circles mark detected anomalies.
  • Smoothness prior modeling: Smoothness-prior modeling balances goodness of fit against a smoothness criterion using a trade-off parameter λ.The smoothness constraint can use a k-th order differential operator, with first differences defined as ∇f(t) = f_t − f_{t−1}.
  • Smoothness prior modeling: The Bayesian interpretation treats the smooth function as having a prior and combines it with the data distribution to obtain a posterior estimate.The formulation is introduced for a homoscedastic Gaussian time series before extension to the proposed model.

C. Deep Generative Models and SGVB Estimator

The paper uses VAE-based latent-variable modeling with neural generative and inference components, extending it sequentially with an RNN backbone for time-series dependencies. SGVB makes the variational objective differentiable and trainable by backpropagation.

  • Deep generative models: Latent-variable modeling represents the marginal likelihood through a prior pθ(z) and conditional data distribution pθ(x|z).The latent variable captures variation in observed data.
  • Deep generative models: VAE approximates the intractable posterior with qφ(z|x) and optimizes an evidence lower bound combining KL divergence with expected reconstruction log likelihood.The generative and inference models are trained jointly.
  • SGVB estimator: SGVB rewrites the stochastic latent variable as z = gφ(ϵ, x), enabling estimation of the variational lower bound and gradient-based training.The auxiliary variable ϵ has an independent marginal distribution p(ϵ).
  • SISVAE: SISVAE adds a smooth prior to sequential VAE density estimation and supports anomaly detection through either absolute reconstruction loss or reconstruction probability.The model is designed for multidimensional time-series point anomalies.
  • Sequential VAE: Sequential VAE models use an RNN hidden-state backbone conditioned on previous observations and latent variables to capture temporal features.This extends VAE-style latent modeling to sequential time-series data.

1) Generative model:

SISVAE factorizes its generative model into conditional observation likelihoods and temporally structured latent-state priors. Neural feature extractors and a GRU recurrence parameterize the evolving distributions.

  • The generative model conditions each observation on previous observations and latent states through pθ(xt|z≤t, x<t).
  • The latent-state prior depends on the previous hidden state, introducing temporal structure into the latent representation.
  • For multivariate Gaussian outputs, µx,t and σx,t parameterize the generated data distribution's mean and variance.
  • A gated recurrent unit updates hidden states within the generative model's recurrence.
  • Feature extractors for observed data and latent variables are implemented with neural networks.

2) Inference model:

The inference model uses variational inference and recurrent structure, while SISVAE adds a probabilistically coherent smoothness prior to reduce anomaly-induced reconstruction bias. Training combines inference, reconstruction, and smoothness terms using SGVB and mini-batch optimization.

  • Variational inference constructs an approximate posterior qφ over latent variables conditioned on the observed data and trains it with the AEVB algorithm.
  • The inference posterior for zt depends on the current observation xt and the previous hidden state ht−1, with neural networks parameterizing its statistics.
  • Anomaly Detection with Sequential VAE: Sequential VAE reconstruction can become vulnerable to anomalies because fitting contaminated data biases the reconstructed mean or inflates its variance.
  • Anomaly Detection with Sequential VAE: SISVAE extends smoothness regularization to time-varying probability distributions, requiring reconstructed distributions to transition smoothly over time.
  • Anomaly Detection with Sequential VAE: The proposed transition cost measures distributional smoothness between consecutive reconstructions using a distance metric, chosen here as KL-divergence for Gaussian distributions.
  • The final objective combines inference loss, expected reconstruction loss, and a variational smoothness regularizer, optimized with SGVB.

1) Preparing dataset:

SISVAE processes long multivariate time series in short sliding-window chunks, then uses probabilistic reconstruction and reconstruction-based anomaly scores. Reconstruction probability is introduced to handle non-stationary variance, where absolute error can misclassify points.

  • Preparing dataset: Long sequences are divided into short chunks with sliding windows because recurrent training over entire sequences can suffer from vanishing or exploding gradients.The windows use size W and step size s, and the resulting chunks form the training dataset.
  • Computing anomaly scores: A trained model encodes each input chunk into sequential latent variables and decodes them as Gaussian distributions for probabilistic reconstruction.The decoded sequence is parameterized by reconstructed means and variances at each time step.
  • Computing anomaly scores: Absolute reconstruction error uses distance from the reconstructed mean, but assumes stationary variance and can fail when observation noise changes over time.Under increasing noise, equally distant points may differ in whether they are anomalous.
  • Computing anomaly scores: SISVAE therefore computes anomaly scores from reconstruction probability, averaging probabilities from latent samples and using log probability for numerical stability.Sequential latent dependencies are handled with Sequential Monte Carlo.
  • Computing anomaly scores: For a full multivariate series, the method slices the data into non-overlapping chunks and reconstructs each chunk separately.The resulting anomaly scores are computed over the processed chunks.

V. EXPERIMENTS

The experiments evaluate unsupervised point-level anomaly detection for correlated multivariate time series using threshold-independent metrics and comparisons with deep, sequential, and traditional models. SISVAE variants isolate the effects of reconstruction criterion, smoothness regularization, and latent temporal structure.

  • Protocols and settings: The experiments address unsupervised point-level anomaly detection for multi-dimensional correlated time series.The evaluation compares SISVAE with peer VAE methods and traditional statistical models.
  • Performance metrics: Anomaly scores are thresholded to produce labels, while AUROC, AUPRC, and best F1-score summarize performance across possible thresholds.AUROC and AUPRC are computed from threshold-varying rates, and the reported metrics are independent of a specific threshold.
  • Performance metrics: Because anomaly classes may be heavily imbalanced, the study reports both AUROC and AUPRC, with AUPRC described as more discriminative for false positives.The metrics provide complementary views of detection performance.
  • Peer methods: The comparison includes Donut, STORN, History Average, ARMA, and LDS, alongside SISVAE and its variants.STORN uses independent sequential latent variables, while ARMA and LDS represent traditional time-series models.
  • Model variants: SISVAE-p uses reconstruction probability with sliding windows, SISVAE-e uses absolute reconstruction error, and SISVAE-0 removes smoothness regularization.VAE-s adds the smoothness regularization to vanilla VAE to test latent temporal structure.
  • Implementation details: The implementation uses window size W = 120, hidden dimension 200, latent dimension 40, and smoothness regularization λ = 0.5.These settings are used across the reported datasets for the corresponding models.

B. Experiments on Synthetic Data

Synthetic experiments test SISVAE under varying anomaly proportions and smoothness strengths, then examine training convergence. The proposed variational smoothness regularizer improves stability and supports higher AUPRC than unregularized or mean-regularized alternatives.

  • Synthetic experiments: The synthetic study varies anomaly proportions and the smoothness hyper-parameter λ to examine SISVAE detection behavior.It uses correlated 100-dimensional Gaussian-process sequences with non-stationary Gaussian noise.
  • Effects of anomaly proportion: As anomaly proportions increase, unregularized SISVAE-0 and STORN show declining AUPRC, while the proposed regularizer is evaluated for robustness.The experiment attributes the influence of anomalies to unsupervised training on both normal and anomalous data.
  • Effects of regularization hyper-parameter: Best SISVAE performance occurs at λ = 0.5 on the synthetic dataset with 2% anomalies.The tested values were λ = {0.01, 0.05, 0.1, 0.2, 0.5, 0.8, 1.0}.
  • Convergence study: The convergence comparison trains unregularized, mean-regularized, and proposed KL-regularized SISVAE models on synthetic data with 2% anomalies.AUPRC is recorded throughout training.
  • Convergence study: After 100 iterations, unregularized and mean-regularized models fluctuate between 0.45 and 0.85 AUPRC, whereas KL-regularized SISVAE fluctuates within a smaller interval.The reported fluctuation is associated with conflict between decoder reconstruction and encoder inference objectives.
  • Convergence study: The KL-regularized model converges stably and reaches a higher AUPRC score, while unregularized training remains unstable.The regularizer weakens the reconstruction objective and guides generated sequences toward temporal smoothness.

C. Experiments on Real-world Data

On real-world multivariate datasets, SISVAE-p generally outperforms competing unsupervised models, while model choice depends on the required precision–recall trade-off and ranking budget.

  • Threshold based anomaly detection: SISVAE-p outperforms other models in most datasets and metrics under threshold-based evaluation.The evaluation computes AUROC, AUPRC, and F1-score across possible detection thresholds.
  • Overall model comparison: SISVAE-p outperforms other methods notably when models are ranked by mean and standard deviation of dataset-specific performance ranks.The ranking aggregates performance across datasets and metrics.
  • Ranking based anomaly detection: SISVAE-p maintains stable precision@K as K increases, making its detections more precise when only a limited number of anomalies can be investigated.At K = 10, STORN, LDS, and SISVAE-p perform well, while Donut fails on A2, A3, and A4.
  • Model characteristics: SISVAE-p outperforms LDS and STORN at most detection thresholds, while Donut assigns very large anomaly scores to some normal points in Yahoo A2, A3, and A4.These abnormal scores cause Donut’s precision–recall curves to increase before decreasing.
  • The trade-off between recall and precision: When recall is fixed at 0.95, Donut outperforms SISVAE-p in precision on Yahoo A2 and A4, whereas SISVAE-p has higher recall under a precision requirement above 0.95.The preferred operating point depends on whether precision or recall is more important.

D. Case study

Case studies show SISVAE-p is robust to contaminated training data and detects both point-level and subsequence anomalies while producing smoother density and anomaly-score estimates.

  • False alarms on normal time series: On a normal Yahoo-A1 sequence, SISVAE-p and Donut avoid the three false alarms reported by STORN.SISVAE-p’s anomaly score remains stably low, whereas Donut’s score fluctuates more under noisy data.
  • Detection of point-level anomalies: For a point anomaly, all three models identify the anomalous point, but STORN and Donut also report false alarms while SISVAE-p recovers a smoother density.The point anomaly corrupts the density estimates of STORN and Donut but not SISVAE-p’s estimate.
  • Detection of sub-sequence level anomalies: For window-sized anomalies, SISVAE-p detects most anomalous subsequences, while STORN and Donut suffer from false negatives and false positives.SISVAE-p estimates a smooth probability density over time, unlike the notably anomaly-influenced densities of the baselines.
  • Detection of sub-sequence level anomalies: The case studies attribute SISVAE-p’s robust detection behavior to its smoothness prior regularizer, which helps recover the underlying density.The reported examples cover normal sequences, point-level anomalies, and subsequence-level anomalies.

E. Further Comparison with Donut

A Mackey–Glass simulation compares SISVAE-p with Donut on nonlinear univariate data containing point and subsequence anomalies, finding stable SISVAE-p performance across latent dimensions and concluding that SISVAE-p performs effectively.

  • Simulation study on uni-variate time series: The Mackey–Glass simulation uses a nonlinear univariate benchmark generated for 5000 iterations and collected as a length-5000 time series.The study compares SISVAE-p with Donut in the one-dimensional setting.
  • Anomaly construction: The simulation inserts point anomalies at 0.3% of time steps using combined Poisson noise, Gaussian noise, and a fixed bias.Each point anomaly combines Poisson noise with λ = 1, Gaussian noise with zero mean and unit variance, and 0.2 fixed bias.
  • Anomaly construction: The simulation inserts two subsequence anomalies with random lengths from 10–28, replacing the true signal with Gaussian-process samples using an RBF kernel.The Gaussian process has unit variance and length-scale 0.3.
  • Latent-dimension comparison: Donut performs best at latent dimension z dim = 5, while SISVAE-p performance remains relatively stable as z dim changes.The reported ROC and PRC compare the two methods on the generated data.
  • Conclusion: The paper concludes that SISVAE-p benefits from its sequential network structure and smoothness prior regularizer in this comparison.The conclusion also reports that SISVAE outperforms state-of-the-art unsupervised models on multi-dimensional time series across synthetic and real-world datasets.
Loading 2102.01331v1…