PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networks

Tolga Ergen, Ali Hassan Mirza, Suleyman Serdar Kozat

arXiv:1710.09207v1eess.SPcs.LGstat.ML

TL;DR

Anomaly detection for variable-length sequences requires a suitable decision function without relying on fixed-length inputs. The paper maps sequences to fixed-length LSTM representations, jointly trains them with OC-SVM or SVDD detectors, and reports significant gains over conventional methods. It also extends the framework to supervised settings and GRUs.

  • Problem

    The paper addresses unsupervised anomaly detection when data sequences have variable lengths and conventional OC-SVM or SVDD methods process only fixed-length sequences.

  • Method

    The approach uses LSTM processing and pooling to obtain fixed-length representations, then jointly optimizes the LSTM and OC-SVM or SVDD parameters with gradient-based or quadratic-programming training.

  • Results

    The experiments report significant performance gains over conventional methods, with higher AUC scores for LSTM and GRU methods on the http dataset.

  • Takeaways & Limitations

    The resulting algorithms process variable-length sequences and support unsupervised, semi-supervised, and fully supervised anomaly detection frameworks.

Abstract

from arXiv · show

We investigate anomaly detection in an unsupervised framework and introduce Long Short Term Memory (LSTM) neural network based algorithms. In particular, given variable length data sequences, we first pass these sequences through our LSTM based structure and obtain fixed length sequences. We then find a decision function for our anomaly detectors based on the One Class Support Vector Machines (OC-SVM) and Support Vector Data Description (SVDD) algorithms. As the first time in the literature, we jointly train and optimize the parameters of the LSTM architecture and the OC-SVM (or SVDD) algorithm using highly effective gradient and quadratic programming based training methods. To apply the gradient based training method, we modify the original objective criteria of the OC-SVM and SVDD algorithms, where we prove the convergence of the modified objective criteria to the original criteria. We also provide extensions of our unsupervised formulation to the semi-supervised and fully supervised frameworks. Thus, we obtain anomaly detection algorithms that can process variable length data sequences while providing high performance, especially for time series data. Our approach is generic so that we also apply this approach to the Gated Recurrent Unit (GRU) architecture by directly replacing our LSTM based structure with the GRU based structure. In our experiments, we illustrate significant performance gains achieved by our algorithms with respect to the conventional methods.

I. INTRODUCTION

The paper introduces LSTM-based anomaly detectors for unlabeled variable-length sequences, combining fixed-length recurrent representations with OC-SVM or SVDD scoring. It jointly optimizes the recurrent and scoring parameters, extends the framework beyond unsupervised learning, and reports gains over conventional methods.

  • Method: LSTM processing converts each variable-length sequence into a fixed-length representation for downstream OC-SVM or SVDD anomaly scoring.The method applies the LSTM to sequence columns and uses mean pooling, while also pooling the state information.
  • Extensions: The algorithms extend from unsupervised learning to semi-supervised and fully supervised frameworks.The contribution list explicitly includes these extensions of the LSTM-based anomaly detection derivations.
  • Method: The authors jointly train the LSTM architecture and OC-SVM or SVDD formulation using quadratic-programming and gradient-based optimization methods.The gradient method modifies the original formulations and proves convergence of the modified formulations to the originals.
  • Evaluation: The generic approach is also applied to GRU architectures and is reported to improve performance over conventional OC-SVM and SVDD methods.The experiments involve real and simulated data, while the GRU application tests replacing the LSTM structure with another recurrent architecture.
  • Motivation: The framework targets unsupervised anomaly detection for variable-length sequences, where most observed sequences are assumed normal.The scoring function assigns nominal and anomalous outputs to sequences whose lengths may vary.

III. NOVEL ANOMALY DETECTION ALGORITHMS

This section formulates OC-SVM- and SVDD-based anomaly detection and then provides joint optimization updates for the overall structure.

  • The section first formulates anomaly detection approaches based on OC-SVM and SVDD algorithms.
  • It then provides joint optimization updates to train the parameters of the overall structure.
  • The algorithms therefore combine anomaly-detector formulations with training of the complete model structure.

A. Anomaly Detection with the OC-SVM Algorithm

The paper formulates anomaly detection with OC-SVM by mapping variable-length sequences into LSTM-derived fixed-length representations and jointly optimizing the detector and LSTM parameters. Orthogonality constraints regularize the LSTM parameters to avoid trivial solutions and preserve temporal modeling.

  • OC-SVM separates anomalies from normal data using a hyperplane applied to LSTM-derived fixed-length sequence representations.
  • The optimization learns the separating hyperplane parameters and LSTM parameters jointly because the fixed-length representations depend on the LSTM parameters.
  • Orthogonality constraints on LSTM parameters limit parameter norms, helping avoid overfitting and trivial solutions such as θ = 0.
  • The paper also permits alternative constraints, including Frobenius-norm constraints, while retaining the same optimization procedure.

1) Quadratic Programming Based Training Algorithm:

The quadratic-programming method alternates between optimizing OC-SVM dual variables and LSTM parameters. It uses SMO for the SVM update and orthogonality-constrained optimization for the recurrent parameters.

  • The method converts the constrained OC-SVM problem into a dual quadratic program with Lagrange multipliers and LSTM constraints.
  • SMO updates selected dual variables while fixing the others, projects updated values into their feasible interval, and repeats until the dual parameters converge.
  • After each dual update, the algorithm fixes the dual variables and updates LSTM parameters using optimization with orthogonality constraints.
  • The resulting procedure is a quadratic-programming-based training algorithm for the LSTM anomaly detector.

2) Gradient Based Training Algorithm:

The gradient-based method updates OC-SVM and LSTM parameters simultaneously by replacing the nondifferentiable objective with a smooth approximation. The paper proves that this approximation and its optimum converge to those of the original formulation.

  • The gradient method updates OC-SVM and LSTM parameters simultaneously, addressing the convergence limitation of separated consecutive updates.
  • Because the original OC-SVM objective is nondifferentiable, the method uses a differentiable smoothing function Sτ with τ > 0.
  • The method updates w and ρ with first-order gradients and optimizes LSTM parameters under orthogonality constraints.
  • As τ increases, Sτ uniformly converges to G, and the resulting approximation Fτ uniformly converges to the original objective F.
  • For fixed θ, the optimized parameters wτ and ρτ are unique, and Fτ(wτ, ρτ, θ) converges to the minimum of F(w, ρ, θ).

B. Anomaly Detection with the SVDD algorithm

The SVDD formulation learns an LSTM representation together with a hypersphere-based anomaly detector. Its training procedure provides joint updates for the LSTM and SVDD parameters and uses a scoring function after optimization.

  • The SVDD-based method jointly learns the LSTM parameters and hypersphere parameters through joint updates.
  • SVDD seeks a hypersphere enclosing normal data while leaving anomalous data outside.
  • The trade-off parameter λ balances the hypersphere radius R^2 against total misclassification error, with θ denoting LSTM parameters and ξ slack variables.
  • After solving the constrained optimization problem, anomalies are detected using the resulting scoring function.

1) Quadratic Programming Based Training Algorithm:

The SVDD training procedure alternates optimization of fixed LSTM parameters and SVDD parameters, using quadratic programming and iterative updates until convergence.

  • The quadratic programming method first fixes LSTM parameters, then optimizes SVDD parameters based on those fixed representations.
  • The method derives a dual SVDD formulation using Lagrange multipliers and solves its constrained optimization through SMO updates.
  • The SVDD formulation imposes orthogonality constraints on W and R and a unit-norm constraint on b.
  • The algorithm alternates parameter updates until convergence, then evaluates the anomaly detector using the converged parameters.
  • Algorithm 3 initializes LSTM and dual SVDD parameters, sets a convergence threshold, iterates updates, and detects anomalies afterward.

2) Gradient Based Training Algorithm:

The gradient-based SVDD method replaces the nondifferentiable objective component with a smooth approximation, updates model parameters by gradients, and proves convergence properties for fixed LSTM parameters.

  • Because G(·) is nondifferentiable, the gradient method uses Sτ(·) and optimizes the resulting modified objective.
  • The method updates ˜c, R, and θ iteratively, using SGD for ˜c and R while maintaining the stated orthogonality constraints.
  • The framework extends to semi-supervised and supervised settings by incorporating labeled instances and taking l = 0 for the supervised case.
  • Theorem 2 states that, for fixed θ, the solutions ˜cτ and Rτ are unique and the modified objective converges to the minimum of the original objective.
  • Strict convexity establishes global uniqueness for ˜cτ and R2τ under fixed accompanying parameters.

IV. SIMULATIONS

The simulations evaluate the proposed LSTM and GRU anomaly detectors on variable-length digit sequences and several real benchmark datasets using gradient- and quadratic-programming training variants.

  • The experiments begin with a variable-length digit dataset and continue with occupancy, Hong Kong Exchange rate, http, and Alcoa stock price benchmarks.
  • Algorithm 4 initializes LSTM and SVDD parameters, iteratively updates them using gradient-based rules, and detects anomalies after convergence.
  • The real-data experiments compare GRU-based algorithms with the corresponding LSTM-based algorithms.
  • The study names the gradient and quadratic-programming variants LSTM-GSVM, LSTM-QPSVM, LSTM-GSVDD, and LSTM-QPSVDD, with GRU labels replacing the LSTM prefix.

A. Anomaly Detection for Variable Length Data Sequences

On variable-length digit sequences, the proposed LSTM detectors outperform conventional mean-based OC-SVM and SVDD methods on AUC, with gradient-trained variants generally strongest.

  • Variable writing speeds produce digit sequences with significantly different lengths, which the proposed generic structure can process.
  • AUC measures anomaly-detection performance by summarizing the ROC curve relating true-positive and false-positive rates.
  • Conventional OC-SVM and SVDD methods obtain significantly lower AUC scores than the introduced LSTM methods when sequence means provide fixed-length inputs.
  • For the 0-normal and 9-anomaly scenario, LSTM-GSVM slightly exceeds LSTM-QPSVM, while LSTM-GSVDD achieves significantly higher AUC than LSTM-QPSVDD.
  • For the 1-normal and 7-anomaly scenario, LSTM-GSVM and LSTM-GSVDD achieve the highest AUC scores in the SVM and SVDD cases, respectively.

B. Benchmark Real Datasets

On four real benchmark datasets, the proposed LSTM and GRU anomaly detectors generally outperform conventional SVM and SVDD methods, with dataset-dependent differences between training strategies.

  • Benchmark scope: The experiments compare LSTM and GRU detectors with conventional methods on occupancy, HKE rate, http, and Alcoa stock price datasets.The benchmark section uses AUC scores and includes GRU implementations alongside the LSTM-based algorithms.
  • Occupancy dataset: On the occupancy dataset, GRU-GSVDD achieves the highest AUC, while LSTM-GSVM and LSTM-QPSVM provide comparable scores.Both recurrent architectures outperform conventional SVM and SVDD algorithms, and gradient-based training outperforms quadratic programming-based training.
  • HKE rate dataset: On the HKE rate dataset, LSTM-GSVM achieves the highest AUC, and gradient-based training outperforms quadratic programming-based training.The recurrent methods outperform conventional methods by preserving sequential information in the time series.
  • http dataset: On the http dataset, GRU-QPSVDD achieves the highest AUC, while quadratic programming outperforms gradient-based training.All methods achieve high, closely spaced AUC scores, so the proposed algorithms show only slight improvement.
  • Alcoa stock price dataset: On the Alcoa stock price dataset, GRU-QPSVDD achieves the highest AUC, while recurrent methods considerably outperform conventional methods.The gradient-based training method provides higher performance than quadratic programming-based training in this experiment.

V. CONCLUDING REMARKS

The paper introduces recurrent anomaly detectors that represent variable-length sequences before applying OC-SVM or SVDD scoring. It jointly trains the representation and detector, extends the framework across supervision settings, and reports effective performance, including GRU variants.

  • Contribution: The paper introduces a generic LSTM structure that processes variable-length data sequences by producing fixed-length representations.The representations are then used by anomaly detectors based on OC-SVM and SVDD.
  • Joint training: The method jointly optimizes LSTM parameters and the final OC-SVM or SVDD scoring function using gradient-based and quadratic-programming-based training.The framework also extends to semi-supervised and fully supervised settings.
  • Optimization: The gradient-based formulation modifies the original OC-SVM and SVDD objectives and provides convergence results to the original formulations.This supports using gradient-based optimization while relating the modified criteria to the actual ones.
  • Extensions: The generic approach is also applied to GRU architectures, yielding anomaly detectors designed for variable-length sequences and time-series data.The paper reports high performance from the resulting algorithms in extensive experiments.
Loading 1710.09207v1…