Source-linked AI summary
Deep Structured Energy Based Models for Anomaly Detection
Shuangfei Zhai, Yu Cheng, Weining Lu, Zhongfei Zhang
TL;DR
The paper addresses how to perform anomaly detection by modeling complex data distributions with deep, structured energy-based models. It proposes architectures and score-matching training for multiple data types, and reports that DSEBMs consistently match or outperform competing methods on benchmark tasks.
Problem
Anomaly detection needs effective distribution models that generalize across static, sequential, and spatial data while supporting scalable training and statistically sound decisions.
Method
DSEBMs use deterministic deep neural networks as structured energy functions and train them with score matching, using energy score and reconstruction error as anomaly criteria.
Results
DSEBMs consistently match or outperform competing anomaly-detection algorithms across static, sequential, and spatial benchmark datasets.
Takeaways & Limitations
The study extends EBMs to multiple deep architectures and data structures while simplifying training through score matching instead of maximum likelihood.
Takeaways & Limitations
The recurrent formulation simplifies the score gradient by omitting cross-time terms, while the convolutional formulation similarly uses an omitted gradient simplification.
Abstract
from arXiv · showhide
In this paper, we attack the anomaly detection problem by directly modeling the data distribution with deep architectures. We propose deep structured energy based models (DSEBMs), where the energy function is the output of a deterministic deep neural network with structure. We develop novel model architectures to integrate EBMs with different types of data such as static data, sequential data, and spatial data, and apply appropriate model architectures to adapt to the data structure. Our training algorithm is built upon the recent development of score matching \cite{sm}, which connects an EBM with a regularized autoencoder, eliminating the need for complicated sampling method. Statistically sound decision criterion can be derived for anomaly detection purpose from the perspective of the energy landscape of the data distribution. We investigate two decision criteria for performing anomaly detection: the energy score and the reconstruction error. Extensive empirical studies on benchmark tasks demonstrate that our proposed model consistently matches or outperforms all the competing methods.
1. Introduction
The paper frames anomaly detection as distribution characterization and proposes DSEBMs to model complex data with structured deep architectures, efficiently trained through score matching. It evaluates energy and reconstruction-based criteria across data types and reports competitive benchmark performance.
- Anomaly detection identifies patterns that do not conform to expected normal patterns, motivating models that characterize the underlying data distribution.
- Deep architectures can capture complex distributions and learn multiple levels of increasingly abstract representations.Properly regularized autoencoders are cited as examples that characterize distributions more effectively than shallow PCA or K-Means methods.
- The paper identifies four development questions: deep distribution modeling, adaptation to static, sequential, and spatial data, scalable training, and statistically sound decision criteria.
- DSEBMs use deterministic deep neural networks as energy functions and support fully connected, recurrent, or convolutional structures for different data types.The architectures target static vectors, sequences such as audio, and spatial data such as images.
- Score matching replaces maximum likelihood training to avoid the complicated sampling procedures required by energy-based models.
- Energy score and reconstruction error are evaluated as anomaly criteria, with DSEBMs consistently matching or outperforming competing algorithms on several benchmarks.
2. Background
The background introduces EBMs as flexible density models whose energy functions define normalized probabilities, then connects score matching with autoencoder training. This connection enables efficient end-to-end training without sophisticated sampling.
- Energy Based Models: Energy-based models define probability densities through an energy function and a partition function that normalizes the density.The energy is the negative log probability associated with an input, while the partition function ensures integration to one.
- Energy Based Models: EBMs are flexible because their energy functions can be parameterized in arbitrary sensible ways, but maximum likelihood is intractable because of the partition function.Training therefore typically relies on MCMC methods to approximate the required integration.
- Restricted Boltzmann Machines: RBMs are EBM examples for continuous inputs, with energies parameterized by weights, biases, and a softplus function.Multiple RBMs can be stacked into deep RBMs, although training remains harder than training deterministic neural networks.
- Denoising Autoencoders and Score Matching: Autoencoders learn to reconstruct inputs, using a reconstruction function typically composed of an encoder and decoder with shared parameters.Denoising autoencoders instead reconstruct original inputs from randomly corrupted versions and can be trained with standard SGD.
- Denoising Autoencoders and Score Matching: Score matching estimates non-normalized densities by minimizing the expected distance between model and true score functions rather than directly maximizing training-instance probabilities.The model score equals the input gradient of log density, or the negative input gradient of energy.
- Denoising Autoencoders and Score Matching: The score-matching objective can take the form of an autoencoder objective, allowing arbitrary differentiable EBMs to be trained end-to-end with SGD.This avoids sophisticated sampling procedures and layer-wise pretraining.
3. Deep Structured EBMs
DSEBMs adapt deterministic deep energy functions to fully connected, recurrent, and convolutional structures for static, sequential, and spatial data. Their reconstruction functions are derived from energy gradients so the models can be trained with score matching and SGD.
- Structured architectures: DSEBMs use fully connected, recurrent, or convolutional architectures to match the structure of static, sequential, or spatial data.RNNs are used for sequences, while CNNs are used for spatially structured inputs.
- Fully Connected EBMs: The fully connected formulation defines a multilayer energy function and includes an input prior that penalizes inputs far from a bias vector.
- Fully Connected EBMs: For fully connected EBMs, the reconstruction function is derived through the chain rule and inserted into an autoencoder objective for standard training.
- Recurrent EBMs: Recurrent EBMs factorize sequence probability across time and model each conditional distribution with an EBM whose parameters depend on previous inputs through an RNN.To reduce parameter requirements, the recurrent model fixes W_t across time and updates biases through the RNN.
- Recurrent EBMs: Score matching for recurrent EBMs uses a simplified gradient that omits cross-time gradient terms, followed by backpropagation through time.The paper reports that the simplification yields a more succinct objective and works well in practice.
- Convolutional EBMs: Convolutional EBMs directly use convolutional networks, optionally with pooling or fully connected layers, and train end-to-end with score matching.This differs from prior CNN-RBM combinations that alternately convolve and stack RBM layers.
- Convolutional EBMs: The convolutional formulation derives reconstruction functions for convolution and max-pooling layers, with analogous treatment possible for mean pooling.
4. Deep Structured EBMs for Anomaly Detection
The paper derives two anomaly-detection criteria from a trained DSEBM: energy thresholds and reconstruction error, which corresponds to the energy gradient norm. Energy is more reliable in a one-dimensional example, while reconstruction error remains reasonable in high dimensions despite a specific false-positive case.
- Low-probability samples are selected as outliers by thresholding their energy above Eth.Although exact probability computation is intractable, the normalization term is constant with respect to x, making probability and energy thresholding equivalent.
- Reconstruction-error anomaly detection is motivated by the connection between energy-based models and regularized autoencoders.Earlier unregularized autoencoders used reconstruction error for anomaly detection but could not be interpreted as density models.
- High reconstruction errors correspond to large energy gradient norms, so samples exceeding Errorth are classified as outliers.Inliers usually lie near energy local minima, where the gradient is close to zero.
- In a 1D example, energy and reconstruction error correctly classify x1 and x3, but reconstruction error falsely classifies x2 as an inlier.The false positive occurs because x2 lies near a local energy maximum, where the gradient is also small; energy correctly identifies it as an outlier.
- Despite this failure mode, reconstruction error remains a reasonable criterion because outliers near local maxima become exponentially unlikely as dimensionality increases.
5. Experimental Evaluation
The evaluation tests DSEBMs on static, sequential, and spatial data using energy and reconstruction-error criteria, comparing them with established baselines. Across these settings, DSEBM-e generally performs best, while reconstruction error is competitive but weaker except on Thyroid.
- Experimental setup: Experiments cover static, sequential, and spatial datasets, using fully connected, recurrent, and convolutional EBMs, respectively.The study compares DSEBM-e and DSEBM-r with established static and sequential anomaly-detection baselines.
- Static Data: On static datasets, DSEBMs are comparable to or better than the best baselines overall, with DSEBM-e best on Usenet recall and F1 and KDD99 F1.On Thyroid, they perform slightly worse than OC-SVM and RKDE, which the authors associate with the dataset’s low dimensionality.
- Sequential Data: DSEBM-e achieves the highest mean precision and mean F1 in most sequential cases, while DSEBM-r is generally second best.On FITNESS, DSEBMs improve over other baselines by 4% in mean precision, 6% in mean recall, and 5% in mean F1.
- Spatial Data: DSEBM-e is best overall on image datasets for mean recall and mean F1, improving F1 by 3.5% and 2.3% over the best-performing baselines.The reported pattern remains consistent as the outlier ratio varies from 0.1 to 0.4.
- Energy VS. Reconstruction Error: DSEBM-e consistently outperforms DSEBM-r except on Thyroid, supporting energy score as the more accurate criterion than reconstruction error.In the Caltech-101 illustration, all seven visually similar outliers have small reconstruction errors but exceed the energy threshold.
6. Related Work
Prior anomaly-detection work includes reconstruction-, probability-density-, and compact-data-model approaches. The paper distinguishes its direct density-modeling path from CNN-based density-ratio estimation and notes that prior score-matched EBMs were mostly shallow.
- Existing anomaly-detection methods include reconstruction-based, probability-density-based, and compact data-model approaches.
- The paper’s approach differs from CNN-based inlier detection by directly modeling the density rather than estimating density ratios.
- Most prior EBMs trained with score matching were constrained to shallow models, limiting applications to relatively simple tasks.
7. Conclusion
The paper trains deep structured energy-based models for anomaly detection across fully connected, recurrent, and convolutional structures. Score matching simplifies training, while experiments across static, sequential, and spatial datasets show performance that consistently matches or outperforms state-of-the-art methods.
- DSEBMs extend energy-based models to fully connected, recurrent, and convolutional structures.
- Score matching replaces maximum likelihood estimation to significantly simplify DSEBM training.
- Systematic experiments across static, sequential, and spatial datasets show DSEBMs consistently match or outperform state-of-the-art anomaly-detection algorithms.
- The paper presents extensive evaluation of deep structured models for anomaly detection across three data structures.