Source-linked AI summary
Clustering and Unsupervised Anomaly Detection with L2 Normalized Deep Auto-Encoder Representations
Caglar Aytekin, Xingyang Ni, Francesco Cricri, Emre Aksu
TL;DR
The paper addresses the challenge of learning useful deep representations for clustering and unsupervised anomaly detection. It constrains auto-encoder representations with l2 normalization during training and applies k-means clustering to them. The resulting representations support improved clustering, while the clustering-based anomaly detector outperforms reconstruction-error-based deep methods.
Problem
Deep unsupervised clustering methods commonly use auto-encoder representations, while reconstruction-error anomaly detection depends on an assumption that can fail for some normal and anomalous class configurations.
Method
The paper l2 normalizes auto-encoder activations during training, clusters the resulting representations with k-means, and uses cluster-based normality for unsupervised anomaly detection.
Results
The normalized representations improve clustering accuracy, and the proposed anomaly detector outperforms reconstruction-error-based deep anomaly detection methods.
Takeaways & Limitations
Simple k-means on l2 normalized auto-encoder representations can achieve high clustering accuracy and support superior unsupervised anomaly detection without additional clustering losses.
Abstract
from arXiv · showhide
Clustering is essential to many tasks in pattern recognition and computer vision. With the advent of deep learning, there is an increasing interest in learning deep unsupervised representations for clustering analysis. Many works on this domain rely on variants of auto-encoders and use the encoder outputs as representations/features for clustering. In this paper, we show that an l2 normalization constraint on these representations during auto-encoder training, makes the representations more separable and compact in the Euclidean space after training. This greatly improves the clustering accuracy when k-means clustering is employed on the representations. We also propose a clustering based unsupervised anomaly detection method using l2 normalized deep auto-encoder representations. We show the effect of l2 normalization on anomaly detection accuracy. We further show that the proposed anomaly detection method greatly improves accuracy compared to previously proposed deep methods such as reconstruction error based anomaly detection.
I. INTRODUCTION
The paper constrains auto-encoder representations to be l2 normalized during training, improving clustering accuracy across dense and convolutional auto-encoders. It then applies k-means to these representations for unsupervised anomaly detection, outperforming reconstruction-error approaches.
- l2 normalization during auto-encoder training greatly improves clustering accuracy regardless of whether the auto-encoder is dense or convolutional.
- k-means on the constrained representations improves accuracy by a large margin compared with baselines using additional clustering losses or none.
- The proposed anomaly detector clusters l2 normalized deep representations instead of relying on reconstruction error.
- The clustering-based anomaly detector greatly improves performance over reconstruction-error-based and other deep anomaly detection strategies.
A. Deep Unsupervised Anomaly Detection
Deep anomaly detection methods commonly use auto-encoder reconstruction error, but that assumption depends on how normal and anomalous classes are defined. This paper instead proposes clustering deep representations without annotations.
- Existing deep anomaly detectors train auto-encoders on all data and use reconstruction error as an anomaly indicator.
- These methods assume anomalies are fewer than normal samples, so auto-encoders reconstruct anomalies less accurately.
- The reconstruction-error assumption works when normal samples come from one class and anomalies from multiple classes.
- The assumption fails when normal samples span multiple classes and anomalies come from one specific class.
- The paper proposes unsupervised anomaly detection by clustering deep auto-encoder representations, achieving superior performance to reconstruction-error-based anomaly detection.
B. Regularization and Normalization in Neural Networks
The paper distinguishes its unit-ball activation constraint from conventional normalization and weight regularization. It fixes each activation vector's norm rather than estimating statistics or penalizing weights.
- Weight norm regularization penalizes large neural-network weights to reduce over-fitting and improve generalization.
- Batch normalization uses batch statistics, whereas layer normalization computes statistics from summed inputs within each training sample.
- The proposed unit-ball constraint is parameter-free because it sets each activation vector's norm to 1.
- Unlike l2 weight regularization, the constraint applies to activations and fixes their norms rather than penalizing large norms.
A. Clustering on l2 Normalized Deep Auto-Encoder Representations
The method trains an auto-encoder with l2-normalized representations, then applies k-means directly to those representations. During training, this constraint produces more separable and compact clusters than post-training normalization or no normalization.
- Auto-encoder representations: The encoder maps each input through layered computations to produce an auto-encoder representation, which the decoder reconstructs.The encoder and decoder may use dense or convolutional layers, filtering, activations, and optional pooling or up-sampling operations.
- l2 normalization: Auto-encoder training minimizes reconstruction error, with an additional l2 constraint applied directly to the encoded representation.The constraint places representations on the unit ball and defines the constrained loss Lc.
- l2 normalization: l2 normalization makes Euclidean distances depend on vector angles rather than lengths and can regularize the entire auto-encoder.The paper motivates this design for clustering methods such as conventional k-means.
- Empirical comparison: On MNIST, l2 normalization during training produces more separable and compact clusters than no normalization or normalization applied after training.The digit 7 forms a clear single cluster only with the training-time constraint, whereas post-training normalization does not help.
- Inference: After constrained training, clustering is performed directly with k-means without adding another clustering loss.The inference procedure is illustrated in Fig. 2a.
B. Unsupervised Anomaly Detection using l2 Normalized Deep Auto-Encoder Representations
The anomaly detector clusters l2-normalized auto-encoder representations and scores samples by their similarity to cluster centroids. Samples whose normality scores fall below a threshold are classified as abnormal.
- Detection pipeline: The auto-encoder is trained on the entire dataset, including normal and abnormal samples, without annotations or supervision.The training uses the constrained loss from the l2-normalized representation method.
- Detection pipeline: The detector clusters l2-normalized representations with k-means and treats the centroids as representations of normal clusters, assuming anomalies are fewer than normal clusters.Anomalies may introduce some error into the learned centroids.
- Scoring: Each sample receives a normality score based on its dot-product similarity to l2-normalized cluster centroids.Because both representations and centroids are l2 normalized, the score lies in [0, 1].
- Decision rule: Samples are detected as abnormal when their normality score vi is below a threshold τ ∈ [0, 1].The threshold provides the decision rule for converting scores into anomaly predictions.
A. Clustering
The section evaluates clustering accuracy for dense and convolutional auto-encoder representations using a standard mapped accuracy metric. Across these settings, l2 normalization is compared with unnormalized and alternative normalization approaches, with the reported results favoring l2 normalization.
- Evaluation: Clustering accuracy is evaluated by maximizing agreement between ground-truth labels and cluster assignments under one-to-one mappings.The maximization can be performed with the Hungarian algorithm.
- Experimental setup: The experiments compare auto-encoder representations with and without an l2 normalization constraint in dense and convolutional architectures.Dense auto-encoders are evaluated on MNIST, while convolutional auto-encoders are evaluated on MNIST and USPS.
- Dense auto-encoders: Dense auto-encoder clustering is tested with the architecture used in DEC and IDEC, using a 10-dimensional encoded representation.The encoder uses four hidden layers with 500, 500, 2000, and 10 neurons.
- Convolutional auto-encoders: Convolutional auto-encoder clustering is tested with the DCEC model structure and compared against reported baseline results.The model uses convolutional encoding layers followed by a dense layer of dimension 10.
- Normalization comparison: l2 normalization outperforms batch and layer normalization by a large margin, while the latter methods provide no noticeable improvement over the CAE+k-means baseline.On MNIST, batch and layer normalization produce a significant accuracy decrease.
B. Anomaly Detection
The paper evaluates clustering-based anomaly detection using l2-normalized auto-encoder representations and compares it with reconstruction-error baselines. Across the reported experiments, clustering generally performs better, while similar-looking digits remain a source of confusion.
- Evaluation: The method evaluates anomaly detection by thresholding scores to form ROC curves and measuring performance with AUC.The experiments repeat random partitions ten times and report mean AUC.
- Results: The clustering-based method drastically outperforms reconstruction-error anomaly detection for the CAE structure.
- Results: For digits 1, 7, and 9, reconstruction error performs very poorly, especially for digit 1, because simple shapes remain easy to reconstruct.The paper identifies this as a drawback of reconstruction-error detection.
- Effect of l2 normalization: In 9/10 anomaly-class cases, l2 normalization performs better than no normalization, with digit 9 as the exception.Digits 4 and 9 also show lower performance, potentially because similar handwriting causes clustering confusion.
- Comparison with baselines: Against dense auto-encoder reconstruction-error methods, the proposed method outperforms both variants in 9/10 cases, except against VAE on digit 4.The comparison is harder for the proposed method because anomalous samples are included during training, whereas the baselines train only on normal data.
V. CONCLUSION
The paper concludes that l2-normalized deep auto-encoder representations form separable Euclidean clusters that support effective k-means clustering and unsupervised anomaly detection. It reports improved anomaly-detection performance over reconstruction-error approaches.
- Clustering: l2-normalized deep auto-encoder representations cluster well in Euclidean space, enabling high-accuracy k-means clustering without additional clustering losses.
- Anomaly detection: The proposed anomaly detection method clusters l2-normalized deep auto-encoder representations without supervision.
- Results: The l2 normalization constraint drastically increases anomaly-detection performance, while reconstruction-error methods perform weakly under this anomaly definition.