Source-linked AI summary
Neural Transformation Learning for Deep Anomaly Detection Beyond Images
Chen Qiu, Timo Pfrommer, Marius Kloft, Stephan Mandt, Maja Rudolph
TL;DR
Anomaly detection beyond images lacks clear, manually designed transformations for data such as time series and tabular records. NeuTraL AD learns transformations and an encoder jointly with a deterministic contrastive objective, improving detection accuracy across these domains, including a 10-point AUC gain on epilepsy time series.
Problem
For anomaly detection beyond images, it is unclear which transformations are useful, and designing them manually is difficult.
Method
NeuTraL AD jointly learns transformations and an encoder so transformed data preserve semantic information while different views remain distinguishable.
Results
NeuTraL AD improves anomaly detection over the state of the art across time-series and tabular data, including an epilepsy AUC increase from 82.6% to 92.6% (+10%).
Takeaways & Limitations
Learning domain-specific transformations extends self-supervised anomaly detection to time series and tabular data.
Takeaways & Limitations
Alternative transformation-learning losses can reach constant or identity edge cases that undermine anomaly detection unless constrained or regularized.
Abstract
from arXiv · showhide
Data transformations (e.g. rotations, reflections, and cropping) play an important role in self-supervised learning. Typically, images are transformed into different views, and neural networks trained on tasks involving these views produce useful feature representations for downstream tasks, including anomaly detection. However, for anomaly detection beyond image data, it is often unclear which transformations to use. Here we present a simple end-to-end procedure for anomaly detection with learnable transformations. The key idea is to embed the transformed data into a semantic space such that the transformed data still resemble their untransformed form, while different transformations are easily distinguishable. Extensive experiments on time series demonstrate that our proposed method outperforms existing approaches in the one-vs.-rest setting and is competitive in the more challenging n-vs.-rest anomaly detection task. On tabular datasets from the medical and cyber-security domains, our method learns domain-specific transformations and detects anomalies more accurately than previous work.
1. Introduction
NeuTraL AD addresses the difficulty of choosing transformations for non-image anomaly detection by learning transformations and an encoder jointly. Experiments report improved accuracy on time-series and tabular data.
- For time series and tabular data, useful transformations are difficult to design manually.
- NeuTraL AD jointly learns transformations and an encoder so transformed samples preserve semantic information while different views remain distinguishable.
- The method uses a deterministic contrastive loss that requires neither additional regularization nor adversarial training and can directly provide anomaly scores.
- AUC increased from 82.6% to 92.6% (+10%) on an epilepsy time-series dataset.
- On Arrhythmia, the F1 score increased by +2.9 percentage points to an accuracy of 60.3.
2. Related Work
Prior self-supervised anomaly detection largely uses predefined transformations or learned representations, while NeuTraL AD learns transformations for anomaly detection beyond images. Its single-sample objective supports anomaly scoring and transformation learning.
- Image-based self-supervised anomaly detection commonly predicts applied transformations or uses representations learned from transformed views.
- Extending transformation-based anomaly detection beyond images is difficult because suitable transformations depend on the data domain.
- NeuTraL AD learns transformations rather than relying on fixed choices and achieves consistently higher performance on the same tabular datasets.
- NeuTraL AD determines which view is the original using only one sample, which supports test-time anomaly scoring and transformation learning.
- Unlike prevalent time-series methods that score individual time points, NeuTraL AD receives whole time series and produces sequence-level anomaly detection.
3. Neural Transformation Learning for Deep Anomaly Detection
NeuTraL AD jointly learns transformations and an encoder with a deterministic contrastive objective that balances semantic preservation and view diversity. The resulting loss also serves as a deterministic anomaly score and avoids trivial transformation solutions.
- Proposed Method: NeuTraL AD is a two-component pipeline consisting of learnable transformations and an encoder trained jointly with deterministic contrastive loss.
- Proposed Method: The DCL pulls each transformed sample toward its original embedding while pushing embeddings of different transformations apart.
- Anomaly Score: At test time, the training loss functions as a deterministic anomaly score, with higher scores indicating greater likelihood of anomaly.
- Transformation Requirements: The transformations must preserve relevant semantics and produce diverse views; constant and identical transformations violate these requirements.
- Theory: The DCL avoids the constant and identity edge cases that can undermine alternative transformation-learning losses.
- Empirical Behavior: After training, normal samples tend to receive low anomaly scores while anomalies receive higher scores, making held-out classes distinguishable.
4. Empirical Study: Deep Anomaly Detection of Time Series and Tabular Data
The empirical study evaluates NeuTraL AD for whole-sequence time-series and tabular anomaly detection using one-vs-rest and more challenging n-vs-rest protocols. Across these settings, learned transformations generally outperform shallow and deep baselines, while performance becomes harder to maintain as nominal variability increases.
- Evaluation setup: The study compares NeuTraL AD with shallow, deep, self-supervised, and time-series-specific anomaly-detection baselines.Evaluation uses standard one-vs-rest and the more challenging n-vs-rest protocol.
- Evaluation setup: The experiments target whole-sequence anomaly detection for time series, alongside tabular anomaly detection across representative application domains.The paper distinguishes sequence-level detection from novelty detection at individual time points.
- One-vs-rest results: NeuTraL AD outperforms all shallow baselines and deep-learning baselines in 4 of 5 time-series one-vs-rest experiments.On RS, transformation prediction with fixed, hand-crafted transformations performs better, showing that manually designed transformations succeed only sometimes.
- N-vs-rest results: As the number of normal classes increases, anomaly-detection performance drops because greater nominal variability makes the task more challenging.This pattern is reported for the n-vs-rest experiments and for deep-learning methods in the n = N −1 setting.
- N-vs-rest results: NeuTraL AD outperforms all baselines on NATOPS and all deep-learning baselines on SAD, but LOF outperforms it on SAD when n > 3.The comparison concerns n-vs-rest experiments with varying numbers of normal classes.
- Design choices: With sufficiently many transformations, NeuTraL AD becomes robust to transformation parametrization and achieves competitive results even with small K.Performance has higher variance when K ≤4, while transformation-based methods require roughly K times the memory of other deep-learning methods.
5. Conclusion
The paper proposes self-supervised anomaly detection with learnable transformations and a deterministic contrastive objective. NeuTraL AD improves state-of-the-art performance across time series and tabular data.
- NeuTraL AD learns transformations for self-supervised anomaly detection beyond images.
- Its deterministic contrastive loss encourages transformed views to preserve semantic information while remaining diverse.
- NeuTraL AD improves over the state of the art on time series and tabular anomaly detection.
A. Proofs for Section 3.2
This appendix compares training objectives for neural transformation learning by testing whether they produce transformations that preserve sample semantics and maintain view diversity.
- The appendix evaluates loss functions by whether learned transformations satisfy semantic preservation and diversity requirements.
- The constant edge-case removes dependence on the original sample, violating the semantic requirement.
- The identity edge-case reproduces every sample perfectly, violating the diversity requirement.
- The comparison covers NeuTraL AD’s deterministic contrastive loss, transformation prediction loss, and SimCLR loss.
A.1. Proof of Proposition 1
The proof shows that transformation prediction can be minimized by transformations whose encoded outputs identify only the transformation index, not the input sample.
- Optimizing transformation prediction jointly over transformation and encoder parameters is tested against the two transformation requirements.
- The constant edge-case can approach the global minimum of the transformation prediction loss.
- In this edge-case, fφ(Tk(x)) = Cck, where ck is a one-hot vector encoding the kth transformation position.
- As C grows, the negative-log-probability loss approaches its lower bound of 0.
A.2. Proof of Proposition 2
The proof shows that SimCLR can be minimized using identity transformations when the encoder is sufficiently flexible, producing identical views rather than diverse ones.
- The analysis asks whether optimizing SimCLR over transformations and the encoder yields transformations satisfying both requirements.
- The identity edge-case Tk(x) = x with an adequate encoder is a minimizer of the SimCLR loss.
- SimCLR separates alignment and uniformity terms, and minimizing both provides a sufficient condition for minimizing the loss.
- With a sufficiently flexible encoder, an equivalent minimum can rely only on identity transformations.
- The construction defines ˜fφ using T1 and uses it with identity transformations to obtain the same minimal loss.
A.3. Proof of Proposition 3
Proposition 3 rules out constant and identity transformations as minimizers of the DCL objective, establishing that these edge cases cannot satisfy the intended learning behavior. The proof uses nonzero gradients for constant transformations and compares identity with a lower-loss counterexample.
- Proposition 3, Part 1: Constant transformations do not minimize DCL for any finite constant or as the constant tends to infinity.The proposition considers transformations fφ(Tk(x)) = Cck, with ck encoding the transformation index.
- Proposition 3, Part 1: At the constant edge case, the gradient cannot vanish because the required condition would imply a non-integral number of transformations K.The proof concludes that Equation (16) is nonzero because K is defined as an integer.
- Proposition 3, Part 2: Identity transformations do not minimize DCL for an adequate encoder fφ.The identity edge case sets Tk(x) = x for every transformation.
- Proposition 3, Part 2: Under identity transformations, DCL equals K times the cross-entropy of the uniform distribution, corresponding to random guessing.An adequate encoder can achieve lower loss, and trained anomaly scores become smaller than K log K.
- Proposition 3, Part 2: For K = 2, the counterexample has loss 0.627, lower than the identity edge case’s loss 1.386.The counterexample uses z1 ⊥ z, z2 ⊥ z, and z1 = −z2.
B.2. Implementations of baselines
The baseline implementations use standard libraries or published code, with selected hyperparameters tuned over explicitly specified ranges. The time-series baseline additionally uses hand-crafted transformations for its classification-based comparison.
- OC-SVM, IF, and LOF use scikit-learn implementations with default parameters.
- Deep SVDD, DROCC, and DAGMM use published implementations with encoders similar to NeuTraL AD.DAGMM selects the best mixture-component count from 4 through 12.
- GOAD uses published code while selecting its affine-transformation output dimension r from 22 through 26.The best-performing value is selected.
- The classification-based time-series baseline uses hand-crafted compositions including flipping along the time axis.
- The RNN baseline uses two recurrent layers and two ReLU-activated linear layers, while LSTM-ED uses an implementation taken from the web.
C. Tabular datasets
The tabular evaluation uses four datasets spanning medical diagnosis and cybersecurity, with dataset-specific feature processing and anomaly-label conventions. Figures also visualize learned transformations for representative spoken-digit and NATOPS examples.
- The tabular evaluation contains Arrhythmia, Thyroid, KDDCUP, and KDDCUP-Rev datasets.
- Medical datasets: Thyroid uses six continuous attributes, with hyperfunction as abnormal and the other two classes as normal.
- Cybersecurity datasets: KDDCUP uses 120 dimensions after one-hot encoding seven categorical attributes, with attack samples labeled normal and non-attack samples abnormal.
- Cybersecurity datasets: KDDCUP-Rev reverses the anomaly convention and subsamples attack data to 25% of the non-attack count.
- Transformation visualizations: Figures 7 and 8 visualize K = 4 learned transformations for normal and abnormal examples from spoken Arabic digits and NATOPS.
D.2. Results for tabular data
For tabular data, learned transformations provide visual explanations of anomaly scores by showing which masks and transformation terms contribute most. The visualizations also expose a limitation: diverse transformations are learned, but their anomaly-detection usefulness is not guaranteed.
- Thyroid explanations: Thyroid visualizations show one normal example, three anomalies, four learned masks, and transformation-specific score contributions.
- Thyroid explanations: The normal thyroid example has a very low score, whereas anomaly scores are mainly contributed by different transformation terms.
- Thyroid explanations: Mask M4 focuses on the fourth attribute and contributes high scores for all listed anomalies, while M2 is less useful for anomaly detection.
- Scope of learned transformations: NeuTraL AD can learn diverse transformations, but without labels it is not guaranteed that every learned transformation is useful for anomaly detection.
- Score visualizations: Normal data are not dominated by one transformation, whereas anomalies are mainly dominated by T3 and T4.
- Score visualizations: Simplex visualizations compare whether T1, T3, or T4 dominates scores and encode score magnitude through transparency.