Source-linked AI summary
Anomaly Detection via Reverse Distillation from One-Class Embedding
Hanqiu Deng, Xingyu Li
TL;DR
Unsupervised anomaly detection needs teacher-student representations that remain discrepant on anomalies, but similar architectures can limit that distinction. This paper introduces Reverse Distillation with a teacher encoder, student decoder, and one-class bottleneck embedding, reporting significant gains across anomaly detection, localization, and novelty detection.
Problem
Similar or identical teacher and student architectures can hinder diverse anomalous representations in knowledge-distillation anomaly detection.
Method
Reverse Distillation uses a teacher encoder, student decoder, and trainable one-class bottleneck embedding to restore multiscale teacher representations from compact embeddings.
Results
The method significantly outperformed previous methods in anomaly detection, anomaly localization, and novelty detection.
Takeaways & Limitations
Reverse Distillation provides a knowledge-distillation paradigm that improves one-class knowledge transfer across anomaly detection tasks.
Takeaways & Limitations
Localization is relatively weak on the transistor dataset because predictions and annotations cover different regions.
Abstract
from arXiv · showhide
Knowledge distillation (KD) achieves promising results on the challenging problem of unsupervised anomaly detection (AD).The representation discrepancy of anomalies in the teacher-student (T-S) model provides essential evidence for AD. However, using similar or identical architectures to build the teacher and student models in previous studies hinders the diversity of anomalous representations. To tackle this problem, we propose a novel T-S model consisting of a teacher encoder and a student decoder and introduce a simple yet effective "reverse distillation" paradigm accordingly. Instead of receiving raw images directly, the student network takes teacher model's one-class embedding as input and targets to restore the teacher's multiscale representations. Inherently, knowledge distillation in this study starts from abstract, high-level presentations to low-level features. In addition, we introduce a trainable one-class bottleneck embedding (OCBE) module in our T-S model. The obtained compact embedding effectively preserves essential information on normal patterns, but abandons anomaly perturbations. Extensive experimentation on AD and one-class novelty detection benchmarks shows that our method surpasses SOTA performance, demonstrating our proposed approach's effectiveness and generalizability.
1. Introduction
The paper proposes Reverse Distillation for unsupervised anomaly detection, replacing conventional encoder-based teacher-student models with heterogeneous encoder-decoder architectures and reverse feature transfer. A one-class bottleneck embedding further supports compact normal-pattern restoration, and experiments report new state-of-the-art performance.
- Unsupervised anomaly detection identifies and localizes anomalies when no prior abnormality information is available, using normal samples for reference.
- Conventional knowledge-distillation anomaly detectors can produce insufficiently discrepant anomaly representations because teacher and student architectures and data flow are similar.
- Reverse Distillation uses a teacher encoder and student decoder, feeding the student a low-dimensional embedding to restore the teacher’s multiscale representations.
- The method’s reverse structure separates down-sampling and up-sampling roles, while distillation proceeds from high-level representations toward low-level features.
- Extensive experiments report new state-of-the-art performance, with the reverse-distillation paradigm and OCBE module presented as the main contributions.
- The one-class bottleneck embedding projects high-dimensional teacher features into a compact space that preserves information for anomaly-free representation restoration.
2. Related Work
The related work covers one-class, reconstruction, augmentation, pretrained-feature, and knowledge-distillation approaches to unsupervised anomaly detection. The paper distinguishes Reverse Distillation through its heterogeneous encoder-decoder pair and reverse data flow.
- Classical one-class methods model compact normal distributions using OC-SVM, SVDD, DeepSVDD, or PatchSVDD.
- Generative reconstruction methods train models such as autoencoders or GANs on normal samples, but anomalous regions may also be reconstructed successfully.
- Data-augmentation methods create pseudo anomalies to supervise detection, but can bias models toward pseudo outliers and miss diverse anomaly types.
- Pretrained-feature approaches detect anomalies by memorizing anomaly-free features or measuring similarity with distances such as Mahalanobis distance.
- Knowledge-distillation methods exploit discrepant teacher-student features on anomalies and use strategies such as model ensembles or multilevel feature alignment.
- Reverse Distillation differs from prior knowledge-distillation methods by using heterogeneous teacher-encoder and student-decoder networks with reverse data flow.
3. Our Approach
The approach uses reverse distillation with a frozen teacher encoder, a trainable one-class bottleneck, and a reversed student decoder to restore multiscale normal representations. Anomaly scores arise from teacher–student feature discrepancies across scales, with compact embedding designed to suppress anomalous perturbations.
- 3.1. Reverse Distillation: Reverse distillation pairs a fixed teacher encoder with a reversed student decoder that restores the teacher’s multiscale representations from a bottleneck embedding.The teacher extracts comprehensive features, while the decoder learns restoration using normal training samples.
- 3.1. Reverse Distillation: The teacher uses a frozen pre-trained ImageNet backbone, while the student decoder mirrors its representation dimensions through a symmetrical reversed architecture.ResNet and WideResNet are reported as suitable teacher candidates for extracting rich image features.
- 3.1. Reverse Distillation: Multi-scale distillation compares teacher and student activations with channel-wise cosine similarity, accumulating anomaly maps across feature layers.Low similarity indicates abnormality, and K denotes the number of feature layers used.
- 3.2. One-Class Bottleneck Embedding: The one-class bottleneck embedding projects high-dimensional teacher features into a compact space intended to retain normal information while suppressing unusual perturbations.The OCBE module combines multi-scale feature fusion with one-class embedding and is optimized jointly with the student on normal samples.
- 3.3. Anomaly Scoring: At inference, the teacher can represent abnormal features while the student may fail to restore them, producing discrepant representations for anomaly localization.Multi-scale anomaly maps are upsampled and accumulated into a pixel-level score map; Gaussian smoothing is applied afterward.
- 3.3. Anomaly Scoring: The method ranks first on MVTec according to average texture, object, and overall AUROC scores.The table evaluates images at 256 × 256 resolution and highlights the top two AUROC (%) methods per category.
4. Experiments and Discussions
Experiments evaluate reverse distillation across MVTec anomaly detection and localization, one-class novelty detection, ablations, and complexity. The method achieves state-of-the-art results while retaining strong performance with smaller backbones and low time and memory complexity, although transistor localization remains relatively weak.
- Experimental setup: The evaluation covers MVTec anomaly detection and localization, one-class novelty detection, ablations, and computational complexity.MVTec contains 15 datasets with anomaly-free training images, anomalous test images, and pixel-level localization annotations.
- Anomaly detection: 99.5% texture AUROC and 98.0% object AUROC establish new MVTec anomaly-detection state-of-the-art results.The average outcome exceeds the previous state of the art by 2.5%.
- Anomaly localization: 97.8% AUROC and 93.9% PRO are the average MVTec anomaly-localization scores, surpassing state-of-the-art results.AUROC measures pixel-wise performance, whereas PRO emphasizes region-based behavior.
- Complexity analysis: The approach achieves performance gains with low time and memory complexity compared with pre-trained feature-model approaches.The comparison reports inference time, memory usage, and detection and localization performance on MVTec.
- Limitations: Transistor localization is relatively weak because predictions cover misplaced regions while annotations cover both misplaced and original areas.Using higher-level features improves transistor AUROC from 94.5% with layers two and three to 97.3% with layer three alone.
- One-class novelty detection: The method produces excellent one-class novelty-detection results across MNIST, FashionMNIST, and CIFAR10.Training uses samples from a single class and detects novel samples.
- Ablation studies: The trainable OCE and MFF blocks improve anomaly detection and localization by condensing feature codes and fusing rich features.A pre-trained residual bottleneck can retain anomaly features, reducing teacher-student representation discrepancy.
- Backbones and features: Reverse distillation remains effective with ResNet18, while multi-scale fusion covers more anomaly types by combining local texture and global structure information.The second feature layer provides the best single-layer trade-off between local and global information.
5. Conclusion
The paper concludes that reverse distillation addresses limitations of earlier knowledge-distillation anomaly-detection methods. Trainable one-class embedding and multi-scale fusion further improve one-class knowledge transfer, with gains across anomaly detection, localization, and novelty detection.
- Conclusion: Reverse distillation boosts the teacher-student model’s response to anomalies in anomaly detection.The paradigm uses an encoder-decoder structure and reverse knowledge distillation.
- Conclusion: Trainable one-class embedding and multi-scale feature fusion improve one-class knowledge transfer.These blocks are introduced within the reverse-distillation framework.
- Conclusion: Experiments show significant improvements over previous methods in anomaly detection, anomaly localization, and novelty detection.The conclusion reports gains across all three evaluated task types.