Source-linked AI summary
Multiresolution Knowledge Distillation for Anomaly Detection
Mohammadreza Salehi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad Hossein Rohban, Hamid R. Rabiee
TL;DR
The paper targets weak generalization and imprecise localization in unsupervised anomaly detection trained only on normal data. It distills intermediate knowledge from an ImageNet-pretrained expert into a compact cloner and uses their activation discrepancy with interpretability methods. Across diverse datasets and tasks, it reports superior or competitive results, including domains far from ImageNet.
Problem
Normal-only training and limited sample sizes make it difficult to learn generalizable features that also discriminate normal from anomalous inputs, while existing methods often lack precise localization.
Method
The method distills intermediate activation values from an ImageNet-pretrained source network into a smaller cloner trained on normal data, then measures their discrepancy and gradients for detection and localization.
Results
The method achieves superior results on many datasets and remains competitive on others across anomaly detection and localization, including domains far from ImageNet.
Takeaways & Limitations
Intermediate multi-layer distillation offers a distinctive anomaly criterion with precise localization without intensive region-based training and testing.
Takeaways & Limitations
The cloner can face difficulty transferring the source network’s knowledge, addressed here with generic rotation and scaling augmentations.
Abstract
from arXiv · showhide
Unsupervised representation learning has proved to be a critical component of anomaly detection/localization in images. The challenges to learn such a representation are two-fold. Firstly, the sample size is not often large enough to learn a rich generalizable representation through conventional techniques. Secondly, while only normal samples are available at training, the learned features should be discriminative of normal and anomalous samples. Here, we propose to use the "distillation" of features at various layers of an expert network, pre-trained on ImageNet, into a simpler cloner network to tackle both issues. We detect and localize anomalies using the discrepancy between the expert and cloner networks' intermediate activation values given the input data. We show that considering multiple intermediate hints in distillation leads to better exploiting the expert's knowledge and more distinctive discrepancy compared to solely utilizing the last layer activation values. Notably, previous methods either fail in precise anomaly localization or need expensive region-based training. In contrast, with no need for any special or intensive training procedure, we incorporate interpretability algorithms in our novel framework for the localization of anomalous regions. Despite the striking contrast between some test datasets and ImageNet, we achieve competitive or significantly superior results compared to the SOTA methods on MNIST, F-MNIST, CIFAR-10, MVTecAD, Retinal-OCT, and two Medical datasets on both anomaly detection and localization.
1. Introduction
The paper addresses anomaly detection’s generality and localization challenges by distilling intermediate representations from a pretrained source into a compact cloner, then exploiting their discrepancy.
- Anomaly detection must handle both broad and subtle anomalies while precisely localizing defective regions, a combination not yet fully achieved across datasets.
- Limited normal-only training data makes generalizable representation learning difficult, while many existing methods work well only in one anomaly setting.
- The method distills intermediate activations from an ImageNet-pretrained source into a simpler cloner trained only on normal data, producing divergent behavior on anomalies.
- Gradient-based discrepancy analysis yields precise anomaly heatmaps without region-based training or testing, supporting real-time localization.
- A compact cloner transfers comprehensive source knowledge while concentrating on features that distinguish normal from anomalous inputs.
- The method uses computationally inexpensive, stable training and reports superior or competitive results across diverse anomaly detection and localization datasets.
2. Related Work
Prior anomaly-detection methods use reconstruction, compact latent spaces, pretrained features, or interpretability techniques, but face limitations in robustness, localization, or computational cost.
- Autoencoder-based methods reconstruct normal inputs more accurately than anomalies, but reported approaches can fail on industrial or complex datasets.
- Compact latent-space methods retain useful normal information, yet normal-only optimization can converge to trivial solutions and require unprincipled early stopping.
- The proposed framework combines source–cloner discrepancy with interpretability algorithms to produce pixel-precise anomaly localization maps.
- GT performs well in one-class settings but fails on subtle-anomaly datasets such as MVTecAD.
- Using only pretrained VGG last-layer features can lead to bad local minima, while patch-based Gaussian modeling remains weak on diverse detection and localization tasks.
- Gradient-based interpretability methods estimate pixel importance, with GuidedBackprop filtering negative gradients and SmoothGrad averaging maps under small input perturbations.
3. Method
The method distills multiple intermediate representations from a pretrained source network into a smaller cloner trained only on normal images, then uses source–cloner discrepancies for detection and gradient-based maps for localization.
- 3. Method: The cloner learns the source network’s normal-data manifold by matching representations at multiple critical layers rather than only the final layer.Multiple hints expose several abstraction levels and make the optimization more well-posed.
- 3. Method: Knowledge transfer combines Euclidean activation-value matching with directional similarity between source and cloner activation vectors.Directional similarity matters in ReLU networks because equal Euclidean distances can produce different downstream neuron activations.
- 3. Method: The total loss combines value and directional terms, with λ calibrated so both constituent terms have the same scale and training can fully converge.The scaling is estimated from the initial errors of the untrained network.
- 3. Method: A smaller cloner concentrates on normal-data features, while anomalies produce a discrepancy between source and cloner behavior that is thresholded for detection.The source and cloner both process each test input, but the cloner has learned only the normal point of view.
- 3. Method: Localization uses gradients of the total loss with respect to the input, followed by Gaussian smoothing and morphological opening to produce cleaner anomaly maps.Opening combines erosion and dilation to remove small sporadic noises.
- 3. Method: Ablations report that more distillation layers improve anomaly detection, compact cloners outperform equal-sized ones by approximately 3%, and the combined loss performs best across anomaly types.The combined loss avoids the failures observed when using only directional or Euclidean matching.
- 3. Method: Among tested interpretability methods, SmoothGrad highlights anomalous parts better before filtering, but filtered methods perform almost the same, so simple gradients are used thereafter.Simple gradients avoid SmoothGrad’s additional computational cost.
4. Experiments
Experiments evaluate the method across seven datasets and anomaly detection and localization tasks, reporting stable training and strong performance across industrial, natural-image, and medical domains.
- Evaluation protocol: Training stability is assessed by averaging models sampled every 10 epochs after convergence and reporting variances, while the ImageNet-pre-trained source sees none of the test datasets.The source network is therefore evaluated without exposure to tested-dataset data.
- Datasets: The method is tested on seven datasets spanning handwritten digits, fashion products, natural images, industrial objects and textures, and medical images.MVTecAD contains over 5k high-resolution images across 15 object and texture categories.
- Anomaly detection: AUROC comparisons show SOTA results on MNIST and Fashion-MNIST, with a 13% advantage over SOTA on CIFAR-10 without dataset-dependent fine-tuning.The evaluation compares against generative, self-supervised, and autoencoder-based methods.
- MVTecAD: ∼10%: the method surpasses SOTA on MVTecAD anomaly detection while performing well in both one-class and defect-detection settings.The reported comparison uses AUROC and describes a large margin over other methods.
- Medical datasets: On Retinal-OCT, the method outperforms all other SOTA methods, while medical-dataset experiments also report SOTA-level results and low variance on brain MRI.On HeadCT, generic augmentation brings performance close to LSA despite limited training data.
5. Conclusion
The conclusion presents intermediate-feature distillation from an ImageNet-pre-trained expert into a compact cloner as a framework for anomaly detection and localization. It reports superior results across diverse datasets without intensive region-based procedures.
- 5. Conclusion: Intermediate knowledge from an ImageNet-pre-trained expert is distilled into a compact cloner using anomaly-free data, with discrepancies supporting anomaly detection and localization.The framework uses different behavior between expert and cloner networks on different samples.
- 5. Conclusion: Interpretability methods produce localization maps without intensive region-based training and testing, while the framework achieves superior results across domains far from ImageNet.The conclusion covers multiple tasks and datasets rather than a single benchmark.
Appendix
The appendix reports additional ablation studies across MVTecAD, CIFAR10, and MNIST classes.
- Appendix: Additional ablation studies examine different classes of MVTecAD, CIFAR10, and MNIST.The passage identifies the datasets but does not report specific ablation outcomes.
A. Intermediate Knowledge
The appendix examines how the choice of critical distillation layers affects performance, finding that more intermediate hints improve the framework.
- A. Intermediate Knowledge: Performance improves when more intermediate hints are included in distillation.Using only the last layer performs roughly like a random detector on some MVTecAD classes, with AUC=50%.
B. Distillation Effect (Compact Cloner)
A more compact cloner network outperforms a cloner matching the source network’s size, based on class-detailed comparisons on MVTecAD and CIFAR-10.
- The more compact cloner network outperforms an equal-sized cloner on MVTecAD and CIFAR-10 class-detailed comparisons.The comparison is reported in Tables 9 and 10.
C. Ldir and Lval
Combining directional and MSE losses produces the best performance, indicating that direction-wise activation knowledge complements an MSE-based objective.
- Ltotal, combining directional and MSE losses, achieves the best performance across the reported MVTecAD and CIFAR-10 class comparisons.The results are reported as class-detailed AUROC comparisons.
- The results highlight the positive impact of adding direction-wise activation knowledge to an MSE approach.
D. Localization using Interpretability Methods
SmoothGrad provides the strongest unfiltered anomaly localization among the tested interpretability methods, while Gaussian filtering makes their performance nearly equivalent.
- SmoothGrad highlights anomalous parts better than the other tested interpretability methods before Gaussian filtering.GBP performs weaker than the other methods in the reported MVTecAD localization results.
- After Gaussian filtering, the interpretability methods perform almost the same on MVTecAD pixel-wise AUROC.
- More intermediate distillation layers improve anomaly-detection performance on MVTecAD and MNIST.The improvement is reported in the class-detailed AUROC results for both datasets.
- The combined Ltotal objective performs well with equal-sized and smaller cloners, whereas individual directional or Euclidean losses fail in one case.Smaller cloner networks also perform better in general on MVTecAD and CIFAR-10.