PaperlayerView exact revision on arXiv

Source-linked AI summary

PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation

Tal Reiss, Niv Cohen, Liron Bergman, Yedid Hoshen

arXiv:2010.05903v3cs.CVcs.LG

TL;DR

Anomaly detection has relied heavily on self-supervised features despite limited normal data and underused pretrained representations. PANDA combines pretrained features with simple detectors and adapts them using collapse-resistant early stopping and elastic regularization. It substantially outperforms state-of-the-art methods across one-class classification, outlier exposure, and anomaly segmentation settings.

  • Problem

    Anomaly detection needs high-quality features, but pretrained deep features are underused and naive one-class feature adaptation can cause catastrophic collapse.

  • Method

    PANDA combines pretrained features with simple anomaly methods and uses adaptive early stopping and elastic regularization to combat collapse during target-distribution adaptation.

  • Results

    PANDA outperforms the state of the art across OCC, outlier exposure, and anomaly segmentation, reaching 96.2% vs. 90.1% ROCAUC on CIFAR10 without outlier exposure.

  • Takeaways & Limitations

    Pretrained features are preferable for most practical anomaly-detection purposes, and simple baselines can outperform more complex methods.

  • Takeaways & Limitations

    The work requires strong pretrained feature extractors, which are not currently available for all data modalities.

Abstract

from arXiv · show

Anomaly detection methods require high-quality features. In recent years, the anomaly detection community has attempted to obtain better features using advances in deep self-supervised feature learning. Surprisingly, a very promising direction, using pretrained deep features, has been mostly overlooked. In this paper, we first empirically establish the perhaps expected, but unreported result, that combining pretrained features with simple anomaly detection and segmentation methods convincingly outperforms, much more complex, state-of-the-art methods. In order to obtain further performance gains in anomaly detection, we adapt pretrained features to the target distribution. Although transfer learning methods are well established in multi-class classification problems, the one-class classification (OCC) setting is not as well explored. It turns out that naive adaptation methods, which typically work well in supervised learning, often result in catastrophic collapse (feature deterioration) and reduce performance in OCC settings. A popular OCC method, DeepSVDD, advocates using specialized architectures, but this limits the adaptation performance gain. We propose two methods for combating collapse: i) a variant of early stopping that dynamically learns the stopping iteration ii) elastic regularization inspired by continual learning. Our method, PANDA, outperforms the state-of-the-art in the OCC, outlier exposure and anomaly segmentation settings by large margins.

1. Introduction

PANDA shows that pretrained features combined with simple anomaly methods outperform more complex approaches, while feature adaptation for one-class classification requires explicit safeguards against collapse.

  • Motivation: Limited normal training data restricts representation quality, motivating the use of features pretrained on external datasets.Prior methods largely rely on self-supervised learning with limited normal data, partly because external features may not generalize.
  • Simple pretrained-feature baselines: Pretrained features with k-nearest-neighbor retrieval significantly outperform previous methods on anomaly detection and segmentation, including distant target domains.The paper presents simple baselines using large externally pretrained features rather than complex task-specific architectures.
  • Feature adaptation: Feature adaptation in one-class classification often causes catastrophic collapse, mapping normal and anomalous samples to the same point and preventing separability.DeepSVDD avoids collapse through architectural constraints, but these restrict network expressivity and the pretrained models available off-the-shelf.
  • Results: 96.2% vs. 90.1% ROCAUC on CIFAR10 without outlier exposure demonstrates PANDA's improvement over the state of the art.With outlier exposure, CIFAR10 reaches 98.9% vs. 95.6%; anomaly segmentation on MVTec reaches 96.0% vs. 89.0%.
  • PANDA contributions: PANDA combats collapse with adaptive early stopping and elastic regularization, while simple constant-duration early stopping already achieves top performance.The adaptive method selects stopping iterations per sample, and elastic regularization postpones collapse.
  • Critical analyses: Pretrained features strictly dominate RotNet-based self-supervised features, while DeepSVDD adaptation does not improve over simple data whitening.The authors conclude that pretrained features are preferable for most practical purposes.

2. A General Framework and Simple Baselines for Anomaly Detection and Segmentation

The framework initializes features, optionally adapts them on normal target data, and then scores samples by local feature-space normality; simple baselines skip adaptation.

  • Three-stage framework: The framework has three stages: initialize a feature extractor, adapt it to normal target data, and learn an anomaly-scoring function.The scoring stage uses extracted training features to estimate the density of normal data around a test sample.
  • Feature initialization: The initial extractor may come from external-dataset pretraining or self-supervised learning such as auto-encoding, rotation, or jigsaw prediction.External pretrained extractors can be used off-the-shelf.
  • Anomaly scoring: Anomaly scores are high for test samples located in low-density regions of the adapted feature space.The scoring function measures the density of normal data around each test representation.
  • Simple baselines: The simple baselines skip feature adaptation and use pretrained representations directly for anomaly detection or segmentation.Both methods are designed to be simple to implement, with a K-Means-based speedup described in the appendix.
  • Deep Nearest Neighbors: Deep Nearest Neighbors uses final-pooling-layer ResNet features and assigns each test image the average distance to its k nearest normal images.The ResNet is pretrained on ImageNet.
  • SPADE: SPADE extracts per-pixel features from the first three ResNet blocks and concatenates descriptors across layers to combine low-level and semantic information.The method uses an ImageNet-pretrained ResNet and associates each pixel with spatial feature-map descriptors.

3. Feature Adaptation for Anomaly Detection

PANDA adapts pretrained features for one-class anomaly detection while addressing catastrophic collapse. It contrasts compactness-based adaptation, joint optimization, and early-stopping or regularization strategies that preserve anomaly separation.

  • Pretrained-feature baselines with kNN substantially outperform previous anomaly detection and segmentation methods, motivating feature adaptation.The approach uses externally trained features and simple retrieval rather than relying only on limited normal training data.
  • Existing feature-adaptation methods: Compactness adaptation minimizes distances between training features and a center c, aiming to make normal features more compact.DeepSVDD uses c, typically the average pretrained feature, as the reference center.
  • Existing feature-adaptation methods: DeepSVDD avoids the trivial mapping ψ(x)=c by removing layer biases, but its adaptation does not outperform simple feature whitening.The architectural restriction also limits network expressivity and the pretrained models usable off the shelf.
  • Existing feature-adaptation methods: Joint optimization combines compactness with the original ImageNet classification loss, requiring retention of roughly 50k labelled pretraining images.The method weights the two losses with α, but jointly training both tasks may reduce anomaly-detection accuracy.
  • PANDA: PANDA controls collapse through constant early stopping, sample-wise early stopping, and elastic weight consolidation inspired by continual learning.Sample-wise early stopping selects among checkpoints using the largest ratio of target-to-training-center distance, while EWC regularizes pretrained weights according to Fisher information.
  • PANDA: EWC preserves anomaly separation during adaptation: anomalous samples can remain relatively farther from the normal center even when both normal and anomalous distances decrease.Figure 3 reports that EWC mitigates collapse, and anomaly-scoring experiments use kNN unless otherwise stated.

4. Image Anomaly Detection

Across image anomaly detection and segmentation, pretrained features with simple methods outperform self-supervised and state-of-the-art alternatives, while adaptation and suitable outlier exposure can provide further gains.

  • High-Level Results: Pretrained features significantly outperform self-supervised features across all evaluated datasets for both anomaly detection and segmentation.The evaluation spans datasets with varied sizes, domains, resolutions, and symmetries.
  • Feature Adaptation: Feature adaptation significantly improves performance on larger datasets, but unregularized adaptation can peak higher before collapsing.PANDA-EWC postpones collapse, while PANDA-SES is especially useful when unregularized training fails.
  • Outlier Exposure: Outlier exposure achieves near-perfect performance on CIFAR10/100 when its outliers resemble anomalies, but hurts Fashion MNIST and CatsVsDogs.The difference is attributed to the similarity of the external 80M Tiny Images data to anomalies versus normal data.
  • High-Level Results: Simple pretrained-feature baselines using kNN outperform previous anomaly detection and segmentation methods, including on distant target domains.The raw deep ResNet features without adaptation also outperform all compared methods on Fashion MNIST and DIOR.
  • Anomaly Segmentation: SPADE, a simple no-training baseline, outperforms previous anomaly segmentation methods, including methods using trained or pretrained features.The authors found MVTec too small for reliable feature adaptation in this setting.
  • Efficiency: kNN improves anomaly detection over distance to the center by around 2% on average, while 50-mean approximation raises segmentation speed from 2.7 to 41 frames per second.The approximation incurs an approximately 0.5% ROCAUC decrease.

5. Conclusion and Outlook

PANDA combines strong pretrained-feature baselines with feature adaptation that mitigates catastrophic collapse and outperforms current methods. Its main boundary is dependence on strong pretrained extractors, which are unavailable for some modalities.

  • Conclusion: Simple pretrained-feature baselines outperform the state of the art for image anomaly detection and segmentation.The paper further improves these baselines by adapting pretrained features while mitigating catastrophic collapse.
  • Conclusion: PANDA adapts pretrained features and mitigates catastrophic collapse, addressing limitations of existing anomaly-detection methods.The paper leaves optimal image anomaly-detection performance without any auxiliary data unaddressed.
  • Outlook: The main limitation is the requirement for strong pretrained feature extractors.Generic extractors are not currently available for all data modalities, including some time-series and audio settings.

A. Pretrained Features, RotNet Auxiliary Tasks and Generalization

RotNet auxiliary-task features can generalize better while weakening anomaly discrimination, because anomalous samples also perform better on the auxiliary task. Pretrained features instead outperform self-supervised alternatives, including on distant domains.

  • RotNet Auxiliary Tasks: RotNet uses rotation-prediction accuracy as an anomaly signal after training on normal samples.The method assumes anomalous images have worse rotation-prediction accuracy than normal images.
  • RotNet Auxiliary Tasks: Better rotation-classification performance does not necessarily improve anomaly detection because generalizable features also improve anomalous samples.This reduces the normal–anomalous auxiliary-task gap and can degrade anomaly detection.
  • Analysis: The mutual-information analysis explains why class-specific traits may contribute little to rotation prediction when general traits already determine the rotation.Under that condition, the predicted rotation carries limited additional class information.
  • Comparison: Using RotNet features for transfer learning performs worse than both MHRot and PANDA, while an ensemble of rotations reaches comparable MHRot performance.The reported MHRot re-implementation achieves 89.7%.

B.1. PANDA

PANDA implementation adapts the final blocks of an ImageNet-pretrained ResNet152 and compares early stopping, elastic regularization, and outlier exposure variants using kNN-based scoring.

  • Optimization: PANDA finetunes the two final blocks of an ImageNet-pretrained ResNet152 with SGD, weight decay 5 · 10^-5, momentum 0.9, and gradient clipping 10^-3.Training duration is normalized using a constant number of minibatches containing 32 samples each.
  • Early Stopping: PANDA early stopping uses a 2.3k-minibatch checkpoint, while sample-wise early stopping evaluates checkpoints through 150k image samples.Network copies are saved every five epochs.
  • Anomaly Scoring: Unless otherwise specified, anomaly scores use kNN with k = 2 nearest neighbours.PANDA-SES additionally normalizes feature sets using typical kNN distances from normal training features.
  • Outlier Exposure: PANDA outlier exposure finetunes the final block using 48k synthetic outliers from 80 Million Tiny Images after removing CIFAR10/100 images.The procedure uses 75 epochs, learning rate 0.1, gradient clipping 10^-3, momentum 0.9, and no weight decay.

B.2. Anomaly Detection Baselines

The baselines use pretrained Wide-ResNet50 × 2 features with nearest-neighbor retrieval, multilayer feature combinations, and Gaussian-smoothed pixel scores. Comparisons include OC-SVM, DeepSVDD, MHRot, and outlier exposure.

  • OC-SVM uses an RBF kernel, with ν and γ optimized to maximize ROCAUC.
  • DeepSVDD experiments resize images to 32 × 32 pixels and use the official PyTorch CIFAR10 configuration.
  • MHRot comparisons use the published method, with implementations adapted for high- and low-resolution images.
  • The feature extractor is a Wide-ResNet50 × 2 pretrained on ImageNet.
  • Feature extraction: Features are drawn from ResNet blocks at 56 × 56, 28 × 28, and 14 × 14 resolutions with equal weights.
  • Feature combination: The ensemble combines per-layer pixel anomaly scores and is more robust than concatenating resampled per-pixel features.

C. Datasets

The evaluation spans standard, small, and symmetry-containing datasets, with MVTec also supporting layer-selection comparisons for anomaly segmentation.

  • Standard datasets: Standard datasets include CIFAR10, Fashion MNIST, CIFAR100, and DogsVsCats.
  • Segmentation evaluation: Table 8 evaluates MVTec anomaly segmentation accuracy using different ResNet layers and reports PRO percentages.
  • Standard datasets: DogsVsCats uses the ASIRRA data, with the first 10,000 images per class for training and the last 2,500 for testing.
  • Small datasets: Small-dataset evaluations include 102 Category Flowers, Caltech-UCSD Birds 200, and MVTec across varied domains and manufacturing errors.
  • Symmetric datasets: Symmetric-dataset evaluations use microscopy white-blood-cell images and DIOR aerial images.

D. Choosing the Layers to Finetune

Finetuning later ResNet blocks preserves performance better than finetuning all layers, which is prone to feature collapse even with continual learning.

  • Collapse: Finetuning all layers is prone to feature collapse, even with continual learning.
  • Block selection: Finetuning blocks 3 and 4, or blocks 2, 3, and 4, produces similar performance.
  • Block selection: Finetuning only block 4 reaches 94.8 ROC AUC, compared with 94.6 for whitening.
  • Recommendation: The paper recommends finetuning blocks 3 and 4 because block-4-only finetuning performs similarly to linear whitening.

E. SPADE: Detailed Results

SPADE is evaluated with per-pixel ROCAUC and per-region PRO, and the reported results show significant improvement over baselines on both metrics. Qualitative examples illustrate detected anomalous pixels.

  • Metrics: Per-pixel ROCAUC is computed by scanning thresholds over anomaly scores for pixels across all test images.
  • Metrics: PRO addresses ROCAUC’s bias toward large anomalies by measuring connected-region pixel coverage across false-positive rates from 0 to 0.3.
  • Baselines: Table 10 compares Deep SVDD with PCA whitening for anomaly-detection performance using ROC AUC percentages.
  • Quantitative results: SPADE significantly outperforms baselines in both ROCAUC and PRO according to Tables 11 and 12.
  • Qualitative results: Figure 5 shows anomalous images, normal training images, detected masks, and predicted anomalous pixels for flowers and birds.
Loading 2010.05903v3…