PaperlayerView exact revision on arXiv

Source-linked AI summary

Mean-Shifted Contrastive Loss for Anomaly Detection

Tal Reiss, Yedid Hoshen

arXiv:2106.03844v2cs.CVcs.LG

TL;DR

Small anomaly detection datasets make self-supervised features weak, while outlier exposure can fail when external data do not resemble anomalies. The paper transfers and fine-tunes generic pre-trained representations, analyzes the failure of standard contrastive objectives, and introduces MSC, achieving state-of-the-art performance including 98.6% ROC-AUC on CIFAR-10.

  • Problem

    Small anomaly detection datasets limit self-supervised representation learning, and outlier exposure can fail when external datasets do not resemble the anomalies.

  • Method

    The paper transfers generic pre-trained representations to anomaly detection and introduces the Mean-Shifted Contrastive Loss for fine-tuning them.

  • Results

    The method achieves state-of-the-art anomaly detection performance, including 98.6% ROC-AUC on CIFAR-10.

  • Takeaways & Limitations

    Mean-shifted contrastive adaptation is effective for anomaly detection with pre-trained features.

  • Takeaways & Limitations

    MSC does not improve over standard contrastive losses when training from scratch without strong pre-trained initialization.

Abstract

from arXiv · show

Deep anomaly detection methods learn representations that separate between normal and anomalous images. Although self-supervised representation learning is commonly used, small dataset sizes limit its effectiveness. It was previously shown that utilizing external, generic datasets (e.g. ImageNet classification) can significantly improve anomaly detection performance. One approach is outlier exposure, which fails when the external datasets do not resemble the anomalies. We take the approach of transferring representations pre-trained on external datasets for anomaly detection. Anomaly detection performance can be significantly improved by fine-tuning the pre-trained representations on the normal training images. In this paper, we first demonstrate and analyze that contrastive learning, the most popular self-supervised learning paradigm cannot be naively applied to pre-trained features. The reason is that pre-trained feature initialization causes poor conditioning for standard contrastive objectives, resulting in bad optimization dynamics. Based on our analysis, we provide a modified contrastive objective, the Mean-Shifted Contrastive Loss. Our method is highly effective and achieves a new state-of-the-art anomaly detection performance including $98.6\%$ ROC-AUC on the CIFAR-10 dataset.

1 Introduction

Anomaly detection benefits from effective representations, but small datasets and mismatched external anomalies limit common approaches. This paper transfers generic pre-trained representations and introduces MSC after finding that standard contrastive fine-tuning is poorly suited to them.

  • Motivation: High-dimensional anomaly detection depends on feature quality because density estimators often perform poorly on raw image features.The paper frames representation learning as central to estimating which samples are normal or anomalous.
  • Motivation: Small anomaly detection datasets without labeled anomalies can produce weak self-supervised features.Generic external datasets provide off-the-shelf supervision without additional annotation costs.
  • Motivation: Outlier exposure can fail when external data resemble normal samples less than the true anomalies do.This motivates transferring representations rather than using external data to simulate anomalies.
  • Approach: Transferring generic pre-trained representations and fine-tuning them on normal training data can improve anomaly detection performance.Prior results found even a simple ImageNet-based kNN detector outperformed nearly all self-supervised methods.
  • Contribution: Standard contrastive methods initialized with pre-trained weights do not improve anomaly detection, motivating the Mean-Shifted Contrastive loss.The paper attributes this to poor suitability for normal data concentrated in a compact subspace and reports better OCC performance than center loss.

1. We analyze the standard contrastive loss for fine-tuning

The paper analyzes why standard contrastive fine-tuning fails with pre-trained representations and proposes the Mean-Shifted Contrastive Loss as an alternative objective. It reports state-of-the-art anomaly detection performance, including 98.6% ROC-AUC on CIFAR-10.

  • Analysis: Pre-trained representations for one-class classification are poorly initialized for standard contrastive fine-tuning.The paper analyzes this failure as a central contribution.
  • Method: The Mean-Shifted Contrastive Loss is proposed as an alternative objective for adapting features to anomaly detection.Its importance for feature adaptation is explicitly analyzed.
  • Results: 98.6% ROC-AUC on CIFAR-10 is reported as an example of the method’s state-of-the-art anomaly detection performance.The paper presents this result as evidence from extensive experiments.

2 Related Work

Related work spans classical anomaly detection, self-supervised representation learning, and feature adaptation using externally learned representations. The paper builds on these approaches while focusing on adapting pre-trained features for one-class classification.

  • Classical anomaly detection: Classical anomaly detection uses reconstruction, density estimation, and one-class classification paradigms.Examples include PCA and kNN, Gaussian mixture and kernel density models, and OC-SVM or related methods.
  • Self-supervised learning: Self-supervised deep learning trains representations from unlabeled data using auxiliary tasks with automatically generated labels.Reviewed tasks include frame prediction, colorization, puzzle solving, and rotation prediction.
  • Feature adaptation: Feature adaptation initializes one-class classification models with handcrafted or externally learned representations and tunes them using anomaly-detection objectives.Prior work includes ImageNet pre-training and SVDD-based adaptation.

3 Background: Learning Representations for One-Class Classification

One-class classification learns representations from normal-only training data and scores new samples for anomaly detection. The background contrasts center loss and contrastive objectives while motivating adaptation of pre-trained features.

  • 3.1 Preliminaries: One-class classification trains on normal samples only, learns a representation, and applies an anomaly score threshold to new samples.The representation may be initialized with external or self-supervised pre-trained weights before tuning on training data.
  • 3.2 Self-supervised Objectives for OCC: Self-supervised objectives are reviewed as tools for learning representations in one-class classification.The section introduces the objectives before discussing their behavior with pre-trained features.
  • Center Loss: Center loss pulls normal features toward a predetermined center in feature space.Its objective is Lcenter(x) = ∥φ(x) − c∥2.
  • Center Loss: Center loss can cause catastrophic collapse, making normal and anomalous samples indistinguishable.All representations may collapse to the same center.
  • Contrastive Loss: Contrastive learning brings augmented views of the same image together while separating features of different normal samples.The objective operates on pairs of augmented examples from a mini-batch.
  • Contrastive Loss: Contrastive methods achieve strong OCC performance without externally trained network weights.This provides the motivation for considering contrastive objectives in feature adaptation.
  • Figure 1: Figure 1 tracks pairwise and augmentation cosine similarities across training epochs for contrastive and MSC objectives.The caption distinguishes pairwise image similarity from image–augmentation similarity for the two objectives.
  • Feature adaptation: Fine-tuning pre-trained weights on normal data can strengthen feature extractors, but directly replacing center loss with contrastive loss causes poor OCC results.The paper states that the representation collapses immediately under this modification.

4 Modifying the Contrastive Loss for Anomaly Detection

Standard contrastive learning fails to adapt ImageNet-pretrained features for one-class anomaly detection because normal data occupy a compact feature subspace. Mean-Shifted Contrastive Loss instead measures angles around the normal-feature center, preserving compactness while improving separation between normal and anomalous samples.

  • Adaptation failure: ImageNet-pretrained features degrade quickly when optimized with the standard contrastive loss for one-class anomaly detection.The reported failure occurs despite contrastive learning's strong performance in other visual recognition settings.
  • Adaptation failure: Standard contrastive training improves feature uniformity but fails to increase similarity between augmented views, leaving features insufficiently discriminative.The analysis tracks uniformity and augmentation distance across training epochs.
  • Why standard contrastive learning fails: Normal data occupy a compact region of the pretrained feature space, whereas standard contrastive learning pushes representations toward the entire origin-centered sphere.This mismatch makes optimization focus on spreading features rather than preserving useful pretrained structure.
  • Mean-shifted objective: Mean-Shifted Contrastive Loss computes feature angles relative to the center of normal features rather than the origin.The method mean-shifts normalized representations by subtracting the normal-feature center before applying the contrastive objective.
  • Mean-shifted objective: Unlike standard contrastive learning, the mean-shifted objective preserves distance to the center while maximizing angles between negative pairs.This design avoids increasing the Euclidean distance from the normal-feature center as negative-pair angles grow.
  • Anomaly criterion: For anomaly scoring, cosine similarity is computed between a target image and K selected training exemplars, using either k-nearest-neighbors or k-means selection.A threshold on the resulting anomaly score determines whether the image is normal or anomalous.
  • Empirical behavior: On CIFAR-10 Bird, normal and anomalous angular-distance distributions overlap under standard contrastive learning but not under MSC.The comparison measures angular distance to the center around the origin.

5 Experiments

Experiments evaluate mean-shifted contrastive learning across standard, small-data, cross-domain, architectural, and multi-class anomaly-detection settings. The method improves over prior approaches, while its benefits depend on pretrained initialization and can eventually be affected by catastrophic collapse.

  • Experimental setup: The evaluation covers standard OCC benchmarks, small datasets, and multi-class anomaly detection using common image datasets and pretrained ImageNet features.The experiments use CIFAR-10, CIFAR-100, CatsVsDogs, MVTec, and DIOR, with ResNet152 pretrained on ImageNet as the default feature extractor.
  • Main results: The proposed approach surpasses previous state-of-the-art methods on common one-class classification benchmarks.The comparison includes previous self-supervised and pretrained methods.
  • Small datasets: On small datasets, self-supervised methods perform poorly, whereas transferring pretrained features achieves strong anomaly-detection performance.The paper attributes this difference to the limited sample size available for learning features from scratch.
  • Ablation study: The MSC loss outperforms the evaluated objectives, and combining it with angular center loss produces further improvements.Angular center loss uses angular distance rather than the Euclidean distance of standard center loss.
  • Initialization: The objective does not improve over standard contrastive losses when training from scratch because random initialization removes the pretrained distance prior.The paper characterizes MSC as directed toward anomaly detection from pretrained features.
  • Cross-domain transfer: The method extends to domains substantially different from ImageNet, with results on DIOR and MVTec providing evidence of transfer across distant domains.The paper also relates this finding to earlier evidence from Reiss et al. (2021).
  • Optimization behavior: Pretrained adaptation methods can undergo catastrophic collapse after very long training, although MSC collapses more slowly and remains ahead of PANDA throughout the curve.This establishes early stopping as a relevant practical boundary for the evaluated adaptation setting.
  • Architectures: Mean-shifted contrastive learning generalizes across network architectures and reaches 98.6% ROC-AUC on CIFAR-10.PANDA is reported as sensitive to architecture, while the proposed loss gains performance across architectures.

6 Conclusion

The paper investigates feature adaptation methods for anomaly detection, analyzes standard contrastive loss, and introduces Mean-Shifted Contrastive loss to address its limitations.

  • The study investigates feature adaptation methods for anomaly detection.
  • Standard contrastive loss provides poor initialization for one-class classification feature adaptation.
  • Mean-Shifted Contrastive loss is introduced to overcome standard contrastive loss limitations.
  • Extensive experiments verify the method’s outstanding anomaly detection performance.

A.1 Dataset Descriptions

The evaluation uses standard image datasets, small-domain datasets, and competing-method implementations configured through official repositories or reported results.

  • The standard datasets include CIFAR-10, coarse-grained CIFAR-100, and DogsVs-Cats.DogsVs-Cats is split into 10,000 training and 2,500 test images per class.
  • Small-domain evaluation includes MvTec industrial products with subtle in-class manufacturing-error anomalies and DIOR aerial images.
  • DROC results are taken from its paper, while other competing methods are evaluated using official repositories or selected configurations.
  • DeepSVDD uses its official PyTorch implementation with the CIFAR-10 configuration after resizing images to 32 × 32 pixels.
  • MRot uses the current GitHub implementation for high-resolution images and a modified architecture for low-resolution images.
  • CSI and PANDA are run using the exact protocols described in their official repositories.

A.3 Implementation details

Implementation fine-tunes ImageNet-pre-trained ResNet backbones with normalization and the Mean-Shifted Contrastive objective, using fixed optimization settings and substantial per-class runtime.

  • ResNet152 fine-tunes its two last blocks for 25 epochs, while ResNet18 fine-tunes the full backbone for 20 epochs.
  • Both settings use ImageNet-pre-trained backbones, an additional ℓ2 normalization layer, and Lmsc with temperature τ = 0.25.
  • Training each dataset class takes approximately 3 hours on a single NVIDIA RTX-2080 TI.

A.5 Per-class results

The experiments report per-class results across CIFAR-10, CIFAR-100, and CatsVsDogs, while analyzing objective collapse and temperature effects on CIFAR-10.

  • Per-class results: Per-class results are presented for CIFAR-10, CIFAR-100, and CatsVsDogs.
  • Objective collapse: Contrastive loss is unsuitable for one-class classification feature adaptation because it causes very fast catastrophic collapse.
  • Objective collapse: PANDA-EWC postpones collapse but does not prevent it, whereas PANDA-ES initially improves accuracy before features degrade.
  • Objective collapse: Mean-shifted contrastive loss dominates PANDA in the collapse evaluation.
  • Temperature analysis: The temperature parameter affects sample-distribution uniformity on the hypersphere and the weight assigned to hard negative samples.
  • Temperature analysis: Figure 6 compares pairwise image similarity and temperature ablations for standard versus mean-shifted contrastive loss on CIFAR-10 Airplane.
Loading 2106.03844v2…