PaperlayerView exact revision on arXiv

Source-linked AI summary

Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detection

Jia Guo, Shuai Lu, Weihang Zhang, Fang Chen, Huiqi Li, Hongen Liao

arXiv:2405.14325v5cs.CV

TL;DR

Multi-class UAD seeks a unified model, but its performance remains behind class-separated methods because diverse normal patterns can encourage identity mapping or over-generalization. Dinomaly uses a minimal reconstruction framework built from pure Transformer components and four simple design elements. It achieves state-of-the-art results across MVTec AD, VisA, and Real-IAD, with further gains from scaling.

  • Problem

    Unified multi-class UAD remains substantially behind class-separated models, while diverse multi-class normal patterns make identity mapping and over-generalization difficult to avoid.

  • Method

    Dinomaly is a minimalist reconstruction-based UAD framework using pure Transformer blocks, foundation features, a noisy MLP bottleneck, Linear Attention, and loose reconstruction.

  • Results

    99.6%, 98.7%, and 89.3% image-level AUROC are achieved on MVTec AD, VisA, and Real-IAD, respectively, with base-size Dinomaly.

  • Takeaways & Limitations

    Dinomaly demonstrates that a unified model can match or exceed recent class-separated UAD records without severe performance degradation across complicated multi-class scenarios.

  • Takeaways & Limitations

    Dinomaly targets sensory anomaly detection and is not suitable for semantic anomaly detection.

Abstract

from arXiv · show

Recent studies highlighted a practical setting of unsupervised anomaly detection (UAD) that builds a unified model for multi-class images. Despite various advancements addressing this challenging task, the detection performance under the multi-class setting still lags far behind state-of-the-art class-separated models. Our research aims to bridge this substantial performance gap. In this paper, we introduce a minimalistic reconstruction-based anomaly detection framework, namely Dinomaly, which leverages pure Transformer architectures without relying on complex designs, additional modules, or specialized tricks. Given this powerful framework consisted of only Attentions and MLPs, we found four simple components that are essential to multi-class anomaly detection: (1) Foundation Transformers that extracts universal and discriminative features, (2) Noisy Bottleneck where pre-existing Dropouts do all the noise injection tricks, (3) Linear Attention that naturally cannot focus, and (4) Loose Reconstruction that does not force layer-to-layer and point-by-point reconstruction. Extensive experiments are conducted across popular anomaly detection benchmarks including MVTec-AD, VisA, and Real-IAD. Our proposed Dinomaly achieves impressive image-level AUROC of 99.6%, 98.7%, and 89.3% on the three datasets respectively, which is not only superior to state-of-the-art multi-class UAD methods, but also achieves the most advanced class-separated UAD records.

1. Introduction

Dinomaly addresses the performance gap between unified multi-class anomaly detection and class-separated models with a minimalist Transformer reconstruction framework. It combines simple mechanisms to prevent identity mapping and achieves strong results across three benchmarks.

  • Motivation: Unified multi-class anomaly detection reduces the storage burden of one-model-per-category systems but makes normal-pattern distributions harder to model.Diverse intra-normal patterns can harm detection performance when represented in one unified model.
  • Motivation: A substantial performance gap remains between state-of-the-art multi-class and class-separated anomaly detection methods.Earlier approaches also rely on delicately designed modules and architectures with limited universality and ease of use.
  • Approach: Dinomaly uses pure Transformer blocks and introduces Dropout-based noise, Linear Attention, and loose reconstruction to prevent identical information from being reconstructed.The framework relies on Self-Attentions and MLPs without increasing complexity or computational burden.
  • Results: 99.6%, 98.7%, and 89.3% image-level AUROC are achieved on MVTec AD, VisA, and Real-IAD, respectively, using base-size Dinomaly.These results surpass previous state-of-the-art multi-class methods by a large margin.
  • Results: 99.8%, 98.9%, and 90.1% are reached after scaling model size on MVTec AD, VisA, and Real-IAD, respectively.Scaling down parameters and input size can provide more efficient solutions for computation-constrained scenarios.

2. Related Work

Multi-class UAD research developed unified approaches to avoid training separate models, but commonly faces identical shortcuts that allow both normal and anomalous samples to be recovered. Recent work addresses this issue with attention masking, synthetic anomalies, diffusion, cross-reconstruction, and Transformer-based methods.

  • Multi-Class UAD: UniAD introduced multi-class anomaly detection with a unified model for different object classes.The unified setting differs from conventional class-specific UAD models.
  • Multi-Class UAD: Multi-class UAD commonly suffers from identical shortcuts, allowing both anomaly-free and anomalous samples to be effectively recovered during inference.Prior work attributes this phenomenon to diverse normal patterns encouraging generalization to unseen patterns.
  • Prior Approaches: Recent methods address multi-class UAD with neighbor-masked attention, synthetic anomalies, vector quantization, diffusion models, and state space models.These approaches represent several architectural and training strategies for mitigating unified-model weaknesses.
  • Prior Approaches: ViTAD develops a unified feature-reconstruction framework with Transformer building blocks, while ReContrast uses cross-reconstruction between two encoders.LafitE and DiAD use diffusion models, whereas OmniAL uses synthesized pseudo anomalies for anomaly localization.

3. Method

Dinomaly is a reconstruction-based multi-class UAD framework built from Transformer components, using unfocused attention and loosened reconstruction constraints to avoid reproducing anomalous patterns.

  • Dinomaly Framework: Dinomaly uses a pretrained ViT encoder, an MLP bottleneck, and a Transformer decoder to reconstruct encoder features.The decoder is trained with cosine similarity and is expected to reconstruct normal regions but fail on anomalous ones.
  • Foundation Transformers: Foundation ViTs provide universal, discriminative features, and Dinomaly systematically evaluates their pretraining strategies, sizes, and input resolutions.The paper investigates scaling behavior in UAD models using foundational Vision Transformers.
  • Unfocused Linear Attention: Softmax-free Linear Attention is selected because its inability to focus helps prevent attention from attending to identical information.The paper contrasts this property with the identity-mapping risk of attention or convolutional operations that over-concentrate on corresponding input locations.
  • Unfocused Linear Attention: Linear Attention reduces complexity from O(N^2d) to O(Nd^2) and spreads attention across the image instead of focusing on exact query regions.This unfocused behavior supplies more long-range information for feature restoration and reduces identical reconstruction of unseen patterns.
  • Loose Reconstruction: Loose reconstruction groups feature maps across encoder layers rather than enforcing strict layer-to-layer supervision.The authors argue that stronger layer-to-layer supervision can make the decoder mimic the encoder too closely, harming discrepancy-based anomaly detection.
  • Loose Reconstruction: Loose Loss discards or downweights well-restored feature points during training using hard-mining global cosine loss.Points with smaller cosine distance receive a one-tenth gradient scale, while total loss averages the resulting losses across feature pairs.

4. Experiments

Dinomaly is evaluated across multi-class anomaly-detection benchmarks, compared with existing methods, and tested through ablations of its components, model sizes, input sizes, and ViT foundations. It achieves strong results across datasets while benefiting from scaling and foundation-model quality.

  • Experimental Settings: MVTec-AD, VisA, and Real-IAD provide multi-class evaluation settings with 15, 12, and 30 object classes, respectively.The datasets contain normal-only training data and test sets with normal and anomalous images.
  • Comparison to Multi-Class UAD SoTAs: Dinomaly surpasses compared methods across datasets and metrics, achieving image-level AUROC values of 99.6/98.7/89.3 on MVTec-AD, VisA, and Real-IAD.The reported image-level results are accompanied by pixel-level metrics and improvements over previous SoTAs.
  • Comparison to Class-Separated UAD SoTAs: Dinomaly remains comparable to class-separated SoTAs, with nearly no performance drop on MVTec-AD and VisA and a moderate drop on Real-IAD.The Real-IAD setting contains more classes and views than the other evaluated datasets.
  • Ablation Study: Noisy Bottleneck and Loose Loss directly improve performance, while Linear Attention and Loose Constraint provide additional performance gains in ablations.The baseline uses a noiseless MLP bottleneck, Softmax Attention, dense layer-to-layer supervision, and global cosine loss.
  • Model Scalability: Scaling Dinomaly's model size improves performance, and increasing the training schedule can improve results without increasing inference costs.ViT-Small already produces state-of-the-art results, while ViT-Large reaches a higher record.
  • Input Scalability: Increasing input size benefits Dinomaly for anomaly localization, whereas compared methods degrade when their input size increases.Dinomaly retains state-of-the-art performance with smaller images while benefiting from larger inputs for localization.
  • ViT Foundations: Dinomaly is robust to most pretrained ViT foundations, with almost all evaluated backbones producing image-level AUROC above 98%.Anomaly-detection performance is strongly correlated with ImageNet linear-probing accuracy, while MAE is a notable exception without fine-tuning.

5. Conclusion

Dinomaly addresses underperforming multi-class UAD with a minimalist Transformer framework built around four elements. Experiments show superiority over unified multi-class models and recent class-separated models across challenging benchmarks.

  • Conclusion: Dinomaly combines Foundation Transformer, Noisy MLP Bottleneck, Linear Attention, and Loose Reconstruction without additional modules or specialized tricks.These elements are presented as the key components for improving multi-class UAD performance.
  • Conclusion: Experiments on MVTec-AD, VisA, and Real-IAD show Dinomaly outperforming previous unified multi-class models and recent class-separated models.The results support implementing a unified model without severe performance degradation in complicated scenarios.

A. Additional Related Work

The related work covers reconstruction, pseudo-anomaly, feature-statistics, and sensory-versus-semantic approaches to unsupervised anomaly detection. These methods differ in how they model anomalies, normal features, and application scope.

  • Reconstruction Methods: Pixel-reconstruction methods assume networks reconstruct normal regions well but reconstruct anomalous regions poorly.Examples include auto-encoders, variational auto-encoders, and generative adversarial networks.
  • Pseudo-Anomaly Methods: Pseudo-anomaly methods synthesize defects on normal images to convert UAD into supervised classification or segmentation tasks.CutPaste, DRAEM, and related methods use handcrafted perturbations or masks to imitate anomalies.
  • Feature-Statistics Methods: Feature-statistics methods memorize or model normal features and match test samples against them, increasing computation during training and inference.The computational burden is especially pronounced for large training sets.
  • Scope of Application: The paper focuses on sensory anomaly detection, where normal and anomalous samples are the same objects except for regional or structural anomalies.Semantic anomaly detection instead compares samples from semantically different classes, such as animals and vehicles.

B. Full Implementation Details

The implementation uses a DINOv2-R ViT encoder, dropout-based bottleneck noise, grouped loose reconstruction, and high-resolution inputs. Training schedules and optimization settings vary across datasets and evaluation settings.

  • Architecture: The reconstruction uses middle layers of 12-layer ViT models, selected layers of the 24-layer ViT-Large, and an eight-layer decoder.The selected encoder layers feed the bottleneck and reconstruction process.
  • Optimization: StableAdamW with AMSGrad trains the model for 10,000 iterations on MVTec-AD and VisA, 50,000 on Real-IAD, and 5,000 per class in class-separated UAD.The learning rate warms up from 0 to 2e-3 and cosine-anneals to 2e-4.
  • Comparison Sources: Most compared multi-class SoTA results are drawn directly from the ADer benchmark paper.The implementation details acknowledge ADer as the source of most comparison results.
  • Additional Settings: Foundation comparisons use ViT-Base models with different patch sizes and evaluate a resize-then-center-crop input scheme.Table A1 compares pretrained foundations, while Table A2 studies input-size ablations on MVTec-AD.

C. Additional Ablation and Experiment

Additional experiments show that Dinomaly’s performance depends on strong pretrained foundations, suitable input resolution, and its proposed reconstruction components, while remaining scalable and robust across settings.

  • Pre-Trained Foundations: Nearly all evaluated foundation models achieve SoTA-level results within Dinomaly, with image-level AUROC higher than 98%.Combined contrastive-learning and masked-image-modeling foundations generally outperform single-strategy models, while MAE produces the worst results.
  • Input Size: Larger input sizes benefit anomaly localization, although smaller inputs are sufficient for image-level anomaly detection.The method remains robust to input size, but center-crop preprocessing can make edge anomalies unreachable.
  • Ablations: Ablations on VisA confirm that Dinomaly’s proposed components contribute to anomaly-detection performance beyond MVTec-AD.The evaluated components include Noisy Bottleneck, Linear Attention, Loose Constraint, and Loose Loss.
  • Noise Bottleneck: Dropout is robust across rates and provides a more robust, elegant noise injector than feature jitter without introducing new modules.Both Dropout and Feature Jitter work in the Noisy Bottleneck, but Dropout is less sensitive to the noisy-scale hyperparameter.
  • Reconstruction Constraint: Group-to-group reconstruction outperforms layer-to-layer supervision, while mixing low- and high-level features harms anomaly localization.One-group reconstruction performs similarly to two-group reconstruction on image-level metrics when all layers are added.
  • Attention vs. Convolution: Attentions outperform convolutions as decoder spatial mixers, especially for anomaly localization, while convolutions still produce SoTA results.This comparison supports both the effectiveness and universality of the proposed framework.
  • Scalability and Robustness: Dinomaly remains scalable across model sizes and input sizes, and repeated runs show robustness to random seeds.Its components can also be adapted to CNN-based RD4AD, where Noisy Bottleneck and Loose Loss improve performance.

D. Additional Dataset

Dinomaly generalizes beyond its primary benchmarks to additional multi-class anomaly-detection datasets, including industrial and medical settings, though Uni-Medical has a stated suitability caveat.

  • Additional Datasets: Dinomaly is evaluated under multi-class UAD on MPDD, BTAD, and Uni-Medical to test generalization across additional datasets.MPDD covers six industrial categories, BTAD covers three industrial products, and Uni-Medical covers brain CT, liver CT, and retinal OCT.
  • Results: Dinomaly demonstrates superior performance against previous SoTAs on MPDD and BTAD under multi-class UAD.The reported comparison is presented in Table A13.

E. Results Per-Category

The appendix reports per-class image-level and pixel-level results for MVTec-AD, VisA, and Real-IAD, alongside randomly selected anomaly-map visualizations.

  • Pixel-Level Results: Per-class pixel-level anomaly-localization results are reported for MVTec-AD, VisA, and Real-IAD using AUROC, AP, F1-max, and AUPRO.These results are provided in Tables A15, A17, and A19.
  • Visualizations: Anomaly maps are visualized on all three datasets using randomly chosen samples without artificial selection.The visualizations appear in Figures A1–A3.

G. Limitation

The stated limitations concern computational efficiency, task scope, and excluded UAD settings. Dinomaly targets sensory anomaly detection rather than semantic anomaly detection, and several other settings remain unevaluated.

  • Computational Efficiency: Transformer computation can hinder inference speed in low-computation scenarios.The paper suggests distillation, pruning, and hardware-friendly attention as future efficiency directions.
  • Task Scope: Dinomaly is designed for sensory anomaly detection and is not suitable for semantic anomaly detection.The paper notes that methods designed for sensory and semantic anomaly detection generally do not transfer competitively between these task types.
  • Excluded Settings: Zero-shot UAD, few-shot UAD, and UAD with noisy training sets are not included in the work.These settings are identified as directions outside the reported evaluation scope.
Loading 2405.14325v5…