PaperlayerView exact revision on arXiv

Source-linked AI summary

AnomalyDiffusion: Few-Shot Anomaly Image Generation with Diffusion Model

Teng Hu, Jiangning Zhang, Ran Yi, Yuzhen Du, Xu Chen, Liang Liu, Yabiao Wang, Chengjie Wang

arXiv:2312.05767v2cs.CV

TL;DR

Industrial anomaly inspection is limited by scarce anomaly data, and existing generators often lack authenticity or accurate mask alignment. AnomalyDiffusion uses few-shot latent diffusion with disentangled spatial conditioning and adaptive attention re-weighting; experiments report improved authenticity, diversity, and downstream inspection performance.

  • Problem

    Scarce anomaly data limits industrial anomaly inspection, while existing generators struggle with authentic synthesis or accurate image-mask alignment.

  • Method

    AnomalyDiffusion uses a pretrained latent diffusion model with Spatial Anomaly Embedding and Adaptive Attention Re-weighting for few-shot anomaly generation.

  • Results

    AnomalyDiffusion outperforms state-of-the-art anomaly-generation models in authenticity and diversity and achieves 99.1% pixel-level AUROC and 81.4% AP for MVTec localization.

  • Takeaways & Limitations

    Generated anomaly data effectively improves downstream anomaly inspection tasks, including anomaly localization.

Abstract

from arXiv · show

Anomaly inspection plays an important role in industrial manufacture. Existing anomaly inspection methods are limited in their performance due to insufficient anomaly data. Although anomaly generation methods have been proposed to augment the anomaly data, they either suffer from poor generation authenticity or inaccurate alignment between the generated anomalies and masks. To address the above problems, we propose AnomalyDiffusion, a novel diffusion-based few-shot anomaly generation model, which utilizes the strong prior information of latent diffusion model learned from large-scale dataset to enhance the generation authenticity under few-shot training data. Firstly, we propose Spatial Anomaly Embedding, which consists of a learnable anomaly embedding and a spatial embedding encoded from an anomaly mask, disentangling the anomaly information into anomaly appearance and location information. Moreover, to improve the alignment between the generated anomalies and the anomaly masks, we introduce a novel Adaptive Attention Re-weighting Mechanism. Based on the disparities between the generated anomaly image and normal sample, it dynamically guides the model to focus more on the areas with less noticeable generated anomalies, enabling generation of accurately-matched anomalous image-mask pairs. Extensive experiments demonstrate that our model significantly outperforms the state-of-the-art methods in generation authenticity and diversity, and effectively improves the performance of downstream anomaly inspection tasks. The code and data are available in https://github.com/sjtuplayer/anomalydiffusion.

1 Introduction

Industrial anomaly inspection is constrained by scarce anomaly samples, while existing generation methods struggle with authenticity or mask alignment. AnomalyDiffusion addresses these issues with few-shot diffusion-based generation and improves downstream inspection performance.

  • Motivation: Scarce anomaly samples make industrial anomaly inspection challenging, and existing approaches mainly use unsupervised or few-shot supervised learning.These methods perform well for detection but have limited localization performance and cannot handle classification.
  • Existing limitations: Model-free methods produce synthetic anomalies with poor authenticity, whereas GAN-based methods generally require many anomaly samples.The few-shot DFMGAN instead produces anomalies that are not accurately aligned with anomaly masks.
  • Proposed approach: AnomalyDiffusion uses a pretrained latent diffusion model to generate anomalies on normal samples with specified masks from few-shot anomaly data.Its Spatial Anomaly Embedding separates anomaly appearance from location, enabling anomalous image-mask pair generation at desired positions.
  • Proposed approach: Adaptive Attention Re-weighting allocates more attention to less noticeable generated-anomaly regions, improving alignment between anomalies and masks.The mechanism dynamically adjusts cross-attention maps during denoising.
  • Results: Extensive experiments show higher generation authenticity and diversity than state-of-the-art anomaly generation models, with effective gains in downstream inspection.On MVTec localization, the reported scores are 99.1% pixel-level AUROC and 81.4% AP.

2 Related Work

Prior anomaly-generation work includes latent diffusion and anomaly-inspection methods, but existing generators face data, authenticity, mask-generation, or alignment limitations. AnomalyDiffusion combines latent diffusion with spatial anomaly conditioning and adaptive attention re-weighting.

  • Generative models: Latent diffusion models improve image generation by compressing diffusion space and learning strong prior information from large-scale datasets.The paper positions LDMs as the generative foundation for AnomalyDiffusion.
  • Anomaly inspection: Anomaly inspection comprises anomaly detection, localization, and classification, with prior methods using reconstruction or deep feature modeling.These approaches form major categories of existing inspection methods discussed by the paper.
  • AnomalyDiffusion: AnomalyDiffusion uses spatial anomaly embedding and adaptive attention re-weighting to control anomaly location and improve image-mask alignment.The framework separates appearance and location information while focusing denoising attention on less noticeable anomaly regions.
  • Anomaly generation: Existing GAN-based generators require substantial anomaly data, while DFMGAN lacks authentic generation and accurate alignment with anomaly masks.SDGAN and Defect-GAN also cannot generate anomaly masks according to the cited related-work passage.

3 Method

AnomalyDiffusion generates anomaly images from normal samples and masks using latent diffusion, separating anomaly appearance from location and adaptively improving mask alignment. It also learns to generate additional anomaly masks and achieves high-quality, diverse outputs on MVTec.

  • The model takes an anomaly-free sample and anomaly mask as inputs, generating anomalies within the mask while preserving the remaining region.
  • Spatial Anomaly Embedding: Spatial Anomaly Embedding combines a learned anomaly embedding for appearance with a mask-encoded spatial embedding for location.The combined embedding conditions the diffusion model.
  • Diffusion Model: The diffusion model uses a forward noise process and an iterative backward denoising process to generate conditioned anomaly images.The forward process adds noise, while the backward process predicts Gaussian means and variances.
  • Spatial Anomaly Embedding: Masked textual inversion learns anomaly appearance embeddings from anomaly regions rather than entire images, while a shared encoder maps masks into spatial embeddings.The spatial encoder uses ResNet-50 features, feature-pyramid fusion, and fully connected networks to produce textual tokens.
  • Adaptive Attention Re-Weighting: Adaptive Attention Re-weighting computes mask-region weights from pixel differences between generated and normal images, emphasizing less noticeable anomalies during denoising.The mechanism addresses incomplete filling of irregular or multiple-region masks and improves alignment.
  • Experiments: On MVTec, the model is reported to achieve the best IS and IC-LPIPS, indicating high-quality and diverse generated anomaly data.
  • Mask Generation: Textual inversion learns mask embeddings that generate extensive anomaly masks for each anomaly type, increasing mask availability for training.

4 Experiments

Experiments evaluate AnomalyDiffusion for anomaly generation and downstream inspection on MVTec, including quality, diversity, detection, localization, classification, and component ablations. The model produces authentic, diverse, mask-aligned anomalies and improves downstream localization, while removing proposed components reduces localization performance.

  • Experiment settings: The MVTec experiments compare anomaly generation quality, diversity, and downstream inspection performance against multiple baseline groups.Baselines include models generating anomalous image-mask pairs and models generating specific anomaly types.
  • Generation quality: AnomalyDiffusion outperforms state-of-the-art anomaly generation models in generation authenticity and diversity.The comparison evaluates IS and IC-LPIPS using generated anomaly images for each anomaly category.
  • Generation quality: AnomalyDiffusion generates high-quality, authentic anomalies that accurately align with their corresponding masks, unlike several competing methods.DFMGAN sometimes fails to produce authentic anomalies and maintain mask alignment, while other baselines show poor or confusing outputs.
  • Downstream inspection: 81.4% AP and 99.1% AUROC are achieved for pixel-level anomaly localization with a simple U-Net trained on generated anomaly data.The result is reported against existing anomaly detection methods and indicates strong downstream localization performance.
  • Ablation study: Removing any of SAE, masked diffusion loss, or AAR causes a noticeable decline in anomaly localization performance.The ablation study trains U-Nets on generated anomalous image-mask pairs and compares pixel-level localization results.

5 Conclusion

The paper concludes that AnomalyDiffusion generates anomalous image-mask pairs by separating anomaly appearance from location and using adaptive attention to improve alignment. Experiments report better generation and downstream inspection performance, while future work targets higher-resolution generation with a stronger diffusion model.

  • Conclusion: AnomalyDiffusion generates anomalous image-mask pairs using anomaly and spatial embeddings in the textual space of an LDM.The anomaly embedding represents appearance, while the spatial embedding represents location.
  • Conclusion: Adaptive attention re-weighting focuses the model on areas with less noticeable generated anomalies, improving alignment between anomalies and masks.This mechanism addresses incomplete mask coverage and alignment problems during generation.
  • Conclusion: Experiments show that AnomalyDiffusion outperforms existing anomaly generation methods and that its generated data improves downstream anomaly inspection.The conclusion states this result without specifying an additional metric.
  • Future work: Future work will explore a more potent diffusion model to enhance generated-anomaly resolution and potentially improve downstream performance.This is the paper’s stated future direction.

B Data Augmentation

The data augmentation procedure addresses few-shot spatial overfitting by transforming paired anomaly images and masks together. These transformations disrupt position information while keeping the anomaly within the image.

  • Motivation: Fewer than 10 training images per anomaly category can cause spatial embedding to retain observed anomaly positions and overfit scarce spatial-encoder data.The limitation concerns the ability to disentangle anomaly appearance from location under very limited data.
  • Augmentation procedure: Random cropping, translation, and rotation are applied jointly to each anomalous image and its corresponding mask.The paired transformations preserve image-mask correspondence during augmentation.
  • Augmentation procedure: Recording the anomaly region’s maximum and minimum coordinates keeps the anomaly inside the image during augmentation.The coordinate constraint is used while transforming paired image-mask data.

C.1 Training Details

Training details specify token counts and optimization settings for spatial anomaly embeddings, then describe mask generation through classifier-free guidance. The experiments use standard metrics for generation and anomaly inspection evaluation.

  • Training spatial anomaly embedding: Each anomaly type receives an 8-token anomaly embedding, while a shared spatial encoder produces a 4-token spatial embedding.Training uses batch size 4 and learning rate 0.005, sampling four anomalous image-mask pairs per iteration.
  • Mask generation: A trained 4-token mask embedding is supplied as a text condition to guide latent-diffusion mask generation.Classifier-free guidance is used for this generation process.
  • Mask generation: The classifier-free guidance equation combines unconditional and mask-conditioned noise predictions using scale s.The implementation sets s = 5, matching Textual Inversion.
  • Metrics: IS and IC-LPIPS evaluate generated-image quality and diversity, while AUROC and AP evaluate anomaly detection and localization.Higher IS, IC-LPIPS, AUROC, and AP indicate better performance for their respective evaluation purposes.

D.1 Ablation on Spatial Anomaly Embedding

Textual Inversion entangles anomaly appearance with training-location information, limiting arbitrary placement. Spatial Anomaly Embedding separates these factors so masks can specify anomaly locations.

  • D.1 Ablation on Spatial Anomaly Embedding: Textual Inversion captures anomaly locations together with anomaly types, causing generated anomalies to remain tied to training-sample positions.This limits anomaly generation when anomalies must appear at arbitrary positions.
  • D.1 Ablation on Spatial Anomaly Embedding: Spatial Anomaly Embedding disentangles anomaly appearance from location using separate anomaly and spatial embeddings.The spatial embedding is encoded from the input anomaly mask, while the anomaly embedding represents appearance.
  • D.1 Ablation on Spatial Anomaly Embedding: Anomaly localization AP decreases quickly when the anomaly rate falls below 30%.The paper attributes this to limited training data, often only 1–2 instances per category, making anomaly information difficult to capture.
  • D.1 Ablation on Spatial Anomaly Embedding: Increasing anomaly and spatial embedding lengths raises model parameters but produces similar downstream anomaly-localization performance.This indicates that the model is not sensitive to these hyperparameters within the tested settings.

D.4 Ablation on SAE and AAR

SAE improves anomaly authenticity and AAR improves mask filling, while their combination achieves both properties. Direct cross-attention-map replacement is unstable and less effective than the proposed approach.

  • D.4 Ablation on SAE and AAR: The full SAE + AAR model generates authentic anomalies that fill the anomaly mask, unlike either component used alone.SAE alone improves authenticity without filling the mask, whereas AAR alone fills the mask but sacrifices authenticity.
  • D.4 Ablation on SAE and AAR: Directly replacing the cross-attention map can conflict with the attention values designed for the original map, producing unstable results.This motivates the adaptive attention re-weighting design.
  • D.4 Ablation on SAE and AAR: Textual Inversion + Prompt-to-Prompt generates anomalies less authentically than Spatial Anomaly Embedding and differs substantially from real-anomaly ground truth.The comparison evaluates reconstruction from masked real anomaly images.
  • D.4 Ablation on SAE and AAR: The proposed model significantly outperforms Textual Inversion + Prompt-to-Prompt on downstream anomaly localization.The comparison trains a U-Net on generated anomalous image-mask pairs.

F More qualitative experiments

Across supplementary qualitative comparisons, AnomalyDiffusion produces diverse, authentic anomalies that accurately align with masks, while competing methods show authenticity or alignment failures.

  • F More qualitative experiments: DFMGAN occasionally fails to maintain alignment between generated anomalies and masks, including for the metal nut-bent example.This contrasts with the proposed model’s reported alignment accuracy.
  • F More qualitative experiments: AnomalyDiffusion generates anomalies with the highest diversity and authenticity while accurately aligning them with anomaly masks.These generated pairs can effectively support downstream anomaly inspection tasks.

G More Quantitative experiments

Supplementary experiments evaluate generated-data methods for pixel-level localization, image-level detection, and localization using F1-max, with the proposed model achieving the best reported F1-max localization performance.

  • G More Quantitative experiments: The supplementary evaluation adds the Per Region Overlap metric to compare anomaly-generation methods on downstream anomaly localization.The experiment trains a U-Net on generated datasets from Crop&Paste, DRAEM, PRN, DFMGAN, and AnomalyDiffusion.
  • G More Quantitative experiments: AnomalyDiffusion achieves the best anomaly-localization performance with F1-max in the supplementary comparison.Table 12 reports comparisons among anomaly localization models using F1-max.
  • G More Quantitative experiments: DiffAug generates anomalies that are the same as training samples in the qualitative comparison.The comparison is presented as part of anomaly-generation quality evaluation.
Loading 2312.05767v2…