Source-linked AI summary
Divide-and-Assemble: Learning Block-wise Memory for Unsupervised Anomaly Detection
Jinlei Hou, Yingying Zhang, Qiaoyong Zhong, Di Xie, Shiliang Pu, Hong Zhou
TL;DR
Unsupervised image anomaly detection lacks abnormal training data, and subtle anomalies can be overwhelmed by reconstruction error from normal pixels. The paper proposes divide-and-assemble reconstruction with block-wise memory and adversarial semantic features, achieving state-of-the-art performance on common anomaly-detection benchmarks.
Problem
Unsupervised image anomaly detection is challenging because abnormal training data are unavailable and subtle anomalies contribute little to aggregated reconstruction error.
Method
The Divide-and-Assemble framework uses multi-scale block-wise memory to model block-level normal patterns and adversarially learned discriminator features to detect anomalies semantically.
Results
The framework demonstrates excellent reconstruction and state-of-the-art performance on common unsupervised anomaly-detection benchmarks.
Takeaways & Limitations
Controlling feature-map block granularity balances good reconstruction of normal samples with poor reconstruction of abnormal samples, while semantic features complement image-space detection.
Takeaways & Limitations
Comparison with MemAE is constrained because MemAE uses a different evaluation protocol in which only part of the test set is used.
Abstract
from arXiv · showhide
Reconstruction-based methods play an important role in unsupervised anomaly detection in images. Ideally, we expect a perfect reconstruction for normal samples and poor reconstruction for abnormal samples. Since the generalizability of deep neural networks is difficult to control, existing models such as autoencoder do not work well. In this work, we interpret the reconstruction of an image as a divide-and-assemble procedure. Surprisingly, by varying the granularity of division on feature maps, we are able to modulate the reconstruction capability of the model for both normal and abnormal samples. That is, finer granularity leads to better reconstruction, while coarser granularity leads to poorer reconstruction. With proper granularity, the gap between the reconstruction error of normal and abnormal samples can be maximized. The divide-and-assemble framework is implemented by embedding a novel multi-scale block-wise memory module into an autoencoder network. Besides, we introduce adversarial learning and explore the semantic latent representation of the discriminator, which improves the detection of subtle anomaly. We achieve state-of-the-art performance on the challenging MVTec AD dataset. Remarkably, we improve the vanilla autoencoder model by 10.1% in terms of the AUROC score.
1. Introduction
Unsupervised reconstruction-based anomaly detection struggles to separate subtle anomalies from normal content. DAAD addresses this by controlling feature-map block granularity and combining multi-scale memory with semantic adversarial features.
- Unsupervised anomaly detection models normality without abnormal training samples, then identifies inference samples that are outliers of that distribution.
- Autoencoders can lose image details through down-sampling, causing blurry outputs and large reconstruction errors even for normal samples.
- Existing memory-augmented models decode feature maps per pixel, which can reconstruct anomalies accurately because small blocks share patterns with normality.
- A proper medium block size can reconstruct similar normal patterns while reconstructing abnormal patterns poorly, maximizing their error separation.
- DAAD divides encoded feature maps into grids and uses matching block-wise memory modules across multiple scales to retrieve normal patterns for decoding.
- Adversarially learned semantic features complement low-level reconstruction errors, improving detection of subtle anomalies whose pixels occupy small regions.
- Extensive benchmark experiments report excellent reconstruction and state-of-the-art anomaly-detection performance.
2. Related Work
Related work formulates anomaly detection primarily as unsupervised modeling of normality because abnormal samples are scarce. Existing approaches include one-class, clustering, reconstruction-based, and feature-based methods, alongside memory networks.
- Unsupervised Learning for Anomaly Detection: One-class classification methods are natural choices when abnormal samples are unavailable during training.
- Unsupervised Learning for Anomaly Detection: Unsupervised clustering methods, including k-means and Gaussian-based approaches, are also used for anomaly detection.
- Reconstruction-based methods: Reconstruction-based methods detect anomalies through higher reconstruction error, typically using autoencoders trained on normal samples.
- Feature-based methods: Feature-based methods use pre-trained ImageNet representations to compare test-image semantics with normal samples or guide variational autoencoder training.
- Memory Networks: Memory networks use specialized memory banks to capture long-term dependencies, but limited memory capacity can prevent complete recording of sequential content.
3. Approach
DAAD implements image reconstruction as divide-and-assemble: block-wise memory modulates reconstruction through multi-scale feature-map divisions, while adversarial semantic features supplement pixel reconstruction for anomaly detection.
- Framework overview: The framework uses an encoder-decoder network whose encoder maps input x to latent z and decoder GD reconstructs the image.The architecture is a symmetric bow-tie network with skip connections inspired by UNet.
- Block-wise memory module: Block-wise memory divides multi-scale feature maps into grids, converts blocks into queries, and reassembles retrieved features before decoding.Division rates are shared across scales, but queries at different levels use separate memory banks.
- Block-wise memory module: The memory bank records prototypical normal patterns and retrieves them through attention weights computed from query-item similarity.Each memory item has fixed dimension matching the query, and addressing produces a normal representation for each block.
- Adversarially learned representation: Adversarial learning appends a discriminator after the decoder, with the encoder-decoder acting as the generator.During inference, discriminator features are extracted from both the input and reconstructed images.
- Loss function: The alignment loss matches discriminator feature representations of input and reconstructed images so feature similarity can contribute to anomaly detection.The extracted features come from the discriminator’s last convolutional layer.
- Anomaly score: During inference, the anomaly score fuses high-dimensional reconstruction error with low-dimensional alignment error.Both scores are linearly scaled to [0, 1], and γ controls their relative contribution.
4. Experiments
Experiments evaluate DAAD on image anomaly-detection benchmarks and ablations. DAAD improves over vanilla autoencoders, while medium block granularity and semantic alignment strengthen performance.
- Setup: The experiments evaluate DAAD on MVTec AD, MNIST, and CIFAR10, using image-level AUROC as the principal criterion.MVTec AD contains 5,354 high-resolution color images across 15 classes, including fine-grained anomalies.
- MVTec AD: 10.1% absolute AUROC gain over the vanilla AE is achieved on MVTec AD.DAAD surpasses most reconstruction-based methods and remains comparable to AESc.
- MNIST and CIFAR10: DAAD achieves excellent performance on MNIST and CIFAR10 under the one-class novelty-detection protocol.The comparison is against existing reconstruction-based methods.
- Ablation Study: DAAD gains 5.1% AUROC over AE, while DAAD+ adds 5%, producing a total 10.1% improvement.The multi-scale block-wise memory module improves separability, and adversarially learned representation adds further improvement.
- Block Size Ablation: A medium division rate of 8 gives the best AUROC, whereas small or large division rates are sub-optimal.Larger division rates create smaller blocks, and reconstruction quality varies accordingly.
- Adversarial Representation: Adversarial training alone does not improve DAAD, but alignment loss and integrating the alignment score into anomaly scoring provide a performance boost.The alignment score particularly helps categories that are difficult to reconstruct, such as leather and carpet.
5. Conclusion
The paper concludes that controlling feature-map block granularity balances reconstruction of normality and anomaly. Adversarially learned semantic representations complement image-space reconstruction to detect subtle anomalies.
- Conclusion: Existing memory-augmented methods struggle to improve anomaly detection on complex datasets such as MVTec AD.The paper identifies ineffective performance improvement as a limitation of prior approaches.
- Conclusion: DAAD controls the granularity of feature-map building blocks to balance good reconstruction of normality with poor reconstruction of anomaly.This division-and-assembly perspective is presented as central to reconstruction-based anomaly detection.
- Conclusion: Adversarially learned feature representations enable anomaly detection in low-dimensional semantic space alongside image-space detection.The two representations are described as complementary.
- Conclusion: Results on common anomaly-detection benchmarks validate the effectiveness and versatility of the proposed framework.The conclusion states this outcome without narrowing it to a single dataset.
Appendix
Appendix material examines memory size, video-dataset generalization, and DAAD architecture. Increasing memory size improves AUROC until saturation, while the appendix documents model structures and settings.
- Impact of Memory Size: AUROC steadily improves as memory size increases and saturates after 500 memory items.Further increases do not cause obvious performance degradation.
- Video Datasets: The method is evaluated for generalization on the UCSD-Ped2 and CUHK Avenue video anomaly-detection datasets.Video frames are resized to 256 × 256 and the memory bank size is set to 2,000.
- Structure of DAAD: DAAD combines skip connections with a block-wise memory module to balance reconstruction of normality and anomaly.The appendix identifies this as the framework structure.
- Structure of the Discriminator: DAAD+ uses a discriminator architecture following DCGAN, with a flattened feature-extractor representation of dimension 100 for 256 × 256 inputs.This describes the discriminator structure used for adversarially learned representation.