Source-linked AI summary
Inpainting Transformer for Anomaly Detection
Jonathan Pirnay, Keng Chai
TL;DR
Anomaly detection by reconstruction is challenged by models that reconstruct anomalous regions, while CNN receptive fields limit distant contextual information. The paper proposes InTra, a convolution-free Transformer that inpaints covered patches from long image-patch sequences, achieving competitive detection and improved segmentation on MVTec AD without extra training data.
Problem
Existing reconstruction and inpainting methods can reconstruct anomalies, while CNN receptive fields limit distant contextual information needed for larger anomalous regions.
Method
InTra splits images into patches and uses a convolution-free Transformer with self-attention to reconstruct covered patches from long neighboring-patch sequences.
Results
InTra achieves detection performance on par with current state-of-the-art and slightly better segmentation performance on MVTec AD without extra training data.
Takeaways & Limitations
Global-context patch reconstruction enables anomaly detection and localization without relying on convolutional operations or extra training data.
Takeaways & Limitations
InTra underperforms on Cable and capsule because large-area constitution defects are hard to detect and typography reconstruction is insufficient.
Abstract
from arXiv · showhide
Anomaly detection in computer vision is the task of identifying images which deviate from a set of normal images. A common approach is to train deep convolutional autoencoders to inpaint covered parts of an image and compare the output with the original image. By training on anomaly-free samples only, the model is assumed to not being able to reconstruct anomalous regions properly. For anomaly detection by inpainting we suggest it to be beneficial to incorporate information from potentially distant regions. In particular we pose anomaly detection as a patch-inpainting problem and propose to solve it with a purely self-attention based approach discarding convolutions. The proposed Inpainting Transformer (InTra) is trained to inpaint covered patches in a large sequence of image patches, thereby integrating information across large regions of the input image. When training from scratch, in comparison to other methods not using extra training data, InTra achieves results on par with the current state-of-the-art on the MVTec AD dataset for detection and surpassing them on segmentation.
1 Introduction
The paper frames anomaly detection as patch inpainting and replaces convolutional reconstruction with self-attention that incorporates distant context. InTra reconstructs covered patches from surrounding sequences and reaches competitive MVTec AD detection and segmentation performance without extra training data.
- Anomaly detection is difficult because anomalous samples are scarce and anomalies can have unexpected shapes and textures, limiting supervised methods.
- Convolutional autoencoders can reconstruct anomalies well because they generalize strongly, causing misdetection.
- Inpainting methods cover image regions during self-supervised training, but limited CNN receptive fields make large anomalous regions difficult to remove.
- InTra splits images into patches and uses a convolution-free Transformer to reconstruct covered patches from long sequences of neighboring patches.
- Global context lets individual patch reconstructions avoid simply mimicking local neighborhoods, producing high anomaly scores for spacious anomalous regions.
- On MVTec AD, InTra is trained from scratch with approximately 55M parameters using only 60–400 images per category and no extra training data.
2 Related Work
Prior reconstruction methods model normal images with CNN autoencoders or inpainting schemes, but local receptive fields can limit global context. Transformer-based patch representations provide an alternative for incorporating longer-range image dependencies.
- Reconstruction-based anomaly detection models the manifold of defect-free images and compares defective inputs with their reconstructions.
- CNN autoencoders often generalize to anomalies despite training only on defect-free samples, motivating inpainting schemes that hide input regions.
- Large-area anomalies can remain problematic when inpainting does not cover them sufficiently, motivating Transformer-based global context instead of CNNs.
- Self-attention relates sequence elements through weighted dependencies and processes long-range dependencies globally.
- Convolutional architectures provide locality, while their restricted receptive fields limit global context in convolutional autoencoders.
- Vision Transformer represents images as sequences of embedded square patches and achieves results comparable to state-of-the-art CNNs on some tasks.
3 Inpainting Transformer for Anomaly Detection
InTra formulates anomaly detection as patch inpainting with a convolution-free Transformer that uses broad spatial context. It reconstructs each patch, assembles a full image, and derives anomaly maps from reconstruction differences.
- Patch Embeddings and Multihead Feature Self-attention: The method splits images into square patches and trains a Transformer to reconstruct one covered patch from the remaining patches in a local window.Patch positions are embedded so the model can use spatial information while attending across the sequence.
- Patch Embeddings and Multihead Feature Self-attention: Each input sequence replaces the target patch with a learnable inpainting embedding and combines patch content with positional embeddings.The resulting sequence contains the target placeholder and all non-target patches in the selected window.
- Network Architecture and Training: The network is a stack of Transformer blocks using multihead feature self-attention, MLPs, pre-normalization, residual connections, and long residual links between early and late layers.The final sequence is averaged and mapped through a learnable affine transformation to produce the reconstructed flattened patch.
- Network Architecture and Training: Training compares original and reconstructed patches with pixel-wise L2 loss, structural similarity, and gradient magnitude similarity.The loss combines these terms using individual scaling parameters α and β.
- Inference and Anomaly Detection: During inference, the model reconstructs every patch using a centered window where possible, then combines the patches into a full reconstructed image.The difference between the original and reconstructed images is used to compute the anomaly map.
- Inference and Anomaly Detection: The anomaly map uses multiscale smoothed gradient differences, while the image-level detection score is the pixel-wise maximum of the map.Smoothing is used to improve robustness to small, poorly reconstructed anomalous regions.
4 Experiments
Experiments evaluate InTra on MVTec AD using reconstruction-based detection and segmentation metrics, compare it with established methods, and examine architectural choices and category-specific weaknesses.
- MVTec AD contains high-resolution manufacturing images from 5 texture and 10 object categories, with around 60–400 normal training samples per category and pixel-level test annotations.
- ROC AUC is reported at image level for detection and pixel level for segmentation.
- The model uses 13 attention blocks, 8 heads per block, latent dimension D = 512, and approximately 55M learnable parameters.
- InTra is compared with RIAD and CutPaste without using extra training data.
- 0.2% separates CutPaste’s superior detection result from InTra, while InTra improves segmentation by 0.6%.
- Cable anomalies involving missing product components and capsule typography reconstruction produce underperforming results.
- Ablation studies compare the regular architecture with variants such as removing long residual connections, with average detection and segmentation results reported in Table 2.
5 Conclusion
The conclusion presents InTra as a self-attention-based inpainting approach for visual anomaly detection and localization. It reports state-of-the-art MVTec AD results without extra training data while emphasizing sensitivity to hyperparameter choices.
- InTra uses Transformer-based inpainting and discards convolutions to incorporate global context into image reconstructions.
- Hyperparameters including image size, patch sequence length, and patch dimension strongly affect overall performance.
- InTra reaches state-of-the-art results on MVTec AD without using extra training data.
A Appendix
The appendix provides qualitative InTra results across the remaining MVTec AD categories. Figures organize examples by category and show originals, reconstructions, and anomaly maps.
- Figure 3 presents carpet, grid, leather, tile, wood, bottle, cable, and capsule examples from top to bottom.
- Each Figure 3 row contains category examples, with image groups showing original, reconstruction, and anomaly map images.
- Figure 4 continues the qualitative results for hazelnut, metal nut, pill, screw, toothbrush, transistor, and zipper.