Source-linked AI summary
MemSeg: A semi-supervised method for image surface defect detection using differences and commonalities
Minghui Yang, Peng Wu, Jing Liu, Hui Feng
TL;DR
Industrial surface-defect detection is challenging because abnormal samples are rare and diverse, while semi-supervised approaches traditionally rely only on normal data. MemSeg combines simulated abnormal samples with a memory pool of normal patterns in an end-to-end segmentation network. It reports SOTA MVTec AD performance and inference speed suited to real-time industrial requirements.
Problem
Industrial anomalies vary widely and are difficult to cover during data collection, while using only normal samples does not sufficiently teach models differences from non-normal samples.
Method
MemSeg uses simulated anomalies to learn normal–non-normal differences and a memory pool of normal patterns to guide end-to-end abnormal-region segmentation.
Results
99.56% image-level AUROC and 98.84% pixel-level AUROC are reported as SOTA on MVTec AD, with inference at 31.34 images per second on an NVIDIA RTX 3090 GPU.
Takeaways & Limitations
MemSeg simplifies semi-supervised surface-defect detection into a straightforward semantic-segmentation task while meeting reported real-time industrial requirements.
Takeaways & Limitations
MemSeg depends on spatially aligned detection targets; randomly located or oriented targets require more memory samples and higher dataset quality and quantity.
Abstract
from arXiv · showhide
Under the semi-supervised framework, we propose an end-to-end memory-based segmentation network (MemSeg) to detect surface defects on industrial products. Considering the small intra-class variance of products in the same production line, from the perspective of differences and commonalities, MemSeg introduces artificially simulated abnormal samples and memory samples to assist the learning of the network. In the training phase, MemSeg explicitly learns the potential differences between normal and simulated abnormal images to obtain a robust classification hyperplane. At the same time, inspired by the mechanism of human memory, MemSeg uses a memory pool to store the general patterns of normal samples. By comparing the similarities and differences between input samples and memory samples in the memory pool to give effective guesses about abnormal regions; In the inference phase, MemSeg directly determines the abnormal regions of the input image in an end-to-end manner. Through experimental validation, MemSeg achieves the state-of-the-art (SOTA) performance on MVTec AD datasets with AUC scores of 99.56% and 98.84% at the image-level and pixel-level, respectively. In addition, MemSeg also has a significant advantage in inference speed benefiting from the end-to-end and straightforward network structure, which better meets the real-time requirement in industrial scenarios.
1. Introduction
MemSeg addresses industrial surface-defect detection with an end-to-end semi-supervised segmentation network that learns from simulated differences and memorized normal patterns. On MVTec AD, it reports SOTA accuracy and real-time-oriented inference speed.
- Motivation: Surface-defect detection is difficult for supervised learning because abnormal samples are rare and anomalies take diverse forms.Semi-supervised methods instead require only normal samples during training.
- Limitations of Existing Methods: Existing reconstruction methods can reconstruct abnormal regions correctly, while embedding-based methods require costly feature matching during inference.Embedding-based models also rely on pre-trained feature extraction and may be insufficiently adapted to anomaly detection.
- MemSeg: MemSeg is an end-to-end memory-based segmentation network that directly determines abnormal image regions without reconstructing the input.Its design exploits small intra-class variation in products from the same production line.
- Difference Modeling: Simulated anomalies expose differences between normal and non-normal images during training, helping MemSeg learn a more robust decision boundary.The simulated anomalies need not match real-world anomalies exactly, and inference uses normal and simulated abnormal images without auxiliary tasks.
- Commonality Modeling: MemSeg compares inputs with memory-pool samples representing general normal patterns to support abnormal-region localization.Multi-scale feature fusion and novelty spatial attention coordinate memory and input-image information.
- Results: 99.56% image-level AUROC and 98.84% pixel-level AUROC are reported as SOTA on MVTec AD, while inference reaches 31.34 images per second on an NVIDIA RTX 3090 GPU.The dataset contains 75 different forms of anomalies in real scenarios.
2. Related Works
Prior semi-supervised defect-detection methods either reconstruct images or match features against normal samples. Anomaly simulation methods improve explicit difference learning, but earlier strategies and pipelines retain limitations that MemSeg targets.
- Reconstruction-Based Methods: Reconstruction-based methods train autoencoders or GANs to reconstruct normal images accurately and detect anomalies through reconstruction error.Their assumption is that abnormal images produce larger reconstruction errors.
- Anomaly Simulation-Based Methods: DRAEM creates textural anomalies by adding texture images, whereas CutPaste and AnoSeg paste image patches to simulate structural anomalies.Existing strategies consider structural or textural anomalies one-sidedly and may simulate inefficiently when foreground and background are difficult to distinguish.
- Pipeline Limitations: AnoSeg and DRAEM still reconstruct inputs, while CutPaste detects defects at image level and relies on GradCAM or Gaussian density estimation for pixel localization.MemSeg instead uses anomaly simulation without reconstruction as an auxiliary task.
- Embedding-Based Methods: Embedding-based methods extract features with pre-trained ImageNet networks and calculate anomaly scores from distances between test and normal-sample features.These methods use feature-space comparisons to obtain abnormal regions.
3. Method
MemSeg is an end-to-end U-Net-based segmentation framework that combines simulated anomalies with memory information to localize industrial surface defects. Its anomaly simulation, memory comparison, spatial attention, and multi-scale fusion modules support learning and precise localization.
- Framework: MemSeg uses U-Net with a ResNet18 encoder to segment abnormal regions end-to-end during inference.Simulated abnormal samples and memory information assist training, while localization is performed directly by the network.
- Anomaly Simulation Strategy: The anomaly simulation strategy combines target foreground, textural, and structural information to generate simulated abnormal samples.Perlin-noise masks define contiguous regions, while DTD textures and rearranged input-image content provide textural and structural anomalies.
- Anomaly Simulation Strategy: Most simulated abnormal regions are generated on the target foreground, maximizing similarity between simulated and real abnormal samples.The simulated image overlays a noisy foreground onto the original image background.
- Memory Module: MemSeg stores features from a small number of normal memory samples and compares input features with them to obtain difference information.The most similar memory sample supplies the best difference information, where larger positional differences indicate higher abnormality probability.
- Feature Fusion and Attention: Best difference information is concatenated with high-level input features, fused across scales, and passed to the U-Net decoder through skip connections.Spatial attention maps derived from the difference information reinforce abnormal-region localization.
4. Experiments
MemSeg is evaluated on semi-supervised anomaly-detection datasets using image-level and pixel-level ROC-AUC, with specified memory-sample and training settings.
- The evaluation covers MVTec AD, BeanTech AD, and a toy dataset for semi-supervised anomaly detection.
- MVTec AD contains 15 categories, about 60–400 normal training samples per category, and realistic test anomalies with varied textures and scales.
- Performance is measured using image-level and pixel-level ROC-AUC.
- MemSeg uses ResNet18 as its encoder and a convolutional upsampling decoder with bilinear interpolation, batch normalization, and ReLU activation.
- Hyperparameter search sets the learning rate to 0.04, focal-loss V to 4, and two objective-function weights to 0.6 and 0.4.
- The method generally uses 30 randomly selected memory samples, with category-specific adjustments for screws and toothbrushes.
4.3. Comparison with Existing Methods
MemSeg is compared with established anomaly-detection methods on MVTec AD and BeanTech AD, including localization quality and dataset-level performance.
- MemSeg outperforms most existing methods in the reported AUC comparisons.
- The MVTec AD comparison reports ROC-AUC in image-level and pixel-level form across methods including SPADE, PaDiM, DRAEM, CutPaste, P-SVDD, and P-SVDD-C.
- The weakest MVTec AD image-level category is screw, while transistor is weakest at pixel-level because some configurations and anomalies are difficult to localize.
- On BeanTech AD, MemSeg reaches 97.1% pixel-level AUC versus PaDiM’s 97.3%, while achieving the best image-level AUC among the compared models.
- Before thresholding, MemSeg’s anomaly localization is reported as more precise, with boundaries closer to ground truth than comparison methods.
4.4. Impact of Anomaly Simulation Strategy
The experiments assess MemSeg’s anomaly-simulation components and examine how simulated abnormal features relate to real abnormal features.
- Removing textural simulation, structural simulation, or foreground enhancement decreases AUC compared with the complete anomaly-simulation strategy.
- Table 3 reports image-level and pixel-level AUC for the complete strategy and for variants missing each anomaly-simulation component.
- For most categories, simulated abnormal and real abnormal samples show some overlap in the encoder-feature distribution after training.
- MemSeg does not require simulated abnormal samples to have the same distribution as real abnormal samples, and decoder skip connections can correct weak bottleneck separability.
4.5. Impact of Different Losses
The loss-function experiments compare L1 loss with focal loss for anomaly localization under pixel-level class imbalance.
- MemSeg’s loss-function ablation reports AUC scores for different loss-function combinations on MVTec AD.
- Using L1 loss and focal loss simultaneously produces more precise segmented-image edges than using L1 loss alone.
- Pixel-level imbalance arises because normal and simulated abnormal images are sampled at a 1:1 ratio while abnormal regions occupy only a small image fraction.
- Focal loss is introduced to focus training on abnormal regions that are underrepresented at the pixel level.
4.6. Impact of Module Components
MemSeg’s memory module, multi-scale strategy, spatial attention, and CA each affect anomaly localization, with memory size requiring careful tuning. Visualizations indicate that memory-derived difference information becomes more certain after multi-scale fusion and spatial attention.
- Module ablations: Memory information significantly affects abnormal localization, while increasing memory samples improves accuracy only within a suitable range.Too many memory samples reduce training samples and slightly decrease AUC scores.
- Module ablations: The mean AUC scores are evaluated across 13 MVTec AD categories while varying the number of memory samples.The evaluation excludes screw and toothbrush categories.
- Module ablations: Multi-scale strategy, spatial attention, and CA contribute significantly to anomaly detection performance, with multi-scale strategy having the greatest impact.
- Attention visualization: Memory-derived difference information provides a blurred abnormal-region guess, while multi-scale fusion reduces heat-map noise and makes the guess more certain.
- Attention visualization: The spatial attention-map generation process visualizes the effects of the memory module, multi-scale strategy, and spatial attention on defect localization.
4.7. Evaluation with A Toy Dataset
MemSeg generalizes to previously unseen regular anomalies generated with varied shapes, colors, and spatial properties. On the toy dataset, it achieves precise localization with an AUC score close to 100%.
- Toy-dataset evaluation: MemSeg achieves an AUC score close to 100% when localizing unknown anomalies in the toy dataset.The trained model is applied directly to abnormal samples never seen during training.
- Toy-dataset construction: The toy dataset uses rectangles, triangles, lightning bolts, stars, hearts, and circles with randomized size, color, position, angle, and aspect ratio.
- Toy-dataset evaluation: AUC scores are compared among PatchCore, SPADE, PaDiM, and MemSeg on the toy dataset.
- Toy-dataset construction: The toy dataset tests MemSeg’s generalization to regular noise using normal MVTec AD test-set samples.
4.8. Inference Speed
MemSeg is designed for faster industrial inference by avoiding reconstruction and complex feature matching. On an NVIDIA RTX 3090, it processes images substantially faster than the compared embedding-based methods.
- Inference speed: 0.0319s per image is MemSeg’s inference time, compared with 0.319s for PaDiM and 0.339s for SPADE.The reported MemSeg time represents a 10-fold inference-speed improvement.
- Inference design: MemSeg segments abnormal regions end-to-end without reconstructing input samples.This avoids the conventional reconstruction procedure used by reconstruction-based models.
- Industrial deployment: Complex feature matching in embedding-based methods is difficult to apply in industrial scenarios with high real-time requirements.
5. Conclusion
MemSeg frames semi-supervised surface defect detection as semantic segmentation by combining anomaly simulation with memory information. Its high performance and real-time operation are constrained when target locations or directions vary randomly.
- Conclusion: MemSeg simplifies semi-supervised surface defect detection into a semantic segmentation task using anomaly simulation and memory information.
- Conclusion: MemSeg achieves SOTA performance while meeting real-time requirements in industrial scenarios.
- Limitation: Random target locations or directions, such as for MVTec AD screws, require more memory samples and place higher demands on dataset quality and quantity.