Source-linked AI summary
Few-Shot Defect Image Generation via Defect-Aware Feature Manipulation
Yuxuan Duan, Yan Hong, Li Niu, Liqing Zhang
TL;DR
Industrial inspection lacks sufficient defect images, motivating data augmentation in the challenging few-shot setting. DFMGAN pretrains a data-efficient StyleGAN2 on defect-free images and adds defect-aware residual blocks to manipulate masked defect regions. On MVTec AD, it generates realistic and diverse defects and improves downstream inspection, including roughly 10% gains over the runner-up in classification accuracy.
Problem
Scarce and costly defect images hinder fully supervised industrial inspection and motivate generating additional defect samples.
Method
DFMGAN pretrains a StyleGAN2 backbone on defect-free images, then learns defect masks and masked feature manipulation through defect-aware residual blocks using limited defect images.
Results
On MVTec AD, DFMGAN generates realistic, diverse defect images and achieves the highest classification accuracy on all test partitions, with generally 10% improvement over the runner-up.
Takeaways & Limitations
Defect-aware regional transfer supports few-shot defect generation and provides useful augmentation for downstream defect inspection tasks.
Takeaways & Limitations
DFMGAN may perform poorly when defects substantially change object contours, including destructive missing-part defects and additive defects extending object areas.
Abstract
from arXiv · showhide
The performances of defect inspection have been severely hindered by insufficient defect images in industries, which can be alleviated by generating more samples as data augmentation. We propose the first defect image generation method in the challenging few-shot cases. Given just a handful of defect images and relatively more defect-free ones, our goal is to augment the dataset with new defect images. Our method consists of two training stages. First, we train a data-efficient StyleGAN2 on defect-free images as the backbone. Second, we attach defect-aware residual blocks to the backbone, which learn to produce reasonable defect masks and accordingly manipulate the features within the masked regions by training the added modules on limited defect images. Extensive experiments on MVTec AD dataset not only validate the effectiveness of our method in generating realistic and diverse defect images, but also manifest the benefits it brings to downstream defect inspection tasks. Codes are available at https://github.com/Ldhlwh/DFMGAN.
1 Introduction
Industrial defect inspection is constrained by scarce, costly defect images, while existing synthetic approaches struggle to produce realistic and diverse defects. DFMGAN addresses few-shot defect generation by adapting a defect-free StyleGAN2 backbone with defect-aware feature manipulation.
- Motivation: Defect-image scarcity and collection cost make it difficult to obtain diverse samples for industrial inspection.Existing systems often rely on defect-free data, limiting tasks such as defect classification.
- Motivation: Manual artifacts, patch copying, and region pasting can generate defect images, but the results are unrealistic and insufficiently diverse.
- Method: The method trains a StyleGAN2 backbone on defect-free data, then attaches defect-aware residual blocks that produce masks and manipulate features within defect regions.
- Motivation: GAN-based defect generation is difficult in few-shot settings because GANs are susceptible to data shortage.Prior GAN-based approaches commonly depend on hundreds or thousands of defect images.
- Contribution: DFMGAN generates realistic and diverse defect images from limited defect images by transferring critical regions rather than whole images.The method is presented as the first few-shot defect image generation approach on MVTec AD.
- Contribution: Experiments on MVTec AD validate DFMGAN for defect image generation and its benefits for downstream defect inspection tasks.
2 Related Work
Prior work addresses defect inspection and limited-data image generation through reconstruction, pseudo-defects, data-efficient GANs, and whole-image transfer. DFMGAN instead targets few-shot defect generation by transferring and manipulating defect regions, aiming for realistic and diverse outputs.
- Defect Inspection: Defect inspection research includes reconstruction-based methods, GAN-based methods, and pseudo-defect localization approaches under data insufficiency.
- Image Generation on Limited Data: Data-efficient GAN research improves generation on small datasets through augmentation, parameter limitation, transferable latent regions, or cross-domain consistency.
- Defect Image Generation: Earlier defect-image methods create pseudo-defects by adding cutouts, copying patches, or pasting defect regions onto defect-free images.
- Defect Image Generation: Only two previous generative augmentation methods, SDGAN and Defect-GAN, were identified, but they required large texture datasets.
- Defect Image Generation: DFMGAN is introduced as the first few-shot defect-generation method designed to render realistic images with diversity across objects and defects.
3 Method
DFMGAN transfers a defect-free StyleGAN2 backbone to few-shot defect generation by training defect-aware residual modules that localize and manipulate defect regions. The fixed backbone preserves object features while added modules learn diverse defect appearances from limited defect images.
- Training strategy: DFMGAN uses two training stages: pretraining a StyleGAN2 backbone on defect-free images, then training attached defect modules on a few defect images.The second stage fixes the backbone and trains 3.7M added parameters, compared with 23.2M in the first stage.
- Training strategy: The backbone maps object codes to diverse defect-free images, while defect-aware residual blocks adapt its features to generate defect images.The backbone uses StyleGAN2 with adaptive differentiable augmentation and encodes rich object features before defect adaptation.
- Defect-aware feature manipulation: Only masked defect regions receive residual features, leaving non-defect features unchanged during feature manipulation.This region-specific design focuses adaptation on defects while retaining the backbone’s object representation.
- Discriminators and objectives: A realism discriminator and a defect-matching discriminator supervise generated images and image-mask correspondence, respectively.The defect-matching discriminator uses concatenated image-mask pairs and has 1.5M parameters versus 24M for the original discriminator.
- Defect-aware feature manipulation: Residual blocks begin at 64×64 and use ToMask-generated masks to control where residual features alter the backbone feature maps.Masks are upsampled at 128 and 256 resolutions, while lower-resolution synthesis blocks remain unchanged to preserve coarse object structure.
- Discriminators and objectives: Mode seeking loss encourages different defect codes to produce different defect masks when the object code is fixed.The method uses differences between masks rather than images because defect appearances can contain unexpected artifacts.
- Discriminators and objectives: The overall objective combines StyleGAN2 losses, defect-mask matching loss, and mode seeking loss while alternating optimization of the generator and discriminators.The mode seeking component is part of the second-stage objective for improving variation in generated defects.
4 Experiment
Experiments on MVTec AD evaluate DFMGAN for few-shot defect image generation and downstream defect classification. DFMGAN generates realistic, diverse samples and improves classification performance, while baseline methods exhibit overfitting, limited diversity, or realism and boundary problems.
- 4.1 Dataset: MVTec AD: MVTec AD provides 200–400 defect-free images and typically 10–25 defect images per category, supporting the few-shot experimental setting.DFMGAN trains a backbone on defect-free images, then trains an individual model for each associated defect category at 256 × 256 resolution.
- 4.2 Defect Image Generation: DFMGAN is evaluated with KID for realism and diversity, supplemented by clustered LPIPS as a standalone diversity metric.KID is reported between 5,000 generated defect images and corresponding dataset images; lower KID indicates better performance.
- 4.2 Defect Image Generation: DFMGAN outperforms all other methods on both KID and clustered LPIPS across hazelnut defect categories despite severely insufficient training data.This indicates strong image-generation quality and diversity under the few-shot setting.
- 4.2 Defect Image Generation: Qualitatively, DFMGAN balances realism and diversity, whereas Finetune and DiffAug overfit, CDC produces unrealistic similar images, and SDGAN and Defect-GAN fail to render realistic samples.Crop&Paste largely reuses dataset defect appearances and can place defects outside object boundaries.
- 4.2 Defect Image Generation: DFMGAN generates paired defect-free and defective images whose pixel-level masks precisely delimit the defect regions.With the fixed backbone, it can also generate defect-free samples by ignoring defect-aware residual features.
- 4.3 Data Augmentation for Defect Classification: DFMGAN achieves the highest defect-classification accuracy across all tested partitions, generally improving by 10% over the runner-up.The classification experiment uses generated data to classify unseen defect samples in the hazelnut category.
5 Conclusion
DFMGAN generates diverse, high-quality defect images from only a handful of defect samples by learning defect masks and manipulating object features within those regions. Experiments on MVTec AD also show benefits for downstream defect inspection.
- DFMGAN generates diverse, high-quality defect images from just a handful of defect samples.
- Defect-aware residual blocks learn reasonable defect masks and manipulate object features within the masked regions.
- Experiments on MVTec AD demonstrate strong generation ability and benefits for downstream defect inspection tasks.
1 Implementation Detail
The implementation follows StyleGAN2 with ADA and adds DFMGAN-specific mapping, residual, and discriminator modules. Training uses specified optimization settings and GPU protocols, with ablation results reported separately.
- DFMGAN follows the StyleGAN2 implementation with adaptive data augmentation as its backbone.
- The implementation adds a defect mapping network, defect-aware residual blocks, and a defect matching discriminator.
- The defect mapping network uses two fully connected layers with 512 output channels.
- Defect-aware residual blocks use two convolutional layers, with a 64-resolution ToMask module producing a single-channel mask.
- The defect matching discriminator concatenates image and mask inputs, requiring four input channels in its first block.
- Training uses 512-dimensional random codes and modulation weights, batch size 32, and Adam with learning rate 0.0025.
- Defect generation trains for 400 kimgs on two GPUs, while classification trains ResNet-34 for 50 epochs on one GPU.
2 Additional Experiment
Additional experiments test DFMGAN across ablations, extreme 5-shot and 1-shot settings, significance tests, and multiple MVTec AD categories. The results show strong diversity, robust few-shot generation, and informative augmentation for classification, while KID can favor overfit or defect-free outputs.
- Experiments examine architectural variants, extreme 5-shot and 1-shot settings, statistical significance, and additional MVTec AD categories.
- Interpolating z_defect changes defects on the same object while fixing z_object, supporting separate control and greater image diversity.
- In 5-shot and 1-shot experiments, DFMGAN maintains satisfying performance without much deterioration, unlike baselines that repeat or fail to render realistic samples.
- Welch’s t-tests report p values of 2.53 × 10^-18 for KID and 2.23 × 10^-10 for clustered LPIPS against Defect-GAN.
- DFMGAN does not always achieve the best KID because overfit or defect-free outputs can receive favorable distribution scores.
- DFMGAN obtains the highest clustered LPIPS in 56 of 73 cases, indicating broadly strong diversity across categories.
- DFMGAN-augmented datasets achieve the best classification accuracy in 36 of 42 cases and rank second in the remaining six.
- The authors argue that slightly less realistic but more diverse defects can be more informative for augmentation than images resembling few training samples.
3 Future Work
The authors identify limitations involving defects that substantially alter object contours and the current unconditional formulation. They suggest improved contour handling and conditional models that share information across defect categories as future directions.
- DFMGAN may underperform on destructive defects that remove object parts because it is not trained to fill missing regions with reasonable backgrounds.Examples include severe cracks in hazelnuts.
- DFMGAN may also struggle with additive defects that extend object areas because it must generate defects over non-object background features.Misplaced transistors are given as an example.
- The presented model is unconditional and generates various defect types one at a time.A conditional version using extra category codes showed sub-optimal performance for some defect categories, despite good performance on others.
- A conditional model that shares information across defect categories may be desirable if the reported performance issues are resolved.