PaperlayerView exact revision on arXiv

Source-linked AI summary

Defect Transformer: An Efficient Hybrid Transformer Architecture for Surface Defect Detection

Junpu Wang, Guili Xu, Fuju Yan, Jinjin Wang, Zhengsheng Wang

arXiv:2207.08319v1cs.CV

TL;DR

Cluttered surfaces and pseudo-defects make pixel-wise defect detection difficult, while CNNs and transformers each have limitations in modeling global context and preserving local detail. DefT combines CNN and transformer components in an encoder-decoder architecture to model both kinds of information. Across three defect datasets, the authors report performance, data efficiency, training efficiency, and generalization comparable to or better than existing CNN- and transformer-based methods.

  • Problem

    Cluttered surfaces and pseudo-defects complicate defect recognition, while CNNs’ local modeling and transformers’ limited local structural information pose challenges for pixel-wise detection.

    Source: §I · pp. 1–2

  • Method

    DefT combines CNN and transformer properties in an encoder-decoder architecture to model local and long-range information for pixel-wise surface defect detection.

    Source: §I · p. 2

  • Results

    Across three defect datasets, DefT is reported as comparable to or better than existing CNN- and transformer-based methods in detection performance, data efficiency, training efficiency, and generalization.

    Source: §VI · p. 10

  • Takeaways & Limitations

    The reported results support DefT as a competitive approach across the evaluated defect datasets and four reported evaluation dimensions.

    Source: §VI · p. 10

Abstract

from arXiv · show

Surface defect detection is an extremely crucial step to ensure the quality of industrial products. Nowadays, convolutional neural networks (CNNs) based on encoder-decoder architecture have achieved tremendous success in various defect detection tasks. However, due to the intrinsic locality of convolution, they commonly exhibit a limitation in explicitly modeling long-range interactions, critical for pixel-wise defect detection in complex cases, e.g., cluttered background and illegible pseudo-defects. Recent transformers are especially skilled at learning global image dependencies but with limited local structural information necessary for detailed defect location. To overcome the above limitations, we propose an efficient hybrid transformer architecture, termed Defect Transformer (DefT), for surface defect detection, which incorporates CNN and transformer into a unified model to capture local and non-local relationships collaboratively. Specifically, in the encoder module, a convolutional stem block is firstly adopted to retain more detailed spatial information. Then, the patch aggregation blocks are used to generate multi-scale representation with four hierarchies, each of them is followed by a series of DefT blocks, which respectively include a locally position-aware block for local position encoding, a lightweight multi-pooling self-attention to model multi-scale global contextual relationships with good computational efficiency, and a convolutional feed-forward network for feature transformation and further location information learning. Finally, a simple but effective decoder module is proposed to gradually recover spatial details from the skip connections in the encoder. Extensive experiments on three datasets demonstrate the superiority and efficiency of our method compared with other CNN- and transformer-based networks.

I. INTRODUCTION

Surface defect detection must distinguish genuine defects from pseudo-defects on cluttered surfaces, a challenge for both CNNs’ local modeling and transformers’ preservation of fine spatial structure. DefT combines CNN and transformer components to model local and long-range relationships, and the authors evaluate it on three benchmarks.

I. INTRODUCTION

  • Rusts, impurities, and wrinkles can resemble genuine defects on cluttered surfaces, making correct recognition difficult even for humans.The paper identifies these error-prone artifacts as pseudo-defects.Source: §I · p. 1
  • CNNs capture low-to-high-level features but their intrinsic locality limits explicit access to global context, while transformers can lose fine spatial detail and local-structure bias.The paper also notes that convolution-free transformers may require large datasets and longer training to learn local visual structure implicitly.Source: §I · p. 2
  • Pixel-wise prediction makes self-attention’s quadratic cost in input resolution a challenge, while lower-resolution attention can sacrifice performance.The paper also identifies positional-order modeling as a challenge for self-attention.Source: §I · p. 2
  • DefT combines CNN and transformer components in an encoder-decoder network to learn local and long-range relationships for pixel-wise surface defect detection.Its convolutional stem retains local spatial information, four patch aggregation stages generate multi-scale features, and the decoder recovers defect details from encoder skip connections.Source: §I · p. 2
  • The authors report experiments on three surface defect benchmarks comparing DefT with transformer- and CNN-based models.The stated goal is to demonstrate performance and generalization.Source: §I · p. 2

II. RELATED WORKS

The related-work section reviews traditional defect-detection approaches, including statistical, spectral, and model-based methods. It characterizes these methods as dependent on hand-crafted features and assumptions that may not generalize across industrial scenarios.

II. RELATED WORKS

  • The section introduces its review of traditional and CNN-based methods before turning to vision transformers.This is the stated scope of the related-work review.Source: §II · pp. 2–3

A. Traditional defect detection methods

  • Traditional defect-detection methods use hand-crafted features and prior knowledge, and are grouped into statistical, spectral, and model-based approaches.Statistical methods use image statistics, spectral methods use transform-domain features, and model-based methods formulate feature distributions mathematically.Source: A. Traditional defect detection methods · p. 3
  • These approaches depend on assumptions, selected features, or transform domains that may not suit different textures and industrial scenarios.The paper also notes computational complexity for model-based methods and difficulty with stochastic surface variations for spectral methods.Source: A. Traditional defect detection methods · p. 3

B. CNN-based defect detection methods

CNN-based methods learn features across levels and support pixel-wise defect detection, but convolution’s locality limits explicit long-range context. Transformer methods offer global interaction, yet preserving local structure and fine spatial detail remains important for dense prediction.

B. CNN-based defect detection methods

  • CNNs learn low-to-high-level features, and FCN- and UNet-style architectures have advanced pixel-wise defect detection and detail recovery.The section describes UNet as a dominant segmentation architecture.Source: B. CNN-based defect detection methods · p. 3
  • CNN receptive fields constrain segmentation in challenging cases, while enlarging context with large kernels, atrous or deformable convolutions, and attention can add cost or design complexity.The section concludes that convolution still gathers information locally and lacks explicit long-range dependency modeling.Source: B. CNN-based defect detection methods · p. 3

C. Transformer-based object detection methods

  • Vision transformers model global interactions, and pyramid structures adapt them to pixel-wise prediction by generating multi-level features.The cited work also includes transformer encoders and decoders for segmentation.Source: C. Transformer-based object detection… · p. 3
  • Convolution-free transformers can neglect local structure needed for fine spatial details, motivating hybrid designs that insert convolution or pooling into transformer components.The section describes convolution for positional encoding and feed-forward networks, and pooling or convolution for reducing attention computation.Source: C. Transformer-based object detection… · pp. 3–4
  • Prior transformer-based defect work includes sewer-defect localization and severity analysis and semi-supervised anomaly detection, while few works use fully supervised transformers for high-quality pixel-wise detection.The passage identifies this as the gap for surface defect detection.Source: C. Transformer-based object detection… · p. 4

III. METHODOLOGY

DefT is an encoder-decoder hybrid that combines CNN and transformer properties for surface defect detection. Its encoder retains local information, builds multi-scale features, and uses overlapping convolutional patch aggregation to preserve neighboring spatial relations.

III. METHODOLOGY

  • The methodology presents DefT as a hybrid transformer architecture for surface defect detection.The stated design aims to combine CNN strengths in dense prediction with transformer long-range dependency modeling.Source: §III · p. 4, A. Network Architecture · p. 4

A. Network Architecture

  • A UNet-like encoder uses a convolutional stem and four stages of patch aggregation and DefT blocks, then sends hierarchical features through skip connections to the decoder.The DefT blocks contain LPB, LMPS, and CFFN components; the resulting feature strides are 2, 4, 8, 16, and 32 relative to the input.Source: A. Network Architecture · p. 4

1) Stem Block:

  • The convolutional stem uses smaller 2×2 patches than cited patchify stems to retain finer spatial information for pixel-wise detection.It is implemented with a stride-2 3×3 convolution followed by two stride-1 3×3 convolutions.Source: 1) Stem Block · p. 4

2) Patch Aggregation Block:

  • Overlapping convolutional patch aggregation replaces non-overlapping linear patch embedding to preserve spatial relations around neighboring patches.The described aggregation downsamples resolution by 2× using a 3×3 convolution with stride 2 and padding 1.Source: 2) Patch Aggregation Block · pp. 4–5

3) DefT Block:

DefT blocks combine local position encoding, efficient multi-scale global attention, and convolutional feature transformation. Pooling reduces the spatial scale of attention keys and values, while convolutional operations add local spatial information.

3) DefT Block:

  • Each DefT block comprises LPB, LMPS, and CFFN, with layer normalization and residual connections at the ends of LMPS and CFFN.The block models local and long-range relations among multi-scale features.Source: 3) DefT Block · p. 5
  • The locally position-aware block uses adjacent patches to dynamically learn local spatial information, addressing transformers’ lack of positional awareness.It derives position encoding from a 3×3 convolution with zero padding.Source: 3) DefT Block · p. 5
  • Lightweight multi-pooling self-attention applies pyramid pooling to keys and values, reducing their spatial scale while modeling multi-scale contextual information.The pooled features are flattened and concatenated before attention.Source: 3) DefT Block · pp. 5–6
  • LMPS reduces computational load and memory usage while adding multi-scale contextual modeling.Its keys and values have shorter sequences while maintaining the contextual abstraction of the input.Source: 3) DefT Block · p. 6
  • The convolutional feed-forward network adds a 3×3 convolution to the feed-forward network to further learn locality during feature transformation.The 3×3 convolution supports information interaction within a local region.Source: 3) DefT Block · p. 6

C. Decoder module

The decoder progressively restores defect details by merging upsampled decoder features with features from symmetrical encoder stages. Repeated convolutional refinement reduces information loss from downsampling, and the final feature map produces the detection result.

C. Decoder module

  • At each scale, the decoder upsamples its feature map twice, concatenates it with a feature map from the symmetrical encoder stage, then convolves and halves the channels.The merge operation is repeated across stages to refine defect details and reduce information loss caused by downsampling.Source: C. Decoder module · p. 6
  • The final refined feature map passes through a one-channel convolution and sigmoid function to generate the detection result.Source: C. Decoder module · p. 6
  • The decoder uses simple alternating upsampling and convolution operations rather than the intricate components cited as [45].The paper attributes the reported performance of this design to the encoder’s local and global attention representations.Source: C. Decoder module · p. 6

D. Training strategy

DefT is trained end to end on resized and cropped training samples using SGD and a hybrid loss. At inference, test images are uniformly scaled before prediction.

D. Training strategy

  • Training samples are resized to 256×256, cropped to 224×224, and randomly fed to the network in batches of 8.Source: D. Training strategy · p. 6
  • Training uses SGD for 700 epochs with momentum 0.9, weight decay 0.0001, and an initial learning rate of 0.003 that decreases stepwise under a poly policy.The policy uses power 0.9; training also uses the hybrid loss in and deep supervision.Source: D. Training strategy · p. 6
  • At inference, test images are uniformly scaled to 256×256 before DefT predicts the detection result.Source: D. Training strategy · p. 6

IV. EXPERIMENTS

The experiments compare DefT with other deep learning models on three datasets. The datasets cover steel, fabric, and rail surfaces, with differing image counts and training and test splits.

IV. EXPERIMENTS

  • The experiments report comparisons with other deep learning models on three datasets.Source: §IV · p. 6

1) Datasets Description:

  • SD-saliency-900 contains 900 annotated steel defect images; the described comparison uses 810 images for training and 900 for testing.The images include cluttered backgrounds, low contrast, and noise interference.Source: 1) Datasets Description · pp. 6–7
  • The fabric defect dataset contains 1,600 images, split into 1,200 training and 400 test images.The passage identifies fabric wrinkles and weak defect features as challenges.Source: 1) Datasets Description · p. 7
  • NRSD-MN contains 3,936 man-made and 165 natural no-service rail-surface images; 2,971 man-made images are selected for training and the rest for testing.The images include diverse defect shapes and rust disturbance.Source: 1) Datasets Description · p. 7

2) Datasets Description:

The section defines the pixel-level metrics and comparison set, then reports DefT’s visual and quantitative results on SD-saliency-900. DefT’s predictions closely match the ground truth while suppressing background clutter.

2) Datasets Description:

  • Detection is evaluated using FPR, FNR, ACC, MAE, and F1; lower FPR, FNR, and MAE and higher ACC and F1 indicate better performance.The comparison also reports model parameters, FLOPs, inference speed, and PR and F-measure curves.Source: 2) Datasets Description · p. 7

3) Competing methods:

  • The comparison includes pure-transformer, hybrid-transformer, generic CNN, and defect-specific CNN methods, all retrained for the defect-detection tasks.The listed methods are SETR and Swin-unet; TransUNet, SegFormer, and MedT; UNet; and MCnet and EDRNet.Source: 3) Competing methods · p. 7

1) Comparison on SD-saliency-900 dataset:

  • DefT closely matches ground truth, precisely highlights defects, suppresses background clutter, and outperforms all competitors in FNR, F1, and ACC on SD-saliency-900.UNet misses low-contrast defects and detects impurities; transformer methods can yield blurry boundaries.Source: 1) Comparison on SD-saliency-900 dataset · p. 7, Figure 5 · p. 7

2) Comparison on Fabric defect dataset:

On the fabric-defect and NRSD-MN datasets, DefT is competitive with specialized defect detectors while addressing errors such as missed defects and rust-related false detections. The reported NRSD-MN comparison gives improvements over the second-best model on four metrics.

2) Comparison on Fabric defect dataset:

3) Comparison on NRSD-MN dataset:

4) Efficiency comparison:

DefT is reported to offer a favorable speed-accuracy trade-off, and component ablations on SD-saliency-900 indicate gains from the proposed design choices. The evaluation also includes PR and F-measure curves for model configurations.

4) Efficiency comparison:

  • DefT has 30.56M parameters and processes a test image in 0.17s without TensorRT, with less or comparable parameters and computation and a better speed-accuracy trade-off.Its inference speed and segmentation accuracy consistently outperform MedT and SegFormer, though its speed is not the shortest.Source: 4) Efficiency comparison · p. 8, Table V · p. 9

A. Ablation Analysis

  • On SD-saliency-900, the ablations report MAE gains of 11.9% from smaller CSB patches, 5.5% from PAB, and 5.2% from LPB; LMPS changes MAE from 0.0146 to 0.0136.Replacing the baseline FFN with CFFN reaches a new state-of-the-art; integrating all components gives the best detection results with affordable computational overhead.Source: A. Ablation Analysis · p. 9, Table VI · p. 9
  • PR and F-measure curves also indicate that all components are needed for the best detection performance.The curves compare different model configurations.Source: A. Ablation Analysis · p. 9, Figure 8 · p. 9

B. Effect of position encoding

DefT’s position encoding improves detection accuracy over absolute and relative encoding baselines. Across three defect datasets, the paper reports comparable or better detection performance, data efficiency, training efficiency, and generalization than existing CNN- and transformer-based methods.

B. Effect of position encoding

  • 1.15% and 0.77% higher detection accuracy than APE and RPE, respectively, with DefT’s LPB and CFFN position encoding.The N/A variant without specialized position encoding remains competitive; the authors attribute this in part to convolutional components elsewhere in the model.Source: Table VII · p. 9, B. Effect of position encoding · p. 9

C. Analysis of model efficiency

  • 13.38% versus 1.01% relative F1-score decline for Swin-Unet and DefT, respectively, when training drops from 700 to 60 epochs.DefT reaches 82.53% F1 after 60 epochs; the experiments compare full-data training at 60, 300, 510, and 700 epochs, and training with 30%, 60%, or 100% of NRSD-MN. With 30% of the data, DefT is competitive with Swin-Unet* trained on 100%.Source: C. Analysis of model efficiency · pp. 9–10, Figure 9 · p. 10

VI. CONCLUSION

  • DefT is comparable to or outperforms existing transformer- and CNN-based methods in detection performance, data efficiency, training efficiency, and generalization.Source: §VI · p. 10
Loading 2207.08319v1…