Source-linked AI summary
Explainable Temporal Attention-based Defect Detection For Fillet Joints in Real-Time Gas Metal Arc Welding Based on Multi-modal Data
Mobina Mobaraki, Mahyar Asadi, Klaske Van Heusden, Guy A. Dumont
TL;DR
Real-time welding defect detection faces processing challenges because existing approaches have limited ability to store long-term spatiotemporal information. The paper proposes a temporal, multi-modal attention-based model, reporting F1-score improvements across five defects and using attention to enhance detection.
Problem
Real-time welding data processing is challenging because existing approaches have limited ability to store long-term spatiotemporal information.
Method
The paper proposes a temporal multi-modal defect detection model that adds temporal features and uses attention techniques for welding defect detection.
Results
The model improves F1 Score by 7.95% for LOP, 6.52% for LOF, 4.21% for porosity, 4.21% for undercut, and 12.5% for cold lap.
Takeaways & Limitations
Adding temporal information improves the F1 score for real-time welding defect detection.
Takeaways & Limitations
Real-time processing approaches are constrained by limited ability to store long-term spatiotemporal information.
Abstract
from arXiv · showhide
Deep learning is an efficient technique to monitor the real time welding process, reducing post-welding repairs and production delays. This paper leverages the monitoring capability by proposing a multi modal temporal attention based deep learning defect detection model for internal defects that are challenging to detect, including porosity, lack of penetration and fusion, undercut, and cold lap during Gas Metal Arc Welding in fillet joints. The model is trained on collected welding images and sound data from an industrial collaborative welding robot. The results show that the attention module can improve the F1 Score to 0.99. We use explainable Artificial Intelligence to interpret the proposed models behavior and dataset distribution, determining potential important areas in image and sound spectrograms and preferred modality to detect each defect. This improves trust and reliability in Artificial Intelligence driven welding inspection.
1. Introduction
The paper addresses real-time GMAW defect detection by combining temporal, multi-modal modeling with explainability for challenging internal defects. It targets limitations in conventional CNNs and interprets both model behavior and dataset structure.
- Motivation: Real-time defect detection is needed to improve robotic welding efficiency, while GMAW fillet joints involve rapidly changing conditions that challenge detection.The target defects include lack of penetration, lack of fusion, porosity, undercut, and cold lap.
- Motivation: Conventional CNNs can struggle with real-time welding data because they have limited ability to retain long-term spatiotemporal information.Losing information from previous time steps can reduce defect detection performance.
- Approach: Attention can enhance detection by prioritizing salient welding information over time and across frequency bands, improving performance over conventional CNNs without attention.The approach can incorporate longer-term weld-pool dynamics and acoustic signatures while reducing noise and redundancy across sources.
- Interpretability: The paper addresses the lack of interpretability studies for GMAW by identifying important image and sound regions and preferred modalities for each defect.Its explainability analysis is intended to clarify model behavior and dataset distribution in GMAW inspection.
- Approach: The proposed model combines temporal multi-modal defect detection with attention focused on important temporal and frequency features.Channel attention emphasizes welding information across time, frequency, and modality, including weld-pool images and acoustic spectrograms.
- Results: Temporal features improved F1 Score by 7.95 %, 6.52 %, 4.21 %, 4.21 %, and 12.5 % for LOP, LOF, porosity, undercut, and cold lap, respectively.The reported gains correspond to the five target defects in that order.
- Results: The attention-enhanced model achieved an F1 Score of 0.99 for all five target defects with a minimum additional computational cost of 0.1 GFLOPs.The paper also uses GradCAM, GradCAM++, XGradCAM, t-SNE, and manual feature analysis to interpret models and dataset modality preferences.
2. Equipment and dataset
The study uses an industrial collaborative welding robot to collect synchronized welding images and sound data from fillet-joint GMAW tests. The dataset covers five targeted defects, with mel spectrograms representing sound and a comparatively large image collection divided into training, validation, and test sets.
- Equipment: An industrial collaborative robot captured molten-metal video and torch-mounted microphone recordings during GMAW on fillet joints.The camera and microphone were attached to the robot and torch, respectively.
- Test design: The experiments deliberately produced defective or defect-free welds across six sections in each of eight welding tests.Targeted defects included lack of penetration, lack of fusion, undercut, cold lap, and porosity.
- Dataset: The image dataset contains 23,543 training, 2,942 validation, and 2,942 test images.The authors describe the dataset as larger and more representative than the 500–3,000-image average reported for other studies.
3.1. Multi-modal models
The multi-modal models combine image and sound information using feature fusion, with fusion choices evaluated for F1 score and inference time. Input-size selection balances detection performance against computational complexity, while adding attention improves F1 score.
- Image and sound feature vectors from uni-modal models were fused using concatenation, addition, linear regression, and contrastive learning.
- The best fusion technique for each defect was selected by comparing F1 Score and inference time.
- 112 × 112 image and spectrogram inputs were chosen because larger inputs increase computational complexity and lengthen training.
- An attention module improves the F1 score to 0.99 with additional computational cost.
3.2. Temporal Models
The temporal model uses 3D ResNet processing to represent consecutive image and sound data as temporal blocks. This design incorporates temporal information into multi-modal defect detection.
- A 3D multi-modal ResNet model includes temporal features by treating time as the channel depth of image and sound inputs.
- The model input format is (batch size, number of channels, number of frames, height, width).
- Sixteen images and corresponding sound data are stacked into a block for every second, and the modified 3D ResNet extracts a 512-dimensional feature vector.
3.3. Attention-based temporal models
The attention-based temporal model assigns importance to consecutive fused image-sound blocks before classification. Its design targets cross-modal temporal interactions while reducing FLOPs relative to Squeeze-and-Excitation methods.
- The proposed model adds an attention technique to the 3D temporal multi-modal model, inspired by video vision transformers.
- 48 % fewer FLOPs are required than with Squeeze-and-Excitation methods.
- The attention is applied to fused feature maps to account for cross-model interactions between image and sound data.
- Image and sound features are concatenated for each block, and a three-layer Multi Layer Perceptron produces a confidence score.
- Softmax-normalized confidence scores become attention weights that represent each block’s importance.
- Weighted multi-modal blocks are summed into a final feature vector, which an SVM classifies as defective or defect-free.
3.4. Model explainability methods
The explainability analysis evaluates GradCAM maps by perturbing image regions and measuring confidence changes. Heatmap selection and percentage-based perturbations identify pixels most relevant to detecting each defect.
- GradCAM back-propagates class scores with respect to feature maps to produce an explanation map highlighting influential image regions.
- The evaluation perturbs original images and predicts outputs from the perturbed images to assess explanation quality.
- Drop in confidence measures the percentage confidence decrease after unnecessary pixels are blocked according to the explanation map.
- Increase in confidence counts cases where focusing only on proposed important pixels increases confidence, with larger values indicating helpful removal of unnecessary details.
- The percentage metric removes 5, 25, or 50 % of proposed important pixels and predicts using the remaining pixels.
- For each defect, heatmaps are selected using confidence behavior and negative percentage metrics to identify areas with the highest detection importance.
- The analysis reports that the multi-modal model can extract required information from its combined modalities.
3.5. Data explainability methods
The study uses t-SNE and silhouette analysis to examine modality and class structure, while pixel-wise image statistics characterize defect-specific visual patterns. These analyses support comparisons of class separability, image variation, and contrasts between defect classes.
- t-SNE analysis: t-SNE represents similarities between neighboring image and sound samples in a lower-dimensional space for multimodal class analysis.The method uses conditional neighbor probabilities to preserve local similarity relationships.
- t-SNE analysis: Perplexity specifies a target number of neighbors in t-SNE, with higher values producing higher variance.
- t-SNE analysis: Silhouette score evaluates how similar each sample is to its own cluster relative to other clusters, including when boundaries are poorly defined or data are imbalanced.
- Image feature analysis: Class mean images are computed pixel-wise from the image samples in each defect class, using image dimensions and the number of class samples.The mean-image procedure is illustrated for lack of penetration.
- Image feature analysis: The analysis considers within-class variation as image standard deviation and between-class contrast as the pixel-wise difference between class mean images.Mean-image contrast is used to identify visual differences between defect classes.
4. Results and Discussion
Temporal and attention-based multimodal models improve defect-detection performance, while explainability and feature analyses identify defect-specific image and sound regions. The analyses also indicate which modality is preferred for each target defect.
- Model performance: Adding temporal information improves F1 scores for detecting lack of penetration, lack of fusion, undercut, cold lap, and porosity.
- Model performance: Attention further improves F1 scores for lack of penetration by 4.21% and lack of fusion by 1%.The reported improvement is attributed to focusing on sub-millimeter defects while filtering irrelevant spatial and temporal information.
- Model performance: The attention module adds 0.1 GFLOPs, while the temporal attention-based model totals 0.58 GFLOPs and improves lack-of-penetration F1 score by 4%.
- Explainability analysis: GradCAM analysis identifies defect-specific image regions, including the left and right legs for lack of penetration, central and upper regions for lack of fusion, and surrounding molten-metal areas for porosity.
- Explainability analysis: Sound spectrogram analysis emphasizes time-frequency areas, primarily in the first half of the spectrogram, for the five target defects.
- Explainability analysis: The identified regions include the right leg for undercut and the upper regions and molten-metal boundary for cold lap.
- Modality analysis: Image is preferred for detecting lack of penetration, lack of fusion, undercut, and cold lap, whereas sound is preferred for porosity.The preference is based on t-SNE distributions and manual image-feature analysis.
5. Conclusion
The study concludes that temporal information and attention improve multimodal defect detection, reaching an F1 Score of 0.99 for five target defects. Explainable AI identifies important regions and preferred modalities, while the data scope limits direct production-setting evaluation.
- Performance conclusions: Adding temporal information improves F1 scores for the GMAW defect-detection models.
- Performance conclusions: Attention improves defect-detection performance by 4.21% for lack of penetration and 1.02% for lack of fusion, leading to an F1 Score of 0.99 for five target defects.
- Explainability conclusions: Explainable AI identifies important regions in welding images and sound spectrograms and determines the preferred modality for each defect.
- Scope and limitations: The study data were collected in an R&D setting with intentionally created defects, and production-setting real-time performance remains future work.