Source-linked AI summary
A Visual Question Answering Model to Automate Nondestructive Evaluation Image Analysis
Mehrdad Shafiei Dizaji, Hoda Azari
TL;DR
NDE inspection traditionally requires subjective interpretation of images, motivating interactive tools for querying defects and material conditions. This study develops a VQA model that combines visual and language encoders to answer domain-specific questions, achieving consistent BLEU scores across multiple Impact Echo evaluations and supporting targeted technical interpretation.
Problem
Traditional NDE inspection can be subjective, creating a need for more consistent image analysis and interactive answers to technical questions.
Method
The model encodes NDE images and questions separately, combines their representations, and uses a decoder to generate answers from expert-verified domain-specific question-answer data.
Results
0.68 BLEU-1, 0.65 BLEU-2, 0.61 BLEU-3, and 0.51 BLEU-4 were reported consistently across defect-type, location, shape, and severity questions.
Takeaways & Limitations
The framework provided accurate and concise answers about defect characteristics and material conditions from Impact Echo data in the BEAST dataset.
Abstract
from arXiv · showhide
This study introduces a Visual Question Answering model designed specifically for nondestructive evaluation applications. VQA models allow inspectors to interactively query NDE images, asking targeted questions like, Is there a crack or Where is the defect located and receive precise answers from the model. Leveraging deep learning and natural language processing, the developed system integrates image feature extraction (via a ResNet-50 model) and language generation capabilities (via GPT-2) to provide accurate, informative feedback. By enabling direct question-and-answer interactions, this VQA model significantly improves inspection efficiency, reduces potential errors, and enhances usability in practical field scenarios.
1. Introduction
NDE supports safer, longer-lasting infrastructure by detecting hidden defects without damaging structures, but traditional inspection can be subjective. The proposed AI platform combines visual analysis and language generation to make NDE image interpretation easier and more consistent.
- NDE assesses structural integrity, safety, and longevity while detecting hidden defects without damaging materials.
- Early detection of cracks, voids, and internal flaws can reduce costs and help prevent accidents.
- Deep learning identifies visual defect patterns, while NLP generates human-readable explanations for engineers and decision-makers.
- The NDE_Chat Platform combines CNN-based visual feature extraction with transformer-based language models for written descriptions.
- Automated analysis aims to reduce inspector workload and human error while making results faster and more consistent.
2. Visual Question Answering (VQA) Model
The proposed NDE VQA model answers questions about inspection images by encoding visual and textual inputs, fusing their representations, and decoding answers. It uses structured NDE question-answer data and BLEU-based evaluation within an accelerated bridge-testing setting.
- VQA answers questions about scenes using an input image and natural language text.
- The model processes images with CNNs, encodes questions with transformer-based models, fuses modalities through attention, and predicts an answer.
- The NDE dataset consists of image-question-answer triples, with the model trained to generate each answer for its corresponding image and question.
- 2.1. Methodology: Separate visual and language encoders produce a unified multimodal representation that a shared decoder uses to generate the final answer.
- 2.2. Evaluation Measures: BLEU evaluates predicted answers against ground truth using modified n-gram precision, a brevity penalty, and n-grams up to length 4.
- 2.3. Experiment Setup: Bridge Evaluation and Accelerated Structural Testing (BEAST): BEAST is a full-scale bridge facility where accelerated environmental and traffic demands simulate 15–20 years of wear in a few months.
- The VQA dataset transforms annotated image-caption pairs into expert-verified questions and answers about defect types, locations, dimensions, severity, and risk.
2. Results & Discussion
Across four Impact Echo VQA tasks, the model produced consistent BLEU scores while answering questions about defect type, location, shape, and severity. Its responses aligned with expert interpretations, including surface-level low-intensity zones associated with delamination or possible structural damage.
- Defect type: 0.68 BLEU-1, 0.65 BLEU-2, 0.61 BLEU-3, and 0.51 BLEU-4 were reported for the defect-type evaluation.The model identified low-intensity zones near the surface and answered surface-level delamination, although its response was more concise than the expert label.
- Defect location: 0.68 BLEU-1, 0.65 BLEU-2, 0.61 BLEU-3, and 0.51 BLEU-4 were also reported for defect-location questions.The model matched the expert interpretation that defects occurred near the surface in low-intensity zones associated with possible delamination or surface flaws.
- Defect shape: 0.68 BLEU-1, 0.65 BLEU-2, 0.61 BLEU-3, and 0.51 BLEU-4 were maintained for defect-shape questions.Although the image did not clearly label shape, the model suggested that delamination typically appears irregularly, consistent with expert insights.
- Defect severity: 0.68 BLEU-1, 0.65 BLEU-2, 0.61 BLEU-3, and 0.51 BLEU-4 were maintained for severity questions.The model flagged surface-level low-intensity zones as possible indicators of structural damage requiring closer inspection, aligning with the expert answer.
3. Conclusion
The ChatNDE-Figure-to-Caption VQA model, trained and validated on BEAST, provided accurate and concise answers to technical questions about Impact Echo data. BLEU evaluation indicated reliable responses and practical utility for inspection scenarios.
- The VQA model was trained and validated using the BEAST dataset.
- It answered targeted technical questions about defect types, locations, shapes, severity, and material conditions from Impact Echo data.
- BLEU evaluation showed consistently reliable responses, supporting the model’s practical utility in real-world inspection scenarios.
- Future improvements include broader question types and scenarios, multi-step reasoning, and confidence-based answers for field conditions.