Source-linked AI summary
Advanced Steel Microstructural Classification by Deep Learning Methods
Seyed Majid Azimi, Dominik Britz, Michael Engstler, Mario Fritz, Frank Mücklich
TL;DR
Steel microstructural classification is difficult, subjective, and traditionally dependent on separately engineered features and classification. This paper uses pixel-wise FCNN segmentation with max-voting and reports 93.94% accuracy versus 48.89% for the prior state of the art.
Problem
Microstructural classification is challenging because complex constituents are mostly classified manually, while prior methods separate expert-designed feature extraction from classification.
Method
The method performs pixel-wise steel microstructural segmentation with a Fully Convolutional Neural Network followed by max-voting to classify each object.
Results
93.94% classification accuracy is achieved, compared with 48.89% for the previous state-of-the-art method.
Takeaways & Limitations
The results support a more objective and robust approach to steel microstructural classification and steel quality appreciation.
Takeaways & Limitations
The dataset contains only 11 training images and 10 test images, and bainite appearance varies because one test sample was normalized after preparation.
Abstract
from arXiv · showhide
The inner structure of a material is called microstructure. It stores the genesis of a material and determines all its physical and chemical properties. While microstructural characterization is widely spread and well known, the microstructural classification is mostly done manually by human experts, which gives rise to uncertainties due to subjectivity. Since the microstructure could be a combination of different phases or constituents with complex substructures its automatic classification is very challenging and only a few prior studies exist. Prior works focused on designed and engineered features by experts and classified microstructures separately from the feature extraction step. Recently, Deep Learning methods have shown strong performance in vision applications by learning the features from data together with the classification step. In this work, we propose a Deep Learning method for microstructural classification in the examples of certain microstructural constituents of low carbon steel. This novel method employs pixel-wise segmentation via Fully Convolutional Neural Networks (FCNN) accompanied by a max-voting scheme. Our system achieves 93.94% classification accuracy, drastically outperforming the state-of-the-art method of 48.89% accuracy. Beyond the strong performance of our method, this line of research offers a more robust and first of all objective way for the difficult task of steel quality appreciation.
1 Introduction
Steel’s mechanical performance depends strongly on its microstructure, making accurate classification important despite substantial variation in constituent appearance and processing history.
- Steel’s mechanical properties are mainly determined by its microstructure and the distribution, shape, and size of its phases.
- Microstructures vary with alloying, rolling, cooling, heat treatment, and post-treatment conditions.
- Steel microstructures can contain constituents including ferrite, cementite, austenite, pearlite, bainite, and martensite.
- Deep Learning is motivated by its ability to learn high-level features directly from raw input data in computer-vision applications.
2 Related Works
Prior steel microstructural classification relied largely on expert procedures and manual judgment, while imaging trade-offs and limited automation motivated Deep Learning approaches.
- High-resolution SEM imaging is more expensive in time and operating costs than LOM imaging, whereas low-resolution LOM makes substructure distinctions harder.
- Figure 1 illustrates martensite, tempered martensite, bainite, and pearlite as objects against ferrite matrix background in SEM and LOM images.
- Experts traditionally assign microstructure classes by observing sample images, allowing different opinions because expertise levels vary.
- The proposed method transfers Deep Learning’s segmentation success to steel microstructural classification and reports substantial gains over the previous state of the art.
3 Review of recent Deep Learning techniques in computer vision
Recent computer-vision methods use end-to-end neural networks to learn features and classifications from raw images, with CNN extensions supporting semantic segmentation.
- Convolutional layers apply trainable filters and biases across input data to produce feature maps, while pooling performs nonlinear down-sampling.
- Softmax produces categorical class probabilities, and cross-entropy measures differences between true labels and predicted probabilities.
- CNNs perform end-to-end learning by transforming raw inputs such as image pixels into semantic outputs without separating feature extraction and classification.
- Dropout randomly ignores neurons during training to improve CNN generalization, addressing overfitting risks in small datasets.
- Transfer learning initializes networks from large datasets before fine-tuning on a new task, reducing the training data needed for convergence.
- Fully Convolutional Neural Networks remove fully connected layers and add up-sampling to produce semantic segmentations closer to the original image resolution.
4 Methods
The methods compare object-based CNN classification with MVFCNN, which performs pixel-wise segmentation and max-voting to classify steel microstructure objects. The workflow covers systematic cropping, patch processing, stitching, aggregation, and training strategies.
- Object-based CNN classification: Object-based classification crops and separates constituents from registered SEM and LOM images before resizing them for CNN classification.A user-defined LOM intensity threshold produces a binary mask that localizes constituents in the corresponding SEM image.
- Segmentation-based classification: MVFCNN classifies cropped SEM or LOM patches pixel-wise, stitches the segmented patches, and applies max-voting to classify complete microstructure objects.The architecture uses FCNNs with convolutional replacements for fully connected layers, up-sampling, and optional skip layers.
- Segmentation-based classification: FCNN outputs class-specific posterior scores for each pixel, and the highest-scoring class determines the pixel-wise segmentation.The resulting segmented patches are stitched to reconstruct the original input image before object-level aggregation.
- Object-based CNN classification: 224x224px is the fixed VGG16 input size used for resized cropped objects in the object-based CNN workflow.The fixed-size requirement motivates warping each cropped object before CNN classification.
- Training strategy: Different patch strides are used across classes in MVFCNN cropping so that classes have equal numbers of patches despite dataset imbalance.The class with fewer training images receives a smaller stride than the class with more images.
5 Results
Experiments evaluate object-based and pixel-based microstructural classification on registered SEM and LOM steel images, using FCNN segmentation and max-voting for pixel-based classification. The pixel-based MVFCNN approach achieves the strongest reported classification performance, while SEM generally outperforms LOM.
- 5.1 Dataset: The dataset contains 21 registered LOM and SEM images, split into 11 training images and 10 test images.This produces 2831 training objects and 2262 test objects.
- 5.2 Task Definition and Metrics: The task classifies steel microstructural objects by their constituent substructure, with semantic segmentation evaluated using intersection-over-union metrics.Objects are defined relative to a ferritic matrix and foreground constituent, while pixels are classified before object-level aggregation in the pixel-based approach.
- 5.3 Evaluation: 93.94% overall accuracy is achieved by the best MVFCNN approach, compared with 48.89% for the previous state-of-the-art method.The confusion matrix excludes 48 not-segmented objects from this overall-accuracy calculation.
- 5.3 Evaluation: Pixel-based classification improves over object-based approaches by approximately 45 percentage points, whereas object-based methods improve over prior work by at most approximately 18 percentage points.The pixel-based method classifies segmented objects by the majority vote of their pixels.
- 5.3 Evaluation: Fine tuning together with data augmentation gives the best MVFCNN result, although the isolated effect of data augmentation is not significant.The evaluation tables report this pattern for the MVFCNN approach.
- 5.3 Evaluation: SEM images generally provide better segmentation and classification performance than LOM images, although FCNNs can learn pixel-wise classifications from low-resolution LOM images.SEM images also contain illumination and stitching artifacts, while matrix segmentation remains robust against noise from dirt, dust, and sample preparation.
6 Discussion
The discussion finds that MVFCNN-based pixel-wise segmentation outperforms object-based and hand-crafted approaches, while preserving texture information and avoiding resizing. Remaining errors concentrate in visually similar or underrepresented bainite-related cases.
- Resizing can distort object texture, whereas splitting large objects into 224x224 px patches improves performance.
- MVFCNN classification using SEM images is more efficient and accurate than object-based CNNs and hand-crafted features.Pixel-wise segmentation avoids resizing and its associated performance impact.
- 93.94% classification accuracy is achieved by the system described in the paper.
- Bainite has the lowest pixel-wise segmentation accuracy, while matrix has the highest.
- Bainite errors are associated with smaller objects, textured matrix regions, and transformed bainite absent from training data.Additional training data, especially for upper and lower bainite, is identified as a way to reduce misclassification.
7 Conclusion
The conclusion presents pixel-wise FCNN segmentation followed by max voting as an effective approach to steel microstructural classification. It reports strong accuracy, fast prediction, and reduced sensitivity to object size compared with resizing-based CNN pipelines.
- Pixel-wise microstructural segmentation using a trained FCNN is followed by a max-voting scheme.
- The approach performs classification without separate segmentation and feature extraction steps.
- MVFCNN avoids the loss of discriminative texture information caused by resizing objects for object-based CNNs.
- The method achieves high accuracy and very fast prediction.
9 Additional information
The paper reports no competing financial interests.
- The authors declare no competing financial interests.
- The disclosure states that the authors report no competing financial interests.
- The additional information contains a competing-interests declaration.