Source-linked AI summary
Overview: Computer vision and machine learning for microstructural characterization and analysis
Elizabeth A. Holm, Ryan Cohn, Nan Gao, Andrew R. Kitahara, Thomas P. Matson, Bo Lei, Srujana Rao Yarasi
TL;DR
Microstructural quantification traditionally requires humans to choose measurements and devise purpose-built methods, despite its foundational role in linking structure with material behavior. This overview surveys CV methods that encode microstructural images for supervised or unsupervised ML across major image-analysis tasks. It reports that these tools support richer visual metrics and processing-microstructure-property relationship discovery, while interpretability remains an open limitation.
Problem
Traditional microstructural quantification depends on case-specific human choices about what to measure and how to measure it.
Method
The paper surveys CV and ML applications that numerically represent microstructural images and analyze them through classification, segmentation, and related image-analysis tasks.
Results
CV/ML systems support rich quantitative microstructural characterization, including high-accuracy semantic segmentation and visual similarity analysis.
Takeaways & Limitations
These tools provide new approaches for developing visual metrics and discovering processing-microstructure-property relationships.
Takeaways & Limitations
The basis for CV/ML decisions remains difficult to interpret generally, especially when drawing processing-microstructure-property connections.
Abstract
from arXiv · showhide
The characterization and analysis of microstructure is the foundation of microstructural science, connecting the materials structure to its composition, process history, and properties. Microstructural quantification traditionally involves a human deciding a priori what to measure and then devising a purpose-built method for doing so. However, recent advances in data science, including computer vision (CV) and machine learning (ML) offer new approaches to extracting information from microstructural images. This overview surveys CV approaches to numerically encode the visual information contained in a microstructural image, which then provides input to supervised or unsupervised ML algorithms that find associations and trends in the high-dimensional image representation. CV/ML systems for microstructural characterization and analysis span the taxonomy of image analysis tasks, including image classification, semantic segmentation, object detection, and instance segmentation. These tools enable new approaches to microstructural analysis, including the development of new, rich visual metrics and the discovery of processing-microstructure-property relationships.
1 INTRODUCTION: THE QUANTIFICATION OF MICROSTRUCTURE
Microstructural quantification connects structure to composition, processing, and properties, but traditional analysis depends on human choices about what to measure and how. CV and ML offer quantitative, objective approaches that encode image information and identify patterns across diverse microstructures.
- Motivation: Microstructural science links visual structure to material composition, processing history, and properties, beginning with Sorby’s qualitative observations of steel.Modern analysis still often relies on qualitative descriptors such as equiaxed, aligned, rough, dispersed, ordered, and columnar.
- The quantification problem: An overarching microstructural metric remains elusive because individual metrics are developed and applied case by case.The difficulty reflects the diversity of structures and the absence of a universally appropriate measurement.
- What to measure: Selecting what to measure requires judgment because different structures may make grain size, aspect ratio, dendrite spacing, colony size, or porosity the more relevant metric.The relevant feature may depend on the process or property of interest, and multiple metrics may contribute simultaneously.
- How to measure it: Measuring selected features can require difficult segmentation, especially when boundaries are fuzzy or absent from optical images.Examples include carbon-denuded-zone boundaries and grain boundaries that fail to etch deeply enough to form visible dark lines.
- CV and ML: Computer vision aggregates pixel values into high-dimensional image representations, while supervised and unsupervised machine learning find correlations in that representation.Supervised systems use human-determined ground truths; unsupervised methods identify structure without those labels.
- CV and ML: CV and ML can capture richer microstructural information quantitatively, objectively, and generally, assisting, improving, or replacing ad hoc characterization methods.The overview presents applications across varied microstructural image-analysis tasks.
2 METHODS
CV converts microstructural images into numerical feature vectors, while ML extracts classifications, measurements, associations, and other quantitative information from those representations. The methods surveyed include feature-based and CNN representations, supervised and unsupervised learning, and data aggregation for materials discovery.
- 2.1 Computer Vision: Create a numerical representation: CV creates numerical feature vectors by aggregating visual information from image pixels.Feature-based methods use expert-selected filters and descriptors, whereas CNN filters are learned during training.
- 2.1 Computer Vision: Create a numerical representation: CNNs encode images through hierarchical convolutions, rectification, pooling, and feature-learning operations before a classifier or other ML method draws conclusions.Pooling down-samples representations, and the resulting feature-learning stage precedes classification.
- 2.1 Computer Vision: Create a numerical representation: Whole-image CNN feature vectors can be taken from individual layers, while hypercolumn vectors combine activations across layers for each pixel and multiple length scales.Shallow layers represent local environments, whereas deeper layers capture larger length scales; hypercolumns retain information across scales.
- 2.1 Computer Vision: Create a numerical representation: Transfer learning applies CNNs trained on natural images to microstructures, reducing the need to train a CNN from scratch on microstructural data.This transferability is attributed to shared visual features such as edges, blobs, and textures.
- 2.2 Machine Learning: Extract quantitative visual information: ML extracts classifications, metadata associations, measurements, and feature-presence information from high-dimensional feature vectors using supervised or unsupervised approaches.Supervised methods use ground truth; unsupervised methods identify patterns without ground-truth data or human intervention.
- 2.3 Data: The basis for data science: Aggregating and cross-referencing image metadata—including composition, imaging information, processing history, and property measurements—supports discovery across data sources.The metadata may include material system and other image-related data.
3 RESULTS: A TAXONOMY OF MICROSTRUCTURAL ANALYSIS
The paper organizes microstructural image analysis into classification, semantic segmentation, object detection, and instance segmentation tasks. CV/ML feature representations support visual search, classification, direct measurement, and segmentation workflows, while performance varies with image complexity and constituent ambiguity.
- Analysis taxonomy: The taxonomy comprises image classification, semantic segmentation, object detection, and instance segmentation.These tasks respectively identify image content, assign pixels to constituents, locate individual objects, and assign pixels to individual objects.
- Image classification and feature vectors: Feature-vector distances support visual search and clustering, and a linear SVM classified ultrahigh carbon steel microconstituents with 99±1% accuracy.A database of 961 microstructures showed visually similar images near one another in feature space, while t-SNE clusters corresponded to primary microstructural constituents.
- Image classification and feature vectors: Deep regression predicted grain size without segmentation, with 2.3% standard error for perfect polycrystals and 3.9% when 40% of boundaries were missing.The authors report that this measurement approach creates opportunities for fault-tolerant, high-throughput evaluation.
- Semantic segmentation: Multipart CNN segmentation separated dislocation features from precipitate boundaries by combining feature and region-mask segmentations.On a small nickel-based-superalloy dataset, the system initially struggled to learn which single-pixel-wide linear features to ignore.
- Object detection and instance segmentation: Instance segmentation handled dense, overlapping powder particles and achieved 97.5% average precision and 95.4% recall, while satellite-particle identification reached 69.2% precision and 54.5% recall.The lower satellite-particle scores reflected greater subjectivity in identifying satellites than ordinary particle edges.
4 NEXT STEPS: METRICS, APPLICATIONS, AND INTERPRETATION
CV/ML enables quantitative microstructural metrics that capture visual characteristics beyond traditional measurements, while open challenges include 3D reconstruction and interpreting learned visual signatures.
- 4.1 Novel and advanced microstructural metrics: CV/ML groups powder particles by visual similarity and produces quantitative fingerprints capturing size, shape, roughness, and agglomeration.For Inconel-618 particles, k-means statistics and t-SNE density maps provide information beyond traditional powder size distributions.
- 4.1 Novel and advanced microstructural metrics: These particle-level representations support new metrics because computers can process thousands of image patches more objectively and repeatably than humans.The approach combines repetitive computation with human-like visual judgment learned by CV/ML methods.
- 4.2 Unsupervised analysis: Unsupervised analysis can derive image metrics without requiring tedious or sometimes impossible ground-truth annotation.The paper identifies particle cluster statistics from k-means as one route to an unsupervised image metric.
- 4.1 Novel and advanced microstructural metrics: Inferring 3D structure from 2D microstructural images remains an open question for CV/ML approaches.Quantitative stereology remains the conventional framework for this problem.
- 4.3 Interpretability: Opening the black box: Interpretability remains important for analysis tasks that seek processing-microstructure-property connections, because the visual basis of model decisions may be unknown.The paper discusses feature-vector characteristics, filter activations, and salient image regions as partial interpretability approaches.
5 CONCLUSIONS
The overview presents CV/ML as a framework for numerically encoding microstructural images and applying supervised or unsupervised learning across major image-analysis tasks. These systems support richer quantitative characterization and potentially new processing-microstructure-property relationships.
- 5 CONCLUSIONS: CV converts microstructural images into feature vectors that ML algorithms use to find associations and trends in high-dimensional visual representations.The surveyed task taxonomy includes classification, semantic segmentation, object detection, and instance segmentation.
- 5 CONCLUSIONS: Feature-vector similarity enables visual search, sorting, and classification of micrographs.This uses numerical similarity between image representations rather than relying only on predefined measurements.
- 5 CONCLUSIONS: CV/ML can extract information not readily visible to humans, including chemical composition encoded latently in SEM image feature vectors.The approach uses latent information in the learned representation.
- 5 CONCLUSIONS: Semantic segmentation can assign microstructural constituents to pixels with high accuracy and human-like judgment about what to consider or ignore.The capability targets constituent-level characterization in complex images.
- 5 CONCLUSIONS: CV/ML can combine segmentations from different feature types and locate or segment individual objects, including overlapping objects.These capabilities include bounding instances and enabling new object-level image analyses.
- 5 CONCLUSIONS: Across applications, computational systems provide accurate, autonomous, objective, repeatable, and continuously available analyses.The overview links these capabilities to richer visual metrics and discovery of processing-microstructure-property relationships.