Source-linked AI summary
Zero-shot Autonomous Microscopy for Scalable and Intelligent Characterization of 2D Materials
Jingyun Yang, Ruoyan Avery Yin, Chi Jiang, Yuepeng Hu, Xiaokai Zhu, Xingjian Hu, Sutharsika Kumar, Xiao Wang, Xiaohua Zhai, Keran Rong, Yunyue Zhu, Tianyi Zhang, Zongyou Yin, Jing Kong, Neil Zhenqiang Gong, Zhichu Ren, Haozhe Wang
TL;DR
Manual microscopy and the training-data demands of supervised models complicate characterization of emerging 2D materials. ATOMIC combines foundation models for autonomous, zero-shot microscopy and analysis. It reports >99% accuracy without training, with robustness across imaging conditions and material species.
Problem
Microscopy relies on manual inspection, while supervised methods demand high-quality training data that may be infeasible for newly discovered materials.
Source: Main · p. 1Method
ATOMIC uses GPT to control microscopy, SAM to generate masks, and GPT-supervised k-means clustering to analyze 2D materials without training.
Source: Discussion · p. 5Results
>99% accuracy is reported for autonomous characterization without training, with robustness across imaging conditions, fabrication methods, and material species.
Source: Discussion · p. 5Takeaways & Limitations
The authors present a scalable, data-efficient approach with potential to support closed-loop materials discovery and research beyond materials science.
Source: Discussion · p. 5SAM-generated masks can overlap or contain topological holes, causing inaccurate RGB computation when few-layer structures occur within monolayer regions.
Source: S4 Importance of Topological… · p. 17
Abstract
from arXiv · showhide
Characterization of atomic-scale materials traditionally requires human experts with months to years of specialized training. Even for trained human operators, accurate and reliable characterization remains challenging when examining newly discovered materials such as two-dimensional (2D) structures. This bottleneck drives demand for fully autonomous experimentation systems capable of comprehending research objectives without requiring large training datasets. In this work, we present ATOMIC (Autonomous Technology for Optical Microscopy & Intelligent Characterization), an end-to-end framework that integrates foundation models to enable fully autonomous, zero-shot characterization of 2D materials. Our system integrates the vision foundation model (i.e., Segment Anything Model), large language models (i.e., ChatGPT), unsupervised clustering, and topological analysis to automate microscope control, sample scanning, image segmentation, and intelligent analysis through prompt engineering, eliminating the need for additional training. When analyzing typical MoS2 samples, our approach achieves 99.7% segmentation accuracy for single layer identification, which is equivalent to that of human experts. In addition, the integrated model is able to detect grain boundary slits that are challenging to identify with human eyes. Furthermore, the system retains robust accuracy despite variable conditions including defocus, color temperature fluctuations, and exposure variations. It is applicable to a broad spectrum of common 2D materials-including graphene, MoS2, WSe2, SnSe-regardless of whether they were fabricated via chemical vapor deposition or mechanical exfoliation. This work represents the implementation of foundation models to achieve autonomous analysis, establishing a scalable and data-efficient characterization paradigm that fundamentally transforms the approach to nanoscale materials research.
Main
ATOMIC combines foundation models to automate optical microscopy and characterize 2D materials without labeled training data. It reports high accuracy, including under imaging variations and across several materials.
Main
- ATOMIC combines GPT microscope control, zero-shot SAM masks, and GPT-supervised k-means clustering to automate microscopy and 2D-material analysis.The framework uses prompts rather than labeled training data.Source: Main · p. 2, Large Model Controlled Microscopy · p. 2
Spatial Segmentation
- Topological correction separates overlapping SAM masks, raising classification accuracy from 77.92% to 100% for typical images.The correction subtracts an internal multilayer mask from a combined monolayer–multilayer mask.Source: Spatial Segmentation · p. 3
LLM-supervised Clustering
- GPT-4o matched human cluster-count choices on 29 of 35 images (82.9%), versus 11 of 35 (31.4%) for the Elbow method.In one example, GPT-assisted selection raised mask classification accuracy from 67.80% to 98.30%.Source: LLM-supervised Clustering · p. 4
Robustness
- Pixel-level accuracy remained above 99.5% across defective imaging conditions, including defocus, white-balance variation, and exposure inconsistencies.The original image had 95.7% mask-level accuracy.Source: Robustness · p. 4
- The system identifies layer regions and challenging features across exfoliated MoS2, PVD SnSe, CVD WSe2, and CVD graphene.Detected features include adhesive residue, folded regions, and in-plane wrinkles.Source: Robustness · p. 4
- ATOMIC’s zero-shot approach achieves 99.7% pixel-level accuracy (95% CI: 99.4%–100.0%) without training data, outperforming most conventional models and maintaining capability under challenging imaging conditions and across 2D material species.Source: Robustness · p. 5
Discussion
The authors frame ATOMIC as a data-efficient, scalable approach to 2D-material characterization, with potential relevance to materials discovery and adjacent scientific fields.
Discussion
- ATOMIC combines GPT microscope control, SAM masks, and GPT-supervised k-means clustering for autonomous characterization without training.The framework is reported to achieve >99% accuracy, detect subtle features, and remain robust across imaging conditions, fabrication methods, and material species.Source: Discussion · p. 5
- The authors describe the framework as scalable and data-efficient, and suggest foundation models may support closed-loop materials discovery.They also note potential relevance beyond materials science.Source: Discussion · p. 5
Methods
The methods connect microscope control, SAM segmentation, prompt-guided analysis, and fabrication of samples spanning several 2D materials.
Physical Controlling of the Microscopy
- A Python interface links the Zeiss microscope to ZEN software, while GPT function calling translates prompts into executable control commands.Function descriptions are converted into GPT-compatible JSON schemas, and imaging parameters are tuned through ZEN.Source: Physical Controlling of the Microscopy · p. 5
Spacial Segmentation by SAM
- SAM segments raw RGB microscopy images, then overlap-based filtering removes redundant masks while preserving layer inclusion relationships.The resulting masks preserve hierarchical relationships among monolayer, few-layer, and multilayer structures.Source: Spacial Segmentation by SAM · p. 6
Prompt Engineering
- Structured prompts guide microscope control and region search, while a clustering prompt uses calibrated substrate RGB and the Elbow Method to select cluster counts.The prompts specify operational context, spatial parameters, and classification criteria.Source: Prompt Engineering · p. 6
Materials Fabrication
- The study fabricated MoS2, SnSe, WSe2, and graphene samples using CVD, mechanical exfoliation, or PVD procedures.The methods describe CVD MoS2, mechanically exfoliated MoS2, PVD SnSe, CVD WSe2, and LPCVD graphene.Source: Materials Fabrication · pp. 6–7
Contributions
The authors describe their respective roles in developing the microscopy-control, segmentation, and clustering modules, fabricating samples, and preparing the manuscript.
Contributions
- H.W. and Z.R. conceived the study and supervised the work.J.Y., R.A.Y., Z.R., and H.W. developed the microscope-control module.Source: Contributions · p. 7
- J.Y., Y.H., S.K., N.Z.G., and H.W. developed the segmentation module; J.Y., C.J., and H.W. developed GPT-supervised clustering.Source: Contributions · p. 7
- X.Z. (Zhu), X.H., Y.Z., T.Z., C.J., J.K., and H.W. fabricated the characterized materials.J.Y., R.A.Y., X.H., X.Z. (Zhu), Z.R., and H.W. drafted the manuscript; all authors reviewed and revised it.Source: Contributions · p. 7
Supplementary Information Zero-shot Autonomous Microscopy for Scalable and
The microscope-control pipeline converts natural-language instructions into instrument operations through a two-tier GPT agent architecture and standardized API interactions.
S1 Physical Controlling of the Optical Microscope
- CallingGPT converts docstring-annotated functions into GPT-compatible JSON, enabling the two-tier agents to translate natural-language commands into microscope operations.The pipeline uses standardized API interactions.Source: S1 Physical Controlling of the Optical… · p. 16
- The front-end GPT agent extracts high-level commands from natural-language input and sends them to the microscope system through a Flask-based web framework.The structured commands support downstream function-calling.Source: S1 Physical Controlling of the Optical… · p. 16
- The back-end GPT agent maps structured commands to microscope functions using CallingGPT-generated JSON schemas and executes functions such as autofocus and stage movement.GPT function-calling dynamically selects the relevant function.Source: S1 Physical Controlling of the Optical… · p. 16
S2 Accuracy analysis of our foundation model synergy
Monolayer identification reaches 99.7% pixel-level accuracy, while mask-level accuracy varies depending on whether masks are aggregated directly or averaged across images. SAM-based segmentation corrects errors in conventional clustering and contour workflows, and topological correction improves classification in an example with overlapping masks.
S2 Accuracy analysis of our foundation model synergy
- Monolayer precision was 99.4%, while Multilayer recall was 93.5%; confusion between these classes was the primary source of errors.Substrate precision and recall were both 100%, and the Other class had over 95% precision and recall.Source: S2 Accuracy analysis of our foundation… · p. 16
- 99.7% pixel-level accuracy, 97.6% direct-aggregation mask-level accuracy, and 96.2% average mask-level accuracy (95% CI: [93.5%, 98.9%]) were reported for monolayer identification.The evaluation covered 1,337 masks across 28 images.Source: S2 Accuracy analysis of our foundation… · p. 16
S3 Comparison of SAM Segmentation vs. Traditional Computer Vision (CV) Methods
- With k = 4, direct pixel clustering split the substrate into two categories yet still failed to distinguish impurities; overexposure also produced a circular image region.The four intended categories were substrate, monolayer, multilayer, and impurities.Source: S3 Comparison of SAM Segmentation vs.… · p. 17
- SAM-based segmentation corrected contamination and impurity assignments missed by direct clustering, while contour detection on clustered maps omitted multilayers and small flakes.Clustering errors also propagate into contour detection, whereas SAM delineates boundaries before classification.Source: S3 Comparison of SAM Segmentation vs.… · p. 17
S4 Importance of Topological Correction on SAM Generated Masks
- 100% example-image classification accuracy, up from 77.92%, followed topological correction that separates overlapping masks for downstream classification.The correction removes overlap portions from original masks and stores them as new masks.Source: S4 Importance of Topological… · pp. 17–18
S5 Comparison of RGB Color Space vs. HSV Color Space
HSV clustering can distinguish light-yellow impurities in complex CVD-grown MoS2 images but does not separate monolayer from multilayer regions. RGB clustering separates those layer regions, with some impurity–substrate confusion that does not hinder analysis of layer coverage and growth patterns.
S5 Comparison of RGB Color Space vs. HSV Color Space
- HSV-based clustering distinguishes light-yellow impurities in CVD-grown MoS2 images, though not perfectly, but fails to differentiate monolayer and multilayer regions.The layer-separation limitation was observed across multiple images.Source: S5 Comparison of RGB Color Space vs.… · p. 18
- RGB clustering separates monolayer regions from internal multilayer structures, despite some confusion between light-yellow impurities and substrate.That confusion does not hinder analysis of monolayer coverage and multilayer growth patterns.Source: S5 Comparison of RGB Color Space vs.… · p. 18
- For studying MoS2 layer distribution and morphology in optical microscopy images, RGB clustering is more suitable than HSV clustering.HSV misses monolayer–multilayer distinctions, while RGB separates them despite impurity–substrate confusion.Source: S5 Comparison of RGB Color Space vs.… · p. 18
S6 Analysis of RGB Channel Importance
RGB-channel perturbations affect different class boundaries: R noise confuses substrate and monolayer, while B noise disrupts monolayer–multilayer separation; G noise is generally less disruptive. The Elbow method can overfit at high k, while GPT and Elbow can both match human classifications on simpler images.
S6 Analysis of RGB Channel Importance
- At 7%, G-channel noise introduces some monolayer-to-multilayer confusion, although overall clustering remains largely consistent.At 3% and 5%, G-channel perturbations have relatively minor effects on clustering.Source: S6 Analysis of RGB Channel Importance · pp. 18–19
- 7% B-channel noise makes monolayer and multilayer regions almost indistinguishable by obscuring their boundary.At 5%, B-channel noise also causes substantial monolayer-to-multilayer misclassification, associated with similar B-channel values in the two classes.Source: S6 Analysis of RGB Channel Importance · pp. 18–19
- R-channel noise primarily confuses substrate and monolayer, while B-channel noise primarily disrupts monolayer and multilayer distinction.At 5% R noise, substrate and monolayer points are misclassified in both directions; at 7%, confusion extends to layer distinction.Source: S6 Analysis of RGB Channel Importance · pp. 18–19
S7 Classification Comparison between Elbow Method and GPT supervision
- With k=5, the Elbow method unnecessarily splits multilayer and substrate regions, while k=3 gives a clearer classification aligned with human judgment.The example illustrates how excessive k can overfit and reduce interpretability.Source: S7 Classification Comparison between… · p. 19
- On simpler images with well-separated regions, Elbow and GPT selections can both agree with human annotations, but agreement does not guarantee correctness.The Elbow method may still struggle on more challenging or noisy data, where GPT’s adaptability provides an advantage.Source: Figure S11d
S8 AFM Characterization of Thickness
AFM height profiles support ATOMIC’s classification of monolayer, bilayer, and multilayer structures.
S8 AFM Characterization of Thickness
- AFM height profiles confirm the distinction between monolayer, bilayer, and multilayer structures in the classification results.The profiles show first- and second-layer thicknesses corresponding to theoretical values, and multilayer thickness above 100 nm.Source: S8 AFM Characterization of Thickness · p. 19
S9 Supplementary Methods
The supplementary methods describe SAM mask generation and topological correction, alongside GPT prompts for microscope scanning and clustering decisions.
S9.1 Spacial Segmentation by SAM and topological analysis
- SAM mask generation uses IoU, stability, and refinement settings, including a 0.88 prediction-IoU threshold and a 0.95 stability threshold.The configuration also sets box NMS and crop NMS thresholds to 0.7 and retains small masks by setting the minimum mask region area to 0.Source: S9.1 Spacial Segmentation by SAM and… · pp. 19–20
- Topological correction takes binary masks and a minimum-overlap percentage as input and returns filtered masks, updating overlaps and removing marked masks.The algorithm skips masks already marked as None and adds overlap regions to new masks.Source: S9.1 Spacial Segmentation by SAM and… · p. 20
S9.2 Prompt Engineering for GPT
- GPT prompts specify a stepwise scan, image analysis after each movement, mask-area-ratio calculations, and a move to the position with the highest requested-category ratio.A separate prompt asks GPT to choose k from distortion values and percentage changes while avoiding impractical over-segmentation.Source: S9.2 Prompt Engineering for GPT · pp. 20–21
S10 Supplementary Figures
Supplementary figures illustrate microscope control, segmentation and clustering comparisons, noise tests, and AFM layer characterization.
S10 Supplementary Figures
- Figure S1 presents the workflow of microscope control enabled by GPT.Figure S2 compares SAM predictions with human labels across multiple masks.Source: Figure S1 · p. 22, Figure S2 · p. 22
- Figure S4 compares direct clustering and contour detection with SAM segmentation and SAM-based clustering on a CVD-grown MoS2 image.The figure highlights boundary preservation, segmentation accuracy, incorrect clusters, and missed edge detections.Source: Figure S4 · p. 23
- Figure S5 compares clustering with and without topological correction, showing substrate–monolayer confusion and poorly distinguished fine structures without correction.The corrected result adds a few-layer category and better separates monolayer, multilayer, and substrate regions.Source: Figure S5 · p. 24
- Figures S7–S10 show RGB clustering and results after adding 3%, 5%, and 7% Gaussian noise to individual color channels.The noise-test confusion matrices use the noise-free clustering result in Figure S7b as ground truth.Source: Figure S7 · p. 24, Figure S8 · p. 25, Figure S9 · p. 26
- Figures S12–S13 pair AFM images of MoS2 and WSe2 with height profiles for layer characterization.Figure S12 covers monolayer, bilayer, and multilayer MoS2; Figure S13 shows monolayer and bilayer WSe2.Source: Figure S12 · p. 28, Figure S13 · p. 29
S11 Supplementary Tables
Supplementary Tables S1 and S2 report ATOMIC confusion matrices at the mask and pixel levels.
S11 Supplementary Tables
- Table S1 presents the mask-level confusion matrix for ATOMIC.Source: Table S1 · p. 30
- Table S2 presents the pixel-wise confusion matrix for ATOMIC.Source: Table S2 · p. 30