Source-linked AI summary
Deep-Learning-Based Image Segmentation Integrated with Optical Microscopy for Automatically Searching for Two-Dimensional Materials
Satoru Masubuchi, Eisuke Watanabe, Yuta Seo, Shota Okazaki, Takao Sasagawa, Kenji Watanabe, Takashi Taniguchi, Tomoki Machida
TL;DR
Practical implementation of image-recognition tools remains challenging. This paper integrates a Mask-RCNN-based segmentation algorithm with an automated optical microscope to search for 2D materials, enabling efficient detection of various exfoliated crystals, including some materials absent from training.
Problem
Despite advances in image-recognition algorithms, implementing these tools for practical applications remains challenging.
Method
The authors integrate deep-learning algorithms with an automated optical microscope to search for 2D materials on SiO2/Si substrates.
Results
The trained system enabled efficient detection of various exfoliated 2D crystals and correctly detected thin WSe2 and MoSe2 without training on those materials.
Takeaways & Limitations
An automated optical microscope can search for 2D flakes using deep-learning-assisted image analysis.
Takeaways & Limitations
The layer categorization into thin, medium, and thick ranges was considered sufficient for practical use.
Abstract
from arXiv · showhide
Deep-learning algorithms enable precise image recognition based on high-dimensional hierarchical image features. Here, we report the development and implementation of a deep-learning-based image segmentation algorithm in an autonomous robotic system to search for two-dimensional (2D) materials. We trained the neural network based on Mask-RCNN on annotated optical microscope images of 2D materials (graphene, hBN, MoS2, and WTe2). The inference algorithm is run on a 1024 x 1024 px2 optical microscope images for 200 ms, enabling the real-time detection of 2D materials. The detection process is robust against changes in the microscopy conditions, such as illumination and color balance, which obviates the parameter-tuning process required for conventional rule-based detection algorithms. Integrating the algorithm with a motorized optical microscope enables the automated searching and cataloging of 2D materials. This development will allow researchers to utilize unlimited amounts of 2D materials simply by exfoliating and running the automated searching process.
Introduction:
The work addresses the difficulty of deploying image-recognition tools for 2D-materials microscopy, where rule-based detectors require expert retuning as conditions change. It introduces a Mask-RCNN-based deep-learning system integrated with an automated optical microscope to search for 2D materials robustly.
- Motivation: Deep-learning deployment remains challenging because practical applications require jointly developed annotated datasets and software.This challenge persists despite advances in image-recognition algorithms.
- Motivation: Conventional rule-based 2D-materials detectors use handcrafted image features and require expert-adjusted parameters that must be retuned when microscopy conditions change.Deep-learning detectors are motivated by the prospect of generalized detection without fine-tuning these parameters.
- Contribution: The study integrates deep-learning algorithms with an automated optical microscope to search for 2D materials on SiO2/Si substrates.The system uses a Mask-RCNN architecture that detects exfoliated materials while generating a segmentation mask for each object.
- Method: The Mask-RCNN system uses transfer learning from COCO to support development from a relatively small dataset of approximately 2000 optical microscope images.The network generates a segmentation mask for each detected object.
- Result: The resulting detection process is robust to changes in microscopy conditions, a property not achieved by conventional rule-based image-recognition algorithms.This robustness follows from the neural network's generalization ability.
System architectures and functionalities:
The system combines a motorized optical microscope, software pipeline, and Mask-RCNN algorithms to detect and catalog 2D materials. It provides robust, real-time segmentation and supports automated searches for flakes on SiO2/Si substrates.
- System architectures and functionalities:: Mask-RCNN detected 2D flakes with good mask overlap, correctly classified monolayer graphene as “mono,” and resisted contamination from tape residue, particles, and corrugated flakes.The model outputs bounding boxes, class labels, confidences, and segmentation masks, with thickness categories of “mono,” “few,” and “thick.”
- System architectures and functionalities:: Deep-learning detection remained unchanged as illumination varied from I = 220 to 180 or 90, whereas rule-based detection failed when intensity decreased from I = 220 to 200 and 180.The comparison demonstrates greater robustness of the deep-learning algorithms under changing illumination conditions.
- System architectures and functionalities:: Automated searching identified approximately 25 WTe2 flakes with thicknesses of 1–10 layers on a 1 × 1 cm2 SiO2/Si substrate in 1 h.Detected flake positions and shapes are stored in a database for browsing and potential heterostructure assembly.
Model training:
The Mask-RCNN model was trained with staged optimization, augmentation, and transfer learning across multiple 2D materials. Data augmentation reduced overfitting, while shared backbone features improved convergence and test loss.
- Model training:: Optimization proceeded through four 30-epoch stages, progressing from training only network heads to training the backbone and entire model while reducing the learning rate from 10^-3 to 10^-5.Each epoch comprised 500 iterations.
- Model training:: Sharing the backbone across material-specific models contributed to faster convergence and a smaller test loss.The model was first trained on mixed graphene, hBN, MoS2, and WTe2 datasets, then transferred to each material subset for layer-thickness classification.
- Model training:: With data augmentation, training and validation losses decreased monotonically with a small difference, indicating that 2000 images enabled training without overfitting.Without augmentation, training loss reached zero while test loss increased, indicating growing generalization error.
Transfer learning from the trained model for accurate deep-learning inference models:
Transfer learning from a model pretrained on MS-COCO and multiple 2D-material classes accelerated convergence and reduced minimum loss across material-specific models. It also improved inference accuracy by detecting graphene flakes missed by MS-COCO-only transfer and avoiding WTe2 substrate misclassification.
- Transfer learning from the trained model for accurate deep-learning inference models:: The test loss decreased faster and reached a lower minimum with 2D-material and MS-COCO pretraining than with MS-COCO-only pretraining.After 30 epochs, the loss was nearly the same order as the MS-COCO-only model after 80 epochs.
- Transfer learning from the trained model for accurate deep-learning inference models:: For graphene, MS-COCO-only transfer missed some thick graphite flakes, whereas transfer including 2D materials detected them.The comparison used inference results from models initialized with the respective pretrained weights.
- Transfer learning from the trained model for accurate deep-learning inference models:: For WTe2, MS-COCO-only transfer misclassified surrounding SiO2/Si substrate as WTe2, while 2D-material transfer avoided that error.The substrate region surrounding thick WTe2 crystals was not recognized as WTe2 after transfer from the multi-class pretrained model.
- Transfer learning from the trained model for accurate deep-learning inference models:: Pretraining on MS-COCO and 2D materials improved model accuracy across material-specific inference tasks.The learned backbone features common to 2D crystals contributed to performance improvements for each material.
at epoch 120 for each material.
The trained neural network generalized across optical microscope setups with differing imaging conditions, successfully detecting exfoliated graphene despite not being trained on images from those instruments. This supports its use as a general-purpose graphene detector, unlike rule-based methods requiring parameter retuning when optical conditions change.
- Generalization across microscope setups: The model successfully detected exfoliated graphene across three optical microscope setups, despite none of those instruments’ images being used for training.The setups differed in white balance, magnification, resolution, illumination intensity, and illumination inhomogeneity.
- Generalization across microscope setups: The results indicate that the network captured latent general features of graphene flakes and functions as a general-purpose graphene detector irrespective of microscope setup.The model weights from training epoch 120 on the graphene dataset were used for inference.
- Comparison with conventional detection: Conventional rule-based detection algorithms require parameter retuning when optical conditions change, unlike the demonstrated setup-independent neural-network detector.
Discussion and summary:
The study establishes a Mask-RCNN-based automated optical-microscopy system that efficiently detects and segments multiple 2D crystals. Its generalization, transferable weights, and cloud-based deployment support broader automated materials searches and future heterostructure fabrication.
- Generalization to unseen materials: Thin WSe2 and MoSe2 flakes were correctly detected using a model trained on WTe2, demonstrating generalization to materials absent from training.The result indicates that appearance differences between these materials are covered by the network’s generalization ability.
- Deployment and accessibility: Remote server/client inference provides a cloud-based setup without requiring a local GPU, while compatible conventional microscopes can be adapted with client software.The client captures images, sends them to the server, receives inference results, and displays them.
- Broader impact: Distributing the inference system can save time in optical-microscopy-based searches and represents a step toward fully automated van der Waals heterostructure fabrication.The microscope, inference system, searchable database, and CAD software together support browsing crystals and designing heterostructures.
- Automated detection and segmentation: The Mask-RCNN system efficiently detects graphene, hBN, WTe2, and MoS2 while generating segmentation masks for each exfoliated crystal.The automated microscope searches 2D crystals on SiO2/Si substrates and reduces repetitive optical-microscopy tasks.
Optical microscope drivers:
The automated optical microscope drivers were implemented in C++ and Python using a robotic operating system stack and the HALCON image-processing library.
- The microscope drivers used C++ and Python, with software built on a robotic operating system stack and the HALCON image-processing library.
Preparation of training dataset:
A semi-automatic annotation workflow combined preliminary Mask-RCNN predictions with human correction to prepare the training dataset for segmenting 2D crystals. This procedure improved annotation efficiency, reducing labeling time to 20– per image.
- Preparation of training dataset:: A semi-automatic workflow trained Mask-RCNN on ~80 graphene images, generated prediction labels for optical microscope images, and had a human annotator manually correct them.Prediction labels were stored in LabelBox using an API before manual correction.
- Preparation of training dataset:: The workflow greatly enhanced annotation efficiency, allowing each image to be labeled in 20–.
(Supplementary Information)
The supplementary information documents the server/client integration architecture, representative detection outcomes and errors, data augmentation examples, and inference-based detection of additional 2D materials.
- System integration: The server/client architecture integrates the deep-learning algorithms with an automated optical microscope for searching 2D flakes.The architecture is presented specifically for automated optical-microscope searching.
- Detection examples: Supplementary examples show true-positive, false-positive, and false-negative detections for WTe2 and graphene flakes.The examples include annotated false negatives and 10 µm scale bars.
- Data augmentation: Data augmentation examples illustrate transformed optical microscope images containing WTe2 and graphene flakes.The examples show input images alongside augmented images with 20 µm scale bars.
- Additional materials: Deep-learning inference also detects WSe2 and MoSe2 flakes in supplementary optical microscope examples.Both supplementary figures document detections of these additional materials.
Robustness against variations in magnification
The inference remained robust across objective-lens magnifications of 25× and 50×, detecting monolayer graphene flakes at both magnifications.
- Magnification robustness: Inference results were obtained from optical microscope images acquired with 25× and 50× objective lenses.
- Magnification robustness: Monolayer graphene flakes were detected at both 25× and 50× objective-lens magnifications.The evaluation used optical microscope images and corresponding inference results for exfoliated graphene on SiO2/Si.
- Magnification robustness: The magnification comparison used images with 10 µm scale bars.
Fabrication of van der Waals heterostructures using 2D crystals detected by deep
Deep-learning-detected WTe2 crystals were used to fabricate van der Waals heterostructures with graphene and hBN encapsulation.
- Fabrication of van der Waals heterostructures using 2D crystals detected by deep: The fabricated heterostructures used deep-learning-detected WTe2 crystals in contact with graphene flakes and encapsulated between hBN flakes.The optical images show the resulting device structures.
- Fabrication of van der Waals heterostructures using 2D crystals detected by deep: One device contained trilayer WTe2, while another contained five-layer WTe2, with both contacting graphene flakes.The optical microscope images distinguish the two WTe2 thicknesses.
- Fabrication of van der Waals heterostructures using 2D crystals detected by deep: The optical microscope images used 4 µm scale bars.The scale bars apply to the reported heterostructure images.