Source-linked AI summary
AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Use
Yaotian Yang, Yiwen Tang, Yizhe Chen, Xiao Chen, Jiangjie Qiu, Hao Xiong, Haoyu Yin, Zhiyao Luo, Yifei Zhang, Sijia Tao, Wentao Li, Qinghua Zhang, Yuqiang Li, Wanli Ouyang, Bin Zhao, Xiaonan Wang, Fei Wei
TL;DR
STEM images can resolve atoms, but converting a single image into a validated crystal structure remains difficult and existing tools leave gaps in complete structure reconstruction and property prediction. AutoMat combines agentic hypothesis search and closed-loop verification with perception and physics modules. On a 459-pair benchmark, it reports strong lattice, energy, and structure-matching results against existing approaches.
Problem
Although STEM can image atoms at sub-ångström resolution, translating images into quantitative crystal structures still relies on expert annotation, and existing approaches lack an end-to-end pipeline from STEM images to structures and predicted properties.
Source: §1 · pp. 1–2, §4.1 · p. 5Method
AutoMat uses inference-time hypothesis search and closed-loop verification to orchestrate denoising, retrieval-guided reconstruction, and machine-learned-potential energy evaluation.
Source: §1 · p. 2Results
On 459 image–structure pairs, AutoMat achieves 0.11 ± 0.03 Å projected RMSD, formation energy MAE below 350 meV/atom, and 83.2% atomic correspondence success, outperforming named multimodal and domain-specific baselines by an order of magnitude.
Source: §1 · p. 2Takeaways & Limitations
AutoMat integrates microscopy-driven structure reconstruction with property prediction, and the authors present STEM2Mat-Bench as a benchmark for the task.
Source: §6 · p. 9Even with the correct template, downstream steps fail in some cases because projection ambiguity and similar elemental contrast can lead to poor relaxation, inaccurate energies, or misclassification.
Source: §5.3 · p. 9
Abstract
from arXiv · showhide
Reconstructing atomistic crystal structures from a single noisy STEM projection is an ill-posed inverse problem: multiple lattices can explain similar contrast, and purely feed-forward models cannot verify physical validity. We present AutoMat, a failure-aware agentic controller that performs inference-time hypothesis search with closed-loop verification to convert Scanning Transmission Electron Microscopy (STEM) images into simulation-ready crystal structures and downstream properties. AutoMat composes perception and physics modules---pattern-adaptive denoising, physics-guided template retrieval as a state-dependent auxiliary branch, symmetry-constrained atomic reconstruction, and MLIP-based relaxation/validation---and triggers rollback-and-retry when verification fails. For systematic evaluation, we introduce STEM2Mat-Bench, a benchmark dataset containing 450+ annotated samples. Performance is assessed using lattice root-mean-square deviation (RMSD), formation energy mean absolute error (MAE), and structure matching accuracy. Results demonstrate that AutoMat outperforms existing approaches including SOTA models, specialized domain tools, and closed-source multimodal large models. This work establishes a direct pathway from microscopic characterization to atomic-scale modeling, addressing a fundamental challenge in materials science.
1. Introduction
AutoMat addresses the gap between atomic-resolution STEM imaging and quantitative, simulation-ready crystal structures with an agentic reconstruction workflow and a benchmark. It combines image processing, physics-guided reconstruction, and energy evaluation, and reports improved structural fidelity and downstream energy prediction against multimodal baselines.
1. Introduction
- STEM images resolve atoms at sub-ångström resolution, but translating them into quantitative crystal structures still depends on time-consuming expert annotation.The paper frames this as a gap between structural representation and theoretical validation.Source: §1 · p. 1
- Existing work is fragmented across denoising, atom localization, reconstruction, and phase classification, while many tools do not produce complete CIF structures or property predictions from STEM images.The stated gap includes a lack of standardized benchmarks for end-to-end structure–property pipelines.Source: §1 · p. 2
- On STEM2Mat, AutoMat achieves 0.11 ± 0.03 Å projected RMSD, formation energy MAE below 350 meV/atom, and 83.2% atomic correspondence success, outperforming several multimodal and domain-specific baselines by an order of magnitude.The benchmark uses 459 image–structure pairs selected from 2,143 curated monolayer structures.Source: §1 · p. 2
- AutoMat treats reconstruction as inference-time hypothesis search with closed-loop verification, composing denoising, retrieval-guided priors, reconstruction, and relaxation with rollback-and-retry after failed checks.The controller uses failure signals to adaptively retry steps.Source: §1 · p. 2
- MOE-DIVAESR adapts denoising to STEM image patterns, while STEM2CIF uses symmetry heuristics and physical constraints to infer a minimal repeating unit and produce a CIF.The workflow also evaluates relaxed structures with a machine-learned potential.Source: §1 · p. 2
- The paper introduces STEM2Mat with 2,143 simulated monolayer structures and a benchmark split with unified metrics for lattice accuracy, energy fidelity, robustness, and computational efficiency.Code, data, and evaluation tools are publicly available.Source: §1 · pp. 2–3
2. Related Work
Prior work advances microscopy analysis, structure reconstruction, multimodal modeling, and property prediction, but the cited approaches leave gaps in end-to-end conversion of STEM images into explicit crystal structures and predicted properties. AutoMat integrates these tasks and introduces a benchmark for evaluating the pipeline.
Automated Microscopy Image Analysis
- Microscopy analysis tools and multimodal models have distinct limits: AtomAI focuses on segmentation and identification, while MicroscopyGPT generates descriptions but not CIFs or property predictions.SciLink targets experiment–theory orchestration rather than dedicated reconstruction into explicit crystal structures for downstream simulations.Source: §2 · p. 3
STEM Image to Structure Reconstruction
- Earlier STEM reconstruction methods often require multiple images or prior structures, or are limited to single-element systems; few generate CIFs from one image of a complex multielement 2D crystal.Bayesian 3D reconstruction was demonstrated for simple single-element systems.Source: §2 · p. 3
Material Property Prediction Models
- Most property-prediction benchmarks use theoretical datasets rather than evaluating an end-to-end pipeline that starts from experimental images.Machine-learning interatomic potentials are presented as a way to connect imaging with computational property prediction.Source: §2 · p. 3
Summary
- AutoMat integrates STEM-image analysis with material-property prediction and establishes STEM2Mat to evaluate robustness and scalability of automated characterization.The paper positions this integration as addressing gaps across microscopy analysis, reconstruction, multimodal modeling, and property prediction.Source: §2 · p. 3
3. Dataset Construction
The dataset construction yields a curated collection of 2D crystals, simulated STEM images, and a tiered benchmark for evaluating reconstruction under varying material complexity and imaging conditions. Evaluation combines lattice, energy, composition, and structure-success measures.
3.1. Composition and Taxonomy
- The 2,143 curated monolayer crystals span six chemical families and 67 elements, including 76 unary, 1,409 binary, and 658 ternary systems.Structures were selected from C2DB, Materials Project, and OpenCrystal and stored as CIFs with validated lattice vectors and fractional coordinates.Source: §3.1 · p. 3
3.2. Image Simulation and Data Augmentation
- Synthetic iDPC-STEM images were generated from supercells at 0.1 Å/pixel with varied electron doses, lens aberrations, and Poisson detector noise.The image–structure–property samples include noisy projections, ground-truth CIFs, and DFT-level formation energies; structural fingerprints informed balanced train, validation, and test splits.Source: §3.2 · p. 4
3.3. STEM2Mat Benchmark Split and Tiering
- After geometric and chemical screening, 459 unique monolayer structures were retained for blind end-to-end evaluation, with the other 1,693 split into training and validation sets.The benchmark tiers vary composition and imaging noise to assess performance across task difficulty.Source: §3.3 · p. 4, Figure 2 · p. 5
3.4. Evaluation Metrics
- Evaluation reports formation-energy MAE and projected lattice RMSD alongside composition correctness and structure success, which requires both chemical and geometric validity.Formation-energy MAE reflects geometric reconstruction error and intrinsic MLIP prediction error; the benchmark also defines spatial similarity using projected atomic positions after optimal element-wise matching.Source: §3.4 · pp. 4–5
4. Overview of AutoMat
Existing multimodal and domain tools do not complete the path from a single real STEM image to a simulation-ready structure and formation energy. AutoMat combines modular tools with physics-based verification and rollback-and-retry.
4.1. Limitations of Existing Large Multimodal Models and Domain Tools
- Existing multimodal and domain tools cannot yet autonomously produce simulation-ready CIFs and formation energies from one real STEM image; AutoMat targets this end-to-end task.Source: §4.1 · p. 5
4.2. Tools Pool
- The tool pool pairs pattern-adaptive denoising with image-template retrieval, symmetry- and physics-constrained STEM2CIF reconstruction, and MatterSim relaxation and property evaluation.The controller composes these stages from a STEM image to a relaxed crystal with properties.Source: §4.2 · pp. 5–6, Figure 3 · p. 6
4.3. Flexible Tool-Calling Framework
- AutoMat searches tool states with physics-based verification and rollback-and-retry; it invokes template retrieval only when direct reconstruction fails or a stronger prior is needed.The LLM replans from structured tool outputs rather than processing pixels directly.Source: §4.3 · p. 6
5. Experiments and Results
AutoMat outperforms the reported baselines across the benchmark, including challenging Tier 3 cases, and shows preliminary transfer to a real ZSM-5 image. The results also quantify expert-time savings and identify retrieval and projection-related failure modes.
5.1. Quantitative Evaluation
- 332 ± 12 meV/atom formation-energy MAE; AutoMat remains above the 57 meV/atom MLIP lower bound but is stronger than MicroscopyGPT at about 620 meV/atom.Table 1 reports formation-energy MAE across tiers; lower is better.Source: §5.1 · p. 7, Table 1 · p. 7
- AutoMat reaches 0.11 Å projected RMSDxy, 83% composition correctness, and 83.2% structure success overall, outperforming the reported structural baselines.Table 2 defines RMSDxy, composition correctness (C.C.), and structure success rate (S.S.).Source: §5.1 · p. 7, Table 2 · p. 7
- Against a same-tool heuristic script, AutoMat improves overall Energy MAE from 385.42 to 321.57 meV/atom, C.C. from 76.5% to 83.2%, and S.S. from 74.2% to 83.2%.On Tier 3, Energy MAE falls from 585.00 to 333.49 meV/atom and S.S. rises from 62.3% to 73.1%.Source: Table 3 · p. 7, §5.1 · p. 8
- AutoMat outperforms all baselines and maintains strong results in Tier 3 cases with multi-element compositions and low imaging doses.The authors describe this as demonstrating robustness and generalizability across the benchmark.Source: §5.1 · p. 8
5.2. Real iDPC-STEM Case Study and Human-Expert Effort
- On a dose-limited real ZSM-5 image, AutoMat produced a CIF whose projected lattice and channel framework qualitatively agreed with the known model, providing preliminary evidence of transfer.The authors characterize this as transfer from 2D monolayer simulations to a 3D zeolite surface projection.Source: §5.2 · p. 8
- For Tier-2-like samples, AutoMat takes roughly 2 minutes per case, compared with about 6–8 hours for expert interpretation and CIF production.The comparison is reported for the authors’ benchmark hardware and a senior microscopy expert in their group.Source: §5.2 · p. 9
5.3. Error Analysis
- Failures comprise template retrieval errors (39.3%) and downstream errors despite a correct template (60.7%), including projection ambiguity and elemental confusion.The authors identify uncertainty-aware retrieval and 3D-aware modeling as directions for improvement.Source: §5.3 · p. 9
6. Conclusion
AutoMat integrates pattern-adaptive vision models, symmetry-constrained reconstruction, and LLM-driven orchestration to connect microscopy data with atomic modeling. The paper also proposes STEM2Mat-Bench for this task.
6. Conclusion
- AutoMat combines pattern-adaptive vision models, symmetry-constrained reconstruction, and LLM-driven orchestration to align microscopy data with atomic modeling and outperform existing methods in structural and energetic evaluation.The paper proposes STEM2Mat-Bench for evaluating this task.Source: §6 · p. 9
Impact Statement
AutoMat is presented as an assistive tool for reducing manual workload, not as a replacement for expert crystallographic analysis in high-stakes studies. The accompanying appendix describes the benchmark’s elemental coverage and the denoiser’s training pipeline.
Impact Statement
- AutoMat is intended to accelerate candidate generation, consistency checking, and downstream validation, while expert validation remains recommended, especially for out-of-distribution experiments.The paper warns that incorrect reconstructions may propagate into simulations and property predictions.Source: Impact Statement · p. 9
A.1. Dataset Overview
- The dataset has compact unit cells (median N = 6.0) and spans 67 unique elemental species, with coverage peaks among chalcogens and transition metals.Figure 5 presents the atom-count distribution and periodic-table heatmap.Source: §A.1 · p. 12
A. Training pipeline.
- The denoiser’s training pipeline uses eight PCA/k-means clusters, 756,042 patches, 99% validation routing accuracy, and about 50% lower reconstruction error than a single DIVAESR baseline.Six experts were trained jointly; the clusters were grouped into two parent and six child routing categories.Source: A. Training pipeline · p. 12
B. Model Architecture and Objective.
The section describes specialized denoising and super-resolution, template retrieval, and symmetry-constrained conversion into simulation-ready CIF structures. On real ZSM-5 images, AutoMat recovers the framework topology with reported atomic deviations within about 0.2 Å after alignment.
B. Model Architecture and Objective.
- MOE-DIVAESR routes inputs to specialized denoising-plus-super-resolution experts, with a joint objective supporting detail preservation, KL stabilization, and balanced expert use.Each expert is a two-stage cascade; the router selects an expert for each input.Source: B. Model Architecture and Objective · pp. 12–13
A. Image Template Matching.
- Template matching encodes detected atomic centroids as radial-distribution histograms, retrieves candidates, filters by elemental composition, and exports the closest remaining match.The retrieval stage uses the top-k nearest templates before elemental filtering.Source: A. Image Template Matching · p. 13
B. STEM2CIF Conversion.
- STEM2CIF localizes atomic columns, fits a symmetry-constrained lattice, assigns elements from simulated contrast, and exports a primitive-cell CIF for relaxation and property prediction.Lattice fitting uses a stated resolution of 0.1 Å/pixel.Source: B. STEM2CIF Conversion · pp. 13–14
A.8. MicroscopyGPT Baseline
- MicroscopyGPT is evaluated as an off-the-shelf vision–language baseline without further STEM2Mat-Bench fine-tuning; its textual outputs are parsed and converted to CIF.The stated reason for not fine-tuning is to avoid overfitting and keep baselines comparable.Source: §A.8 · p. 15
A.11. Real STEM Image Reconstruction on ZSM-5
- After alignment, framework atomic deviations on real ZSM-5 images are within ∼0.2 Å, with the reconstructed topology matching the reference.Figure 6 juxtaposes the experimental image, AutoMat reconstruction, and DFT-relaxed reference.Source: §A.11 · p. 16, Figure 6 · p. 17
C. Confidence Estimation and Warning Flags.
STEM2CIF assigns per-site confidence from local lattice-fitting quality and neighborhood geometry, then flags regions below a predefined threshold. The signal surfaces some projection ambiguity but does not resolve single-projection depth ambiguity.
C. Confidence Estimation and Warning Flags.
- Per-site confidence combines local fitting quality, represented by fitted R2, with nearest-neighbor distance; low scores trigger a warning for expert inspection before property analysis.Closely packed or overlapping columns tend to receive lower confidence than isolated, well-fitted atoms.Source: C. Confidence Estimation and Warning… · p. 14
- Closely packed iodine columns receive lower confidence than more isolated boron sites, but these signals do not fully resolve single-projection depth ambiguity.The method converts part of the ambiguity into a user-facing uncertainty signal.Source: C. Confidence Estimation and Warning… · p. 14
A.4. Ablation Study
Ablations show that retrieval is especially valuable for difficult cases, while leave-one-family-out tests indicate retained but family-dependent generalization. STEM2CIF itself is not meaningfully ablatable because it is the sole image-to-CIF component.
A.4. Ablation Study
- STEM2CIF cannot be meaningfully ablated: without its image-to-CIF conversion, RMSD, composition correctness, and formation-energy MAE are undefined.It is the only module that converts microscopy images into atomic structures as CIF files.Source: §A.4 · p. 14
- On Tier 3, removing template retrieval raises formation-energy MAE from 333 to 672 meV/atom, particularly affecting hard low-dose, multi-element cases.The authors characterize direct STEM2CIF reconstruction and retrieval-guided reconstruction as complementary paths.Source: §A.4 · p. 14, Table 5 · p. 14
- LOFO removes each held-out chemical family from training and the retrieval library; full-controller degradation is minimal for MXenes and moderate for perovskites.The results indicate meaningful generalization beyond seen families.Source: §A.4 · p. 14, Table 6 · p. 14
- For perovskites, LOFO-NoTemplate formation-energy MAE rises from 356 to 685 meV/atom, showing a larger drop without a correct retrieval prior.This setting measures the direct STEM2CIF path when no correct retrieval prior is available.Source: Table 6 · p. 14, §A.4 · p. 15
A.9. Robustness Across Agents and Retry
The appendix describes state-dependent tool use with retry and repair, and reports stable performance across alternative agent backbones. Its failure analysis separates absent-template retrieval failures from downstream failures despite correct retrieval.
A.9. Robustness Across Agents and Retry
- Tool invocation depends on intermediate quality and validity: reliable denoising can lead directly to STEM2CIF, while unreliable direct reconstruction can trigger retrieval.Environmental tool-error messages are fed into subsequent turns for retry and repair.Source: §A.9 · pp. 15–16
- In one documented run, validation succeeded after direct STEM2CIF reconstruction, so template retrieval was not invoked.The example illustrates that retrieval is not mandatory when the direct path succeeds.Source: §A.9 · p. 16
- Across a stratified 230-case sample, performance remains stable when the agent backbone is replaced by Qwen3, GPT-4o, or Qwen-VL (72B).The authors attribute the gains to the framework rather than a particular backbone.Source: §A.9 · p. 16
A.10. Failure Analysis
- Failure cases divide into 39.3% template-retrieval failures, where the correct structure is absent, and 60.7% downstream failures despite correct retrieval.Downstream cases are attributed to projection effects or extreme contrast imbalance.Source: §A.10 · p. 16
A.12. Limitations and Future Directions
AutoMat currently targets 2D monolayers imaged with iDPC-STEM, and its real-data validation remains preliminary. Future work addresses broader validation, 3D reconstruction, species discrimination, and controller robustness.
A.12. Limitations and Future Directions
- Real-data validation is limited to an initial ZSM-5 feasibility demonstration, not systematic evaluation across multiple real 2D monolayer materials.Public high-quality raw iDPC-STEM data are scarce, and new large-field, dose-limited data require substantial preparation, beamtime, and calibration.Source: §A.12 · p. 17
- The pipeline assumes a single iDPC-STEM projection and targets 2D monolayers, since a single projection cannot uniquely recover general 3D structure.Monolayers' limited z extent substantially reduces projection ambiguity, whereas bulk 3D reconstruction from one projection is intrinsically ill-posed.Source: §A.12 · pp. 17–18
- Element assignment relies on contrast cues and template priors, with greater confusion for similar-Z elements and under low-dose conditions.Source: §A.12 · p. 18
- Template retrieval becomes less robust when pixel scale, defocus, or aberrations are mismatched.Source: §A.12 · p. 18
- The current agent policy uses state-dependent rules that may not remain optimal as the tool pool grows.Source: §A.12 · p. 18
- Planned real-data improvements include evaluating more monolayers with known CIFs, broadening template coverage across imaging conditions, and calibrating pixel scale and drift.Source: §A.12 · p. 18
- For 3D extension, the authors are exploring combining multiple tilt-angle projections through tomography or using depth-sensitive signals from 4D-STEM.Source: §A.12 · p. 18
- Further plans include contrast-sensitive recognition with EELS/EDX fusion for species discrimination and learnable policy optimization with robust retrieval strategies.Source: §A.12 · p. 18