PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial-Frequency Learning for Scanning Transmission Electron Microscopy Image Enhancement

Hesong Li, Ying Fu, Ziqi Wu, Ruiwen Shao, Tao Zhang

arXiv:2610.36487v1cs.CV

TL;DR

STEM images are often obscured by noise and blur, and existing approaches and synthetic datasets have gaps in handling periodic structures and realistic degradations. The paper combines spatial-frequency fusion with calibrated, diverse training data; its AtoMix-trained network produces clearer real-image results across atoms than networks trained on other datasets.

  • Problem

    STEM images can be overwhelmed by noise and blurred by spherical aberration, while existing datasets lack realistic noise and diverse arrangements, imaging modes, and degradations.

    Source: §I · pp. 1–2

  • Method

    The paper combines spatial-frequency fusion that uses periodicity priors with noise calibration from real STEM images and a diverse dataset of arrangements, modes, and degradations.

    Source: §I · p. 2

  • Results

    SFFN trained on AtoMix produces clear results for all atoms in real STEM images, whereas results from other datasets are blurred and fail to recover relatively dark atoms.

    Source: C. Discussion · p. 12

  • Takeaways & Limitations

    Frequency-position guidance enhances periodic signals, while joint spatial- and frequency-domain modeling captures atomic arrangements and local irregularities for image enhancement.

    Source: C. Discussion · p. 11, §VI · p. 14

Abstract

from arXiv · show

Scanning Transmission Electron Microscopy (STEM) enables the observation of atomic arrangements at sub-angstrom resolution and plays a key role in the discovery and characterization of advanced materials. Raw STEM images are often noisy and blurred due to multiple degradations. Although recent deep learning methods improve STEM images, their performance remains limited under severe degradations. In this work, we propose a spatial-frequency learning framework for STEM image enhancement. Specifically, we design a spatial-frequency fusion network that exploits periodic atomic arrangements in the frequency domain and restores local atomic details in the spatial domain. Spatial- and frequency-domain features are enhanced separately and then fused. We also present a STEM noise calibration method for synthesizing more realistic STEM images. The parameters of scan noise, pointwise noise, and background noise are estimated through statistical analysis and fitting of real STEM images. Using these calibrated parameters, we construct a diverse dataset containing a wider range of atomic arrangements, imaging modes, and degradations. Random masks combine different atomic arrangements to simulate interfaces between multiple materials. Experimental results demonstrate that our method achieves state-of-the-art enhancement performance on synthetic datasets and real images of advanced materials while improving atomic localization accuracy.

I. INTRODUCTION

STEM images are degraded by noise and blur, while realistic training data are scarce; the paper addresses both with spatial-frequency fusion and calibrated, diverse synthetic data. It reports stronger enhancement on three datasets and clearer images with more accurate atomic localization.

I. INTRODUCTION

  • SFFN outperforms AtomSegNet [12], FCN [13], AST [14], GAN [15], FADC [16], and SFIN [17] on three STEM enhancement datasets.The PSNR comparison covers TEMImageNet, the SFIN dataset, and AtoMix.Source: Figure 1 · p. 1
  • Noise from probe jitter, signal readout, and background interference, along with spherical aberration, obscures structures; exposure and electron-beam intensity constraints further increase noise.Source: §I · p. 1
  • SFFN combines spatial-domain atomic-contour restoration with frequency-domain periodicity priors, adaptively fusing complementary features to address noise-obscured signals.Source: §I · p. 2
  • The noise-calibration method estimates parameters from real STEM images containing atomic patterns, avoiding the need for uniform calibration frames when synthesizing noisy images.Source: §I · p. 2
  • AtoMix expands dataset coverage to mixed atomic arrangements, HAADF and BF imaging, and complex degradations including spherical aberration, lower brightness, and lower resolution.Source: §I · p. 2
  • Experiments report clearer STEM images and more accurate atomic localization, with application results on advanced materials demonstrating effectiveness.Source: §I · p. 2

II. RELATED WORK

Prior STEM enhancement and synthesis work addresses noise and data needs, but existing approaches leave limitations in degradation removal, frequency modeling, and synthetic-data realism. The paper positions its network and dataset against these gaps.

II. RELATED WORK

  • Traditional filters suppress noise but not background interference or spherical aberration, while supervised deep-learning methods can address these degradations using simulated noisy and ground-truth images.Source: §II · p. 3
  • Unsupervised denoising avoids synthetic training data, but without ideal ground-truth supervision it is limited in removing background interference and correcting atomic-contour distortions.Source: §II · p. 3
  • Existing frequency-domain networks guide spatial processing or select frequencies, rather than directly modeling individual frequency locations as SFFN does.Prior approaches also use fixed spatial-frequency addition or lack explicit cross-domain interaction.Source: §II · p. 3
  • Existing synthetic STEM datasets lack realistic noise and diverse atomic arrangements, imaging modes, and degradation types, limiting performance on complex real-world data.TEMImageNet uses random noise parameters; Cycle-GAN relies on reference STEM images and offers less flexibility for controlling synthesis parameters.Source: §II · p. 3

III. METHOD

The method combines spatial features for localized irregularities with frequency-domain features for periodic atomic structure, then fuses them to enhance STEM images. Its synthesis framework also mixes atomic arrangements and incorporates degradations informed by real-image noise calibration.

A. Formulation and Motivation

  • The image-formation model represents a noisy STEM image as a spherical-aberration-degraded clean image plus background, scan, and pointwise noise.The method models scan- and pointwise-noise variances as functions of local image gradient and pixel intensity, respectively, while retaining additive formulations for computational tractability.Source: A. Formulation and Motivation · p. 3
  • The synthesis framework mixes atomic-arrangement patterns with masks and progressively adds degradations using parameters calibrated from real images.It incorporates mixed arrangements, HAADF and BF modes, and degradations including spherical aberration and intensity loss.Source: A. Formulation and Motivation · p. 4, Figure 4 · p. 5

B. Spatial-Frequency Fusion Network

  • Coordinate-Guided Frequency Convolution transforms features into the frequency domain and processes frequency components using two coordinate channels.The real-valued transform retains half of the frequency-map width because the input is real-valued and the transformed matrix has conjugate symmetry.Source: Figure 3 · p. 4, B. Spatial-Frequency Fusion Network · pp. 4–5
  • Spatial- and frequency-domain branches enhance complementary features, then a convolution fuses their outputs for more comprehensive enhancement.The spatial branch handles localized structures such as interfaces, vacancies, and intercalated ions; the frequency branch targets periodic signatures and can suppress noise in different frequency ranges.Source: B. Spatial-Frequency Fusion Network · p. 4, Figure 3 · p. 4

C. STEM Image Synthesis

The synthesis pipeline creates clean STEM labels from varied atomic arrangements, then simulates degradation using parameters calibrated from real images. It includes mixed material patterns, multiple noise types, and common imaging modes.

C. STEM Image Synthesis

  • The pattern library draws on 37 crystal structures viewed from four orientations to produce 46 periodic pattern units spanning seven major crystal systems.Each selected unit is tiled after random rotation and scaling, with interatomic spacing of approximately 7 to 30 pixels.Source: C. STEM Image Synthesis · pp. 5–6
  • Clean images render atomic arrangements using 2D Gaussian spots, with atom sizes and relative brightness informed by abTEM-derived parameters.The paper uses abTEM to obtain arrangement structures and atom parameters, rather than directly rendering the full dataset with its physics-based simulator.Source: C. STEM Image Synthesis · p. 5
  • AtoMix uses masked combinations of different atomic arrangements, including occasional empty regions to represent vacuum outside material boundaries.Binarized Perlin noise selects the region replaced by a second arrangement; about 10% of atoms are relocated to simulate defects and ion intercalation.Source: C. STEM Image Synthesis · p. 6
  • Synthetic images add spherical-aberration blur and random intensity attenuation before sequentially adding background, scan, and pointwise noise.Atomic radii are doubled to approximate probe broadening, and image intensity is multiplied by a random value between 0.2 and 1.Source: C. STEM Image Synthesis · p. 6
  • Background calibration estimates local brightness ranges from blurred real-image regions and simulates spatial variation with 2D Perlin noise.The local minimum represents background brightness for HAADF images, while the local maximum is used for BF images.Source: C. STEM Image Synthesis · p. 6
  • Scan-noise standard deviation is fit against image-gradient amplitude, while pointwise-noise standard deviation is fit against pixel value and its shape parameter is estimated as a small positive constant.The framework applies to HAADF and BF modes with sign adjustments; AtoMix spans SNRs from −5.09 dB to 10.58 dB across those modes.Source: C. STEM Image Synthesis · pp. 6–7

IV. EXPERIMENTS

The experiments section introduces the setup, compares enhancement methods, and analyzes the proposed network and AtoMix dataset.

IV. EXPERIMENTS

  • The experiments are organized around setup, method comparisons, and analyses of spatial-frequency learning and the AtoMix dataset.Source: §IV · p. 7

A. Experimental Settings

The evaluation uses synthetic and real STEM data, compares traditional and deep-learning methods, and trains SFFN on AtoMix.

A. Experimental Settings

  • Evaluation covers TEMImageNet, SFIN, and AtoMix synthetic data, plus real images of advanced materials and laboratory-captured high-entropy alloys.Enhancement is evaluated with PSNR and SSIM.Source: A. Experimental Settings · p. 7
  • SFFN is compared with traditional filters, supervised deep-learning methods, and unsupervised methods on multi-frame laboratory data.The unsupervised methods UDVD and ZeroIG are compared only on the multi-frame real-world data captured in the laboratory.Source: A. Experimental Settings · p. 7
  • SFFN is trained on AtoMix for 500 epochs with Adam and an L1 loss, using 1,000 training groups and 100 test groups per imaging mode.Training uses a batch size of 8 and is completed on a single NVIDIA 3090 GPU.Source: A. Experimental Settings · p. 7

B. Main Results

SFFN leads synthetic-dataset comparisons and produces clearer, less artifact-prone results on real STEM images. The results also show that training data affects recovery of atoms in real images.

B. Main Results

  • On synthetic examples, SFFN best matches ground-truth brightness on TEMImageNet, separates close atoms on SFIN, and recovers dark and small atoms on AtoMix.Other methods blur the TEMImageNet output, merge close SFIN atoms, or miss some AtoMix atoms.Source: B. Main Results · p. 9
  • Across TEMImageNet, SFIN, and AtoMix, SFFN ranks best, improving PSNR by about 1–2 dB over second-best SFIN.Table II covers enhancement results on the corresponding training and test datasets.Source: B. Main Results · p. 9, Table II · p. 9
  • On real STEM sequences, traditional and unsupervised methods distort atomic shapes and sizes, limiting analysis of atomic positions.The authors attribute this difficulty to the lack of direct ground-truth constraints and trouble removing spherical aberration and background interference.Source: B. Main Results · p. 9
  • On real sequences, AtoMix-trained results are clearer, and SFFN avoids artifacts while preserving atom positions consistent with the material structure.Other supervised methods show artifacts or positional deviations in at least some outputs.Source: B. Main Results · pp. 9–10

C. Discussion

The discussion examines SFFN’s frequency processing, coordinate guidance, and cross-domain fusion, alongside the expanded AtoMix dataset. Ablations and comparisons support the contributions of both feature domains and the dataset’s broader coverage.

C. Discussion

  • Coordinate-guided frequency convolution enhances periodic atomic signals, but also produces background periodic responses; adding the frequency branch raises PSNR from 27.80 dB to 31.60 dB over spatial-only SFFN-V1.The background responses motivate combining frequency-domain and spatial-domain features.Source: C. Discussion · p. 10
  • SFFN convolves Fourier-domain features directly, appends frequency coordinates, and learns concatenation-based fusion; it achieves higher accuracy with 0.9M parameters.Table IV compares frequency-processing methods on AtoMix HAADF data and reports SFFN as best while tying SFIN for fewest parameters.Source: C. Discussion · pp. 10–11
  • Adding coordinate information raises PSNR by 0.25 dB over omitting it, while both coordinate channels contribute substantially to the frequency convolution.The y coordinate contributes slightly more because the frequency feature map is approximately twice as high as it is wide.Source: C. Discussion · p. 11, Figure 11 · p. 11
  • Against no cross-domain interaction, addition raises PSNR by 0.13 dB and SFFN fusion by 1.14 dB; using only one domain also degrades performance.Table III evaluates variants on AtoMix HAADF enhancement.Source: C. Discussion · p. 11
  • AtoMix uses calibrated noise and adds lower resolution and brightness, spherical aberration, and mixed arrangements; training on it better recovers atoms in real images.The dataset comparison describes AtoMix as more diverse and including more complex degradations.Source: C. Discussion · pp. 11–12, Table V · p. 12

V. APPLICATION

The application section evaluates practical utility through real-image denoising of advanced materials and downstream atomic localization.

V. APPLICATION

  • Practical utility is evaluated through denoising real advanced-material images and testing downstream atomic localization.Source: §V · p. 12

A. Enhancement Results on Advanced Materials

SFFN is applied to real images of superconducting materials, batteries, nanodevices, and neuromorphic materials. On a known-structure Pd10Te3 image, it achieves the highest reported denoising and localization metrics.

A. Enhancement Results on Advanced Materials

  • For superconducting material Pd10Te3, SFFN clearly reveals atomic structures in the nucleus and intermediate state where the compared filters and networks fail to visualize atoms.The atoms are only 7 pixels in diameter in the image, posing an enhancement challenge.Source: A. Enhancement Results on Advanced… · p. 12, Figure 14 · p. 13
  • For the NMC-811 battery cathode, SFFN shows slight cation mixing at the edge after long-term contact with Li6PS5Cl, corroborating the cited study’s observations.Source: A. Enhancement Results on Advanced… · p. 12
  • On bent silicon nanoribbons, SFFN presents atomic center positions despite distortions in the HRTEM input, supporting distance measurements and atomic-scale force analysis.The authors note applicability to other atomic-scale images, although HRTEM is globally acquired and lacks STEM scan noise.Source: A. Enhancement Results on Advanced… · p. 12
  • For potassium-ion insertion in Sb4Se6, SFFN distinguishes atomic patterns near and far from the interface and removes spherical-aberration effects.The passage attributes the edge rearrangement to potassium-ion intercalation.Source: A. Enhancement Results on Advanced… · p. 12
  • For the simulated crystal-structure comparison, SFFN has the highest intensity-profile R2 (0.9937), PSNR (24.32 dB), SSIM (0.7798), and atomic-localization IoU (0.5098).The ground-truth image is obtained through atomistic simulation and rendering.Source: A. Enhancement Results on Advanced… · p. 13

B. Localization Results

SFFN achieves the best IoU results across three atomic-localization datasets and produces complete localization results on real high-entropy-alloy images. The section also describes the localization protocol and a real-image comparison with ground truth.

B. Localization Results

  • SFFN achieves the best IoU results on three atomic-localization datasets, and its predictions are consistent with ground truth.The comparison uses Table VI; the supplied text does not report its individual IoU values.Source: Table VI · p. 14, B. Localization Results · p. 14
  • For a real Pd10Te3 image, the denoising result’s brightness-variation curve most closely matches the simulated-and-rendered ground truth.Figure 15 also presents denoising and atomic-localization results.Source: Figure 15 · p. 14
  • The localization protocol trains enhancement networks on images with atoms represented as white disks, then binarizes outputs and locates each atom at its connected region’s center.The section compares network architectures under the same localization protocol.Source: B. Localization Results · pp. 13–14
  • On real high-entropy-alloy images, SFIN and SFFN produce the most complete localization results, consistent with the material’s theoretical structure.Source: B. Localization Results · p. 14

VI. CONCLUSION

The paper presents a spatial-frequency fusion network and the diverse AtoMix dataset for STEM image enhancement. It reports state-of-the-art performance on synthetic datasets and real advanced-material images, and identifies extensions to frequency-domain processing and 4D-STEM as future directions.

VI. CONCLUSION

  • The method achieves state-of-the-art performance on synthetic datasets and real advanced-material images, supporting STEM-based materials research.Source: §VI · p. 14
  • SFFN enhances and fuses spatial- and frequency-domain features, using coordinate-guided frequency convolution to enhance periodic signals.The network exploits frequency-domain information from atomic periodicity.Source: §VI · p. 14
  • AtoMix uses noise parameters derived from real images and includes diverse atomic arrangements, imaging modes, degradations, and masked multimaterial interfaces.Source: §VI · p. 14
  • Future work could combine frequency-domain processing with attention mechanisms or dynamic convolutions, and extend the method to 4D-STEM data.The paper notes that 4D-STEM provides diffraction patterns at probe positions for analyzing local structures, strain, and electromagnetic fields.Source: §VI · p. 14
Loading 2610.36487v1…