PaperlayerView exact revision on arXiv

Source-linked AI summary

Structure-Guided Masked Autoencoders for Ultra-High Resolution Scientific Image Understanding

Enzhi Zhang, Du Wu, Rui Zhong, Cong Ma, Isaac Lyngaas, Amir Koushyar Ziabari, Xiao Wang, Peng Chen, Tao Luo, Toshio Endo, Fumiyoshi Shoji, Kento Sato, Kentaro Uesugi, Takayuki Nonoyama, Ryuji Kiyama, Masahiro Yoshida, Masaru Tezuka, Tetsuya Ishikawa, Satoshi Matsuoka, Masaharu Munetomo, Mohamed Wahib

arXiv:2609.30682v1cs.CV

TL;DR

Ultra-high-resolution scientific images pose a challenge for self-supervised ViTs because labels are scarce, sequences are prohibitively long, and standard MAE does not account for hierarchical microstructure. SGMA combines adaptive quadtree tokenization with structure-conditioned masking; across TEM, XCT, and WSI tasks it improves segmentation over MAE while also accelerating inference.

  • Problem

    Scientific UHR images have costly, scarce annotations and enormous patch sequences, while standard MAE's uniform patches and random masking do not model their hierarchical spatial structures.

    Source: §1 · pp. 1–2

  • Method

    SGMA combines a content-adaptive quadtree tokenizer with Damped Accumulation structure-conditioned sampling to train an MAE-based ViT to reconstruct multiscale scientific image structures.

    Source: §1 · pp. 2–3

  • Results

    Across TEM, XCT, and WSI datasets SGMA consistently outperforms MAE; it gains +16.84 points on PAIP-32K and provides up to 24.8× faster inference.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    SGMA supports downstream scientific analysis, including zero-shot pore-network extraction and hydrogel skeletonization.

    Source: §6 · p. 9

    SGMA's fixed token budget requires dataset-specific tuning, and its edge-based salience can misallocate tokens when gradients are weak, noise dominates, or relevant regions are texture-defined.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Self-supervised pre-training with Vision Transformers, including Masked Autoencoders (MAE), is difficult to apply to gigapixel scientific images. Random masking is poorly matched to the structured, multi-scale morphology of scientific data, while uniform tokenization produces prohibitively long sequences that make $O(N^2)$ attention impractical. We propose SGMA, a structure-guided masked autoencoding framework for ultra-high-resolution scientific images. SGMA couples two components: a content-adaptive quadtree tokenizer that compresses gigapixel images into a fixed-length sequence, and a structure-conditioned masking process that biases reconstruction toward spatially informative regions. To stabilize this process across scales, we introduce Damped Accumulation (DA), which aggregates signal-dependent responses across the tree into a structure canvas used to guide masking. The resulting pre-training task preserves fine microstructure while remaining compatible with standard ViT encoders and MAE-style reconstruction. Across electron microscopy, whole-slide optical microscopy, and X-ray CT datasets, SGMA consistently outperforms MAE baselines. It achieves 95.68% Dice on the 8K x 8K x 28K SpringXCT dataset, improving over the same-architecture MAE baseline by +13.00 points, and 83.21% Dice on the 32K^2 WSI PAIP dataset, improving by +16.84 points, while providing up to a 24.8x inference speedup.

1 Introduction

Ultra-high-resolution scientific images combine rich multiscale structure with scarce labels, while their enormous patch sequences make standard ViT processing intractable. SGMA pairs adaptive quadtree tokenization with structure-conditioned masking for MAE pre-training across scientific imaging domains.

1 Introduction

  • A single 64K² image requires annotation equivalent to 16,384 images at 512 × 512 resolution, and precise scientific labels require costly domain expertise.Source: §1 · p. 1
  • For a 64K² image with 16-pixel patches, standard ViT input reaches 16,384² tokens, making training, fine-tuning, and inference intractable under O(N²) attention.Source: §1 · p. 2
  • Standard MAE lacks hierarchical spatial-structure awareness, while downsampling to fit memory destroys fine details and leaves downstream processing with the full long sequence.Existing long-sequence alternatives may add architectural complexity or alter attention, complicating compatibility with MAE's global sparse-patch reconstruction.Source: §1 · p. 2
  • SGMA targets distinct multiscale structures across TEM, optical WSI, and XCT images, spanning resolutions below 0.1 nm to 12 µm and 200 nm, respectively.Source: §1 · p. 2, Figure 2 · p. 3
  • SGMA jointly uses Structure-Guided Feature Tokenizer tokenization and Damped Accumulation structure-conditioned sampling to guide MAE reconstruction toward detail-rich regions.Its probabilistic, content-adaptive quadtree captures multilevel spatial information, while the sampling process adds scale-dependent Gaussian noise across tree levels.Source: §1 · pp. 2–3

2 Related Work

Masked Image Modeling offers self-supervised reconstruction, but uniform patching and random masks are poorly suited to gigapixel scientific imagery. SGMA uses a quadtree as an adaptive input tokenizer and couples its construction with structure-conditioned sampling, retaining a standard ViT architecture.

2 Related Work

  • Masked Image Modeling methods such as MAE randomly mask image patches for reconstruction, but uniform patching becomes intractable at gigapixel scale and random masking may miss fine scientific structures.Source: §2 · p. 4
  • Long-sequence approaches optimize computation with parallel methods, efficient kernels, approximations, sparse attention, or hierarchical ViTs, but can add complexity or risk performance loss.Source: §2 · pp. 4–5
  • Unlike prior quadtree approaches that add model complexity or require expert hyperparameters, SGFT uses the tree as an adaptive input tokenizer for a standard ViT and interleaves it with structure-conditioned sampling.Source: §2 · p. 5

3 Methodology

SGMA combines adaptive tree tokenization and structure-conditioned information sampling within an MAE-style pre-training framework. The method also adapts the pre-trained encoder for downstream segmentation using patch-level targets.

3 Methodology

  • SGMA comprises preprocessing, multi-scale tree construction, and MAE pre-training, with structure-conditioned modeling implemented through SGFT construction and information sampling.Source: §3 · p. 5

3.1 Structure-Guided Feature Tokenizer

  • Up to 20× faster at N=10^6 tokens than the SHF/AP-style arg max tokenizer at matched tokenization quality, the softmax-batched quadtree selects active tokens by salience.The temperature controls selection softness, while N sets the target sequence length.Source: §3.1 · p. 5

3.2 Structure-Guided Information Sampling

  • Damped accumulation adds independent information along each token’s ancestral path, with depth-scaled local variances keeping canvas accumulation sub-linear with depth.The resulting full-resolution cumulative structure canvas is updated alongside the active token set.Source: §3.2 · pp. 5–6

3.3 SGMA Pre-training Framework

  • SGMA corrupts selected adaptive structural patches, encodes the visible sequence with a standard ViT, and reconstructs only the corrupted patches using clean patches as context.The decoder uses an L2 reconstruction loss.Source: §3.3 · p. 6

3.4 Fine-tuning

  • For segmentation fine-tuning, SGMA discards the reconstruction decoder and trains the encoder and a linear head against tree-transformed ground-truth masks at patch level.Patch-level logits save memory and reduce backpropagation costs.Source: §3.4 · p. 6

4 Experiments

The experiments span electron microscopy, X-ray CT, and whole-slide pathology data at high resolutions. They compare MAE- and SGMA-pre-trained versions of shared architectures alongside other segmentation models.

4.1 High Resolution Scientific Image Datasets

  • The datasets cover nanoscale hydrogel TEM, synthetic-label SpringXCT slices, and real-world liver cancer whole-slide images reaching 64K^2 resolution.SpringXCT-8K uses 8,192 × 8,192 slices; PAIP-32K contains 2,457 whole-slide images.Source: §4.1 · p. 7
  • SpringXCT-8K pre-training uses 308,000 8,192 × 8,192 slices, while fine-tuning data are generated through simulated imaging with ground-truth labels.The fine-tuning set comprises 60 synthetic volumes and corresponding reconstructions.Source: §4.1 · p. 7

4.2 Evaluating Models: Baselines & Proposed

  • The evaluation compares standard MAE and SGMA pre-training on the same SAM and UNETR architectures, with U-Net, UNETR, and Swin-UNet also included.The MAE-SAM setup uses ViT encoders; SGMA-SAM uses symmetrical depatching, and SpringXCT-3D applies SAM 2 to 2D slices.Source: §4.2 · p. 7

5 Evaluation and Results

SGMA improves segmentation across the evaluated datasets and provides faster inference. Its performance depends on an appropriately sized token budget and on edge maps reflecting task-relevant structure.

5.1 High-Precision Image Segmentation

  • 95.68% Dice on SpringXCT-8K improves by +13.00 points over FT-MAE-SAM; 83.21% Dice on PAIP-32K improves by +16.84 points over FT-MAE-SAM 2.Table 1 compares SGMA with MAE pre-training on identical architectures.Source: §5.1 · p. 7, Table 1 · p. 8
  • SGMA-UNETR resolves fine, nm-scale structures, while UNETR with large pre-training patches merges individual structures into a single mass.On BTCV and KiTS19, SGMA-SAM 2 matches or marginally exceeds specialized baselines while running 2.58×–7.15× faster; DA gains are task-dependent, at +0.88 Top-1 for classification and +3.00 Dice for segmentation.Source: §5.1 · p. 8
  • Segmentation comparisons across HydrogelTEM-1K, SpringXCT-8K, and PAIP-32K show blurry or incomplete UNet boundaries, unlike SGMA's crisp, topologically accurate masks.SGMA's adaptive-scale pre-training resolves fine structures in the visual comparison.Source: §5.1 · pp. 7–8

5.2 Compute Efficiency: Training and Inference

  • Up to 24.8× faster inference is reported for SGMA under matched compute, even with a deliberately shorter sequence than its best-Dice configuration.SGMA compresses inputs to a fixed-length sequence, while uniform-grid MAE baselines may need very large patches to avoid OOM and lose fine detail.Source: §5.2 · pp. 8–9, Table 2 · p. 9

5.3 Limitations and Effect of Hyperparameters

  • An undersized sequence budget on heterogeneous inputs, such as PAIP-32K at N=1024, coarsens fine microstructures and lowers Dice, so N must be tuned per dataset.The needed budget depends on input resolution and structural density.Source: §5.3 · p. 9
  • SGFT's Canny-edge salience can misallocate tokens when edges poorly reflect task-relevant structure, including weak-gradient, noisy, or texture-defined images.In those settings, the tree may over-tokenize uninformative areas and under-tokenize regions of interest, bounding gains over uniform patching.Source: §5.3 · p. 9

6 Conclusion

SGMA makes MAE-style pre-training feasible for ultra-high-resolution scientific images while preserving fine-scale structure. It improves segmentation over MAE and supports zero-shot scientific analyses.

6 Conclusion

  • Across TEM, XCT, and WSI datasets, SGMA improves segmentation accuracy over MAE, including +16.84% Dice on 32K pathology and up to 24.8× faster inference.The framework avoids the O(N^2) bottleneck while preserving important fine-scale structures.Source: §6 · p. 9
  • SGMA makes MAE-style pre-training feasible for ultra-high-resolution scientific images.The conclusion describes the framework as avoiding the O(N^2) bottleneck and preserving important fine-scale structures.Source: §6 · p. 9
  • SGMA enables zero-shot analysis, including pore-network extraction and hydrogel skeletonization.Source: §6 · p. 9

A.1 Broader Impacts

SGMA targets ultra-high-resolution scientific imaging and is presented as a means to reduce analysis costs. Clinical use of models derived from pathology images carries additional validation and oversight requirements.

A.1 Broader Impacts

  • SGMA targets ultra-high-resolution scientific images from electron microscopy, X-ray CT, and whole-slide pathology.Source: §A.1 · p. 15
  • The authors state that SGMA can lower compute and annotation costs for gigapixel scientific data and reduce reliance on expensive expert annotation.They identify potential relevance to materials science, biomedical imaging, and structural biology.Source: §A.1 · p. 15
  • Clinical deployment of models derived from pathology datasets requires additional validation, regulatory clearance, and safeguards against over-reliance on automated segmentation.Source: §A.1 · p. 15

A.2 Hyperparameters and Implementation Details

SGMA uses fixed, dataset-specific training settings for masking, token budgets, backbones, and optimization. Its token budget directs finer partitioning toward detail-rich regions, while larger backbones recover finer topology.

Masking Ratio

  • r = 0.75 is used for SGMA pre-training, and HydrogelTEM-1K validation found it gave the best Dice among the tested ratios.Masks preserve the ratio within each quadtree scale; reconstruction quality degrades gracefully as the ratio increases.Source: §A.2 · p. 15

Sequence Length

  • The SGFT token budget ranges up to N = 16,384 for the highest-accuracy runs, with smaller speed-optimized settings varying by dataset.HydrogelTEM-1K uses N = 8,194 for fine-tuning; SpringXCT-8K uses N = 1,024 for speed optimization; PAIP-32K uses N = 2,048 for speed optimization.Source: §A.2 · p. 15
  • As N grows, SGFT allocates more tokens to detail-rich regions while keeping uniform regions coarsely tokenized.The reported visualization compares N ∈{1024, 4096, 8192}.Source: §A.2 · p. 15

Model Size

  • SGMA uses standard ViT backbones, an 8-block pre-training decoder, dataset-specific segmentation heads, and 400 pre-training epochs with AdamW and a cosine schedule.Fine-tuning runs for 100 epochs at a 10× smaller learning rate; mixed precision uses bf16.Source: §A.2 · pp. 15–16
  • On a HydrogelTEM-1K patch, ViT-XL recovers finer network topology than ViT-L under the same masking and quadtree tokenization.The study uses ViT-L for its highest-resolution datasets.Source: §A.2 · p. 15

A.3 Zero-shot Downstream Tasks Evaluation

On a simulated SpringXCT-8K slice, SGMA-SAM 2 recovers fine pore topology better than the listed baselines. SGMA-derived masks also support component and connection analysis on real samples.

Model-size comparison

  • ViT-XL recovers finer HydrogelTEM-1K network topology than ViT-L under the same structure-guided masking and quadtree tokenization.This model-size comparison motivates using larger backbones on higher-resolution datasets.Source: Figure 6 · p. 17

Real-sample analysis

  • On real samples, SGMA-UNETR segmentations yield near-perfect segmentation, while MAE-SAM 2 masks produce noisy statistics that underestimate small-pore volume.The examples include connected-component analysis and stone-distribution mapping for SpringXCT-8K, and zero-shot segmentation with skeleton degree analysis for HydrogelTEM-1K.Source: §A.3 · p. 17

SpringXCT-8K

  • 94.79% Dice is reported for SGMA-SAM 2 on a simulated SpringXCT-8K slice, versus 85.98% for MAE-pretrained SAM 2, 58.38% for UNet, and an improvement over Adaptive Patching.SGMA-SAM 2 uses a 2×2 effective patch with N = 8,194 SGFT tokens; the MAE-pretrained model uses a 128×128 patch, and Adaptive Patching uses N = 8,194 tokens.Source: Figure 7 · p. 17, §A.3 · p. 17

A.5 Comparison with Long-Sequence and Hierarchical Transformer Methods

Across four extensions, SGMA combines input-stage structure-guided tokenization with standard ViTs, speeds tokenizer construction, and reports segmentation and classification results on 3D and whole-slide tasks.

A.5 Comparison with Long-Sequence and Hierarchical Transformer Methods

  • SGMA places sequence reduction at input preprocessing, leaving standard ViT architecture and attention unchanged while producing a fixed-length, structure-conditioned token sequence.The comparison describes its best-case effective complexity as O(log2 N) in input resolution, unlike methods that modify attention or use specialized backbones.Source: §A.5 · pp. 18, 20, Table 3 · p. 19

A.6 SGFT Tokenizer Construction Efficiency

  • At matched tokenization quality, SGFT is about 20× faster than heap-based arg max on a 4,096^2 image at N = 106 tokens and 9.2× faster on 8,192^2.The three tokenization algorithms' small-scale per-pixel MSE agrees within about 3%.Source: §A.6 · pp. 20–21

A.7 Additional 3D Medical Segmentation Baselines

  • SGMA-SAM 2 with 23 patches reports 89.71% Dice on BTCV and 87.25% on KiTS19, with speedups of 7.15× over SAM 2 and 2.58×, respectively.The section characterizes these results as matching or marginally beating the best specialized baselines at lower spatial resolution.Source: §A.7 · p. 21, Table 5 · p. 22

A.8 Additional Task: Classification on PAIP-16K and a DA Ablation

  • On PAIP-16K, SGMA-ViT-2 achieves 83.77% Top-1 accuracy and 81.67% Dice; removing DA costs +0.88 Top-1 and +3.00 Dice.SGMA-ViT-2 improves over SHF-ViT-2 by +3.63 accuracy points and over HIPT by +11.08 points; the reported DA effect is larger for segmentation than classification.Source: §A.8 · p. 22, Table 6 · p. 22

Related papers

Loading 2609.30682v1…