PaperlayerView exact revision on arXiv

Source-linked AI summary

Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning

Colorado J. Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, Trevor Darrell

arXiv:2212.14532v4cs.CV

TL;DR

Remote sensing imagery varies widely in ground scale, while existing pretrained approaches often overlook scale-specific information and generalize poorly across multiple scales. Scale-MAE incorporates ground-area-aware positional encoding and multiscale Laplacian-pyramid reconstruction into masked autoencoder pretraining. It improves classification and building-segmentation transfer performance across remote sensing datasets and evaluation scales.

  • Problem

    Existing MAE-based pretraining approaches overlook scale information and fail to generalize across remote sensing domains with images at multiple scales.

  • Method

    Scale-MAE learns relationships between known scales using GSD-based positional encoding and a Laplacian-pyramid decoder that reconstructs low- and high-frequency image components.

  • Results

    5.6% average nonparametric kNN classification improvement across eight remote sensing datasets and 0.9 mIoU to 1.7 mIoU improvement on SpaceNet building segmentation transfer were obtained.

  • Takeaways & Limitations

    Scale-MAE produces stronger, more robust multiscale representations than standard MAE, SatMAE, and ConvMAE across varied remote sensing scales and tasks.

  • Takeaways & Limitations

    Scale-MAE cannot use input images whose bands have different GSDs and does not yet support differing-resolution bands or SAR modalities.

Abstract

from arXiv · show

Large, pretrained models are commonly finetuned with imagery that is heavily augmented to mimic different conditions and scales, with the resulting models used for various tasks with imagery from a range of spatial scales. Such models overlook scale-specific information in the data for scale-dependent domains, such as remote sensing. In this paper, we present Scale-MAE, a pretraining method that explicitly learns relationships between data at different, known scales throughout the pretraining process. Scale-MAE pretrains a network by masking an input image at a known input scale, where the area of the Earth covered by the image determines the scale of the ViT positional encoding, not the image resolution. Scale-MAE encodes the masked image with a standard ViT backbone, and then decodes the masked image through a bandpass filter to reconstruct low/high frequency images at lower/higher scales. We find that tasking the network with reconstructing both low/high frequency images leads to robust multiscale representations for remote sensing imagery. Scale-MAE achieves an average of a $2.4 - 5.6\%$ non-parametric kNN classification improvement across eight remote sensing datasets compared to current state-of-the-art and obtains a $0.9$ mIoU to $1.7$ mIoU improvement on the SpaceNet building segmentation transfer task for a range of evaluation scales.

1. Introduction

Remote sensing imagery spans wide ground scales, but standard pretrained models generally do not explicitly learn scale relationships. Scale-MAE addresses this with ground-area-aware positional encoding and multiscale reconstruction, improving downstream remote sensing performance.

  • Motivation: Remote sensing images can range from 0.3m to 1km GSD, causing equal-sized pixel images to cover dramatically different areas.This variation changes the spatial ranges of objects and supports applications across atmospheric, hydrologic, agricultural, and environmental monitoring.
  • Problem: Few modern computer vision methods explicitly address multiscale remote sensing imagery, while pretrained models are commonly finetuned for one source-data scale.
  • Contribution: Scale-MAE explicitly learns relationships between data at different known scales during masked-reconstruction pretraining.
  • Method: Scale-MAE uses ground-area-based positional encoding and a Laplacian-pyramid decoder to produce low- and high-frequency representations across scales.The positional encoding reflects land area rather than image resolution, while the decoder reconstructs low and residual high frequency information.
  • Results: 5.6% average nonparametric kNN classification improvement was achieved across eight remote sensing datasets, alongside 0.9 mIoU to 1.7 mIoU gains on SpaceNet building segmentation.The segmentation improvements were measured for a range of evaluation scales, and the classification comparison used current state-of-the-art methods.

2. Related Work

Related work covers masked autoencoders, remote sensing representation learning, super-resolution, and multiscale feature extraction. Scale-MAE differs by combining scale-variant positional encoding with a Laplacian-pyramid decoder in a ViT-based MAE.

  • Representation learning and the Masked Autoencoder: Masked autoencoders pretrain a ViT encoder by masking image patches and reconstructing the input, but commonly used positional encodings do not leverage scale information.
  • Remote Sensing Representation Learning: Remote sensing pretraining has incorporated classification, geospatial metadata, temporal and spectral metadata, modality adapters, and band-selection strategies.
  • Super-resolution: Super-resolution methods learn representations for upsampling imagery to specific or arbitrary resolutions, addressing the small size of remote-sensing objects.
  • Multiscale Features: Multiscale feature methods include spatial pyramids, dense window sampling, CNN feature pyramids, and hybrid CNN-MAE architectures.
  • Multiscale Features: Scale-MAE differs from prior multiscale approaches by using a Laplacian-pyramid decoder to force a ViT encoder to learn multiscale features during MAE pretraining.

3. Scale-MAE

Scale-MAE extends masked autoencoding with positional encodings tied to geographic scale and a Laplacian-pyramid decoder that reconstructs frequency components across scales. Its training pipeline combines GSD-aware encoding, masked ViT reconstruction, and progressive multiscale decoding.

  • GSD Positional Encoding: Scale-MAE scales positional encoding with the land area covered by an image, rather than image resolution alone.The Ground Sample Distance (GSD) represents the physical distance between adjacent pixels and determines image scale.
  • Masked Autoencoder Setup: The ViT encoder receives embedded unmasked patches with positional encodings, while masked patches are restored with learned mask tokens for decoding.The decoder reconstructs the learning target after the removed patches are returned to their original sequence locations.
  • Scale-MAE Decoder: Scale-MAE targets low- and high-frequency components of a higher-resolution ground-truth image instead of reconstructing only the input image.The input is downsampled from the ground-truth resolution, and the frequency components correspond to lower and higher reconstruction scales.
  • GSD Positional Encoding: The GSD positional encoding uses the image GSD g relative to a reference GSD G to account for corresponding pixel ordering across resolutions.The paper nominally sets G to 1m and describes the encoding as an interpolation by the factor G/g.
  • Scale-MAE Decoder: Its progressive Laplacian decoder replaces the traditional MAE decoder with decoding, upsampling, and reconstruction stages for multiscale outputs.Latent feature maps are upsampled to 2x and 4x resolution before Laplacian Blocks reconstruct high- and low-resolution images.

4. Experiments

Scale-MAE is evaluated for representation quality and transfer across datasets, tasks, and image scales. Experiments test kNN classification, transfer classification, semantic segmentation, and ablations against contemporary MAE baselines.

  • Experimental setup: Experiments compare Scale-MAE with SatMAE, ConvMAE, vanilla MAE, and other approaches across remote sensing datasets spanning varied GSDs.The evaluation includes representation quality, transfer tasks, and scale-robustness tests.
  • Experimental setup: Scale-MAE reconstructs masked FMoW images into low-frequency and residual high-frequency components, whose sum forms the final reconstruction.Low-frequency outputs capture color gradients and landscapes, while high-frequency outputs capture edges, roads, and building outlines.
  • Representation quality: Scale-MAE’s kNN advantage increases as evaluation GSD diverges from the original GSD, indicating greater robustness to scale changes.On UC Merced, the largest performance gap occurs at the artificially coarsened 2.4m GSD, while performance is similar at 0.3m.
  • Transfer evaluation: Scale-MAE outperforms SatMAE by 0.9 mIoU, ConvMAE by 1.3 mIoU, and vanilla MAE by 1.0 mIoU on SpaceNet building segmentation.Across downscaled SpaceNet evaluations, the gap grows from 0.9 mIoU at the input GSD to 1.7 mIoU at 50% resolution.
  • Transfer evaluation: Scale-MAE outperforms comparable methods across SpaceNet v1, SpaceNet v2, INRIA Aerial Image, and GID-15 at native resolution.The experiments also include transfer classification on RESISC-45 and full finetuning on FMoW-RGB.

5. Discussion

The discussion examines Scale-MAE’s computational trade-offs, modality and dataset boundaries, and possible extensions to other backbones. It also highlights ablation findings about decoder design and multiscale reconstruction.

  • Computational complexity: Scale-MAE uses a three-layer decoder instead of vanilla MAE’s eight-layer decoder and has 322.9M versus 329.5M parameters with ViT-Large.The smaller parameter count does not eliminate higher GPU memory usage for equal batch sizes.
  • Multi-spectrality and modality: Scale-MAE cannot use input images whose bands have different GSDs because its input tensors must be stacked.The paper reserves extensions to different-resolution bands and modalities, including SAR imagery, for future work.
  • Other backbones: The core Scale-MAE components could be integrated into other architectures, including convolutional backbones, with additional engineering.The Laplacian decoder can be engineered to ingest convolutional feature maps.
  • Ablations: Reconstructing both low- and high-resolution components leads to robust performance in the reported ablation study.The ablation evaluates kNN classification on RESISC-45 at relative GSDs of 50% and 100%.
  • Dataset scope: The evaluation focuses on remote sensing datasets selected for properties relevant to multiscale representation learning rather than covering every available dataset.The paper notes the existence of many generic and highly specific remote sensing datasets.

6. Conclusion

Remote sensing imagery spans diverse resolutions, while existing methods require a new model for each input resolution, making training and multiscale dataset curation difficult. Scale-MAE introduces scale-inclusive positional encodings and progressive multifrequency feature extraction, and reports better performance across scales and benchmarks.

  • 6. Conclusion: Current remote sensing vision methods require training a new model per input resolution, increasing training expense and the burden of curating datasets at multiple scales.
  • 6. Conclusion: Scale-MAE introduces scale invariance into encoders used for diverse downstream tasks.
  • 6. Conclusion: Scale-inclusive positional encodings and progressive multifrequency feature extraction produce better performance than state-of-the-art pretraining methods across multiple scales and benchmarks.
  • 6. Conclusion: The stated goal is to make remote sensing information usable with minimal training iterations.The authors connect this goal to accelerating scientific impact from remote sensing imagery.

A. Datasets

The experiments use selected remote sensing datasets to represent diversity in classes, objects, geographic settings, and resolutions. Dataset selection is intended to support evaluation of multiscale representation learning.

  • A. Datasets: The experiments use ten datasets for land-use/land-cover classification and semantic segmentation.
  • A. Datasets: The datasets were selected for representative properties, including diversity in the kinds of classes and objects represented.
  • A. Datasets: The selection also considered a broad spectrum of geographic and resolution characteristics relevant to remote sensing.

A.1. Diversity in classes

The dataset design prioritizes geographic and class diversity while controlling image-processing differences. The reported dataset statistics organize resolution, GSD, image counts, classes, and task types for the evaluation set.

  • Diversity in classes: Geographic and class diversity is treated as desirable for both pretraining and downstream evaluation.Multiple localities and environments help capture a wider range of remote-sensing classes and features.
  • Dataset statistics: The dataset statistics report resolution, GSD, number of images, number of classes, and task type.Task types include classification and semantic segmentation.
  • Resolution control: The datasets include varied sensor sources processed uniformly to test resolution while excluding processing differences as an experimental factor.
  • Preprocessing control: Images were required to be pansharpened, orthorectified, and projected to the same coordinate reference system.These requirements eliminate large differences in sensor-to-image processing.

B. Laplacian and Upsampling Block Architectures

Figure 7 presents the architectures of the Laplacian Block and Upsampling Block used in Scale-MAE.

  • Figure 7 illustrates the Laplacian and Upsampling block architectures.

B.1. Laplacian Block

Laplacian Blocks reconstruct targets at specific resolutions and frequencies, while Upsampling Blocks progressively increase feature-map resolution.

  • Laplacian Block: Laplacian Blocks reconstruct targets at a specific resolution and frequency.They use Feature Mapping Blocks followed by a final Reconstruction Block.
  • Laplacian Block: Each Laplacian Block contains a chain of Feature Mapping Blocks followed by one Reconstruction Block.The Feature Mapping Blocks distill information at a specific frequency before the final output is generated.
  • Laplacian Block: Two Feature Mapping Blocks are used per Laplacian Block in the experiments.
  • Upsampling Block: Upsampling Blocks progressively double feature-map resolution until reaching the desired target resolution.They use 2x2 transpose convolutions with LayerNorm and GELU between layers; layer count depends on input and output resolutions.

C. Evaluation Details

The evaluation examines Scale-MAE representations through multiscale transfer experiments, including kNN classification and SpaceNet segmentation across different GSDs.

  • Evaluation Details: Scale-MAE was pretrained with a ViT-Large on 363.6k FMoW RGB images varying in resolution and GSD.Inputs use a 448px2 high-resolution crop and a downsampled 224px2 image, with low- and high-frequency reconstruction targets.
  • Evaluation Details: Evaluation tested representation robustness to scale and transfer performance on additional tasks.
  • Evaluation Details: The kNN evaluation compares Scale-MAE, SatMAE, and ConvMAE across varied k values and image resolutions.Table 11 reports the classification results across these settings.
  • Evaluation Details: SpaceNet examples compare ground-truth labels, Scale-MAE, and vanilla MAE at 0.3m and 3.0m GSD.The figure reports better Scale-MAE performance at both higher and lower GSDs.

E.1. Ground sample distance

Ground sample distance (GSD) is the center-to-center distance between adjacent pixels in a remote sensing image.

  • Ground sample distance is the distance between the centers of adjacent pixels.GSD depends on sensor parameters, image parameters, and sensor geometry relative to the imaged object.
Loading 2212.14532v4…