PaperlayerView exact revision on arXiv

Source-linked AI summary

CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders

Anthony Fuller, Koreen Millard, James R. Green

arXiv:2311.00566v1cs.CV

TL;DR

Remote sensing has abundant, spatially aligned multimodal data but sparse labels, creating a need for effective self-supervised representations. CROMA combines radar↔optical contrastive learning with masked reconstruction and spatially biased attention, and reports stronger performance than the previous state of the art across diverse evaluations. Its main scope is static-in-time Sentinel-1 and Sentinel-2 data.

  • Problem

    Remote sensing offers abundant spatially aligned multimodal imagery but sparse labels, motivating self-supervised learning for Earth-observation representations.

  • Method

    CROMA separately encodes masked Sentinel-1 radar and Sentinel-2 optical data, fuses them for reconstruction, and applies radar↔optical contrastive learning with 2D- and cross-attention ALiBi.

  • Results

    CROMA outperforms the previous state of the art across classification and segmentation evaluations, with improvements reported for finetuning, probing, kNN, clustering, and segmentation.

  • Takeaways & Limitations

    CROMA provides optionally unimodal or multimodal representations that can be used across diverse tasks and on larger images at test time.

  • Takeaways & Limitations

    The work focuses on static-in-time Sentinel-1 and Sentinel-2 data; future work will explore other sensors and time-series data.

Abstract

from arXiv · show

A vital and rapidly growing application, remote sensing offers vast yet sparsely labeled, spatially aligned multimodal data; this makes self-supervised learning algorithms invaluable. We present CROMA: a framework that combines contrastive and reconstruction self-supervised objectives to learn rich unimodal and multimodal representations. Our method separately encodes masked-out multispectral optical and synthetic aperture radar samples -- aligned in space and time -- and performs cross-modal contrastive learning. Another encoder fuses these sensors, producing joint multimodal encodings that are used to predict the masked patches via a lightweight decoder. We show that these objectives are complementary when leveraged on spatially aligned multimodal data. We also introduce X- and 2D-ALiBi, which spatially biases our cross- and self-attention matrices. These strategies improve representations and allow our models to effectively extrapolate to images up to 17.6x larger at test-time. CROMA outperforms the current SoTA multispectral model, evaluated on: four classification benchmarks -- finetuning (avg. 1.8%), linear (avg. 2.4%) and nonlinear (avg. 1.4%) probing, kNN classification (avg. 3.5%), and K-means clustering (avg. 8.4%); and three segmentation benchmarks (avg. 6.4%). CROMA's rich, optionally multimodal representations can be widely leveraged across remote sensing applications.

1 Introduction

Remote sensing provides extensive, spatially aligned multimodal data but relatively few labels, motivating self-supervised representations. CROMA combines contrastive and reconstruction objectives to learn representations for diverse Earth-observation uses.

  • Remote sensing deep learning supports applications including environmental monitoring, disaster detection, and socioeconomic estimation.
  • Self-supervised learning is valuable for remote sensing because application-specific models face limited labeling while satellite data are abundant.
  • Contrastive learning aligns positive views and separates negative samples, but can discard information that is not shared between views.
  • Reconstruction objectives capture more information and scale without multiple views, but their representations can require substantial finetuning.
  • Sentinel-2 optical imagery characterizes material composition across wavelengths, while Sentinel-1 SAR measures responses to actively transmitted longer-wavelength pulses.
  • CROMA uses spatially and temporally aligned Sentinel-1 and Sentinel-2 data to combine complementary objectives and learn optionally multimodal representations.

2 Method

CROMA uses separate radar and optical encoders, a multimodal fusion encoder, contrastive learning, masked reconstruction, and spatially biased attention for aligned 2D sensor data. Its design targets sensor-invariant representations, efficient masked pretraining, and image-size extrapolation.

  • CROMA defines optical data as 12-channel Sentinel-2 multispectral imagery and radar data as 2-channel Sentinel-1 SAR backscatter.
  • Model Architecture: Three encoders separately process radar and optical inputs, then encode paired radar-optical patches for joint multimodal representations.
  • Model Architecture: The multimodal encoder fuses sensor information, while mean-pooled unimodal patch encodings produce full-image representations through feedforward networks.
  • ALiBi Background: ALiBi encodes relative positions directly in attention matrices rather than adding position embeddings at the network input.
  • 2D-ALiBi and X-ALiBi: 2D-ALiBi biases self-attention by Euclidean query-key distance, while X-ALiBi applies analogous spatial biases to cross-modal attention.
  • Reconstruction Objective: CROMA masks 75% of radar and optical patches, and a lightweight decoder predicts both sensors’ masked patches from multimodal encodings and positional embeddings.
  • Contrastive Objective: Radar↔optical InfoNCE uses geographically and temporally matched samples as positives and other batch samples as negatives, encouraging sensor-invariant representations.
  • Combined Pretraining Objective: The combined pretraining loss weights contrastive and masked-autoencoding objectives equally.

3 Experiments

CROMA is evaluated against established remote-sensing representation models across classification, clustering, probing, segmentation, and multimodal benchmarks. Results show gains for dense prediction and multimodal representations, while additional analyses examine representation information and design choices.

  • Classification Experiments: CROMA is compared with multispectral foundation models, radar↔optical models, and I-JEPA across four classification benchmarks and multiple probing protocols.Evaluations include finetuning, frozen linear and nonlinear probing, kNN classification, and K-means clustering.
  • Classification Experiments: 76.0% K-means clustering accuracy is reported for CROMA (ViT-B) on EuroSAT, compared with 44.9% for SatMAE (ViT-B), 45.8% for DINO (ViT-S), and 56.9% for radar↔optical (Swin-T).The comparison uses UMAP embeddings and K-means clustering accuracies.
  • Representation Analysis: Sparse probing restricts linear probes to the most informative representation dimensions to assess how information is distributed across learned representations.The experiment ranks dimensions separately for each class and trains binary classifiers on the top k dimensions.
  • Representation Analysis: 49% F1 is achieved by CROMA with only k = 1 representation dimension for one BigEarthNet class analysis.The paper reports broader sparse-probing results in Appendix §A.6.
  • Segmentation Experiments: CROMA outperforms SatMAE on semantic segmentation by averages of 5.4% with ViT-B and 6.4% with ViT-L backbones.The evaluation covers DFC2020, DW-Expert, and MARIDA using frozen patch encodings and linear probes.
  • Multimodal Experiments: Joint radar-optical representations outperform optical-only representations by 1.2% on BigEarthNet and by 4.9% for CROMA-B and 3.5% for CROMA-L on DFC2020.CROMA also outperforms SatViT-V2 and its radar-only and radar-optical representations outperform DeCUR under linear probing.
  • Multimodal Experiments: The ablation table evaluates radar-only, optical-only, and joint radar-optical inputs using both performance and cost.Rows below the default configuration report performance differences between ablated cases and the default.

4 Ablation Analysis

The ablations show that CROMA’s combined objectives and spatial attention biases are central to performance, while 2D-ALiBi enables strong test-time extrapolation to larger images.

  • Ablation Setup: The ablation setup compares frozen unimodal and multimodal representations through linear probing on BigEarthNet classification and DW-Expert-120 segmentation.
  • Ablation Results: Removing either self-supervised objective significantly degrades accuracy on classification and segmentation tasks.
  • Ablation Results: Across six evaluations, 2D-ALiBi without X-ALiBi outperforms PEG by 0.5% and 2D-sinusoidal embeddings by 1.5%.Adding X-ALiBi further improves average performance by 0.2%, particularly for multimodal performance.
  • Extrapolation Results: 2D-ALiBi outperforms PEG by 1.7% on 504 × 504 pixel images.
  • Extrapolation Results: A 0.7% mIoU drop occurs when testing on areas 17.64× larger than training images, extending from 225 to 3,969 patches per image.

5 Related Work

Related work has used self-supervised learning and positional encoding for remote sensing, but prior work had not leveraged relative position encoding in cross-attention.

  • Remote Sensing Representations: Remote sensing representation learning has formed positive pairs using spatial and temporal relationships, among other strategies.
  • Relative Position Encoding for ViTs: Relative position encoding can improve performance and sometimes allow models trained at one image size to extrapolate to larger images without retraining.
  • Relative Position Encoding for ViTs: PEG is a state-of-the-art relative position encoding method using convolution between ViT layers, whereas iRPE shows no extrapolation ability.
  • Relative Position Encoding for ViTs: No prior work identified by the authors leveraged relative position encoding in cross-attention.

6 Conclusion

CROMA jointly uses contrastive and reconstruction objectives to learn unimodal and multimodal Earth Observation representations. Its spatial encoding strategies support larger-image extrapolation, while the work remains focused on static Sentinel-1 and Sentinel-2 data.

  • CROMA learns unimodal and multimodal Earth Observation representations by jointly leveraging contrastive and reconstruction self-supervised objectives.
  • The framework extends a state-of-the-art position encoding method to 2D inputs and cross-attention, enabling extrapolation to larger images at test-time.
  • CROMA’s method can also be leveraged for spatially aligned multimodal data in medical imaging and autonomous vehicles.
  • The main limitation is the focus on static-in-time Sentinel-1 and Sentinel-2 data, leaving higher-resolution sensors and time-series data for future work.

A Appendix

The paper provides code and pretrained models for CROMA through its GitHub repository.

  • Code and pretrained models are available at the CROMA GitHub repository.

A.1 Other Pretraining Experiments

Additional experiments find that several objective, negative-sampling, decoder, positional-encoding, and tuning choices do not improve CROMA uniformly, supporting the selected default configuration.

  • A cross-modal MSE objective between location-matched patch encodings does not improve representations.
  • Equal contrastive and MAE loss weights are selected because reweighting either term does not uniformly improve representations.
  • Hard-negative mixing degrades performance in CROMA across the tested settings.
  • A tiny 1-layer, 512-d decoder performs similarly to a 6-layer, 768-d decoder in these experiments.
  • 2D-sinusoidal embeddings and PEG with shared masking perform poorly, whereas 2D-ALiBi places positional information in the attention matrix.
  • Lower masked tuning with 10%, 25%, or 50% mask ratios does not improve CROMA’s linear probing accuracy.

A.2 Two Reasons for 2D-ALiBi’s Performance

The appendix attributes 2D-ALiBi’s performance to transformation-invariant image representations and spatially diverse patch representations that retain local information.

  • Transformation invariance: 2D-ALiBi outperforms 2D-sinusoidal and PEG even when training and testing use the same image size.Its primary motivation remains enabling test-time extrapolation, but the performance difference also appears without extrapolation.
  • Transformation invariance: 0.992 average cosine similarity shows that 2D-ALiBi representations remain highly invariant to image flipping and rotation.2D-sinusoidal representations have 0.575 average cosine similarity, while PEG performs similarly to 2D-ALiBi on image classification and also learns these invariances.
  • Local information: 2D-ALiBi produces patch representations with greater spatial diversity than PEG, and X-ALiBi further improves that diversity.The analysis measures cosine similarity between patch encodings at different locations across 5,000 optical images.
  • Local information: 2D-sinusoidal embeddings produce the most diverse patch representations but encode positional information directly in patch encodings.The appendix uses patch-location prediction to measure positional information retained by patch encodings.

A.3.1 Data

Pretraining uses SSL4EO, a globally distributed, temporally repeated collection of paired Sentinel-1 and Sentinel-2 imagery.

  • SSL4EO contains Sentinel-1 and Sentinel-2 imagery from 250,000 locations worldwide, each imaged four times across a year.Each location is a 2.64 km × 2.64 km square, yielding 1 million 264 × 264-pixel samples for pretraining.

A.3.2 Implementation

The appendix documents hardware, preprocessing, benchmark construction, evaluation procedures, and distribution-shift concerns to make comparisons more consistent and clarify scope.

  • Implementation: Training uses an 8×A100-80GB DGX server, bfloat16 precision, batch size 7,200 for ViT-B, and cosine-decayed learning rates.The default image pipeline randomly crops 60–180-pixel squares from 264 × 264 inputs and resizes them to 120 × 120 pixels.
  • Evaluation: The evaluation reuses SatMAE conditions where possible, re-evaluates near-state-of-the-art models identically, and adds linear, nonlinear, kNN, and K-means analyses.The authors emphasize held-out validation and shared code and preprocessed datasets as part of improving Earth Observation evaluation.
  • Datasets and protocols: Benchmark protocols retain shared splits where applicable, while some studies use combined validation and test sets to improve ablation reliability.BigEarthNet uses 236,130 combined validation and test samples for ablations; fMoW-Sentinel introduces a 10% training split to reduce tuning costs.
  • Datasets and protocols: Complete-training-set fMoW-Sentinel results are reported without hyperparameter tuning because finetuning the 712,874-sample set is costly.CROMA-B and CROMA-L are each finetuned once with selected reasonable hyperparameters.
  • Datasets and protocols: Canadian Cropland contains 10 agricultural classes and is challenging to separate, while DW-Expert and MARIDA use held-out validation constructions for segmentation evaluation.Canadian Cropland uses 64 × 64 images; DW-Expert and MARIDA are divided into 96 × 96 images.
  • Limitations: Pretraining on settlement-centered SSL4EO samples may underrepresent sparsely populated regions and reduce representation quality there.The stated concern includes the Amazon rainforest, Sahara desert, and Australian outback.
Loading 2311.00566v1…