PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised Learning in Remote Sensing: A Review

Yi Wang, Conrad M Albrecht, Nassim Ait Ali Braham, Lichao Mou, Xiao Xiang Zhu

arXiv:2206.13188v2cs.CV

TL;DR

The paper addresses the shortage of high-quality annotations and the limited development of SSL in remote sensing. It reviews SSL concepts, methods, datasets, and evaluation, then benchmarks modern methods and studies augmentations. The review reports meaningful transferable representations and strong label efficiency, while identifying multimodal and multitemporal learning as a future challenge.

  • Problem

    Remote sensing has abundant data but insufficient high-quality labels, while a didactic and up-to-date SSL introduction for the community is missing.

  • Method

    The paper systematically reviews SSL, proposes an extended taxonomy, benchmarks four contrastive methods on three remote-sensing datasets, and analyzes data augmentations and transfer learning.

  • Results

    Self-supervised pretraining produces meaningful representations across remote-sensing datasets, and fine-tuning with 50% of labels outperforms supervised learning with full labels.

  • Takeaways & Limitations

    SSL shows potential for remote sensing by using unlabeled data and enabling transfer across datasets and downstream applications.

Abstract

from arXiv · show

In deep learning research, self-supervised learning (SSL) has received great attention triggering interest within both the computer vision and remote sensing communities. While there has been a big success in computer vision, most of the potential of SSL in the domain of earth observation remains locked. In this paper, we provide an introduction to, and a review of the concepts and latest developments in SSL for computer vision in the context of remote sensing. Further, we provide a preliminary benchmark of modern SSL algorithms on popular remote sensing datasets, verifying the potential of SSL in remote sensing and providing an extended study on data augmentations. Finally, we identify a list of promising directions of future research in SSL for earth observation (SSL4EO) to pave the way for fruitful interaction of both domains.

I. INTRODUCTION

Self-supervised learning addresses the costly shortage of high-quality annotations by learning representations from unlabeled data for transfer to downstream remote-sensing applications. This review introduces and categorizes SSL, benchmarks representative methods and datasets, and identifies directions for SSL research in Earth observation.

  • Motivation: Large-scale annotation is laborious, time-consuming, and expensive, limiting deep learning when training data require extensive supervision.Remote-sensing datasets also require frequent updates and application-specific labeling.
  • Scope and contributions: The review provides an introduction and literature review, a taxonomy spanning computer vision and remote sensing, and benchmarks on ImageNet and three multispectral satellite datasets.The remote-sensing datasets are BigEarthNet, SEN12MS, and So2Sat-LCZ42.
  • Scope and contributions: The paper connects natural imagery with remotely sensed data and discusses future research directions for self-supervised learning in Earth observation.It aims to bridge the gap between SSL progress in computer vision and its more limited activity in remote sensing.
  • Motivation: Remote sensing faces a trade-off between annotation quantity and quality: large noisy datasets can bias models, while few good labels can cause overfitting.Label noise arises from image quality and the complexity of specific applications.
  • Self-supervised learning: SSL learns representations from unlabeled data using automatically generated supervision, reducing reliance on manual annotation before downstream transfer.Common objectives reconstruct inputs, predict self-produced labels, or contrast semantically similar inputs.

C. Performance Evaluation in Self-Supervised Learning

Self-supervised models are evaluated by transferring learned representations to downstream tasks. The review describes linear probing, K-nearest neighbors, fine-tuning, and qualitative visualization as complementary evaluation procedures.

  • Evaluation procedures: Downstream tasks such as scene classification, semantic segmentation, and object detection assess the generalizability of a pretrained representation.These tasks transfer the pretrained model to specific applications.
  • Evaluation procedures: Linear probing freezes the encoder and trains a linear classifier on its representations to measure embedding linear separability.The procedure evaluates representations without updating the learned encoder parameters.
  • Evaluation procedures: K-nearest neighbors classifies test images through weighted voting among their nearest feature-space neighbors.It is a non-parametric procedure applicable to classification tasks.
  • Evaluation procedures: Fine-tuning initializes downstream-task training with pretrained parameters and supports tasks beyond classification.It is described as the most general quantitative evaluation procedure.
  • Qualitative evaluation: Kernel and feature-map visualizations compare learned filters or hidden representations with supervised-training counterparts to analyze self-supervised features.Kernel visualization targets CNN first-layer kernels, while feature-map visualization examines spatial attention in CNNs and vision transformers.

II. GENERATIVE, PREDICTIVE, AND CONTRASTIVE: A TAXONOMY OF SELF-SUPERVISED LEARNING

The paper organizes self-supervised learning into generative, predictive, and contrastive methods, then reviews generative approaches that reconstruct or generate input data. Generative methods include autoencoders, GANs, and variants applied to remote sensing.

  • The taxonomy distinguishes generative reconstruction or generation, predictive self-generated labels, and contrastive similarity learning.
  • Generative Methods: Autoencoders encode input x into a latent vector z and decode z to reconstruct x, using reconstruction error for self-supervision.
  • Generative Methods: Autoencoders require constraints because an unconstrained encoder-decoder can learn a trivial identity mapping instead of expressive representations.
  • Generative Methods: Sparse, denoising, variational, and masked autoencoders impose alternative constraints or corruption schemes to improve representation learning.
  • Generative Methods: A VAE encodes each input into Gaussian mean and variance vectors, samples a latent representation, and reconstructs the input from that sample.Its loss combines reconstruction with a KL-divergence term that pushes the latent distribution toward a unit Gaussian.
  • Generative Methods: Autoencoders have been widely used for representation learning from multispectral, hyperspectral, and SAR remote sensing data.

2) Generative Adversarial Networks (GAN):

GANs learn through adversarial minimax optimization between a generator that produces synthetic samples and a discriminator that distinguishes real from fake data. Extensions such as AAE, BiGAN, and ALI incorporate or infer latent representations, while remote-sensing applications remain limited.

  • Generative Adversarial Networks (GAN): A GAN generator maps random latent vectors z to fake samples G(z), while the discriminator distinguishes generated samples from real data.
  • Generative Adversarial Networks (GAN): The standard GAN objective is optimized adversarially, with G trying to fool D and D distinguishing real and fake samples.
  • Generative Adversarial Networks (GAN): AAE approximates the inverse generator to obtain latent representations for given samples using an adversarially trained autoencoder.
  • Generative Adversarial Networks (GAN): BiGAN and ALI add an encoder and train a discriminator on joint data-latent tuples rather than only generated data.
  • Generative Adversarial Networks (GAN): GAN-based self-supervised pre-training in remote sensing is limited, while existing works mainly target hyperspectral unmixing, image retrieval, and scene classification.

B. Predictive Methods

Predictive methods use self-generated labels to predict selected properties rather than reconstructing complete images, aiming to emphasize high-level semantic information. Remote-sensing pretext tasks exploit spatial, spectral, temporal, and other semantic contexts, but their design must reflect domain-specific structure.

  • B. Predictive Methods: Predictive methods design pretext tasks with auto-generated labels, focusing on selected data properties instead of pixel-level image reconstruction.
  • B. Predictive Methods: Predictive objectives can reduce emphasis on irrelevant low-level details and incorporate long-range correlations important for image understanding.
  • B. Predictive Methods: Pretext tasks are categorized by context as spatial, spectral, temporal, and other semantic contexts.
  • Spatial Context: Spatial tasks include relative-position prediction, jigsaw puzzles, rotation prediction, and image inpainting.
  • Spatial Context: Remote-sensing studies apply rotation and inpainting to tasks including scene classification, object detection, and semantic segmentation.
  • Spatial Context: Jigsaw puzzles are rarely used in remote sensing, potentially because translational invariance is prominent across many overhead scenes.
  • Spectral Context: Spectral pretext-task design is challenging because multispectral and hyperspectral data encode fine physical-surface details beyond standard RGB.

3) Temporal Context:

Temporal context provides self-supervised signals by exploiting ordering, prediction, and relationships across timestamps. In remote sensing, these signals support change detection and satellite time-series representation learning despite limited satellite video.

  • Temporal Context: Temporal pretext tasks either predict the correct frame sequence after timestamp shuffling or reconstruct masked frames.Frame-order methods include binary order verification and explicit timestamp assignment, while missing-frame prediction estimates future snapshots.
  • Temporal Context: Frame sampling strategies can improve temporal pretext tasks by selecting footage frames with significant motion.The review notes that video frames differ in importance for understanding events.
  • Temporal Context: Remote sensing temporal stamps are important for applications including change detection and crop type classification.Satellite-based video recording is not yet common, but multitemporal observations remain useful for these tasks.
  • Temporal Context: A change-detection method identifies corresponding patches across two snapshots and yields improved, robust differentiation.Its discriminator imitates the discriminator in generative adversarial networks while a generator produces self-supervised representations.
  • Temporal Context: Transformer-based satellite time-series methods predict contaminated observations or masked-patch central pixels to learn spectral-temporal representations.The first approach uses an entire pixel time series, while the later work performs patch-based representation learning.
  • Temporal Context: Designing or integrating suitable pretext tasks remains an important research question.The review identifies pretext-task design as an unresolved issue for self-supervised learning.

1) Negative Sampling:

Negative-sampling methods learn representations by comparing semantically similar and dissimilar samples. Their design centers on selecting positive and negative pairs while addressing the computational cost and staleness of large negative sets.

  • Negative Sampling: Contrastive learning compares an anchor with positive and negative samples to pull similar representations together and repel dissimilar ones.Positive samples should preserve the anchor’s semantics, often through augmentation, while negatives come from other data points.
  • Negative Sampling: Contrastive methods commonly use one positive and N −1 negative samples in an N-way classification objective.In CPC, consecutive timestamps are positive samples and clips from other scenes are negatives.
  • Negative Sampling: Instance discrimination projects each image into a 128-dimensional L2-normalized space and scatters training features across the unit sphere.This treats individual training samples as the basis for learning the feature embedding.
  • Negative Sampling: More negative examples can better cover the underlying distribution, but mini-batch and hardware-memory limits restrict their number.This motivates larger batches or mechanisms that maintain negatives outside the current mini-batch.
  • Negative Sampling: MoCo replaces a memory bank with a momentum encoder and a queue of progressively replaced keys.The dictionary can be much larger than a typical mini-batch, while the momentum encoder avoids maintaining a separate memory bank.
  • Negative Sampling: SimCLR uses end-to-end training with batch size 8196 and emphasizes hard positive sampling through data augmentation.Its augmentation study considered 10 forms of augmentation; the review separately highlights cropping, color jittering, grayscale, blurring, and flipping.

2) Clustering:

Clustering-based self-supervised methods create pseudo-labels or local groups to organize representations without human labels. Later approaches reduce the time and performance costs of separate clustering stages through online or augmentation-based designs.

  • Clustering: DeepCluster iteratively applies K-means clustering to produce pseudo-labels for representation learning.The clustering step provides an alternative to unavailable human labels.
  • Clustering: Self-labeling uses optimal transport for pseudo-labels, whereas local aggregation directly optimizes a local soft-clustering metric.Local aggregation requires no extra readout layer and adds only a small amount of computation.
  • Clustering: Early two-stage clustering methods are time-consuming and poorer-performing than later instance-discrimination methods such as SimCLR and MoCo.Later methods remove the slow clustering stage and use efficient data augmentation strategies.
  • Clustering: Remote sensing studies combine DeepCluster with triplet contrastive learning for multimodal, multitemporal change detection and integrate DeepCluster, BYOL, and MoCo-v2 for segmentation maps.Other comparisons examine DeepCluster, VAE, colorization, and BiGAN for remote sensing image retrieval.

3) Knowledge Distillation:

Knowledge-distillation methods use teacher–student networks to match representations from augmented views without requiring negative samples. BYOL, SimSiam, and related methods differ in how asymmetry, stop-gradient, momentum updates, and additional objectives prevent collapse.

  • Knowledge Distillation: Teacher–student distillation methods optimize similarity between two augmented views and avoid negative samples through asymmetric rules or architectures.This family commonly uses Siamese-like networks to transfer knowledge from student to teacher.
  • Knowledge Distillation: BYOL uses online and target branches, with target parameters updated by exponential moving average and a stop-gradient operation.Its architecture resembles MoCo but does not use negative samples.
  • Knowledge Distillation: SimSiam finds that the stop-gradient operation is the most critical component for stable target representations, while the predictor is not necessary to prevent collapse.It also characterizes itself as SimCLR without negatives, SwAV without online clustering, and BYOL without the momentum encoder.
  • Knowledge Distillation: DINO extends self-distillation to vision transformers using centered, temperature-normalized teacher and student outputs with cross-entropy similarity.Stop-gradient and momentum updates are used, while EsViT adds a region-level contrastive task to DINO’s global view-level task.
  • Knowledge Distillation: Remote sensing applications combine GANs with BYOL-style similarity losses and propose gated self-attention and pyramidal convolution for the architecture.The discriminator encodes fake and real images as two input views in a Siamese setup.
  • Knowledge Distillation: A pixel-level SimSiam approach enforces point-level consistency across views and uses background-swap augmentation to focus on foreground.The method targets change detection.

4) Redundancy Reduction:

Redundancy-reduction methods learn invariant representations while limiting redundancy, offering alternatives to explicit or implicit negative samples for avoiding trivial solutions.

  • Barlow Twins avoids trivial solutions by matching embeddings of distorted views while minimizing redundancy between embedding components.
  • Its objective uses an invariance term and a redundancy-reduction term computed from the cross-correlation matrix of two identical networks.λ controls the trade-off between the terms, while the cross-correlation matrix is computed across the batch dimension.
  • VICReg extends redundancy reduction with variance, invariance, and covariance regularization to avoid the trivial solution.
  • Remote-sensing applications of redundancy-reduction methods include continual-learning pre-training with Barlow Twins and few-shot hyperspectral-image classification.

A. Characterisics and Challenges of Remote Sensing Data

Remote-sensing data differ from natural images through geolocation, physical measurement properties, object density, temporal information, and diverse sensing modalities, requiring specialized SSL designs.

  • Remote-sensing modalities differ substantially in imaging geometry, content, spectral resolution, spatial resolution, and sensor characteristics.Multispectral imagery typically uses 3 to 15 bands, hyperspectral imagery can use hundreds, and SAR has distinct dynamic range, speckle, geometry, and complex-domain properties.
  • Geolocation enables geographic pretext tasks and cooperation with geographical resources such as land-cover databases.
  • Augmentations must preserve physically meaningful properties, because careless transformations can alter information needed to recognize objects.
  • Remote-sensing images contain many repeated objects, making some natural-image pretext tasks and augmentations less suitable.The paper highlights jigsaw puzzles on low-resolution satellite images and rotations that affect building shadows.
  • Temporal information is important across modalities, but remote-sensing timestamps and objectives such as change detection differ from video frames.Seasonal information is used as an augmentation in Seasonal Contrast.

4) Multi-sensor Fusion:

Remote-sensing applications span image-, pixel-, and patch-level tasks, while multimodal sensing motivates self-supervised approaches that account for relationships among modalities and task requirements.

  • 4) Multi-sensor Fusion:: Multi-sensor fusion can treat different modalities as augmented views of the same semantic input for contrastive pretext tasks or data augmentation.
  • 4) Multi-sensor Fusion:: Self-supervision should be selected with the eventual downstream task in mind because methods can affect tasks differently.
  • Image-level tasks emphasize global knowledge, while pixel-level tasks require fine local representations for applications such as semantic segmentation.
  • Patch-level tasks, including object detection, require both object location and labels, benefiting from integrated global and local representations.
  • Patch-based change detection compares image tiles rather than pixels when temporal misalignment or rapid preprocessing and limited bandwidth matter.
  • The review positions its benchmark as a reference study because recent contrastive methods were mainly developed and evaluated on natural images.

A. Methods, Datasets and Implementation Details

The benchmark evaluates four modern SSL methods on three Sentinel-2 datasets using frozen-feature linear probing and fine-tuning, finding meaningful transfer and method-dependent behavior.

  • A. Methods, Datasets and Implementation Details: The benchmark compares MoCo-v2, SwAV, SimSiam, and Barlow Twins across BigEarthNet, SEN12MS, and So2Sat-LCZ42.Each method represents a different contrastive-learning sub-category.
  • A. Methods, Datasets and Implementation Details: The experiments pre-train ResNet-18 encoders for 100 epochs and evaluate with linear probing or fine-tuning on Sentinel-2 data.
  • 1) General Comparsion of SSL Methods:: All evaluated SSL methods produce meaningful representations after direct transfer from natural-image settings to satellite imagery.Frozen features achieve performance comparable to fully supervised training while training far fewer parameters.
  • 1) General Comparsion of SSL Methods:: MoCo-v2 generally outperforms the other methods across datasets, supporting negative sampling as a robust transferable baseline in this benchmark.
  • 1) General Comparsion of SSL Methods:: SimSiam training is unstable: its loss drops rapidly in early epochs and then vibrates in later epochs.
  • Stronger cropping improves BigEarthNet performance, with larger minimum crop sizes producing worse results.

3) Regime of Limited Labels:

Self-supervised pre-training is especially advantageous when labeled data are scarce, including for fine-tuning and cross-dataset transfer. The review also identifies unresolved challenges spanning model theory, augmentations, datasets, modalities, computation, architectures, and task integration.

  • 3) Regime of Limited Labels:: Fine-tuning with self-supervised pre-training outperforms supervised learning across label scenarios, with larger advantages as labeled data decrease.The reported advantage is especially substantial when only tiny amounts of labels are available.
  • 4) Transfer Learning:: On EuroSat, a ResNet18 pre-trained on BigEarthNet transfers representations across datasets, with fine-tuning on 50% of labels surpassing full-label supervised learning.Fine-tuning on 10% of labels already achieves performance close to full-label supervised learning.
  • V. CHALLENGES AND FUTURE DIRECTIONS: Model collapse remains a major challenge because the theory underlying contrastive methods, particularly methods without negative samples, is not well understood.The review reports that SimSiam can easily collapse on Earth observation data and calls for deeper theoretical study.
  • V. CHALLENGES AND FUTURE DIRECTIONS: Pretext tasks and augmentations require remote-sensing-specific evaluation because findings from natural-image benchmarks may not fully apply to multispectral and other modalities.The review calls for studies linking useful choices to different types of remote-sensing data.
  • V. CHALLENGES AND FUTURE DIRECTIONS: Future SSL4EO research must address limited and biased pre-training datasets, multimodal and multitemporal complexity, computational cost, backbone design, and integration of few or weak labels.The review specifically points toward large-scale uncurated data, modality and timestamp balancing, efficient training, vision transformers, and task-oriented weak supervision.
  • VI. CONCLUSION: The article systematically reviews SSL for remote sensing, organizes methods into three categories, benchmarks modern contrastive approaches, and discusses open problems and future directions.Its benchmark and extended analysis cover four modern contrastive methods across three popular remote-sensing datasets.
Loading 2206.13188v2…