PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformers in Remote Sensing: A Survey

Abdulaziz Amer Aleissaee, Amandeep Kumar, Rao Muhammad Anwer, Salman Khan, Hisham Cholakkal, Gui-Song Xia, Fahad Shahbaz khan

arXiv:2209.01206v1cs.CV

TL;DR

Transformers are increasingly explored in remote sensing, but the field lacked a dedicated systematic review of their advances. This paper surveys more than 60 transformer-based methods across VHR, hyperspectral, and SAR imagery, and discusses challenges and open issues. It reports favorable performance across different remote sensing tasks and links this to long-range dependency modeling and representation flexibility.

  • Problem

    Existing surveys cover remote sensing or transformers in broader areas, but the literature lacked a comprehensive survey dedicated to transformers in remote sensing.

  • Method

    The paper reviews transformer-based approaches across VHR, hyperspectral, and SAR imagery, covering tasks including classification, detection, and segmentation.

  • Results

    The survey covers more than 60 research works and observes favorable transformer performance across different remote sensing tasks.

  • Takeaways & Limitations

    Self-attention’s long-range dependency modeling and representation flexibility, together with publicly available transformer backbones, support continued exploration in remote sensing.

Abstract

from arXiv · show

Deep learning-based algorithms have seen a massive popularity in different areas of remote sensing image analysis over the past decade. Recently, transformers-based architectures, originally introduced in natural language processing, have pervaded computer vision field where the self-attention mechanism has been utilized as a replacement to the popular convolution operator for capturing long-range dependencies. Inspired by recent advances in computer vision, remote sensing community has also witnessed an increased exploration of vision transformers for a diverse set of tasks. Although a number of surveys have focused on transformers in computer vision in general, to the best of our knowledge we are the first to present a systematic review of recent advances based on transformers in remote sensing. Our survey covers more than 60 recent transformers-based methods for different remote sensing problems in sub-areas of remote sensing: very high-resolution (VHR), hyperspectral (HSI) and synthetic aperture radar (SAR) imagery. We conclude the survey by discussing different challenges and open issues of transformers in remote sensing. Additionally, we intend to frequently update and maintain the latest transformers in remote sensing papers with their respective code at: https://github.com/VIROBO-15/Transformer-in-Remote-Sensing

I. INTRODUCTION

Remote sensing has adopted transformers as a complement or alternative to CNNs because self-attention can model global, content-dependent relationships beyond CNNs’ local receptive fields. This survey reviews the rapidly growing literature across remote sensing modalities and tasks, while identifying challenges and future directions.

  • CNNs and vision transformers: CNNs capture local interactions efficiently but have limited ability to model long-range dependencies and use content-independent filters.Their spatial connectivity and translation equivariance provide useful architectural biases, while stationary filters constrain global, content-dependent modeling.
  • CNNs and vision transformers: Vision transformers use self-attention to learn relationships among sequence elements and capture global interactions with flexible receptive fields.The mechanism updates each token by aggregating global knowledge from other tokens in the sequence.
  • Transformers in remote sensing: Transformer-based frameworks are increasingly used for remote sensing tasks including classification, change detection, pan sharpening, building detection, and image captioning.Approaches use either ImageNet pre-training or remote sensing pre-training with vision transformers.
  • Survey scope and contributions: The survey provides a holistic review of transformers-based models in remote sensing and covers more than 60 recent research works.It presents an overview of CNNs and transformers, then discusses challenges and research directions.
  • Survey scope and contributions: The review spans very high-resolution, hyperspectral, and synthetic aperture radar imagery across remote sensing research problems.The paper is organized around imaging modalities, model overviews, and transformer applications in VHR, hyperspectral, and SAR imagery.
  • Related surveys: This work addresses a gap left by broader remote sensing, computer vision, and medical imaging surveys by focusing specifically on transformers in remote sensing.The authors describe it as the first comprehensive survey dedicated to transformer progress in VHR, hyperspectral, and SAR image analysis.

III. REMOTE SENSING IMAGING DATA

Remote sensing imagery is characterized by spatial, spectral, radiometric, and temporal resolutions and spans VHR, hyperspectral, and SAR modalities. This section introduces these modalities and the CNN-based foundations reviewed before vision transformers.

  • Remote sensing imaging data: Remote sensing data are characterized by spatial, spectral, radiometric, and temporal resolutions.Spatial resolution concerns pixel size and separable scene detail; spectral resolution concerns wavelength discrimination; radiometric resolution concerns pixel information range; temporal resolution concerns revisit interval.
  • Very high-resolution imagery: VHR imagery supports land-use change detection, object detection, instance segmentation, precision agriculture, and emergency response.
  • Hyperspectral imagery: Hyperspectral imagery captures continuous-spectrum information from ultraviolet to infrared wavelengths using fine spectral resolution.
  • Synthetic aperture radar imagery: SAR imagery is produced by emitting and receiving electromagnetic signals and offers high spatial resolution and all-weather capability.Its applications include disaster management, forestry, and hydrology.
  • Convolutional neural networks: CNNs extract feature maps through convolution and reduce their spatial size through pooling before fully connected classification layers.Their development has supported advances in hardware, optimization, and open-source libraries, while their local inductive biases aid feature learning.

B. Vision Transformers

Vision transformers process image patches with self-attention to model long-range dependencies and have been adapted from NLP and computer vision for remote sensing. The section describes self-attention variants and representative backbones, including ViT, Swin, and PVT.

  • Vision transformers: Vision transformers interpret images as patch sequences and process them with a transformer encoder for recognition.This patch-based design motivated applications across computer vision and remote sensing tasks.
  • Self-attention: Self-attention captures interactions among all patch embeddings by projecting them into queries, keys, and values.The mechanism aggregates global knowledge to update each token and thereby models long-range dependencies.
  • Masked self-attention: Masked self-attention sets attention ratings for future sequence entities to zero during prediction.An upper-triangular mask prevents attending to subsequent entities.
  • Multi-head attention: Multi-head attention concatenates multiple self-attention outputs, with separate learnable weight matrices for each head.A final linear transformation combines the concatenated head outputs.
  • Popular transformer backbones: ViT applies a pure transformer directly to image patches, whereas Swin introduces hierarchical features and local-window attention with cross-window connections.Swin’s restricted windows provide linear computational complexity with respect to input image size.
  • Popular transformer backbones: PVT uses a progressively shrinking pyramid and spatial-reduction attention for pixel-level dense prediction tasks.
  • Vision transformer properties: Self-attention dynamically calculates filters, and vision transformers have been reported as more robust to severe occlusions, domain shifts, and perturbations.

V. TRANSFORMERS IN VHR IMAGERY

Transformers-based methods are increasingly applied to VHR scene classification and object detection, often combining transformers with CNNs or using remote sensing pre-training. Across reviewed benchmarks, classification methods achieve better recognition performance, while detection methods generally show similar accuracy with slight gains from Swin-T.

  • Scope: VHR imagery research reviews transformers-based approaches for scene classification and object detection.Scene classification assigns semantic labels to high-resolution images, while detection recognizes and localizes objects with rectangle or oriented bounding boxes.
  • Scene Classification: CTNet concatenates features from parallel CNN and ViT streams and optimizes them jointly with cross-entropy and center losses.The CNN stream captures local structural information, while the ViT stream captures semantic features.
  • Scene Classification: Remote sensing pre-training trains vision-transformer backbones from scratch on MillionAID before fine-tuning them for tasks including scene classification.The survey contrasts remote sensing pre-training with ImageNet pre-training and presents response-map comparisons between them.
  • Object Detection: Recent detection methods use hybrid CNN-transformer designs, DETR-based architectures, and adaptive point representations for oriented aerial objects.Oriented RepPoints learns point samples for classification, localization, and orientation, using quality assessment and sample assignment to select high-quality training points.
  • Object Detection: Most reviewed DOTA detectors obtain similar mAP, with a slight performance improvement when using the Swin-T backbone.DOTA evaluates oriented bounding-box detection across 2,806 aerial images and 15 object categories.

C. Image Change Detection

Transformers-based approaches address remote sensing change detection by comparing multi-temporal imagery to produce pixel-level binary change maps. The reviewed methods combine CNNs and transformers to integrate semantic, temporal, local, global, and multiscale information, with comparisons reported on WHU and LEVIR.

  • Image Change Detection: Image change detection generates binary pixel-level maps by comparing multi-temporal or bi-temporal remote sensing images.The task supports applications including agriculture, urban planning, and map revision.
  • Methods: UVACD uses a CNN backbone for high-level semantic features and transformers for temporal information interaction in change-feature generation.Other hybrid designs combine CNN encoders with transformers and UNet-style multiscale processing.
  • Methods: Hybrid-TransCD captures fine-grained and large-object features through heterogeneous tokens generated with multiple receptive fields.This design targets both local detail and larger-scale change structures.
  • Evaluation: Change-detection approaches are compared by F1 score on the WHU and LEVIR benchmarks.WHU contains one pair of 0.075m images, whereas LEVIR contains 637 pairs of 0.5m images.
  • Related Dense-Prediction Tasks: The survey also covers transformer-based semantic segmentation, building extraction, and instance segmentation methods.These approaches commonly combine CNN-derived local details with transformer-derived global context or enhance local and global feature information.

E. Others

Transformers have been explored for additional VHR remote-sensing tasks including image captioning and super-resolution. The reviewed approaches commonly combine transformer-based global context modeling with CNN-derived local or multiscale features.

  • Image Captioning: Image captioning generates semantically natural descriptions of remote-sensing images, with transformer methods adapting encoders and decoders for this task.Reported designs include residual connections, dropout, adaptive feature fusion, multiscale CNN features, and topic-token priors.
  • VHR Methods: The survey organizes representative VHR transformer methods by task, datasets, metrics, and methodological highlights.The overview includes change detection, segmentation, building extraction, image captioning, and image super-resolution.
  • VHR Methods: Several VHR approaches combine local and global information through hybrid CNN-transformer architectures, multiscale feature fusion, or self-attention.Examples include frameworks for segmentation, change detection, building extraction, and crop segmentation.
  • Image Captioning: Topic-token mask transformers integrate a topic token into the encoder and use it as a decoder prior to capture global semantic relationships.
  • Image Super Resolution: Remote-sensing image super-resolution reconstructs high-resolution images from low-resolution inputs, including multi-image fusion of the same scene.Reviewed methods use multistage enhancement, conventional super-resolution modules, CNN-transformer hybrids, or residual-block encoders with fusion and super-pixel convolution decoders.

VI. TRANSFORMERS IN HYPERSPECTRAL IMAGING

Hyperspectral imaging represents data with multiple spectral bands and supports diverse remote-sensing applications. The survey reviews transformer-based approaches across hyperspectral imaging tasks.

  • Hyperspectral images contain several spectral bands, and their analysis is important across a wide range of remote-sensing problems.
  • The survey reviews recent transformer-based approaches for different hyperspectral imaging tasks.
  • The hyperspectral imaging review is organized around task-specific transformer approaches.

A. Image Classification

Hyperspectral image classification assigns a category label to each pixel and is studied with pure transformers, hybrid CNN-transformers, and multimodal fusion models. The reviewed methods target spectral, spatial, and long-range dependencies using pixel or patch representations and fused modalities.

  • Hyperspectral image classification assigns a category label to each pixel acquired through hyperspectral sensors.
  • Pure Transformers-based Methods: Pure transformer methods include HSI-BERT for global dependencies and SpectralFormer for pixel-wise or patchwise spectral modeling.HSI-BERT is described as flexible across regions without pre-training, while SpectralFormer is designed to capture spectrally local sequences.
  • Hybrid CNN-Transformers based Methods: Hybrid CNN-transformer methods combine local information from convolutions with long-range dependencies from transformers for hyperspectral classification.CTN uses center position encoding to combine pixel positions with spectral features and obtains local-global features through convolutional transformer blocks.
  • Experimental Comparison: Table VI compares representative CNN, pure-transformer, and hybrid CNN-transformer classifiers on the Indian Pines and Pavia benchmarks using overall accuracy.The comparison reports results with 200 training samples for each category.
  • Multi-modal Fusion Transformers based Methods: Multimodal fusion transformers combine hyperspectral data with modalities such as LiDAR and SAR through fused class tokens and patch tokens.MFT derives class tokens from multimodal data alongside standard hyperspectral patch tokens.

B. Hyperspectral Pansharpening

Hyperspectral pansharpening enhances the spatial resolution of low-resolution hyperspectral imagery using registered panchromatic information while preserving spectral information. Transformer methods address this through spatial-spectral extraction, cross-modal feature learning, and multiscale or multiresolution fusion.

  • Hyperspectral pansharpening enhances low-resolution hyperspectral images spatially using registered panchromatic information while preserving their spectral information.The task supports downstream remote-sensing applications including classification and change detection.
  • Transformer-based pansharpening methods use Swin transformers to extract spatial and spectral information.
  • Cross-modal transformer approaches formulate multispectral and panchromatic features as keys and queries for joint feature learning.An invertible neural module is used to fuse the features for generating pansharpened images.
  • Other approaches use separate panchromatic and hyperspectral feature extractors with attention mechanisms for fusion.
  • The survey summarizes transformer-based approaches across hyperspectral imaging tasks in Table VII.

VII. TRANSFORMERS IN SAR IMAGERY

SAR imagery has distinctive acquisition properties, including robustness to day, night, and fog, and the survey reviews transformer-based approaches for SAR tasks.

  • SAR uses electromagnetic-wave signals transmitted from a sensor platform to image Earth’s surface.

A. SAR Image Interpretation

Transformer-based SAR interpretation methods address classification, segmentation, and detection by modeling long-range dependencies or combining global and local features. The surveyed approaches include pure transformer and hybrid CNN-transformer frameworks across several target types.

  • Classification: ViT-PolSAR treats SAR image-patch pixel values as tokens and uses self-attention to capture long-range dependencies for polarimetric SAR classification.
  • Classification: GLNS combines a lightweight CNN and efficient vision transformer to capture local and global SAR features before classification.
  • Segmentation and Detection: SAR segmentation and detection are challenging because speckle is multiplicative noise that increases with back-scattering radar magnitude.
  • Classification: Figure 13 compares supervised whole-map classification results from Wishart, RBF-SVM, CV-CNN, 3D-CNN, PSENet, SF-CNN, and ViT-PolSAR on ALOS2 San Francisco.
  • Segmentation and Detection: CRTransSar combines CNNs and transformers to capture local and global information for SAR object detection.
  • Segmentation and Detection: A transformer-based method for 3D oil-tank detection uses incidence angle as a prior token and scattering centers to refine predictions.

B. Others

Beyond classification, detection, and segmentation, transformers have been explored for SAR despeckling, change detection, and image registration. These methods use global dependencies, convolution-enhanced representations, or hybrid CNN-transformer matching.

  • Others: Only a few studies explore transformers for SAR problems beyond classification, detection, and segmentation, including image despeckling.
  • SAR Image Despeckling: A despeckling framework learns global dependencies among SAR regions and trains end-to-end on synthetic speckled data with a composite loss.
  • Change Detection in SAR Images: Self-supervised contrastive learning with a convolution-enhanced transformer constructs hierarchical representations to distinguish changes in high-resolution SAR images.
  • SAR Image Registration: A hybrid CNN-transformer method performs dense matching to register large SAR images under weak-texture conditions.
  • Others: The survey provides a holistic overview of transformer techniques across SAR imaging tasks in Table VIII.

VIII. DISCUSSION AND CONCLUSION

The survey reviews transformers across VHR, hyperspectral, and SAR imagery and reports favorable performance across varied tasks, while identifying pre-training, computational cost, dataset scale, and domain adaptation as open challenges.

  • Scope and Findings: The survey covers VHR, hyperspectral, and SAR imagery, including transformer approaches for classification, detection, and segmentation.
  • Scope and Findings: More than 60 transformer-based remote sensing research works are reviewed.
  • Scope and Findings: The survey observes favorable transformer performance across remote sensing tasks, likely related to capturing long-range dependencies and representing features.
  • Open Research Directions: Most recognition approaches use supervised ImageNet or remote-sensing pre-training, leaving large-scale self-supervised pre-training on unlabeled imagery open.
  • Open Research Directions: Hybrid CNN-transformer architectures are common, while global self-attention typically incurs higher computational cost.
  • Open Research Directions: Heavy backbones can improve aerial detection accuracy but significantly slow detector speed, motivating lightweight transformer backbones.
  • Open Research Directions: Many methods use transformers in a plug-and-play manner, motivating domain-specific architectural components and loss formulations.
Loading 2209.01206v1…