PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning Meets SAR

Xiao Xiang Zhu, Sina Montazeri, Mohsin Ali, Yuansheng Hua, Yuanyuan Wang, Lichao Mou, Yilei Shi, Feng Xu, Richard Bamler

arXiv:2006.10027v2eess.IVcs.LGstat.ML

TL;DR

Deep learning has achieved limited adoption beyond optical remote sensing, leaving substantial SAR potential underexploited. This paper introduces relevant models, analyzes SAR-specific pitfalls, reviews six application areas and benchmarks, and identifies future research directions. The review concludes that broader exploitation is mainly limited by inadequate representative datasets and a lack of tailored models for SAR signal characteristics.

  • Problem

    Deep learning in remote sensing remains mostly focused on optical data, while SAR and InSAR lack operational big-data examples and representative benchmarks.

  • Method

    The paper introduces deep learning models, analyzes SAR-specific characteristics and pitfalls, reviews six SAR application fields, and discusses benchmarks and future directions.

  • Results

    The review concludes that full exploitation of deep learning in SAR is mainly limited by insufficient representative benchmark datasets and a lack of tailored models incorporating SAR signal characteristics.

  • Takeaways & Limitations

    Deep learning in SAR remains an under-exploited research field requiring models and datasets tailored to SAR data.

Abstract

from arXiv · show

Deep learning in remote sensing has become an international hype, but it is mostly limited to the evaluation of optical data. Although deep learning has been introduced in Synthetic Aperture Radar (SAR) data processing, despite successful first attempts, its huge potential remains locked. In this paper, we provide an introduction to the most relevant deep learning models and concepts, point out possible pitfalls by analyzing special characteristics of SAR data, review the state-of-the-art of deep learning applied to SAR in depth, summarize available benchmarks, and recommend some important future research directions. With this effort, we hope to stimulate more research in this interesting yet under-exploited research field and to pave the way for use of deep learning in big SAR data processing workflows.

I. MOTIVATION

Deep learning extracts hierarchical representations and is attracting attention across SAR applications, but its potential remains underused. The paper introduces models, discusses SAR-specific pitfalls, reviews applications and benchmarks, and identifies future directions.

  • I. MOTIVATION: Deep learning methods use hierarchical architectures to extract feature representations from raw data for diverse tasks.CNNs learn low- and high-level image features through convolutional and pooling layers.
  • I. MOTIVATION: Early SAR applications include terrain classification, object detection, parameter inversion, despeckling, InSAR, and SAR-optical data fusion.
  • I. MOTIVATION: Deep learning in SAR and InSAR has not reached operational big-data processing or satellite-mission production chains.
  • I. MOTIVATION: The paper introduces common models, analyzes SAR characteristics, reviews six application areas, discusses benchmark datasets, and outlines future directions.
  • I. MOTIVATION: High-quality benchmark datasets constrain what supervised models can learn and have historically supported major advances in deep learning.

1) Convolutional Neural Networks (CNN):

The paper surveys CNNs and related deep learning architectures used for visual and remote-sensing tasks, emphasizing representation learning, skip connections, recurrence, generation, and learning paradigms.

  • 1) Convolutional Neural Networks (CNN):: AlexNet’s 15.3% top-5 test error on ILSVRC-2012 versus 26.2% for the second-best system accelerated attention to CNNs.
  • 1) Convolutional Neural Networks (CNN):: ResNet, U-Net, and DenseNet use skip connections to reduce information loss and vanishing gradients while enabling deeper networks.U-Net concatenates encoder and decoder features for fine-grained segmentation information.
  • 1) Convolutional Neural Networks (CNN):: RNNs process variable-length sequences through recurrent units that combine current inputs with previous-state outputs.Their weights can be replaced with convolutional kernels for visual time-series tasks.
  • 1) Convolutional Neural Networks (CNN):: GANs use a generator and discriminator in a game-theoretic setup to generate samples resembling the training-data distribution.
  • 1) Convolutional Neural Networks (CNN):: VAEs learn output distributions by modeling latent means and variances and adding a KL-divergence term to reconstruction loss.

3) Deep Reinforcement Learning (DeepRL):

Deep reinforcement learning learns actions from state transitions and rewards, while SAR deep learning requires architectures and preprocessing that respect dynamic range, speckle, geometry, and complex phase.

  • 3) Deep Reinforcement Learning (DeepRL):: Reinforcement learning learns preferred actions for states through environmental feedback and rewards.Its model comprises states, actions that transition between states, and a reward system.
  • 3) Deep Reinforcement Learning (DeepRL):: AutoML searches deep-network hyperparameters because billions of combinations make manual selection costly in time, computation, and energy.
  • III. POSSIBLE PITFALLS: SAR images can have dynamic ranges up to 90 dB, so many approaches apply dynamic compression or normalization before CNN processing.
  • III. POSSIBLE PITFALLS: Speckle is multiplicative, and some optimal SAR feature estimators cannot be emulated by the first layer of a standard CNN.Logarithmic mapping makes speckle additive and reduces dynamic range, but a single convolutional layer still only approximates optimal estimators.
  • III. POSSIBLE PITFALLS: Rotation augmentation can create physically impossible SAR imagery because range and azimuth coordinates reflect the image-generation geometry.
  • III. POSSIBLE PITFALLS: SAR phase information requires CNNs, activation functions, and losses to handle complex numbers without distorting phase.In interferometry, arbitrary phase offsets should leave the output meaning unchanged, motivating phase linearity.
  • III. POSSIBLE PITFALLS: For phase regression, losses should account for phase cyclicity and may reduce dependence on signal magnitude.The reference signal is denoted by y.

IV. RECENT ADVANCES IN DEEP LEARNING APPLIED TO SAR

The paper reviews deep learning applications to SAR across six fields, reporting developments chronologically and discussing their advantages and disadvantages.

  • IV. RECENT ADVANCES IN DEEP LEARNING APPLIED TO SAR: The review covers terrain classification, object detection, parameter inversion, despeckling, InSAR, and SAR-optical data fusion.SAR image formation is not explicitly treated.

A. Terrain Surface Classification

Deep learning has advanced SAR and PolSAR terrain classification by learning features beyond hand-designed representations, while newer methods address speckle, spatial structure, complex values, limited labels, and architecture selection.

  • Deep learning automatically learns discriminative terrain-classification features, reducing reliance on problem-specific human-designed features.
  • DCAE combines hand-crafted texture and scale-transformation layers with a trained SAE to extract features while reducing speckle.
  • DSCNN adds HOG descriptors, supervised feature-label penalties, and contractive restrictions to capture relevant information and enhance local invariance.
  • Dual-branch CNNs learn polarization and spatial features through separate channels for PolSAR classification.
  • Complex-valued CNNs directly process complex PolSAR values using complex inputs, weights, operations, and backpropagation.
  • The field has expanded from autoencoder-based representation learning toward methods addressing speckle, spatial structure, complex data, semi-supervision, metric learning, and AutoML.

B. Object Detection

Deep learning performs well across SAR object-detection tasks, including military vehicles, buildings, and ships, but progress is constrained by SAR-specific imaging factors and inconsistent datasets.

  • 99% test accuracy was achieved by the 5-layer A-ConvNets on the MSTAR military-vehicle dataset.
  • Early ship-detection methods used fixed-size patches, which were not robust to variations in ship geometry such as size and shape.
  • 93% accuracy was reported for VHR SAR building segmentation using an FCN followed by CRF-RNN.
  • Individual building segmentation achieved an F1 score of 75.08%, with reconstructed LoD 1 models reaching a mean height error of 2.39 m.
  • SAR object detection has achieved strong results across military vehicle, ship, and building-detection applications using CNN and related architectures.
  • The main challenges are modeling imaging geometry, object size, and speckle noise, alongside the lack of standardized high-quality datasets.

C. Parameter Inversion

Deep learning parameter inversion addresses difficult SAR regression problems, including ice concentration and rough-surface parameter estimation, but remains under-exploited because physical models and domain knowledge are often needed.

  • Ice concentration estimation is challenging because complex interactions between SAR signals and sea ice hinder empirical algorithms.
  • CNNs generated ice concentration maps directly from dual-polarized SAR image patches, using expert-interpreted labels for RadarSat-2 melt-season data.
  • A five-pair convolutional and max-pooling CNN inverted rough-surface parameters from simulated SAR data with reasonable accuracy and efficient training.
  • Parameter estimation remains under-exploited because regression tasks often require appropriate physical models and domain knowledge for SAR data.

D. Despeckling

SAR despeckling uses deep networks to suppress multiplicative speckle while preserving image structure, but training data, over-smoothing, and realistic validation remain important constraints.

  • Speckle makes SAR image processing and interpretation difficult, making despeckling important before downstream applications.
  • Residual CNNs use logarithmic transformations to convert multiplicative speckle into an additive residual-learning problem, with skip connections supporting reconstruction.
  • SAR-DRN combines dilated convolutions, skip connections, and residual learning without requiring a prior noise description model.
  • PDF-based and distribution-aware losses incorporate SAR intensity statistics, gradients, strong scatterers, or Kullback–Leibler divergence into despeckling objectives.
  • Time-series despeckling can achieve comparative performance with state-of-the-art methods, but over-smoothing remains a critical issue.
  • CNN-NLM combines CNN-based similarity weights with nonlocal processing, while other methods adapt encoder–decoder and residual architectures to preserve features.
  • Noise2noise despeckling uses two independently and identically distributed noisy images, while multitemporal inputs require compensation for temporal change.

E. InSAR

Deep learning has been applied across InSAR reconstruction and deformation detection, as well as SAR–optical matching and fusion, but InSAR remains at an early stage with limited data and underused complex information.

  • InSAR derives topographic information by measuring interferometric phase, unwrapping interferograms, and converting absolute phase to height.
  • CNN encoder–decoder models estimate full-resolution DEMs from single-pass radar acquisitions and may address layover using contextual features.
  • CNNs detect subsidence and volcanic deformation from interferograms or deformation maps, including workflows trained on simulated data.
  • Unfolded and real-valued deep networks have been used for tomographic processing and MIMO SAR 3-D imaging, with reported speed or super-resolution advantages.
  • InSAR deep learning remains at an early stage because complex interferogram structure and ground-truth scarcity are insufficiently addressed.
  • SAR-Optical Data fusion: SAR–optical matching networks use separate modality branches, similarity layers, or generated SAR-like images to improve correspondence estimation.
  • SAR-Optical Data fusion: SAR–optical fusion is constrained by scarce high-quality training data and extreme differences in sensor viewing geometries, motivating auxiliary 3D data.

V. EXISTING BENCHMARK DATASETS AND THEIR

The paper emphasizes that deep learning requires large, informative benchmarks, while SAR datasets remain scarce and difficult to annotate because of SAR’s complex properties.

  • Large datasets are indispensable for training and evaluating deep learning models because learned knowledge is bounded by training-data information.
  • SAR images are harder to annotate than RGB images because of their complex properties, and purpose-built large SAR datasets are nearly non-existent.
  • SAR deep learning benchmarks cover image classification, scene classification, semantic segmentation, object detection, and registration or matching.

A. Image/Scene Classification •

The reviewed open SAR datasets support image and scene classification across urban, land-cover, and multimodal settings, with substantial variation in scale, annotation, and sensor coverage.

  • So2Sat LCZ42: So2Sat LCZ42 contains 400,673 dual-pol Sentinel-1 and multispectral Sentinel-2 patch pairs from 42 urban agglomerations and 10 smaller areas, labeled into 17 LCZ classes.It also provides geocoded SLC data, Lee-filtered variants, and expert-based label-uncertainty measurements.
  • OpenSARUrban: The six illustrated OpenSARUrban classes range from dense and low-rise residential buildings to vegetation, including high-rise buildings, villas, and industrial storage areas.The figure orders these classes from top to bottom.
  • OpenSARUrban: OpenSARUrban contains 33,358 Sentinel-1 dual-pol patches from 21 Chinese cities, with 10 urban-scene classes at the finest hierarchical level.Each patch is 100 by 100 pixels with 10 m spacing.
  • Multimodal datasets: SEN12MS provides 180,662 triplets of dual-pol Sentinel-1, multispectral Sentinel-2, and MODIS land-cover patches, each 256 by 256 pixels at 10 m ground sampling distance.The dataset is intended for scene classification, semantic segmentation, and land-cover mapping.
  • MSAW: MSAW uses 0.5 m quad-polarized X-band SAR imagery over 120 km2 of Rotterdam and labels 48,000 building footprints with building heights.The imagery was used for the SpaceNet 6 Challenge.

C. Object Detection •

Open SAR object-detection benchmarks cover vehicles, ships, and urban scenes, but differ markedly in sample volume, sensor configuration, resolution, and realism.

  • Vehicle recognition: MSTAR contains 17,658 X-band SAR chips covering 10 vehicle classes plus a geometric-target class, alongside 100 clutter images.Its centered vehicles and homogeneous clutter make the scenario relatively idealized and unrealistic despite its value for target recognition.
  • Ship detection: OpenSARShip 2.0 contains 34,528 Sentinel-1 ship patches with AIS information and manually verified ship length, width, direction, and type.About one-third of patches come from GRD products and two-thirds from SLC products; it is suitable for object detection.
  • Ship detection: SAR-Ship-Dataset contains 43,819 256-by-256 ship chips from Gaofen-3 and Sentinel-1 imagery, with distinct scales and backgrounds for multi-scale detection.The dataset combines 102 Gaofen-3 and 108 Sentinel-1 images.
  • Ship detection: FUSAR-Ship provides over 5,000 Gaofen-3 ship chips matched to AIS-derived ship information that can trace each chip to a unique vessel.Its space-time matching links imagery with vessel metadata.
  • Ship detection: AIR-SARShip-1.0/2.0 includes 31 or 300 Gaofen-3 SAR images at 1 m and 3 m resolution across spotlight and stripmap modes and varied maritime scenes.Object categories include ships, tankers, and fishing boats, with scenes spanning ports, islands, reefs, and sea surfaces.
  • SAR-optical resources: SEN1-2 and SARptical provide large paired SAR-optical resources, with 282,384 Sentinel-1/Sentinel-2 pairs and 10,108 precisely coregistered VHR pairs, respectively.SEN1-2 supports global, all-season multimodal research, while SARptical targets dense urban interpretation.

VI. CONCLUSION AND FUTURE TRENDS

The paper concludes that deep learning in SAR remains under-exploited because benchmarks are limited and models insufficiently tailored to SAR signal characteristics. It recommends larger datasets, phase-aware and uncertainty-aware methods, scalable optimization, cognitive sensing, and closer interdisciplinary collaboration.

  • Conclusion: Deep learning in SAR is limited primarily by the lack of large representative benchmarks and by models that do not fully account for SAR signal characteristics.The paper frames these as the two main barriers to full exploitation.
  • Future directions: Next-generation spaceborne SAR will combine high resolution with global coverage, motivating new analytics for dynamic-Earth monitoring and geo-parameter retrieval.The paper identifies deep learning as among the most promising methods for this big-SAR-data era.
  • Large and representative benchmark datasets: SAR target detection often relies on MSTAR, which provides only several thousand target samples overall and several hundred per class for training.The paper uses this example to motivate larger and more representative benchmarks.
  • Unsupervised deep learning: Unsupervised learning is proposed to address annotated-data deficiencies through feature learning, representation learning, or clustering.Suggested approaches include autoencoders, variational autoencoders, and deep embedded clustering.
  • Interferometric data processing: Many methods transform SAR into RGB-like data or use intensities only, leaving phase information insufficiently considered.The paper recommends complex-domain CNNs that preserve phase and learn directly from raw data.
  • Quantification of uncertainties: Geo-parameter estimates without uncertainty measures are considered invalid, yet deep models generally fail to quantify predictive uncertainty.The paper highlights Bayesian deep learning for estimating aleatoric and epistemic uncertainty.
  • Large-scale nonlinear optimization problems: Fast solvers are needed for large-scale inversion involving non-convex, nonlinear, and complex-valued optimization problems.Examples include compressive-sensing tomographic inversion and low-rank complex tensor decomposition for InSAR time series.
  • Cognitive sensors: Cognitive SARs could adapt imaging modes through feedback between intelligent receiver processing and transmit waveforms, but remain conceptual and experimental.The proposed sensing loop includes learning from environmental interaction and preservation of radar-return information.
Loading 2006.10027v2…