PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep learning in remote sensing: a review

Xiao Xiang Zhu, Devis Tuia, Lichao Mou, Gui-Song Xia, Liangpei Zhang, Feng Xu, Friedrich Fraundorfer

arXiv:1710.03959v1cs.CVeess.IV

TL;DR

Remote sensing raises distinctive challenges for deep learning, including heterogeneous data, physical quantities, and the tension between data-driven models and expert knowledge. This paper reviews the field’s advances, surveys its challenges, and provides resources for applying deep learning while advocating integration with physical models and remote sensing expertise. The review concludes that deep learning is a powerful toolbox for moving beyond traditional small-scale benchmarks, although current results cannot yet be generalized across remote sensing fields.

  • Problem

    Remote sensing needs deep learning approaches that address heterogeneous data, physical and expert knowledge, and limited generalization across remote sensing fields.

  • Method

    The paper reviews deep learning advances in remote sensing and examines approaches that combine learned representations with physics-based models and expert knowledge.

  • Results

    Deep learning networks tend to improve over classical approaches using hand-crafted features and can help researchers move beyond traditional small-scale benchmarking tasks.

  • Takeaways & Limitations

    Remote sensing researchers should treat deep learning as a toolbox while bringing domain expertise and physical models into its use.

  • Takeaways & Limitations

    Deep learning results in remote sensing cannot yet be generalized across fields, given varying conditions, intra-class variability, culture-dependent features, and limited training samples.

Abstract

from arXiv · show

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.

I. MOTIVATION

Deep learning is expanding into remote sensing as data and computational resources increase, but remote sensing introduces distinctive challenges involving heterogeneous, geo-located, temporal, large-scale, and physically grounded data. The paper reviews emerging advances and provides tools for researchers to apply deep learning.

  • Deep learning foundations: Deep learning has become a widely used model family, with CNNs effective for image recognition, object detection, and semantic segmentation, and RNNs successful for sequential analysis.CNNs learn abstract features through convolutional and pooling layers, while RNNs address sequential data tasks such as action recognition and image captioning.
  • Remote sensing challenges: Remote sensing combines complementary modalities such as optical and SAR data, requiring architectures for cross-modal image matching, fusion, and network transferability.The modalities differ in imaging geometry and content, while joint information extraction may require matching images from different perspectives without an existing 3D model.
  • Remote sensing challenges: Geo-location enables fusion with GIS layers, geo-tagged imagery, and other sensors, while supporting applications including picture localization, location-based services, and reality augmentation.Each pixel corresponds to a spatial coordinate, linking image information to other geographically referenced data.
  • Remote sensing challenges: Continuous acquisition shifts remote sensing from individual-image analysis toward time-series processing that jointly exploits temporal, spatial, and spectral information.The required architectures must use these dimensions together rather than treating images independently.
  • Remote sensing challenges: Global remote sensing creates a big-data problem: algorithms must process growing data volumes quickly and transferably, although metadata and annotations can support semi-automatic dataset generation.Sentinel satellites had acquired about 25 PetaBytes of data, and Copernicus calls for applications across the whole Earth surface.
  • Remote sensing challenges: Because remote sensing retrieves geo-physical and bio-chemical quantities and has process models and expert knowledge, fully expert-free deep learning should be questioned and physical models reintroduced.The paper emphasizes bringing remote sensing expertise into deep learning rather than relying exclusively on data-learned representations.
  • Paper scope and resources: The paper reviews deep learning advances across hyperspectral, SAR, very-high-resolution, data-fusion, and 3D-reconstruction applications and supplies open codes and data repositories.These resources are presented as practical tools for scientists beginning deep learning research in remote sensing.

A. Autoencoder models

The section introduces autoencoders, sparse autoencoders, RBMs, DBNs, and CNN architectures as deep-learning models for learning representations. These models encode inputs into learned features, impose reconstruction or sparsity constraints, or learn hierarchical convolutional representations.

  • 1) Autoencoder and Stacked Autoencoder (SAE):: An autoencoder maps input x to a latent representation h through a nonlinear function and reconstructs it as y through a reverse mapping.The encoding uses weights and biases, while the decoder commonly constrains its weights to the transpose of the encoder weights.
  • 1) Autoencoder and Stacked Autoencoder (SAE):: The reconstruction error is defined as the Euclidean distance between the input x and reconstructed input y, with optimization performed using stochastic gradient descent.The parameters are optimized so the reconstruction approaches the input data.
  • 1) Autoencoder and Stacked Autoencoder (SAE):: A stacked autoencoder wires each layer’s output to the inputs of the following layer, forming a multilayer network of autoencoders.
  • 2) Sparse Autoencoder:: When the latent dimension M is smaller than the input dimension D, an autoencoder learns a low-dimensional compressed representation; when M > D, sparsity can constrain hidden units instead.
  • 2) Sparse Autoencoder:: Sparse autoencoder training minimizes reconstruction error plus a KL-divergence sparsity penalty, whose importance is controlled by λ and whose target activation is ρ.The average activation ρ̂_j measures hidden neuron h_j activation over training samples.
  • 3) Restricted Boltzmann Machine (RBM) & Deep Belief Network (DBN):: An RBM is a stochastic undirected model with visible and hidden layers, symmetric interlayer connections, and no connections within either layer.Its conditional probability distribution is computationally feasible when either x or h is fixed.
  • 3) Restricted Boltzmann Machine (RBM) & Deep Belief Network (DBN):: A single RBM has limited feature-representation ability, whereas stacking RBMs into a DBN enables greater representational power through greedy layerwise training.
  • B. Convolutional neural networks (CNNs).: CNNs learn filters that perform convolutions in the image domain, and AlexNet scaled LeNet into a deeper, larger network for more numerous and complicated objects.

2) VGG Net:

VGG and related deep architectures emphasize deeper hierarchical feature learning, while FCN adapts convolutional networks to dense pixel-level prediction. Remote-sensing applications increasingly use these architectures across hyperspectral and satellite-image analysis.

  • VGG Net: VGG uses small 3 × 3 filters, consistent feature-map sizes within convolutional stages, increasing feature sizes in deeper layers, and scale jittering.These design choices reinforce deeper hierarchical feature representations.
  • Residual Networks: ResNet addresses degradation by adding shortcut connections that bypass stacked convolutional layers through identity mappings.The authors report that optimizing residual mappings is easier than optimizing the original mappings in conventional CNNs.
  • FCN: FCN performs semantic segmentation by producing an output with the same pixel dimensions as its input.Its encoder/decoder structure learns upsampling end to end, while skip connections fuse information from different depths.
  • Remote-sensing applications: Deep learning research in remote sensing increased exponentially after 2014, spanning hyperspectral analysis, SAR interpretation, high-resolution imagery, multimodal fusion, and 3D reconstruction.The review describes the surge as a rapid increase in interest.
  • Hyperspectral image analysis: 3D CNNs perform spatial-spectral convolutions and can model spectral information better than 1D and 2D CNNs.The review attributes this advantage to the third-dimensional convolution operations.

2) Anomaly Detection:

Deep learning has been applied to hyperspectral anomaly detection and several SAR interpretation tasks, but evidence and application maturity differ across tasks. Limited training data and preprocessing choices remain important constraints.

  • Anomaly Detection: Hyperspectral anomaly detection commonly models background statistics because anomalous spectra are difficult to specify in advance.One deep-learning framework trains a multilayer CNN on gray-value differences between neighboring pixel pairs.
  • Anomaly Detection: Deep learning has been widely applied to hyperspectral classification, whereas change and anomaly detection have received comparatively limited attention.The review identifies nonlinear spectral unmixing, enhancement, and time-series analysis as further problems to explore.
  • SAR applications: SAR deep-learning studies primarily address automatic target recognition, terrain surface classification, and parameter inversion.The review presents these as the main application areas in SAR image analysis.
  • Automatic Target Recognition: SAR automatic target recognition is constrained by scarce training samples, which can cause severe overfitting.Data augmentation is reported as necessary and the most critical step for CNN-based SAR ATR.
  • Automatic Target Recognition: AConvNets achieved 99.1% accuracy on the MSTAR dataset under standard operating conditions and were robust under several extended operating conditions.The approach removes fully connected layers and uses data augmentation to counteract overfitting.

2) Terrain surface classification:

Terrain-surface classification studies use deep models to extract spatial and polarimetric features from SAR and PolSAR data. Approaches range from autoencoders and belief networks to CNNs designed for complex-valued inputs.

  • Terrain surface classification: Deep learning automatically extracts features representing both spatial patterns and polarimetric characteristics in high-resolution SAR images.This addresses information that conventional approaches based mainly on pixel-wise target-decomposition parameters may not fully consider.
  • Terrain surface classification: SAR terrain-classification methods include stacked autoencoders, deep convolutional autoencoders, supervised contractive networks, DBNs, and CNNs.Several methods combine learned representations with handcrafted filters, superpixels, clustering, or Markov Random Fields.
  • Terrain surface classification: One reported finding is that speckle reduction produced worse performance in PolSAR classification.The authors suspect that speckle reduction may smooth out useful information.
  • Terrain surface classification: Complex-valued CNNs process PolSAR complex values with complex weights and operations throughout the network.A complex-valued backpropagation algorithm is developed for training.

3) Parameter inversion:

Deep learning methods support remote-sensing parameter inversion, scene classification, and object detection. Across these applications, pretrained or fine-tuned CNN features are widely used, while limited labeled datasets constrain training from scratch.

  • Parameter inversion: CNN regression for ice-concentration estimation from dual-polarization RadarSat-2 data produced more detailed results than operational products.The model was trained with visually interpreted expert labels and mean squared error for continuous-value regression.
  • Scene classification: Deep architectures achieved state-of-the-art performance in high-resolution satellite-image scene classification.The review attributes this to learning increasingly abstract and discriminative semantic features as depth grows.
  • Scene classification: Fine-tuning pretrained CNNs on satellite-image datasets produced better results than directly using pretrained CNN features alone.The review explains that fine-tuning makes features more oriented toward satellite imagery.
  • Limitations: Large CNNs require substantial computational power and very large labeled training sets.Small satellite datasets limit full-network fine-tuning, while smaller networks may become dataset-specific and require retraining.
  • Scene classification: Training large CNNs from scratch on small satellite-image datasets reduced classification accuracy relative to pretrained or fine-tuned networks.The review links this outcome to overfitting and local-minimum problems caused by millions of trainable parameters.
  • Object detection: Object-detection studies use CNN features for vehicles, ships, seals, aircraft, oil tanks, and other targets in high-resolution satellite or aerial imagery.Methods address scale and orientation variation through multiscale features, transferred networks, fine-tuning, and sliding-window detection.

3) Image Retrieval:

Deep learning image retrieval replaces limited low-level representations with learned deep features, achieving strong performance while leaving user-feedback integration and broader study underdeveloped.

  • Image retrieval compares image similarity against a query using feature representations, making feature descriptiveness central to retrieval performance.
  • Low-level handcrafted features remain effective for aerial retrieval, but their limited discriminative ability makes substantial gains difficult.
  • Deep features extracted from pretrained CNN fully connected layers outperform low-level features regardless of the retrieval system.
  • A CNN followed by a three-layer perceptron achieves remarkable performance with low-dimensional deep features after training on a large remote sensing dataset.
  • Sketch-based retrieval learns deep cross-domain features that enable satellite-image retrieval from hand-free sketches.
  • Current work on deep-learning retrieval remains limited, with user-feedback integration identified as an open issue.

1) Pansharpening and Super-Resolution:

Deep learning is applied to pansharpening, super-resolution, dense prediction, and multimodal fusion, ranging from operation replacement to end-to-end learned transformations.

  • Pansharpening and Super-Resolution: Pansharpening improves multispectral spatial resolution by fusing multispectral data with sharper spatial information, extending beyond panchromatic data to thermal and hyperspectral imagery.
  • Pansharpening and Super-Resolution: Deep neural networks are attractive for pansharpening because satellite imagery commonly includes panchromatic bands and training data are abundant.
  • Pansharpening and Super-Resolution: Early CNN pansharpening replaces a conventional upsampling operation, whereas later models learn the complete pansharpening transform end-to-end.
  • Pansharpening and Super-Resolution: Transfer learning fine-tunes a natural-image super-resolution CNN on hyperspectral imagery, injecting task-specific knowledge into pretrained features.
  • Dense Prediction and Fusion: Remote sensing fusion combines multiple sources by stacking them into a single data cube whose learned first-layer filters depend on the source stack.
  • Dense Prediction and Fusion: Dense prediction architectures use fully convolutional or deconvolutional layers to produce spatially resolved outputs rather than only central-pixel labels.
  • Dense Prediction and Fusion: On ISPRS Vaihingen and Potsdam benchmarks, compared architectures stack CIR data with a normalized digital elevation model and report results in Figures 11 and 12.
  • Dense Prediction and Fusion: Learned fusion of land-cover prediction maps outperforms simple averaging of posterior probabilities.

3) Fusing Heterogeneous Sources:

Fusing heterogeneous aerial, ground, and street-level imagery supports joint detection, fine-grained classification, localization, and related geospatial tasks.

  • Fusing Heterogeneous Sources: Multimodal remote sensing extends beyond restrictive single-view settings by combining overhead imagery with ground-based and aerial photographs.
  • Fusing Heterogeneous Sources: Ground and aerial imagery can provide better object detection, localization, or virtual-environment reconstruction.
  • Fusing Heterogeneous Sources: Urban-tree systems detect trees across aerial views and Google Street View panoramas, jointly reducing duplicate and illogical detections.
  • Fusing Heterogeneous Sources: A Pasadena tree inventory validates the detection model, while a GoogleNet-based CNN performs fine-grained tree-species classification.
  • Fusing Heterogeneous Sources: CNN–MRF models perform joint inference over monocular aerial imagery and ground stereo-camera images for fine-grained urban categories.
  • Fusing Heterogeneous Sources: Cross-view localization matches ground images with aerial views using a cross-view Siamese network, supporting localization and related change-detection applications.

E. 3D Reconstruction

Deep learning can improve parts of remote-sensing 3D reconstruction, whose conventional workflow computes camera orientation and triangulates matched image points.

  • 3D data such as DSMs and DTMs form a basic layer for subsequent remote-sensing processing and analysis.
  • A typical 3D reconstruction workflow first estimates camera positions and orientations from matched tie-points through camera resectioning.
  • The workflow then triangulates 3D measurements for point correspondences established through stereo matching.
  • The fundamental reconstruction algorithms are geometrical, and machine learning has historically played little role in this pipeline.
  • Recent machine-learning techniques could significantly improve particular steps in the 3D reconstruction pipeline.

1) Tie points identification and matching:

Deep learning replaces hand-engineered image-matching components with learned descriptors, similarity measures, and eventually end-to-end stereo processing. CNN-based methods outperform established engineered and benchmark methods in tie-point matching and stereo estimation.

  • Tie-point identification and matching: Traditional tie-point workflows detect repeatable image locations and compute patch similarities using engineered features such as SIFT and SURF.These methods remain less accurate than humans, motivating learned alternatives.
  • Tie-point identification and matching: CNNs learn image-patch descriptors from training examples and compute descriptors for detected tie-points.The described network uses five convolutional layers and two fully connected layers.
  • Tie-point identification and matching: Learned CNN descriptors significantly outperform engineered descriptors such as SIFT on standard tie-point matching datasets.The result is reported for experiments on standard datasets.
  • Stereo processing using convolutional neural networks: CNNs can replace the similarity-measure stage of stereo processing by learning patch correspondences instead of using NCC or the Census transform.The learned similarity measure produced a significantly lower error rate than SGM.
  • Stereo processing using convolutional neural networks: 8% error for MC-CNN versus 18.4% for SGM, while the best-ranked CNN-based stereo method reached 5.9%.The ranking is from the Middlebury stereo evaluation benchmark as of May 2017, where CNN-based methods led the board.
  • Stereo processing using convolutional neural networks: End-to-end CNN stereo models can take a stereo image pair as input and directly output a disparity map.This design replaces the individual engineered algorithmic steps with a single encoder-decoder CNN trained on image-depth pairs.

3) Large scale semantic 3D city reconstruction:

Semantic information from CNN classification can be integrated into 3D reconstruction, producing large-scale city models whose 3D points also carry object-class labels. The review also identifies resources intended to make deep learning in remote sensing easier to start with.

  • Large scale semantic 3D city reconstruction: Traditional 3D generation used geometric constraints and treated image data as intensity values without semantic meaning.CNN-based classification makes class information available for the 3D generation process.
  • Large scale semantic 3D city reconstruction: Joint reconstruction assigns occupancy and semantic class labels to voxels, integrating image-derived classes such as vegetation, building, road, and sky.The volumetric method partitions the scene into voxels and jointly estimates occupancy and class labels.
  • Large scale semantic 3D city reconstruction: Class labels influence voxel occupancy and labeling probabilities, making the resulting 3D reconstruction semantically interpretable.Building pixels favor occupied voxels, sky pixels favor unoccupied voxels, and stacked voxels increase the probability of a building label.
  • Large scale semantic 3D city reconstruction: Semantic 3D reconstruction has been successfully applied to aerial imagery to create large-scale city models containing both 3D data and class labels.Building structures receive building labels, and roof structures can also be labeled as roof.
  • Large scale semantic 3D city reconstruction: CNN stereo processing significantly boosted the accuracy and precision of depth estimation, while CNN classifiers enabled semantic 3D reconstruction.The review describes semantic 3D reconstruction as a research area expected to grow significantly.
  • Resources for starting deep learning: The review provides tutorials, open-source frameworks, open remote-sensing datasets, and showcasing code examples to support researchers beginning deep learning.The resources cover tutorials, frameworks, training data, and examples implemented with different libraries.

C. Remote Sensing Data for Training Deep Learning Models

Training deep learning models requires large datasets or, for large architectures, preferably pretrained methods. The review catalogs remote-sensing datasets spanning aerial imagery, semantic labeling, multimodal urban data, and SAR-optical matching, alongside reproducible code examples.

  • Dataset requirements: Large datasets are needed for good generalization when fine-tuning models or training small networks from scratch, while large architectures preferably use pretrained methods.The review notes that public datasets can support training deep neural networks.
  • Aerial image classification datasets: UC Merced provides 256×256 RGB aerial images across 21 land-use classes, with 100 images per class and one label per image.Its single image-level label limits the dataset to whole-image classification.
  • Aerial image classification datasets: AID contains 10,000 annotated aerial images across 30 land-use scene classes and covers more images and categories than UC Merced.The review presents it as aligned with modern deep-learning data requirements.
  • Multimodal datasets: SARptical contains 10,000 precisely co-registered SAR-optical image-patch pairs from dense urban Berlin.The pairs are generated using co-registered 3D InSAR point clouds and support SAR-optical image matching.
  • Showcasing code: The review presents reproducible examples spanning MatConvNet, Caffe, and TensorFlow/Keras, including classification, SAR target recognition, and hyperspectral feature learning.Examples include pretrained models, training and testing code, and a successfully trained CNN model.

V. CONCLUSION AND FUTURE TRENDS

The paper reviews deep learning in remote sensing and identifies technical and application trends for large-scale, data-driven Earth observation. It emphasizes domain expertise, tailored models, shared resources, and continued work on challenges including limited data, benchmarking, transferability, and multimodal imagery.

  • The paper reviews the current state of the art in deep learning for remote sensing.
  • Future research must address limited annotated data, transferability, and benchmarking for fast-growing deep-learning algorithms.The paper raises weakly supervised, semi-supervised, and unsupervised learning, global geo-information extraction, and shared benchmark datasets as open directions.
  • Global-scale use remains challenging because of complex light scattering, atmospheric conditions, intra-class variability, culture-dependent features, and limited training samples.The paper identifies never-ending learning and self-taught approaches as possible responses to large-scale application needs.
  • Deep learning can support remote sensing applications beyond small-scale benchmarks, including large-scale real-life problems and global-change monitoring.The authors describe deep learning as an implicit model that can generalize well and support applications such as monitoring global changes.
  • Remote sensing models must address data that are multimodal, geo-located, multi-aspect, and multi-temporal.The paper calls for models tailored to these properties of big remote sensing data.
  • The authors advocate combining remote sensing expertise with deep learning, incorporating tailored prior knowledge, and sharing data and architectures.They also identify fusion of physics-based modeling and deep neural networks as a promising direction because remote sensing imagery is produced by physical processes.
Loading 1710.03959v1…