PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep learning in remote sensing: a review

Xiao Xiang Zhu, Devis Tuia, Lichao Mou, Gui-Song Xia, Liangpei Zhang, Feng Xu, Friedrich Fraundorfer

arXiv:1710.03959v1cs.CVeess.IV

TL;DR

As science shifts toward data-driven approaches, remote-sensing experts must consider how to position their expertise among data scientists. This review examines deep learning challenges, surveys advances, and provides introductory resources; it reports that deep networks tend to improve on classical approaches using hand-crafted features.

  • Problem

    The shift toward data-driven science raises how remote-sensing experts should position themselves among data scientists.

    Source: §V · p. 41

  • Method

    The paper reviews the state of the art in remote-sensing deep learning, analyzes its challenges, and provides resources for getting started.

    Source: §V · p. 40

  • Results

    Deep networks tend to improve over classical approaches using hand-crafted features.

    Source: §V · p. 40

  • Takeaways & Limitations

    The authors present deep learning as a toolbox for researchers to move beyond traditional small-scale benchmarking tasks.

    Source: §V · p. 41

    The authors caution that results cannot be generalized, particularly amid varying conditions, intra-class variability, culture-dependent features, and limited training samples.

    Source: §V · p. 40

Abstract

from arXiv · show

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.

I. MOTIVATION · II. FROM PERCEPTRON TO DEEP LEARNING

Deep learning’s success has helped drive its adoption in remote sensing, where multimodal, geolocated, temporal, and large-scale data and scientific retrieval needs pose distinct challenges. The paper reviews advances and resources while arguing for remote sensing expertise and physical models alongside deep learning.

I. MOTIVATION

  • CNN representations are effective for image recognition, object detection, and semantic segmentation, while RNNs succeed on sequential tasks such as action recognition and image captioning.CNNs extract mid- and high-level abstract features from raw images; RNNs are presented as a successful branch of deep learning for sequential data.Source: §I · p. 2
  • Optical and SAR data differ in imaging geometry and content, motivating architectures to match images across perspectives and modalities without requiring an existing 3D model, and investigation of network transfer across modalities.Source: §I · pp. 2–3
  • Because remote sensing pixels have geographic coordinates, their information can be fused with GIS layers, geo-tagged social-media images, or other sensors, enabling applications such as picture localization and location-based services.Source: §I · p. 3
  • Continuous acquisition is shifting remote sensing from individual-image analysis to time-series processing, requiring networks that jointly exploit temporal, spatial, and spectral information.The paper notes the Copernicus program’s guarantee of continuous acquisition for decades.Source: §I · p. 3
  • Global, growing data volumes demand fast, transferable algorithms, while abundant annotations and metadata can enable semi-automatic generation of large training datasets.Source: §I · p. 3
  • Remote sensing often retrieves physical or biochemical quantities using process models and expert priors, so the authors question expert-free automation and call to reintroduce physical models.Examples include mass movement rates, soil mineral composition, water constituents, atmospheric trace gases, and terrain elevation or biomass.Source: §I · pp. 3–4
  • The paper reviews remote-sensing deep-learning advances and provides tools for researchers, including open codes and data repositories.The reviewed application areas include hyperspectral imagery, SAR, very high resolution imagery, data fusion, and 3D reconstruction.Source: §I · p. 4

II. FROM PERCEPTRON TO DEEP LEARNING

  • The section introduces the perceptron as an early neural-network model, then recalls classic deep-learning architectures for visual data processing and their more sophisticated training methodologies.Source: §II · p. 4

A. Autoencoder models … 1) AlexNet:

The section introduces autoencoder-based and stochastic models for learned representations, then surveys CNN architectures and AlexNet’s benchmark success and design choices. It describes how these models encode or process image data and how AlexNet combined architectural and training techniques.

1) Autoencoder and Stacked Autoencoder (SAE):

2) Sparse Autoencoder:

  • Sparse autoencoders can learn overcomplete representations (M > D) by adding a KL-divergence sparsity penalty to reconstruction loss.Source: 2) Sparse Autoencoder · p. 5

3) Restricted Boltzmann Machine (RBM) & Deep Belief Network (DBN):

B. Convolutional neural networks (CNNs).

1) AlexNet:

  • AlexNet [2] achieved a 15.4% top-5 test error rate in 2012, the first year a CNN reached that result, and won the ILSVRC.Source: 1) AlexNet · p. 7
  • AlexNet used ReLU to reduce training time, dropout to limit overfitting, and data augmentation that translated and reflected training patches across horizontal and vertical axes.Source: 1) AlexNet · p. 7
  • GPU training helped AlexNet train faster, enabling the use of larger datasets and bigger images.Source: 1) AlexNet · p. 7
  • VGG Nets emphasized simplicity and depth and used 3 × 3 filters.Source: 1) AlexNet · p. 7, 2) VGG Net · p. 7

2) VGG Net: … 1) Hyperspectral Image Classification:

This section surveys deep-learning architectures and approaches for remote-sensing image analysis, outlining VGG, ResNet, and FCN before reviewing hyperspectral classification, including learned filters and an RNN that analyzes pixel sequences.

2) VGG Net:

  • VGG uses 3 × 3 filters, consistent feature-map sizes and filter counts within convolutional layers, deeper features that roughly double in size, and scale jittering during training.Training such a model from scratch requires substantial computational power and a very large labeled training set.Source: 2) VGG Net · p. 8

3) ResNet:

  • He et al.’s 152-layer ResNet won ILSVRC 2015 with a 3.6% error rate; identity shortcuts add convolution outputs to bypass layers and ease optimization.The authors identify degradation—higher training and test errors from simply adding layers to plain networks—as a problem that residual mappings address.Source: 3) ResNet · p. 8

4) FCN:

  • FCN produces pixel-level labels at input-image resolution through learned encoder-decoder upsampling, while skip connections fuse information from different depths for multiscale inference.Its fully convolutional design supports arbitrary-sized inputs by omitting a size-dependent final fully connected layer.Source: 4) FCN · p. 8

III. REMOTE SENSING MEETS DEEP LEARNING

  • Deep-learning papers on remote sensing have increased exponentially since 2014, and the review surveys hyperspectral analysis, SAR and high-resolution image interpretation, multimodal fusion, and 3D reconstruction.The paper frames these as remote-sensing applications achieved by deep learning.Source: §III · p. 9

A. Hyperspectral Image Analysis

  • Hyperspectral sensors’ hundreds of narrow bands enable pixel-material identification, and hyperspectral analysis is important for applications such as land-cover classification, change detection, and object detection.The review identifies hyperspectral analysis as an active remote-sensing research direction over the last decade.Source: A. Hyperspectral Image Analysis · pp. 9–10

1) Hyperspectral Image Classification:

  • Atmospheric scattering, complex light-scattering mechanisms, inter-class similarity, and intra-class variability make hyperspectral classification inherently nonlinear, motivating deep hierarchical features.The review says these abstract features are generally more robust to nonlinear inputs than features from shallow models.Source: 1) Hyperspectral Image Classification · p. 11
  • For crop-type classification, 2D CNNs outperformed 1D CNNs, but their maps smoothed and misclassified some small objects.The review also describes supervised CNNs using spectral-only 1D inputs and 2D CNNs combining spectral and spatial information.Source: 1) Hyperspectral Image Classification · p. 11
  • 3D CNNs perform spatial-spectral convolutions, unlike the spatial-only convolutions of 2D CNNs, and model spectral information better than 1D and 2D CNNs; introduced a supervised, ℓ2-regularized 3D CNN-based model.The passages also describe unsupervised convolutional and Conv-Deconv networks for spectral-spatial feature learning.Source: 1) Hyperspectral Image Classification · pp. 11–12
  • The authors report an extensive study of learned filters.They refer to Fig. 6.Source: 1) Hyperspectral Image Classification · p. 12
  • The RNN proposed in uses a new activation function and a modified gated recurrent unit for hyperspectral image classification.Source: 1) Hyperspectral Image Classification · p. 13
  • The model analyzes hyperspectral pixels as sequential data and uses network reasoning to determine information categories.The proposed RNN is shown in Fig. 7.Source: 1) Hyperspectral Image Classification · p. 13, Figure 7 · p. 13

2) Anomaly Detection: · B. Interpretation of SAR Images · 1) Automatic Target Recognition:

The review presents deep learning as nascent for hyperspectral anomaly and change detection but widely applied to SAR interpretation, especially automatic target recognition (ATR). Across these topics, it describes background modeling, CNN-based methods, and strategies for limited training data.

2) Anomaly Detection:

  • For hyperspectral anomaly detection, where prior knowledge of anomalous spectra is difficult to obtain, background modeling is common; Li et al. use a CNN trained on neighboring-pixel gray-value differences.At test time, the trained CNN detects anomalies by evaluating differences between neighboring pixel pairs.Source: 2) Anomaly Detection · p. 13
  • Deep learning is widely applied to multispectral classification, but change and anomaly detection have only made an initial mark, with further work needed on unmixing, enhancement, and time-series analysis.The review identifies nonlinear spectral unmixing, hyperspectral image enhancement, and hyperspectral time-series analysis as potential problems to explore.Source: 2) Anomaly Detection · p. 13

1) Automatic Target Recognition:

  • For SAR ATR, detection, discrimination, and classification form a staged process, and deep learning applies because each stage can be treated as classification.The stages perform increasingly complex processing and select candidate objects for the next stage.Source: 1) Automatic Target Recognition · p. 14
  • Limited SAR training samples can cause severe overfitting and impair generalization, while data augmentation is considered necessary and critical for CNN-based ATR.Reported strategies include translation, rotation, and interpolation.Source: 1) Automatic Target Recognition · p. 14
  • 99.1% accuracy on the MSTAR dataset under standard operating condition (SOC) was reported for AConvNets, which were also found robust in several extended operating conditions (EOCs).The approach removes fully connected layers from conventional CNNs.Source: 1) Automatic Target Recognition · p. 14
  • Other SAR ATR approaches include a DBN that extracts features with stacked RBMs and a CNN-feature/SVM classifier trained with augmentations that mimic imaging errors.Wagner’s augmentation includes elastic distortions and affine transformations to mimic errors such as changing range or incorrect alignment.Source: 1) Automatic Target Recognition · p. 14
  • SAR CNN studies also address ship-iceberg discrimination, ship discrimination, and harbor ship detection, with simulated training data used to address sample scarcity.The cited studies use TerraSAR-X StripMap imagery, highway networks, and simulation software, respectively.Source: 1) Automatic Target Recognition · p. 15
  • Song et al. use a generative deconvolutional network to map target labels into a feature space, then train a CNN to map SAR images there for interpreting previously unseen targets.The intended system interprets an unseen target in the context of known targets.Source: 1) Automatic Target Recognition · p. 15

2) Terrain surface classification:

In terrain surface classification, deep learning is used to extract spatial patterns alongside polarimetric characteristics, addressing patterns largely overlooked by conventional pixel-wise approaches.

2) Terrain surface classification:

3) Parameter inversion: … 2) Object Detection:

This section reviews deep-learning methods for parameter inversion, scene classification, and object detection, including approaches for detailed ice-concentration estimates and scene-classification results. For object detection, it covers efforts toward rotation invariance, robust and efficient ship detection, and reduced manual annotation using modified CNN features, wavelet-domain features with efficient classification, and image-level weak labels.

3) Parameter inversion:

  • CNNs estimated melt-season ice concentration from expert-labeled dual-pol RadarSat-2 SAR and produced more detailed results than operational products.The task was treated as continuous-value regression with mean squared error loss.Source: 3) Parameter inversion · p. 17

1) Scene Classification:

  • Deep architectures achieved state-of-the-art high-resolution satellite scene-classification performance [82] by learning increasingly abstract, discriminative semantic features.The review contrasts this with mid-level approaches and notes that varied spatial arrangements make scene classification challenging.Source: 1) Scene Classification · pp. 17–18
  • Pre-trained CNN features from intermediate layers yielded impressive scene-classification results [82], while fine-tuning GoogLeNet layers on UC-Merced improved results over direct use [82].Fine-tuning makes learned features more oriented to satellite imagery.Source: 1) Scene Classification · p. 18
  • Training small-scale networks on small satellite image datasets can cause overfitting and local minimum problems; they can orient to training images, and their generalization ability decreases.For each satellite dataset, the network needs to be retrained.Source: 1) Scene Classification · p. 19

2) Object Detection:

  • CNN vehicle detectors used implicit vehicle models with spatial and spectral features, multi-scale feature maps, and graph-based superpixels to extract candidate patches.The multi-scale approach addressed object-size variation, and the patch-based CNN predicted whether a patch contained a vehicle.Source: 2) Object Detection · p. 19
  • Object-detection work also transferred pre-trained CNNs and used weakly supervised coupled CNNs for aircraft detection, mining training examples from image-level labels to reduce human labor.The coupled model proposed regions and localized aircraft simultaneously for large, high-resolution satellite images.Source: 2) Object Detection · pp. 19, 21
  • Cheng et al. add a rotation-invariant layer to AlexNet and train it with a regularization constraint that aligns features of original and rotated samples.The constraint is intended to improve the model’s invariance to object rotation.Source: 2) Object Detection · p. 21
  • Tang et al. combine two wavelet-domain SDA models with ELM for compressed-domain ship detection, yielding robust features under changing conditions and accurate, fast detection.The SDA models extract hierarchical ship features; ELM performs efficient feature pooling and classification.Source: 2) Object Detection · p. 21
  • Han et al. use weakly supervised learning and a DBM, requiring only image-level labels that indicate an object’s presence and reducing manual training-data annotation.The labels specify whether an object is present in the whole image.Source: 2) Object Detection · p. 21

3) Image Retrieval: · D. Multimodal Data Fusion

Remote-sensing image retrieval depends on descriptive image representations, and reviewed CNN approaches improve retrieval or enable sketch-based search. Deep neural networks are presented as candidates for fusion of big, heterogeneous sensor data across three task areas.

3) Image Retrieval:

  • Retrieval systems compare image-feature representations, so retrieval performance depends strongly on how descriptively those features represent visual content.Source: 3) Image Retrieval · p. 21
  • BoVW and VLAD feature coding with low-level hand-crafted features has been effective for aerial retrieval, but their limited discriminative ability constrains further gains.Source: 3) Image Retrieval · p. 21
  • CNN-derived features outperform low-level features regardless of retrieval system, while a deep cross-domain method enables satellite-image retrieval from hand-drawn sketches [111].The sketch-based method is described as learning deep cross-domain features.Source: 3) Image Retrieval · pp. 21–22
  • A CNN followed by a three-layer perceptron, trained on a large remote-sensing dataset, achieves remarkable performance with low-dimensional deep features.Source: 3) Image Retrieval · p. 22
  • Deep retrieval research remains limited, despite the potential of learned high-level features; future systems are anticipated, and incorporating user feedback remains an open issue.Source: 3) Image Retrieval · p. 22

D. Multimodal Data Fusion

  • Growing sensor-data availability makes big, heterogeneous data more tangible for studying environmental processes, while complex relations favor high-capacity models such as deep neural networks.Source: D. Multimodal Data Fusion · p. 22
  • Deep-learning data-fusion work in remote-sensing image analysis is reviewed across pansharpening, feature- and decision-level fusion, and fusion of heterogeneous sources.Source: D. Multimodal Data Fusion · p. 22

1) Pansharpening and Super-Resolution: · 2) Feature and Decision-level Fusion for image classification:

This section reviews deep-learning approaches to pansharpening and super-resolution, and to feature, decision, and multi-source fusion for image classification and change detection, including recurrent detection and joint prediction of classification with other outputs. It also describes gains alongside computational costs and challenges related to unavailable depth data and imperfect image registration.

1) Pansharpening and Super-Resolution:

  • Pansharpening improves multispectral spatial resolution by fusing sharper spatial data, extending from panchromatic bands to thermal and hyperspectral imagery.It is a special instance of super-resolution.Source: 1) Pansharpening and Super-Resolution · p. 22
  • [120] uses a shallow CNN to replace an upsampling operation, whereas learns the entire pansharpening transform end-to-end.The approach then uses a traditional Gram-Schmidt transform.Source: 1) Pansharpening and Super-Resolution · p. 23
  • [122] fine-tunes a natural-image super-resolution CNN for hyperspectral imagery, while uses learned panchromatic relationships to upsample multispectral bands.The approaches transfer learned image features across domains and train autoencoders to predict full-resolution panchromatic imagery, respectively.Source: 1) Pansharpening and Super-Resolution · p. 23

2) Feature and Decision-level Fusion for image classification:

  • Feature-level fusion can stack sources as network inputs; also compares RGB-based CNN features with DSM-inclusive inputs for patch classification using an SVM.The classifier predicts one semantic class per patch and is applied with a sliding window.Source: 2) Feature and Decision-level Fusion… · pp. 23–24
  • Dense prediction incorporates spatial structure by predicting patch labels; fully convolutional and deconvolutional networks were compared on Vaihingen and Potsdam.The reported figures show architectures and Potsdam classification results, but these passages do not state a winning approach.Source: 2) Feature and Decision-level Fusion… · p. 24, Figure 12 · p. 25
  • Other spatial-feature methods stack spatial filters for SAR classification and use a conditional random field to reduce speckle effects.A related approach learns recombinations of filters from hyperspectral bands and DSMs.Source: 2) Feature and Decision-level Fusion… · p. 25
  • For change detection, learns a joint representation that makes changes easier to detect by image differencing, while learns a bottleneck representation and demonstrates versatility across datasets.The approach uses optical imagery; includes optical–SAR image pairs and change detection between optical and SAR.Source: 2) Feature and Decision-level Fusion… · p. 26
  • An end-to-end LSTM-based RNN learns joint spectral feature representations from bi-temporal image sequences for multi/hyperspectral change detection, detects multi-class changes, and shows good transferability.Source: 2) Feature and Decision-level Fusion… · p. 26
  • An end-to-end CNN ensemble that combines land-cover maps from different data sources and abstraction levels improves performance, but requires a complex, computationally demanding architecture.The architecture uses many skip connections and fusion layers.Source: 2) Feature and Decision-level Fusion… · p. 28
  • Learned residual fusion of pretrained model maps outperforms simple averaging of posterior probabilities.The fusion network learns coefficients that adjust the average and favor one map or the other.Source: 2) Feature and Decision-level Fusion… · p. 28
  • Jointly predicting object boundaries and land cover can sharpen classification maps, and the reviewed boundary-aware model produces accurate predictions at a cost of up to 800 million parameters [143].It predicts multiscale edge likelihoods from CIR and height data, then adds detected boundaries as input channels to classification networks.Source: 2) Feature and Decision-level Fusion… · p. 29
  • Joint image classification and DSM regression can estimate height maps from imagery when DSM inputs are unavailable.The model combines dense classification and DSM-prediction regression losses, and uses DSM as an output rather than an input.Source: 2) Feature and Decision-level Fusion… · p. 29
  • A conditional random field model jointly performs image registration, land-use classification, and change detection, and was applied to VHR image pairs and VHR images with video frames.The approach addresses the difficulty of perfect co-registration, particularly for very-high-resolution imagery.Source: 2) Feature and Decision-level Fusion… · p. 29

3) Fusing Heterogeneous Sources:

The section describes multimodal fusion beyond same-viewpoint imagery, focusing on combining ground-based and aerial data for object detection, localization, and virtual-environment reconstruction.

3) Fusing Heterogeneous Sources:

  • Fusing human-perspective ground images with overhead views can support object detection, localization, and virtual-environment reconstruction.The ground images include sources such as Google Street View and Flickr.Source: 3) Fusing Heterogeneous Sources · p. 30
  • For urban trees, uses Faster R-CNN detections from aerial and Street View imagery, jointly filters them with an energy function, and classifies species using GoogleNet.The joint energy function is designed to avoid multiple or illogical detections.Source: 3) Fusing Heterogeneous Sources · p. 30
  • A CNN–MRF approach jointly infers fine-grained categories from monocular aerial imagery and ground images captured by a car-mounted stereo camera.Reported categories include road, sidewalk, background, building, and parking.Source: 3) Fusing Heterogeneous Sources · p. 30
  • Cross-view localization matches ground photos to aerial views with Siamese networks, which also support change detection between matched panoramas and aerial images.This addresses locating ground photos that lack geolocation.Source: 3) Fusing Heterogeneous Sources · p. 30
  • CNN features characterize city-specific imagery and translate Charleston–San Francisco similarities into aerial-image differences; also uses feature similarity for localization.The city-specific comparisons run in both directions between Charleston and San Francisco.Source: 3) Fusing Heterogeneous Sources · p. 30

E. 3D Reconstruction

Generating 3D data from images is important in remote sensing because DSMs or DTMs provide a basic layer for further analysis. The typical workflow combines camera orientation and triangulation, a geometrical pipeline where machine learning has historically played a limited role but has recently improved some steps.

E. 3D Reconstruction

  • 3D data such as DSMs or DTMs is a basic layer for subsequent processing and analysis in remote sensing.Generating 3D data from image data plays an important role in remote sensing.Source: E. 3D Reconstruction · p. 31
  • 3D reconstruction typically requires camera orientation followed by triangulation of stereo-matched point correspondences.Camera orientation estimates the positions and orientations of the image-producing cameras using matched tie-points and camera resectioning; triangulation calculates 3D measurements.Source: E. 3D Reconstruction · p. 31
  • The pipeline’s fundamental algorithms are geometrical and analytical, so machine learning has historically played a limited role, although it has recently improved some steps.Source: E. 3D Reconstruction · p. 31

1) Tie points identification and matching: · 2) Stereo processing using convolutional neural networks:

The review describes CNNs as improving tie-point matching beyond engineered descriptors and leading stereo benchmarks, with approaches ranging from learned patch similarity to end-to-end disparity prediction.

1) Tie points identification and matching:

  • Engineered tie-point methods detect repeatable locations such as corners and compare image patches, but remained less accurate than humans.The review cites SIFT and SURF as examples of engineered features.Source: 1) Tie points identification and… · p. 31
  • CNN descriptors trained on image patches significantly outperformed engineered SIFT descriptors on standard tie-point matching datasets [159].The network used five convolutional layers and two fully connected layers to compute a descriptor for each patch.Source: 1) Tie points identification and… · p. 31
  • Yi et al. extended CNN tie-point matching by detecting point locations and producing a descriptor vector for each point.This approach combines location detection and descriptor output in a deep CNN.Source: 1) Tie points identification and… · p. 32

2) Stereo processing using convolutional neural networks:

  • Stereo processing searches for corresponding pixels, typically across candidate depth values, then optimizes matching costs and filters outliers.The described pipeline uses similarity measures, optimization strategies such as SGM or MRF inference, and heuristic filtering.Source: 2) Stereo processing using… · p. 32
  • 18.4% was SGM's error rate versus 8% for MC-CNN after a CNN learned patch similarity in place of NCC or Census.The review describes this as a significantly lower error rate than SGM.Source: 2) Stereo processing using… · p. 32
  • 5.9% was the error rate of the top-ranking stereo method, and CNN-based methods led the Middlebury board as of May 2017.The cited table identifies the benchmark's top-ranking methods; the passages do not provide its individual cell values.Source: 2) Stereo processing using… · pp. 32–33, Table I · p. 33
  • Mayer et al. modeled stereo processing solely as a CNN that takes a stereo pair and directly outputs a disparity map.End-to-end training on image–depth-map pairs lets data define the processing performed, rather than the architecture specifying the algorithm.Source: 2) Stereo processing using… · p. 33

3) Large scale semantic 3D city reconstruction: … B. Open-source Deep Learning Frameworks

CNN-derived semantics enable 3D reconstruction to use object information, including through joint voxel reconstruction and class labeling. The paper also introduces resources and framework guidance to help researchers start using deep learning in remote sensing.

3) Large scale semantic 3D city reconstruction:

  • CNN-derived class information lets 3D generation move beyond geometric constraints and use image semantics to guide reconstruction.Traditional generation treated images as intensity values without semantic meaning; class-specific parameters can steer the process.Source: 3) Large scale semantic 3D city… · p. 33
  • Haene et al. jointly reconstruct a voxel scene and assign class labels, using image labels and voxel arrangements to inform occupancy and class probabilities.The method partitions the area into cells and seeks an occupied/non-occupied voxel grid that fits the image data.Source: 3) Large scale semantic 3D city… · p. 34
  • CNNs used for stereo processing significantly boosted the accuracy and precision of depth estimation.Source: 3) Large scale semantic 3D city… · p. 34

IV. DEEP LEARNING IN REMOTE SENSING MADE RIDICULOUSLY SIMPLE TO START WITH

  • The paper assembles starting resources for remote-sensing deep learning, including tutorials, open data, model-training guidance, and framework-based code examples.Source: §IV · pp. 34–35

3) Large scale semantic 3D city reconstruction:

  • Blaha et al. produced large-scale semantic 3D city reconstructions with class labels for structures, including roof labels on buildings.The Enschede reconstruction illustrates classes including ground, building, roof, vegetation, and clutter.Source: 3) Large scale semantic 3D city… · p. 34, Figure 14 · p. 35

A. Tutorials

  • The tutorials subsection points readers to books, surveys, code tutorials, videos, and general recommendations for choosing deep-learning approaches [174].Source: A. Tutorials · p. 35

B. Open-source Deep Learning Frameworks

  • The frameworks subsection names Caffe, Torch, Theano, TensorFlow, and Microsoft-CNTK, and frames comparison of their differences, limitations, and investment value as a practical concern.Fig. 15 ranks popular open-source frameworks by developer stars; the paper points readers to a detailed discussion of framework choices.Source: B. Open-source Deep Learning Frameworks · pp. 35–36, Figure 15 · p. 36

C. Remote Sensing Data for Training Deep Learning Models … D. Showcasing

The section catalogs public remote-sensing datasets for training and evaluation, and showcases researcher-provided deep-learning examples that make it easier to get started with CNNs.

C. Remote Sensing Data for Training Deep Learning Models

1) Scene classification (one image is classified into a single label): -

2) Image classification (each pixel of an image is classified into a label): -

3) Registration / matching: -

  • SARptical addresses challenging co-registration of SAR and optical imagery in dense urban areas with 10,000 corresponding patch pairs whose centers are precisely aligned.Source: 3) Registration / matching: - · p. 38

D. Showcasing

  • Examples include a MatConvNet deconvolution network, a fully convolutional SegNet architecture in Caffe, a Caffe AConvNet for SAR target recognition demonstrated on MSTAR, and an unsupervised residual Conv-Deconv network.Source: D. Showcasing · pp. 38–39
  • The residual Conv-Deconv network learns unsupervised spectral-spatial hyperspectral features, which can be fine-tuned for classification or used for object detection with first-block filters.Source: D. Showcasing · p. 39

V. CONCLUSION AND FUTURE TRENDS

The review finds that deep networks tend to improve on classical approaches using hand-crafted features, while technical challenges remain. It advocates combining remote-sensing expertise with deep learning to tackle large-scale, real-life problems.

V. CONCLUSION AND FUTURE TRENDS

  • The review surveys deep learning in remote sensing and reports that deep networks tend to improve over classical approaches using hand-crafted features.Source: §V · p. 40
  • Deep learning research in remote sensing has focused mainly on classification and detection, while further potentially beneficial applications remain an open question.The review notes particular potential where physical models are complicated, nonlinear, poorly understood, or difficult to generalize.Source: §V · p. 40
  • Global-scale transfer remains challenging because of complex scattering, atmospheric conditions, intra-class variability, culture-dependent features, and limited training samples.The review suggests never-ending learning and self-taught learning as possible solutions.Source: §V · p. 40
  • The review raises limited annotation as a challenge and asks whether remote-sensing image understanding can be learned weakly supervised, semi-supervised, or unsupervised.Source: §V · p. 40
  • The review identifies benchmarking fast-growing algorithms as an open question and points to the 2017 IEEE GRSS Data Fusion Contest and Functional Map of the World Challenge datasets.Source: §V · p. 40
  • The review presents fusion of physics-based modeling and deep neural networks as a promising direction for remote sensing.Imagery arises from physical processes, so the authors call for synergy between models encoding process knowledge and artificial intelligence.Source: §V · pp. 40–41
  • The review calls for models tailored to large, multimodal, geolocated, multi-aspect, and multi-temporal remote-sensing data to address large-scale, real-life problems.It presents monitoring global changes and evaluating resource-consumption reduction strategies as opportunities for remote sensing.Source: §V · p. 41
  • The authors urge remote-sensing experts to bring domain expertise into deep learning as prior knowledge tailored to specific problems, and encourage the community to share data and architectures.Source: §V · p. 41

Related papers

Loading 1710.03959v1…