PaperlayerView exact revision on arXiv

Source-linked AI summary

Remote Sensing Image Scene Classification Meets Deep Learning: Challenges, Methods, Benchmarks, and Opportunities

Gong Cheng, Xingxing Xie, Junwei Han, Lei Guo, Gui-Song Xia

arXiv:2005.01094v2cs.CV

TL;DR

Remote sensing scene classification has advanced rapidly, but a comprehensive review of deep-learning achievements remains lacking amid challenging scene variability. This paper surveys more than 160 papers across autoencoder, CNN, and GAN methods, introduces benchmarks, compares representative algorithms on three datasets, and identifies future opportunities. Across the benchmarks, CNN-based methods dominate, while GAN-based methods remain relatively low-performing and the field still has substantial room for improvement.

  • Problem

    A comprehensive review of recent deep-learning methods for remote sensing image scene classification is lacking despite the field’s rapid growth and challenging scene variability.

  • Method

    The paper systematically surveys more than 160 papers covering autoencoder-, CNN-, and GAN-based methods, benchmarks, comparative results, and future directions.

  • Results

    CNN-based methods dominate the three benchmarks, while GAN-based methods perform relatively low and autoencoder-based development has reached a bottleneck.

  • Takeaways & Limitations

    Scene classification has improved substantially through deep learning and large-scale datasets, but considerable work remains to close the gap with human-level performance.

Abstract

from arXiv · show

Remote sensing image scene classification, which aims at labeling remote sensing images with a set of semantic categories based on their contents, has broad applications in a range of fields. Propelled by the powerful feature learning capabilities of deep neural networks, remote sensing image scene classification driven by deep learning has drawn remarkable attention and achieved significant breakthroughs. However, to the best of our knowledge, a comprehensive review of recent achievements regarding deep learning for scene classification of remote sensing images is still lacking. Considering the rapid evolution of this field, this paper provides a systematic survey of deep learning methods for remote sensing image scene classification by covering more than 160 papers. To be specific, we discuss the main challenges of remote sensing image scene classification and survey (1) Autoencoder-based remote sensing image scene classification methods, (2) Convolutional Neural Network-based remote sensing image scene classification methods, and (3) Generative Adversarial Network-based remote sensing image scene classification methods. In addition, we introduce the benchmarks used for remote sensing image scene classification and summarize the performance of more than two dozen of representative algorithms on three commonly-used benchmark data sets. Finally, we discuss the promising opportunities for further research.

I. INTRODUCTION

Remote sensing scene classification labels image patches by semantic content and has become increasingly important as imagery grows more abundant and complex. This survey addresses the field’s challenges and reviews deep-learning methods, benchmarks, and future directions.

  • Remote sensing imagery supports applications including urban planning, hazard detection, environmental monitoring, vegetation mapping, and geospatial object detection.
  • Higher spatial resolution led remote sensing classification to develop parallel pixel-level, object-level, and scene-level branches.Pixel-level methods label pixels, object-level methods recognize objects, and scene-level methods classify image patches semantically.
  • Scene-level analysis became important because pixel- and object-level methods may not adequately capture global scene content when images contain diverse object classes.
  • Early scene-classification methods used handcrafted descriptors, while unsupervised learning offered automatic feature learning but could not fully exploit class information.
  • Deep learning methods increasingly prevailed, with CNN-based approaches achieving advanced classification performance but generally requiring massive annotated training data.
  • The survey responds to a lack of comprehensive review by analyzing challenges, systematically reviewing deep-learning approaches, introducing benchmarks, and discussing future research directions.

II. MAIN CHALLENGES OF REMOTE SENSING IMAGE SCENE CLASSIFICATION

Remote sensing scene classification remains difficult because scenes contain varied, overlapping, differently scaled, and coexisting ground objects. The paper identifies four principal challenges: intraclass diversity, interclass similarity, scale variance, and multiple-object coexistence.

  • The four principal challenges are big intraclass diversity, high interclass similarity, large object or scene-scale variance, and coexistence of multiple ground objects.
  • Intraclass diversity arises because objects within one semantic class vary in style, shape, scale, and spatial distribution.
  • Interclass similarity occurs when different scene classes share objects or semantic content, while ambiguous class definitions further reduce between-class dissimilarity.
  • Object and scene scales vary substantially because sensors operate at different imaging altitudes and scenes contain objects with different intrinsic sizes.
  • Multiple ground objects commonly coexist in one image, making single-label classification insufficient for deep understanding of image contents.

III. SURVEY ON DEEP LEARNING-BASED REMOTE SENSING IMAGE SCENE CLASSIFICATION METHODS

Before deep learning, remote sensing scene classification relied mainly on handcrafted features or representations produced by encoding local features. The survey then examines newer deep-learning approaches in this progression.

  • Pre-deep-learning scene classification mainly used handcrafted features such as Color Histogram, texture descriptors, and GIST.
  • Earlier methods also used representations generated by encoding local features before deep learning became prominent in the field.

A. Autoencoder-Based Remote Sensing Image Scene Classification

Autoencoders learn compressed representations by encoding inputs into hidden layers and reconstructing them, while stacked variants pass each hidden-layer output into the next autoencoder.

  • A. Autoencoder-Based Remote Sensing Image Scene Classification: Autoencoders are shallow, symmetrical networks with input, hidden, and output layers, using encoder and decoder units.Encoding maps the input to a hidden representation; decoding reconstructs the output from that representation.
  • A. Autoencoder-Based Remote Sensing Image Scene Classification: Remote sensing scene classification faces within-class diversity, between-class similarity, scale variation, and coexistence of multiple ground objects.These challenges are illustrated using images from the NWPU-RESISC45 data set.
  • A. Autoencoder-Based Remote Sensing Image Scene Classification: h = f(W · x + b) maps the input x into a hidden representation h through a nonlinear transformation.The encoding weight matrix W and bias vector b parameterize this transformation.
  • A. Autoencoder-Based Remote Sensing Image Scene Classification: Decoding reverses encoding by transforming the hidden layer into a reconstructed output using decoding weights and bias.The reconstructed output is denoted by ˜x, with decoding parameters W′ and b′.
  • A. Autoencoder-Based Remote Sensing Image Scene Classification: Stacked autoencoders compress high-dimensional features through successive hidden layers trained with reconstruction and regularization objectives.In the three-autoencoder example, AE1 feeds AE2 and AE2 feeds AE3; initialization affects convergence and training stability.

2) Autoencoder-based scene classification methods:

Autoencoder-based methods learn mid-level visual representations from unlabeled data and have produced good scene-classification results, but often underuse scene-class information.

  • 2) Autoencoder-based scene classification methods: Autoencoders automatically learn mid-level visual representations from unlabeled data for remote sensing image scene classification.Reported applications include sparse, convolutional sparse, and hierarchical convolutional sparse autoencoder methods.
  • 2) Autoencoder-based scene classification methods: Autoencoder-derived methods have obtained good results in remote sensing image scene classification through unsupervised learning.Their unsupervised formulation supports feature learning without labeled training data.
  • 2) Autoencoder-based scene classification methods: Most autoencoder-based methods cannot learn the best discriminative features because they do not fully exploit scene-class information.This limitation concerns distinguishing different scene classes rather than learning representations from unlabeled data generally.

1) Brief introduction of CNN:

CNNs combine convolutional, pooling, and fully connected layers to learn hierarchical image features, introduce local invariance, and produce class probabilities.

  • 1) Brief introduction of CNN: CNNs are multilayer networks composed of convolutional, pooling, and fully connected layers for visual feature learning.The paper presents CNN architecture through these three layer types.
  • 1) Brief introduction of CNN: Convolutional layers transform feature maps with trainable filters and nonlinear activation functions, capturing features from edges through shapes.As depth increases, the learned representations move across different feature levels.
  • 1) Brief introduction of CNN: Pooling layers apply max, average, or stochastic operations over local regions to create invariance to small shifts and distortions.The paper identifies this invariance as important for object detection and scene classification.
  • 1) Brief introduction of CNN: Fully connected layers summarize lower-level features, while a softmax classifier converts their outputs into probabilities over classes.Dropout can operate on fully connected layers to reduce overfitting from their many parameters.

2) CNN-based scene classification methods:

CNN-based methods brought end-to-end, high-level feature learning to remote sensing scene classification through feature extraction, fine-tuning, and training from scratch. Their main limitation is the need for numerous annotated samples.

  • 2) CNN-based scene classification methods: CNNs offer end-to-end feature learning and extract high-level visual features unavailable to handcrafted feature methods.Their use in remote sensing image analysis expanded after successful large-scale visual classification applications.
  • 2) CNN-based scene classification methods: Pre-trained CNNs can serve as feature extractors by generating image representations that are then classified or fused for scene recognition.Reported frameworks include two-stage pipelines and fusion of deep features from VGGNet.
  • 2) CNN-based scene classification methods: Fine-tuning pre-trained CNNs is preferable to full training when remote sensing data sets are small.Fine-tuning adapts an already trained network to target data when training a new CNN from scratch is impractical.
  • 2) CNN-based scene classification methods: Training CNNs from scratch avoids some limitations of pre-trained features, which may not fully suit target data characteristics or be convenient to modify.Knowledge distillation and lightweight CNN designs are among the approaches discussed in this setting.
  • 2) CNN-based scene classification methods: CNN-based methods generally require numerous annotated samples for fine-tuning or training networks from scratch.This data demand limits their applicability in settings with insufficient annotation.

2) GAN-based scene classification methods:

GAN-based methods have been introduced for remote sensing scene classification mainly to generate samples or learn features adversarially, addressing the need for labeled data. Their reported performance remains below CNN-based methods, although self-supervised learning offers future potential.

  • GAN-based scene classification methods mainly use adversarial training for sample generation or feature learning.GANs model real-sample distributions and generate new samples, helping address the need for massive labeled data.
  • Only a small number of GAN-based scene classification studies have been reported compared with CNN-based methods.
  • GAN-based scene classification performance is inferior to CNN-based methods on the reported benchmarks.
  • Most GAN-based methods cannot be trained end-to-end because they require labels for an additional classifier.
  • GANs’ self-supervised feature-learning capacity remains a promising direction for future scene classification research.

B. AID Data Set

The AID dataset is a large, multisource aerial-scene benchmark with 30 classes and 10,000 images spanning multiple resolutions. It is commonly evaluated using 20% or 50% training splits, with overall accuracy as the survey’s criterion.

  • AID contains 30 scene classes and 10,000 aerial images cropped from Google Earth at 600 × 600 pixels.Each class contains 220 to 420 images.
  • AID is multisource, with images captured by different sensors.
  • AID is multiresolution, with class-dependent pixel resolutions ranging from about 8 m to about 0.5 m.
  • Common AID evaluations use 20% or 50% of images for training and the remainder for testing.
  • The survey evaluates classifiers using overall accuracy, defined over the entire test set.Average accuracy and confusion matrices are also described, but the survey uses only overall accuracy because confusion matrices require substantial space.

B. Performance Comparison

Across three benchmarks, deep-learning scene-classification performance advanced from plateaued autoencoder methods to CNN methods, while GAN methods remained comparatively weak. Benchmark scale and difficulty shape the observed results, with NWPU-RESISC45 remaining the most challenging dataset.

  • 27 deep-learning methods were compared across UC-Merced, AID, and NWPU-RESISC45: 3 autoencoder-based, 22 CNN-based, and 2 GAN-based.
  • Autoencoder-based methods plateaued on standard benchmarks because their relatively simple structures limited feature-learning capability.
  • CNN feature-extractor methods outperformed autoencoder-based methods, while fine-tuning off-the-shelf CNNs generalized better than using CNNs only as feature extractors.
  • GAN-based methods had relatively low performance on the three benchmarks, whereas CNN-based methods continued to dominate and retained improvement potential.
  • UC-Merced CNN results exceeded 99% OA with 80% training data, AID reached about 97% with 50%, and NWPU-RESISC45 reached about 96% with 20%.NWPU-RESISC45 remains more challenging than UC-Merced and AID.
  • CNN performance depends strongly on training-data quantity, motivating larger and more challenging benchmarks for data-driven algorithms.

VI. FUTURE OPPORTUNITIES

The paper identifies future opportunities spanning richer representations, broader labels and scales, larger datasets, reduced supervision, efficient models, limited-sample learning, and cross-domain generalization. These directions address persistent gaps between current systems, open-world imagery, and human scene-recognition ability.

  • Future work should improve discriminative representations amid intraclass diversity and interclass similarity.Suggested approaches include multi-task learning and designing or fusing CNNs.
  • Learning multi-scale features remains an open problem because scenes and objects appear at different scales.
  • Multi-label classification is needed because bird’s-eye imagery commonly contains multiple ground-object classes.Single-label classification therefore cannot deeply represent intricate image content.
  • Larger-scale datasets are needed because current models classify known dataset categories but are blind in principle to classes outside those datasets.
  • Fully training deep CNNs is impracticable with currently available datasets, so methods rely mainly on fine-tuning or pretrained feature extraction.Deep CNNs can have millions of parameters and overfit tens of thousands of training samples.
  • Unsupervised, compact, limited-sample, few-shot, and zero-shot learning could reduce annotation, computation, and scalability constraints.The paper notes that advanced CNNs require large labeled datasets, high-performance GPUs, or many training iterations.
  • Cross-domain classification remains difficult because performance drops significantly when training and testing distributions differ across platforms or sensors.

VII. CONCLUSIONS

The paper surveys the rapid development of remote sensing image scene classification through its challenges, deep learning methods, benchmarks, and future research opportunities.

  • The review first discusses the main challenges facing remote sensing image scene classification.
  • It surveys autoencoder-, convolutional neural network-, and generative adversarial network-based scene classification methods.
  • It introduces mainstream benchmarks, summarizes method performance on three widely used data sets, analyzes the results, and identifies promising research opportunities.
Loading 2005.01094v2…