PaperlayerView exact revision on arXiv

Source-linked AI summary

Remote Sensing Image Scene Classification: Benchmark and State of the Art

Gong Cheng, Junwei Han, Xiaoqiang Lu

arXiv:1703.00121v1cs.CV

TL;DR

Remote sensing scene classification lacks a systematic review and sufficiently large, diverse datasets for developing and comparing data-driven methods. This paper reviews the field, introduces NWPU-RESISC45, and benchmarks representative approaches, reporting comparative performance results as a baseline for future research.

  • Problem

    Existing scene-classification datasets are often small, lack image variation and diversity, and exhibit saturated accuracy, limiting development of new approaches, especially deep-learning methods.

  • Method

    The paper reviews about 170 publications, proposes the public NWPU-RESISC45 benchmark, and evaluates representative scene-classification methods on it.

  • Results

    Deep CNN features outperform all handcrafted features in the benchmark, while the evaluated methods provide a useful performance baseline.

  • Takeaways & Limitations

    NWPU-RESISC45 provides a large-scale public benchmark for developing and evaluating data-driven remote sensing scene-classification algorithms.

Abstract

from arXiv · show

Remote sensing image scene classification plays an important role in a wide range of applications and hence has been receiving remarkable attention. During the past years, significant efforts have been made to develop various datasets or present a variety of approaches for scene classification from remote sensing images. However, a systematic review of the literature concerning datasets and methods for scene classification is still lacking. In addition, almost all existing datasets have a number of limitations, including the small scale of scene classes and the image numbers, the lack of image variations and diversity, and the saturation of accuracy. These limitations severely limit the development of new approaches especially deep learning-based methods. This paper first provides a comprehensive review of the recent progress. Then, we propose a large-scale dataset, termed "NWPU-RESISC45", which is a publicly available benchmark for REmote Sensing Image Scene Classification (RESISC), created by Northwestern Polytechnical University (NWPU). This dataset contains 31,500 images, covering 45 scene classes with 700 images in each class. The proposed NWPU-RESISC45 (i) is large-scale on the scene classes and the total image number, (ii) holds big variations in translation, spatial resolution, viewpoint, object pose, illumination, background, and occlusion, and (iii) has high within-class diversity and between-class similarity. The creation of this dataset will enable the community to develop and evaluate various data-driven algorithms. Finally, several representative methods are evaluated using the proposed dataset and the results are reported as a useful baseline for future research.

I. INTRODUCTION

Remote sensing scene classification is motivated by increasingly diverse earth-observation imagery, but existing datasets and methods lack systematic review and sufficiently challenging benchmarks. The paper reviews prior work, proposes NWPU-RESISC45, and evaluates representative methods on it.

  • Motivation: Increasingly diverse airborne and satellite imagery creates demand for intelligent land-use and land-cover scene classification.Available instruments produce imagery with different spatial, spectral, and temporal resolutions.
  • Motivation: Pixel- and object-level analysis does not fully provide semantic understanding of scene meanings and contents.The paper contrasts individual-pixel analysis and object-level segmentation with semantic-level scene understanding.
  • Research gap: Existing datasets suffer from small scale, limited variation and diversity, and saturated accuracy, leaving a systematic literature review lacking.The paper reports nearly 100% accuracy on UC Merced with deep ConvNets features as an example of saturation.
  • Contributions: NWPU-RESISC45 contains 31,500 images across 45 scene classes, with 700 images per class.The dataset is publicly available and was created by Northwestern Polytechnical University.
  • Contributions: NWPU-RESISC45 combines large scale with variation in imaging conditions and high within-class diversity and between-class similarity.Reported variations include translation, spatial resolution, viewpoint, object pose, illumination, background, and occlusion.
  • Contributions: The paper reviews about 170 publications, categorizes methods, and benchmarks representative state-of-the-art approaches on the proposed dataset.The benchmark results are reported as a useful performance baseline for future research.

II. A REVIEW ON REMOTE SENSING IMAGE SCENE CLASSIFICATION DATASETS

The review covers publicly available high-resolution remote sensing scene-classification datasets, including UC Merced, WHU-RS19, and SIRI-WHU. These datasets differ in scale, class coverage, resolution, and collection source.

  • Dataset overview: Publicly available high-resolution datasets have been introduced to support remote sensing scene-classification research and method evaluation.The section reviews datasets developed by different groups.
  • UC Merced Land-Use Dataset: UC Merced contains 2,100 overhead images in 21 land-use classes, with 100 256×256 images per class at 0.3 m resolution.Its classes include visually overlapping residential categories that differ mainly in structure density.
  • WHU-RS19 Dataset: WHU-RS19 contains 19 RGB scene classes extracted from Google Earth imagery with spatial resolution up to 0.5 m.Its classes include airport, beach, bridge, commercial area, farmland, forest, residential area, river, and viaduct.
  • SIRI-WHU Dataset: SIRI-WHU consists of 2,400 images across 12 scene classes, with 200 200×200 images per class at 2 m resolution.The dataset was collected from Google Earth by the RS_IDEA Group at Wuhan University.

D. RSSCN7 Dataset

RSSCN7 is a multi-scale remote sensing scene dataset, while the surrounding review places it among datasets and handcrafted-feature methods with varying class complexity and representation limits. Handcrafted descriptors capture complementary visual cues but remain difficult to fuse effectively.

  • RSSCN7 Dataset: RSSCN7 contains 2,800 Google Earth images in seven scene classes, with four image scales represented by 100 images per class and scale.Each image is 400×400 pixels, and scale variation is identified as its main challenge.
  • Related datasets: RSC11 contains 11 complicated scene classes from high-resolution imagery of several US cities, including visually similar categories.Its RGB imagery includes classes such as dense forest, harbor, high buildings, railway, and storage tanks.
  • Related datasets: The Brazilian coffee dataset has only two scene classes despite substantial within-class variation from crop management, plant age, and spectral distortion.Its image tiles are 64×64 pixels and cropped from SPOT satellite imagery.
  • Method taxonomy: Scene-classification methods are grouped into handcrafted-feature, unsupervised-feature-learning, and deep-feature-learning categories.The grouping is based on the features used in feature-space classification.
  • Handcrafted features: Handcrafted methods use color, texture, shape, spatial, or spectral information designed with domain expertise.Examples include color histograms, texture descriptors, GIST, SIFT, HOG, and fused local-global representations.
  • Handcrafted features: Color histograms, texture, and GIST describe global properties, whereas SIFT and HOG represent local gradient or shape information.SIFT variants are described as invariant to scale, rotation, and illumination changes.
  • Handcrafted-feature limitations: Effective fusion of complementary handcrafted features remains an open problem, and manually designed features can lose descriptive power on challenging scenes.The paper links this limitation to the influence of human feature design on representational capability.

B. Unsupervised Feature Learning Based Methods

Unsupervised feature learning replaces exclusively manual feature design with representations learned from unlabeled data. The reviewed approaches include PCA, k-means, sparse coding, and autoencoders, but their lack of category labels can limit class discrimination.

  • Overview: Unsupervised feature learning learns basis functions or filters from unlabeled inputs, which may be handcrafted features or raw pixel intensities.The learned outputs form feature representations intended to be better suited to the classification problem.
  • PCA: PCA learns a linear projection whose principal components preserve the structure of the data distribution.The method uses orthogonal basis vectors, but its linear description power limits more abstract representations.
  • K-means clustering: K-means clustering divides data items into k clusters by iteratively assigning items to nearby centroids and updating those centroids.It is treated as unsupervised because class-membership labels are unavailable.
  • Sparse coding: Sparse coding learns an over-complete dictionary from unlabeled samples and represents inputs as efficient linear combinations of dictionary atoms.Coefficient vectors or reconstruction residuals can serve as new feature representations.
  • Autoencoder: Autoencoders learn compressed representations by minimizing reconstruction error between encoded inputs and their decoded reconstructions.The method uses a symmetrical neural network with encoding and decoding layers.
  • Limitation: Unsupervised feature learning can perform well for land-use classification, but omitting category labels may limit discrimination between classes.The review motivates labeled data for achieving better classification performance.

C. Deep Feature Learning Based Methods

Deep feature learning methods replace hand-engineered features with representations learned by multilayer networks. The review focuses mainly on stacked autoencoders and convolutional neural networks for remote sensing scene classification.

  • Deep learning models learn feature representations from data using trainable multilayer networks rather than hand-engineered features.These methods can discover intricate structures and discriminative information in high-dimensional data, while deeper layers show semantic abstraction.
  • The review mainly covers stacked autoencoders and convolutional neural networks among the available deep learning models.Other models mentioned include deep belief nets, deep Boltzmann machines, and stacked autoencoders.
  • SAE: An SAE is a multilayer autoencoder whose successive layers transform outputs from earlier layers into intermediate feature representations.Greedy layer-wise training is followed by joint fine-tuning of all layers with a smaller learning rate.
  • CNNs: CNNs process multidimensional arrays such as multispectral images and include representative architectures such as AlexNet, VGGNet, GoogLeNet, SPPNet, and ResNet.

2) CNNs:

CNNs use layered architectures to extract increasingly expressive image features, while normalization and fully connected layers support final prediction. The benchmark comparison table emphasizes dataset scale and spatial-resolution variation across remote sensing datasets.

  • CNN architecture: Convolutional layers extract image features from low-level edges and lines to deeper structures, objects, and shapes.
  • CNN architecture: Pooling layers follow convolutional layers to support CNN processing of image features.
  • Dataset comparison: Table 1 compares datasets by images per class, scene classes, total images, spatial resolution, image sizes, and year.
  • CNN architecture: Normalization layers aim to improve generalization, while fully connected layers summarize lower-level information for the final decision.Fully connected layers contain many parameters, so dropout is used to help prevent over-fitting.
  • Applications: CNN research in remote sensing includes applications such as geospatial object detection and land use scene classification.A rotation-invariant CNN was proposed to address object rotation variations in detection.

IV. THE PROPOSED NWPU-RESISC45 DATASET

The paper proposes NWPU-RESISC45 to address limitations in existing remote sensing scene-classification datasets. It constructs a publicly available benchmark with broad scene coverage, substantial scale, and varied imagery.

  • Motivation: Existing datasets are limited by small class and image counts, insufficient scene variation and diversity, and saturated classification accuracy.The authors connect these limitations with the need for datasets containing abundant and diverse images to avoid over-fitting.
  • Motivation: NWPU-RESISC45 is proposed as a freely and publicly available large-scale benchmark for remote sensing image scene classification.
  • Dataset construction: The dataset’s 45 scene classes combine 30 widely used categories with 15 additional classes selected through literature and database searches.
  • Dataset construction: Here, “scene” includes generalized land-use, land-cover, man-made-object, and natural-landscape classes with varied spatial patterns.
  • Dataset characteristics: 31,500 remote sensing images are divided into 45 scene classes, with 700 images per class and 256×256-pixel RGB images.Most classes span approximately 30 m to 0.2 m per pixel in spatial resolution; images were extracted from Google Earth by remote sensing interpretation experts.
  • Dataset characteristics: Example images were selected across weather, season, illumination, imaging conditions, and scales, producing variation in viewpoint, pose, resolution, background, and occlusion.
  • Dataset characteristics: NWPU-RESISC45 provides larger image, class, and spatial-resolution coverage than other publicly available scene-classification datasets.
  • Dataset characteristics: The dataset is designed with high within-class diversity and between-class similarity, including fine-grained pairs such as circular and rectangular farmland.

1) Large scale:

NWPU-RESISC45 is designed at a substantially larger scale than existing scene-classification datasets and is intended to support evaluation of representative methods on a large benchmark. Its imagery also incorporates extensive variation and challenging class relationships.

  • 2) Rich image variations:: Images were selected across weather, seasons, illumination, imaging conditions, and scales, yielding rich variation in translation, viewpoint, pose, resolution, background, and occlusion.
  • 3) High within class diversity and between class similarity:: High within-class diversity and between-class similarity make the new dataset challenging compared with simpler datasets where deep methods reached saturated accuracy.
  • Evaluation: Representative scene-classification approaches are evaluated on NWPU-RESISC45 because their performance on a large-scale dataset was previously unclear.

A. Representative Methods

The evaluation spans handcrafted, unsupervised, deep CNN, and fine-tuned CNN representations, with twelve feature types selected for comparison.

  • Representative Methods: Twelve features include handcrafted color histograms, LBP, and GIST; unsupervised BoVW variants; CNN features; and fine-tuned CNN features.The selected CNN models are AlexNet, VGGNet-16, and GoogLeNet.
  • Representative Methods: Color histograms use 192-dimensional RGB representations formed from 64 bins per channel and L1 normalization.The feature is computed directly in RGB color space.
  • Representative Methods: LBP describes texture by comparing each central pixel with its neighbors and histogramming the resulting binary patterns.The descriptor aggregates local-pattern frequencies over image subregions.
  • Representative Methods: GIST represents global scene layout using multiscale-oriented Gabor filters averaged over a 4×4 grid, producing a 512-dimensional vector.The filters use 8 orientations and 4 scales.
  • Representative Methods: BoVW encodes local image descriptors into visual words and pools them into a global histogram, while SPM adds increasingly fine spatial subregions.LLC instead uses locality-constrained projections and max pooling; its implementation shares BoVW’s codebook and feature dimension.
  • Representative Methods: The deep-learning comparisons include AlexNet, VGGNet-16, and GoogLeNet features, plus fine-tuned versions of all three models.Fine-tuning uses the new dataset without data augmentation and different learning rates for the final and earlier layers.

B. Experimental Setup

Experiments compare twelve feature types under two training-test splits, using prescribed codebook sizes, pretrained or fine-tuned CNNs, and linear SVM classification.

  • Experimental Setup: The experiments use 10%-90% and 20%-80% training-test splits, corresponding to 70 or 140 training images per class.The associated test sets contain 630 or 560 images per class.
  • Experimental Setup: BoVW, BoVW+SPM, and LLC use densely sampled 16×16-pixel SIFT patches on an 8-pixel grid, with codebooks of 500, 1000, 2000, or 5000.These settings examine how visual codebook size affects classification performance.
  • Experimental Setup: AlexNet, VGGNet-16, and GoogLeNet are extracted from ImageNet-pretrained models and also fine-tuned on the proposed dataset.The CNN implementations use CPU memory and a GTX Titan X GPU for acceleration.
  • Experimental Setup: All twelve feature types are classified using linear one-vs-all SVMs with the default C=1 setting.A test image receives the label of the classifier with the highest response.
  • Experimental Setup: Figure 3 compares BoVW, BoVW+SPM, and LLC across four visual codebook sizes under 10% and 20% training ratios.The reported plots use overall accuracy.

C. Evaluation Metrics

Evaluation uses overall accuracy and confusion matrices, with repeated experiments supporting mean and standard-deviation reporting.

  • Evaluation Metrics: Overall accuracy divides correctly classified samples by the total number of samples, while average accuracy averages per-class accuracies.The confusion matrix provides a tabular view of classification behavior.
  • Evaluation Metrics: Because every class contains the same number of images, overall and average accuracy have equal values in this dataset.The paper therefore reports overall accuracy and confusion matrix rather than average accuracy.
  • Evaluation Metrics: Each training-test ratio is evaluated ten times, with the mean and standard deviation reported for overall accuracy and confusion matrix results.Repeated evaluation is used to obtain reliable metric results.

D. Experimental Results

Codebook size affects BoVW-family accuracy, while deep CNN features substantially outperform earlier representations and fine-tuning yields the highest reported accuracy.

  • Experimental Results: For both 10% and 20% training ratios, BoVW and LLC perform best with codebook size 5000, whereas BoVW+SPM performs best with size 500.Subsequent BoVW-family evaluations use these optimal parameter settings.
  • Experimental Results: The results compare handcrafted, unsupervised, deep CNN, and fine-tuned CNN features under 10% and 20% training ratios.Tables 3–6 report overall accuracies for these four feature groups.
  • Experimental Results: Deep CNN features outperform handcrafted and unsupervised methods by at least 30% performance improvement.The paper presents this as evidence of deep learning’s superiority over the previous methods evaluated.
  • Experimental Results: Fine-tuning the three off-the-shelf CNN models further boosts accuracy by at least six percentage points and produces the highest accuracy.Confusion matrices are also shown for color histograms, BoVW, VGGNet-16, and fine-tuned VGGNet-16.
  • Experimental Results: The paper combines a comprehensive review, a benchmark dataset, and representative-method evaluation to provide a performance baseline for future research.The conclusion frames the benchmark as supporting development and evaluation of new data-driven algorithms.
Loading 1703.00121v1…