PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Low Dimensional Convolutional Neural Networks for High-Resolution Remote Sensing Image Retrieval

Weixun Zhou, Shawn Newsam, Congmin Li, Zhenfeng Shao

arXiv:1610.03023v2cs.CV

TL;DR

HRRSIR needs powerful feature representations because traditional hand-crafted low-level features are difficult to design for complex remote sensing imagery. The paper evaluates pretrained CNN-layer features and proposes LDCNN, a remote-sensing-trained low-dimensional CNN; the schemes achieve state-of-the-art performance, with LDCNN producing compact features and improving selected datasets.

  • Problem

    Traditional HRRSIR relies on time-consuming hand-crafted low-level features, while pretrained CNNs are trained on ImageNet rather than remote sensing data.

  • Method

    The paper extracts fully connected and convolutional features from pretrained CNNs and trains an LDCNN combining convolution layers with a three-layer perceptron on remote sensing data.

  • Results

    The two schemes achieve state-of-the-art HRRSIR performance, while LDCNN yields 30-D features and improves the first scheme’s best results on RSD by 26% and RSSCN7 by 8.3%.

  • Takeaways & Limitations

    CNN-derived features can provide comparative or superior HRRSIR performance to hand-crafted features, including compact representations from LDCNN.

  • Takeaways & Limitations

    UCMD is relatively small and its retrieval results have become saturated, limiting its suitability for evaluating deep-feature performance.

Abstract

from arXiv · show

Learning powerful feature representations for image retrieval has always been a challenging task in the field of remote sensing. Traditional methods focus on extracting low-level hand-crafted features which are not only time-consuming but also tend to achieve unsatisfactory performance due to the content complexity of remote sensing images. In this paper, we investigate how to extract deep feature representations based on convolutional neural networks (CNN) for high-resolution remote sensing image retrieval (HRRSIR). To this end, two effective schemes are proposed to generate powerful feature representations for HRRSIR. In the first scheme, the deep features are extracted from the fully-connected and convolutional layers of the pre-trained CNN models, respectively; in the second scheme, we propose a novel CNN architecture based on conventional convolution layers and a three-layer perceptron. The novel CNN model is then trained on a large remote sensing dataset to learn low dimensional features. The two schemes are evaluated on several public and challenging datasets, and the results indicate that the proposed schemes and in particular the novel CNN are able to achieve state-of-the-art performance.

1. Introduction

HRRSIR addresses the challenge of managing large volumes of high-resolution remote sensing data by retrieving relevant images from large databases. This paper investigates CNN-based deep feature representations as an alternative to difficult-to-design hand-crafted features.

  • High-resolution remote sensing imagery enables detailed interpretation but creates a persistent data organization and management challenge.
  • HRRSIR retrieves relevant images from large databases and depends on both feature extraction and similarity measurement.
  • Traditional HRRSIR methods mainly use low-level global and local features, including spectral, shape, texture, and invariant features.
  • Hand-crafted features are challenging to design as powerful representations because they remain essentially low-level.
  • The paper proposes two CNN-based schemes: extracting features from pretrained layers and learning low-dimensional features with a novel CNN.

2. Convolutional Neural Networks (CNN)

CNNs combine convolutional, pooling, and fully connected layers to learn image representations, while the paper evaluates several ImageNet-pretrained architectures for HRRSIR feature extraction.

  • CNN architectures generally consist of convolutional, pooling, and fully connected layers.
  • Convolutional filters generate feature maps, while pooling subsamples spatial dimensions and reduces feature-map size.
  • ReLU generates non-negative features from convolutional and fully connected feature maps.
  • The evaluated pretrained models include AlexNet, CaffeRef, VGGF, VGGM, and VGGS.
  • The VGG variants share a 4096-D second fully connected layer, while variant VGGM models provide 128-D, 1024-D, and 2048-D alternatives.

3. Deep Feature Representations for HRRSIR

The paper develops two CNN-based HRRSIR feature schemes: pretrained-layer extraction and a low-dimensional CNN designed for remote sensing data.

  • 3.1 First Scheme: Features Extracted from the Pre-trained Layers: The first scheme extracts fully connected and convolutional features directly from pretrained CNN layers after mean subtraction.
  • 3.1 First Scheme: Features Extracted from the Pre-trained Layers: Fc1 and Fc2 produce feature representations that can be used directly for similarity measurement, while convolutional activations require aggregation.
  • 3.1 First Scheme: Features Extracted from the Pre-trained Layers: BOVW, VLAD, and IFK aggregate high-dimensional local convolutional descriptors into compact feature vectors.
  • 3.2 Second Scheme: Features Extracted by the Novel Low Dimensional CNN: The second scheme addresses pretrained-model mismatch, noncompact 4096-D features, and fully connected layers’ overfitting tendency.
  • 3.2 Second Scheme: Features Extracted by the Novel Low Dimensional CNN: LDCNN combines five conventional convolution layers, a three-layer perceptron-based mlpconv layer, global average pooling, and a softmax classifier.
  • 3.2 Second Scheme: Features Extracted by the Novel Low Dimensional CNN: Global average pooling averages each feature map to produce an n-dimensional feature vector for HRRSIR, where n is the number of image classes.

4. Experiments and Analysis

The experiments evaluate the proposed HRRSIR schemes on several publicly available remote sensing image datasets using datasets, experimental setup, and detailed result analyses.

  • The evaluation uses several publicly available remote sensing image datasets and reports the datasets, experimental setup, and results in detail.

4.1 Dataset

The study evaluates HRRSIR features on four public datasets spanning different scales, resolutions, class counts, and levels of challenge. UCMD is relatively small and saturated, whereas RSD, RSSCN7, and AID provide more challenging evaluation settings.

  • UCMD: UCMD contains 21 classes with 100 256×256-pixel images per class at about one-foot resolution.The images were cropped from large aerial images downloaded from USGS.
  • RSD: RSD contains 19 classes and 1005 600×600-pixel images collected from Google Earth.
  • RSSCN7: RSSCN7 contains seven land-use classes, with 400 images per class sampled across four zoom scales.Each image is 400×400 pixels, with 100 images per scale.
  • AID: AID contains 30 scene types and 10,000 600×600-pixel images, with 220 to 420 images per class.Its scale and multi-resolution make it challenging.
  • Dataset difficulty: UCMD is relatively small and performance on it has saturated, while the other three datasets are more challenging because of image scale, size, and pixel resolution.

4.2 Experimental Setup

The experiments extract CNN features under model-specific input settings, evaluate retrieval across all datasets, and use Euclidean distance after L2 normalization. Performance is measured primarily with ANMRR and mAP, with P@k used when those measures disagree.

  • Feature extraction: Images are resized to 227×227 pixels for AlexNet and CaffeRef and to 224×224 pixels for the other CNNs.
  • Feature extraction: K-means constructs dictionaries for convolutional features, with sizes 1000 for BOVW and 100 each for VLAD and IFK.
  • LDCNN training: LDCNN transfers and fixes VGGM convolutional-layer weights to accelerate training with limited labeled remote-sensing images.The model removes layers after VGGM’s last pooling layer and retains the preceding layers.
  • Retrieval protocol: The evaluation uses 2100 UCMD, 1005 RSD, 2800 RSSCN7, and 10,000 AID queries, with Euclidean distance between L2-normalized feature vectors.
  • Performance measures: ANMRR and mAP evaluate retrieval performance, while P@k is an auxiliary measure when ANMRR and mAP produce opposite results.Lower ANMRR is better, whereas higher mAP and precision indicate better retrieval.

4.3 Results of the First Scheme

The first scheme evaluates fully connected and convolutional features from pre-trained CNNs, including ReLU variants and PCA-compressed representations. Results show model-, layer-, aggregation-, activation-, and dataset-dependent behavior, while low-dimensional features retain competitive performance and deep features outperform prior UCMD methods.

  • Fully-connected features: 12% lower ANMRR and 14% higher mAP are achieved by VGGM_Fc2 versus VGGM128_Fc2 on UCMD.On RSD, CaffeRef_Fc2 achieves about 18% lower ANMRR and 21% higher mAP than VGGM128_Fc2.
  • Fully-connected features: ANMRR and mAP can rank features oppositely, so P@k further compares cases such as RSSCN7 and AID.For RSSCN7, VGGM_Fc1 is slightly better than VGGF_Fc2 within 100 retrieved images; for AID, VGGS_Fc1 is slightly better than VGGS_Fc2.
  • Fully-connected features: VGGM outperforms its three lower-dimensional variants across the four datasets, while Fc2 generally performs better than Fc1.The variants nevertheless reduce storage and query costs because of their lower feature dimensions.
  • Convolutional features: IFK is best among convolutional aggregations on UCMD for most evaluated CNNs, with VD16 achieving ANMRR 0.407, while BOVW generally performs worst.The discussion attributes BOVW’s weakness to ignoring spatial information important in remote-sensing images.
  • ReLU effects: ReLU improves some UCMD VLAD and IFK results but generally decreases convolutional-feature performance on the four datasets.For fully connected features, Fc1 improves without ReLU, whereas Fc2_ReLU often matches or exceeds Fc2 except for selected VGGM variants.
  • Low-dimensional features: Low-dimensional PCA-compressed Fc features remain competitive with 4096-D features, with 16-D best on RSD and RSSCN7, 32-D on UCMD, and 64-D on AID.These compact representations are reported to reduce storage cost and improve retrieval efficiency.
  • Comparison with prior methods: Deep features achieve ANMRR 0.375 on UCMD and improve the reported state of the art by a significant margin.

4.4 Results of the Proposed LDCNN

LDCNN produces compact 30-D features and improves retrieval over the first scheme on RSD and RSSCN7, but not on UCMD. Its performance pattern aligns with dataset similarity to the AID training data, while requiring fewer parameters than the compared VGGM models.

  • 30-D features from LDCNN are more compact than Fc and Conv features extracted from pre-trained CNNs.The feature dimensionality is explicitly reported as 30.
  • 26% and 8.3% improvements over the first scheme are reported on RSD and RSSCN7, respectively.
  • LDCNN performs slightly better than fine-tuned Fc features but does not surpass the first scheme or fine-tuned VGGM on UCMD.
  • LDCNN’s stronger results on RSD and RSSCN7 are attributed to their greater similarity to the AID training dataset than UCMD.The comparison concerns image size and spatial resolution.
  • LDCNN has about 2.6 times fewer parameters than VGGM-Finetune and 2.7 times fewer than VGGM.The comparison counts weights and biases in the relevant layers.

5. Discussion

The experiments support both proposed schemes as effective for HRRSIR, with deep CNN features matching or exceeding hand-crafted features. LDCNN offers compact representations and dataset-dependent performance, while ReLU affects Fc and Conv features differently.

  • Deep features from pre-trained CNNs and LDCNN achieve comparative or superior performance to state-of-the-art hand-crafted features.
  • Fc1 outperforms Fc2 without ReLU but underperforms it with ReLU, indicating that nonlinearity changes their relative retrieval performance.
  • ReLU decreases Conv-feature performance on all four datasets except UCMD.The discussion relates this pattern to feature-distribution differences from traditional hand-crafted features.
  • LDCNN combines conventional convolutional layers with a three-layer perceptron and transfers convolutional parameters from VGGM to speed training.
  • LDCNN generates compact 30-D vectors, outperforming the first scheme and fine-tuned VGGM on RSD and RSSCN7 but performing worse on UCMD.The discussion attributes the UCMD result to differences between UCMD and AID in image size and spatial resolution.

6. Conclusions

The paper presents two CNN-based schemes for extracting deep features for HRRSIR and evaluates fully connected, convolutional, and low-dimensional representations. LDCNN addresses the domain difference between ImageNet and remote sensing data and may also serve other remote sensing tasks.

  • The first scheme extracts Fc and Conv features from pre-trained CNN layers, using feature aggregation for Conv features before similarity measurement.
  • The first scheme also evaluates how ReLU affects Fc and Conv features to identify effective representations.
  • LDCNN uses conventional convolution layers and a three-layer perceptron, trained on a large remote sensing dataset rather than relying only on ImageNet pre-training.The motivation is that ImageNet differs substantially from remote sensing data.
  • LDCNN generates 30-D features that achieve remarkable performance on several remote sensing datasets.
  • Although designed for HRRSIR, LDCNN can also be applied to scene classification and object detection.
Loading 1610.03023v2…