PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual descriptors for content-based retrieval of remote sensing images

Paolo Napoletano

arXiv:1602.00970v5cs.CV

TL;DR

Remote-sensing images vary in texture and object content, leaving descriptor choice uncertain. The paper evaluates descriptor families across four retrieval schemes and two datasets; CNN features outperform hand-crafted features across the tested schemes, with domain-adapted and NetVLAD features showing specific advantages.

  • Problem

    Remote-sensing images vary from fine- to coarse-grained textures and object-containing scenes, so the most suitable visual descriptor is unclear.

    Source: §1 · p. 2

  • Method

    The study compares global and local hand-crafted descriptors and CNN features across four retrieval schemes and two public remote-sensing datasets.

    Source: §1 · p. 2

  • Results

    CNN descriptors outperform global and local hand-crafted descriptors across both datasets and retrieval schemes, while SatResNet-50 improves on multimedia-pretrained CNNs and NetVLAD suits fine-grained textures and objects.

    Source: §5 · pp. 24–25

  • Takeaways & Limitations

    The findings highlight remote-sensing domain adaptation as important and identify NetVLAD as a strong option for images with fine-grained textures and objects.

    Source: §5 · pp. 24–25

    Nearest-image search can be slow because of large datasets, high-dimensional vectors, and memory access; compact representations may improve efficiency.

    Source: §4.5 · p. 20

Abstract

from arXiv · show

In this paper we present an extensive evaluation of visual descriptors for the content-based retrieval of remote sensing (RS) images. The evaluation includes global hand-crafted, local hand-crafted, and Convolutional Neural Network (CNNs) features coupled with four different Content-Based Image Retrieval schemes. We conducted all the experiments on two publicly available datasets: the 21-class UC Merced Land Use/Land Cover (LandUse) dataset and 19-class High-resolution Satellite Scene dataset (SceneSat). The content of RS images might be quite heterogeneous, ranging from images containing fine grained textures, to coarse grained ones or to images containing objects. It is therefore not obvious in this domain, which descriptor should be employed to describe images having such a variability. Results demonstrate that CNN-based features perform better than both global and and local hand-crafted features whatever is the retrieval scheme adopted. Features extracted from SatResNet-50, a residual CNN suitable fine-tuned on the RS domain, shows much better performance than a residual CNN pre-trained on multimedia scene and object images. Features extracted from NetVLAD, a CNN that considers both CNN and local features, works better than others CNN solutions on those images that contain fine-grained textures and objects.

1. Introduction

Remote-sensing images vary from fine- to coarse-grained textures and object-containing scenes, making descriptor choice unclear. The paper compares descriptor families across four retrieval schemes and two public datasets.

1. Introduction

  • Heterogeneous remote-sensing imagery makes it unclear which visual descriptor best represents its content.Images may contain fine-grained textures, coarse-grained textures, or objects.Source: §1 · p. 2
  • The evaluation compares global and local hand-crafted descriptors with CNN intermediate representations trained for generic or remote-sensing recognition.The descriptors are evaluated alongside four retrieval schemes.Source: §1 · p. 2
  • Experiments use the 21-class UC Merced Land Use/Land Cover and 19-class High-resolution Satellite Scene datasets.The paper also considers multiple similarity measures for the first three retrieval schemes and histogram intersection for active-learning RF.Source: §1 · p. 2

2. Background and Related Works

The background frames CBIR as a pipeline that represents images, retrieves similar items, presents ranked results, and incorporates relevance feedback. It reviews global, local, and learned descriptors alongside basic retrieval and feedback strategies.

2. Background and Related Works

  • A CBIR system indexes image content, retrieves similar images, visualizes ranked results, and supports relevance feedback.Feedback can be selected manually by a user or automatically by the system.Source: §2 · p. 3

2.1. Indexing

  • Global hand-crafted features describe whole-image color, texture, and shape, while local descriptors represent salient patches and can be aggregated as visual-word histograms.CNNs provide learned representations; a pretrained network can serve as a fixed feature extractor for another task.Source: §2.1 · pp. 3–4

2.2. Retrieval

  • Basic retrieval ranks database images by comparing their visual descriptors with the query descriptor using a distance or similarity measure.The background lists Euclidean, cosine, Manhattan, and χ2-square measures.Source: §2.2 · p. 4

2.3. Relevance Feedback

  • Relevance feedback expands an initial query with images judged relevant, either automatically from top-ranked results or manually by the user.Active-learning feedback uses an SVM and iteratively proposes uncertain and diverse images for user feedback.Source: §2.3 · pp. 5–6

3. Methods and materials

The evaluation compares visual descriptors and retrieval schemes on two datasets with known ground truth. Retrieval ranks images by descriptor distance to a query.

3. Methods and materials

  • For a query, the k most relevant database images are those with the smallest distances between their feature vectors and the query vector.The tested distances are Euclidean, cosine, Manhattan, and χ2-square.Source: §3 · p. 6
  • The study evaluates visual descriptors and different retrieval schemes on two public datasets with known ground truth.The descriptors and schemes are detailed in the following sections.Source: §3 · p. 6

3.1. Visual descriptors

The paper evaluates normalized global and local hand-crafted descriptors and CNN representations for remote-sensing image retrieval. CNNs include generic pretrained networks, NetVLAD, and a ResNet-50 fine-tuned on aerial scenes.

3.1. Visual descriptors

  • The evaluated descriptors include global and local hand-crafted features and CNNs, with some hand-crafted features computed on color and grayscale images.Feature vectors are L2-normalized.Source: §3.1 · p. 6

3.1.1. Global hand-crafted descriptors •

  • Global descriptors encode image-wide color, texture, shape, and spatial information using histograms, co-occurrence features, wavelets, Gabor, Gist, HOG, granulometry, and LBP.The evaluated examples include both compact vectors, such as 8-dimensional DT-CWT features, and higher-dimensional vectors, such as 1536-dimensional spatial RGB histograms.Source: §3.1.1 · pp. 6–7

3.1.2. Local hand-crafted descriptors

  • Local features include SIFT and LBP representations, with SIFT encoded as BoVW, VLAD, or Fisher vectors and descriptors sampled at keypoints or on dense grids.The SIFT BoVW codebooks contain 1024 visual words; dense SIFT VLAD and Fisher vectors have 25,600 and 40,960 dimensions, respectively.Source: §3.1.2 · p. 7

3.1.3. CNN-based descriptors

  • Generic CNN descriptors are intermediate network representations extracted by removing the final softmax nonlinearity and last fully connected layer.The tested architectures span several accuracy-speed trade-offs and include AlexNet-, VGG-, GoogleNet-, and ResNet-family networks.Source: §3.1.3 · pp. 7–8
  • NetVLAD combines VGG VeryDeep 16 with a VLAD layer and was trained for place recognition using panoramic Street View imagery.Source: §3.1.3 · p. 8
  • SatResNet-50, initialized from ILSVRC2015-pretrained ResNet-50 and fine-tuned on AID using the remaining images after selecting 20 per class, achieved 96.34% Top-1 and 99.34% Top-5 accuracy.Source: §3.1.3 · p. 9
  • The fine-tuned SatResNet-50 serves as a feature extractor using 2048-component fully connected layer activations.Source: §3.1.3 · p. 9

3.2. Retrieval schemes

The evaluation compares four retrieval schemes, using several distance functions and relevance feedback to examine visual descriptors across different retrieval setups.

3.2. Retrieval schemes

  • Active-learning-based RF simulates user interaction by selecting relevant and non-relevant images from ground truth, using histogram intersection as its distance measure.The scheme follows Demir and Bruzzone (2015).Source: §3.2 · p. 9
  • Basic image retrieval returns images ranked by similarity to the query, using Euclidean, Cosine, Manhattan, or χ2-square distance.Source: §3.2 · p. 9
  • Pseudo-RF treats the first n images returned by the initial query as relevant, with n varied from 1 to 10.Source: §3.2 · p. 9
  • Manual RF uses the first n actually relevant images in the initial results, with n varied from 1 to 10.Source: §3.2 · p. 9

3.3. Remote Sensing Datasets

The evaluation uses two remote-sensing datasets with different image sizes, resolutions, and scene coverage; the differences may affect descriptor performance across datasets.

3.3. Remote Sensing Datasets

  • LandUse contains 2,100 256 × 256 RGB images across 21 land-use classes, drawn from USGS aerial orthoimagery at 30 cm pixel resolution.There are 100 images per class.Source: §3.3 · p. 10
  • SceneSat contains 1,005 600 × 600 RGB satellite-scene images across 19 classes, with about 50 images per class.Source: §3.3 · p. 10

3.3.1. Differences between LandUse and SceneSat

  • SceneSat depicts larger land areas and more heterogeneous content, while LandUse has a different zoom level that makes objects more readily recognizable.The authors expect descriptor performance to differ across the datasets.Source: §3.3.1 · p. 11

3.4. Retrieval measures

The paper evaluates retrieval effectiveness with ranking, precision, recall, and average-precision measures, and estimates query cost independently of computer architecture.

3.4. Retrieval measures

  • Retrieval effectiveness is assessed with ANMRR, Precision, Recall, and MAP, while EQC measures query cost independently of computer architecture.Source: §3.4 · p. 11, §3.4.4 · p. 13

3.4.1. Average Normalized Modified Retrieval Rank (ANMRR)

  • ANMRR summarizes the number and rank of relevant retrieved items, accommodates queries with different ground-truth sizes, and ranges from zero to one, with lower values better.The average rank is computed per query; the penalty constant K(q) is commonly 2NG(q).Source: §3.4.1 · pp. 11–12

3.4.2. Precision and Recall

  • Precision is the fraction of retrieved images that are relevant, while recall is the fraction of relevant images successfully retrieved.Precision may be measured at a cutoff rank; the interpolated precision-recall curve uses 11 recall levels and the highest precision at or above each level.Source: §3.4.2 · p. 12

3.4.3. Mean Average Precision (MAP)

  • MAP averages query-level average precision, which uses precision at each retrieved rank weighted by whether the item at that rank is relevant.Source: §3.4.3 · pp. 12–13

3.4.4. Equivalent Query Cost

  • EQC estimates descriptor query cost from descriptor length, using a basic cost defined for a descriptor of length B; B is set to 5, the shortest evaluated descriptor.The measure is based on distance computation being linear in the number of components.Source: §3.4.4 · p. 13

4. Results

Across datasets, retrieval schemes, and measures, CNN-based descriptors rank highest overall. Results also show that the best descriptor and retrieval strategy depend on image content and computational cost.

4.1. Feature evaluation using the basic retrieval scheme

  • On both datasets, SatResNet-50 improves on Bag of Dense SIFT and DT-CWT on LandUse and Bag of Dense SIFT (FV) on SatScene across most reported retrieval metrics.On LandUse, SatResNet-50 has P@100 = 64.42; on SatScene, it has P@50 = 68.02.Source: §4.1 · pp. 13–14, Table 1 · p. 14, Table 2 · p. 15

4.2. Feature evaluation using the pseudo-RF retrieval scheme

  • With pseudo-RF at n = 5, both datasets improve over basic retrieval for every descriptor, with gains largest for CNN and local hand-crafted descriptors.For Vgg M, MAP improves by about 5% on LandUse and about 7% on SceneSat.Source: §4.2 · p. 17, Figure 7 · p. 17, Table 3 · p. 18

4.3. Feature evaluation using the manual-RF retrieval scheme

  • Manual RF at n = 5 outperforms both basic retrieval and pseudo-RF on both datasets, with the largest gains for CNN and local hand-crafted descriptors.For Vgg, MAP improves by about 9% on LandUse and about 14% on SceneSat.Source: §4.3 · p. 18, Table 4 · p. 19

4.4. Feature evaluation using the active-learning-based-RF retrieval scheme

  • Active-learning RF improves all descriptors on LandUse, but on SceneSat improves only hand-crafted descriptors and newer CNNs such as ResNet or NetVLAD.NetVLAD is the best descriptor on both datasets in this scheme; the authors relate its performance to its less-sparse feature vector and the SVM used by the scheme.Source: §4.4 · p. 20, Table 5 · p. 21

4.5. Average rank of visual descriptors across RS datasets

  • CNN-based descriptors occupy the first 13 of 38 average-rank positions across datasets, retrieval schemes, and measures.Local hand-crafted descriptors are penalized by large feature vectors; Dense SIFT (FV) has 40,960 dimensions.Source: §4.5 · p. 20, Table 8 · p. 24
  • Active-learning RF is not always advantageous: for the top five descriptors it performs worse globally than pseudo-RF and has higher EQC.For other descriptors, active-learning RF can perform better than pseudo-RF.Source: §4.5 · p. 20, Table 8 · p. 24

4.6. Comparison with the state of the art

  • The Vgg M 128 offers a stronger performance-cost trade-off than the cited texture descriptor, with about 25% lower ANMRR at about twice the computational cost.Bag of Dense SIFT (VLAD) has performance close to CNN-based descriptors, with ANMRR = 0.460 and EQC = 5120.Source: §4.6 · pp. 22–24

5. Conclusions

The paper evaluates global and local hand-crafted descriptors and CNN features with four CBIR schemes on two remote-sensing datasets. CNN descriptors perform better on average across both datasets and all retrieval schemes; the conclusions also highlight NetVLAD’s performance on images with fine-grained textures and objects.

5. Conclusions

  • The evaluation covers global and local hand-crafted descriptors and CNN features paired with basic, pseudo-RF, manual-RF, and active-learning-based RF schemes on two public datasets.The datasets differ in image size and resolution.Source: §5 · p. 24
  • CNN-based descriptors outperform global and local hand-crafted descriptors on average across both datasets and all retrieval schemes.The RS-domain-adapted ResNet-50 also improves on CNNs pretrained on multimedia scene and object images.Source: §5 · p. 24
  • NetVLAD works better on images containing fine-grained textures and objects, particularly LandUse classes such as chaparral, harbor, and runway.The paper identifies NetVLAD as a CNN that considers local features.Source: §5 · p. 25
  • Active-learning-based relevance feedback is the best-performing retrieval scheme on average, but costs at least 4 times other RF schemes and 20 times a basic query.The cost comparison is per query.Source: §5 · p. 25
  • Future work is proposed to test whether compact feature representations, such as hashes or polysemous codes, can speed up image search.Source: §5 · p. 25

Funding

The authors acknowledge NVIDIA Corporation for donating the Tesla K40 GPU used in part of the experiments.

Funding

  • NVIDIA Corporation donated the Tesla K40 GPU used for part of the experiments.Source: Funding · p. 25

Related papers

Loading 1602.00970v5…