Source-linked AI summary
PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrieval
Weixun Zhou, Shawn Newsam, Congmin Li, Zhenfeng Shao
TL;DR
Remote sensing image retrieval lacks large, suitable benchmarks for developing and comparing feature representations. This paper introduces PatternNet and evaluates handcrafted and deep-learning methods under consistent conditions to establish baseline results for future research.
Problem
Remote sensing retrieval lacks large labeled datasets suitable for deep-learning methods, and existing benchmarks can exhibit saturated retrieval performance.
Method
The paper constructs PatternNet as a large-scale remote sensing retrieval benchmark and evaluates handcrafted and deep-learning features under consistent experimental conditions.
Results
The evaluation establishes baseline results for existing handcrafted and deep-learning remote sensing image retrieval methods on PatternNet.
Takeaways & Limitations
PatternNet is presented as the largest publicly available remotely sensed evaluation dataset for remote sensing image retrieval.
Takeaways & Limitations
The paper notes that remote sensing lacks datasets with the large amounts of labeled data required by deep-learning methods.
Abstract
from arXiv · showhide
Remote sensing image retrieval(RSIR), which aims to efficiently retrieve data of interest from large collections of remote sensing data, is a fundamental task in remote sensing. Over the past several decades, there has been significant effort to extract powerful feature representations for this task since the retrieval performance depends on the representative strength of the features. Benchmark datasets are also critical for developing, evaluating, and comparing RSIR approaches. Current benchmark datasets are deficient in that 1) they were originally collected for land use/land cover classification and not image retrieval, 2) they are relatively small in terms of the number of classes as well the number of sample images per class, and 3) the retrieval performance has saturated. These limitations have severely restricted the development of novel feature representations for RSIR, particularly the recent deep-learning based features which require large amounts of training data. We therefore present in this paper, a new large-scale remote sensing dataset termed "PatternNet" that was collected specifically for RSIR. PatternNet was collected from high-resolution imagery and contains 38 classes with 800 images per class. We also provide a thorough review of RSIR approaches ranging from traditional handcrafted feature based methods to recent deep learning based ones. We evaluate over 35 methods to establish extensive baseline results for future RSIR research using the PatternNet benchmark.
1. Introduction
Rapid growth in remote sensing data makes efficient image retrieval essential, while handcrafted features and existing datasets limit progress. The paper addresses these gaps by introducing PatternNet, reviewing RSIR methods, and providing consistent benchmark evaluations.
- Motivation: Rapidly increasing remote sensing data volumes create a challenge of efficiently accessing images of interest.Higher acquisition rates enable considerable daily data collection, making large collections difficult to manage.
- Feature representations: Retrieval performance depends heavily on feature effectiveness, but handcrafted features require time-consuming, heuristic design and may be suboptimal.Conventional RSIR uses global or local low-level visual features, with local features offering robustness to occlusion and invariance to viewing angle and lighting.
- Deep learning: Supervised feature learning methods such as CNNs outperform unsupervised methods such as k-means and auto-encoders by a wide margin.Deep learning learns feature representations from data, motivating larger datasets for RSIR.
- Contributions: The paper introduces PatternNet as a publicly available, high-resolution, large-scale benchmark dataset for RSIR and evaluates handcrafted and deep-learning features under consistent conditions.It also provides a comprehensive review of RSIR methods and extensive baseline results for future research.
2. Remote Sensing Image Retrieval Methods
RSIR methods are organized around feature representations, broadly spanning handcrafted and deep-learning features, with hybrid combinations also considered. Traditional approaches use global or local handcrafted descriptors, while CNN-based features usually outperform them but require large labeled datasets that are scarce in remote sensing.
- RSIR retrieval performance depends heavily on feature-representation effectiveness, motivating extensive development of powerful representations.
- Existing RSIR representations comprise handcrafted features and deep learning features, although hybrid combinations are also considered.
- Traditional handcrafted features: Traditional methods use global or local handcrafted low-level features, including color, texture, shape, and patch-centered local descriptors.Color and texture features are used more widely than shape features, while local features are extracted from patches centered at interesting points.
- Traditional handcrafted features: Local features often outperform global features across remote sensing tasks, though the two types can be complementary and combined.
- Mid-level representations: Mid-level encoding methods such as BOVW, VLAD, and IFK transform numerous high-dimensional local features into more practical representations for large-scale RSIR.BOVW encodes local features into a compact global image representation; VLAD-based representations are more discriminative than BOVW in almost all land cover classes.
- Deep learning features: CNN-based features usually outperform competing features and achieve state-of-the-art performance on several evaluation datasets, but they require large amounts of labeled data.The lack of such datasets in remote sensing makes achieving satisfactory performance difficult and time consuming.
3. PatternNet: A Large-Scale Dataset for Remote Sensing Image Retrieval
PatternNet is introduced as a large-scale, high-resolution remote sensing dataset collected specifically for remote sensing image retrieval (RSIR). It addresses limitations of existing benchmarks through retrieval-focused collection, larger class coverage, higher resolution, limited background, and substantial visual diversity.
- Motivation: Existing RSIR benchmarks are limited because UCMD is small and saturated, while AID and NWPU-RESISC45 were collected for scene classification and often contain irrelevant background.UCMD has 21 classes with 100 images per class, and retrieval performance on it has saturated.
- Dataset construction: PatternNet images were collected from Google Earth imagery or the Google Map API for US cities and generally have higher resolution than comparable datasets.Its highest spatial resolution is around 0.062m and its lowest is around 4.693m.
- Dataset characteristics: PatternNet is designed so that the class of interest covers most of each image, leaving very little background for retrieval.Higher spatial resolution helps the classes of interest constitute a larger portion of the image.
- Dataset characteristics: PatternNet covers many US cities and varying spatial resolutions, producing high intra-class diversity and high inter-class similarity.Examples include wastewater treatment plants for intra-class diversity and basketball courts versus tennis courts for inter-class similarity.
4. Baseline Methods
The baseline evaluation covers handcrafted low-level features, SIFT-based mid-level encodings, unsupervised feature learning, and CNN representations on PatternNet. The CNN baselines include established pre-trained architectures and an efficient low-dimensional CNN designed for limited labeled data.
- Low-level features: Handcrafted baselines include global statistics, color histograms, Gabor texture, GIST, LBP, and PHOG features.Simple statistics use grayscale mean and standard deviation, while PHOG represents spatial layout through a spatial pyramid.
- Mid-level features: Mid-level representations aggregate SIFT descriptors using BOVW, VLAD, and IFK encoding approaches.BOVW quantizes descriptors into a global visual-word histogram; VLAD and IFK construct higher-dimensional encoded representations.
- Unsupervised feature learning: The evaluated UFL framework learns representations from unlabeled images through local extraction, unsupervised learning, encoding, and sparse pooling.An auto-encoder learns local features, which are then pooled into a global feature vector.
- Convolutional neural networks: CNNs generally produce more discriminative features but require many labeled samples, and their fully connected outputs are usually 4096-D.These large feature vectors create computational and storage challenges for large-scale RSIR, motivating the more efficient LDCNN.
5. Experiments and Results
Experiments on PatternNet compare handcrafted, mid-level, unsupervised, pretrained CNN, and LDCNN features using standard retrieval metrics. Results favor Gabor texture among low-level features, VLAD among mid-level features, ResNet50 among CNNs, and compact LDCNN features overall.
- Experimental setup: Experiments evaluate handcrafted and deep learning feature-based methods for remote sensing image retrieval on PatternNet.ANMRR, mAP, P@k, and precision-recall curves are used, with each image serving as a query.
- Handcrafted low-level features: Gabor texture features achieve the best handcrafted low-level performance, while simple statistics perform worst and retrieval performance decreases as k increases.The low-level comparison includes ANMRR, mAP, and P@k for k=5, 10, 50, 100, and 1000.
- Handcrafted mid-level features: VLAD and IFK outperform BOVW, improving ANMRR by about 7% and 4%, respectively, while VLAD performs best for most classes.BOVW-128 is selected among BOVW configurations, but VLAD and IFK have higher dimensionality, increasing storage cost and reducing retrieval efficiency.
6. Conclusions · Reference
The paper presents PatternNet as the largest publicly available remotely sensed evaluation dataset constructed for remote sensing image retrieval (RSIR). It also surveys and evaluates traditional handcrafted and recent deep learning approaches on PatternNet to establish baseline results for future research.
- 6. Conclusions: PatternNet is presented as the largest publicly available remotely sensed evaluation dataset constructed for RSIR.
- 6. Conclusions: PatternNet is expected to advance the state-of-the-art in RSIR.
- 6. Conclusions: PatternNet is particularly intended to support deep learning based methods requiring large amounts of labeled training data.
- 6. Conclusions: The paper surveys traditional handcrafted features and recent deep learning features for RSIR.
- 6. Conclusions: The surveyed RSIR approaches are evaluated on PatternNet.
- 6. Conclusions: The evaluations establish baseline results to inform future research.