PaperlayerView exact revision on arXiv

Source-linked AI summary

AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification

Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang

arXiv:1608.05167v1cs.CV

TL;DR

Aerial scene classification needs stronger evidence because existing datasets are small and saturated, limiting evaluation of new methods. This paper reviews prior approaches, constructs and evaluates the AID benchmark, and reports that high-level methods outperform low-level methods across datasets while AID is more challenging due to greater intra-class variation and smaller inter-class dissimilarity.

  • Problem

    Existing aerial scene classification datasets are small and saturated, while inconsistent evaluation settings make the state of the art unclear.

  • Method

    The paper reviews aerial scene classification methods, constructs the large-scale AID dataset, and evaluates representative approaches under varied protocols.

  • Results

    High-level methods perform better than low-level methods on all evaluated datasets, while AID is more challenging because of higher intra-class variation and smaller inter-class dissimilarity.

  • Takeaways & Limitations

    AID provides a benchmark resource and baseline results intended to advance aerial scene classification research.

  • Takeaways & Limitations

    Low-level methods have difficulty depicting high-diversity, non-homogeneous aerial scenes, and mid-level methods may lack flexibility because they rely on handcrafted descriptors.

Abstract

from arXiv · show

Aerial scene classification, which aims to automatically label an aerial image with a specific semantic category, is a fundamental problem for understanding high-resolution remote sensing imagery. In recent years, it has become an active task in remote sensing area and numerous algorithms have been proposed for this task, including many machine learning and data-driven approaches. However, the existing datasets for aerial scene classification like UC-Merced dataset and WHU-RS19 are with relatively small sizes, and the results on them are already saturated. This largely limits the development of scene classification algorithms. This paper describes the Aerial Image Dataset (AID): a large-scale dataset for aerial scene classification. The goal of AID is to advance the state-of-the-arts in scene classification of remote sensing images. For creating AID, we collect and annotate more than ten thousands aerial scene images. In addition, a comprehensive review of the existing aerial scene classification techniques as well as recent widely-used deep learning methods is given. Finally, we provide a performance analysis of typical aerial scene classification and deep learning approaches on AID, which can be served as the baseline results on this benchmark.

1 Introduction

Aerial scene classification addresses semantic understanding of complex high-resolution aerial imagery, but progress is constrained by unclear comparisons and small, saturated datasets. The paper responds with a review, the AID benchmark, baseline evaluations, and released resources.

  • Problem: Aerial scene classification assigns aerial images semantic categories by modeling spatial arrangements and structural patterns.It provides a higher-level interpretation than pixel- or object-oriented classification.
  • Challenges: Existing progress is difficult to assess because methods use different datasets and experimental settings, while unreleased code impedes fair reproduction.The paper states that the field’s state of the art is therefore not absolutely clear.
  • Challenges: Existing benchmark datasets contain at most about two-thousand images and inadequately represent high intra-class diversity and complex spatial patterns.The paper identifies UC-Merced and WHU-RS19 as examples of these limited datasets.
  • Contributions: The paper reviews existing aerial scene classification approaches and evaluates representative methods under varied protocols on AID.The resulting evaluations are intended to serve as baseline results for future work.
  • Contributions: AID is introduced as a new large-scale aerial-image dataset with high intra-class diversity and low inter-class dissimilarity.The authors describe it as, to their knowledge, the largest dataset and a resource for evaluating and advancing aerial image analysis.
  • Contributions: The dataset and implementation codes are made publicly downloadable to support reproduction and further research.The paper identifies the released codes as general tools for other researchers.

2 A review on aerial scene classification

The review organizes aerial scene classification from handcrafted low-level and mid-level representations to adaptive deep-learning methods. It highlights both the strengths of holistic representations and the limitations imposed by handcrafted features and small datasets.

  • Overview: Scene classification models aerial images holistically, assuming that scenes of the same type share statistically holistic visual characteristics.This distinguishes it from bottom-up pixel- and object-level classification.
  • Low-level methods: Low-level methods use spectral, texture, and structural descriptors, but struggle with high-diversity and non-homogeneous spatial distributions.They perform well mainly on scenes with uniform structures and spatial arrangements.
  • Mid-level methods: Mid-level methods encode local image attributes into holistic representations, including BoVW histograms that count visual-word occurrences.Feature concatenation and spatial-relation modeling extend the basic BoVW framework.
  • Mid-level methods: Topic models add higher-order semantic relationships among visual words, while unsupervised feature learning seeks more adaptive representations.Examples include LDA, sparse coding, manifold learning, and sparse auto-encoders.
  • Deep learning methods: Deep learning methods learn global features through multi-stage architectures and often formulate aerial scene classification end to end.They can use pretrained networks as global feature extractors, local feature extractors, or newly trained networks.
  • Deep learning methods: Training large deep networks on small aerial datasets can reduce accuracy because millions of parameters make overfitting and local minima more likely.The review motivates adapting network size or training strategy to dataset scale.

3 Aerial Image Datasets (AID) for aerial scene classification

AID is introduced as a large-scale aerial scene dataset designed to address the limited size, saturation, and insufficient challenge of existing benchmarks. It contains 10,000 images across 30 classes, with diverse sources, regions, resolutions, and challenging visual variation.

  • Motivation: Existing benchmarks are limited by small sample sizes, few scene types, and near-saturated accuracies, obscuring reliable state-of-the-art comparisons.The paper also notes that differing experimental settings and unavailable code hinder fair reproduction and comparison.
  • AID construction: AID combines imagery from multiple sensors, countries, regions, times, seasons, and resolutions.Pixel resolution ranges from about 8 meters to about half a meter, with images fixed at 600 × 600 pixels.
  • Dataset properties: AID targets higher intra-class variation through changes in object scale, orientation, imaging conditions, and scene appearance.Figure 2 illustrates multi-scale imagery, different building styles, and different imaging conditions within the same scene type.
  • Dataset properties: AID targets smaller inter-class dissimilarity by including scenes that share objects, textures, or structural distributions.Examples include stadium versus playground, bare land versus desert, and resort versus park.
  • AID construction: AID contains 10,000 aerial images spanning 30 semantic scene classes.The dataset is annotated by remote-sensing interpretation specialists.

4 Baseline methods

The paper reviews and benchmarks low-, mid-, and high-level aerial scene classification methods. These methods range from hand-crafted patch or scene descriptors to coded local features and deep CNN representations.

  • Methods with low-level scene features: Low-level methods describe aerial images using spectral, texture, structural, or spatial-envelope features such as SIFT, LBP, color histograms, and GIST.The benchmark evaluates four representative low-level methods.
  • Methods with mid-level scene features: Mid-level methods encode local descriptors into scene representations using BoVW, SPM, LLC, pLSA, LDA, IFK, or VLAD.The evaluation combines three local descriptors with seven feature-coding approaches.
  • Methods with mid-level scene features: BoVW represents images by visual-word frequencies, while spatial or locality-aware variants retain region structure or constrain coding to nearby dictionary elements.SPM uses increasingly coarse grids, whereas LLC applies locality constraints and max pooling.
  • Methods with mid-level scene features: Topic and Fisher-based approaches alter feature representations through latent topics, Dirichlet priors, or gradients of Gaussian-mixture likelihoods.LDA adds a Dirichlet prior, while IFK produces a gradient-based feature vector.
  • Methods with high-level scene features: High-level methods use learned CNN features, with CaffeNet, VGG-VD-16, and GoogLeNet selected as representative deep-learning baselines.GoogLeNet uses multi-scale Inception modules and has a 22-layer architecture with one final fully connected layer.

5 Experimental studies

The experimental study evaluates representative low-, mid-, and high-level scene classification methods as baselines on AID.

  • Baseline evaluation: The benchmark includes four low-level methods, seven mid-level coding methods, and three high-level deep-learned methods.The selected baselines cover SIFT, LBP, CH, GIST; BoVW, SPM, LLC, pLSA, LDA, IFK, VLAD; and CNN models.

5.1 Parameter Settings

The parameter settings define feature extraction for low-, mid-, and high-level methods, then use a linear classifier to estimate performance on held-out images.

  • Low- and mid-level settings: Low-level extraction uses grid-sampled patches and fixed descriptor settings, including a 4 × 8 × 4 × 4, 512-dimensional GIST representation.SIFT uses 16 × 16-pixel patches with an 8-pixel spacing step; GIST uses four scales, eight orientations, and a 4 × 4 grid.
  • Low- and mid-level settings: Mid-level features combine BoVW, SPM, LLC, pLSA, LDA, IFK, or VLAD with SIFT, LBP, and color-histogram local descriptors.Grid sampling is used to obtain local patches.
  • High-level settings: High-level features are extracted from pretrained ILSVRC 2012 CNNs and L2 normalized before classification.CaffeNet and VGG-VD-16 produce 4096-dimensional vectors, while GoogLeNet produces a 1024-dimensional vector.
  • Classification protocol: A liblinear supervised classifier is trained on training-set features and evaluated using testing-set features.The linear classifier is chosen because it can quickly train on large-scale datasets.

5.2 Evaluation protocols

The evaluation uses overall accuracy and confusion matrices, with dataset-specific training-set ratios and fixed image splits for comparison.

  • Overall accuracy measures correctly predicted images divided by total predicted images, providing whole-dataset classification performance.
  • Confusion matrices visualize class-level performance, with columns representing predicted classes and rows representing actual classes.
  • Training-set ratios vary by dataset: RSSCN7 and AID use 20% and 50%, UC-Merced uses 50% and 80%, and WHU-RS19 uses 40% and 60%.
  • For confusion matrices, fixed training images are used with training ratios of 50% for UC-Merced, 40% for WHU-RS19, and 20% for RSSCN7 and AID.

5.3 Experimental results

Experiments compare low-, mid-, and high-level features across four aerial-scene datasets using overall accuracy and confusion matrices. Results favor high-level features overall, while AID provides more precise evaluation and contains challenging fine-grained scenes.

  • Results with low-level methods: SIFT performs about 20% lower OA than the best low-level features across datasets, while GIST leads UC-Merced and CH leads WHU-RS19 and AID.
  • Results with mid-level methods: Dictionary size strongly affects coding results: BoVW improves with larger dictionaries, whereas IFK and VLAD are relatively insensitive and LDA, pLSA, and SPM eventually decline.
  • Results with mid-level methods: IFK with SIFT performs best among 21 mid-level features across all datasets, combining SIFT robustness and invariance with IFK’s generative and discriminative properties.
  • Results with high-level methods: High-level methods outperform low- and mid-level methods on both datasets, indicating stronger discriminative features and generalization from natural-image pretraining.
  • Results with high-level methods: AID has more than ten times as many testing images as UC-Merced and WHU-RS19, producing much smaller OA standard deviations and more precise evaluation.
  • Confusion matrix: High-level confusion matrices are cleaner than low- and mid-level matrices across WHU-RS19 and RSSCN7, reflecting improved class separation.
  • Confusion matrix: AID is more suitable for aerial scene classification than the other datasets because it contains fine-grained and challenging scene types.

5.4 Discussion

AID is more challenging than other aerial scene classification datasets, while high-level methods perform better across datasets. Its larger sample count also enables more precise method evaluation.

  • High-level scene classification methods outperform low-level methods across all evaluated datasets.
  • These findings motivate investigating more effective high-level methods on challenging datasets.
  • AID is more challenging than other datasets because it has higher intra-class variation and smaller inter-class dissimilarity.
  • AID's larger number of sample images supports more precise evaluation of classification methods.

6 Conclusion

The paper reviews aerial scene classification and introduces AID to address saturated existing datasets. It evaluates representative methods under varied protocols and provides public data, code, and baseline results for future work.

  • Existing popular aerial scene classification datasets have saturated results, limiting further progress.
  • AID is introduced as a large-scale benchmark intended to advance aerial scene classification algorithms.
  • Representative classification approaches are evaluated on AID using various experimental protocols to establish baseline results.
  • The dataset and code are publicly available for downloading to promote development of aerial scene classification.
Loading 1608.05167v1…