PaperlayerView exact revision on arXiv

Source-linked AI summary

Recurrently Exploring Class-wise Attention in A Hybrid Convolutional and Bidirectional LSTM Network for Multi-label Aerial Image Classification

Yuansheng Hua, Lichao Mou, Xiao Xiang Zhu

arXiv:1807.11245v2cs.CV

TL;DR

Aerial multi-label classification must account for multiple objects and their class dependencies, which existing independent-class approaches underexplore. The paper introduces CA-Conv-BiLSTM, an end-to-end network combining feature extraction, class attention, and bidirectional LSTM dependency modeling. Experiments on UCM and DFC15 validate the model quantitatively and qualitatively, including improved reported scores over comparison methods.

  • Problem

    Existing aerial-image methods often assume one label or model multiple classes independently, underexploring co-occurrence relationships among object classes.

  • Method

    CA-Conv-BiLSTM combines fine-grained feature extraction, class-specific attention learning, and bidirectional LSTM modeling to produce structured multiple object labels.

  • Results

    Experiments on UCM and DFC15 multi-label datasets validate CA-Conv-BiLSTM quantitatively and qualitatively, with reported improvements over comparison methods.

  • Takeaways & Limitations

    Modeling class dependencies bidirectionally within an end-to-end framework supports structured multi-label prediction for high-resolution aerial imagery.

Abstract

from arXiv · show

Aerial image classification is of great significance in remote sensing community, and many researches have been conducted over the past few years. Among these studies, most of them focus on categorizing an image into one semantic label, while in the real world, an aerial image is often associated with multiple labels, e.g., multiple object-level labels in our case. Besides, a comprehensive picture of present objects in a given high resolution aerial image can provide more in-depth understanding of the studied region. For these reasons, aerial image multi-label classification has been attracting increasing attention. However, one common limitation shared by existing methods in the community is that the co-occurrence relationship of various classes, so called class dependency, is underexplored and leads to an inconsiderate decision. In this paper, we propose a novel end-to-end network, namely class-wise attention-based convolutional and bidirectional LSTM network (CA-Conv-BiLSTM), for this task. The proposed network consists of three indispensable components: 1) a feature extraction module, 2) a class attention learning layer, and 3) a bidirectional LSTM-based sub-network. Particularly, the feature extraction module is designed for extracting fine-grained semantic feature maps, while the class attention learning layer aims at capturing discriminative class-specific features. As the most important part, the bidirectional LSTM-based sub-network models the underlying class dependency in both directions and produce structured multiple object labels. Experimental results on UCM multi-label dataset and DFC15 multi-label dataset validate the effectiveness of our model quantitatively and qualitatively.

1. Introduction

Aerial images commonly contain multiple object labels, but existing methods often treat classes independently and underuse their dependencies. The paper proposes an end-to-end architecture that combines feature extraction, class attention, and bidirectional dependency modeling, alongside a new DFC15 multi-label dataset.

  • Most aerial-image classification studies assign one label, although real images commonly contain multiple object labels and holistic object descriptions are useful.
  • Image-level multi-label annotations offer a lower-cost alternative to pixel-wise segmentation masks and bounding-box annotations.
  • The Challenges of Multi-label Classification: High-resolution imagery makes holistic feature extraction difficult, while existing multi-label methods often predict classes independently despite meaningful co-occurrences.
  • The Motivation of Our Work: The proposed CA-Conv-BiLSTM integrates feature extraction, class attention learning, and bidirectional LSTM modeling of high-order class dependencies in one end-to-end network.
  • The Motivation of Our Work: The method treats multi-label classification as structured output, making each label prediction dependent on others rather than framing labels as separate regression outputs.
  • The Motivation of Our Work: The study introduces DFC15 multi-label dataset, containing aerial images at 5 cm spatial resolution for evaluating multi-label classification networks.

2. Methodology

The methodology treats aerial-image multi-label classification as structured prediction, modeling class dependencies rather than predicting labels independently. CA-Conv-BiLSTM combines feature extraction, class-specific attention, and bidirectional recurrent modeling in an end-to-end architecture.

  • Class dependency: Existing methods commonly predict each class independently, despite strong and asymmetric co-occurrence relationships among aerial-image labels.Examples include ships with sea and buildings with pavement; P(Cp|Cr) captures directional class dependence.
  • Architecture: The proposed CA-Conv-BiLSTM comprises a feature extraction module, class attention learning layer, and bidirectional LSTM-based recurrent sub-network.These components extract image features, learn discriminative class-specific features, and model class dependencies, respectively.
  • Feature extraction: The feature extraction module adapts CNN architectures such as VGG-16 to produce high-level feature maps while preserving spatial resolution for later class-specific processing.The model discards the final two pooling layers and uses dilation rate 2 in the last convolutional block; GoogLeNet and ResNet variants are also evaluated.
  • Class attention learning: The class attention learning layer applies class-specific 1×1 convolutions to feature maps and vectorizes the resulting attention maps into class-wise feature vectors.Each attention map is a linear combination of feature-map channels and is connected to its corresponding recurrent unit.
  • Class attention learning: Class attention maps highlight discriminative regions for different categories and show almost no activation for absent classes.The paper illustrates these maps for bare soil, building, and water in an aerial image.
  • Class dependency learning: The bidirectional LSTM assigns time steps to classes and learns interclass relationships in both directions from class-specific feature vectors.Using two reversed recurrent streams addresses the bidirectional nature of class dependencies.

3. Experiments and Discussion

The experiments evaluate the proposed network on two high-resolution aerial multi-label datasets using quantitative and qualitative analyses. Figure 6 illustrates the UCM dataset’s scene categories and corresponding multiple object labels.

  • The proposed network is evaluated on two high-resolution aerial datasets with quantitative and qualitative analyses.The experimental section describes the datasets, training strategies, and subsequent evaluations.
  • UCM examples pair each scene category with its corresponding multiple object labels.The images are 256 × 256 pixels with one-foot spatial resolution.

3.1. Data description

The experiments use UCM and DFC15-derived multi-label datasets with different image sources, resolutions, and labeling procedures. UCM assigns object labels to cropped scene images, whereas DFC15 aggregates pixel-level labels after sliding-window cropping.

  • UCM multi-label dataset: UCM contains 2,100 256×256 aerial images across 21 scene categories, with 100 one-foot-resolution images per category.The images were manually cropped from USGS National Map aerial ortho imagery.
  • UCM multi-label dataset: UCM multi-label relabeling assigns each image one or more labels from 17 primitive object classes.Some labels, including airplane, building, and tank, occur at different semantic levels in the original and relabeled datasets.
  • DFC15 multi-label dataset: DFC15 multi-label is derived from seven 10,000 × 10,000-pixel tiles acquired over Zeebrugge at 5 cm spatial resolution.The source DFC15 data provide pixel-level labels for eight object classes.
  • DFC15 multi-label dataset: DFC15 tiles are cropped into 600 × 600-pixel images with a 200-pixel stride, excluding images containing unclassified pixels before aggregating pixel labels.The resulting image-level labels are formed from all pixel labels within each image.
  • DFC15 multi-label dataset: DFC15 images have lower inter-image similarity and greater difficulty than UCM images because they are cropped consecutively from vast regions at extremely high resolution.The dataset evaluation uses a randomly selected 80% training split, with the remainder used for evaluation.

3.2. Training details

Training uses pretrained CNN features, initialized class-attention filters, and randomly initialized bidirectional LSTM weights. The network is optimized with Adam using binary cross entropy and regularized with early stopping.

  • Initialization: The feature extraction module is initialized from ImageNet-pretrained CNNs and fine-tuned during training.Class-attention convolutional filters use Glorot uniform initialization.
  • Initialization: The bidirectional 2048-d LSTM weights are randomly initialized uniformly in [−0.1, 0.1].This initialization applies to all weights in the bidirectional LSTM layer.
  • Optimization: Training uses Adam with Nesterov momentum, β1 = 0.9, β2 = 0.999, ϵ = 1e −08, and an initial learning rate of 1e −04.The learning rate is decayed by 0.1 when validation accuracy saturates.
  • Optimization: The network uses binary cross entropy, trains for up to 100 epochs on one NVIDIA Tesla P100 16GB GPU, and uses batch size 32.Training stops when the loss fails to decrease for five epochs.
  • Label encoding: Multiple image labels are encoded as multi-hot binary sequences whose length equals the number of candidate labels.A 1 indicates label presence and 0 indicates label absence.

3.3. Results on UCM Multi-label Dataset

On UCM, the CA-Conv-LSTM framework improves over standard CNNs by modeling class dependencies, while bidirectional modeling further improves performance. Attention maps generally highlight discriminative regions for present classes, though visually similar categories remain difficult.

  • Quantitative Results: CA-Conv-LSTM improves mean F1 over standard CNN baselines by incorporating class dependencies.CA-GoogLeNet-LSTM reaches a mean F1 of 81.78%, while CA-GoogLeNet-LSTM obtains the best mean F2 of 85.16%.
  • Quantitative Results: CA-ResNet-LSTM raises mean F1 by 1.68% and mean F2 by 3.08% over ResNet.
  • Quantitative Results: CA-Conv-BiLSTM outperforms CA-Conv-LSTM in mean F1 and exceeds Conv-RBFNN by up to 0.98% in mean F1 and 2.80% in mean F2.Its higher example-based recall but lower example-based precision contributes to the relatively higher mean F2 score.
  • Qualitative Results: Grass and tree are frequently confused, while bare soil can be missed because these categories have similar visual patterns.
  • Qualitative Results: Class attention maps strongly activate discriminative regions for positive classes and show little activation when corresponding objects are absent.Examples include building, grass, pavement, and tree regions in UCM images.

3.4. Results on DFC15 Multi-label Dataset

On DFC15, bidirectional class-dependency modeling improves multi-label classification across the reported comparisons. Attention maps are class-specific overall, but similar visual patterns and border artifacts expose remaining weaknesses.

  • Quantitative Results: CA-ResNet-BiLSTM improves mean F1 by 4.87% over CA-ResNet-LSTM and 5.55% over ResNet.
  • Quantitative Results: CA-VGG-BiLSTM achieves a mean F1 of 76.25%, while CA-GoogLeNet-BiLSTM reaches 78.25%.
  • Quantitative Results: CA-Conv-BiLSTM improves over Conv-RBFNN by up to 5.29% in mean F1 and 4.18% in mean F2.
  • Qualitative Results: Attention maps highlight impervious regions and cars, supporting the presence of discriminative class-specific features.
  • Qualitative Results: Some negative labels produce strong activations, and border artifacts remain prominent in several attention maps.A skip-connection scheme for refining the maps provides negligible improvements.

4. Conclusion

The paper concludes that CA-Conv-BiLSTM combines feature extraction, class attention, and bidirectional sequence modeling for structured aerial-image multi-label prediction. Results on UCM and DFC15 support its quantitative and qualitative effectiveness.

  • CA-Conv-BiLSTM extracts fine-grained features, learns class-specific attention, and models class dependency in both directions.
  • The network predicts multiple object labels as ordered, structured sequences rather than discrete independent outputs.
  • Experiments on UCM and DFC15 validate the model quantitatively and qualitatively, with mean F2 increased by at most 0.0446 over competitors.
Loading 1807.11245v2…