PaperlayerView exact revision on arXiv

Source-linked AI summary

Relation Network for Multi-label Aerial Image Classification

Yuansheng Hua, Lichao Mou, Xiao Xiang Zhu

arXiv:1907.07274v3cs.CV

TL;DR

Aerial multi-label classification needs to capture high-level image semantics and label correlations, which prior approaches have addressed only to a limited extent. The paper proposes a network that learns label-specific features, localizes discriminative regions, and reasons over label relations. The authors report the best mean F1 and F2 scores across variant backbones on both UCM and AID multi-label datasets.

Abstract

from arXiv · show

Multi-label classification plays a momentous role in perceiving intricate contents of an aerial image and triggers several related studies over the last years. However, most of them deploy few efforts in exploiting label relations, while such dependencies are crucial for making accurate predictions. Although an LSTM layer can be introduced to modeling such label dependencies in a chain propagation manner, the efficiency might be questioned when certain labels are improperly inferred. To address this, we propose a novel aerial image multi-label classification network, attention-aware label relational reasoning network. Particularly, our network consists of three elemental modules: 1) a label-wise feature parcel learning module, 2) an attentional region extraction module, and 3) a label relational inference module. To be more specific, the label-wise feature parcel learning module is designed for extracting high-level label-specific features. The attentional region extraction module aims at localizing discriminative regions in these features and yielding attentional label-specific features. The label relational inference module finally predicts label existences using label relations reasoned from outputs of the previous module. The proposed network is characterized by its capacities of extracting discriminative label-wise features in a proposal-free way and reasoning about label relations naturally and interpretably. In our experiments, we evaluate the proposed model on the UCM multi-label dataset and a newly produced dataset, AID multi-label dataset. Quantitative and qualitative results on these two datasets demonstrate the effectiveness of our model. To facilitate progress in the multi-label aerial image classification, the AID multi-label dataset will be made publicly available.

I. INTRODUCTION

Aerial image labels can describe either a broad scene or its multiple objects; image-level multi-label annotation offers a comprehensive view without the expense of pixel-wise or bounding-box labels. The paper addresses the challenge of extracting high-level features and using label correlations, which prior work has often left underexplored.

I. INTRODUCTION

  • Multi-label image classification provides a more comprehensive picture of aerial-image contents using only image-level labels, unlike pixel-wise or bounding-box annotation.The latter require substantial expertise and labor, making their datasets expensive and difficult to access.Source: §I · p. 1
  • Images sharing a scene label can contain different object labels, and objects such as buildings and pavement can provide cues about nearby human activity.The cited examples contrast rivers near human activity with rivers more likely in the wild.Source: §I · pp. 1–2

A. Challenges of Identifying Multiple labels

  • High-resolution aerial images pose a feature-extraction challenge because their spatial context is complicated, while handcrafted methods may fail to extract high-level semantics.The passage identifies an efficient high-level feature extractor as desirable.Source: A. Challenges of Identifying Multiple… · p. 2
  • Label correlations are described as essential but sparsely addressed; RNN approaches model dependencies through chains and implicitly, with performance tied to long-term memorization.Ships can suggest water or sea, while cars can suggest pavements.Source: A. Challenges of Identifying Multiple… · p. 2

B. Related Work

  • Prior work includes methods using spatial relations between patches, CNN-based multi-label prediction, data augmentation, CNN-RNN label dependencies, and prior knowledge such as word embeddings or knowledge graphs.The cited examples include spatial-structure SVMs and CRFs, as well as recurrent and knowledge-based approaches.Source: B. Related Work · p. 2

C. The Motivation of Our Work

The paper proposes an end-to-end aerial-image multi-label network that localizes label-specific discriminative regions and explicitly models semantic label dependencies. It also introduces a relabeled AID multi-label dataset described as more challenging than UCM because of its image resolutions, scene diversity, and sample count.

C. The Motivation of Our Work

  • The attention-aware label relational reasoning network combines label-wise feature parcels, attentional region extraction, and label relational inference.The paper presents relation networks as a way to reason about an entity in relation to all other entities, and uses a spatial transformer to reduce irrelevant semantic features.Source: C. The Motivation of Our Work · p. 2
  • A learnable spatial transformer extracts and re-coordinates attentional regions from label-wise feature parcels in a proposal-free fashion.The regions are assumed to contain discriminative information and yield attentional feature parcels.Source: C. The Motivation of Our Work · p. 2
  • The paper produces the AID multi-label dataset by relabeling AID images; it is described as more challenging than UCM because it has more varied spatial resolutions, scenes, and samples.The stated comparison is with the UCM multi-label dataset.Source: C. The Motivation of Our Work · pp. 2–3

A. Network Architecture

The network extracts a feature parcel for each object label, localizes discriminative regions to form attentional parcels, and reasons over relations among those parcels to predict label presence. Its feature-parcel module uses a CNN backbone and learns K feature maps specific to each label.

A. Network Architecture

  • For each label, the network extracts a high-level feature map X_l, forms an attentional feature parcel A_l from discriminative regions, then reasons over relations among parcels to predict object presence.The architecture has label-wise feature parcel learning, attentional region extraction, and label relational inference modules.Source: A. Network Architecture · p. 3

B. Label-wise Feature Parcel Learning

  • A CNN backbone feeds a 1 × 1 convolutional layer with KL filters; its output is divided into L label-specific parcels, each containing K feature maps.The feature maps are learned end-to-end to extract discriminative semantics for each label.Source: B. Label-wise Feature Parcel Learning · p. 3
  • Higher-resolution X_l is described as beneficial for localizing discriminative regions, so the VGG-16 implementation discards its final max-pooling layer to retain a 14 × 14 output.The VGG-16 weights are initialized from ImageNet pretraining and updated during training.Source: B. Label-wise Feature Parcel Learning · pp. 3–4

C. Attentional Region Extraction Module

The attentional region extraction module selects discriminative, label-specific regions from feature parcels to reduce the influence of less informative regions. A learnable spatial transformation localizes these regions and produces attentional feature parcels.

C. Attentional Region Extraction Module

  • The module extracts discriminative regions from label-wise feature parcels to reduce noise from less informative regions.Weakly activated regions have looser relevance to their corresponding labels than highlighted regions.Source: C. Attentional Region Extraction Module · p. 4
  • A learnable spatial transformation maps each feature parcel’s grid to an attentional grid with the same number of points, using scaling and translation.The grid coordinates are normalized to [−1, 1], and the transformation is learned separately for each object label.Source: C. Attentional Region Extraction Module · p. 4
  • Because the transformation is label-specific, the extracted attentional regions can reflect how different objects are distributed in aerial images.The paper describes the resulting attentional regions as capable of improving the effectiveness of label-wise features.Source: C. Attentional Region Extraction Module · p. 4
  • The implementation predicts the transformation from a vectorized feature parcel, samples the attentional grid by bilinear interpolation, and re-coordinates it into a regular grid.A localization layer estimates the transformation matrix; the final attentional parcel has the same spatial structure as the original feature parcel.Source: C. Attentional Region Extraction Module · p. 4

D. Label Relational Inference Module

The label relational inference module predicts label presence by reasoning over pairwise relations between attentional feature parcels. It accumulates relations with a summation-based design rather than an LSTM chain, and learns relations from data without requiring prior relation knowledge.

D. Label Relational Inference Module

  • Pairwise relations take two attentional feature parcels as input; a 1 × 1 convolution models their relation, and pooling plus an MLP produces the pairwise relation.The convolution is used instead of an MLP for relation reasoning to explore spatial information.Source: D. Label Relational Inference Module · p. 4
  • For each label, the module combines its pairwise relations with all other labels by summing them before an MLP predicts label presence.A global average pooling layer transforms each pairwise output into a vector before element-wise addition.Source: D. Label Relational Inference Module · p. 5
  • The network learns label relations from data without requiring prior knowledge of which object relations to compute.The authors report that learned relations align with prior knowledge in their experiments.Source: D. Label Relational Inference Module · p. 5
  • Summation replaces an LSTM-style chain, which the authors say increases robustness by avoiding dependence on long short-term memory learning effectiveness.Relations are learned separately for each object-label pair, and the authors contrast this explicit design with the implicit RNN approach in.Source: D. Label Relational Inference Module · p. 5

III. EXPERIMENTS AND DISCUSSION

The experiments evaluate the model on the UCM and proposed AID multi-label datasets. The UCM multi-label dataset assigns object labels to 2,100 aerial images and uses an 80/20 train-test split within each scene class.

III. EXPERIMENTS AND DISCUSSION

  • The experiments evaluate the model on the UCM and proposed AID multi-label datasets.The section introduces dataset descriptions before discussing training strategies and experimental results.Source: §III · p. 5

1) UCM multi-label dataset:

  • The UCM multi-label dataset assigns 17 candidate object labels to aerial images collected in the UCM dataset.Some terms, including building, tank, and airplane, are scene-level labels in but refer to objects in.Source: 1) UCM multi-label dataset · pp. 5–6
  • Fig. 5 shows UCM scene-category examples alongside their associated object labels.The examples include scenes such as tennis court, harbor, and agricultural, each paired with object labels.Source: Figure 5 · p. 6
  • UCM contains 2,100 manually cropped images at one-foot resolution and 256 × 256 pixels, with 80% of samples per scene class used for training and 20% for testing.Table I lists image counts by object category and the numbers assigned to training and test sets.Source: 1) UCM multi-label dataset · p. 6

2) AID multi-label dataset:

The AID multi-label dataset was created by assigning object labels to selected AID images, and the section also specifies how the network was initialized and trained.

2) AID multi-label dataset:

  • AID images are 600 × 600 pixels with spatial resolutions from 0.5 to 8 m/pixel, and the dataset is described as more challenging than UCM.AID has 10,000 images collected from worldwide Google Earth imagery; scene categories contain 220–420 images each.Source: 2) AID multi-label dataset · pp. 6–7
  • The examples span varied scenes and object-label combinations, including desert labeled with sand and river labeled with multiple objects.The figure shows samples from scene categories and their associated object labels.Source: Figure 6 · p. 7
  • The AID multi-label dataset contains 3,000 manually relabeled images from 30 scenes, with 80% used for training and the remainder for testing.The passage says images were selected and labeled through extensive human visual inspection.Source: 2) AID multi-label dataset · p. 7

B. Training Details

  • The network uses ImageNet-pretrained backbone weights, Glorot uniform initialization for other convolutional and relational-module weights, and an identity transformation for the attention module.The full network is trained end to end, including backbone fine-tuning.Source: B. Training Details · p. 7
  • Training uses multi-hot labels and binary cross-entropy with Adam and Nesterov momentum; 10% of training samples form the validation set.The 17-label sequence marks absent objects with 0 and present objects with 1; training uses 90% of the training samples for parameter learning.Source: B. Training Details · p. 8
  • Training runs for 100 epochs with batches of 32 and stops when validation loss rises continuously for five epochs.The reported implementation uses TensorFlow-1.12.0 and an NVIDIA Tesla P100 GPU with 16GB memory.Source: B. Training Details · p. 8

C. Experimental Setup

The experiments compare AL-RN-CNN against CNN and BiLSTM-based competitors using multiple backbones and classification metrics. On UCM, the ResNet variant achieves the strongest reported F1 and F2 scores, and cross-validation indicates limited score variance.

C. Experimental Setup

  • The proposed network is also evaluated with GoogLeNet and ResNet backbones adapted to retain higher spatial resolution.The setup removes global pooling and fully connected layers from these backbone variants.Source: C. Experimental Setup · p. 8
  • The comparison includes a standard CNN, CNN-RBFNN, and CA-CNN-BiLSTM, with output sequences thresholded at 0.5 for final predictions.The CNN baseline replaces its single-label softmax layer with a sigmoid layer.Source: C. Experimental Setup · p. 8

1) Quantitative analysis:

  • Evaluation uses mean F1 and F2 as principal indexes, alongside example-based and label-based precision and recall.The evaluation also includes example-based true-positive, false-positive, and false-negative counts.Source: 1) Quantitative analysis · p. 8
  • AL-RN-VGGNet improves mean F1 and F2 by 7.16% and 5.64% versus VGGNet, and mean F1 by 5.92% versus CA-VGG-BiLSTM.AL-RN-GoogLeNet improves mean F1 by 4.56% versus GoogLeNet and 3.42% versus CA-GoogLeNet-BiLSTM.Source: 1) Quantitative analysis · pp. 8–9
  • AL-RN-ResNet achieves the best UCM mean F1 score, 0.8676, and F2 score, 0.8667, among the compared models.The table reports classification comparisons on the UCM multi-label dataset.Source: Table III · p. 9, 1) Quantitative analysis · p. 9
  • Five-fold cross-validation of AL-RN-ResNet yields variances of 0.38% for mean F1 and 0.71% for mean F2.The authors characterize these variances as limited compared with improvements brought by the network.Source: 1) Quantitative analysis · p. 11

2) Qualitative analysis:

Qualitative visualizations show label-specific activations, attentional regions, and pairwise label relations. The examples indicate that regions cover objects of interest and that learned label relations can be directional.

2) Qualitative analysis:

  • Feature parcels highlight discriminative regions for positive labels, though one baseball field is misidentified as tanks and may lead to incorrect predictions.The displayed feature maps use red for strong activations and blue for weak activations.Source: 2) Qualitative analysis · p. 12
  • Most learned attentional regions cover objects of interest, including dispersed buildings and cars in the examples.The figure presents regions for positive labels across example scenes.Source: 2) Qualitative analysis · p. 12, Figure 11 · p. 12
  • Pairwise relations show car–pavement links in both prediction directions, while tree strongly suggests bare soil but not vice versa.The relation visualizations are row-normalized, with white indicating null values.Source: 2) Qualitative analysis · p. 12, Figure 12 · p. 12

1) Quantitative analysis:

On the AID multi-label dataset, AL-RN-CNN outperforms competitors on most metrics across backbones. The results include higher mean F1 and F2 scores and a large gain in mean label-based recall.

1) Quantitative analysis:

  • AL-RN-GoogLeNet achieves mean F1 0.8817 and mean F2 0.8825, the highest scores compared with GoogLeNet and CA-GoogLeNet-BiLSTM.The passage also reports that AL-RN-ResNet surpasses the second-best model by 1.09% in mean F1 and 0.51% in mean F2.Source: 1) Quantitative analysis · pp. 12–13
  • AL-RN-VGGNet improves mean F1 by 2.57% and mean F2 by 2.71% over the baseline model on AID.Against CA-VGG-BiLSTM, it gains 1.41% in mean F1 and 1.43% in mean F2.Source: 1) Quantitative analysis · p. 12
  • AL-RN-CNNs outperform baseline CNNs by up to 18.30% in mean label-based recall on AID.The authors conclude that explicitly modeling label relations can improve robustness and retrieval ability.Source: 1) Quantitative analysis · p. 13

2) Qualitative analysis:

Visualizations show label-specific features and attentional regions highlighting semantic content, alongside pairwise label relations that contribute to predictions. The relational inference module also outperforms LR-CNN variants, and the paper concludes with a three-module network evaluated on UCM and AID.

2) Qualitative analysis:

  • Feature maps highlight label-related semantics, while attentional regions capture objects such as cars and trees; AID attentional regions are coarser than UCM's because AID has lower spatial resolution.Ponds are highlighted for water recognition, while residential and industrial areas activate building feature maps.Source: 2) Qualitative analysis · p. 13
  • Tree and pavement contribute significantly to identifying car, while car suggests a high probability of pavement; building relations can assist prediction of car, pavement, and tree.Source: 2) Qualitative analysis · p. 13

F. Discussion on the Relational Inference Module

  • AL-RN-VGGNet improves mean F1 by 3.59% and mean F2 by 3.82% over LR-VGGNet on UCM; AL-RN-GoogLeNet improves mean F1 by 3.25% and mean F2 by 1.28% over LR-ResNet on AID.Across variant backbones, the network achieves the best mean F1 and F2 on both datasets; changing convolutional filter size lets gθlm encode label relations through different fields of view.Source: F. Discussion on the Relational… · p. 13

IV. CONCLUSION

  • The proposed network combines label-wise feature parcels, attentional region extraction, and pairwise label-relation inference, and is evaluated on UCM and AID.The conclusion reports better classification results than other deep learning methods and identifies weakly supervised object detection and semantic segmentation as potential future applications.Source: §IV · p. 13
Loading 1907.07274v3…