PaperlayerView exact revision on arXiv

Source-linked AI summary

BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval

Gencer Sumbul, Arne de Wall, Tristan Kreuziger, Filipe Marcelino, Hugo Costa, Pedro Benevides, Mário Caetano, Begüm Demir, Volker Markl

arXiv:2105.07921v2cs.CV

TL;DR

Existing remote-sensing benchmarks often lack large-scale multi-modal images with multi-label annotations, while some land-cover classes are difficult to describe from single-date imagery. The paper builds BigEarthNet-MM with paired Sentinel-1/Sentinel-2 patches and a revised 19-class nomenclature, finding that direct training generally outperforms ImageNet transfer learning and that ResNet outperforms VGG in classification.

  • Problem

    Existing remote-sensing benchmarks largely provide single-modal images or single-label annotations, and some CLC classes are difficult to identify from single-date imagery.

    Source: §I · pp. 1–2

  • Method

    The paper constructs BigEarthNet-MM from 590,326 paired Sentinel-1 and Sentinel-2 patches, assigns CLC-based multi-labels, and introduces a revised 19-class nomenclature.

    Source: §I · p. 2, §II · p. 2

  • Results

    Direct learning from BigEarthNet-MM achieves the highest class-based F2 score for every class versus ImageNet transfer learning, while ResNet achieves the highest scores across evaluated metrics versus VGG.

    Source: 1) Comparison among the Strategies of… · p. 4, 2) Comparison of State-of-the-Art CNN… · p. 4

  • Takeaways & Limitations

    BigEarthNet-MM provides a public resource for multi-modal multi-label remote-sensing image classification, retrieval, and further learning studies.

    Source: §IV · pp. 4–5

    BigEarthNet-MM is limited for applications requiring time-series data, and some Sentinel-1 patches may contain radio-frequency-interference or other dataset-related artefacts.

    Source: §IV · p. 5

Abstract

from arXiv · show

This paper presents the multi-modal BigEarthNet (BigEarthNet-MM) benchmark archive made up of 590,326 pairs of Sentinel-1 and Sentinel-2 image patches to support the deep learning (DL) studies in multi-modal multi-label remote sensing (RS) image retrieval and classification. Each pair of patches in BigEarthNet-MM is annotated with multi-labels provided by the CORINE Land Cover (CLC) map of 2018 based on its thematically most detailed Level-3 class nomenclature. Our initial research demonstrates that some CLC classes are challenging to be accurately described by only considering (single-date) BigEarthNet-MM images. In this paper, we also introduce an alternative class-nomenclature as an evolution of the original CLC labels to address this problem. This is achieved by interpreting and arranging the CLC Level-3 nomenclature based on the properties of BigEarthNet-MM images in a new nomenclature of 19 classes. In our experiments, we show the potential of BigEarthNet-MM for multi-modal multi-label image retrieval and classification problems by considering several state-of-the-art DL models. We also demonstrate that the DL models trained from scratch on BigEarthNet-MM outperform those pre-trained on ImageNet, especially in relation to some complex classes, including agriculture and other vegetated and natural environments. We make all the data and the DL models publicly available at https://bigearth.net, offering an important resource to support studies on multi-modal image scene classification and retrieval problems in RS.

I. INTRODUCTION

Remote sensing archives need accurate classification and retrieval, but existing benchmarks largely provide single-modal images with single-label annotations. BigEarthNet-MM addresses this gap with large-scale multi-modal, multi-label image pairs and an alternative nomenclature for difficult classes.

  • Remote sensing archives are growing, increasing demand for accurate scene classification and content-based image retrieval.Source: §I · p. 1
  • Most existing public benchmarks for remote sensing scene classification and retrieval contain single-modal images and single-label annotations.Source: §I · p. 1
  • Multi-modal, multi-label benchmarks are needed, but large-scale annotation is time-consuming, complex, and costly.Source: §I · p. 1
  • ImageNet transfer learning may be poorly suited to remote sensing because image characteristics and semantic content differ between the domains.Source: §I · p. 1
  • BigEarthNet-MM introduces 590,326 Sentinel-1 and Sentinel-2 image-patch pairs plus an alternative nomenclature evolving the original CLC labels.Source: §I · p. 2

II. DESCRIPTION OF BIGEARTHNET-MM

BigEarthNet-MM pairs Sentinel-2 and Sentinel-1 patches from closely matched European acquisitions. The archive is built from carefully selected and sensor-specific pre-processing pipelines.

  • 590,326 Sentinel-1 and Sentinel-2 patch pairs cover 10 European countries and derive from 125 low-cloud Sentinel-2 tiles acquired between June 2017 and May 2018.Source: §II · p. 2
  • Sentinel-2 tiles were atmospherically corrected with ESA’s sen2cor before being divided into non-overlapping patches at 10m, 20m, and 60m resolutions.Source: §II · p. 2
  • Sentinel-1 patches were constructed from 325 closely timed GRD products jointly covering the original Sentinel-2 tile areas.Source: §II · p. 2
  • Sentinel-1 processing included orbit correction, noise removal, radiometric calibration, terrain correction, and conversion of backscatter to decibels.Source: §II · p. 2

A. Class-Nomenclature of BigEartNet-MM

BigEarthNet-MM assigns CLC 2018 multi-labels to paired images and reorganizes the detailed CLC hierarchy into a 19-class nomenclature tailored to the archive’s image properties.

A. Experimental Design

The experiments evaluate multi-modal, multi-label retrieval and classification using stacked Sentinel-1 and Sentinel-2 bands, multiple metrics, and comparisons between learning strategies and CNN architectures.

Experimental Design

  • Sentinel-1 VV and VH bands and Sentinel-2 10m and 20m bands were stacked into one input volume for each image pair.Source: A. Experimental Design · p. 3
  • The classification evaluation uses Hamming loss, one-error, recall, and F2-Score.Source: A. Experimental Design · p. 3
  • Table I compares class-based F2 scores for ImageNet transfer learning with direct learning from BigEarthNet-MM in multi-modal multi-label classification.Source: Table I · p. 4

1) Comparison among the Strategies of Learning directly from BigEarthNet-MM and Transfer Learning from the ImageNet:

Direct learning from BigEarthNet-MM outperforms ImageNet transfer learning for classification and retrieves semantically more similar image pairs. The advantage is especially pronounced for complex land-use and land-cover classes.

2) Comparison of State-of-the-Art CNN Models:

ResNet models outperform VGG models in multimodal multi-label classification, while increasing depth within a model family has little effect on performance.

IV. DISCUSSION AND CONCLUSION

BigEarthNet-MM provides a large multimodal benchmark with a new CLC-based nomenclature for remote-sensing classification and retrieval research. It supports several learning settings but remains constrained for applications requiring time-series information and may contain Sentinel-1 artefacts.

  • BigEarthNet-MM contains 590,326 Sentinel-1 and Sentinel-2 image-patch pairs with a new CLC-based class nomenclature.Source: §IV · p. 4
  • The archive supports research on class-imbalanced multimodal data, transfer learning, and unsupervised, self-supervised, and semi-supervised learning.Source: §IV · p. 5
  • BigEarthNet-MM is limited for applications requiring time-series data to accurately describe classes such as Non-irrigated arable land and Permanently irrigated land.Source: §IV · p. 5
  • Some Sentinel-1 patches may contain artefacts from Radio-Frequency-Interference or other dataset-related issues independent of preprocessing.Source: §IV · p. 5

Related papers

Loading 2105.07921v2…