Source-linked AI summary
BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval
Gencer Sumbul, Arne de Wall, Tristan Kreuziger, Filipe Marcelino, Hugo Costa, Pedro Benevides, Mário Caetano, Begüm Demir, Volker Markl
TL;DR
Existing remote-sensing benchmarks often lack large-scale multi-modal images with multi-label annotations, while some land-cover classes are difficult to describe from single-date imagery. The paper builds BigEarthNet-MM with paired Sentinel-1/Sentinel-2 patches and a revised 19-class nomenclature, finding that direct training generally outperforms ImageNet transfer learning and that ResNet outperforms VGG in classification.
Problem
Existing remote-sensing benchmarks largely provide single-modal images or single-label annotations, and some CLC classes are difficult to identify from single-date imagery.
Method
The paper constructs BigEarthNet-MM from 590,326 paired Sentinel-1 and Sentinel-2 patches, assigns CLC-based multi-labels, and introduces a revised 19-class nomenclature.
Results
Direct learning from BigEarthNet-MM achieves the highest class-based F2 score for every class versus ImageNet transfer learning, while ResNet achieves the highest scores across evaluated metrics versus VGG.
Takeaways & Limitations
BigEarthNet-MM provides a public resource for multi-modal multi-label remote-sensing image classification, retrieval, and further learning studies.
Takeaways & Limitations
BigEarthNet-MM is limited for applications requiring time-series data, and some Sentinel-1 patches may contain radio-frequency-interference or other dataset-related artefacts.
Abstract
from arXiv · showhide
This paper presents the multi-modal BigEarthNet (BigEarthNet-MM) benchmark archive made up of 590,326 pairs of Sentinel-1 and Sentinel-2 image patches to support the deep learning (DL) studies in multi-modal multi-label remote sensing (RS) image retrieval and classification. Each pair of patches in BigEarthNet-MM is annotated with multi-labels provided by the CORINE Land Cover (CLC) map of 2018 based on its thematically most detailed Level-3 class nomenclature. Our initial research demonstrates that some CLC classes are challenging to be accurately described by only considering (single-date) BigEarthNet-MM images. In this paper, we also introduce an alternative class-nomenclature as an evolution of the original CLC labels to address this problem. This is achieved by interpreting and arranging the CLC Level-3 nomenclature based on the properties of BigEarthNet-MM images in a new nomenclature of 19 classes. In our experiments, we show the potential of BigEarthNet-MM for multi-modal multi-label image retrieval and classification problems by considering several state-of-the-art DL models. We also demonstrate that the DL models trained from scratch on BigEarthNet-MM outperform those pre-trained on ImageNet, especially in relation to some complex classes, including agriculture and other vegetated and natural environments. We make all the data and the DL models publicly available at https://bigearth.net, offering an important resource to support studies on multi-modal image scene classification and retrieval problems in RS.
I. INTRODUCTION
Remote sensing archives need accurate classification and retrieval, but existing benchmarks largely provide single-modal images with single-label annotations. BigEarthNet-MM addresses this gap with large-scale multi-modal, multi-label image pairs and an alternative nomenclature for difficult classes.
- Remote sensing archives are growing, increasing demand for accurate scene classification and content-based image retrieval.
- Most existing public benchmarks for remote sensing scene classification and retrieval contain single-modal images and single-label annotations.
- Multi-modal, multi-label benchmarks are needed, but large-scale annotation is time-consuming, complex, and costly.
- ImageNet transfer learning may be poorly suited to remote sensing because image characteristics and semantic content differ between the domains.
- BigEarthNet-MM introduces 590,326 Sentinel-1 and Sentinel-2 image-patch pairs plus an alternative nomenclature evolving the original CLC labels.
II. DESCRIPTION OF BIGEARTHNET-MM
BigEarthNet-MM pairs Sentinel-2 and Sentinel-1 patches from closely matched European acquisitions. The archive is built from carefully selected and sensor-specific pre-processing pipelines.
- 590,326 Sentinel-1 and Sentinel-2 patch pairs cover 10 European countries and derive from 125 low-cloud Sentinel-2 tiles acquired between June 2017 and May 2018.
- Sentinel-2 tiles were atmospherically corrected with ESA’s sen2cor before being divided into non-overlapping patches at 10m, 20m, and 60m resolutions.
- Sentinel-1 patches were constructed from 325 closely timed GRD products jointly covering the original Sentinel-2 tile areas.
- Sentinel-1 processing included orbit correction, noise removal, radiometric calibration, terrain correction, and conversion of backscatter to decibels.
A. Class-Nomenclature of BigEartNet-MM
BigEarthNet-MM assigns CLC 2018 multi-labels to paired images and reorganizes the detailed CLC hierarchy into a 19-class nomenclature tailored to the archive’s image properties.
- Each geographically matched Sentinel-1 and Sentinel-2 pair receives one or more labels extracted from the CORINE Land Cover 2018 map.
- CLC provides harmonized European land-cover and land-use maps with approximately 85% reported accuracy and a 44-class Level-3 hierarchy.
- BigEarthNet-MM image patches represent 43 CLC classes under the original nomenclature.
- The alternative nomenclature rearranges CLC Level-3 labels into 19 classes, retaining 10 original classes and grouping 22 classes into 9 new classes.
- Image pairs have between 1 and 12 labels, while 96.80% have no more than 5 labels.
A. Experimental Design
The experiments evaluate multi-modal, multi-label retrieval and classification using stacked Sentinel-1 and Sentinel-2 bands, multiple metrics, and comparisons between learning strategies and CNN architectures.
- Experimental Design: Sentinel-1 VV and VH bands and Sentinel-2 10m and 20m bands were stacked into one input volume for each image pair.
- Experimental Design: The classification evaluation uses Hamming loss, one-error, recall, and F2-Score.
- Experimental Design: Table I compares class-based F2 scores for ImageNet transfer learning with direct learning from BigEarthNet-MM in multi-modal multi-label classification.
1) Comparison among the Strategies of Learning directly from BigEarthNet-MM and Transfer Learning from the ImageNet:
Direct learning from BigEarthNet-MM outperforms ImageNet transfer learning for classification and retrieves semantically more similar image pairs. The advantage is especially pronounced for complex land-use and land-cover classes.
- Direct learning from BigEarthNet-MM achieves the highest class-based F2 score for every class compared with ImageNet transfer learning.
- More than 12% and 25% higher F2 scores are obtained for Industrial or commercial units and Complex cultivation patterns, respectively, with direct learning.
- More than 54% and 69% F2-score improvements occur for Moors, heathland and sclerophyllous vegetation and Agro-forestry areas, respectively.
- Direct learning retrieves semantically more similar pairs, while ImageNet transfer learning retrieves mismatched Urban fabric and Arable land pairs as early as the fifth result.
- The authors attribute direct learning’s advantage to better spectral characterization, reduced semantic mismatch, and adaptation to remote-sensing imagery.
2) Comparison of State-of-the-Art CNN Models:
ResNet models outperform VGG models in multimodal multi-label classification, while increasing depth within a model family has little effect on performance.
- ResNet provides the highest scores across all reported multimodal multi-label classification metrics.
- ResNet50 achieves more than 2% higher recall and F2 scores than VGG models.
- Increasing model depth does not significantly affect performance, with similar scores across depth variants within each architecture.
IV. DISCUSSION AND CONCLUSION
BigEarthNet-MM provides a large multimodal benchmark with a new CLC-based nomenclature for remote-sensing classification and retrieval research. It supports several learning settings but remains constrained for applications requiring time-series information and may contain Sentinel-1 artefacts.
- BigEarthNet-MM contains 590,326 Sentinel-1 and Sentinel-2 image-patch pairs with a new CLC-based class nomenclature.
- The archive supports research on class-imbalanced multimodal data, transfer learning, and unsupervised, self-supervised, and semi-supervised learning.
- BigEarthNet-MM is limited for applications requiring time-series data to accurately describe classes such as Non-irrigated arable land and Permanently irrigated land.
- Some Sentinel-1 patches may contain artefacts from Radio-Frequency-Interference or other dataset-related issues independent of preprocessing.