Source-linked AI summary
MLRSNet: A Multi-label High Spatial Resolution Remote Sensing Dataset for Semantic Scene Understanding
Xiaoman Qi, PanPan Zhu, Yuebin Wang, Liqiang Zhang, Junhuan Peng, Mengfan Wu, Jialong Chen, Xudong Zhao, Ning Zang, P. Takis Mathiopoulos
TL;DR
Remote-sensing datasets commonly use single labels despite scenes containing multiple land-cover classes, while available resources may have limited coverage for complex scene understanding. The paper constructs MLRSNet, a large high-resolution multi-label dataset, and evaluates deep-learning methods for image classification and retrieval. The authors conclude that MLRSNet can serve as a benchmark for these tasks.
Problem
Existing remote-sensing datasets commonly use single-label annotation, while limited data distribution and unique data sources can reduce interclass diversity and complicate robust scene understanding.
Method
The paper develops MLRSNet, a large high-resolution multi-label remote-sensing dataset for semantic scene understanding, and evaluates deep-learning methods on image classification and retrieval.
Results
MLRSNet can be adopted as a benchmark dataset for performance evaluations of multi-label image retrieval and scene classification.
Takeaways & Limitations
MLRSNet provides a data resource for evaluating and advancing methods in semantic scene understanding.
Takeaways & Limitations
The authors note that blurry boundaries between some data pairs, such as airport and island or parkway, remain in the dataset.
Abstract
from arXiv · showhide
To better understand scene images in the field of remote sensing, multi-label annotation of scene images is necessary. Moreover, to enhance the performance of deep learning models for dealing with semantic scene understanding tasks, it is vital to train them on large-scale annotated data. However, most existing datasets are annotated by a single label, which cannot describe the complex remote sensing images well because scene images might have multiple land cover classes. Few multi-label high spatial resolution remote sensing datasets have been developed to train deep learning models for multi-label based tasks, such as scene classification and image retrieval. To address this issue, in this paper, we construct a multi-label high spatial resolution remote sensing dataset named MLRSNet for semantic scene understanding with deep learning from the overhead perspective. It is composed of high-resolution optical satellite or aerial images. MLRSNet contains a total of 109,161 samples within 46 scene categories, and each image has at least one of 60 predefined labels. We have designed visual recognition tasks, including multi-label based image classification and image retrieval, in which a wide variety of deep learning approaches are evaluated with MLRSNet. The experimental results demonstrate that MLRSNet is a significant benchmark for future research, and it complements the current widely used datasets such as ImageNet, which fills gaps in multi-label image research. Furthermore, we will continue to expand the MLRSNet. MLRSNet and all related materials have been made publicly available at https://data.mendeley.com/datasets/7j9bv9vwsx/2 and https://github.com/cugbrs/MLRSNet.git.
1. Introduction
Remote-sensing scene images contain complex, multiple land-cover classes, but many existing datasets use single labels or limited class coverage. MLRSNet addresses this gap with a large, high-resolution multi-label dataset and evaluations for image classification and retrieval.
- Motivation: Large-scale training data can substantially improve CNN performance on image-recognition tasks, motivating richer annotated resources for semantic scene understanding.Prior work linked deep-learning performance to training-data quantity and showed gains from large multi-perspective samples.
- Limitations of Existing Datasets: Many existing remote-sensing scene datasets remain insufficient because of high intra-class diversity, low inter-class variation, limited image counts, or few classes.SEN12MS, for example, contains 180,662 triplets but no more than 17 classes under a selected classification scheme.
- Motivation: Most existing remote-sensing datasets assign each image a single label, limiting representation of scenes containing multiple semantic classes.Multi-label annotation can capture label correlations such as roads with cars and grass or water with golf courses.
- Dataset: MLRSNet provides 109,161 high-resolution images across 46 categories, with each image tagged by several of 60 predefined class labels.Images have a fixed size of 256 × 256 pixels, with varied pixel resolutions; category sizes range from 1,500 to 3,000 samples.
- Evaluation: The paper evaluates multi-label deep-learning approaches on image classification and image retrieval, and reports better performance for these tasks.The authors conclude that MLRSNet can serve as a benchmark for evaluating multi-label image retrieval and scene classification.
- Contribution: MLRSNet is presented as a large high-resolution multi-label resource with abundant multi-label information and high intraclass diversity.The dataset is intended to support evaluation and advancement of semantic scene-understanding methods.
2. MLRSNet: A Multi-label High Spatial Resolution Remote Sensing
MLRSNet is a large-scale, high-resolution, multi-label remote sensing dataset designed for overhead semantic scene understanding. It combines hierarchical labels, broad visual diversity, quality-controlled annotations, and benchmark tasks for image classification and retrieval.
- Multi-resolution overhead imagery: MLRSNet spans approximately 10-meter to 0.1-meter pixel resolutions and uses fixed 256 × 256-pixel images from satellite or aerial sensors.
- Hierarchy: The dataset organizes labels hierarchically into 3 first-class categories, 46 second-class categories, and 60 third-class labels.
- Each image receives between 1 and 13 labels from 60 predefined class labels because remote sensing scenes contain non-mutually-exclusive object classes.
- MLRSNet contains 109,161 high-resolution remote sensing images annotated across 46 scene categories, with 1,500–3,000 samples per category.
- Diversity: Its diversity strategy covers geographic and seasonal distributions, weather, viewpoints, capture times, resolution, object pose, illumination, backgrounds, and occlusion.
- Construction and quality control: Quality-controlled construction involved scene collection, database quality control, and diversity improvement, with ambiguous category boundaries receiving additional annotation review.
- Diversity evaluation: In a relative-diversity evaluation across 20 shared categories, MLRSNet was most diverse, with averages of 0.78 versus 0.56 for AID and 0.69 for NWPU-RESISC45.
3. Scene Classification
This section evaluates CNNs for multi-label remote-sensing scene classification using MLRSNet, with sigmoid outputs converted into 60-label predictions. Fine-tuned models perform well overall, especially MLRSNet-DenseNet201 as training data increases.
- MLRSNet provides a benchmark for evaluating CNN-based remote-sensing scene classification.
- Eight pretrained CNN architectures, including VGGNet, ResNet, and DenseNet variants, are fine-tuned for classification.
- Sigmoid outputs assign scores to 60 classes, and thresholding at 0.5 produces the final multi-label prediction.
- Three training configurations use 20%, 30%, or 40% of MLRSNet for training, with 10% validation and the remainder for testing.
- MLRSNet-DenseNet201 achieves significantly better metric values, while DenseNet201 and DenseNet169 exceed 0.80 F1 score within 10 epochs at a 20% training ratio.
- Performance generally improves with more training data, and repeated trials are summarized with standard-deviation error bars.
4. Image Retrieval
This section uses MLRSNet to benchmark multi-label image retrieval with fine-tuned CNN features and similarity-based ranking. DenseNet201 performs best overall, while multi-label retrieval better matches shared scene objects than single-label retrieval.
- MLRSNet serves as a benchmark for evaluating retrieval performance under multi-label scene annotations.
- CNN features extracted from the last network layer are compared with Euclidean distance across query and retrieval images.
- The retrieval protocol tests 5% vs. 55%, 10% vs. 50%, and 15% vs. 45% query-to-database splits.
- Retrieval is evaluated using ANMRR, mAP, and P@k for k = 10, 50, 100, and 500.
- MLRSNet-DenseNet201 has the strongest retrieval performance across query percentages, with higher mAP and P@k and lower ANMRR than competing models.
- DenseNet201’s mAP improves by about 1% over DenseNet169 and by 6.26%–7.94% over ResNet50.
- Retrieval quality varies by category: basketball court and commercial area are difficult, potentially because of high intra-class diversity.
- Multi-label retrieval better matches shared objects such as swimming pools and water than single-label retrieval.
5. Conclusion
MLRSNet is introduced as a large-scale, diverse, hierarchical, multi-label high-spatial-resolution dataset for overhead semantic scene understanding. Experiments support its use as a benchmark for multi-label scene classification and image retrieval, while future work extends it to additional recognition tasks.
- MLRSNet is a multi-label high spatial resolution remote sensing dataset for semantic scene understanding from the overhead perspective.
- The dataset is characterized by hierarchy, large scale, diversity, and multi-label annotation.
- Experiments evaluate different deep neural networks on multi-label scene classification and image retrieval.
- MLRSNet can serve as a benchmark for performance evaluations of multi-label image retrieval and scene classification.
- MLRSNet complements large object-centric datasets such as ImageNet and is planned for expansion to semantic, instance segmentation, and ground object recognition.