PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mapping

Junshi Xia, Naoto Yokoya, Bruno Adriano, Clifford Broni-Bediako

arXiv:2210.10732v1cs.CVcs.LG

TL;DR

Sub-meter land-cover benchmarks lack regional diversity and fine annotation quality. OpenEarthMap addresses this gap with a globally distributed, manually labeled dataset and evaluates segmentation and domain-adaptation methods, finding that models trained on it generalize across regions while revealing challenging adaptation settings.

  • Problem

    Existing sub-meter semantic-segmentation benchmarks lack regional diversity and sufficiently fine land-cover annotations.

  • Method

    The paper constructs OpenEarthMap from manually annotated imagery across 97 regions, then evaluates semantic-segmentation and unsupervised-domain-adaptation methods.

  • Results

    Models trained on OpenEarthMap generalize across the globe, while domain-adaptation experiments identify challenging settings for further development.

  • Takeaways & Limitations

    OpenEarthMap provides a globally diverse benchmark for high-resolution land-cover mapping and supports evaluation under varied regional and domain conditions.

  • Takeaways & Limitations

    The paper identifies extending DAFormer or developing a new UDA method with U-Net-EfficientNet-B4 as an open direction.

Abstract

from arXiv · show

We introduce OpenEarthMap, a benchmark dataset, for global high-resolution land cover mapping. OpenEarthMap consists of 2.2 million segments of 5000 aerial and satellite images covering 97 regions from 44 countries across 6 continents, with manually annotated 8-class land cover labels at a 0.25--0.5m ground sampling distance. Semantic segmentation models trained on the OpenEarthMap generalize worldwide and can be used as off-the-shelf models in a variety of applications. We evaluate the performance of state-of-the-art methods for unsupervised domain adaptation and present challenging problem settings suitable for further technical development. We also investigate lightweight models using automated neural architecture search for limited computational resources and fast mapping. The dataset is available at https://open-earth-map.org.

1. Introduction

OpenEarthMap addresses regional disparity and coarse annotation quality in sub-meter semantic-segmentation benchmarks by assembling globally diverse, manually labeled imagery and evaluating models for mapping and domain adaptation.

  • Motivation: Existing sub-meter semantic-segmentation benchmarks are limited by regional bias toward developed countries and coarse land-cover annotations.High-resolution imagery and redistribution constraints make broader coverage difficult, while manual labeling at sub-meter resolution is costly.
  • Dataset: OpenEarthMap contains 5000 images from 97 regions in 44 countries across six continents, with eight manually labeled land-cover classes at 0.25–0.5m GSD.The dataset combines adopted benchmark imagery with newly collected public imagery to reduce regional imbalance.
  • Dataset: The dataset uses eight classes: bareland, rangeland, developed space, road, tree, water, agriculture land, and building.These classes align with existing sub-meter land-cover products and benchmarks.
  • Annotation: Sixteen annotators labeled and quality-checked the imagery, requiring an average of 2.5 hours per image.Each image was labeled by one person and checked by at least two others.
  • Annotation: Human annotators agreed on 78% of pixels across repeated labels for 200 images, below the 96% agreement reported for Cityscapes.The comparison indicates that high-resolution remote-sensing annotation is particularly challenging.

3. Land Cover Semantic Segmentation

The paper evaluates CNN- and Transformer-based segmentation architectures on OpenEarthMap, including lightweight networks found through neural architecture search. Results identify strong baseline models, compact accuracy–efficiency trade-offs, and performance behavior under limited training data.

  • U-Net, U-NetFormer, FT-U-NetFormer, DeepLabV3, HRNet, SETR, SegFormer, and UPerNet were evaluated with multiple CNN and Transformer backbones.
  • U-NetFormer and FT-U-NetFormer share the top positions, while U-Net-EfficientNet-B4, UPerNet-Swin-B, and FT-U-NetFormer are recommended considering performance, parameters, and FLOPs.The recommended models balance segmentation quality with computational size.
  • Neural Architecture Search: SparseMask and FasterSeg automatically discovered compact segmentation networks for OpenEarthMap.
  • Neural Architecture Search: FasterSeg found the lightest-weight network, while SparseMask reduced computational complexity but had lower inference speed.
  • Learning from Limited Labels: With 50% of the training images, machine predictions reached almost the same level as human annotation.Human annotation accuracy was measured as IoU between two independent annotations.
  • Learning from Limited Labels: With 10% of the training data, U-Net-EfficientNet-B4 outperformed the three Vision Transformer methods by about 6-15% across class-specific IoUs.The paper attributes this advantage to CNN inductive bias under limited data.

4. Unsupervised Domain Adaptation

The paper evaluates metric-based, adversarial, and self-training methods for unsupervised domain adaptation across regional and continent-wise shifts. Domain gaps vary substantially, while SegFormer and especially DAFormer provide strong results in challenging settings, with Oceania remaining difficult as a source domain.

  • The UDA evaluation compares metric-based MCD, adversarial methods, and self-training methods including CBST, IAST, and DAFormer.
  • Regional-level UDA: In regional UDA, Oracle settings performed best, source-only settings performed worst, and DAFormer achieved the highest mIoU of 62.35%.
  • Continent-wise UDA: SegFormer generally outperformed U-Net-EfficientNet-B4 in 26 of 30 challenging continent-wise UDA settings, contrary to semantic segmentation and regional UDA results.
  • Continent-wise UDA: DAFormer improved over SegFormer in many continent-wise cases, doing so in 20 of 30 settings.
  • Continent-wise UDA: Oceania produced the lowest transferred results when used as the source domain, and limited Oceania data caused poor DAFormer results in some target settings.

5. Cross-Dataset Evaluation

The cross-dataset evaluation tests OpenEarthMap as a pretraining source against LoveDA and DeepGlobe using a common U-Net-EfficientNet-B4 model. OpenEarthMap pretraining increasingly outperforms the alternatives as fine-tuning proceeds and gives LoveDA a stronger starting point.

  • The study compares U-Net-EfficientNet-B4 models trained from scratch or fine-tuned using OpenEarthMap, LoveDA, and DeepGlobe under similar settings.
  • OpenEarthMap increasingly outperforms both comparison models as the number of training epochs increases.
  • When fine-tuned on LoveDA, the OpenEarthMap-pretrained model achieves more than a 20% increase in initial IoU and remains higher as epochs increase.

6. Demonstration on Out-of-Sample Imagery

The paper evaluates OpenEarthMap on out-of-sample NAIP imagery using a Chesapeake Bay land cover map as reference. The OpenEarthMap model produces similar classification and resolution while showing finer spatial segmentation than the LoveDA model.

  • The demonstration uses an NAIP image resampled at 0.5m GSD and compares predictions with a 13-class Chesapeake Bay land cover map.
  • The OpenEarthMap result is similar to the Chesapeake Bay reference in classification and resolution.
  • OpenEarthMap provides finer spatial segmentation than the LoveDA model in the out-of-sample comparison.

7. Conclusion and Societal Impacts

OpenEarthMap addresses limited regional diversity and annotation quality in sub-meter land-cover benchmarks through broad geographic coverage and detailed labels. Experiments show both worldwide adaptation potential and remaining challenges in domain adaptation, limited-data training, and lightweight modeling.

  • OpenEarthMap covers 97 regions from 44 countries across 6 continents with finely detailed annotations for global high-resolution land-cover mapping.
  • Models trained on OpenEarthMap can adapt and generalize across the globe on out-of-sample imagery.
  • Continent-wise domain gaps and limited training data remain challenging problem settings for further technical development.
  • NAS-based lightweight models support investigation of mapping with resource-limited devices.

Supplementary Material

The supplementary material documents the dataset classes, baseline methods, training settings, and additional experimental and mapping results.

  • The supplement describes the land-cover classes used in OpenEarthMap.
  • It provides an overview of baseline methods for semantic segmentation and unsupervised domain adaptation.
  • It reports training settings, additional task results, and further land-cover mapping examples.

S1. Land Cover Classes

OpenEarthMap defines eight visually interpretable land-cover classes, including a subdivision of urban areas into building, road, and developed space. Regional class proportions vary substantially across the dataset.

  • The urban class is subdivided into building, road, and developed space for visual interpretation at sub-meter resolution.
  • The eight classes are bareland, rangeland, developed space, road, tree, water, agriculture land, and building.
  • Bareland denotes unvegetated sand, rocks, or other earthen accumulations, whereas rangeland covers uncultivated or ungrazed herbaceous vegetation and bushes.
  • Developed space includes built surfaces and artificial grass areas, while roads include transport lanes and runways made from asphalt, concrete, or soil.
  • Tree, water, agriculture land, and building cover vegetation structures, water bodies, cultivated or grazing areas, and residential, commercial, or industrial structures.
  • Class proportions differ across the 97 regions, as shown by the t-SNE visualization and representative regional bar graphs.

S2. Land Cover Semantic Segmentation

The supplementary segmentation study compares CNN- and Transformer-based baselines, visualizes their predictions, and investigates training choices and compact architecture search. The experiments also examine lightweight models for resource-constrained mapping.

  • Baseline methods: The baseline suite includes CNN architectures such as U-Net, DeepLabV3, HRNet, K-Net, and ConvNeXt, alongside Transformer-based models.
  • Baseline methods: The Transformer baselines include UNetFormer, FT-U-NetFormer, SETR, SegFormer, and UPerNet with ViT, Twins, and Swin Transformer backbones.
  • Training settings: All baselines use PyTorch and are trained with 512×512 random crops for 200 epochs under specified optimizer and loss settings.
  • Evaluation: Compact-model performance is reported using class IoUs and mIoU with and without test-time augmentation.
  • Results: Visual comparisons show model-specific differences in recognizing roads, bareland, and water bodies across representative regions.
  • Ablation study: The study evaluates the effects of ImageNet pre-training, optimizers, input size, and loss functions using UPerNet-Swin-B.
  • Lightweight models: SparseMask and FasterSeg search for compact segmentation architectures using gradient-based strategies, with pruning or teacher-student co-searching.
  • Lightweight models: The compact architectures require about 0.8 GPU days with SparseMask and 2 GPU days with FasterSeg for architecture search.

S3. Unsupevised Domain Adaptation

The paper evaluates metric-based, adversarial, and self-training methods for unsupervised domain adaptation, with DAFormer showing strong qualitative transfer and challenging class- and continent-dependent behavior.

  • S3.1. Brief Overview of the Baselines: The benchmark compares metric-based MCD, adversarial methods, and self-training methods, including DAFormer, for unsupervised domain adaptation.The evaluated methods include DeepLabV2-based baselines and the transformer-based DAFormer.
  • S3.3. Results: DAFormer better recognized small roads, buildings, trees, developed space, and water than several source-only or self-training baselines in visual examples.The improvements varied by scene, with DAFormer especially effective for tiny structures and narrow roads.
  • S3.3. Results: U-Net-EfficientNet-B4, SegFormer, and K-Net generally occupied the top positions across source-only and oracle settings, while DeepLabV2 performed worst.SegFormer achieved the best oracle performance, whereas U-Net-EfficientNet-B4 led the source-only setting.
  • S3.3. Results: Continent-wise transfer depended strongly on source and target continents; limited Oceania source imagery produced the worst transfers, whereas Oceania targets performed better.The results cover eight source-target combinations and compare class-specific IoUs and mIoUs.
  • S3.3. Results: Building was the easiest transferred class, while bareland was the most challenging; DAFormer improved results particularly for difficult classes such as bareland and water.Reported building IoUs were 63.4–76.8 for U-Net-EfficientNet-B4 and 64.3–76.3 for SegFormer, while bareland IoUs were much lower.

S4. Mapping for Out-of-Sample Images

OpenEarthMap models produced visually detailed, semantically consistent maps and reasonably high-resolution predictions on images from regions not included in the dataset.

  • S4. Mapping for Out-of-Sample Images: On Chesapeake Bay imagery, the OpenEarthMap-trained U-Net-EfficientNet-B4 produced fine spatial details and semantically consistent land cover maps.The comparison used models trained on OpenEarthMap, LoveDA, DeepGlobe, and DynamicEarthNet with the same implementation details.
  • S4. Mapping for Out-of-Sample Images: The DynamicEarthNet comparison used original 1m GSD imagery because it performed better than processing at 3m GSD.Additional comparisons considered 0.5cm, 1m, and 3m inference resolutions.
  • S4. Mapping for Out-of-Sample Images: A U-Net-EfficientNet-B4 trained on OpenEarthMap generated reasonably high-resolution land cover maps for out-of-sample images from France, China, Ecuador, and Tanzania.The evaluated sources were MiniFrance, LoveDA, SIGTIERRAS, and the Zanzibar Mapping Initiative.
  • S4. Mapping for Out-of-Sample Images: Because these test images were not included in OpenEarthMap, the visual results support the model’s generalization capability.The evidence is based on out-of-sample mapping examples rather than a quantitative benchmark result.

S5. Attribution of Source Data

The supplementary attribution materials document the source-data provenance for OpenEarthMap’s 97 regions, while the supplied figure captions describe related visual analyses rather than attribution details.

  • S5. Attribution of Source Data: Table S6 summarizes the attribution of source data for all 97 OpenEarthMap regions.The table is the direct source-data attribution reference for the dataset.
  • S5. Attribution of Source Data: OpenEarthMap labels use the same license as their original RGB images when those source licenses vary by dataset.This licensing approach applies across the collected source data.
  • S5. Attribution of Source Data: Labels for regions whose RGB imagery is public-domain or lacks an explicit license are released under CC BY-NC-SA 4.0.The stated license is Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International.
Loading 2210.10732v1…