PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenEarth-OVS44: A Multimodal Remote Sensing Dataset for Open-Vocabulary Segmentation

Yimin Wei, Hongruixuan Chen, Hao Zheng, Hao Liu, Jian Song, Yuting Zhu, Junshi Xia, Naoto Yokoya

arXiv:2610.29642v1cs.CV

TL;DR

Remote-sensing OVS lacks benchmarks with enough category detail and multimodal coverage to assess fine-grained recognition. OpenEarth-OVS44 provides a 44-class dataset with RGB-only, cloudy RGB-SAR, and cross-region evaluation. Training on it improves performance over the strongest training-free baseline on both seen and unseen categories.

  • Problem

    Existing RS OVS benchmarks have limited category coverage and coarse taxonomies, constraining fine-grained semantic evaluation.

    Source: §1 · p. 1

  • Method

    The authors introduce a 44-class benchmark with expert-verified segments, registered RGB-SAR pairs, and RGB-only, cloudy multimodal, and held-out cross-region tracks.

    Source: §1 · pp. 2–3

  • Results

    Up to 7.1 mIoU points on unseen categories and 27.6 mIoU points on seen categories: trained models outperform the strongest training-free baseline.

    Source: §1 · p. 3

  • Takeaways & Limitations

    OpenEarth-OVS44 provides a basis for evaluating fine-grained remote-sensing OVS across modalities, weather conditions, and geographic regions.

    Source: §1 · p. 2, §5 · p. 9

    Current models show weaker performance on rare categories such as tower, chimney, and expressway toll station, where CAT-Seg fails to establish effective visual-text correspondence.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

In remote sensing, open-vocabulary segmentation is particularly important for flexible and scalable land-cover understanding across diverse regions, as it improves generalization to novel categories without requiring exhaustive pixel-level annotations. Despite its great promise, progress in this area remains largely constrained by existing datasets with limited category coverage and overly coarse taxonomies, and therefore struggles to support fine-grained semantic understanding required in real-world applications such as urban functional zoning, carbon accounting, and post-disaster assessment. To address these challenges, we present OpenEarth-OVS44, a multimodal benchmark with 290,587 expert-verified segments from 2,832 RGB images and 1,341 strictly registered RGB-SAR pairs at sub-meter spatial resolution, covering 40 cities across France, Japan, and the United States under a unified 44-class taxonomy organized into 9 base categories. Three evaluation tracks are provided: an RGB-only fine-grained benchmark, a multimodal track with diverse synthetic cloudy conditions for weather-robust evaluation, and a held-out cross-region track. Extensive experiments on representative open-vocabulary segmentation models demonstrate that training on OpenEarth-OVS44 substantially improves fine-grained segmentation performance over the strongest training-free baseline, with improvements of up to 7.1 mIoU points on unseen categories and 27.6 mIoU points on seen categories. The dataset and code will be released.

1 Introduction

Remote-sensing OVS lacks fine-grained multimodal benchmarks, limiting measurement of distinctions needed in applications. OpenEarth-OVS44 addresses this gap with a 44-class, multi-weather dataset and experiments showing gains from training on it.

Motivation

  • Existing remote-sensing OVS benchmarks have limited coverage—up to 17 RGB classes and 8 RGB-SAR classes—and merge functionally distinct categories.Such coarse taxonomies can conceal fine-grained recognition failures, despite their relevance to real-world applications.Source: §1 · p. 1
  • Fine-grained multimodal annotation is difficult because class cues can be subtle, RGB-SAR interpretation requires expert cross-validation, and boundary cases need adjudication.These constraints make scaling annotation pipelines more challenging than simply increasing their size.Source: §1 · p. 2

Main contributions

  • The authors evaluate training-free methods and five trained models, reporting that current training-free RS OVS models struggle with fine-grained tasks.The evaluation also includes four unimodal OVS models and a multimodal framework.Source: §1 · pp. 2–3
  • OpenEarth-OVS44 contains 290,587 manually annotated segments across 2,832 RGB images and 1,341 paired RGB-SAR images, with 44 classes and three evaluation tracks.The tracks cover RGB-only evaluation, multimodal cloudy conditions, and held-out cross-region testing.Source: §1 · pp. 2–3
  • Up to 7.1 mIoU points on unseen categories and 27.6 mIoU points on seen categories: trained models outperform the strongest training-free baseline.The paper establishes baselines by training five representative OVS models on OpenEarth-OVS44.Source: §1 · p. 3

2 Related Work

Remote-sensing OVS has expanded from unimodal to RGB-SAR methods, but evaluations remain limited by coarse-taxonomy benchmarks. OpenEarth-OVS44 addresses this benchmark gap with fine-grained categories and registered multimodal data.

Open-Vocabulary Segmentation Model in RS

  • Remote-sensing OVS includes training-free and training-required methods, with MM-OVSeg introducing optical-SAR fusion for cloudy conditions.Earlier methods are unimodal and degrade under low visibility; current methods remain evaluated on coarse-taxonomy benchmarks.Source: §2 · p. 3

Open-Vocabulary Segmentation Benchmarks

  • Natural-image OVS uses large-scale semantic segmentation datasets, while remote-sensing benchmarks are commonly derived from existing datasets with seen/unseen splits.The cited natural-image example, COCO-Stuff, has 171 classes.Source: §2 · p. 3
  • Existing RS OVS benchmarks offer up to 17 RGB classes or 8 RGB-SAR classes, merging distinct entities and leaving fine-grained weather-robust evaluation without a benchmark.OpenEarth-OVS44 provides 44 manually verified categories and strictly registered RGB-SAR pairs for unified unimodal and multimodal evaluation.Source: §2 · p. 3

3 OpenEarth-OVS44 Benchmark

OpenEarth-OVS44 is designed to evaluate fine-grained remote-sensing OVS under semantic granularity, optical degradation, and geographic shift. It combines expert-reviewed annotations with RGB-only, cloudy RGB-SAR, and held-out cross-region settings.

3.1 Overview

  • The benchmark contains 290,587 segments from 2,832 RGB images, including 1,341 registered SAR counterparts, across 40 cities and 44 classes organized into 9 base categories.Its three settings are Optical for France and US imagery, Cloudy for synthetic thin- and thick-cloud RGB-SAR evaluation, and Cross for held-out Japan imagery.Source: §3.1 · pp. 3–4

3.2.1 Data Collection

  • Optical data come from NAIP, GSI, and IGN, while SAR data come from Umbra Space; experts manually registered and cross-validated the modalities.The stated average RGB-SAR registration RMSE is 1.369 pixels, comparable to an established registered RGB-SAR benchmark [44].Source: §3.2.1 · pp. 4–5

3.2.2 Annotation and Quality Control

  • The seen/unseen split contains 31 seen and 13 unseen classes; nine Road and Water classes and four underrepresented categories are held out from training.The four additional unseen categories are windmill, beach, barren/exposed soil, and expressway toll station; their annotations receive no direct class-level supervision.Source: §3.2.2 · p. 5
  • Three PhD researchers reviewed annotations and two senior experts performed final adjudication, yielding 290,587 expert-verified segments.The review process required approximately 512 person-hours.Source: §3.2.2 · p. 5

3.2.3 Data Split

  • Optical uses 1,936 France/US images, Cloudy uses 1,341 registered triplets across 43 shared categories, and Cross tests on 896 Japan images excluded from training and validation.Cloudy pairs synthetic thin- and thick-cloud RGB variants with unchanged SAR; Cross evaluates 40 categories, including 29 seen and 11 unseen.Source: §3.2.3 · pp. 5–6

3.3 Dataset Statistics and Comparison

  • Compared with existing RS OVS datasets, OpenEarth-OVS44 offers finer semantic granularity while retaining paired RGB-SAR observations for unified RGB-only, weather-robust, and cross-region evaluation.Its taxonomy includes underrepresented concepts such as windmill, container crane, and expressway toll station.Source: §3.3 · p. 6, Table 3 · p. 6

4 Benchmark Experiments

The experiments evaluate representative OVS models across three settings and show that training on OpenEarth-OVS44 improves fine-grained segmentation, while performance remains challenged by coarse taxonomies, cloudy conditions, and rare classes.

4.1 Baselines and Experimental Setups

  • The evaluation covers natural-image, unimodal remote-sensing, training-free remote-sensing, and multimodal OVS methods, with training-required models trained on the corresponding splits.SegEarth-OV and SegEarth-OV3 are evaluated without additional training; mIoU is the primary metric.Source: §4.1 · p. 6

4.2 Main Results

  • SegEarth-OV, the strongest training-free baseline, scores 14.4, 6.1, and 16.3 overall mIoU on Optical, Cloudy, and Cross, respectively.SegEarth-OV3 performs even lower across all three settings, so adopting a stronger frozen vision backbone alone is insufficient for fine-grained recognition.Source: §4.2 · p. 6
  • 91.4 and 91.8 IoU on coarse Road and Water fall to 6.4 and 12.9 mIoU after those categories are split into fine-grained subclasses.The evaluation compares coarse and fine-grained taxonomies for SegEarth-OV while holding other inference settings constant.Source: §4.2 · p. 7, Table 5 · p. 7
  • 34.4 and 26.8 overall mIoU are achieved by CAT-Seg (ViT-L/14) on OpenEarth-OVS44-Optical and OpenEarth-OVS44-Cross, respectively; training also improves unseen-category performance over the training-free baseline.On OpenEarth-OVS44-Cloudy, CAT-Seg (ViT-L/14) exceeds that baseline by 5.3 mIoU points on unseen categories; RSKT-Seg (ViT-L/14) gains 26.4 mIoU points on seen categories.Source: §4.2 · p. 7, Table 4 · p. 7
  • 27.9 overall mIoU on OpenEarth-OVS44-Cloudy makes MM-OVSeg (ViT-L/14) outperform all unimodal models there.All unimodal models degrade under cloudy conditions; the passage attributes MM-OVSeg's improvement to complementary SAR information.Source: §4.2 · p. 7
  • Seen-category performance remains notably higher than unseen-category performance, and CAT-Seg (ViT-L/14) performs poorly on rare classes such as tower, chimney, and expressway toll station.The discussion identifies stronger visual-text alignment for rare classes and more informative, less ambiguous prompts as directions for improvement.Source: §4.2 · pp. 7–8

5 Conclusion

OpenEarth-OVS44 is a fine-grained multimodal benchmark for remote-sensing open-vocabulary segmentation. Experiments report improvements from training on it across evaluation tracks, alongside limitations in current methods.

5 Conclusion

  • OpenEarth-OVS44 contains 290,587 annotated segments from 2,832 RGB images and 1,341 paired RGB-SAR images at sub-meter resolution, covering 44 categories across 40 cities in three countries.Its tracks comprise RGB-only fine-grained evaluation, multimodal evaluation with synthetic cloudy conditions, and held-out cross-region evaluation.Source: §5 · pp. 8–9
  • Figure 4 presents per-class IoU results for existing models on OpenEarth-OVS44-Optical, distinguishing seen and unseen classes by purple and blue class names.Source: Figure 4 · p. 9
  • Training five OVS models on OpenEarth-OVS44 yields improvements of up to 7.1 mIoU points on unseen categories and 27.6 mIoU points on seen categories across the evaluation tracks.Experiments on seven advanced OVS models also highlight potential and inherent limitations of current methods.Source: §5 · p. 9

Related papers

Loading 2610.29642v1…