PaperlayerView exact revision on arXiv

Source-linked AI summary

OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation

Kaiyu Li, Zepeng Xin, Zixuan Jiang, Jing Fu, Lanxuan Xue, Lingyu Zhang, Xiangyong Cao

arXiv:2607.27278v2cs.CV

TL;DR

Existing open-vocabulary EO benchmarks provide limited evidence because they cover narrow categories or query forms. OVEarth-Bench broadens category and query diversity under unified zero-shot mask and box localization, finding that MLLM-based methods perform strongest overall while EO-specific methods lag leading general models.

  • Problem

    Existing open-vocabulary EO benchmarks cover narrow category vocabularies or limited query forms, incompletely assessing genuine open-vocabulary capability.

  • Method

    OVEarth-Bench evaluates hierarchical category breadth and diverse vocabulary, referring, and reasoning queries with human-verified mask and box annotations under unified zero-shot localization.

  • Results

    MLLM-based methods achieve the strongest overall performance across query types, while EO-specific models generally fall behind the strongest general methods.

  • Takeaways & Limitations

    Broader category coverage makes model comparisons less sensitive to category selection, supporting joint evaluation of category breadth, query diversity, localization, and presence prediction.

  • Takeaways & Limitations

    Accurate localization of positive queries does not necessarily guarantee reliable rejection of absent concepts or presence judgments.

Abstract

from arXiv · show

Open-vocabulary Earth observation (EO) aims to localize geospatial concepts specified in natural language rather than a fixed label set. Existing benchmarks, however, usually cover narrow category vocabularies or limited query forms. To fill this gap, we introduce OVEarth-Bench, which extends existing evaluation in two directions: category breadth, through broad hierarchical category coverage with positive and negative expressions, and query diversity, through vocabulary, referring, and reasoning queries. The benchmark supports mask and box localization under a unified zero-shot protocol. We evaluate a broad set of general and EO-specific methods. The evaluation reveals that: (1) the performance of current methods remains limited, while broader category coverage yields more stable model rankings; (2) MLLM-based methods achieve the strongest overall performance; and (3) EO-specific methods generally underperform general models and rarely match the strongest methods. These findings provide guidance for future open-vocabulary EO method design and highlight the importance of developing more realistic, diverse, high-quality, and large-scale benchmarks for reliable evaluation. Our data and evaluation package are released at https://earth-insights.github.io/OVEarth-bench.

Introduction

OVEarth-Bench addresses incomplete open-vocabulary EO evaluation by expanding category breadth and query diversity beyond conventional narrow-category benchmarks. Its systematic evaluation finds that rankings are sensitive to category composition, MLLM-based methods lead, and EO specialization alone offers no consistent advantage.

  • Motivation: EO spans diverse semantic concepts and scales, motivating open-vocabulary localization beyond predefined label sets.The covered concepts include land cover, transportation networks, industrial facilities, public services, and agricultural structures.
  • Limitations: Existing benchmarks provide limited fine-grained and long-tail category coverage and cannot simultaneously evaluate vocabulary, referring, and reasoning queries.This leaves genuine open-vocabulary EO capability incompletely assessed.
  • Benchmark: OVEarth-Bench evaluates category breadth with 172 categories and 1,346 unique vocabulary strings, alongside diverse query forms.The benchmark uses a hierarchical taxonomy and positive and negative queries, and is built from newly collected EO images.
  • Findings: Model rankings are substantially affected by narrow category composition, MLLM-based methods dominate leading results, and EO specialization alone has no consistent advantage.The findings also motivate combining MLLM generalization capabilities with richer EO-specific corpora.
  • Evaluation: 49 general and EO-specific model variants are systematically evaluated across segmentation and detection paradigms under zero-shot evaluation.The evaluated families include general contrastive segmentation, MLLM-based segmentation, detection-and-grounding, and EO-specific methods.

Related Work

Related work spans traditional remote-sensing benchmarks with narrow category spaces, language-conditioned evaluations with diverse query forms, and open-vocabulary methods built on language-aligned representations. EO-specific approaches further adapt these methods to overhead imagery and domain characteristics such as small objects and arbitrary orientation.

  • Traditional RS benchmarks: Traditional remote-sensing benchmarks evaluate semantic, box-level, or instance segmentation within narrow category spaces.These datasets capture EO challenges including scale, orientation, and density variation, but remain tied to limited category vocabularies.
  • Language-conditioned benchmarks: Language-conditioned benchmarks evaluate visual grounding, referring segmentation, and implicit or reasoning-oriented queries.RSVGD studies box-level grounding, while RISBench, RRSIS-D/RefSegRS, EarthReason, and LaSeRS broaden target specification.
  • Language-conditioned benchmarks: Existing language-conditioned resources reuse public benchmark images, limiting source diversity and risking overlap with pre-training data.They also assess category recognition, absentconcept rejection, spatial reference, and functional reasoning in separate datasets rather than one shared benchmark.
  • Open-vocabulary methods: Open-vocabulary methods use language-aligned representations to localize concepts beyond a fixed inference vocabulary.CLIP-based dense methods adapt image–text alignment to pixel prediction, while MLLM-based models provide another method family.
  • EO-specific methods: EO-specific methods adapt open-vocabulary segmentation or vision–language pretraining to overhead appearance, small objects, and arbitrary orientation.Other EO-specific approaches emphasize category-rich localization and related remote-sensing localization or segmentation settings.

OVEarth-Bench

OVEarth-Bench evaluates open-vocabulary Earth observation through broad category coverage and diverse natural-language queries, using newly collected imagery with unified mask and box localization. Its construction combines hierarchical taxonomy design, multi-stage annotation quality control, and three query-specific evaluation tasks.

  • Benchmark scope: OVEarth-Bench jointly evaluates category and query diversity on newly collected images with mask and box annotations under zero-shot localization.The benchmark uses natural-language queries without benchmark-specific adaptation or a fixed inference label set.
  • Category design: The taxonomy contains seven top-level domains and 172 reviewed categories spanning plants, services, industry, transportation, terrain, water infrastructure, and special objects.Categories were designed from land-use standards, EarthNets inventories, and OpenStreetMap tags before image collection.
  • Query annotation: A two-stage LLM pipeline generates positive, negative, referring, and reasoning queries, then screens absent-object and spatial-direction errors before expert review.Stage 2 flagged 3.9% of negative candidates as present and auto-corrected 28.6% of referring expressions.
  • Data collection and scale: The dataset contains 590 mask annotations from 520 images across 172 categories, with 425 unique positive and 1,029 unique negative vocabulary items.All images are newly collected through POI retrieval and cover six continents, with denser sampling in Asia, Europe, and North America.
  • Evaluation tasks: The benchmark defines open-vocabulary, referring, and reasoning tasks, each evaluated with segmentation masks, horizontal boxes, or oriented boxes as separate localization modalities.Positive queries require localization, negative queries require rejection, referring queries use spatial context, and reasoning queries describe function without naming the category.

Experiments

Experiments evaluate 49 model variants under a unified zero-shot protocol across vocabulary, referring, and reasoning queries for mask and box localization. Current performance remains limited, MLLM-based methods lead overall, EO specialization offers limited gains, and rankings are sensitive to category breadth and query form.

  • Evaluation setup: 49 model variants spanning general contrastive, MLLM-based, detection or grounding, and EO-specific open-vocabulary approaches are evaluated zero-shot without benchmark training, fine-tuning, or threshold selection.Detection outputs are converted to masks with SAM2.1-L, and reported box results use HBB for a common format.
  • Overall performance: 38.75%, 41.32%, and 37.96% are the highest segmentation ma-IoU scores for vocabulary, referring, and reasoning queries, all achieved by Rex-Omni+SAM.The best box-localization scores are 24.53%, 35.56%, and 26.12% for mi-F10.5, and 14.23%, 21.47%, and 14.91% for mi-F10.5:0.95, respectively.
  • Model comparison: Only one EO-specific variant ranks among the top ten vocabulary-segmentation results, and none ranks among the top ten for referring or reasoning queries.The best EO-specific results remain 7.07–15.31 segmentation points and 7.46–23.58 detection points below task maxima.
  • Model comparison: MLLM-based methods account for nine of the ten best vocabulary-segmentation ma-IoU results and all ten best results for referring and reasoning queries.Their advantage may reflect broad visual-semantic alignment and transferable language understanding from heterogeneous image-text, grounding, segmentation, and multimodal pretraining.
  • Ranking sensitivity: With five categories, the median Spearman’s ρ is 0.84 and median top-five recovery is 0.6, showing that narrow category sets can alter model rankings.The analysis repeatedly samples smaller category sets and compares their rankings with the full-benchmark ranking.
  • Query diversity: Query-form effects are model-dependent: SegEarth-R2 declines on referring expressions, UniGeoSeg improves, and reasoning descriptions improve some models while reducing others.Comparisons use annotation targets shared across query types to isolate query-form effects.
  • Presence discrimination: Qwen3-VL-4B+SAM reaches MCC 0.60, while every other model with defined MCC scores at most 0.35; vocabulary ma-IoU and MCC have Spearman ρ = 0.107.Thus, localizing present concepts does not imply reliable rejection of absent concepts.

Conclusion … Stage 2: Verification

OVEarth-Bench provides a unified zero-shot evaluation of category and query diversity in open-vocabulary Earth observation, while its prompt pipeline generates and verifies positive, negative, and referring-expression queries. The benchmark supports comprehensive recognition, reasoning, and localization assessment, with MLLM-based methods achieving the strongest overall performance across query types.

  • Conclusion: OVEarth-Bench unifies category and query-diversity evaluation with broad taxonomy, diverse queries, and spatial annotations for recognition, reasoning, and localization.The benchmark is designed for comprehensive open-vocabulary EO evaluation under a unified zero-shot protocol.
  • Conclusion: MLLM-based methods achieve the strongest overall performance across query types.The conclusion identifies MLLM-based approaches as the leading method family in the experiments.
  • A Query Generation Prompt Templates: The query-generation pipeline uses GPT-5-mini with structured JSON output.This implementation supports consistent machine-readable generation of benchmark query content.
  • Stage 1: Generation: Stage 1 prompts an expert remote-sensing annotator to generate open-vocabulary segmentation prompts and challenging absent negative candidates.The annotator analyzes red-outlined target regions in satellite or aerial imagery.
  • Stage 1: Generation: Stage 1 requires exactly two short positive category phrases, including the canonical category name in at least one phrase.The prompt favors standard category nouns over visual descriptions and discourages forced attributes such as color or material.
  • Stage 2: Verification: Stage 2 verifies negative candidates and checks referring expressions for factual correctness, especially spatial relations and direction words.Verification inspects the entire image because remote-sensing scenes may contain very small or densely packed objects.
  • Stage 2: Verification: Stage 2 records ABSENT or PRESENT statuses for negatives and returns exactly two final referring expressions after quality-control correction.Each referring expression receives a valid or corrected status with issue and corrected-phrase fields.

B Dataset Statistics · Image Statistics

OVEarth-Bench images span diverse spatial extents and resolutions, with most tiles being very-high-resolution despite a small number of coarse-resolution outliers. Image widths and GSD values show substantially different median and mean statistics.

  • Image Statistics: Image widths have a median of 1,385 pixels.
  • Image Statistics: Image widths have a mean of 1,626 pixels.
  • Image Statistics: GSD values have a median of 0.30 m/pixel.
  • Image Statistics: GSD values have a mean of 2.02 m/pixel.
  • Image Statistics: The GSD median–mean gap is caused by a small number of coarse-resolution outliers.
  • Image Statistics: Most images are very-high-resolution Earth observation tiles.

Annotation Statistics

OVEarth-Bench combines broad vocabulary coverage with diverse referring and reasoning annotations across 172 categories. Its box-based subset is dominated by single-object annotations and very small targets, while language queries are generally concise.

  • Vocabulary and query coverage: The benchmark contains 1,067 positive phrases, 2,168 negative phrases, 732 referring expressions, and 1,056 reasoning queries across 172 categories.After within-polarity deduplication, the phrases yield 425 unique positive and 1,029 unique negative expressions; 1,346 unique vocabulary strings occur after accounting for 108 cross-polarity overlaps.
  • Box and object statistics: The detection subset contains 4,810 bounding boxes, with a median of 1 and mean of 8.2 boxes per annotation.The distribution is highly skewed: 14 annotations contain more than 40 boxes.
  • Box and object statistics: Small targets dominate the box annotations: 4,414 boxes (91.8%) occupy less than 1% of the image area, while 17 boxes (0.4%) occupy more than half.Most open-vocabulary phrases contain one to three words.

C Evaluation Metrics · Segmentation Metrics

This section defines the segmentation metrics used in OVEarth-Bench and introduces the open-source ovearth-eval package for reproducible evaluation. Metrics are computed over positive queries, while negative queries are handled separately through presence detection.

  • C Evaluation Metrics: The open-source Python package ovearth-eval provides reproducible comparisons and formal definitions for all reported metrics.The package accompanies the benchmark evaluation protocol.
  • Segmentation Metrics: Localization metrics are computed only on positive queries with non-empty ground-truth foreground sets.For each positive sample, Gi denotes the ground-truth foreground pixels and Pi the predicted foreground pixels.
  • Segmentation Metrics: Missing, null, empty, or all-zero predictions are represented by empty Pi and assigned zero precision, recall, and IoU.This convention ensures failed predictions receive zero overlap-based scores.
  • Segmentation Metrics: Per-query precision, recall, and IoU are averaged across all N positive query samples to obtain ma-P, ma-R, and ma-IoU.Q-mIoU denotes the ma-IoU reported in the main table.
  • Segmentation Metrics: Negative queries are excluded from mask-overlap aggregation and evaluated separately through presence detection.Thus, segmentation overlap metrics focus exclusively on positive queries.
  • Segmentation Metrics: The benchmark additionally reports mi-IoU by pooling intersection and union areas over all positive query samples.This provides an aggregate overlap measure across positive samples rather than averaging per-query ratios.

Robustness to Evaluation Aggregation … Query-Form Effects

The evaluation shows that model rankings are robust to aggregation choices and category sampling, while query form does not impose a uniform difficulty ordering. MLLM-based methods dominate the strongest results, whereas EO-specific methods rarely do.

  • Robustness to Evaluation Aggregation: Model rankings remain stable across query-, annotation-, and category-macro IoU aggregation, preserving the main comparisons despite local rank changes.The highest-ranked model is unchanged across all three tasks, and category aggregation does not alter the overall comparison.
  • Detection Metrics: Micro detection metrics match instances across predicted and ground-truth boxes at IoU thresholds, pooling true positives, false positives, and false negatives.The protocol reports mi-P0.5, mi-R0.5, mi-F10.5, and mi-F10.5:0.95, with missing predictions counted as false negatives.
  • Presence Detection Metric (MCC): MCC evaluates binary presence predictions for imbalanced positive and negative queries, returning null when denominator marginals make the metric undefined.The task contains 1,067 positive and 2,168 negative queries, and localization outputs determine present-versus-absent predictions.
  • Category-Subset Stability: 150 categories recover the full-benchmark ranking with median Spearman correlation 1.00 (0.99–1.00) and median top-five recovery 1.0 (0.8–1.0).With five categories, the corresponding medians are 0.84 (0.58–0.94) and 0.6 (0.2–0.8), increasing as category coverage grows.
  • Model and Variant Comparisons: Direct MLLM segmentation methods provide eight of the ten best results for every query type, while MLLM-based methods provide nine of ten vocabulary results and all ten referring and reasoning results.Only one EO-specific method enters the vocabulary top ten, and none enters the referring or reasoning top ten.
  • Query-Form Effects: Query-form rankings remain broadly correlated across shared models and annotation targets, but direct score differences vary in sign across models.The analysis controls for model overlap and shared spatial targets to separate query-form effects from model and target differences.
  • Query-Form Effects: Referring scores exceed vocabulary scores for 20 of 31 models and fall below them for 11, while reasoning scores exceed vocabulary scores for 17 of 30 and fall below them for 13.Reasoning scores exceed referring scores for 12 of 31 models and fall below them for 19, showing that no query form is uniformly easier or harder.

Comparison of Mask-Generation Paradigms

Mask-generation paradigms show no uniform winner: the highest score comes from a grounding-based model, while the second-best grounding model trails its segmentation-based counterpart. On non-instanceable targets, results likewise do not reveal a consistent segmentation advantage.

  • Comparison of Mask-Generation Paradigms: Rex-Omni+SAM leads the evaluated paradigms, but LocateAnything+SAM trails the corresponding second-best segmentation-based model across all three query types.Rex-Omni+SAM exceeds the best segmentation-based models by 5.44, 1.11, and 1.88 points, whereas LocateAnything+SAM trails by 7.87, 2.99, and 4.12 points on vocabulary, referring, and reasoning queries, respectively.
  • Effect of target instanceability: On non-instanceable targets, the four selected models do not show a consistent segmentation advantage.Grounding-based models score higher on vocabulary and referring queries, while reasoning-query scores overlap closely.
  • Effect of target instanceability: 53.88% and 47.81% are the vocabulary ma-IoUs for Rex-Omni+SAM and LocateAnything+SAM, versus 48.96% and 49.45% for SAMTok-Qwen3-VL-4B-co and X2SAM.On referring queries, the grounding-based models score 58.55% and 54.44%, above the segmentation-based models at 51.10% and 48.05%.
  • Possible explanations for the grounding ceiling: Grounding models may benefit from substantially larger region-language training corpora than many mask-decoding MLLMs.LocateAnything uses 12M images, more than 138M language queries, and 785M boxes, while Rex-Omni uses approximately 22M images during supervised fine-tuning.
  • Possible explanations for the grounding ceiling: Bounding-box prediction may align more closely with autoregressive MLLMs’ native token interface than dense-mask prediction.Dense masks typically require an additional mask decoder, mask tokenizer, or pixel-level feature alignment, introducing specialized output interfaces and additional training requirements.

Category Overlap with Other Benchmarks · E Detection Subset: Excluded Categories

OVEarth-Bench categories are broadly aligned with established EO benchmarks while extending their coverage beyond represented concepts. Its detection and OBB subsets exclude area-type or linear features unsuitable for instance-level bounding-box evaluation, along with four post-hoc annotation exclusions.

  • Category Overlap with Other Benchmarks: Eight benchmark inventories are manually aligned to OVEarth-Bench using exact/synonym, hierarchy-compatible, or unmatched categories.Part–whole relations are generally not treated as matches.
  • Category Overlap with Other Benchmarks: Non-specific catch-all and background labels, including other-vehicle and clutter/background, are excluded from semantic-overlap denominators.This exclusion applies to the overlap calculation rather than the category alignment itself.
  • Category Overlap with Other Benchmarks: All categories in six of the eight benchmarks have an OVEarth-Bench match.FAIR1M and xView are the two exceptions.
  • Category Overlap with Other Benchmarks: 31/34 (91%) and 53/60 (88%) are the respective OVEarth-Bench match rates for FAIR1M and xView.These figures are reported for FAIR1M and xView, respectively.
  • Category Overlap with Other Benchmarks: 36 of 172 OVEarth-Bench categories have exact/synonym matches, 54 have hierarchy-compatible matches, and 82 are absent from the selected benchmarks.The reverse comparison indicates broad retention of established EO concepts alongside substantial extension.
  • E Detection Subset: Excluded Categories: Twelve categories are excluded from detection and OBB evaluation because they are area-type or linear features that are non-instanceable or unsuitable for bounding-box evaluation.The categories are pipeline, common cropland, vineyard, arbor, grass, bareland, ice, asphalt road, mountain road, railway, port container, and river.
  • E Detection Subset: Excluded Categories: Four additional annotations—heliostat ×2, fish raft house ×1, and moat ×1—are excluded after post-hoc review.The passage states that the review determined their instances w…; the supplied text is truncated.

F Model Descriptions and Experimental Settings … Inference Settings

The paper evaluates a broad set of general, multimodal, detection, grounding, and remote-sensing open-vocabulary models under a unified zero-shot inference protocol. Model descriptions emphasize their architectural designs, training strategies, and task-specific adaptations, while inference uses official settings without adaptation or threshold selection.

  • General Open-Vocabulary Segmentation Models: The evaluation includes CLIP-derived dense prediction methods, including training-free adaptations and models that modify attention, transfer structural features, or use fine-tuning.CLIP performs zero-shot localization by matching image patches with text embeddings, while methods such as MaskCLIP, ClearCLIP, ProxyCLIP, CASS, GEM, and SCLIP alter dense prediction mechanisms.
  • General Open-Vocabulary Segmentation Models: The benchmark also evaluates SAM 3 and an agent that converts referring or reasoning queries into noun phrases for iterative mask generation and selection.Qwen2.5-VL-7B-Instruct proposes non-repeated phrases, invokes SAM 3, and inspects returned masks against the original image and query.
  • Multimodal Large Language Models: MLLM-based segmentation methods span decoder-guided, mask-token, memory-based, mask-tokenization, and text-generation architectures evaluated with multiple backbone variants.The evaluated methods include LISA, Sa2VA, UniPixel, SAMTok, X2SAM, SELF1E, ALToLLM, and Text4Seg, with training ranging from public segmentation data to approximately 5M mask-text instruction pairs.
  • Detection and Grounding Models: Detection and grounding baselines combine vision-language fusion, generative box decoding, or unified coordinate-token prediction for open-vocabulary localization.GroundingDINO, YOLO-World2, LocateAnything, and Rex-Omni use distinct combinations of detector backbones, language models, pretraining data, and coordinate representations.
  • RS Open-Vocabulary Models: Remote-sensing-specific methods address semantic segmentation, referring and reasoning segmentation, detection, domain transfer, and attribute-based representation learning.The set includes GSNet, Pi-Seg, SegEarth variants, SegEarth-R models, SkySense-O, GeoPixel, RemoteSAM, UniGeoSeg, LAE-DINO, OpenRSD, and SLIP-RS.
  • Inference Settings: All methods are evaluated without training, adaptation, threshold selection, or exposure to ground-truth categories, masks, or boxes at inference.Queries are submitted independently, predictions are mapped to original-image coordinates, selected masks are unioned for segmentation, and detector instances remain separate for box evaluation.
  • Inference Settings: CLIP-based dense prediction models use prescribed backbones, image resizing, and sliding-window settings, while detectors retain official confidence and NMS thresholds.CAT-Seg, GSNet, and Pi-Seg use ViT-L/14 with 1008-pixel resizing and 336-pixel windows; several other CLIP models use ViT-B/16 with 896-pixel resizing and 224-pixel windows.
  • Inference Settings: Experiments run on NVIDIA A100 or 4090 GPUs using default hyperparameters from official implementations or papers.The reported inference configuration specifies hardware and implementation-derived settings rather than task-specific tuning.

G Discussion and Limitations

OVEarth-Bench shows that broader category coverage stabilizes model comparisons, while query-form analysis reveals model-specific behavior and weak alignment between positive localization and absent-concept rejection. MLLM-based methods lead especially on referring and reasoning queries, motivating EO-specialized enhancements while highlighting taxonomy, sampling, reasoning, memorization, and localization limitations.

  • Findings: Broader category coverage makes model comparisons less sensitive to which specific categories are selected.Query-form analysis also reveals model-specific variations despite broadly similar performance rankings.
  • Findings: MLLM-based methods achieve most of the strongest results, particularly on referring and reasoning queries requiring compositional spatial understanding or functional inference.The passage attributes this descriptively to multimodal pretraining and instruction tuning that provide transferable visual–semantic and linguistic representations beyond narrow EO supervision.
  • Future directions: Future methods should combine general MLLMs’ semantic breadth with EO-specific spatial competence through multiscale encoding, orientation-aware decoding, and EO-focused instruction tuning.The proposed tuning should cover long-tail categories, spatial relations, functional descriptions, and hard negative queries without eroding general language transferability.
  • Limitations: The taxonomy is not exhaustive, geographic sampling is uneven, and reasoning queries cover functions and visible context rather than the full scope of geospatial reasoning.New images reduce direct benchmark memorization risk but cannot guarantee complete absence from foundation-model pretraining corpora.
  • Limitations: The evaluation focuses on categories rather than localization across taxonomy levels, and HBB-based detection comparisons limit conclusions about orientation-aware localization.Only a limited subset of evaluated methods natively produces OBBs, while data-collection costs make some categories difficult to represent.
Loading 2607.27278v2…