OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation

Introduction

• EO’s broad semantic space makes open-vocabulary localization important, but conventional datasets and limited query forms leave existing benchmarks unable to assess genuine capability comprehensively (Li et al. 2025a; Lam et al. 2018; Wang et al. 2021; Zhang et al. 2021; Xia et al. 2023; Zhan, Xiong, and Yuan 2023; Dong et al. 2024; Yuan et al. 2024; Li et al. 2025c; Xin et al. 2026).The missing query coverage includes vocabulary phrases, referring expressions, and reasoning queries.Source: Introduction · p. 1
• OVEarth-Bench uses 172 hierarchical categories, 1,346 vocabulary strings, positive and negative queries, diverse language forms, and verified masks with reviewed HBB and OBB annotations.It is built from newly collected EO images and supports mask and box localization under a unified zero-shot protocol.Source: Introduction · pp. 1–2
• Across 49 general and EO-specific model variants, rankings were sensitive to category composition, MLLM-based methods led, and EO specialization alone gave no consistent advantage.Accurate localization also did not necessarily mean reliable rejection of absent concepts; the findings inform future method design.Source: Introduction · pp. 1–2
• Combining MLLMs’ broad generalization capabilities with richer EO-specific corpora is presented as a promising direction.EO images have unique characteristics, motivating this combination.Source: Introduction · p. 2

Remote-sensing Localization Benchmarks

• Traditional remote-sensing datasets cover semantic segmentation, box localization, and instance segmentation, but remain tied to predefined taxonomies despite EO-specific challenges such as scale and orientation variation.Representative datasets include those of Rottensteiner et al. 2012, Wang et al. 2021, Xia et al. 2023, Cheng et al. 2014, Xia et al. 2018, Lam et al. 2018, Li et al. 2020, Zhang et al. 2021, Sun et al. 2022, and Waqas Zamir et al. 2019.Source: Remote-sensing Localization Benchmarks · p. 2
• Language-conditioned benchmarks add visual grounding, referring segmentation, and reasoning descriptions, but reuse public images and assess recognition, rejection, spatial reference, and reasoning separately.Examples include RSVGD, RISBench (Dong et al. 2024), RRSIS-D/RefSegRS (Yuan et al. 2024), EarthReason (Li et al. 2025c), and LaSeRS (Xin et al. 2026); image reuse limits source diversity and risks pre-training overlap (Zhou et al. 2026b).Source: Remote-sensing Localization Benchmarks · p. 2

Open-Vocabulary Understanding Methods

• General open-vocabulary methods adapt language-aligned representations for pixel prediction, combine language understanding with mask generation, or localize text-specified regions.Examples include CAT-Seg (Cho et al. 2024), ClearCLIP (Lan et al. 2024a), ProxyCLIP (Lan et al. 2024b), LISA (Lai et al. 2024), and detection and grounding methods (Liu et al. 2024).Source: Open-Vocabulary Understanding Methods · p. 2
• EO-specific methods target overhead imagery and varied queries, yet conventional or task-specific evaluations leave category coverage and query diversity insufficiently characterized, motivating OVEarth-Bench’s unified zero-shot protocol.Examples span open-vocabulary segmentation and vision–language pretraining, category-rich localization, and referring or reasoning-guided segmentation.Source: Open-Vocabulary Understanding Methods · p. 2

2. Image Collection

• The image-collection materials name QGIS, Google Earth, Amap, and Search.Source: §2 · p. 3
• The National land-use standard is identified in the image-collection materials.Source: §2 · p. 3
• One listed category group is Business, Service & Residential.Source: §2 · p. 3
• Another listed category group is Industrial, Mining & Storage.Source: §2 · p. 3
• The collection covers 172 categories described as reliably discriminable in optical remote-sensing imagery.Source: §2 · p. 3

• Query generation pairs an LLM-generation stage with model verification, checking granularity and matching, negative absence, and spatial grounding.The pipeline also excludes non-instances.Source: §4 · p. 3
• Missed instance-able objects are annotated for detection.Source: §4 · p. 3
• The benchmark includes positive and negative phrases, referring expressions, and reasoning queries.Source: §4 · p. 3
• Human mask annotation and review feed instance review and box derivation, including HBB and OBB boxes for detection and grounding.Source: Figure 3 · p. 3, §4 · p. 3

OVEarth-Bench

• The benchmark jointly evaluates category and query diversity on newly collected images with mask and box annotations.Its construction includes taxonomy design, image collection, mask annotation, query construction, and box derivation.Source: OVEarth-Bench · p. 3

Taxonomy and Category Design

• The category hierarchy is defined before image collection to guide sampling by an explicit semantic scope rather than annotation convenience.It draws on three complementary sources, though the supplied passage does not identify them fully.Source: Taxonomy and Category Design · p. 3

Image Collection

• All images are newly collected through POI retrieval and patch extraction, covering six continents with some same-location seasonal pairs.Sampling is denser in Asia, Europe, and North America, and sparser where suitable high-quality imagery was harder to obtain.Source: Image Collection · p. 3

Mask Annotation

• Annotators create polygon masks, using SAM 2 for simple targets and manual annotation for complex, concave, or ambiguous targets.A manager reviews category assignment, boundaries, and missed instances, returning uncertain cases for correction.Source: Mask Annotation · p. 4

Query Generation Pipeline

• A two-stage LLM pipeline generates positive, negative, referring, and reasoning queries, screens candidate errors, and sends all language annotations for manual review.Stage 2 flagged 3.9% of negative candidates as present and corrected 28.6% of referring expressions; expert reviewers changed or removed 57.3% of referring expressions and modified 16.0% of negative phrases.Source: Query Generation Pipeline · p. 4

HBB and OBB Annotations

• For instanceable targets, the benchmark derives horizontal boxes from polygon envelopes and minimum-area oriented boxes using rotating calipers.Area-type and linear categories, including cropland and road sections, are excluded as non-instanceable.Source: HBB and OBB Annotations · p. 4

Evaluation Protocol

• The three query types define open-vocabulary, referring, and reasoning tasks, evaluated separately with masks, horizontal boxes, or oriented boxes.The protocol reports macro segmentation metrics to reduce large-target bias and uses MCC to assess presence/absence discrimination across positive and negative vocabulary queries.Source: Evaluation Protocol · pp. 4–5

Dataset Statistics

• OVEarth-Bench contains 590 mask annotations from 520 images across 172 categories, plus 4,810 detection boxes.It also includes 425 unique positive and 1,029 unique negative vocabulary phrases, 732 referring expressions, and 1,056 reasoning queries; 91.8% of boxes cover less than 1% of an image.Source: Dataset Statistics · p. 4, Figure 5 · p. 5

Experimental Setup

• The zero-shot evaluation covers 49 model variants, with all providing vocabulary outputs and 32 and 31 providing referring and reasoning outputs.Detection outputs are converted to masks with SAM2.1-L; reported box results use HBB. Methods use default hyperparameters, with no benchmark training, fine-tuning, or threshold selection.Source: Experimental Setup · p. 5

Results and Analysis

• 38.75%, 41.32%, and 37.96% ma-IoU are the vocabulary, referring, and reasoning segmentation maxima, all achieved by Rex-Omni+SAM.The maxima are reported for OVEarth-Bench’s zero-shot segmentation evaluation.Source: Table 2 · p. 5, Results and Analysis · pp. 5–6, Table 3 · p. 6
• MLLMs account for 9 of 10 vocabulary and all top-ten referring and reasoning ma-IoU results, while only one EO-specific variant reaches the vocabulary top ten and none reaches the other two.The best EO-specific results trail task maxima by 7.07–15.31 segmentation points and 7.46–23.58 detection points; related-baseline gains occur but are inconsistent. Broader, heterogeneous pretraining may help MLLMs with long-tail categories and spatial or functional descriptions.Source: Results and Analysis · p. 6
• At 5, 20, and 50 categories, median rank correlations are 0.84, 0.95, and 0.98; five-category top-five recovery is 0.6, rising to 0.8 at 20.The results show that narrow category sets make model comparisons dependent on which concepts are sampled.Source: Results and Analysis · p. 6, Figure 6 · p. 7
• Rex-Omni+SAM exceeds the best direct segmentation model by 5.44, 1.11, and 1.88 ma-IoU points, while LocateAnything+SAM trails the second-best by 7.87, 2.99, and 4.12.The crossed pattern establishes no general advantage for grounding-based pipelines or direct segmentation; training scale, backbone, and supervision format jointly impact performance.Source: Results and Analysis · p. 7
• SELF1E-8B gains 7.52–10.12 ma-IoU points over SELF1E-2B; Sa2VA-Qwen3-VL-4B gains 1.08, 5.46, and 2.82, while UniPixel-7B trails UniPixel-3B by 3.46–6.04.Because variants may differ beyond parameter count, larger published variants do not consistently perform better. Query-form effects also vary: SegEarth-R2 declines and UniGeoSeg improves on referring expressions, while reasoning descriptions help some models and reduce others; overall rankings remain similar.Source: Results and Analysis · p. 7
• Qwen3-VL-4B+SAM reaches 0.60 MCC, while every other model with defined MCC scores at most 0.35; vocabulary ma-IoU and MCC rankings have Spearman ρ = 0.107.Accurately localizing present concepts does not imply reliable rejection of absent concepts.Source: Results and Analysis · p. 7
• 24.53%, 35.56%, and 26.12% mi-F10.5 and 14.23%, 21.47%, and 14.91% mi-F10.5:0.95 are the box-localization maxima across those query types.The values correspond respectively to vocabulary, referring, and reasoning queries.Source: Results and Analysis · p. 6, Table 5 · p. 7

Conclusion

• MLLM-based methods achieve the strongest overall performance across query types.The conclusion identifies this as a key experimental finding.Source: Conclusion · p. 7
• EO-specific models can beat comparable general baselines but still trail the strongest general methods, while grounding- and segmentation-based methods both perform competitively.The conclusion says these results demonstrate the potential of both grounding and segmentation approaches.Source: Conclusion · p. 7
• OVEarth-Bench unifies zero-shot evaluation of category and query diversity with a broad taxonomy, varied queries, and spatial annotations for recognition, reasoning, and localization.The benchmark is presented as a foundation for future model development.Source: Conclusion · p. 7

A Query Generation Prompt Templates

• The query-generation pipeline uses GPT-5-mini with structured JSON output.Source: A Query Generation Prompt Templates · p. 10

Stage 1: Generation

• Stage 1 asks an expert remote-sensing annotator to generate two vocabulary phrases, two category-naming referring expressions, two implicit function-or-purpose prompts, and 3–5 similar absent negatives.The target is indicated by a red outline in satellite or aerial imagery; positive phrases should avoid forced attributes.Source: Stage 1: Generation · p. 10

Stage 2: Verification

• Stage 2 checks whether negatives are present and whether referring expressions are factually correct, correcting inaccurate or overly specific location claims into safer expressions.Review includes spatial relations and direction words, with careful inspection for small or densely packed objects.Source: Stage 2: Verification · p. 10
• Stage 2 returns each negative’s ABSENT or PRESENT status with a reason, quality-control results for each referring expression, and exactly two final referring expressions.Expressions are marked valid or corrected, with original phrase, issue, and corrected phrase recorded.Source: Stage 2: Verification · p. 10

Image Statistics

• Image widths have a median of 1,385 pixels and mean of 1,626, while GSD has a median of 0.30 m/pixel and mean of 2.02, reflecting coarse-resolution outliers.Most images are very-high-resolution tiles, and many are close to square crops.Source: Image Statistics · p. 10

Annotation Statistics

• OVEarth-Bench contains 1,067 positive and 2,168 negative phrases, 732 referring expressions, and 1,056 reasoning queries, yielding 1,346 unique vocabulary strings across 172 categories.After deduplication within each polarity, there are 425 unique positive and 1,029 unique negative phrases; 108 strings occur with both polarities.Source: Annotation Statistics · p. 10, Figure 8 · p. 11
• Most annotations support multiple query types: 478 (81%) include open-vocabulary, referring, and reasoning queries, while only 5 (1%) contain open-vocabulary phrases alone.The other reported combinations are open-vocabulary and reasoning queries in 90 annotations (15%), and open-vocabulary and referring queries in 17 (3%).Source: Annotation Statistics · pp. 10–11
• The detection subset contains 4,810 bounding boxes, with a median of 1 and mean of 8.2 boxes per annotation; 14 annotations contain more than 40 boxes.The box-count distribution is highly skewed.Source: Annotation Statistics · p. 11, Figure 8 · p. 11
• Small targets dominate: 4,414 boxes (91.8%) occupy less than 1% of image area, while only 17 (0.4%) occupy more than half.The OBB rotation-angle distribution also reflects the arbitrary orientations characteristic of remote-sensing objects.Source: Annotation Statistics · p. 11, Figure 8 · p. 11
• Language annotations are concise: most open-vocabulary phrases have one to three words, while referring expressions average 9.2 words and reasoning queries 11.6.The annotation statistics also report OBB rotation angles, illustrating the arbitrary orientations characteristic of remote-sensing objects.Source: Annotation Statistics · p. 11, Figure 8 · p. 11

C Evaluation Metrics; Segmentation Metrics; Robustness to Evaluation Aggregation

• The released ovearth-eval package formalizes mask metrics over positive queries, and query-, annotation-, and category-level aggregation produces highly consistent model rankings.Query-level averaging measures performance across language expressions; annotation-macro IoU weights annotated targets equally, and category-macro IoU weights represented categories equally. The leading model is unchanged across the three tasks, and the broad model comparison is not altered by aggregation.Source: C Evaluation Metrics · p. 11, Segmentation Metrics · p. 11, Robustness to Evaluation Aggregation · pp. 12–13

Detection Metrics; Presence Detection Metric (MCC)

• Detection uses maximum-cardinality one-to-one box matching and reports micro precision, recall, and F1 at IoU 0.5 plus mean micro F1 across thresholds; presence detection uses MCC for imbalanced positive and negative queries.Missing boxes count as false negatives; MCC ranges from −1 to 1 and is null when a denominator marginal is zero.Source: Detection Metrics · pp. 13–14, Presence Detection Metric (MCC) · p. 14

Category-Subset Stability

• With five categories, median rank correlation is 0.84 and top-five recovery is 0.6, versus 0.99 and 1.0 with 100 categories, showing narrow subsets can change leading-model conclusions.The analysis samples 1,000 subsets at each size; at 20 categories the medians are 0.95 and 0.8, while at 150 they reach 1.00 and 1.0.Source: Category-Subset Stability · p. 14

Model and Variant Comparisons

• Figure 10 compares the ten best-performing segmentation models by output interface and EO-specific status, and shows full-benchmark ma-IoU differences between larger and smaller variants across query types, with positive values favoring larger variants.Source: Figure 10 · p. 15

Query-Form Effects

• No query form is uniformly easier: referring scores exceed vocabulary for 20 of 31 models, but reasoning exceeds referring for only 12 of 31.From vocabulary to referring, SegEarth-R2 drops 7.43 ma-IoU points, while UniGeoSeg and LocateAnything+SAM rise 7.80 and 8.12 points.Source: Query-Form Effects · p. 15

Comparison of Mask-Generation Paradigms

• Rex-Omni+SAM leads the grounding-based paradigm, exceeding the best segmentation-based model by 5.44, 1.11, and 1.88 ma-IoU points across vocabulary, referring, and reasoning, respectively.The second-best grounding model trails its segmentation-based counterpart by 7.87, 2.99, and 4.12 points, so the highest observed score does not establish a general paradigm advantage.Source: Comparison of Mask-Generation Paradigms · pp. 15–16, Table 11 · p. 16
• On non-instanceable targets, the selected grounding models beat the selected segmentation models on referring queries, while reasoning scores overlap closely; every selected model scores higher on this subset than on remaining targets.The comparison covers only 12 area-type or linear categories and two selected models per paradigm, so it is descriptive rather than a paradigm-level result.Source: Figure 11 · p. 16, Comparison of Mask-Generation Paradigms · p. 17

Category Overlap with Other Benchmarks; E Detection Subset: Excluded Categories

• OVEarth-Bench matches all categories in six of eight prior datasets, while 82 of its 172 categories are unrepresented by those benchmarks.The detection subset excludes 12 area-type or linear categories and four annotations deemed unreliable at the available resolution, leaving 4,810 boxes for detection scoring.Source: Category Overlap with Other Benchmarks · p. 18, E Detection Subset: Excluded Categories · p. 18

General Open-Vocabulary Segmentation Models

• General segmentation methods range from CLIP’s zero-shot patch–text matching (Radford et al. 2021) to dense-prediction adaptations including MaskCLIP (Ding, Wang, and Tu 2022), CAT-Seg (Cho et al. 2024), ClearCLIP (Lan et al. 2024a), ProxyCLIP (Lan et al. 2024b), and CASS (Kim et al. 2025).These methods use frozen CLIP features in different ways; CAT-Seg is fine-tuned on COCO-Stuff, while the other cited adaptations are training-free.Source: General Open-Vocabulary Segmentation… · p. 18
• GEM (Bousselham et al. 2024) and SCLIP (Wang, Mei, and Yuille 2024) alter CLIP self-attention for dense prediction, while SAM 3 (Carion et al. 2025) supports text-prompted concept segmentation.For referring and reasoning queries, SAM3-Agent uses Qwen2.5-VL-7B-Instruct to propose phrases and select or reject SAM 3 masks, with a 100-round cap; selected masks are unioned.Source: General Open-Vocabulary Segmentation… · pp. 18–19

Multimodal Large Language Models

• The evaluated multimodal models connect language instructions to segmentation through mechanisms including LISA’s <SEG>-to-SAM decoder (Lai et al. 2024), Sa2VA’s instruction tokens (Yuan et al. 2025), and UniPixel’s object memory bank (Liu et al. 2026).The Sa2VA variants use Qwen2.5-VL or Qwen3-VL backbones, and the evaluated UniPixel variants have 3B and 7B parameters.Source: Multimodal Large Language Models · p. 19
• Other evaluated multimodal approaches include SAMTok’s discrete mask tokens, X2SAM’s dual-branch architecture (Wang et al. 2026b), SELF1E’s decoder-free [SEG] token, ALToLLM’s adaptive-length mask tokens (Wang et al. 2026c), and Text4Seg’s text-generation formulation (Lan et al. 2025).Their training includes mask-text instruction tuning, joint image/video training, reinforcement learning, or LoRA fine-tuning, depending on the model.Source: Multimodal Large Language Models · p. 19

Detection and Grounding Models

• Detection and grounding methods include zero-shot GroundingDINO (Liu et al. 2024), YOLO-World2, generative LocateAnything (Wang et al. 2026d), and coordinate-token MLLM Rex-Omni (Jiang et al. 2026).LocateAnything is trained on over 138M queries and 785M boxes; Rex-Omni predicts spatial locations as discrete coordinate-token sequences.Source: Detection and Grounding Models · p. 20

RS Open-Vocabulary Models

• Remote-sensing segmentation approaches include GSNet (Ye, Zhuge, and Zhang 2025), Pi-Seg (Li et al. 2026), SegEarth-OV and SegEarth-OV3 (Li et al. 2025b; Li et al. 2025d), SegEarth-R1 (Li et al. 2025c), and SkySense-O.Their approaches range from training-free CLIP or SAM 3 adaptations to trained models; Pi-Seg uses semantic perturbation modules, while SegEarth-OV3 fuses semantic and instance predictions with a presence score.Source: RS Open-Vocabulary Models · pp. 20–21
• Other remote-sensing models include GeoPixel (Shabbir et al. 2025), UniGeoSeg (Ni et al. 2026), LAE-DINO (Pan et al. 2025), OpenRSD (Huang et al. 2025), and SLIP-RS (Wang et al. 2026a).They cover pixel grounding, instruction-driven segmentation, open-vocabulary detection, and attribute-grounded pretraining; OpenRSD predictions are converted from oriented boxes to axis-aligned envelopes for the paper’s common HBB comparison.Source: RS Open-Vocabulary Models · p. 21

Inference Settings

• The evaluation protocol tests each query independently without training, adaptation, threshold selection, or access to ground-truth targets, and defines how mask and box predictions are scored.Multiple masks for a query are unioned for segmentation; detector instances remain separate for box scoring, and missing or empty outputs count as empty predictions.Source: Inference Settings · p. 21

Discussion

• Broader category coverage makes model comparisons less sensitive to category selection, while query forms vary by model and positive-query localization correlates weakly with rejecting absent concepts.The findings motivate jointly evaluating category breadth, query diversity, localization accuracy, and presence prediction.Source: G Discussion and Limitations · p. 22
• MLLM-based methods achieve most of the strongest results, especially on referring and reasoning queries, but varied training data, scale, interfaces, modules, and inference budgets prevent causal attribution.The authors suggest multimodal pretraining and instruction tuning may provide transferable visual-semantic and linguistic representations; they characterize this explanation as descriptive, not causal.Source: G Discussion and Limitations · p. 22

Future directions

• Promising directions combine general MLLM semantics with EO spatial competence through high-resolution multiscale encoding, orientation-aware decoding, and EO-focused instruction tuning.Tuning should cover long-tail categories, spatial relations, functional descriptions, and hard negatives without eroding general language transferability; presence prediction should be calibrated with explicit negative supervision and uncertainty-aware evaluation.Source: G Discussion and Limitations · p. 22
• Controlled studies should align MLLM backbone, training data, and inference budget while varying EO adaptation and output representations to isolate performance gains and when mask generation suits box grounding plus segmentation.Source: G Discussion and Limitations · p. 22

Limitations

• The taxonomy is not exhaustive, geographic sampling is uneven, reasoning queries cover functions and visible context rather than full geospatial reasoning, and new images cannot guarantee absence from pretraining.Source: G Discussion and Limitations · p. 22
• Evaluation covers categories rather than taxonomy levels and relies mainly on HBB comparisons, while sparse samples for some categories can make category-level results sensitive to individual examples.Only a limited subset of evaluated methods natively produces OBBs; future benchmark releases are intended to address or mitigate these limitations.Source: G Discussion and Limitations · p. 22