PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoRefer-Bench: A Benchmark from Referring Pixels to Verifiable Geospatial Reasoning

Shuaishuai Cao, Min Huang, Meng Tang, Xuan Liu, Youjin Wang, Hui Lin

arXiv:2609.29541v1cs.CVcs.AI

TL;DR

Overhead referring segmentation requires resolving spatial relations among similar objects, but overlap-based benchmarks cannot verify that resolution. GeoRefer-Bench pairs executable query programs with metric scene graphs and scores exact instance-set matches. Across evaluated models, performance falls substantially on deeper queries, with the strongest reaching 60.5 at two-hop reference.

  • Problem

    In overhead imagery, queries can refer to sets of similar objects or no object, while overlap-based scores cannot verify that the stated spatial relation was resolved.

    Source: §1 · pp. 1–2

  • Method

    GeoRefer-Bench represents queries as executable logical forms over metric scene graphs and uses Exact Query Success to test whether predicted instances match the derived answer set.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    The strongest evaluated system reaches 60.5 at two-hop reference, while most evaluated systems fall below 5 at that reasoning depth.

    Source: §8 · p. 15

  • Takeaways & Limitations

    EQS exposes failures in instance selection, answer cardinality, and rejection that overlap can reward without certifying relational grounding.

    Source: §8 · p. 14

    The model comparison is one pass on one benchmark with one checkpoint per system and uses the primary scene-disjoint split, so it does not establish architecture-general or transfer performance.

    Source: §8 · p. 14

Abstract

from arXiv · show

Referring segmentation in overhead imagery is inherently relational: a query may ask for the buildings north of the road or the pond closest to a residential area, so the correct referent can contain one object, several objects, or none. Existing benchmarks mainly score mask overlap, which cannot verify whether a model actually resolved the stated spatial relation. We introduce GeoRefer-Bench, a benchmark for verifiable geospatial referring segmentation. Each query is represented by an executable logical form over a metric scene graph, and predictions are evaluated with Exact Query Success (EQS), which is satisfied only when the returned instance set exactly matches the set denoted by the query. GeoRefer-Bench contains 700 whole 2048x2048 UAV scenes (2.94 Gpx) at 12.5 and 25 cm ground sampling distance, 26,217 instances, 142,796 spatial relations, and 20,916 executable queries spanning five reasoning levels. It further includes three paraphrases per query, 24.0% unanswerable queries, 2,477 counterfactual pairs, and five leakage-controlled evaluation splits. An independent audit re-derives object geometry, mask ownership, relation values, query execution, and split provenance, finding zero issues across all 700 scenes. Relation-blind strategies can retain non-trivial mIoU while achieving at most 22.7 EQS overall, showing that overlap alone does not certify relational grounding. Across fifteen current models, the strongest reaches 74.1 EQS but drops from 98.9 at level 1 to 60.5 at level 5, while ten models score below 5 EQS on two-hop queries. GeoRefer-Bench turns geospatial referring segmentation from mask matching into verifiable reference resolution.

1 Introduction

Overhead referring segmentation requires resolving spatial relations among similar objects and returning the exact set they denote, which may contain none. GeoRefer-Bench makes that resolution executable and evaluates instance identity rather than relying on mask overlap.

1 Introduction

  • Overhead queries distinguish similar rooftops or ponds by spatial relations, so the answer is a set of unknown size that may be empty.Unlike a salient foreground object in consumer photographs, the referent may not be visually distinctive on its own.Source: §1 · p. 1
  • Prior work improved appearance-oriented models and allowed multiple or absent targets, but the benchmark measurement remained centered on overlap.The introduction positions GeoRefer-Bench as addressing measurement as well as task cardinality.Source: §1 · p. 2
  • 29.7 mIOU over answerable queries and 36.9 at level 3 reward returning every instance of the named category, while that strategy earns 0.0 at levels 2 and 4 under an exact criterion.This contrast shows that overlap can reward a relation-blind prediction without certifying that the query was resolved.Source: §1 · p. 2
  • Editing a query program varies composition, relation, answerability, or wording while preserving controlled changes to its answer.The construction supports deeper queries, counterfactuals, unanswerable queries, and paraphrases without a new annotation campaign.Source: §1 · p. 3
  • GeoRefer-Bench represents queries as executable logical forms over metric scene graphs and scores whether the returned instance set exactly matches the program’s answer.The benchmark retains individual instances and the selecting program, enabling query answers to be derived rather than drawn.Source: §1 · pp. 2–3
  • The benchmark releases 700 whole scenes, 26,217 instances, 142,796 relations, and 20,916 executable queries, with an independent audit reporting zero issues across all scenes.The audit re-derives geometry, mask ownership, relation values, query execution, and split provenance.Source: §1 · p. 3

2 Related work

Prior referring-segmentation work emphasized appearance and mask overlap, while remote-sensing datasets generally used cropped patches. GeoRefer-Bench instead evaluates whole-scene relation resolution and derives answers from measured geometry.

2 Related work

  • Earlier referring segmentation used single-object annotations and pixel metrics, while generalized referring work relaxed the assumptions of one target and always having a target [Liu et al., 2023].The paper distinguishes relaxing cardinality from verifying which instances a sentence denotes.Source: §2 · p. 3
  • Remote-sensing grounding datasets report mask overlap on patches, whereas geographic relations here are computed in a metric coordinate reference system with its transform released.This includes making north a ground direction and distances metric rather than measured in degrees.Source: §2 · p. 3, Table 1 · p. 4
  • Program-execution benchmarks motivate deriving answers from scene graphs, but GeoRefer-Bench measures its graph from geometry and returns masks rather than answer tokens.A promptable segmenter supplies reviewed boundary hypotheses; geometry supplies the answers.Source: §2 · pp. 3–4

3 Task and metric

The task treats a query as a program over objects and spatial relations in a metric scene graph, with the answer set derived by execution. Exact Query Success tests whether predicted instances match that set, using a read-out for mask-only predictions.

3 Task and metric

  • Each scene contains a 2048×2048 orthorectified image, discrete object masks and geometry, region anchors, and directed relations computed in a metric CRS.Relations include compass directions, distance predicates measured in metres, and containment or intersection; geodetic metadata records GSD, CRS, and north.Source: §3 · p. 4
  • Queries pair an English sentence with a logical form over objects and relations, using category, attribute, relation, conjunction, or two-hop forms.HOP selects an intermediate anchor by one relation before applying a second; proximity is represented with REL or HOP rather than ATTR.Source: §3 · p. 4
  • 5.2% of queries use near or far, whose thresholds are calibrated to scene window size rather than fixed across ground samplings.Excluding these 1,093 queries changes no evaluated rank or score by more than half a point.Source: §3 · p. 4
  • The shared resolver derives answer sets from the logical form and scene graph, permits empty answers, and rejects any released query whose stored answer disagrees with execution.Models receive the image, sentence, and geodetic metadata, but not the logical form or relation graph.Source: §3 · p. 5
  • EQS requires the correct instance count and a one-to-one mask match at IoU ≥ 0.5, with no extra prediction.The matching uses Hungarian assignment; a union of candidates cannot pass because its cardinality is wrong.Source: §3 · p. 5
  • For mask-only systems, the read-out maps predicted foreground to catalogued instances by coverage, while any nonempty foreground fails an unanswerable query.Coverage is used so a single blob across several buildings can count as multiple instances; rejection accuracy tests whether unanswerable queries receive an empty prediction.Source: §3 · p. 5
  • The scorer reports diagnostic metrics alongside EQS, and separately tracks query programs, sentence realizations, and paraphrase consistency.Unless a protocol specifies otherwise, models are scored on one canonical realization per logical query; the other two realizations assess consistency under paraphrase.Source: §3 · p. 5

4 Constructing the benchmark

GeoRefer-Bench is constructed from reviewed whole-scene instances, a metric scene graph, and executable queries whose answers are checked against the graph. Its scenes, controls, and split protocols are designed to test reference resolution across multiple spatial reasoning levels.

Construction pipeline

  • Five construction stages cover scene selection, expert-reviewed instance annotation, metric graph construction, query generation, and verification of geometry, answers, and split provenance.Queries are retained only when executing their logical forms reproduces their stored answers; models propose boundaries but do not determine instance membership.Source: §4 · p. 6

Imagery and splits

  • The release contains 700 whole 2048-pixel scenes at 0.25 m and 0.125 m sampling, with a primary split that prevents native-pixel overlap across train, validation, and test.The two sampling regimes also change physical field of view, and the five protocols include urban-to-rural, cross-scale, reasoning-level, and unseen-pair evaluations.Source: §4 · pp. 6–7

Instance annotation

  • Experts accept, edit, or reject segmenter proposals; discrete targets use whole-scene and tiled passes, while region anchors use whole-scene passes only.Discrete instances are stored in a uint16 index map, while overlapping regions use bit planes; the vocabulary excludes categories not reliably countable or stable at the released scales.Source: §4 · p. 6

Metric scene graph

  • Direction, distance, and topology are computed in a projected metric CRS, with anchors restricted to nameable objects and regions that do not dominate the scene.The graph averages 203.99 edges per scene, with a median of 79.5 and up to 2,289 in a dense village block.Source: §4 · p. 6

Queries and controls

  • Queries span five structural reasoning levels and are checked for answer-count, execution, negativity, and relation-redundancy consistency; controls include unanswerable queries, counterfactual pairs, and three paraphrases per form.Unanswerable queries make up 24.0% of the set, and 2,477 counterfactual pairs flip one relation; unverified LLM rewrites could silently change a sentence’s meaning.Source: §4 · pp. 6–7

Query difficulty

  • Queries are short but require selection among many candidates: 29.2 same-category instances on average, while answers cover 1.58% of the scene on average and 0.33% at the median.Difficulty tiers use candidate count, reasoning level, answer area, and answer cardinality; they comprise 4,373 easy, 8,904 medium, and 7,639 hard queries.Source: §4 · p. 7

5 What an auditor found, and what it now certifies

The released build has zero validator issues across 700 scenes, and an independent audit reports exact provenance for all 700. The audit covers machine-checkable geometry, mask ownership, relations, query execution, and source windows, while category naming and sentence naturalness remain matters for expert review.

Audit findings

  • The released build has 0 validator issues across 700 scenes, and an independent provenance audit found 700/700 scenes exact for the machine-checkable release properties.The audit re-derives object geometry, mask ownership, relation values, query execution, and source-window provenance; category names and sentence naturalness are handled by expert review, not certified by the audit.Source: §5 · p. 7

6 Diagnostic reference points: what relation-blind selection scores

Relation-blind selection strategies can earn substantial overlap scores without reliably returning the denoted instance set. Their results show why exact-set evaluation distinguishes relational selection from category-level mask coverage.

Exact-set diagnostics

  • No relation-blind strategy exceeds 22.7 EQS overall; above L1, the best reaches 30.4 at L2 and 5.5 at L4, versus 24.0 for a permanently silent system.L1 scores 100 because naming the category is exactly what that level requires.Source: §6 · p. 8

Overlap versus exact selection

  • The category-union strategy earns 29.7 mIOU overall and 36.9 at L3, but its L3 exact-set score is 9.2 and it is exactly right zero times at L2 and L4.Overlap can reward enclosing plausible candidates even when the returned instance set is not the one denoted by the query.Source: §6 · p. 8

Failure profiles

  • The largest-instance strategy is strongest overall because a size prior suits superlative queries; one-instance strategies fare better on L5 than L4, whose answers average two objects.The category union scores zero at both L4 and L5, while L5’s single-object answers can be satisfied by a one-object strategy.Source: §6 · p. 8

7 Benchmarking current models

The benchmark evaluates fifteen released models on full-scene relational queries, reporting performance across reasoning depth, answerability, difficulty, and error type. Reasoning-oriented models lead, but instance selection remains substantially harder than category naming.

7 Benchmarking current models

  • Fifteen released models are evaluated on 1,881 queries from 130 scene-disjoint test scenes, using the whole scene and sentence without exposing the logical form.Scores are reported across reasoning depth, difficulty, answerability, and error type.Source: §7 · p. 8

7.1 Compared methods

  • The comparison spans reasoning segmentation, geospatial foundation and grounding models, classical referring segmentation, and generalist segmentation MLLMs, without GeoRefer-Bench fine-tuning.Each system uses its released checkpoint, input size, and expected prompt format.Source: §7.1 · pp. 8–9

7.2 Quantitative results

  • 74.1 EQS overall is SegEarth-R2’s score, while L1 scores are at least 86.0 and L5 scores range from 60.5 to 2.4.Reasoning segmentation leads overall; classical and generalist models score 25.5–38.3 overall despite L1 scores of 92.7–97.2.Source: §7.2 · p. 9, Table 3 · p. 10

7.3 How much of the ordering is real

  • 89 of 105 model pairs remain separated after Holm correction, while category reweighting and removing near/far queries preserve the broad ranking.The corrected leaderboard has a four-point resolution; the controls yield Kendall’s τ of 0.96 and 0.99, respectively.Source: §7.3 · p. 9, Table 5 · p. 11

7.4 What makes a query hard

  • Within L3, mean EQS falls from 49.0 to 34.1 as same-category candidates increase, while candidate-count penalties persist across answer-area terciles.Answer area has a smaller effect when ambiguity is held fixed.Source: §7.4 · p. 10, Table 4 · p. 11

7.5 How predictions change with depth

  • Predictions stay strong on category naming but split on superlatives and relations, with whole-category masks and empty predictions among the visible failures.EQS is reported by level and answerability alongside mIoU, since a single mean can hide which level failed.Source: §7.5 · pp. 10–11

7.6 Predictions across scenes and resolutions

  • The plates pair whole scenes with cropped windows and compare urban 12.5 cm and rural 25 cm scenes, where candidate counts and answer sizes differ.They mark masks outside the window as out and empty predictions as none.Source: §7.6 · pp. 11–12, Figure 2 · p. 12

7.7 Error taxonomy and unanswerable sentences

  • Boundary drift retains IoU of 0.35–0.64, but sibling, whole-category, and empty-mask reference errors score zero to 0.27 despite overlap from whole-category masks.For unanswerable queries, rejection accuracy ranges from 32.8 to 65.4, and the examples require rejecting a present category whose relation is unsatisfied.Source: §7.7 · p. 12, Figure 3 · p. 13

8 Discussion and conclusion

The evaluated systems handle category-level reference well but struggle with compositional queries, ambiguity, and rejecting plausible referents. The paper also identifies limits on what its single-split evaluation and dataset can establish, while releasing materials to reproduce its analyses.

8 Discussion and conclusion

  • At least 86.0 at L1 for every model, with the best reasoning-oriented remote-sensing systems reaching 84.3 at L2 and 77.5 at L3, indicates that category-level and one-relation reference are achievable.The authors characterize the benchmark as not being a segmentation-quality test in disguise.Source: §8 · p. 13
  • 36.9 mIOU against EQS ≤22.7 for relation-blind selection shows that overlap can reward predictions that fail to resolve the query’s identity.The error taxonomy distinguishes boundary quality, instance selection, answer cardinality, and rejection; Figure 4 separates pixel-level boundary drift from identity failures.Source: §8 · p. 14, Figure 4 · p. 14
  • The fifteen-model comparison covers one checkpoint per system on the primary scene-disjoint split, not the released geographic, cross-scale, depth, or unseen-pair protocols.The cross-scale protocol changes ground sampling and physical field of view together, so its results cannot isolate sampling effects.Source: §8 · p. 14
  • 60.5 EQS at two-hop reference is the strongest system’s score, while most of the fifteen systems fall below 5; the field’s failures arise in compositional reference, ambiguity, and rejection.The conclusion reports that the small separations within the comparison are not significant under its paired scene-level bootstrap.Source: §8 · pp. 14–15

Reproducibility statement

  • The release includes scenes, masks, graphs, queries, diagnostic records, five splits, and the tools needed to regenerate reported analyses; its validator reports zero issues across 700 scenes.Split guarantees are tested against native pixel windows of the source orthophotos.Source: Reproducibility statement · pp. 15–16

Ethics statement

  • The dataset uses rural and urban UAV orthophotography from Jiangxi, China, and excludes vehicles because their segmentation was not stable enough across scenes and resolutions for instance-level answers.The paper notes that sentences come from geometry rather than annotator-supplied judgments, and that it offers no formal anonymity guarantee.Source: Ethics statement · p. 16
Loading 2609.29541v1…