PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoReason: Bridging Logical Reasoning and Spatial Fidelity in Remote Sensing Segmentation

Cheng Fan, Han Wu, Tingzhang Luo, Minjing Dong, Yehui Tang, Jianyuan Guo

arXiv:2610.23416v1cs.CV

TL;DR

Remote-sensing reasoning segmentation must handle cluttered scenes and extreme scale variation, while delayed [SEG] generation can leave visual evidence diluted during long reasoning. GeoReason anchors [SEG] in the prompt and combines bounding-box reasoning with latent patch alignment. It reports state-of-the-art performance on EarthReason and RRSIS-D.

  • Problem

    Remote-sensing reasoning segmentation remains challenging in cluttered scenes and across extreme scale variations, with delayed [SEG] generation associated with diluted visual evidence and localization errors.

    Source: §1 · p. 2

  • Method

    GeoReason injects [SEG] into the prompt as an anchor and combines bounding-box-based Location-aware CoT with PTA alignment of token states to target patches.

    Source: §1 · p. 3

  • Results

    GeoReason achieves state-of-the-art performance on EarthReason and RRSIS-D for reasoning and referring segmentation.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The combined design preserves visual evidence, improves grounding, and strengthens the segmentation-token representation in remote-sensing reasoning segmentation.

    Source: §5 · p. 10

    Experiments mainly use Qwen-VL and SAM-family models and emphasize single-target segmentation; transfer to other backbones or decoders and more complex settings remain unevaluated.

    Source: F Limitations · p. 22

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have significantly advanced instruction-driven perception through the "Embedding-as-Mask" paradigm. However, extending this capability to Remote Sensing (RS) reasoning segmentation remains exceptionally challenging under cluttered geographical contexts and extreme scale variations, often leading to Spatial Fidelity Degradation and Target Misidentification. We identify a critical yet underexplored bottleneck: the sequential alienation of the segmentation token [SEG] from its initial visual grounding within the autoregressive generation paradigm. Produced only at the end of long reasoning chains, [SEG] relies on visual evidence that has been progressively diluted by linguistic abstraction and autoregressive noise. This information decay is particularly catastrophic for RS targets with minuscule pixel footprints, where even marginal fidelity loss leads to localization failure. To address these bottlenecks, we propose GeoReason, a novel framework designed to fortify logical deduction and preserve spatial integrity. First, we introduce an Anticipatory Prior mechanism that injects the [SEG] token directly into the initial user query, shifting it from a terminal output to a primary condition. This paradigm shift transforms the token into a persistent spatial anchor, preventing information decay during long-form linguistic inference. Second, we enhance visual reasoning via a Location-aware Chain-of-Thought (CoT) that enforces coarse-to-fine localization using bounding boxes, coupled with a Latent Patch-to-Token Alignment (PTA) Loss that explicitly aligns the latent segmentation token with physical RS textures. Extensive experiments demonstrate that GeoReason consistently outperforms state-of-the-art methods on the Earthreason and RRSIS-D benchmarks. Code is available at https://github.com/Fanc0126/GeoReason.

1 Introduction

Embedding-as-Mask lets MLLMs support instruction-driven segmentation, but remote-sensing scenes make precise reasoning difficult. GeoReason addresses degraded masks and target misidentification by anchoring [SEG] early and combining spatial reasoning with latent alignment.

1 Introduction

  • Remote-sensing imagery combines overhead views, extreme scale variation, and clutter, requiring logical inference and precise localization.The introduction names functional attributes and spatial relationships as reasoning demands.Source: §1 · p. 2
  • Current MLLM-based methods can identify a target yet produce fragmented masks with imprecise boundaries, or lose track of the target entirely.The latter failure is described as segmentation drifting to an incorrect object.Source: §1 · p. 2
  • The paper attributes these failures to [SEG] being generated only after a long reasoning chain, as linguistic abstraction and autoregressive noise dilute visual evidence.Shortening or removing reasoning can partially alleviate degradation, but difficult samples may still require multi-step inference.Source: §1 · p. 2
  • The Anticipatory Prior injects [SEG] into the user query as a persistent spatial anchor, preserving visual evidence through long-form reasoning.The prompt-side token is described as a deterministic input that avoids cumulative linguistic abstraction and autoregressive noise.Source: §1 · p. 3
  • Location-aware CoT uses intermediate bounding boxes for coarse-to-fine localization, while PTA aligns [SEG] hidden states with ground-truth target patches.The paper presents these as explicit spatial grounding and implicit latent supervision for cluttered remote-sensing scenes.Source: §1 · p. 3
  • GeoReason is reported to achieve state-of-the-art performance on remote-sensing reasoning-segmentation benchmarks while improving reasoning accuracy and segmentation quality.The introduction identifies EarthReason and RRSIS-D as the benchmarks.Source: §1 · p. 3

2 Related Work

Prior MLLM segmentation work uses latent [SEG] embeddings or text-generated spatial prompts, while remote-sensing methods have progressed from referring segmentation toward reasoning tasks. GeoReason combines explicit spatial grounding with latent token supervision.

2 Related Work

  • Embedding-as-Mask methods use a special [SEG] token whose hidden state prompts a mask decoder, whereas text-based approaches generate spatial prompts such as boxes or points.The passage situates these as the two main MLLM segmentation paradigms.Source: §2 · p. 3
  • GeoReason’s framework injects [SEG] into the query, predicts intermediate bounding boxes before the final [SEG], and aligns prompt- and answer-side token states with target patches.The overview presents these as the Anticipatory Prior, Location-aware CoT, and PTA loss.Source: Figure 3 · p. 4
  • Remote-sensing language-guided segmentation has moved from referring-expression tasks toward reasoning segmentation, where logical deduction remains difficult in cluttered scenes.SAM-based sparse-prompt methods are described as effective for referring-style grounding but limited on multi-step reasoning.Source: §2 · p. 4
  • Visual reasoning work includes image interaction, intermediate box prediction, and latent-token alignment, motivating a combination of explicit grounding and token-level supervision.The cited examples span iterative cropping or zooming, spatial anchoring, and alignment with image representations.Source: §2 · p. 4

3 Our Method

GeoReason combines an MLLM with SAM2 for remote-sensing reasoning segmentation, using a prompt-side [SEG] anchor to preserve spatial fidelity and additional reasoning and latent-alignment components to improve localization.

3.1 System Architecture

  • GeoReason combines an MLLM with SAM2, treating each [SEG] token’s last-layer hidden state as a mask query projected into SAM2’s space.During inference, candidate masks from prompt-side and answer-side tokens are united.Source: §3.1 · p. 4

3.2 Anticipatory Prior via Prompt-side [SEG] Token

  • A prompt-side [SEG] token establishes segmentation intent before generation and remains a stable spatial reference for answer-side tokens.The answer-side token can inherit this prior rather than reconstructing segmentation cues after a long reasoning chain.Source: §3.2 · p. 5

3.3 Enhanced Visual Reasoning for Remote Sensing

  • Location-aware CoT asks the model to predict a bounding box before the final [SEG], providing coarse localization before dense mask decoding.The box is an intermediate spatial hypothesis for cluttered scenes and scale variation.Source: §3.3 · p. 5
  • PTA aligns supervised [SEG] embeddings with target-image patches, encouraging foreground-specific visual evidence in the embedding.The loss complements mask supervision by shaping the latent token representation.Source: §3.3 · pp. 5–6

3.4 Optimization Objective

  • Training combines language, mask, and latent-alignment supervision, applying segmentation-related losses to prompt-side and answer-side [SEG] tokens.The objective balances text, mask, and PTA losses with λtext, λmask, and λpta.Source: §3.4 · p. 6

4 Experiments

GeoReason is evaluated on EarthReason reasoning segmentation and RRSIS-D referring segmentation, with its strongest reported gains on EarthReason. Mechanism analyses and ablations examine how prompt-side spatial anchoring and complementary components contribute to performance.

4.2 Main Results

  • GeoReason establishes a new EarthReason reasoning-segmentation state of the art, improving test cIoU to 80.60% versus 70.45% for the backbone-matched baseline.It also improves val/test gIoU by +2.57/+3.29 and val/test cIoU by +4.33/+5.99 over the strongest alternatives on those metrics.Source: §4.2 · p. 6, Table 1 · p. 7
  • On RRSIS-D, GeoReason achieves the best validation and test gIoU and remains competitive in cIoU; gains mainly reflect better mask quality.The paper relates this pattern to PTA improving [SEG] discriminability in referring scenarios with little explicit reasoning.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 Mechanistic Analysis of the Anticipatory Prior

  • Figure 5 analyses attention to image tokens and foreground/background alignment; with CoT, baseline attention falls from 0.246 to 0.212, while GeoReason prompt-side [SEG] reaches 0.270.GeoReason’s answer-side [SEG] retains 0.245 attention, and its foreground similarity and foreground-background margin rise to 0.096 and 0.056 from 0.044 and 0.035.Source: Figure 5 · p. 7, §4.3 · pp. 7–8
  • In Figure 6’s case, GeoReason produces a cleaner target mask, while the final answer-side [SEG] focuses strongly on the prompt-side [SEG].Across samples, the prompt-side token ranks 2.01 on average and appears among the top-three attended prompt tokens in 99.69% of samples.Source: Figure 6 · p. 8, §4.3 · p. 8

4.4 Ablation Studies

  • On EarthReason, adding AP alone raises test cIoU from 70.45% to 78.87%; PTA alone raises it to 73.01%, while BBox alone falls slightly below baseline.PTA is identified as the main source of improvement on RRSIS-D, where mask quality matters more than long-form reasoning.Source: §4.4 · p. 8
  • Moving the extra [SEG] from the prompt into the answer stream yields a 74.14 average score, while Zero-Reasoning scores 74.63% versus 72.79% for the full reasoning baseline and remains below GeoReason’s 78.61%.The diagnostic case contrasts GeoReason selecting a protected silo with Zero-Reasoning drifting to an exposed open area.Source: Table 4 · p. 9, §4.4 · p. 9, Figure 7 · p. 10
  • The baseline reaches 72.79% Avg versus 80.84% for Oracle Referring; GeoReason reaches 78.61%, leaving a 2.23-point residual to that ceiling.Adding PTA to Oracle Referring raises performance to 81.78% Avg and 84.14% test cIoU, indicating gains beyond the reasoning setting.Source: Table 5 · p. 9, §4.4 · p. 9
  • The full model performs best on both benchmarks; partial AP+PTA and AP+BBox variants remain competitive but trail it on EarthReason.The components address distinct error sources: AP preserves spatial evidence, BBox supports coarse localization, and PTA sharpens the latent segmentation representation.Source: §4.4 · p. 9

5 Conclusion

GeoReason combines a prompt-side [SEG] anchor, Location-aware CoT, and PTA loss, and reports state-of-the-art performance on EarthReason and RRSIS-D. The implementation supplements training with filtered remote-sensing VQA and captioning data.

5 Conclusion

  • GeoReason reports state-of-the-art performance on EarthReason and RRSIS-D for reasoning and referring segmentation.The framework combines a prompt-side [SEG] anchor, Location-aware CoT, and PTA loss.Source: §5 · p. 10

A.1 Data Composition and Preprocessing

  • Training uses GeoChatInstruct and VRSBench VQA and captioning subsets, with overlapping RRSIS-D validation and test images removed.After filtering, the auxiliary sets contain 21.2k GeoChatInstruct samples and 34.7k VRSBench samples; these tasks do not use [SEG] or generate masks.Source: §A.1 · p. 16

A.2 Training Configurations

This material reports training settings and prompt-side procedures, then analyzes how extended reasoning may dilute visual evidence and shift the segmentation query. It also describes the benchmarked runtime and token overhead.

A.2 Training Configurations

  • The vision encoder stays frozen, while the MLP adapter and SAM2 mask decoder are updated; the LLM uses LoRA with rank 128 and α = 256.The model uses AdamW, a 4 × 10−5 initial learning rate, and λtext, λmask, and λpta values of 1.0.Source: §A.2 · p. 16
  • 1223.6 ms vs. 774.1 ms latency and 73.3 vs. 45.1 tokens: GeoReason exceeds the Qwen3VL-4B + SAM2 baseline on EarthReason validation.The reported overhead stems primarily from autoregressive reasoning and bounding-box tokens; training takes approximately 9 hours on four NVIDIA RTX Pro 6000 GPUs.Source: §A.2 · p. 16

A.3 Prompt Templates and Formats

  • Training supervises both prompt-side and answer-side [SEG] tokens; inference decodes their candidate masks and unions them for the final segmentation.The prompt templates request bounding boxes in <think> tags; their coordinates are derived from ground-truth masks using the minimum axis-aligned box covering target pixels.Source: §A.3 · pp. 16–17

B.1 Analysis of Visual-Evidence Dilution

  • Earlier studies report declining visual attention or perception performance during extended reasoning, including a 6.8-point maximum drop on perception tasks when switching to CoT.The cited studies include Sun et al. [41], Tian et al. [42], MME-CoT [12], and OPERA [11].Source: §B.1 · p. 17
  • The analysis identifies visual-token competition and mask-query drift as two ways longer reasoning may dilute visual evidence in embedding-as-mask segmentation.Additional reasoning tokens enlarge the competing linguistic context, while the segmentation query and attention logits can also change.Source: §B.1 · pp. 17–18
  • With image features and decoder fixed within each model, changes to the answer-side [SEG] representation can alter the decoder’s latent condition without changing the segmentation architecture.The analysis measures image-attention mass and segmentation-query representation drift across reasoning-prefix lengths.Source: §B.1 · pp. 17–18

B.2 Controlled Prefix-Length Experiments

In a controlled EarthReason test-set experiment, the baseline loses more image attention and exhibits greater answer-side [SEG] drift as reasoning prefixes lengthen; GeoReason attenuates both changes.

B.2 Controlled Prefix-Length Experiments

  • The experiment retains 0%, 25%, 50%, 75%, or 100% of one saved GeoReason response trajectory’s tokens, then appends the same final answer for both models.It uses one canonical trajectory per EarthReason test instance and measures image attention and answer-side [SEG] hidden-state drift.Source: §B.2 · p. 18, Table 6 · p. 18
  • 61.7%: GeoReason reduces the baseline’s absolute endpoint image-attention attenuation; its full-prefix hidden-state drift is 0.0495 versus 0.2076 for the baseline, a 76.2% reduction.Image attention changes from 0.2593 to 0.2446 for GeoReason and from 0.2389 to 0.2003 for the baseline.Source: §B.2 · p. 18, Table 6 · p. 18

C Extended Ablation Studies on PTA Loss

The PTA ablations favor a GLU-based alignment head, positive-pixel mask supervision, average pooling, and cosine similarity. Removing the PTA head is associated with a 4.67% test cIoU decrease on EarthReason.

C Extended Ablation Studies on PTA Loss

Architecture of the PTA Head

  • −4.67% in EarthReason test cIoU: removing the PTA head lowers performance, while the GLU-based head outperforms a simple MLP.The authors argue that a projection head preserves the [SEG] token’s representational capacity for subsequent mask generation.Source: C Extended Ablation Studies on PTA Loss · p. 18, Table 7 · p. 19

Granularity of Spatial Supervision

Pooling Strategy

Loss Formulation

D Additional Benchmark Results on GeoSeg-Bench

GeoReason achieves the highest gIoU on both GeoSeg-Bench tasks without GeoSeg-1M fine-tuning, and the paper then introduces qualitative analyses of its remote-sensing reasoning segmentation.

D Additional Benchmark Results on GeoSeg-Bench

  • Evaluation uses the main-experiment checkpoint on GeoSeg-Bench’s Referring and Reasoning subsets, with no GeoSeg-1M fine-tuning, and compares against eleven methods plus fine-tuned Geopixel and Earthmind.Source: D Additional Benchmark Results on… · p. 19
  • 10.16 and 3.66 percentage points: GeoReason improves Referring and Reasoning gIoU, respectively, over Earthmind, the next-best method without GeoSeg-1M fine-tuning.Source: D Additional Benchmark Results on… · p. 19
  • Referring cIoU improves by 5.97 points, while Reasoning cIoU is nearly tied at 30.99 versus 31.01; GeoReason also surpasses fine-tuned baselines on multiple metrics.Source: D Additional Benchmark Results on… · p. 19

E Qualitative Visualizations and Analysis

E.1 Case Study

In a representative case, GeoReason localizes the intended target and produces a more accurate mask than the baseline; further examples illustrate how its components address common failure modes.

E.1 Case Study

  • GeoReason localizes the intended target and produces the correct mask, while the baseline attends to an incorrect structure; Fig. 8 shows its stronger target-focused [SEG] response.Source: §E.1 · pp. 19–20, Figure 8 · p. 20
  • The intermediate bbox supplies a coarse spatial anchor before mask decoding, while the Anticipatory Prior and PTA loss make [SEG] more target-sensitive.Source: §E.1 · p. 20

E.2 Analysis of Typical Failure Modes

  • For spatial fidelity degradation, target misidentification, and boundary redundancy, the analysis associates prompt-side anchoring, bbox localization, and PTA alignment with the respective failure modes.Source: Figure 9 · p. 21, §E.2 · p. 22

F Limitations

The study’s evaluation scope is limited to particular model families and mainly single-target benchmarks.

F Limitations

  • Main experiments use Qwen-VL and SAM-family models, and benchmarks mainly emphasize single-target segmentation; transfer to other backbones or decoders and more complex settings remain unevaluated.Source: F Limitations · p. 22

Related papers

Loading 2610.23416v1…