PaperlayerView exact revision on arXiv

Source-linked AI summary

TerraScope: Pixel-Grounded Visual Reasoning for Earth Observation

Yan Shu, Bin Ren, Zhitong Xiong, Xiao Xiang Zhu, Begüm Demir, Nicu Sebe, Paolo Rota

arXiv:2603.19039v1cs.CV

TL;DR

VLMs struggle to ground complex EO spatial reasoning in precise pixels, especially for fine-grained geospatial analysis. TerraScope interleaves reasoning tokens with segmentation masks, supports optical, SAR, multimodal, and temporal inputs, and is trained and evaluated with Terra-CoT and TerraScope-Bench. The framework achieves precise, interpretable spatial analysis across diverse geospatial reasoning tasks, while retaining limitations involving hallucinations and increased context length.

  • Problem

    Existing VLMs struggle with fine-grained EO reasoning requiring pixel-accurate spatial analysis, while current benchmarks inadequately assess such capabilities.

  • Method

    TerraScope generates segmentation masks alongside reasoning traces, injects masked visual features during reasoning, and adaptively handles optical, SAR, multimodal, and temporal EO data.

  • Results

    TerraScope achieves precise and interpretable spatial analysis across diverse geospatial reasoning tasks, with experiments validating its effectiveness.

  • Takeaways & Limitations

    Terra-CoT and TerraScope-Bench provide scalable pixel-grounded training and evaluation using pixel-accurate masks and dual answer-and-mask metrics.

  • Takeaways & Limitations

    TerraScope may hallucinate reasoning traces or inaccurate masks, and interleaved mask-reasoning generation increases context length during training and inference.

Abstract

from arXiv · show

Vision-language models (VLMs) have shown promise in earth observation (EO), yet they struggle with tasks that require grounding complex spatial reasoning in precise pixel-level visual representations. To address this problem, we introduce TerraScope, a unified VLM that delivers pixel-grounded geospatial reasoning with two key capabilities: (1) modality-flexible reasoning: it handles single-modality inputs (optical or SAR) and adaptively fuses different modalities into the reasoning process when both are available; (2) multi-temporal reasoning: it integrates temporal sequences for change analysis across multiple time points. In addition, we curate Terra-CoT, a large-scale dataset containing 1 million samples with pixel-level masks embedded in reasoning chains across multiple sources. We also propose TerraScope-Bench, the first benchmark for pixel-grounded geospatial reasoning with six sub-tasks that evaluates both answer accuracy and mask quality to ensure authentic pixel-grounded reasoning. Experiments show that TerraScope significantly outperforms existing VLMs on pixel-grounded geospatial reasoning while providing interpretable visual evidence.

1 University of Trento 2 BIFOLD and TU Berlin 3 Technical University of Munich 4 MBZUAI

Figure 1 contrasts direct VLM answers, textual Chain-of-Thought, and TerraScope’s interleaved pixel-grounded reasoning, alongside its dataset and benchmark.

  • TerraScope combines pixel-level grounding masks with textual input to form an interleaved Chain-of-Thought.
  • The figure also presents the Terra-CoT 1M dataset and TerraScope benchmark.

1. Introduction

The introduction identifies a gap in fine-grained, pixel-accurate EO reasoning and presents TerraScope, Terra-CoT, and TerraScope-Bench as a unified response.

  • Existing VLMs struggle with fine-grained geospatial reasoning requiring pixel-accurate spatial analysis.
  • Natural-image grounding methods do not directly transfer to EO because land-cover distributions are continuous and EO reasoning often spans multiple sensors and times.
  • TerraScope grounds each reasoning step in explicitly localized pixel-level visual evidence rather than relying solely on language-domain reasoning.
  • TerraScope-Bench contains 3,837 expert-verified questions covering optical, SAR, joint optical-SAR, single-date, and multi-temporal scenarios.
  • Terra-CoT provides 1M instruction-tuning samples with pixel-accurate masks embedded in reasoning traces, while TerraScope-Bench evaluates answer accuracy and mask quality.

2. Related Works

Related work spans EO-specific VLMs, EO benchmarks, and visual Chain-of-Thought methods, but existing evaluations still lack rigorous pixel-accurate geospatial assessment.

  • EO-specific VLMs adapt general-purpose models through domain-specific data curation and model adaptation.
  • TerraScope’s overview combines interleaved textual reasoning and segmentation masks with multimodal and multi-temporal EO reasoning.
  • Existing EO benchmarks cover conversational, region-level, high-resolution, multi-task, multi-sensor, and bi-temporal capabilities.
  • Visual Chain-of-Thought methods interleave visual evidence with textual reasoning chains or iteratively focus on selected image regions.
  • Prior benchmarks do not rigorously assess pixel-accurate geospatial inference needed for detailed spatial analysis.

3. Method

TerraScope integrates pixel-level masks with textual reasoning, enabling visual evidence to be interleaved throughout its reasoning process. Its framework supports optical-SAR modality selection, temporal disambiguation, and hierarchical synthesis of pixel-grounded chain-of-thought data.

  • Pixel-Grounded Visual Reasoning: TerraScope interleaves generated segmentation masks and masked visual features with textual reasoning steps before producing the final answer.The framework explicitly selects visual features from regions identified at each reasoning step.
  • Pixel-Grounded Visual Reasoning: The model aligns pixel masks to the visual token grid and selects tokens whose corresponding spatial regions are covered by more than 50% of the mask.Selected features are projected, flattened, and fed into the language model alongside text embeddings.
  • Multi-Modal and Temporal Reasoning: TerraScope uses text-guided, token-level modality selection to choose between optical and SAR features for each visual token position.Optical and SAR images are encoded independently, and cross-attention with the question produces modality relevance scores.
  • Multi-Modal and Temporal Reasoning: Temporal indicators in the form “Image: ti” specify which observation supplies each mask and its masked visual tokens.This mechanism addresses temporal disambiguation when reasoning over multiple observations.
  • Terra-CoT Dataset: Terra-CoT is created through grounded captioning and hierarchical synthesis, producing L1 spatial-grounding and L2 multi-step spatial or semantic reasoning traces.The pipeline first uses annotated data to train an annotation model, then synthesizes diverse questions from generated masks and captions.

4. TerraScope-Bench

TerraScope-Bench targets fine-grained spatial reasoning that existing coarse-grained EO benchmarks inadequately assess. It comprises curated samples and evaluates model performance through benchmark tasks and grounding IoU comparisons.

  • Motivation: Existing EO benchmarks emphasize scene classification and captioning, leaving fine-grained pixel-level spatial reasoning inadequately assessed.Above 10m resolution, objects span few pixels and land-use boundaries become ambiguous.
  • Benchmark Design: TerraScope-Bench comprises 3,837 carefully curated samples for evaluating pixel-grounded geospatial reasoning.The supplied passages identify the benchmark scale but do not provide the quantitative results table values.
  • Evaluation: Grounding IoU compares models’ segmentation-mask quality, complementing answer-accuracy evaluation on TerraScope-Bench and related datasets.Table 1 reports quantitative performance on TerraScope-Bench, Landsat30AU, and DisasterM3, while Figure 5 presents grounding IoU.
  • Benchmark Design: The benchmark includes six task categories: coverage percentage, absolute area, distance, comparative area, boundary relationship, and building change estimation.The task counts are 855, 855, 129, 855, 855, and 288 respectively.
  • Benchmark Design: Questions and answers are generated from pixel-level segmentation annotations using spatial properties such as coverage ratios, areas, distances, and boundary relationships.Templates, LLM rephrasing, distractors, and human review support varied multiple-choice questions.

5. Experiments

TerraScope is evaluated on multiple EO benchmarks, including TerraScope-Bench and zero-shot external benchmarks, with ablations of its visual reasoning strategies. Existing VLMs struggle with precise spatial reasoning, while Terra-CoT fine-tuning substantially improves performance across tasks.

  • Evaluation setup: TerraScope is evaluated on TerraScope-Bench and two representative EO benchmarks in zero-shot settings.The external benchmarks include LandSat30-AU and DisasterM3.
  • Main findings: Existing VLMs struggle with fine-grained geospatial reasoning, especially area percentage estimation requiring precise spatial analysis.Both proprietary and open-source models achieve near-random performance on such tasks.
  • Main findings: EO-specific VLMs do not significantly outperform general VLMs on TerraScope-Bench despite training on large-scale EO data.The paper hypothesizes that existing EO datasets emphasize high-resolution imagery, limiting handling of lower-resolution data.
  • Main findings: Reasoning models perform better on tasks requiring external knowledge but remain insufficiently grounded in pixel-level visual evidence.The paper associates their textual reasoning with hallucinations and weak fine-grained spatial perception.
  • Main findings: Fine-tuning InternVL3 and GLM-4.1V-Think on Terra-CoT leads to substantial performance gains across all tasks.This result is presented as evidence of Terra-CoT’s effectiveness for pixel-grounded reasoning.

Model CA AQ CR BRD DM

The ablations show that precise segmentation-based grounding and trained multimodal selection are central to TerraScope’s reasoning. Pixel-grounded masks improve answer correctness, while optical-SAR fusion improves segmentation and handles modality-specific data quality.

  • Pixel-grounded reasoning: Precise pixel-level grounding through segmentation masks is essential for effective visual reasoning in EO.Bounding-box grounding underperforms especially when land-cover regions have irregular boundaries and shapes.
  • Pixel-grounded reasoning: Correct predictions achieve mean IoU 0.628 versus 0.443 for incorrect predictions, with Pearson correlation r = 0.607, p < 0.001.The correlation remains r = 0.70-0.80 across task types.
  • Multi-modal reasoning: Multi-modal fusion substantially improves performance over the optical-only baseline.The comparison includes optical-only, concatenated, test-only text-guided, and train-plus-test text-guided settings.
  • Multi-modal reasoning: Text-guided modality selection reduces context length by selecting the relevant modality instead of processing both.It maintains competitive performance while improving efficiency relative to processing concatenated optical and SAR features.
  • Multi-modal reasoning: Training modality selection is essential because enabling selection only at test time yields no improvement.The model must learn during training when and how to leverage different modalities.
  • Multi-modal reasoning: In cloud-contaminated regions, SAR fusion improves segmentation where optical imagery is corrupted.The selection mechanism prioritizes optical tokens in cloud-free regions and SAR tokens in cloud-covered areas.

6. Qualitative Results

Qualitative examples show TerraScope combining structured textual reasoning with precise masks across area estimation, distance measurement, and temporal counting tasks.

  • Qualitative examples: TerraScope decomposes challenging spatial questions into interpretable textual reasoning sub-steps while generating precise masks for relevant regions.The examples cover area percentage estimation, distance measurement, and temporal counting VQA.

7. Conclusion

The paper presents TerraScope as a unified framework for interpretable, pixel-grounded EO reasoning across modalities and time. It also contributes Terra-CoT and TerraScope-Bench to support scalable training and dual evaluation of answers and masks.

  • Conclusion: TerraScope grounds reasoning traces in segmentation masks for precise spatial analysis, multi-temporal change reasoning, and adaptive optical-SAR use.The framework is presented as a unified vision-language approach for earth observation.
  • Conclusion: Terra-CoT contains 1M instruction-tuning samples with pixel-accurate masks embedded in reasoning chains.The dataset is intended to support scalable pixel-grounded training.
  • Conclusion: TerraScope-Bench is introduced as a benchmark for pixel-grounded geospatial reasoning with answer and segmentation-mask evaluation.The benchmark is described as the first of its kind in the paper.
  • Conclusion: Extensive experiments validate TerraScope across diverse geospatial reasoning tasks.The conclusion summarizes the experimental evidence without specifying a single metric.

Supplementary Material

The supplementary material details TerraScope’s limitations, inference pipeline, and benchmark coverage. It also describes pixel-grounded generation through interleaved masks, visual features, and reasoning tokens.

  • Limitations and Future Work: TerraScope may hallucinate reasoning traces or masks, has increased context length from interleaved generation, and currently supports only optical and SAR imagery.The authors identify multispectral, hyperspectral, and longer temporal reasoning as future extensions.
  • Comparison to Concurrent Works: TerraScope differs from general interleaved reasoning models through EO-specific multi-modal and multi-temporal reasoning capabilities.Concurrent models are described as having limited transferability to earth observation.
  • Comparison to Concurrent Works: Agent-based methods underperform TerraScope because orchestration can introduce hallucinations and their grounding is weaker than pixel-level segmentation.The cited comparison contrasts tool-mediated reasoning with box-level or SAM-based grounding.
  • Details of TerraScope: During inference, TerraScope generates text autoregressively and invokes segmentation when the language model emits a [SEG] token.The resulting mask-conditioned visual features are injected back into the generation sequence through KV-cache updates.
  • Details of TerraScope: Masked visual features are selected from regions covering more than 50% of a token and capped at λ = 128 using spatially uniform sampling.For multi-modal inputs, optical or SAR features are selected adaptively using text-relevance scores.
  • Details of TerraScope-Bench: TerraScope-Bench covers diverse task categories, sensor combinations, temporal settings, and question semantics for pixel-grounded reasoning evaluation.The supplementary distribution analysis presents task, image-source, and question-word distributions.

D.2. Data Annotations for TerraScope-Bench

TerraScope-Bench is built from pixel-level annotations through automated question generation, language refinement, and expert validation. The accompanying data curation process produces reasoning-oriented samples with masks and spatially derived answers.

  • Data Annotation Pipeline: TerraScope-Bench derives answers from segmentation masks for area, coverage, ranking, distance, boundary, and building-change questions.The construction pipeline begins with heuristic answer generation from pixel-level annotations.
  • Coverage Percentage: Coverage questions use Pc = Nc/Ntotal × 100% and require Pc ≥ 5% to exclude negligible regions.Ranking questions similarly restrict included classes to those meeting the 5% threshold.
  • Boundary Relationship Detection: Boundary relationships are determined by dilating one class mask with a 3×3 structuring element and testing intersection with the other mask.Small or highly fragmented regions are filtered to preserve clear adjacency judgments.
  • Question Refinement: GPT-4o rephrases template questions into natural language and generates plausible distractors, using two choices for binary tasks and four for other tasks.The prompt preserves the original meaning while making distractors numerically or semantically close to the ground truth.
  • Expert Validation: Four geoscience and disaster-assessment experts validate mask accuracy, answer correctness, distractor quality, and question clarity.Only samples receiving unanimous high-quality consensus are retained, yielding 3,837 expert-verified samples.
  • Cap-CoT Curation: Cap-CoT uses mask-overlaid images, labels, and metadata to generate captions with explicit spatial reasoning chains.The reasoning is instructed to mention segmented regions, spatial relationships, and approximate coverage or area.

G.2. Inference Time Analysis

The supplementary experiments examine inference speed, memory, and transfer across earth-observation benchmarks. TerraScope preserves competitive or specialized performance while adding measurable resource costs for pixel-level grounding.

  • Inference Time: 2.4s vs 2.6s: TerraScope infers faster than GLM-4.1V-9B despite generating additional segmentation masks.The authors attribute this to structured reasoning outputs and efficient mask-feature insertion into the KV cache.
  • Inference Time: 0.85s: InternVL-3 remains fastest, but it lacks TerraScope’s reasoning transparency and pixel-level grounding.The comparison distinguishes direct answer generation from mask-producing reasoning.
  • Memory Usage: 22.4GB vs 18.2GB: TerraScope uses approximately 22% more memory than InternVL-3 for single-mask inference.The SAM-2 decoder accounts for 3.9GB, and each additional mask adds approximately 0.7GB.
  • Landsat30-AU: Across Landsat30-AU’s eight task types, TerraScope is competitive and particularly strong on cloud-occlusion assessment and fine-object detectability.The reported pattern supports transfer of pixel-grounded reasoning to broader EO understanding tasks.
  • Additional Benchmarks: On RSVQA-LR, TerraScope performs slightly below EarthDial, while on BigEarthNet scene classification it achieves competitive accuracy comparable to EarthDial.The authors relate the RSVQA-LR difference to the larger VQA training datasets used by competing models.
  • DisasterM3: TerraScope is the only evaluated model supporting optical-SAR inputs through adaptive modality selection on DisasterM3.On optical-SAR pairs, it leverages complementary information from heterogeneous modalities for damage assessment.

I. More Ablation Studies

The ablations examine grounded pretraining, multimodal reasoning, and qualitative behavior, while failure analysis identifies limits from RGB-only inputs and segmentation errors. Adaptive modality selection offers a practical efficiency trade-off without materially sacrificing reasoning quality.

  • Effectiveness of Two-Stage Training: Grounded pretraining establishes foundational pixel-level visual understanding and improves performance on tasks requiring accurate segmentation.The perception-focused training still lacks complex reasoning capabilities on challenging multi-step spatial tasks.
  • Ablations of Multimodal Reasoning: Multimodal encoding substantially improves segmentation and reasoning compared with optical-only encoding by providing complementary optical-SAR representations from the beginning.Masked feature injection during reasoning also outperforms optical-only injection.
  • Ablations of Multimodal Reasoning: Adaptive selection maintains comparable segmentation quality and nearly equivalent reasoning performance while reducing context length, memory consumption, and inference time.Concatenation achieves slightly higher answer accuracy, whereas adaptive selection dynamically retains only the most informative modality at each spatial location.
  • Additional Visualizations: Qualitative results demonstrate pixel-grounded reasoning for optical imagery, optical-SAR fusion, and temporal change detection with interpretable reasoning traces.The examples span segmentation masks and spatial analysis across diverse scenarios.
  • Failure Cases and Analysis: TerraScope is limited by RGB-only processing and segmentation errors on small or low-contrast objects, which can propagate into unreliable spatial answers.The missing multispectral information makes spectrally similar land-cover types difficult to distinguish, while inaccurate masks compromise subsequent reasoning.
Loading 2603.19039v1…