Source-linked AI summary
SegEarth-R1: Geospatial Pixel Reasoning via Large Language Model
Kaiyu Li, Zepeng Xin, Li Pang, Chao Pang, Yupeng Deng, Jing Yao, Guisong Xia, Deyu Meng, Zhi Wang, Xiangyong Cao
TL;DR
Remote-sensing segmentation often depends on explicit instructions, while complex implicit queries require reasoning over spatial context, domain knowledge, and intent. This paper introduces geospatial pixel reasoning, EarthReason, and SegEarth-R1, and reports state-of-the-art performance on reasoning and referring segmentation.
Problem
Conventional remote-sensing workflows rely on explicit tasks and precise instructions, limiting their fit for complex implicit queries requiring spatial context, domain knowledge, and user-intent reasoning.
Source: §1 · p. 1Method
The paper introduces geospatial pixel reasoning, builds EarthReason with manually annotated image masks and implicit question-answer pairs, and proposes SegEarth-R1 with a hierarchical encoder, LLM, and spatial-correlation mask generator.
Source: §1 · pp. 1–2Results
SegEarth-R1 achieves state-of-the-art performance on reasoning and referring segmentation, outperforming traditional and LLM-based methods.
Source: §1 · p. 3Takeaways & Limitations
Geospatial pixel reasoning frames remote-sensing mask generation as inference from implicit queries, supported by the EarthReason benchmark.
Source: §1 · pp. 1–2
Abstract
from arXiv · showhide
Remote sensing has become critical for understanding environmental dynamics, urban planning, and disaster management. However, traditional remote sensing workflows often rely on explicit segmentation or detection methods, which struggle to handle complex, implicit queries that require reasoning over spatial context, domain knowledge, and implicit user intent. Motivated by this, we introduce a new task, \ie, geospatial pixel reasoning, which allows implicit querying and reasoning and generates the mask of the target region. To advance this task, we construct and release the first large-scale benchmark dataset called EarthReason, which comprises 5,434 manually annotated image masks with over 30,000 implicit question-answer pairs. Moreover, we propose SegEarth-R1, a simple yet effective language-guided segmentation baseline that integrates a hierarchical visual encoder, a large language model (LLM) for instruction parsing, and a tailored mask generator for spatial correlation. The design of SegEarth-R1 incorporates domain-specific adaptations, including aggressive visual token compression to handle ultra-high-resolution remote sensing images, a description projection module to fuse language and multi-scale features, and a streamlined mask prediction pipeline that directly queries description embeddings. Extensive experiments demonstrate that SegEarth-R1 achieves state-of-the-art performance on both reasoning and referring segmentation tasks, significantly outperforming traditional and LLM-based segmentation methods. Our data and code will be released at https://github.com/earth-insights/SegEarth-R1.
1 Introduction
Remote-sensing segmentation methods built for fixed taxonomies and explicit instructions struggle with implicit queries that require spatial context, domain knowledge, and user-intent reasoning. The paper introduces geospatial pixel reasoning, the EarthReason benchmark, and SegEarth-R1, reporting state-of-the-art results on reasoning and referring segmentation.
1 Introduction
- Geospatial pixel reasoning infers target masks from implicit queries that require reasoning over spatial context and domain knowledge.This differs from conventional workflows that rely on fixed taxonomies and precise user instructions.Source: §1 · pp. 1–2
- EarthReason contains 5,434 manually annotated image-mask pairs across 28 categories, with spatial resolutions from 0.5m to 153m and over 30,000 implicit question-answer pairs.It includes empty-target cases and varying spatial scales.Source: §1 · p. 2
- Remote-sensing images pose challenges for transferred natural-image methods through extreme scale variation, dense small objects, ultra-high resolution, and spatial-correlation needs.For example, identifying informal settlements can involve roof irregularities, road fragmentation, and adjacency to legal land-use zones.Source: §1 · p. 2
- SegEarth-R1 uses aggressive visual-token compression, description projection for language and multi-scale feature fusion, and a mask pipeline that queries description embeddings directly.Its mask generator is tailored to spatial correlation.Source: §1 · p. 2
- SegEarth-R1 achieves state-of-the-art performance on reasoning and referring segmentation, outperforming traditional and LLM-based methods.The model combines a hierarchical visual encoder, an LLM for instruction parsing, and a mask generator designed for spatial correlation.Source: §1 · pp. 2–3
2 Related Work
Prior referring-segmentation methods developed increasingly rich language-visual interactions, including for remote sensing, but still rely on explicit target instructions. LLM-based segmentation expanded pixel-level reasoning, while remote-sensing systems remained focused on explicit prompts; SegEarth-R1 addresses implicit-query reasoning.
2.1 Referring Segmentation
- Referring segmentation progressed from CNN-RNN/LSTM approaches to attention-based cross-modal alignment, including VLT and LAVT.Remote-sensing methods such as RMSIN, FIANet, CroBIM, and RSSep address scale, orientation, feature alignment, or boundary challenges, but use explicit instructions.Source: §2.1 · p. 3
2.2 LLM-based Segmentation
- LLM segmentation methods connect language to pixel masks, but prior remote-sensing models focus on explicit text guidance or grounded conversation rather than implicit-query reasoning.SegEarth-R1 is presented as the first work to reason about target regions from implicit queries, with components designed for remote-sensing challenges.Source: §2.2 · p. 3
3 Benchmark Geospatial Pixel Reasoning Dataset—EarthReason
EarthReason differs from related datasets by requiring reasoning to identify targets and by using manually annotated classification-source imagery across varied scales. Its generation pipeline collects images, creates question-answer pairs, labels masks, and reserves categories for evaluation.
3.1 Comparison with Related Dataset
- EarthReason queries do not explicitly name the target mask, and its classification-source imagery is manually annotated across diverse spatial resolutions and image sizes.The dataset contains 4.46× more data than the natural-image reasoning dataset ReasonSeg.Source: §3.1 · p. 4
3.2 Dataset Generation Pipeline
- Image collection selects 28 Million-AID categories, adds 800 fMoW images and 200 empty-target images, then yields 5,434 images after low-quality examples are removed.The source selection was intended to avoid potential data leakage in future unified remote-sensing segmentation models.Source: §3.2 · p. 4
- GPT-4o1 generates question-answer pairs using each image and its scene category, and GPT-3.5 rephrases them to increase question and answer diversity.The prompts combine remote-sensing images with categories from Million-AID and fMoW.Source: §3.2 · p. 4
- Remote-sensing and vision experts annotate masks from scratch and cross-validate them, using SAM-H for simple targets and finely marked polygons for complex ones.The authors contrast this with prior referring and reasoning datasets that use off-the-shelf masks or bounding boxes.Source: §3.2 · p. 5
- 2,371, 1,135, and 1,928 images form the training, validation, and test sets, respectively, with some categories held out of training for evaluation.Training images average six questions and three answers; average question and answer lengths are 20.86 and 26.76 words.Source: §3.2 · p. 5
4 Baseline Geospatial Pixel Reasoning Method—SegEarth-R1
SegEarth-R1 combines a hierarchical visual encoder, an LLM for instruction interpretation, and a mask generator for spatial correlation. Its design addresses remote-sensing image scale and spatial reasoning through multi-scale features, visual-token compression, and description-driven mask prediction.
4 Baseline Geospatial Pixel Reasoning Method—SegEarth-R1
- SegEarth-R1 uses a visual encoder, an LLM, and a mask generator for image features, instruction interpretation, and spatially correlated mask prediction.Each component incorporates design considerations for remote-sensing images.Source: §4 · p. 5
4.1 Hierarchical Visual Encoder
- A Swin Transformer builds features at 1/4, 1/8, 1/16, and 1/32 resolution to retain small-object detail while capturing deeper contextual semantics.The progressive hierarchy addresses extreme target-scale variation and densely distributed small objects.Source: §4.1 · p. 5
4.2 Large Language Model and Input Schema
- Because ultra-high-resolution remote-sensing images challenge billion-level LLM processing, SegEarth-R1 compresses visual tokens so the LLM handles simple semantic correlations.The model jointly embeds visual tokens and textual instructions in a unified LLM input space.Source: §4.2 · p. 6
4.2.1 Visual Token
- Redundancy analysis examines pixel statistics and spatial structure; the visual connector uses convolutional downsampling and Layer Normalization.The analysis uses intensity entropy and inter-patch SSIM; convolutional blocks reduce feature-map size, while Layer Normalization stabilizes cross-modal training.Source: §4.2.1 · pp. 6–7
- Remote-sensing datasets exhibit higher redundancy than natural images in the reported analysis.The figure compares redundancy across remote-sensing datasets and natural images.Source: Figure 3 · p. 6
4.2.2 Text Instruction
- Geospatial reasoning instructions and referring-segmentation instructions use the same question-answer format, with the task name changed in the prompt template.The geospatial template includes an image, an instruction description, and an answer.Source: §4.2.2 · p. 7
4.3 Mask Generation with Spatial Correlation
- The mask generator projects the full description into a query that interacts with multi-scale visual features, then predicts a mask without mask-token matching or score prediction.The description is averaged into a global vector, cross-attended with flattened visual features, and mapped to a query; the output is supervised with focal and dice losses.Source: §4.3 · p. 7
5 Experiments
SegEarth-R1 is evaluated on geospatial pixel reasoning and referring segmentation, with experiments reporting benchmark comparisons and ablations of its components and design choices.
5.1 Settings
- Evaluation covers EarthReason geospatial pixel reasoning and the RefSegRS and RRSIS-D referring datasets; gIoU and cIoU are reported, with cIoU preferred for stability.Models are trained on each dataset’s training set and evaluated on its validation and test sets.Source: §5.1 · p. 7
5.2 Geospatial Pixel Reasoning Results
- 3.64% and 2.45%: SegEarth-R1 surpasses PSALM on both test-set metrics and achieves the best results among evaluated methods on EarthReason.It also uses fewer visual tokens and mask-generator queries, resulting in lower inference cost.Source: Table 2 · p. 8, §5.2 · p. 8
5.3 Referring Segmentation Results
- The referring-segmentation results are reported in tables for RRSIS-D and RefSegRS, and qualitative EarthReason results are shown in Figure 5.The supplied passages identify the tables and figure but do not provide referring-segmentation outcome values.Source: Table 3 · p. 8, Table 4 · p. 9, Figure 5 · p. 9
5.4 Ablation Study
- Each proposed component improves EarthReason performance by 0.85% to 0.9%; together, they yield substantial gains over the PSALM baseline.The T.C. Connector and Query D.E. also reduce computational overhead.Source: §5.4 · p. 9, Table 5 · p. 9
- SegEarth-R1 maintains consistently high performance across tested LLMs, with Qwen2.5 (0.5B) achieving competitive results on RRSIS-D.The authors note its potential for edge deployment.Source: §5.4 · p. 9, Table 6 · p. 9
- Increasing the T.C. Connector layer number, which controls visual-token quantity, does not improve performance on EarthReason.The authors assign spatial image-instruction correlations to the mask generator and semantic correlations to the LLM.Source: §5.4 · p. 9, Table 7 · p. 9
6 Conclusion
The conclusion presents geospatial pixel reasoning and EarthReason as contributions, alongside SegEarth-R1 and its reported performance across reasoning and referring segmentation.
6 Conclusion
- Geospatial pixel reasoning asks models to infer masks from implicit natural-language queries using spatial context and domain knowledge.EarthReason is presented as the first large-scale benchmark emphasizing complex reasoning scenarios.Source: §6 · p. 10
- SegEarth-R1 combines a hierarchical visual encoder, an LLM for instruction parsing and semantic correlation, and a mask generator for spatial correlation.Source: §6 · p. 10
- The paper reports state-of-the-art performance for SegEarth-R1 on geospatial pixel reasoning and referring segmentation.The conclusion identifies environmental monitoring and disaster response as potential applications.Source: §6 · p. 10
A.1 Annotation of EarthReason
EarthReason’s annotation process combines generated reasoning queries with expert-produced masks and quality checks, and its statistics describe category balance and held-out categories.
A.1 Annotation of EarthReason
- Each EarthReason sample pairs an image and mask with six reasoning queries and answers; GPT-4o and GPT-3.5 generate textual annotations, while remote-sensing and vision experts annotate masks.Experts cross-validate annotations and re-annotate samples that fail quality standards; SAM-H assists with some simple targets.Source: §A.1 · p. 17
- GPT-4o generates a reasoning question–answer pair from an image and category label, and GPT-3.5 expands questions into six variations and answers into three alternatives.The prompt-construction processes are illustrated in Figures 6 and 7.Source: §A.1 · p. 17, Figure 6 · p. 17, Figure 7 · p. 18
- Figure 8 contrasts RRSIS-D semi-automatic masks, including visible errors and query mismatch, with EarthReason’s expert manual annotations.Source: §A.1 · p. 17, Figure 8 · p. 18
A.2 EarthReason Statistics
- EarthReason covers 28 categories with a relatively balanced distribution; four categories are excluded from training, and 119 empty-target samples are included to mitigate hallucinations.The held-out categories are basketball court, island, lake, and stadium.Source: §A.2 · p. 17
B.1 Details of Training Hyper-parameters
Training uses different supervision losses for referring segmentation and geospatial pixel reasoning, while the paper lists model-training hyperparameters in Table 8.
B.1 Details of Training Hyper-parameters
- Referring-segmentation training uses focal and dice losses, while geospatial pixel reasoning additionally uses LLM cross-entropy loss for text-answer generation.Source: §B.1 · p. 18
C.1 More Qualitative Results on EarthReason
On EarthReason, SegEarth-R1 better understands long reasoning instructions and generates more accurate masks than other models.
C.1 More Qualitative Results on EarthReason
- SegEarth-R1 demonstrates better understanding of long reasoning instructions on EarthReason than other models.Figure 10 presents the comparison.Source: §C.1 · p. 19
- SegEarth-R1 produces more accurate mask generation on EarthReason than other models.Figure 10 presents the comparison.Source: §C.1 · p. 19
- Figure 10 compares SegEarth-R1 with other models on EarthReason.Source: §C.1 · p. 19
C.2 More Qualitative Results on RRSIS-D
On RRSIS-D, SegEarth-R1 better understands direct geographical attributes than PSALM; the paper attributes this improvement to direct interaction between description embeddings and image features.
C.2 More Qualitative Results on RRSIS-D
- SegEarth-R1's improved understanding of location, color, and size is attributed to removing indirect mask-token prediction.Description embeddings directly interact with image features to generate masks.Source: §C.2 · pp. 19–20
- Figure 11 compares SegEarth-R1 with PSALM on RRSIS-D, where SegEarth-R1 better understands direct geographical attributes.The attributes include location, color, and size.Source: §C.2 · p. 19, Figure 11 · p. 21