PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing

Akashah Shabbir, Mohammed Zumri, Mohammed Bennamoun, Fahad S. Khan, Salman Khan

arXiv:2501.13925v1cs.CV

TL;DR

Remote-sensing LMMs are limited by coarse grounding, low-resolution inputs, and insufficient domain-specific grounded data for fine-grained spatial understanding. GeoPixel addresses these gaps with an end-to-end high-resolution pixel-grounding model and the GeoPixelD dataset, achieving superior performance across reported RS grounded-conversation evaluations while still exhibiting mask ambiguities in crowded or visually similar scenes.

  • Problem

    Remote-sensing LMMs face coarse spatial grounding, low-resolution constraints, and a lack of granular domain-specific grounded data for complex high-resolution imagery.

  • Method

    GeoPixel combines adaptive high-resolution image processing with a SAM-2-based grounding encoder, pixel decoder, and <SEG> token, supported by the GeoPixelD grounded-conversation dataset.

  • Results

    GeoPixel demonstrates superior performance to the baselines across RS grounded-conversation metrics and maintains strong performance in multi-target scenarios.

  • Takeaways & Limitations

    GeoPixel provides fine-grained pixel-grounded interpretation and generation for high-resolution remote-sensing imagery, including multi-target descriptions with corresponding masks.

  • Takeaways & Limitations

    GeoPixel can produce erroneous, fragmented, overlapping, or confused instance masks when object presence, quantity, or annotation type is ambiguous, especially in crowded scenes.

Abstract

from arXiv · show

Recent advances in large multimodal models (LMMs) have recognized fine-grained grounding as an imperative factor of visual understanding and dialogue. However, the benefits of such representation in LMMs are limited to the natural image domain, and these models perform poorly for remote sensing (RS). The distinct overhead viewpoint, scale variation, and presence of small objects in high-resolution RS imagery present a unique challenge in region-level comprehension. Moreover, the development of the grounding conversation capability of LMMs within RS is hindered by the lack of granular, RS domain-specific grounded data. Addressing these limitations, we propose GeoPixel - the first end-to-end high resolution RS-LMM that supports pixel-level grounding. This capability allows fine-grained visual perception by generating interleaved masks in conversation. GeoPixel supports up to 4K HD resolution in any aspect ratio, ideal for high-precision RS image analysis. To support the grounded conversation generation (GCG) in RS imagery, we curate a visually grounded dataset GeoPixelD through a semi-automated pipeline that utilizes set-of-marks prompting and spatial priors tailored for RS data to methodically control the data generation process. GeoPixel demonstrates superior performance in pixel-level comprehension, surpassing existing LMMs in both single-target and multi-target segmentation tasks. Our methodological ablation studies validate the effectiveness of each component in the overall architecture. Our code and data will be publicly released.

1. Introduction

GeoPixel addresses the difficulty of fine-grained remote-sensing understanding by combining high-resolution analysis with pixel-level grounding and grounded descriptions. The paper also introduces GeoPixelD and a benchmark for evaluating spatially grounded RS conversations.

  • Research gap: General-domain multimodal models struggle with remote-sensing imagery because of overhead viewpoints, object-scale variation, orientation changes, and difficult spatial footprints.Narrow bridges and other challenging structures can be especially difficult to segment in high-resolution imagery.
  • Research gap: Existing RS vision-language models commonly use bounding-box coordinates, which provide coarse localization and add computational burden as the number of objects increases.The paper argues that this representation is inadequate for associating precise object semantics with imagery.
  • GeoPixel: GeoPixel generates detailed natural-language responses paired with geospatial object-segmentation masks for high-resolution RS images.Its design includes adaptive local-global image partitioning and supports inputs up to 4k resolution.
  • GeoPixelD: GeoPixelD contains 53,816 grounded phrases linked to 600,817 object masks, using hierarchically structured annotations for scene-level and object-level descriptions.A semi-automated pipeline combines prior-informed visual prompting with LMMs and verification and filtering steps.
  • Evaluation benchmark: The benchmark contains 5,427 manually validated referring-expression and segmentation-mask pairs covering 61,384 annotated objects, with descriptions averaging 647 characters.It is intended to evaluate complex spatially grounded interpretation and response generation in RS imagery.

2. Related Work

Related work progresses from general multimodal and coarse region-grounding systems toward pixel grounding, but RS models remain constrained by low resolution and limited pixel-level understanding. GeoPixel is positioned against these gaps in spatial precision and high-resolution interpretation.

  • Grounding LMMs: Grounding LMMs associate textual outputs with object locations, progressing from coarse bounding boxes toward more fine-grained pixel-grounding approaches.LISA, PixelLM, and GLaMM use segmentation tokens to condition mask generation, while other systems assign segmentation tasks to specialist components.
  • Grounding LMMs: Existing pixel-grounding models do not adequately interpret RS top-down perspectives or complex spatial arrangements and often accept inputs limited to 224×224 dimensions.Restricted input size constrains field of view and spatial perception.
  • Comparison with GeoPixel: Table 1 characterizes current RS-LMM grounding primarily through region outputs based on bounding-box coordinates, with no listed model providing pixel grounding through detailed segmentation masks.Pixel grounding is presented as important for fine-grained spatial interpretation.
  • RS LMMs: RS LMMs support functions including captioning, region-specific grounding, video captioning, multisensor interpretation, scene-graph generation, and relation reasoning.Examples include RSGPT, GeoChat, SkyEyeGPT, EarthGPT, EarthDial, and SkySenseGPT.
  • RS LMMs: Many existing RS models operate at low resolution and lack pixel-level understanding and grounding capabilities.This limits their ability to associate generated text precisely with image content.

3. Method

GeoPixel addresses RS grounding and resolution limits with adaptive high-resolution image processing, multimodal alignment, and pixel-mask decoding. Its architecture combines patch and global visual features with LLM-generated segmentation-token embeddings.

  • Architecture: GeoPixel combines an adaptive image divider, vision encoder, LLM, grounding vision encoder, and pixel decoder for high-resolution perception and pixel-level grounding.These five components jointly support high-resolution perception, fine-grained interpretation, and grounding.
  • High Resolution Understanding: The adaptive divider up-scales and pads input images to a grid, then partitions them into non-overlapping patches while retaining a resized global view.Patch dimensions are determined by the vision encoder’s base resolution and maximum patch count.
  • High Resolution Understanding: Patch and global features are concatenated with separator tokens and projected into InternLM2 7B through a two-layer vision projector.The projected features replace the <IMAGE> placeholder in the multimodal input sequence.
  • High Resolution Understanding: Partial LoRA selectively adapts vision tokens for multimodal alignment while preserving the LLM’s capabilities and reducing computational cost.The modality-specific plug-in applies low-rank adaptations only to visual tokens.
  • Pixel Grounding: A frozen SAM-2 grounding encoder and trained pixel decoder combine image features with projected <SEG> embeddings to generate segmentation masks.The text projection layer maps 4096-dimensional LLM embeddings into the pixel decoder’s 256-dimensional input space; dynamic masking accommodates variable image-token lengths.

4. GeoPixelD-RS Pixel Grounding Dataset

GeoPixelD builds hierarchical, mask-aligned RS descriptions through holistic, instance, and group-level annotation. Its semi-automated pipeline combines SOM prompting with spatial and categorical priors, followed by filtering, paraphrasing, and manual test-set curation.

  • Dataset Structure: GeoPixelD provides holistic scene, individual instance, and dense object-group annotations to represent RS context and object-level details across spatial scales.The dataset is designed for fine-grained understanding of top-down imagery and complex inter-object relationships.
  • Holistic Image Annotation: Holistic captions are generated with IXC using short, constrained prompts to produce concise, context-rich image descriptions.The constrained format is intended to reduce redundancy in subsequent annotation stages.
  • Individual Instance Annotation: SOM prompting marks prominent objects, while RS-specific spatial and categorical priors guide the model toward the intended object and its details.The priors include category names, locations derived from grid overlap, and marked numbers.
  • Cluster Annotation: A three-stage positional analysis uses 3×3, 2×2, and half-image grids to localize dispersed or grouped objects.The 3×3 grid assigns unique cell identifiers and gives the central region greater spatial weight.
  • Unifying Annotations and Language Marking: The training subset contains 16,795 holistic, 36,793 instance-specific, and 17,023 group annotations covering 600,817 objects.Annotations are filtered for aerial-perspective inconsistencies, text-marked with identifiers linked to masks, and paraphrased while preserving marked phrases.
  • Evaluation: GeoPixel outperforms the compared models across all reported RS-GCG metrics in Table 2.The comparison includes pretrained and GeoPixelD-finetuned baselines, as well as zero-shot and finetuned GLaMM variants.

5. Experiments

Experiments compare GeoPixel with adapted grounding baselines on RS-GCG and RRSIS, then examine resolution, annotation complexity, data complexity, projection training, and failure cases. GeoPixel performs strongly across grounded conversation and referring-expression segmentation, while dense and ambiguous scenes remain challenging.

  • Comparative Results: GeoPixel outperforms LISA†, PixelLM†, and GLaMM-ft across RS-GCG metrics, including Uni-Target, MultiTarget, and Overall evaluations.The metrics include CIDEr, METEOR, AP50, mIoU, and recall.
  • Comparative Results: GeoPixel-ft demonstrates superior performance to recent approaches on the RRSIS-D test and validation sets for referring remote sensing image segmentation.The task generates segmentation masks from textual referring expressions.
  • Ablation Study: At P = 9, CIDEr increases from 14.6 to 20.5 and METEOR from 23.1 to 24.3 as inference patches increase.Moderate mAP and mIoU gains indicate improved localization while lower-resolution inference can remain competitive after high-resolution pretraining.
  • Ablation Study: GeoPixel selects instance, semantic, or hybrid masks according to object size, distribution, and annotation requirements.Instance masks delineate individual objects, semantic masks represent broader categories, and mixed annotations combine both representations.
  • Ablation Study: Mixed annotations produce the lowest mask recall, while visually dense scenes create difficulty in generalizing masking decisions across scale and spatial variability.Semantic-only cases also show the lowest mIoU, reflecting challenges in covering all category instances and grouping them under unified masks.
  • Ablation Study: More complex annotation data in Set-1B improves segmentation accuracy and descriptive detail, and training the vision projection layer improves some metrics.Set-1B includes individual masks for larger same-class objects and grouped semantic masks for smaller objects; projection training highlights feature alignment.
  • Limitations and Challenges: GeoPixel sometimes produces erroneous, fragmented, overlapping, or spatially confused masks, especially for densely populated or crowded images.The reported failures involve ambiguity in object presence, quantity, annotation type, and instance delineation.
  • Limitations and Challenges: Future work targets more robust masking and dynamic resolution adjustment to improve segmentation accuracy in complex scenes.The paper also identifies integration of SAR or infrared imagery as a possible extension.

6. Conclusion

GeoPixel is an end-to-end architecture for high-resolution remote sensing analysis that combines adaptive image partitioning with pixel-level grounding. It supports precise geospatial descriptions and addresses low-resolution and coarse object-grounding limitations.

  • GeoPixel is designed specifically for the challenges of high-resolution remote sensing image analysis.
  • Its architecture combines adaptive image partitioning with pixel-level grounding for geospatially aware descriptions.
  • GeoPixel provides fine-grained visual understanding to address low-resolution constraints and coarse object-grounding in current multimodal models.

A. GeoPixelD dataset

GeoPixelD is built through a semi-automated annotation pipeline that uses instance-level data and compares marking strategies for grounded conversation generation. The pipeline also uses prior-informed prompting and paraphrasing to preserve concise object-mask associations.

  • Preprocessing and Marking: The pipeline starts from instance-level iSAID annotations and crops images into 800 x 800 pixel patches.Objects are selected using an area threshold to keep markers from obscuring substantial portions of objects.
  • Preprocessing and Marking: Numerical markers placed directly on objects preserve visual clarity better than bounding boxes and contours, which add superfluous information.The comparison evaluates bounding boxes, masks, contours, and numerical markers for effects on model accuracy and object fidelity.
  • Prompting: Prior-informed SOM prompting supplies marker positions and object categories to reduce ambiguity and guide the vision-language model.Without priors, the model may produce inaccurate descriptions and hallucinated markers in complex remote sensing scenes.
  • Prompting: The SOM comparison includes open-source and proprietary models, with incorrect and correct outputs marked in red and green, respectively.
  • Annotation Unification: LLM paraphrasing combines holistic, individual, and cluster annotations into concise descriptions while preserving numerical object-mask associations during reordering.Objects are indexed numerically to track their associated masks.
  • Grounded Conversation Generation: GeoPixel interprets referring expressions with varied lengths, spatial descriptors, and object characteristics to generate segmentation masks across scale variations.The qualitative examples include descriptors such as “right,” “lower right,” and “red.”
Loading 2501.13925v1…