PaperlayerView exact revision on arXiv

Source-linked AI summary

RemoteSAM: Towards Segment Anything for Earth Observation

Liang Yao, Fan Liu, Delong Chen, Chuanyi Zhang, Yijun Wang, Ziyun Chen, Wei Xu, Shimin Di, Yuhui Zheng

arXiv:2505.18022v3cs.CV

TL;DR

Existing remote-sensing foundation models struggle to unify heterogeneous tasks and achieve fine-grained pixel-level prediction, while referring-segmentation datasets have limited semantic coverage. RemoteSAM combines an automated, VLM-assisted 270K image-text-mask dataset with a referring-segmentation-centered unified architecture. It reports stronger performance across multiple perception tasks, including over 3.0% mIoU gains on RRSISD and RisBench and a 35% accuracy gain over GeoChat in multi-label classification.

  • Problem

    Existing systems rely on task-specific architectures or text-based unification and have limited semantic coverage, restricting unified fine-grained pixel-level perception.

  • Method

    RemoteSAM combines automated VLM-assisted curation of RemoteSAM-270K with a referring expression segmentation paradigm that supports multiple tasks in one architecture.

  • Results

    RemoteSAM establishes strong results across classification, detection, segmentation, and grounding, including more than 3.0% mIoU gains on RRSISD and RisBench and a 35% accuracy gain over GeoChat.

  • Takeaways & Limitations

    Referring-segmentation-based task unification provides a parameter-efficient foundation model for diverse Earth-observation visual perception tasks.

  • Takeaways & Limitations

    Text-based unification approaches remain intrinsically limited for dense pixel-level prediction because natural language is unsuitable for representing dense outputs.

Abstract

from arXiv · show

We aim to develop a robust yet flexible visual foundation model for Earth observation. It should possess strong capabilities in recognizing and localizing diverse visual targets while providing compatibility with various input-output interfaces required across different task scenarios. Current systems cannot meet these requirements, as they typically utilize task-specific architecture trained on narrow data domains with limited semantic coverage. Our study addresses these limitations from two aspects: data and modeling. We first introduce an automatic data engine that enjoys significantly better scalability compared to previous human annotation or rule-based approaches. It has enabled us to create the largest dataset of its kind to date, comprising 270K image-text-mask triplets covering an unprecedented range of diverse semantic categories and attribute specifications. Based on this data foundation, we further propose a task unification paradigm that centers around referring expression segmentation. It effectively handles a wide range of vision-centric perception tasks, including classification, detection, segmentation, grounding, etc, using a single model without any task-specific heads. Combining these innovations on data and modeling, we present RemoteSAM, a foundation model that establishes new SoTA on several earth observation perception benchmarks, outperforming other foundation models such as Falcon, GeoChat, and LHRS-Bot with significantly higher efficiency. Models and data are publicly available at https://github.com/1e12Leon/RemoteSAM.

1. Introduction

RemoteSAM unifies heterogeneous Earth-observation perception tasks around referring expression segmentation, supported by an automatically curated, semantically diverse dataset. It improves pixel-level performance while retaining broad task compatibility and parameter efficiency.

  • Motivation: Existing task-specific-head methods limit knowledge sharing and require task-specific fine-tuning, while text-based approaches are intrinsically limited for dense pixel-level outputs.The limitations motivate a unified architecture supporting heterogeneous input-output interfaces.
  • Modeling: The model is designed for seamless upward compatibility from fundamental pixel-level outputs to region-level and image-level tasks.Natural-language understanding enables adaptation through text prompts while preserving pixel-level output granularity.
  • Modeling: RemoteSAM unifies pixel-level segmentation, region-level grounding, and image-level counting within one architecture centered on referring expression segmentation.Dense pixel-level outputs are converted into formats required by other vision-centric tasks.
  • Data: 270K image-text-mask triplets form RemoteSAM-270K, with broad category coverage and linguistically varied attributes including colors, states, and spatial relations.The dataset is built through automated curation, VLM-generated referring expressions, and iterative pseudo-label refinement with mixed teacher models.
  • Results: More than 3.0% mIoU improvements over existing methods are reported on both RRSISD and RisBench for referring expression segmentation.The comparison concerns fine-grained pixel-wise prediction benchmarks.
  • Results: RemoteSAM achieves state-of-the-art semantic segmentation without fine-tuning and a 35% accuracy gain over GeoChat in multi-label classification.The model uses an order-of-magnitude smaller parameter count, from billions to millions.

2. Related Work

Prior remote-sensing foundation models address specialized representation, temporal, or scale challenges, while referring-segmentation datasets remain difficult to use at foundation-model scale. RemoteSAM-270K responds with multi-source dataset construction and VLM-generated semantic expressions.

  • Remote Sensing Foundation Models: Remote-sensing foundation models target multimodal signals, fine-grained spatial detail, geospatial relationships, and temporal dynamics absent from ordinary natural-image settings.The related work includes RingMo, SatMAE, Scale-MAE, and BFM as responses to these domain challenges.
  • Remote Sensing Foundation Models: RingMo introduces generative self-supervision with multimodal augmentation and scene-aware contrastive learning for complex remote-sensing scenes.SatMAE instead models temporal evolution through temporal embeddings and cross-time mask reconstruction.
  • Remote Sensing Foundation Models: Scale-MAE uses scale-aware masking and a hierarchical decoder to learn scale-invariant geospatial features, while BFM uses mixture-of-experts and distributed training.These methods address multi-scale representation and large-model construction, respectively.
  • Referring Segmentation: Existing referring remote-sensing segmentation methods include RRSIS, which defines the task and provides RefSegRS, but direct transfer from natural-image methods is difficult.LGCE adds language-guided cross-scale enhancement for small-object segmentation.
  • Referring Segmentation: RemoteSAM-270K expands category coverage and attribute diversity through multi-source integration and semantically dense VLM-generated referring expressions.Figure 3 presents the dataset as a generalized referring-segmentation resource.

3. RemoteSAM-270K Dataset

RemoteSAM-270K addresses limited semantic coverage in remote sensing referring segmentation datasets with a scalable automated curation pipeline. It combines diverse data sources, VLM-generated expressions, automatically produced masks, and quality filtering to create a broad dataset.

  • Dataset scale and coverage: The dataset expands category coverage and attribute diversity to address limitations in existing remote sensing referring segmentation datasets.
  • Data construction: The curation pipeline integrates diverse remote sensing datasets and standardizes their formats before generating pixel-wise annotations.
  • Data construction: Triplet generation combines existing annotations, category-based expressions with class-specific masks, and null-mask samples for absent categories.
  • Data quality: Expert models generate pseudo-label masks, while SigLIP2 filters samples by similarity between mask-related image regions and expressions.
  • Dataset scale and coverage: RemoteSAM-270K contains over 270,000 image-text-mask triplets spanning 297 categories and 16 fine-grained attribute types.
  • Semantic analysis: RSVocab-1K analysis shows RemoteSAM-270K has superior category completeness and richer attribute expressiveness.
  • Attribute expansion: VLM-generated captions provide more detailed attributes and flexible syntax than rule-based expressions, supporting richer semantic representations.

4. RemoteSAM

RemoteSAM uses pixel-level referring expression segmentation as a unified foundation for converting visual inputs into outputs across multiple task levels. Its framework supports segmentation, grounding, detection, classification, captioning, and counting through mask-based task transitions.

  • Architecture: Pixel-level masks provide RemoteSAM’s foundational computation unit for preserving spatial precision while supporting region- and image-level tasks.The framework uses referring segmentation outputs as the basis for higher-level task interfaces.
  • Architecture: RemoteSAM predicts a segmentation mask from an image-text query using visual and textual encoders followed by a cross-modal fusion-decoder.The model is trained by minimizing segmentation loss between the predicted and ground-truth masks.
  • Task transition: Task-specific functions convert the base predicted mask into outputs such as bounding boxes for detection and class probabilities for multi-label classification.This transition framework preserves a shared mask-based representation while adapting outputs to each task’s required space.
  • Pixel-level tasks: Semantic segmentation aggregates masks generated from generalized referring expressions for every category in the task’s category set.For each class, the model uses an expression such as “All {c} in the image,” then filters predictions using category confidence and τseg.
  • Region-level tasks: Visual grounding converts each predicted referring mask into a bounding box with a mask-to-bbox procedure.The box is defined by the minimum and maximum predicted-mask coordinates.
  • Image-level tasks: Image-level outputs derive from semantic segmentation and detection: classification selects or thresholds class confidence, captioning combines categories, counts, and spatial relations, and counting totals detected instances.Image classification selects the class with the highest confidence, while multi-label classification marks classes whose confidence exceeds τcls.

5. Experiments

Experiments evaluate RemoteSAM’s semantic coverage and unified handling of pixel-, region-, and image-level Earth observation tasks. Results show strong generalization and competitive or superior performance across the evaluated benchmarks, with parameter efficiency and no task-specific decoders.

  • Experimental scope: RemoteSAM was evaluated for semantic coverage through unseen-dataset segmentation and zero-shot classification, and for task unification across pixel-, region-, and image-level tasks.The downstream evaluation included semantic segmentation, visual localization, multi-label classification, and object counting.
  • Semantic coverage: 16.9% higher performance than SegEarth-OV on Vaihingen demonstrates strong cross-domain generalization on an unseen dataset.The results also indicate open-set identification capacity.
  • Semantic coverage: RemoteSAM-270K outperformed GroundedSAM2 in SATIN zero-shot recognition across more than 250 categories and varied resolutions and viewpoints.SATIN was used to assess category diversity and recognition performance.
  • Pixel-level tasks: 71.75% mIoU on RRSISD surpassed RS2-SAM2 by 5.03%, while RisBench improved by 3.21 percentage points.These referring segmentation results were attributed to the extensive semantic coverage of the training dataset.
  • Pixel-level tasks: 91.80% accuracy on Potsdam established state-of-the-art semantic segmentation without task-specific tuning.Dataset-specific training further reached 67.01% and 93.54% on the reported evaluations.
  • Region-level tasks: RemoteSAM performed as accurately as specialized models for visual grounding while using substantially fewer parameters than other vision-language models.This supports unified region-level processing without requiring a specialized model for the task.
  • Image-level tasks: 94% and 95% accuracy on DIOR and DOTA surpassed GeoChat by 29% and 35%, respectively, in multi-label classification.The authors associate the improvement with comprehension of complex spatial relationships.
  • Image-level tasks: 62.0% and 40.9% accuracy were achieved for object counting, significantly outperforming approaches such as LHRS-Bot.Counting tests image-level compositional perception and reasoning.

6. Conclusion

RemoteSAM is a unified Earth observation foundation model that addresses pixel-level limitations through a referring segmentation-centered paradigm. Its 270K-triplet dataset and semantic diversity support strong performance across perception tasks with parameter efficiency.

  • Conclusion: RemoteSAM uses a referring segmentation-based paradigm to address pixel-level limitations in existing remote sensing foundation models.The model is presented as a unified visual foundation model for Earth observation.
  • Conclusion: Extensive evaluations show superiority in classification, detection, segmentation, and grounding with significant parameter efficiency.The conclusion frames segmentation-centric architectures as unified backbones for multimodal Earth observation intelligence.

A. Quantitative comparison results for remaining tasks

Additional evaluations report strong performance on referring segmentation, image captioning, object detection, and complex-scene classification. Comparisons with GroundedSAM2 further support generalization associated with scaled semantic coverage.

  • Referring Segmentation: RemoteSAM achieved top results across nearly all evaluated metrics on three referring segmentation datasets, with only minor deviations in a few cases.The detailed comparisons are reported in Tables 8–10.
  • Image Caption: 12.370 CIDEr on UCM-Captions was competitive with generic foundation models in zero-shot image captioning.The reported captioning strategy was rule-based.
  • Complex Scenes Classification: Category-level comparisons with GroundedSAM2 across SATIN complex-scene datasets provide evidence that scaled semantic coverage translates to improved generalization.The comparisons cover UCM Multilabel, AID Multilabel, MultiScene, and MLRSNet.

B.1. Task1: Referring Segmentation

Figure 14 presents RemoteSAM’s referring segmentation task within the paper’s unified task framework.

  • Task1: Referring Segmentation: Task1 is defined as referring segmentation.The figure labels the first task as “Referring Segmentation.”

B.2. Task2: Semantic Segmentation

Figure 15 presents Task2 as semantic segmentation.

  • Task2 is identified as semantic segmentation.
  • Figure 15 corresponds to the semantic-segmentation task.
  • The section concerns semantic segmentation as Task2.

B.3. Task3: Object Detection

The supplied figure captions identify object detection, object counting, visual grounding, classification, and image captioning as successive task sections.

  • Task3 is object detection.
  • Task4 is object counting, while Task5 is visual grounding.
  • The supplied examples mention a purple groundtrackfield, a smaller dam, a black vehicle, a small ship, and several scene categories.
  • Task6 and Task7 address multi-label classification and image classification, respectively.
  • Task8 is image captioning.

C. Ablation studies

The ablations examine backbone selection, CLIP filtering, classification strategies and balance factors, and EPOC refinement for object detection.

  • Backbone Ablation: BERT with Swin-Base achieves the highest backbone-ablation performance, reaching 76.21% oIoU and 64.79% mIoU.
  • Backbone Ablation: 75.51% oIoU and 63.00% mIoU result when Transformer replaces BERT with Swin-Base retained.
  • Backbone Ablation: The ablation attributes the further decline with ConvNext-B to less effective spatial-hierarchy capture than Swin-Base.
  • Effectiveness of CLIP Filtering: CLIP-filtered pseudo-labels achieve 74.14% mIoU against manually annotated masks in a 100-image sample.
  • Classification Ablations: The classification ablations compare mask-level and probability-level multi-label strategies using area and pooled-confidence thresholds.
  • Balance Factor Analysis: λ=0.5 maximizes multi-label classification accuracy, whereas λ=1 maximizes image classification accuracy.
  • EPOC Refinement: EPOC refinement resolves limitations of M2B when processing adjacent targets in object detection.
Loading 2505.18022v3…