PaperlayerView exact revision on arXiv

Source-linked AI summary

SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images

Kaiyu Li, Shengqi Zhang, Yujie Wang, Yupeng Deng, Zhi Wang, Deyu Meng, Xiangyong Cao

arXiv:2512.08730v2cs.CV

TL;DR

Training-free open-vocabulary remote-sensing segmentation needs better localization without the complex pipelines common in CLIP-based approaches. SegEarth-OV3 adapts SAM 3 without training by fusing its semantic and instance outputs, filtering absent categories, and verifying changes across image pairs. Across semantic segmentation, change detection, and 3D segmentation benchmarks, the method demonstrates strong remote-sensing performance, while its performance is bounded on lower-resolution GF-2 imagery.

  • Problem

    Training-free open-vocabulary remote-sensing segmentation faces coarse localization and complex multi-module pipelines, especially for dense scenes with small targets.

  • Method

    SegEarth-OV3 adapts SAM 3 without training by fusing semantic and instance predictions, filtering absent categories with presence scores, and verifying changes at instance and pixel levels.

  • Results

    Across 20 semantic-segmentation, 3 change-detection, and 3D segmentation datasets, the method demonstrates strong capability; on eight segmentation benchmarks, it reaches 53.4% average mIoU versus CorrCLIP’s 40.7%.

  • Takeaways & Limitations

    SAM 3 shows potential as a unified, training-free foundation for open-vocabulary remote-sensing segmentation and derived tasks.

  • Takeaways & Limitations

    On lower-resolution GF-2 imagery, SAM 3 may struggle with blurred textures and continuous macroscopic land-cover categories because of a domain gap from natural-image training.

Abstract

from arXiv · show

Most existing methods for training-free open-vocabulary semantic segmentation are based on CLIP. While these approaches have made progress, they often face challenges in precise localization or require complex pipelines to combine separate modules, especially in remote sensing scenarios where numerous dense and small targets are present. Recently, Segment Anything Model 3 (SAM 3) was proposed, unifying segmentation and recognition in a promptable framework. In this paper, we present a comprehensive exploration of applying SAM 3 to the remote sensing open-vocabulary tasks (i.e., 2D semantic segmentation, change detection, and 3D semantic segmentation) without any training. First, we implement a mask fusion strategy that combines the outputs from SAM 3's semantic segmentation head and the Transformer decoder (instance head). This allows us to leverage the strengths of both heads for better land coverage. Second, we utilize the presence score from the presence head to filter out categories that do not exist in the scene, reducing false positives caused by the vast vocabulary sizes and patch-level processing in geospatial scenes. Furthermore, we extend our method to open-vocabulary change detection by a joint instance- and pixel-level verification strategy built directly upon our fused logits. We evaluate our method on extensive remote sensing datasets and tasks, including 20 segmentation datasets, 3 change detection datasets, and a 3D segmentation dataset. Experiments show that our method achieves promising performance, demonstrating the potential of SAM 3 for remote sensing open-vocabulary tasks. Our code is released at https://github.com/earth-insights/SegEarth-OV-3.

1 Introduction

SegEarth-OV3 explores SAM 3 as a training-free alternative for remote-sensing open-vocabulary tasks, combining its complementary heads and presence prediction to address geospatial challenges. It extends this design to open-vocabulary change detection and reports evaluation across broad remote-sensing benchmarks.

  • Motivation: Remote-sensing open-vocabulary segmentation must recognize arbitrary text-described categories while producing dense pixel-level understanding for applications such as urban planning and disaster monitoring.
  • Motivation: SAM 3 unifies promptable concept segmentation with a presence head, Transformer decoder, and semantic head for predicting category existence, discrete-instance masks, and continuous semantic regions.
  • Method: SegEarth-OV3 fuses semantic-head continuity for amorphous land cover with Transformer-decoder boundary detail for countable objects, matching the duality of remote-sensing scenes.
  • Method: Presence-guided filtering suppresses categories absent from local image patches, addressing false positives caused by large vocabularies and limited geographic coverage.
  • Applications: The framework extends to open-vocabulary change detection through joint instance- and pixel-level verification that filters registration noise and recognition inconsistencies.
  • Evaluation: Evaluation covers 20 semantic-segmentation datasets, 3 change-detection datasets, and a 3D point-cloud segmentation dataset, with results indicating strong SAM 3 capability enhanced by the proposed improvements.

2 Related Work

Related work is dominated by CLIP-based training-free methods and increasingly complex combinations with visual foundation models. SegEarth-OV3 instead builds on SAM 3’s unified outputs to reduce pipeline complexity while addressing remote-sensing geometry and open-vocabulary change detection.

  • OVSS Methods: Training-based OVSS methods adapt pretrained VLMs through mask classification or dense feature adaptation, whereas training-free methods directly modify pretrained models for dense prediction.
  • OVSS Methods: CLIP-based methods often produce coarse localization because CLIP is pretrained for image-level recognition, motivating auxiliary structural guidance, upsampling, or dual-stream designs.
  • OVSS Methods: Some prior approaches rely on separate heavy models for segmentation and recognition, creating significant system complexity.
  • Remote-Sensing Challenges: Remote-sensing OVSS introduces extreme scale variation and arbitrary orientations that can degrade methods designed for natural images.
  • Open-Vocabulary Change Detection: OVCD research progresses from generic SAM-feature change localization and category-specific CLIP methods toward arbitrary-category frameworks based on decoupled comparison and identification.
  • Open-Vocabulary Change Detection: SegEarth-OV3 uses SAM 3 features and logits as constraints to filter pseudo-changes and registration errors without complex pipelines or heavy post-processing.

3 Methods

SegEarth-OV3 uses SAM 3’s prompt-conditioned heads and presence scores to build training-free open-vocabulary segmentation and change detection for remote sensing. The method combines semantic and instance predictions, suppresses absent categories, and verifies changes at both pixel and instance levels.

  • SAM 3 Architecture: SAM 3 provides a vision-text prompt interface with presence, semantic segmentation, and query-based instance heads for concept-specific prediction.The presence head predicts concept existence, the semantic head produces dense probability maps, and the instance head outputs query predictions with confidence scores.
  • Inference Pipeline: The inference pipeline processes vocabulary categories sequentially, assigns each pixel its highest-probability category, and optionally labels low-confidence pixels as background.Multiple prompts for one category are resolved by selecting the highest probability.
  • Instance Aggregation: The Transformer decoder aggregates discrete instance predictions into category-level maps, preserving localization in dense clusters of small, identical objects.Maximum weighted aggregation consolidates query-level predictions into a unified semantic layer.
  • Dual-Head Mask Fusion: Dual-head fusion combines dense semantic coverage with instance-level boundaries, capturing both amorphous regions and small objects.The semantic head supplies global coverage, while the instance head preserves fine-grained localization; max fusion combines their strengths.
  • Presence-Guided Filtering: Presence-guided filtering suppresses categories absent from localized image patches, reducing false positives from large vocabularies and geospatial ambiguity.The method uses SAM 3’s global presence score as a soft gate on category probability maps.
  • Extension to OVCD: For open-vocabulary change detection, joint pixel- and instance-level verification filters noise while retaining sensitivity to true temporal changes.Pixel comparison uses SAM 3 visual features and semantic probability differences, while instance comparison addresses spatial misalignment; both confirmations are required.

4 Experiments

SegEarth-OV3 is evaluated across broad remote-sensing segmentation, change-detection, and 3D benchmarks, with strong results for its training-free approach. The experiments also examine component contributions and domain-specific limitations.

  • Semantic segmentation: 53.4% average mIoU on eight remote-sensing semantic-segmentation benchmarks surpasses CorrCLIP by +12.7% mIoU and exceeds reported training-based OVSS methods.The comparison uses Table 1, whose baselines include methods tuned on remote-sensing data and a fully supervised Oracle.
  • Single-class extraction: 86.9% IoU on WHUAerial and 72.4% on Inria exceed SegEarth-OV by +37.7% and +27.8%, while road and flood extraction reach 49.6% and 75.6% IoU.These results cover nine single-class extraction benchmarks involving buildings, roads, and floods.
  • Ablation studies: Dual-head fusion improves over single-head baselines by +12.0% on LoveDA, +10.1% on CHN6-CUG, and 2.9% on xBD.The Transformer decoder contributes instance detail, while the semantic head provides continuous-region coverage.
  • Specialized images: Performance is lower on GF-2 GID images, where 42.2% mIoU trails SegEarth-OV’s 46.3%, consistent with a domain gap from lower resolution and blurred, continuous land-cover textures.On GF-7 Building and Low-Grade Road, SegEarth-OV3 reaches 58.7% and 60.2% IoU, respectively.
  • Open-vocabulary change detection: SegEarth-OV3 achieves the highest performance across WHU-CD, LEVIR-CD, and S2Looking, including 88.0% F1 and 78.6% IoU on WHU-CD.The joint instance- and pixel-level verification strategy is built on SegEarth-OV3 and supports zero-shot change detection without temporal training.
  • 3D segmentation: 61.26% mIoU on STPLS3D-WMSC surpasses several fully supervised 3D models, indicating that 2D SegEarth-OV3 representations transfer across viewpoints with weighted voting.The 3D evaluation projects 2D predictions onto point clouds for assessment.

5 Conclusion

SegEarth-OV3 adapts SAM 3 for open-vocabulary remote sensing interpretation across semantic segmentation, change detection, and 3D segmentation without training. Across 24 benchmarks, it establishes a strong training-free baseline and sometimes outperforms fully supervised models.

  • Across 24 remote sensing benchmarks, SegEarth-OV3 establishes a strong training-free baseline for multi-dimensional remote sensing tasks.The evaluation covers semantic segmentation, change detection, and 3D point cloud segmentation.
  • SegEarth-OV3 fuses semantic and instance outputs and applies presence-guided filtering for remote sensing semantic segmentation.
  • The framework extends to open-vocabulary change detection through joint instance- and pixel-level verification of fused predictions.
  • In certain scenarios, SegEarth-OV3 outperforms fully supervised models, supporting SAM 3’s potential for open-vocabulary remote sensing tasks.
Loading 2512.08730v2…