PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Efficient Open-Vocabulary Segmentation in the Remote Sensing

Bingyu Li, Haocheng Dong, Da Zhang, Zhiyuan Zhao, Junyu Gao, Xuelong Li

arXiv:2509.12040v3cs.CVcs.AI

TL;DR

OVRSIS remains underexplored because standardized evaluation is absent and existing methods inadequately model remote-sensing characteristics. The paper introduces OVRSISBench and RSKT-Seg, whose integrated design improves segmentation performance and inference speed over existing approaches.

  • Problem

    OVRSIS lacks a standardized benchmark, while existing methods show limited transferability and insufficient modeling of remote-sensing-specific characteristics.

  • Method

    The paper constructs OVRSISBench and proposes RSKT-Seg, integrating RS-CMA, RS-Fusion, and RS-Transfer for efficient remote-sensing segmentation.

  • Results

    Evaluations on OVRSISBench show that RSKT-Seg outperforms existing baselines while achieving 2× faster inference.

  • Takeaways & Limitations

    OVRSISBench provides a standardized reference for comparing methods, while RSKT-Seg advances the task through remote-sensing-specific modeling and efficient inference.

  • Takeaways & Limitations

    Existing OVRSIS methods still inadequately model remote-sensing characteristics such as rotation invariance and large-scale spatial context.

Abstract

from arXiv · show

Open-Vocabulary Remote Sensing Image Segmentation (OVRSIS), an emerging task that adapts Open-Vocabulary Segmentation (OVS) to the remote sensing (RS) domain, remains underexplored due to the absence of a unified evaluation benchmark and the domain gap between natural and RS images. To bridge these gaps, we first establish a standardized OVRSIS benchmark (\textbf{OVRSISBench}) based on widely-used RS segmentation datasets, enabling consistent evaluation across methods. Using this benchmark, we comprehensively evaluate several representative OVS/OVRSIS models and reveal their limitations when directly applied to remote sensing scenarios. Building on these insights, we propose \textbf{RSKT-Seg}, a novel open-vocabulary segmentation framework tailored for remote sensing. RSKT-Seg integrates three key components: (1) a Multi-Directional Cost Map Aggregation (RS-CMA) module that captures rotation-invariant visual cues by computing vision-language cosine similarities across multiple directions; (2) an Efficient Cost Map Fusion (RS-Fusion) transformer, which jointly models spatial and semantic dependencies with a lightweight dimensionality reduction strategy; and (3) a Remote Sensing Knowledge Transfer (RS-Transfer) module that injects pre-trained knowledge and facilitates domain adaptation via enhanced upsampling. Extensive experiments on the benchmark show that RSKT-Seg consistently outperforms strong OVS baselines by +3.8 mIoU and +5.9 mACC, while achieving 2x faster inference through efficient aggregation. Our code is \href{https://github.com/LiBingyu01/RSKT-Seg}{\textcolor{blue}{here}}.

Introduction

OVRSIS remains underexplored because remote sensing introduces domain-specific challenges and lacks standardized evaluation. The paper addresses these gaps with OVRSISBench and the RSKT-Seg framework.

  • OVRSIS extends open-vocabulary segmentation to remote sensing, enabling segmentation of novel classes from text descriptions.
  • The field lacks a standardized benchmark, while prior evaluations use limited datasets or inconsistent experimental setups.
  • Directly transferring classic OVS models from natural images to remote sensing causes a significant performance drop.
  • Existing OVRSIS methods often underuse remote sensing priors and overlook rotation invariance, large-scale context, and spectral diversity.
  • RSKT-Seg combines RS-CMA, RS-Fusion, and RS-Transfer to model remote sensing characteristics while targeting high accuracy and fast inference.

Related Works

Related work has progressed from conventional semantic segmentation toward open-vocabulary remote sensing methods, but existing approaches remain exploratory and incompletely incorporate remote sensing knowledge.

  • Semantic segmentation evolved from fully convolutional networks toward encoder-decoder architectures such as SegNet and U-Net.
  • OVRSIS is an emerging field with preliminary exploratory research and few established methods.
  • Existing OVRSIS methods include incremental adaptations, specialist-generalist modeling, and train-free approaches across remote sensing datasets.
  • RSKT-Seg incorporates rotational invariance and remote sensing domain knowledge, achieving satisfactory performance across multiple datasets.

Benchmark: OVRSISBench

OVRSISBench provides a unified open-vocabulary benchmark by adapting diverse remote sensing datasets and enforcing cross-dataset evaluation for unseen-class generalization.

  • OVRSISBench reformulates widely used remote sensing datasets under an open-vocabulary setting for standardized evaluation.
  • The benchmark adapts eight datasets spanning urban, agricultural, and high-resolution aerial scenes.
  • Training uses DLRSD and iSAID, while evaluation covers all eight datasets with consistent splits across methods.
  • Its cross-dataset transfer protocol trains on DLRSD and iSAID and evaluates on target datasets with partially disjoint category sets.
  • Evaluation uses mean Intersection over Union and mean Accuracy, with classic OVS methods performing worse than dedicated OVRSIS methods under the same backbone.

Method: RSKT-Seg

RSKT-Seg transfers remote-sensing knowledge into open-vocabulary segmentation through rotation-aware cost aggregation, efficient spatial and class fusion, and feature-guided upsampling.

  • RS-CMA: RS-CMA computes vision-language similarities across four rotated image views and combines them with DINO-based features to form rotation-aware, domain-aware cost maps.The input is augmented at 0°, 90°, 180°, and 270°; CLIP encodes each view, while DINO encodes the original image.
  • RS-Fusion: RS-Fusion enhances the cost map along spatial and class dimensions using SET and CET transformer blocks.SET aggregates spatial context through cross-attention, while CET processes class-wise interactions and captures dependencies across categories.
  • RS-Fusion: A spatial-reduction module projects concatenated cost-map and intermediate CLIP/DINO features into a compressed tensor for faster inference.The reduced feature serves as key and value in the spatial transformer block.
  • RS-Transfer Upsample: RS-Transfer Upsample concatenates the upsampled fused cost map with intermediate RemoteCLIP, CLIP, and DINO features before projection and refinement.The final prediction is produced at the original image resolution with shape Nt × H × W.
  • Training: The model is trained with standard cross-entropy loss against one-hot segmentation masks to produce pixel-level class predictions.Minimizing the loss encourages accurate pixel-level class predictions.

Experiment

Across eight OVRSIS datasets, RSKT-Seg consistently outperforms classic OVS and existing OVRSIS methods, while ablations support its components and efficient inference.

  • Main results: RSKT-Seg outperforms classic OVS and existing OVRSIS methods across 8 OVRSIS datasets.It remains robust across training datasets, backbones, and varied remote sensing scenarios.
  • Component ablation: 48.72 m-mIoU is achieved on DLRSD when R-I Cost Map and DINO Cost Map are combined, compared with 46.42 for the corresponding configuration.Adding RS-Transfer and RS-Fusion produces the best ablation results.
  • Remote sensing knowledge: Remote-sensing-pretrained rsDINO is more effective than natureDINO for enhancing performance on DLRSD.Adding the DINO component significantly improves performance, with rsDINO providing stronger results.
  • Fusion strategies: Mean, concatenation, and separate cost-map fusion strategies have similar effects on m-mIoU, m-fwIoU, and m-mACC.The reported metric values vary only slightly across the three strategies.
  • Aggregation layers: N=5 for DLRSD and N=2 for iSAID are selected as optimal Cost Aggregation Module configurations.The selection is based on experiments analyzing different aggregation layers.
  • Inference speed: 65.11 ms average inference time makes RSKT-Seg faster than Cat-Seg and OVRS across the evaluated datasets.The speed comparison reports inference time and FPS for multiple methods.

Visualization

Visualization comparisons show that RSKT-Seg produces more accurate remote sensing segmentations, including fewer misclassifications and more precise object boundaries.

  • Cost-map visualization: RS-Fusion refines CLIP and DINO cost maps across classes such as impervious surfaces, trees, and buildings.The visualizations represent areas of higher or lower classification confidence.
  • Segmentation visualization: RSKT-Seg shows fewer misclassifications and more precise object boundaries than other models when trained on DLRSD and tested on Vaihingen.Each visualization row represents a model and each column represents an image sample.

Conclusion

The paper concludes that OVRSISBench addresses evaluation inconsistency, while RSKT-Seg combines remote sensing-specific modeling with efficient inference to improve OVRSIS performance.

  • Conclusion: OVRSISBench provides a unified benchmark for evaluating open-vocabulary remote sensing segmentation.The benchmark supports consistent assessment of methods in the field.
  • Conclusion: RSKT-Seg integrates RS-CMA, RS-Fusion, and RS-Transfer to address remote sensing segmentation challenges.The framework combines rotation-aware aggregation, cost-map fusion, and remote sensing knowledge transfer.
  • Conclusion: RSKT-Seg outperforms baselines while achieving 2× faster inference.The conclusion presents accuracy and efficiency as the framework’s principal outcomes.

B: The analysis of computational efficiency

The efficiency analysis reduces spatial and class dimensions before Transformer attention, lowering sequence lengths and computational cost while preserving the proposed fusion pipeline.

  • Efficiency results: RSKT-Seg reports 65.11 ms average inference time and lower training-time burden than slower comparison methods.The efficiency comparison highlights suitability for resource-constrained scenarios.
  • Spatial reduction: RS-Fusion reduces spatial feature dimensions before cross-attention to lower the quadratic attention cost.The spatial reduction changes the feature-map sequence length before attention-score computation.
  • Class reduction: RS Class Fusion Transformer uses average pooling to reduce the class dimension before self-attention.This shortens the sequence involved in class-wise attention computation.

C: Datasets introduction

The benchmark draws on multiple remote sensing datasets spanning urban, land-cover, and aerial imagery, with varied category counts and train-validation splits.

  • Dataset composition: The benchmark includes datasets ranging from 2254 to 20,102 images and spanning 5 to 17 semantic categories.Reported datasets include Vaihingen, Potsdam, UDD5, and other remote sensing collections.
  • Dataset diversity: The dataset collection supports evaluation across diverse remote sensing scenes and category sets.
  • Dataset composition: UAVid contains 300 high-resolution unmanned-aerial images annotated with 8 semantic categories, split into 200 training and 100 validation images.
  • Dataset composition: LoveDA contains 18,000 images across 7 land-cover classes, with 14,168 used for training and 3,832 for validation.
  • Dataset composition: VDD contains 7,992 images labeled across 7 categories, using 5,994 for training and 1,998 for validation.

D: Evaluation metric

Evaluation uses standard segmentation metrics and compares cost-map processing and model outputs across datasets, categories, and baselines. The reported comparisons include per-class and overall mIoU, alongside visual failure cases and qualitative differences.

  • Evaluation metrics: mIoU averages intersection-over-union across classes, while fwIoU weights classes by frequency and mACC evaluates per-class accuracy.
  • Cost-map analysis: Cost-map and cost-aggregation comparisons cover categories in DLRSD and Vaihingen, including airplane, buildings, cars, impervious surfaces, building, and tree.
  • Quantitative results: RSKT-Seg reaches an overall mIoU of 34.33 on Vaihingen when trained on DLRSD.
  • Quantitative results: On Vaihingen, RSKT-Seg achieves 35.27 mIoU for Impervious surfaces and 46.71 mIoU for Tree.
  • Qualitative comparison: Qualitative comparison shows RSKT-Seg produces sharper building boundaries, clearer vegetation distinctions, and more coherent impervious-surface segmentation than the baseline.
  • Failure cases: Shadows cause misclassification, while height differences between low vegetation and trees remain difficult to distinguish.

I: Limitation and Future Work

The model remains vulnerable to shadows and height-related confusion between low vegetation and trees. Future work proposes adding depth information to improve distance and height awareness while reducing shadow effects.

  • Limitations: Shadows interfere with object-boundary and category recognition, causing sub-optimal segmentation of impervious surfaces, trees, and cars.
  • Limitations: The model confuses low vegetation with trees because it cannot reliably distinguish their height-related features.
  • Future work: Future work proposes introducing depth modality to improve distance sensing, height discrimination, and robustness to shadows.
Loading 2509.12040v3…