PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM-Assisted Remote Sensing Imagery Semantic Segmentation with Object and Boundary Constraints

Xianping Ma, Qianqian Wu, Xingyu Zhao, Xiaokang Zhang, Man-On Pun, Bo Huang

arXiv:2312.02464v2cs.CV

TL;DR

SAM provides powerful segmentation masks but lacks semantic labels for remote sensing semantic segmentation. This paper introduces a streamlined framework that uses SAM-generated objects and boundaries as auxiliary constraints, and reports effectiveness on ISPRS Vaihingen and LoveDA Urban.

  • Problem

    SAM generates segmentation results without class information, creating a gap for using its raw outputs in remote sensing semantic segmentation.

  • Method

    The framework uses SAM-Generated Objects and Boundaries to add object consistency and boundary preservation losses to a general semantic segmentation model.

  • Results

    Experiments on ISPRS Vaihingen and LoveDA Urban demonstrate the effectiveness of the proposed method.

  • Takeaways & Limitations

    The approach directly leverages SAM’s raw output without semantic information, additional class prompts, or task-specific segmentation designs.

Abstract

from arXiv · show

Semantic segmentation of remote sensing imagery plays a pivotal role in extracting precise information for diverse down-stream applications. Recent development of the Segment Anything Model (SAM), an advanced general-purpose segmentation model, has revolutionized this field, presenting new avenues for accurate and efficient segmentation. However, SAM is limited to generating segmentation results without class information. Consequently, the utilization of such a powerful general vision model for semantic segmentation in remote sensing images has become a focal point of research. In this paper, we present a streamlined framework aimed at leveraging the raw output of SAM by exploiting two novel concepts called SAM-Generated Object (SGO) and SAM-Generated Boundary (SGB). More specifically, we propose a novel object loss and further introduce a boundary loss as augmentative components to aid in model optimization in a general semantic segmentation framework. Taking into account the content characteristics of SGO, we introduce the concept of object consistency to leverage segmented regions lacking semantic information. By imposing constraints on the consistency of predicted values within objects, the object loss aims to enhance semantic segmentation performance. Furthermore, the boundary loss capitalizes on the distinctive features of SGB by directing the model's attention to the boundary information of the object. Experimental results on two well-known datasets, namely ISPRS Vaihingen and LoveDA Urban, demonstrate the effectiveness of our proposed method. The source code for this work will be accessible at https://github.com/sstary/SSRS.

I. INTRODUCTION

Remote sensing semantic segmentation supports downstream geoscience applications, but SAM-generated masks lack semantic labels and face domain disparities between natural and remote sensing imagery. The proposed framework uses SAM-generated object and boundary information through object consistency and boundary preservation losses without requiring task-specific segmentation designs or semantic prompts.

  • Remote sensing semantic segmentation assigns semantic labels to pixels for environmental monitoring, land cover mapping, and disaster management.
  • SAM enables zero-shot segmentation but generates masks without semantic labels, while differences between natural and remote sensing images hinder direct application.
  • The framework exploits SAM-Generated Objects and Boundaries to provide object- and boundary-level information for semantic segmentation.
  • Object consistency loss constrains predicted values within segmented objects, while boundary preservation loss directs optimization toward object boundaries without semantic information or additional class prompts.
  • The approach does not require specific semantic segmentation models, training strategies, or pseudo-label generation, and experiments cover two datasets and four representative models.

II. RELATED WORKS

SAM is a prompt-driven foundation model that produces segmentation masks through image, prompt, and mask-decoder components. Related remote-sensing work adapts SAM, uses learning strategies or prompts, and addresses differences between natural and remote-sensing imagery.

  • SAM combines an image encoder, prompt encoder, and mask decoder to generate segmentation masks from image features and user interactions.
  • SAM supports mask, point, box, and text prompts, with APIs that allow segmentation masks to be obtained through concise programmatic calls.
  • Existing research addresses SAM’s remote-sensing limitations through model adaptation, few-shot or zero-shot learning, and prompt learning.
  • Remote sensing differs from natural imagery in acquisition context, spectral and spatial resolution, object scale and coverage, and content complexity.

C. Object-based methods in Remote Sensing

Object-based remote-sensing methods use object or boundary information to complement pixel-based segmentation. The proposed framework uses frozen SAM preprocessing to generate SGO and SGB, then incorporates them into auxiliary training losses alongside conventional segmentation loss.

  • Remote-sensing semantic segmentation commonly uses pixel-based and object-based approaches that learn features and predict categories at distinct scales.
  • The proposed framework feeds image X into a general segmentation model and adds object consistency and boundary preservation losses computed from SGO and SGB.
  • Prior object-based methods use object representations, superpixels, adaptive merging, or graph convolutions to strengthen regions, edges, or object relationships.
  • SAM preprocessing generates object masks across grid locations, limits their number and size, and stores them in SGO with integer object identifiers.
  • SGO assigns zero to unsegmented pixels and boundaries, while object pixels receive identifiers from 1 to K; a boundary prior is derived from SGO.

B. Network Training

The framework trains a semantic segmentation model with its standard cross-entropy loss plus SAM-derived object and boundary constraints. SGO and SGB are used only to compute these auxiliary losses, so the network and training strategy require no additional modifications.

  • The semantic segmentation model predicts P for each pixel and category, and minimizes a cross-entropy-based loss against ground-truth labels Y.
  • SGO and SGB are used to compute auxiliary losses without requiring additional network modifications or training-strategy adjustments.
  • The proposed objective combines semantic segmentation loss Lseg, object consistency loss Lobj, and boundary preservation loss Lbdy.The weighting coefficients λo and λb balance the three losses.

1) Object Consistency Loss:

The object consistency loss uses SAM-generated object regions to encourage consistent semantic predictions within each segmented object. It extracts object-specific predictions, computes their mean feature, and applies mean squared error across objects.

  • For each object in SGO, the method extracts a mask M_i and filters the semantic segmentation prediction P to obtain an object feature.The mask contains pixels whose SGO value equals the object identifier i.
  • The object’s average feature is computed by aggregating its spatial pixels and normalizing by the number of points N_i, with an added one preventing a zero denominator.The resulting F_i^avg represents the expected mean value of pixels in the i-th object.
  • Lobj is computed across all objects using mean squared error between object predictions and their corresponding average features.
  • The loss directly leverages detailed SAM-generated object masks without requiring semantic information from those regions.

2) Boundary Preservation Loss:

The boundary preservation loss uses boundary information derived from SAM-generated objects to guide semantic segmentation toward more accurate object boundaries. It is defined as one minus the boundary F1 score.

  • The method derives SGB from SGO by setting boundary pixels to zero, thereby retaining detailed object-boundary information.
  • Boundary F1 evaluates boundary detection using precision and recall computed from the model prediction P and the generated boundary map Y_b.The cited precision and recall jointly assess boundary accuracy.
  • The overall training objective combines this boundary loss with semantic segmentation and object consistency losses.

1) ISPRS Vaihingen:

The experiments use ISPRS Vaihingen and LoveDA Urban, which differ in resolution, class composition, and label accuracy. Performance is assessed with mean F1 and mean Intersection over Union, computed over the main foreground classes.

  • ISPRS Vaihingen contains 16 very-high-resolution orthophotos with NIRRG channels at 9 cm ground sampling distance and six labeled classes including clutter.Its 12 training patches and four test patches are used for experimentation.
  • LoveDA Urban contains 1833 RGB images of size 1024 × 1024 pixels at 30 cm ground sampling distance and seven land-cover categories.
  • The datasets differ in sampling resolution, object categories, and label accuracy, with Vaihingen having more precise ground truth and fewer categories than LoveDA Urban.
  • Performance is evaluated using mean F1 score and mean Intersection over Union, computed for the five foreground classes in ISPRS Vaihingen.The clutter or background class is treated as a sparse class.

C. Implementation details

The experiments use Meta AI’s SAM interface to generate SGO and SGB, with fixed thresholds and standard SGD training settings. Qualitative comparisons visualize four segmentation models on 512 × 512 Vaihingen samples.

  • Qualitative comparison: Fig. 6 compares ABCNet, CMTFNet, UNetformer, and FTUNetformer on two 512 × 512 ISPRS Vaihingen samples per model.Purple boxes highlight differences among the qualitative results.
  • Evaluation: Table I reports five foreground-class accuracies and three overall metrics for the ISPRS Vaihingen experiments.Category accuracy is presented in F1/IoU form, with bold values marking the best results.
  • Training settings: The experiments set the threshold K and S values to 50 and used stochastic gradient descent for optimization.Training used a learning rate of 0.01, momentum of 0.9, decay coefficient of 0.0005, and batch size of 10.

D. Performance Comparison

Across the ISPRS Vaihingen comparisons, adding SAM-derived object and boundary information improves segmentation for multiple models and classes. The gains appear in both quantitative metrics and qualitative object and boundary delineation.

  • Class-level results: UNetformer+SAM improved Car performance by 1.08% in F1 and 1.78% in IoU compared with UNetformer.Building performance increased by 0.49% in F1 and 0.91% in IoU.
  • FTUNetformer comparison: FTUNetformer+SAM achieved an mF1 of 91.08% and mIoU of 84.01%, increasing 0.84% and 1.39% over FTUNetformer.The framework also improved FTUNetformer’s Car and building results.
  • Qualitative results: The method precisely segmented complete objects, refined low-vegetation and building boundaries, and addressed challenging tree, building, and impervious-surface categories.These qualitative comparisons are shown for the four evaluated segmentation models.
  • Cross-dataset evaluation: Table II reports LoveDA Urban experiments using the same F1/IoU category format and overall evaluation structure.The table complements the Vaihingen comparisons with a second dataset.

2) Performance Comparison on the LoveDA Urban:

On LoveDA Urban, SAM-assisted models show dataset-level and class-level gains, especially for categories with regular shapes or uncomplicated borders. Qualitative results also show more complete object identification, while sensitivity analysis examines the two loss weights.

  • LoveDA Urban results: UNetformer+SAM improved building, water, and agriculture by 6.82%/8.39%, 2.89%/3.96%, and 23.1%/16.47% in F1/IoU over UNetformer.The corresponding UNetformer+SAM accuracies were 75.91%/61.17%, 80.55%/67.43%, and 43.64%/27.91%.
  • LoveDA Urban results: FTUNetformer+SAM achieved mF1 of 65.74% and mIoU of 50.55%, with increases of 0.4% and 1.43% over UNetformer.The reported increases over FTUNetformer were 1.07% for mF1 and 0.97% for mIoU.
  • Qualitative results: LoveDA Urban visualizations show improved building identification and more complete water and agriculture objects.These observations align with the reported mF1 and mIoU improvements.
  • Sensitivity analysis: The sensitivity analysis varies λo for object consistency and λb for boundary information to examine their separate contributions.The experiments use UNetformer because the two hyper-parameters show no strong correlation.
  • Ablation context: Tables III and IV present ablation studies for the object consistency and boundary preservation losses on the two datasets.The tables report category accuracy in F1/IoU form with bold values marking the best results.
  • Complexity context: Table V reports model complexity using FLOPs, parameters, memory footprint, running time, FPS, and Vaihingen mIoU.The analysis uses two 256 × 256 images on one NVIDIA GeForce RTX 4090 GPU.
  • Sensitivity analysis: Setting λo ≥2.0 noticeably degraded performance, while λo ∈[0.1, 1.0] produced lower sensitivity to changes.The experiments therefore used λo = 1.0 unless otherwise specified.

F. Ablation Study

Ablation experiments show that object consistency and boundary preservation losses contribute separately, while their combination provides a versatile framework across categories and models. Complexity analysis indicates no added inference cost, although training takes longer.

  • Loss ablation: Independent use of the object consistency and boundary preservation losses improved overall semantic segmentation performance.The ablation experiments used UNetformer on the ISPRS Vaihingen and LoveDA Urban datasets.
  • Loss ablation: Boundary preservation alone matched the combined loss for some impervious-surface and building results, indicating task-dependent loss behavior.The paper identifies building detection as a task where boundary preservation may fully exploit SAM-based preprocessing.
  • Loss ablation: Combining both losses is necessary for semantic segmentation involving ground objects with highly complex boundaries.The study reports disparate behaviors across categories, motivating the combined framework.
  • Complexity analysis: The framework adds no model complexity or inference time because SGO and SGB are generated before loss calculation without task-specific modules.During inference, the proposed approach operates identically to the original model.
  • Complexity analysis: The additional loss functions lengthen training time because gradient back-propagation takes longer to compute.The paper characterizes this increase as a marginal trade-off for the observed improvements.
  • Conclusion: The conclusion presents object consistency and boundary preservation losses as a simple, versatile strategy for using SAM’s raw output with different network structures.Validation covered two public datasets and four semantic segmentation models.
Loading 2312.02464v2…