PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM-Assisted Remote Sensing Imagery Semantic Segmentation with Object and Boundary Constraints

Xianping Ma, Qianqian Wu, Xingyu Zhao, Xiaokang Zhang, Man-On Pun, Bo Huang

arXiv:2312.02464v2cs.CV

TL;DR

SAM-generated masks lack semantic labels, complicating their use for remote-sensing semantic segmentation. This paper uses SAM-generated objects and boundaries to add object-consistency and boundary-preservation losses; on LoveDA Urban, UNetformer+SAM improved building F1/IoU by 6.82%/8.39% versus UNetformer.

  • Problem

    SAM's generated masks lack semantic labels, a challenge for its use in remote-sensing semantic segmentation.

    Source: §I · p. 1

  • Method

    The framework uses SAM-generated objects and boundaries to add object-consistency and boundary-preservation losses to semantic segmentation training.

    Source: §I · p. 2

  • Results

    On LoveDA Urban, UNetformer+SAM improved building F1/IoU by 6.82%/8.39%, respectively, versus UNetformer.

    Source: 2) Performance Comparison on the… · p. 10

  • Takeaways & Limitations

    The auxiliary losses facilitate improvements across different semantic segmentation network structures without requiring additional modules.

    Source: §V · p. 12

Abstract

from arXiv · show

Semantic segmentation of remote sensing imagery plays a pivotal role in extracting precise information for diverse down-stream applications. Recent development of the Segment Anything Model (SAM), an advanced general-purpose segmentation model, has revolutionized this field, presenting new avenues for accurate and efficient segmentation. However, SAM is limited to generating segmentation results without class information. Consequently, the utilization of such a powerful general vision model for semantic segmentation in remote sensing images has become a focal point of research. In this paper, we present a streamlined framework aimed at leveraging the raw output of SAM by exploiting two novel concepts called SAM-Generated Object (SGO) and SAM-Generated Boundary (SGB). More specifically, we propose a novel object loss and further introduce a boundary loss as augmentative components to aid in model optimization in a general semantic segmentation framework. Taking into account the content characteristics of SGO, we introduce the concept of object consistency to leverage segmented regions lacking semantic information. By imposing constraints on the consistency of predicted values within objects, the object loss aims to enhance semantic segmentation performance. Furthermore, the boundary loss capitalizes on the distinctive features of SGB by directing the model's attention to the boundary information of the object. Experimental results on two well-known datasets, namely ISPRS Vaihingen and LoveDA Urban, demonstrate the effectiveness of our proposed method. The source code for this work will be accessible at https://github.com/sstary/SSRS.

I. INTRODUCTION

The paper proposes a streamlined way to use SAM’s object and boundary information for remote-sensing semantic segmentation despite SAM’s lack of semantic labels and reduced effectiveness on remote-sensing imagery.

I. INTRODUCTION

  • SAM’s masks lack semantic labels, and the differences between natural and remote-sensing images compromise its effectiveness, although it can recognize objects in remote-sensing imagery.SGO and SGB provide object- and boundary-level detail, respectively.Source: §I · pp. 1–2, Figure 1 · p. 2
  • Prior approaches adapt or enhance SAM, use few-shot or zero-shot learning, or apply prompt learning, but often require complex fine-tuning or dataset-specific prompts.The paper cites SAM-CD [28], RS-CLIP, and SAMRS as examples of these approaches.Source: §I · p. 1
  • Object consistency loss encourages consistent predictions within SGO regions, while boundary preservation loss uses SGB information to guide attention to object boundaries.Both losses use the semantic-segmentation output directly, so no additional decoder segmentation heads are needed.Source: §I · p. 2
  • The framework uses SAM’s raw SGO and SGB outputs without requiring task-specific semantic-segmentation model, training-strategy, or pseudo-label designs.It is intended for direct use across different remote-sensing datasets and semantic-segmentation models.Source: §I · p. 2
  • Experiments on ISPRS Vaihingen and LoveDA Urban with four representative segmentation models support the framework’s adaptability across datasets and general models.The authors present the approach as potentially applicable to tasks combining SAM with remote-sensing semantic segmentation.Source: §I · p. 2

II. RELATED WORKS · A. Segment Anything Model · B. Segment Anything Model in Remote Sensing

SAM is a prompt-driven foundation model that can segment unseen images, while remote-sensing methods face image-domain differences and limits in turning SAM-generated regions into multiclass semantic predictions. Prior work explores prompting, dataset generation, and adaptation, motivating a broadly usable approach that exploits SAM-generated objects and boundaries.

A. Segment Anything Model

  • SAM is a vision-transformer-based foundation model that uses user prompts to generalize segmentation to unseen images and objects, addressing the need for extensive annotations.Its framework comprises an image encoder, prompt encoder, and mask decoder.Source: A. Segment Anything Model · p. 2
  • The comparison figure depicts the proposed preprocessing approach as freezing all SAM parameters and generating SGO and SGB using point prompts.It contrasts this approach with the SAM framework schematic.Source: Figure 2 · p. 3
  • SAM supports mask, point, box, and text prompts; its decoder predicts masks and IoU scores, and its APIs offer automatic, box, and point segmentation modes.The decoder maps image and prompt embeddings with an output token to a mask.Source: A. Segment Anything Model · p. 3
  • Related applications combine SAM masks with raw medical images [44], integrate its pretrained encoder with UNet [45], or use SAM-generated pseudo-labels [46] [47].These approaches have extended SAM across domains through image enhancement, feature extraction, and pseudo-label generation.Source: A. Segment Anything Model · p. 3

B. Segment Anything Model in Remote Sensing

  • Remote-sensing images differ from natural images in acquisition and context, including spectral and spatial resolution, object scale and coverage, and content complexity.Source: B. Segment Anything Model in Remote… · p. 3
  • SAMRS uses rotated-box prompts to create a remote-sensing segmentation dataset, while Text2Seg [36] uses multiple foundation models and text prompts for semantic segmentation.Prompt-learning methods [34] [36] [37] [38] require dataset-specific prompt selection; few- or zero-shot methods [32] show adaptability but can be sensitive to fine-tuning.Source: B. Segment Anything Model in Remote… · p. 3
  • Because SGO lacks semantic labels, existing methods are limited to binary classification [28] [37] or multiclass predictions using class-specific prompts [26], motivating accessible multiclass use of SGO and SGB.Source: B. Segment Anything Model in Remote… · p. 3

C. Object-based methods in Remote Sensing · III. METHODOLOGY · A. SAM-based Pre-processing

The paper places its approach among object-based remote-sensing methods and adds SAM-generated object and boundary information to semantic-segmentation training. SAM’s grid prompts produce filtered object masks and a boundary prior used with the segmentation loss.

C. Object-based methods in Remote Sensing

  • Object-based remote-sensing methods include pixel- and object-based approaches, with prior work integrating object concepts into remote-sensing tasks.OCNN proposed an object-based CNN framework for land-use classification; the supplied passage ends mid-sentence.Source: C. Object-based methods in Remote… · p. 3
  • Prior methods use superpixels or graph convolutions, while this approach uses SAM’s raw object output without specialized learnable modules.The paper says specialized modules can increase implementation complexity and potential instability; its approach integrates with existing segmentation models.Source: C. Object-based methods in Remote… · p. 4

III. METHODOLOGY

  • The framework adds SAM-generated object and boundary losses to the segmentation loss, and uses all three to optimize the model.The object and boundary losses are computed from SGO and SGB, respectively.Source: §III · p. 4, Figure 3 · p. 4

A. SAM-based Pre-processing

  • SAM’s grid-prompt technique generates masks across plausible image locations, which the method treats as individual enclosed objects.A threshold K limits the maximum number of objects in an image.Source: A. SAM-based Pre-processing · p. 4
  • A threshold S filters very small masks; the resulting SGO assigns object identifiers from 1 to K and zero to unsegmented and boundary pixels.SGO is denoted Yo ∈ RH×W, with pixel values in [0, K].Source: A. SAM-based Pre-processing · p. 5
  • The method derives SGB by merging the exterior boundaries of the objects, assigning boundary pixels 255 and all other pixels 0.SGB is denoted Yb ∈ RH×W.Source: A. SAM-based Pre-processing · p. 5

B. Network Training

The framework uses a classical encoder–decoder for semantic segmentation and trains it with a composite objective combining cross-entropy, object consistency, and boundary preservation losses. The SAM-derived losses require no network or training-strategy modifications.

B. Network Training

  • The framework uses classical encoder–decoder networks, such as UNetformer, as its semantic segmentation model, which predicts a per-pixel distribution over ground-object categories.The output is denoted P ∈ RH×W×C, where C is the number of categories.Source: B. Network Training · p. 5
  • Training minimizes a cross-entropy-based segmentation loss between predictions P and ground truth Y.Source: B. Network Training · p. 5
  • The learning objective combines segmentation, object consistency, and boundary preservation losses, with λo and λb balancing the three losses.SGO and SGB are used only to compute loss functions, so the framework requires no network or training-strategy modifications.Source: B. Network Training · p. 5

1) Object Consistency Loss:

The object consistency loss uses SAM-generated object regions to encourage pixel-level consistency in the segmentation model’s predictions. It computes object features and their mean values, then uses mean squared error to form the loss across objects.

Object Consistency Loss

  • For each object in Yo, the method extracts its mask and obtains an object feature by filtering model predictions to that region.The paper denotes the input as X and the model output as P.Source: 1) Object Consistency Loss · p. 5
  • It averages each object’s feature over its pixels, adding one to the pixel count to prevent a zero denominator.The average represents the expected mean value of the pixels in that object.Source: 1) Object Consistency Loss · p. 5
  • The loss applies mean squared error across objects and leverages SAM-generated regions to use the detailed masks in SGO.MSE denotes the mean squared error function.Source: 1) Object Consistency Loss · pp. 5–6

2) Boundary Preservation Loss: · IV. EXPERIMENTS AND DISCUSSION

The boundary preservation loss uses boundary information in SAM-generated objects to guide semantic predictions, and is combined with semantic segmentation and object consistency losses in the training objective.

2) Boundary Preservation Loss:

A. Datasets … B. Evaluation Metrics

Experiments use ISPRS Vaihingen and LoveDA Urban, which differ in resolution, categories, and label accuracy; performance is evaluated with mean F1 and mean IoU. The setup specifies dataset splits, sliding-window processing, and dataset-specific class inclusion.

1) ISPRS Vaihingen:

  • Vaihingen contains 16 NIRRG orthophotos at 9 cm ground sampling distance, averaging 2500 × 2000 pixels, with five foreground classes and clutter.The foreground classes are impervious surface, building, low vegetation, tree, and car.Source: 1) ISPRS Vaihingen · p. 6
  • The ISPRS Vaihingen dataset comprises 16 true orthophotos averaging 2500 × 2000 pixels, split into 12 training patches and 4 test patches.Source: 1) ISPRS Vaihingen · p. 6

2) LoveDA Urban:

  • LoveDA Urban comprises 1833 RGB images of 1024 × 1024 pixels at 30 cm ground sampling distance, covering seven land-cover categories.The images were collected in Nanjing, Changzhou, and Wuhan; categories are background, building, road, water, barren, forest, and agriculture.Source: 2) LoveDA Urban · p. 6
  • LoveDA Urban is divided into 1156 training images and 677 test images.Training indices span 1366–2521 and test indices span 3514–4190.Source: 2) LoveDA Urban · p. 6
  • The datasets differ in sampling resolution, category count, and label accuracy; sliding windows of 256 × 256 use stride 256 for training and stride 32 for testing.The smaller testing stride averages predictions in overlapping regions to mitigate border effects.Source: 2) LoveDA Urban · p. 6

B. Evaluation Metrics

  • Evaluation uses mF1 and mIoU; Vaihingen scores its five foreground classes but excludes clutter/background, while LoveDA Urban scores all seven categories.Per-class F1 and IoU are computed from true positives, false positives, and false negatives, then averaged across the included classes to obtain mF1 and mIoU.Source: B. Evaluation Metrics · p. 6

C. Implementation details

Experiments used SAM-generated objects and boundaries with specified generation thresholds, and trained all models using SGD with shared hyperparameters and simple augmentations.

C. Implementation details

  • SGO and SGB generation used crop NMS and box NMS thresholds of 0.5 and a prediction-IoU threshold of 0.96; higher prediction-IoU thresholds tend to yield more dependable raw output.The thresholds K and S were both set to 50.Source: C. Implementation details · pp. 6, 8
  • Experiments ran in PyTorch on a single NVIDIA GeForce RTX 4090 GPU with 24GB RAM.Source: C. Implementation details · p. 6
  • All models were trained with SGD at learning rate 0.01, momentum 0.9, decay 0.0005, and batch size 10, with random rotation and flipping augmentations.The loss weights λo and λb were selected based on sensitivity analysis in Section IV-E.Source: C. Implementation details · p. 8

D. Performance Comparison · 1) Performance Comparison on the Vaihingen dataset:

On Vaihingen, the proposed framework improves segmentation performance over baseline models, with gains in overall metrics, object classes, and visual delineation of objects and boundaries.

D. Performance Comparison

  • The framework was benchmarked with ABCNet, CMTFNet, UNetformer, and FTUNetformer, spanning CNN-based ResNet and self-attention-based Swin Transformer backbones.The quantitative results are reported in Tables I and II.Source: D. Performance Comparison · p. 8

1) Performance Comparison on the Vaihingen dataset:

  • UNetformer+SAM outperforms UNetformer across all five classes, including Car gains of 1.08% F1 and 1.78% IoU and Building gains of 0.49% and 0.91%.The paper attributes these class gains to the relatively simple, standardized shapes of cars and buildings.Source: 1) Performance Comparison on the… · p. 8
  • 91.08% mF1 and 84.01% mIoU mark overall gains of 0.84% and 1.39%, respectively, over FTUNetformer.Against FTUNetformer, Car improves by 2.63% in F1 and 4.33% in IoU, while Building improves by 0.27% and 0.51%; Tree improves notably, but Low Vegetation gains are diminished.Source: Table 2.63 · p. 8
  • Visual comparisons show more complete object segmentation, refined Building and Low Vegetation boundaries, and improved segmentation of Tree, Building, and Impervious Surface.Purple boxes mark areas of interest in Fig. 6.Source: 1) Performance Comparison on the… · p. 8

2) Performance Comparison on the LoveDA Urban: · E. Sensitivity Analysis

On LoveDA Urban, adding SAM improved reported overall and selected class metrics, while performance varied across categories with intricate boundaries. Sensitivity experiments examined the object-consistency and boundary-loss weights and selected λo = 1.0 and λb = 0.1.

2) Performance Comparison on the LoveDA Urban:

  • FTUNetformer+SAM reached mF1 65.74% and mIoU 50.55%, gains of 0.4% and 1.43% over UNetformer, while its gains over FTUNetformer were 1.07% and 0.97%.These results support the reported overall improvement, although the passage notes that improvements were not consistent for every category.Source: 2) Performance Comparison on the… · p. 10
  • UNetformer+SAM’s building, water, and agriculture F1/IoU were 75.91%/61.17%, 80.55%/67.43%, and 43.64%/27.91%, with respective gains over UNetformer of 6.82%/8.39%, 2.89%/3.96%, and 23.1%/16.47%.The reported gains are in F1/IoU order.Source: 2) Performance Comparison on the… · p. 10
  • Road, forest, and barren results fluctuated between SAM-assisted baselines, which the paper attributes to intricate boundaries, varying sizes, and barren’s small proportion.UNetformer+SAM improved road and forest but decreased barren performance, whereas FTUNetformer+SAM showed the opposite trend for those categories.Source: 2) Performance Comparison on the… · p. 10
  • Visualization examples showed improved building identification across methods and more complete identification of water and agriculture objects.The passage says these observations align with the mF1 and mIoU improvements reported in Table II.Source: 2) Performance Comparison on the… · p. 10

E. Sensitivity Analysis

  • Sensitivity analysis varied λo, which weights object consistency, and λb, which weights boundary information, in separate experiments using UNetformer.The passage states that the hyperparameters have no strong correlation.Source: E. Sensitivity Analysis · p. 10
  • Performance was less sensitive to λo ∈[0.1, 1.0], but setting λo ≥2.0 noticeably degraded results; experiments used λo = 1.0.Smaller λo reduces the influence of object information, while larger values can overemphasize it.Source: E. Sensitivity Analysis · pp. 10–11, Figure 8 · p. 10
  • For λb ∈[0.01, 1.0], a small value made boundary information insignificant, performance peaked at λb = 0.1, and further increases degraded performance.Experiments used λb = 0.1, keeping it in a similar order of magnitude to λo = 1.0.Source: Figure 8 · p. 10, E. Sensitivity Analysis · p. 11

F. Ablation Study · G. Model Complexity Analysis · V. CONCLUSION

The paper reports that object and boundary losses improve segmentation while adding no inference-time complexity, and concludes that SAM’s raw outputs can support semantic segmentation across models and datasets.

F. Ablation Study

  • Independent use of the object and boundary losses improved overall semantic-segmentation performance with UNetformer on two datasets, supporting the value of SAM’s raw output.The ablations are reported in Tables III and IV.Source: F. Ablation Study · p. 11
  • For ISPRS Vaihingen impervious surface and building, boundary loss alone matched the combined loss, but complex-boundary segmentation requires both losses.The losses behaved differently across categories, and their combination was proposed as a versatile framework with robust generalization.Source: F. Ablation Study · p. 11

G. Model Complexity Analysis

  • The framework adds no model complexity or inference time: SGO and SGB are generated before loss calculation, and inference operates like the original model.The complexity comparison is reported in Table V, and inference behavior is illustrated in Fig. 3 (b).Source: G. Model Complexity Analysis · p. 12
  • Training takes longer because back-propagation computes two additional losses, a marginal increase the authors consider a justifiable trade-off.The framework requires no additional task-specific modules.Source: G. Model Complexity Analysis · p. 12

V. CONCLUSION

  • The auxiliary object-consistency and boundary-preservation losses improve segmentation across network structures without additional modules, with validation on two public datasets and four models.The framework is designed to exploit SAM’s raw output alongside general remote-sensing segmentation models.Source: §V · p. 12
  • Object-consistency loss is presented as the first loss function to use SGO directly despite its lack of semantic information.The authors describe the work as a preliminary exploration of SAM’s raw output and its potential for remote sensing.Source: §V · p. 12

Related papers

Loading 2312.02464v2…