PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting Segment Anything Model for Change Detection in HR Remote Sensing Images

Lei Ding, Kun Zhu, Daifeng Peng, Hao Tang, Kuiwu Yang, Lorenzo Bruzzone

arXiv:2309.01429v4cs.CV

TL;DR

Direct use of vision foundation models is unsatisfactory for remote-sensing change detection because their natural-image priors do not fit remote-sensing imagery. The paper proposes SAM-CD, which adapts FastSAM with convolutional feature adaptation and task-agnostic semantic learning. Across four benchmark datasets, SAM-CD improves accuracy over state-of-the-art methods and remains accurate with limited training samples, while still requiring fully supervised training and having relatively slow inference.

  • Problem

    Vision foundation models often perform unsatisfactorily on VHR remote-sensing change detection because their natural-image inductive bias limits adaptation to remote-sensing scenes.

  • Method

    SAM-CD adapts FastSAM features with convolutional adaptors and adds task-agnostic semantic learning to model semantic latent representations in bi-temporal images.

  • Results

    SAM-CD achieves significant accuracy improvements over state-of-the-art methods on four benchmark datasets and obtains fairly accurate results with limited training samples.

  • Takeaways & Limitations

    The approach can better discriminate semantic changes from temporal differences while providing sample-efficient learning comparable to semi-supervised change detection methods.

  • Takeaways & Limitations

    SAM-CD still requires fully supervised training, performs below semi-supervised state-of-the-art methods with very limited samples, and has relatively slow inference speed.

Abstract

from arXiv · show

Vision Foundation Models (VFMs) such as the Segment Anything Model (SAM) allow zero-shot or interactive segmentation of visual contents, thus they are quickly applied in a variety of visual scenes. However, their direct use in many Remote Sensing (RS) applications is often unsatisfactory due to the special imaging characteristics of RS images. In this work, we aim to utilize the strong visual recognition capabilities of VFMs to improve the change detection of high-resolution Remote Sensing Images (RSIs). We employ the visual encoder of FastSAM, an efficient variant of the SAM, to extract visual representations in RS scenes. To adapt FastSAM to focus on some specific ground objects in the RS scenes, we propose a convolutional adaptor to aggregate the task-oriented change information. Moreover, to utilize the semantic representations that are inherent to SAM features, we introduce a task-agnostic semantic learning branch to model the semantic latent in bi-temporal RSIs. The resulting method, SAMCD, obtains superior accuracy compared to the SOTA methods and exhibits a sample-efficient learning ability that is comparable to semi-supervised CD methods. To the best of our knowledge, this is the first work that adapts VFMs for the CD of HR RSIs.

I. INTRODUCTION

Change detection in VHR remote-sensing images remains difficult because training data are scarce and temporal imaging differences can obscure semantic changes. The paper adapts vision foundation models, particularly FastSAM, to improve accuracy while reducing dependence on large training sets.

  • CD segments content changes in multi-temporal remote-sensing images and supports applications including environmental monitoring, urban management, disaster alerting, and LCLU monitoring.
  • Large, high-quality annotated datasets are difficult to collect, especially for small or rare LCLU classes.Training-set construction requires separated acquisition dates and broad observation coverage.
  • Differences in illumination, imaging angle, season, and sensing platform make semantic changes difficult to distinguish from temporal differences.
  • The paper proposes SAM-CD, adapting FastSAM to remote-sensing scenes and using pretrained priors to embed multi-temporal LCLU representations.
  • SAM-CD introduces task-agnostic semantic learning that models semantic latent representations and uses temporal constraints without explicit semantic supervision.

A. Change Detection in RSIs

Remote-sensing change detection has evolved from hand-crafted feature analysis to CNN-, recurrent-, attention-, graph-, and transformer-based approaches. Vision foundation models offer broad recognition capabilities, but their natural-image bias limits direct use on remote-sensing imagery.

  • Pre-deep-learning change detection analyzed texture, object-based, or angular features to segment changes.
  • CNN-based methods commonly use siamese encoders to extract multi-temporal features and a shared decoder to produce change representations.
  • VHR remote-sensing change detection must distinguish semantic changes from seasonal variation while mitigating spatial misalignment and illumination differences.
  • Recent methods model temporal dependencies using recurrent networks, crosstime self-attention, and graph convolutional networks.
  • Vision transformers are used either as temporal-feature backbones or to model temporal dependencies in VHR remote-sensing change detection.
  • Vision foundation models are pretrained on web-scale or millions of annotated images to provide generalized or zero-shot visual recognition.
  • Foundation models often struggle with small and irregular remote-sensing objects and do not provide semantic classes associated with their masks.

III. PROPOSED METHOD

SAM-CD replaces conventional temporal-difference encoding with foundation-model semantic features for VHR remote-sensing change detection. It adapts FastSAM features through convolutional modules, fuses multiple scales, and adds semantic learning alongside change prediction.

  • A. Overview: FastSAM is used as a frozen encoder because it exposes low-level spatial features and requires fewer computational resources, while other foundation models can replace it.
  • A. Overview: Conventional change detection encodes paired temporal images with siamese networks and decodes their temporal differences into a binary change map.
  • A. Overview: SAM-CD uses a vision foundation model encoder to extract universal semantic features, which a decoder compares to identify categorical changes.
  • B. FastSAM Adaptor: Convolutional adaptors tailor FastSAM’s natural-image features to remote-sensing ground objects and aggregate task-oriented information.
  • B. FastSAM Adaptor: The adaptor processes FastSAM features at 1/32, 1/16, 1/8, and 1/4 spatial scales using convolution, batch normalization, and ReLU.
  • B. FastSAM Adaptor: Channel reduction removes redundancy because remote-sensing imagery contains fewer object categories than natural images.
  • B. FastSAM Adaptor: An UNet-like decoder fuses adapted multi-scale features, emphasizing low-level information important for segmenting changed areas.

C. Task-agnostic Semantic Learning

SAM-CD adds a task-agnostic semantic learning branch that aligns bi-temporal latent representations in unchanged areas and embeds semantic focus into change features.

  • C. Task-agnostic Semantic Learning: Jointly learning semantics and changes lets SAM-CD model semantic latent representations alongside temporal differences.The branch follows a multi-task perspective and conducts semantic and change learning simultaneously.
  • C. Task-agnostic Semantic Learning: The method transforms adapted FastSAM features into candidate latent representations and uses temporal constraints rather than categorical labels for supervision.This avoids requiring semantic labels for each acquisition date in binary change detection.
  • C. Task-agnostic Semantic Learning: A temperature-controlled softmax normalizes each candidate latent, producing bi-temporal representations for semantic comparison.The temperature parameter controls the output-feature probability distribution.
  • C. Task-agnostic Semantic Learning: The temporal constraint loss measures similarity between normalized bi-temporal semantics in unchanged areas, excluding changed areas using the ground-truth change label.Unchanged areas are annotated as 1 in c, while changed areas are excluded from loss calculations.
  • C. Task-agnostic Semantic Learning: Attention operations embed the learned semantic focus into change features before mapping them to a change map, making the detection results semantic-aware.The attention operation uses channel-wise concatenation, sigmoid normalization, and convolutional modules.

D. Implementation Details

The implementation uses FastSAM’s visual encoder within SAM-CD, with convolutional modules for semantic and change embeddings and standard PyTorch training settings.

  • D. Implementation Details: FastSAM is built on YOLO-v8, and SAM-CD uses its default 68 Mb version for higher accuracy.SAM-CD uses only FastSAM’s visual encoder and discards the prompt decoders because change detection differs from typical segmentation.
  • D. Implementation Details: SAM-CD’s semantic embedding block is a single 1×1 convolutional layer with the semantic channel count k empirically set to 8.The setting reflects the typically small number of interesting LCLU classes in change-detection applications.
  • D. Implementation Details: The change embedding module uses six residual convolutional-block layers.This design follows the implementation practice described for the change-detection branch.
  • D. Implementation Details: Training runs for 50 epochs in PyTorch with stochastic gradient descent, a decaying learning rate, and random flipping and cropping augmentations.Inference additionally uses test-time augmentation with eight operations.
  • D. Implementation Details: Implementation details and released code are provided through the authors’ SAM-CD repository.The passage directs readers to the project’s released code for further details.

IV. EXPERIMENTAL RESULTS

The experimental section evaluates SAM-CD through ablations, label-efficient learning tests, and comparisons with state-of-the-art change-detection methods across four benchmark datasets.

  • IV. EXPERIMENTAL RESULTS: Experiments assess the SAM-CD architecture through ablation studies, label-efficient learning tests, and comparisons with state-of-the-art methods.These experiments are organized to test architecture effectiveness, label efficiency, and benchmark performance.
  • IV. EXPERIMENTAL RESULTS: The evaluation uses four benchmark datasets: LEVIR-CD, WHU-CD, CLCD, and S2Looking.The datasets cover high-resolution remote-sensing change-detection settings with varied geographic and imaging conditions.
  • IV. EXPERIMENTAL RESULTS: LEVIR-CD contains 637 image pairs from six Texas cities, with 31,333 annotated changed objects at 0.5m spatial resolution.Its changes are mostly associated with construction growth, and the pairs are split into 445 training, 64 validation, and 128 test samples.
  • IV. EXPERIMENTAL RESULTS: WHU-CD covers 20.5 km2 of Christchurch and captures building changes associated with post-earthquake urban growth at 0.2m resolution.The observed building count increased from 12,796 to 16,077 between the acquisition periods.
  • IV. EXPERIMENTAL RESULTS: CLCD comprises 600 Gaofen-2 image pairs from Guangdong collected in 2017 and 2019, with multiple LCLU change types annotated.The training, validation, and test splits contain 320, 120, and 120 image pairs, respectively.
  • IV. EXPERIMENTAL RESULTS: S2Looking contains 5,000 image pairs from rural areas worldwide and includes many pairs captured at different imaging angles.Its images have 0.5 to 0.8m spatial resolution, and differing angles cause spatial misalignment.

2) Evaluation Metrics:

The paper evaluates change-detection accuracy using precision, recall, F1, IoU, and overall accuracy under settings aligned with prior work.

  • 2) Evaluation Metrics:: Change-detection accuracy is assessed with precision, recall, F1, intersection over union, and overall accuracy.These metrics follow the evaluation practice used in the cited literature.
  • 2) Evaluation Metrics:: Precision is calculated as TP divided by TP plus FP, while recall is calculated as TP divided by TP plus FN.TP, FP, and FN denote true positives, false positives, and false negatives.
  • 2) Evaluation Metrics:: Overall accuracy is calculated as TP plus TN divided by TP plus FP plus TN plus FN.TN denotes true negatives.
  • 2) Evaluation Metrics:: Quantitative results use the same evaluation settings and calculation implementation as the cited prior study.This alignment applies to the evaluation settings reported in this section.

B. Ablation Study

The ablation study evaluates SAM-CD components, visual encoders, semantic-temperature selection, and FastSAM feature-layer choices on Levir-CD. Results show gains from FastSAM and progressive improvements from the proposed techniques, with FastSAM-x and all four feature layers selected as defaults.

  • Component ablation: 0.23%, 1.83% and 1.11%: adding plain FastSAM to the baseline increases OA, mIoU and F1, respectively.The baseline uses two siamese ResNet34 encoders and a decoder, with parameter size close to FastSAM.
  • Component ablation: SAM-CD techniques progressively improve segmentation, especially for dark buildings and constructions that resemble their surroundings.
  • Visual encoder comparison: More than 100 times higher FLOPs: SAM encoders are substantially more expensive than FastSAM encoders in SAM-CD.The comparison uses different SAM and FastSAM visual encoders on Levir-CD; SAM requires a fixed 1024×1024 spatial input.
  • Visual encoder comparison: FastSAM encoders achieve better accuracy than the tested SAM-b and SAM-l counterparts, and FastSAM-x gives the highest accuracy.The authors attribute FastSAM’s advantage to access to low-level spatial features that are crucial for change detection, and adopt FastSAM-x by default.
  • Semantic-temperature selection: T = 3 yields the optimal accuracy, as larger T values soften latent distributions and better present semantics.T controls the diversity of semantic representations in the softmax function.
  • Feature-layer selection: All four FastSAM feature layers produce the highest accuracy; l1 and l4 also perform relatively well, while omitting l1 significantly decreases accuracy.The layers correspond to 1/4, 1/8, 1/16 and 1/32 of the input spatial size; the findings indicate both semantic and spatial features matter.
  • Semantic latent visualization: The learned semantic latent attends to buildings, ponds, roads and low vegetation without corresponding human semantic annotations.The latent is visualized with warm colors for high values and cold colors for low values.

C. Comparative experiments with Fully-supervised CD Methods

SAM-CD outperforms fully supervised state-of-the-art methods across four benchmark datasets, with particular gains on complex land-cover and land-use changes. Its qualitative results also show better detection of non-salient changes and fewer false alarms.

  • CGNet outperforms most literature methods across four datasets, but these methods have limited capacity for land-cover and land-use changes and spatially shifted scenes.
  • SAM-CD achieves robust accuracy improvements over state-of-the-art methods across all four benchmark datasets.
  • 1.31% mF1 and 1.7% mIoU gains over state-of-the-art accuracy are reported on CLCD.
  • SAM-CD obtains the best recall on Levir-CD, WHU-CD, and CLCD, indicating sensitivity to non-salient semantic changes.
  • Compared with CGNet and EATDer, SAM-CD better detects non-salient changes and produces fewer false alarms in complex scenes.
  • The FastSAM encoder contributes 68m parameters, while SAM-CD's proposed network modules contribute 2.49m parameters.

D. Comparative experiments with Semi-supervised CD Methods

SAM-CD remains competitive with semi-supervised change-detection methods despite using plain supervised training. It performs especially well with moderate or full training proportions, although performance drops at only 5% supervision.

  • SAM-CD is trained with plain supervised learning, without contrastive, adversarial, or spectral-wise augmentation strategies.
  • SAM-CD outperforms most semi-supervised change-detection methods, except the recent UniMatch.
  • On Levir-CD, SAM-CD achieves the highest accuracy with 40% and 100% of training data and the second-best accuracy with 5%, 10%, and 20%.
  • UniMatch achieves the highest accuracy at 5%, 10%, and 20% training proportions through contrastive learning and feature-level augmentation.
  • SAM-CD's accuracy drops when only 5% of training samples are used, suggesting that adapting FastSAM and training the change head still require supervised data.

V. CONCLUSION

SAM-CD uses FastSAM features and temporal constraints to model semantic information for change detection in very-high-resolution remote-sensing images. Across four benchmark datasets, it improves accuracy and remains sample-efficient, but still requires fully supervised training and has relatively slow inference.

  • V. CONCLUSION: SAM-CD models semantic latent representations to distinguish semantic changes from seasonal and imaging-condition differences.
  • V. CONCLUSION: FastSAM extracts visual features from ground objects, while temporal constraints supervise task-agnostic semantic representations.
  • V. CONCLUSION: SAM-CD achieves significant accuracy improvements over state-of-the-art methods on four benchmark datasets.
  • V. CONCLUSION: SAM-CD exhibits sample-efficient learning and obtains fairly accurate results with a limited number of training samples.
  • V. CONCLUSION: SAM-CD still requires fully supervised training, and with very limited samples its accuracy is below state-of-the-art semi-supervised methods.
  • V. CONCLUSION: SAM-CD has relatively slow inference compared with literature methods, motivating parameter reduction and removal of redundant priors.
Loading 2309.01429v4…