PaperlayerView exact revision on arXiv

Source-linked AI summary

A New Learning Paradigm for Foundation Model-based Remote Sensing Change Detection

Kaiyu Li, Xiangyong Cao, Deyu Meng

arXiv:2312.01163v2cs.CV

TL;DR

Remote-sensing change detection faces limited labeled data and constrained gains from increasingly large task-specific models. BAN adapts frozen foundation-model features through bridging modules into a flexible bi-temporal CD branch, improving existing methods with few additional learnable parameters. The reported experiments support foundation-model adaptation as a promising direction for remote-sensing change detection.

  • Problem

    Limited labeled data constrains further improvement of deep-learning change-detection models, while transferring foundation-model knowledge presents domain and bi-temporal input-structure challenges.

  • Method

    BAN combines a frozen foundation model, bridging modules that select and align features, and a model-agnostic Bi-TAB that can be an existing CD model or hand-crafted blocks.

  • Results

    BAN improves existing CD methods with only a few additional learnable parameters, including up to 4.08% IoU improvement on corresponding customized models.

  • Takeaways & Limitations

    BAN demonstrates the feasibility of adapting foundation models for remote-sensing change detection and provides a basis for subsequent investigation.

Abstract

from arXiv · show

Change detection (CD) is a critical task to observe and analyze dynamic processes of land cover. Although numerous deep learning-based CD models have performed excellently, their further performance improvements are constrained by the limited knowledge extracted from the given labelled data. On the other hand, the foundation models that emerged recently contain a huge amount of knowledge by scaling up across data modalities and proxy tasks. In this paper, we propose a Bi-Temporal Adapter Network (BAN), which is a universal foundation model-based CD adaptation framework aiming to extract the knowledge of foundation models for CD. The proposed BAN contains three parts, i.e. frozen foundation model (e.g., CLIP), bi-temporal adapter branch (Bi-TAB), and bridging modules between them. Specifically, BAN extracts general features through a frozen foundation model, which are then selected, aligned, and injected into Bi-TAB via the bridging modules. Bi-TAB is designed as a model-agnostic concept to extract task/domain-specific features, which can be either an existing arbitrary CD model or some hand-crafted stacked blocks. Beyond current customized models, BAN is the first extensive attempt to adapt the foundation model to the CD task. Experimental results show the effectiveness of our BAN in improving the performance of existing CD methods (e.g., up to 4.08\% IoU improvement) with only a few additional learnable parameters. More importantly, these successful practices show us the potential of foundation models for remote sensing CD. The code is available at \url{https://github.com/likyoo/BAN} and will be supported in our Open-CD.

I. INTRODUCTION

Remote-sensing change detection is important but improvement has slowed under limited labeled data. BAN adapts frozen foundation-model knowledge to existing CD models through a bridging module and a flexible Bi-TAB.

  • Change detection identifies semantic or specific changes between bi-temporal remote-sensing images and supports applications including urban expansion, disaster assessment, and cropland protection.
  • 90% F1-score was reached on LEVIR-CD by 2020, but subsequent improvement was only ∼1% to 2% as models grew to hundreds of millions of parameters.The paper identifies this as a performance bottleneck under limited data.
  • Foundation models accumulate broad knowledge through large-scale training and can transfer it across domains while reducing task-specific training-data requirements.
  • BAN combines a frozen foundation model, a bi-temporal adapter branch, and bridging modules to adapt foundation-model knowledge for change detection.The framework feeds bi-temporal images through a frozen model and combines its features with customized CD components.
  • The bridging module selects and aligns general features before injecting them into remote-sensing CD features using cross-domain dot-product attention.
  • 4.08% average improvement was reported for corresponding customized CD models, with only a few additional learnable parameters.Bi-TAB can use an existing CD model or hand-crafted stacked blocks, giving BAN a plug-and-play structure.

II. RELATED WORK

Related work spans CNN- and Transformer-based change detection, foundation models, and parameter-efficient adaptation. BAN extends these directions by using arbitrary existing CD models as its task-specific branch within a universal foundation-model framework.

  • CNN-based models improved change detection through automatic feature extraction, while ViT-based methods introduced patch-sequence modeling and global dependency capture.
  • Limited data can prevent pure Transformer CD models from reaching their full potential without inductive bias, motivating token-based designs such as BiT.
  • BAN treats existing customized CD models as interchangeable Bi-TAB branches and uses the foundation model to improve them.
  • Vision foundation models accumulate knowledge through scaling across data modalities and proxy tasks; examples include ViT, CLIP, and RemoteCLIP.RemoteCLIP is described as an early remote-sensing visual-language foundation model but remains limited by its data volume.
  • Parameter-efficient transfer learning introduces limited trainable parameters or selects subsets of pretrained parameters instead of fully fine-tuning the model.
  • Unlike the customized Fast-SAM-based SAM-CD model, BAN is presented as a universal framework for adapting foundation models to change detection.

A. Review of ViT

ViT converts images into patch tokens and processes them with Transformer blocks. In BAN, a shared frozen foundation model handles both temporal images, while resolution differences motivate an alignment strategy.

  • A. Review of ViT: ViT divides an image into non-overlapping P × P patches, encodes them into D-dimensional vectors, and adds a class token and positional encoding.The patch embedding contains N = HW/P^2 tokens.
  • A. Review of ViT: A ViT basic block alternates multi-head self-attention and feed-forward layers with layer normalization and residual connections.
  • A. Review of ViT: BAN processes the two bi-temporal images simultaneously with a shared-parameter, siamese frozen foundation model.
  • A. Review of ViT: High-resolution remote-sensing inputs create a mismatch because foundation models are commonly trained at 224×224 or 336×336 resolution.BAN resizes inputs for the foundation model while retaining 256 × 256 or 512 × 512 resolution for Bi-TAB.
  • A. Review of ViT: BAN mainly uses CLIP’s ViT-L/14 image encoder as its foundation-model component.

B. Bi-TAB

Bi-TAB is a model-agnostic branch that extracts task-specific change-detection features while receiving aligned, selected general features from a foundation model through bridging modules.

  • Bi-TAB concept: Bi-TAB is not a fixed network; it can use existing CNN- or Transformer-based change-detection models or hand-crafted stacked blocks.The authors recommend existing customized models so BAN can benefit from evolving change-detection architectures.
  • Bi-TAB concept: Siamese Bi-TAB backbones receive an additional general-feature bank from the foundation model for the corresponding temporal phases.This parallel structure allows foundation-model information to be injected into the bi-temporal processing streams.
  • Bridging motivation: The bridging module addresses feature selection and scale alignment before injecting foundation-model information into Bi-TAB.Foundation features may contain unnecessary knowledge and are downsampled by at least 14×, creating a resolution mismatch.
  • Bridging operation: The module normalizes and linearly projects foundation features, computes cross-feature attention, and resamples the selected output to Bi-TAB resolution.The attention mechanism uses an affinity matrix and softmax weights to filter useful general characteristics.

D. Loss Function

BAN can use the underlying Bi-TAB loss directly, with pixel-wise cross-entropy applied to change-detection predictions and, for auxiliary SCD tasks, semantic segmentation.

  • Loss Function: BAN directly follows the plain Bi-TAB loss function.The paper applies this formulation to BAN without introducing a separate loss design.
  • Loss Function: Pixel-wise cross-entropy is used for BiT and ChangeFormer change-detection tasks.The loss is defined over pixel labels and predictions at positions (h, w).
  • Loss Function: For SCD with auxiliary tasks, cross-entropy loss is also applied to the semantic-segmentation sub-task.This extends the loss beyond the primary change-detection prediction.

IV. EXPERIMENTS

BAN is evaluated extensively on five RGB-image change-detection datasets without extra data. The experiments cover dataset configurations and BAN’s forward propagation from bi-temporal inputs to a predicted change mask.

  • IV. EXPERIMENTS: Five RGB-image CD datasets are used to evaluate BAN and all compared methods without extra data.The datasets are used for both training and evaluation under the stated experimental protocol.
  • IV. EXPERIMENTS: LEVIR-CD provides 637 bi-temporal image pairs with 31,333 annotated change instances and fixed training, validation, and testing splits.Its image pairs are 1024×1024 at 0.5 m/pixel, divided into 445, 64, and 128 pairs.
  • IV. EXPERIMENTS: S2Looking contains 5,000 rural-area image pairs with 65,920 annotated change instances and maps for new and demolished buildings.The dataset uses side-looking satellite images captured at varying angles from nadir.
  • IV. EXPERIMENTS: WHU-CD is formed by cropping two large images from 2012 and 2016 into 256×256 patches, split into 5947/743/744 training, validation, and testing patches.The images depict the same area at two time points.
  • IV. EXPERIMENTS: BAN’s forward propagation takes a pair of bi-temporal images as input and produces a predicted change mask using foundation, change-detection, and bridging modules.The algorithm identifies the foundation model, Bi-TAB change-detection model, and bridging modules as the main components.
  • IV. EXPERIMENTS: BANDON and Landsat-SCD broaden evaluation with off-nadir aerial imagery, domain-split testing, and semantic change categories.BANDON includes in-domain and out-domain test subsets, while Landsat-SCD includes no-change and ten semantic-change types.

B. Implementation details

The implementation uses Open-CD with cross-entropy, AdamW, poly learning-rate decay, data augmentation, and CLIP ViT-L/14 by default. Evaluation uses task-specific metrics for binary and semantic change detection.

  • B. Implementation details: BAN is implemented in the PyTorch-based Open-CD toolkit with cross-entropy loss and AdamW optimization.The base learning rate is 0.0001, with a 10× rate in Bi-TAB’s prediction head.
  • B. Implementation details: Training uses a poly learning-rate schedule, random cropping where specified, flipping, and photometric distortion.The crop size is 512×512 for LEVIR-CD, S2Looking, and BANDON, with no extra crop for WHU-CD and Landsat-SCD.
  • B. Implementation details: Experiments run for dataset-specific iteration or epoch budgets on NVIDIA GeForce RTX 4090 and RTX A6000 GPUs with batch size 8.The stated budgets are 80k and 40k iterations for LEVIR-CD, S2Looking, and BANDON, and 100 and 50 epochs for WHU-CD and Landsat-SCD.
  • B. Implementation details: CLIP’s ViT-L/14 is the default foundation model, and ARIS is enabled by default unless otherwise specified.This configuration defines the default foundation-model setup for the reported experiments.
  • B. Implementation details: BCD evaluation uses IoU_c, F_c^1, overall accuracy, precision_c, and recall_c on the change category.TP, TN, FP, and FN are calculated for the change category to address class imbalance.
  • B. Implementation details: SCD evaluation uses mIoU and separated kappa, then combines them into a weighted score with weights 0.7 and 0.3.IoU_u measures the non-change category, while the kappa calculation covers bi-temporal semantic segmentation.

D. Comparison and Analysis

BAN improves binary change detection across datasets and customized models, with gains also visible in challenging visual cases. Its benefits are achieved with limited additional learnable parameters.

  • Model efficiency: 3.23% IoU_c improvement is obtained by a 1.44M-parameter stacked-blocks model under BAN.This result supports gains from general knowledge even for a lightweight hand-crafted CD model.
  • Cross-dataset results: 1.83% and 2.36% improvements in F_c are reported for BiT and CF-b1 on the challenging S2Looking dataset.S2Looking contains off-nadir aerial images, while BANDON-BCD focuses on urban scenes.
  • BANDON-BCD: 8.65% IoU_c improvement raises BiT from 44.43% to 53.08% on BANDON-BCD.BAN-CF reaches 54.02% IoU_c from 48.84%, the best BCD performance without auxiliary annotations.
  • Visual analysis: BAN better detects buildings resembling the background and special-appearance changes in LEVIR-CD visualizations.Models without BAN often confuse cargo-box changes with building changes because of similar appearance.

2) Semantic Change Detection:

BAN also improves semantic change detection and its auxiliary semantic segmentation task. The gains appear across Landsat and BANDON experiments, including challenging ambiguous targets.

  • Landsat: 1.54% and 2.14% weighted Score improvements are achieved by CF-SCD-b0 and CF-SCD-b1 on Landsat.Their Sek improvements are 1.87% and 2.87%, respectively.
  • Auxiliary semantic segmentation: BAN more accurately detects ambiguous targets in BANDON-SCD semantic segmentation visualizations.The cited examples involve blurring, noise, and entanglement of image content with the background.
  • BANDON-SCD: 4.68% IoU_c improvement raises BAN-CF-SCD-b0 to 56.95% on BANDON-SCD.Compared with BAN-CF-B0 without SCD annotation, the improvement is 2.93%.
  • BANDON-SCD: 59.70% IoU is achieved by BAN-CF-SCD with MiT-b2, only 0.57% below MTGCD-Net.MTGCD-Net uses additional auxiliary annotations for roof-to-footprint offsets and bi-temporal matching flows.

3) Cross-domain Change Detection:

BAN mitigates the performance damage caused by cross-domain inference and improves customized CD models on out-domain data.

  • 11.96% improvement is achieved by BAN-BiT on the BANDON out-domain testing set over plain BiT.Plain BiT achieves 35.24% IoU c on out-domain data, 9.19% below its in-domain result.
  • 6.00% improvement is achieved by BAN-CF over CF on the BANDON out-domain testing set.The improvement is larger than on the in-domain testing set, suggesting stronger generalization support.

4) Semi-supervised Change Detection:

BAN performs strongly when trained with partial labels and can also operate with different foundation-model input resolutions, though its semi-supervised comparison does not use unlabeled data.

  • Semi-supervised Change Detection: BAN is superior to other fully supervised and semi-supervised methods at 5% and 10% labeled data, except SemiCD.BAN was trained only with partial labels and did not use unlabeled data.
  • Semi-supervised Change Detection: 81.05% IoU c is achieved with 20% of labeled data, outperforming SemiCD by 6.25%.
  • Semi-supervised Change Detection: 85.26% IoU c is achieved with 40% of labeled data, outperforming SemiCD by 8.06%.
  • Foundation-model input resolution: 2.86 images of 1024 × 1024 per second are inferred at 224 × 224 foundation-model input resolution with batch size 1 on one A6000 GPU.

2) Different Foundation Models:

The experiments examine how foundation-model choices and Bi-TAB capacity affect BAN, while concluding that the framework can benefit from stronger foundation models and extensible task-specific modules.

  • Different foundation models: Different foundation models provide different types of general knowledge because their training data and training strategies differ.The compared sources include ImageNet-21k, LAION-2B CLIP, and remote-sensing-trained RemoteCLIP models.
  • Scalable Bi-TABs: 2.46% IoU c improvement is achieved by BAN-CF-b0 over CF, increasing to 2.88% with BAN-CF-b1 and 3.21% with BAN-CF-b2.
  • Conclusion: BAN is highly extensible because both its foundation model and Bi-TAB are model-agnostic, allowing it to benefit from stronger foundation models.
  • Conclusion: BAN can be extended beyond RGB CD to multispectral and hyperspectral images.
Loading 2312.01163v2…