PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Spatio-Temporal Modeling for the Semantic Change Detection in Remote Sensing Images

Lei Ding, Jing Zhang, Kai Zhang, Haitao Guo, Bing Liu, Lorenzo Bruzzone

arXiv:2212.05245v4cs.CV

TL;DR

SCD must identify changed areas and their before-and-after semantic categories, but semantic learning is difficult with limited change samples and incomplete temporal modeling. The paper proposes SCanNet, combining SCanFormer with a semantic learning scheme that exploits spatio-temporal constraints. SCanNet improves critical-change detection and bi-temporal semantic consistency, achieving state-of-the-art accuracy on two benchmark datasets.

  • Problem

    SCD requires semantic information from changed areas even though semantic labels are limited and prior spatio-temporal modeling remains partial, one-sided, or shallow.

  • Method

    SCanNet combines triple CNN encoder-decoders, SCanFormer for joint semantic-change dependencies, and task-specific semantic learning based on bi-temporal consistency.

  • Results

    SCanNet achieves accuracy improvements over state-of-the-art methods, including a lead of over 1% in Fscd, while improving critical-change discrimination and temporal semantic information.

  • Takeaways & Limitations

    The proposed architecture and learning scheme improve change detection, semantic extraction, and consistency between bi-temporal results.

  • Takeaways & Limitations

    Rare semantic changes with very few samples remain a challenge left for future studies.

Abstract

from arXiv · show

Semantic Change Detection (SCD) refers to the task of simultaneously extracting the changed areas and the semantic categories (before and after the changes) in Remote Sensing Images (RSIs). This is more meaningful than Binary Change Detection (BCD) since it enables detailed change analysis in the observed areas. Previous works established triple-branch Convolutional Neural Network (CNN) architectures as the paradigm for SCD. However, it remains challenging to exploit semantic information with a limited amount of change samples. In this work, we investigate to jointly consider the spatio-temporal dependencies to improve the accuracy of SCD. First, we propose a Semantic Change Transformer (SCanFormer) to explicitly model the 'from-to' semantic transitions between the bi-temporal RSIs. Then, we introduce a semantic learning scheme to leverage the spatio-temporal constraints, which are coherent to the SCD task, to guide the learning of semantic changes. The resulting network (SCanNet) significantly outperforms the baseline method in terms of both detection of critical semantic changes and semantic consistency in the obtained bi-temporal results. It achieves the SOTA accuracy on two benchmark datasets for the SCD.

I. INTRODUCTION

Semantic Change Detection identifies both changed areas and their before-and-after semantic transitions in remote sensing imagery. The paper addresses missed or false alarms and inconsistent bi-temporal predictions by jointly modeling spatio-temporal dependencies with SCanNet and task-specific semantic learning.

  • Motivation: SCD extracts changed areas and their ’from-to’ semantic transitions from multi-temporal remote sensing data.This supports applications including land-cover monitoring, resource management, and disaster alarming.
  • Motivation: Unlike BCD, SCD predicts semantic maps for both observation times, enabling detailed analysis of land-cover transitions.
  • Related challenges: Triple-branch CNNs became a widely accepted SCD paradigm by separately modeling temporal semantics and bi-temporal change information.
  • Challenges: SCD remains difficult because non-salient changes and appearance variations cause missed or false alarms, while weak temporal correlation produces contradictory results.An example contradiction is a region marked as changed while receiving identical semantic classes at both times.
  • Contributions: SCanNet combines triple encoder-decoder CNNs with SCanFormer to learn semantic transitions and deep temporal correlations.Its semantic learning scheme uses bi-temporal consistency as extra supervision and formulates constraints for change and no-change areas.

A. Binary Change Detection

SCD research evolved from comparison and probabilistic approaches toward end-to-end CNN and transformer-based models. The remaining challenge is jointly modeling spatial and temporal semantic-change dependencies rather than only partial or one-sided correlations.

  • Semantic Change Detection: BCD methods detect where changes occur, whereas SCD additionally identifies what semantic transition occurred between the observations.SCD therefore supports land-cover and land-use transition analysis beyond binary change maps.
  • Semantic Change Detection: Early SCD methods used post-classification comparison, compound classification, and neural networks to classify multi-temporal land-cover transitions.
  • CNN-based SCD: A triple-branch CNN architecture separately exploits temporal semantics and binary change information, while SSCDl uses disentangled semantic segmentation and change detection with late fusion.
  • Open challenge: Temporal attention, channel attention, and learnable transforms improve feature representations but model bi-temporal dependencies only partially, one-sidedly, or shallowly.
  • Proposed direction: SCanNet addresses this gap with triple CNNs for feature extraction, a Transformer for joint spatio-temporal dependencies, and semantic learning for task-specific change-semantic correlations.

A. CNN Architecture for SCD

The TED framework processes two input images through separate temporal encoder-decoder branches and a change branch to generate semantic and change representations. It extends the prior SSCDl design while aiming to preserve more spatial detail.

  • Task formulation: SCD maps a bi-temporal image pair to two semantic change maps that encode changed areas and semantic categories at both times.
  • TED architecture: TED uses temporal branches for semantic information and a separate change branch that detects changes from bi-temporal features.The temporal and change representations are later forwarded to corresponding heads.
  • TED architecture: The encoder networks embed each input image into multi-scale features, while necks enlarge and concatenate features before head processing.
  • TED architecture: The change branch receives high-level semantic features from both temporal branches and outputs a change feature that is spatially aligned with temporal features.
  • Comparison: Compared with SSCDl, TED is reported to retain spatial details more effectively.

B. SCanFormer: ’Semantic-Change’ dependency modeling with Transformer

SCanFormer extends TED with a Transformer head that jointly models semantic-change dependencies across the spatio-temporal domain. It uses tokenized temporal and change features with cross-shaped-window attention for more efficient long-range context modeling.

  • Motivation: TED separately embeds temporal and change features, but this disentanglement does not model their correlation.Bi-temporal semantic consistency can help distinguish change from no-change, while transition patterns can improve semantic recognition.
  • Architecture: SCanFormer is inserted into TED as the heads, producing the hybrid CNN-Transformer network SCanNet.CNNs extract spatially preserved features, while SCanFormer models correlations in embedded semantic space.
  • Token modeling: SCanFormer concatenates temporal and change features and flattens them into semantic tokens before applying stacked attention blocks.Each block contains self-attention and MLP units with residual organization and normalization.
  • Attention design: Cross-shaped-window self-attention replaces full self-attention to model long-range context more efficiently for high-resolution remote sensing images.Full self-attention has computation complexity quadratic to image size, whereas CSWin partitions features into horizontal and vertical stripes.
  • Attention design: The attention operation uses projected query, key, and value tokens plus a learnable relative-position parameter matrix.

C. Semantic Learning with Temporal Consistency Constraints

The semantic learning scheme uses temporal consistency and pseudo-labeling to exploit semantic information in both unchanged and changed areas. Adding the consistency objective jointly considers temporal semantic information and improves discrimination of critical areas.

  • Motivation: Limited semantic labels in changed regions motivate using bi-temporal consistency as prior information to guide semantic learning.Changed areas have semantic labels, whereas unchanged areas are usually annotated only as no-change.
  • Pseudo-labeling: Pseudo-labeling infers semantic labels for unchanged areas when bi-temporal semantic probabilities are sufficiently similar.Cosine similarity measures the semantic similarity, and threshold T controls the pseudo-label generation.
  • Pseudo-labeling: The pseudo semantic objective is calculated for unchanged areas while excluding changed areas from its loss calculation.The objective is illustrated in Fig. 4(a), which uses temporal consistency to exploit semantic information in no-change areas.
  • Temporal consistency: Temporal consistency requires semantic agreement in no-change areas and semantic differences in changed areas.The consistency objective is calculated using the ground-truth change label and is evaluated in both area types.
  • Objective: The overall loss incorporates semantic, pseudo-semantic, and semantic-consistency objectives to jointly train the network.By adding Lsc, the temporal semantic information from the two images is jointly considered, improving discrimination of critical areas.

D. Implementation Details

SCanNet uses ResNet-based Siamese temporal encoders, a residual change-feature block, decoder necks, and a lightweight SCanFormer configuration. Training uses stochastic gradient descent with scheduled learning-rate decay and geometric augmentation.

  • Network implementation: The CNN encoders use ResNet blocks following established segmentation practice.The implementation describes these encoders as part of the TED component.
  • Network implementation: Temporal encoders share weights because the input images come from the same domain, helping avoid over-fitting and align extracted features.The change-feature embedding uses a ResNet-like block with six residual layers, while decoder necks use decoder blocks.
  • SCanFormer: The SCanFormer uses context range s=2, two self-attention layers, and feature-dimension parameter K=2.These settings are selected considering the input feature size and feature dimensions while limiting computation.
  • Training: Training runs for 50 epochs with batch size 8, initial learning rate 0.1, and Nesterov stochastic gradient descent.The learning rate follows 0.1∗(1−iterations/total iterations)^1.5, with random flipping and rotation for augmentation.
  • Implementation: The methods are implemented with PyTorch, and the authors provide released code for further implementation details.The code repository is identified in the implementation section.

IV. DATASET DESCRIPTION AND EXPERIMENTAL SETTINGS

The experimental section defines the datasets, evaluation metrics, and experimental settings used to assess SCanNet.

  • The experiments cover dataset descriptions, evaluation metrics, and experimental settings.

A. Dataset

Experiments use SECOND and Landsat-SCD, which span high-resolution urban scenes and mid-resolution desert-margin imagery. The evaluation combines overall accuracy with metrics designed to assess change and semantic segmentation performance.

  • Datasets: SECOND and Landsat-SCD provide benchmark settings across high-resolution city regions and a mid-resolution desert-margin area.Their differing spatial resolutions and observed scenes support evaluation in different scenarios.
  • SECOND: SECOND contains 4662 spatially matched RSI pairs from several Chinese cities, with 512×512 images and 0.5–3m ground sampling distance.The dataset includes manual semantic change maps associated with each image.
  • Landsat-SCD: Landsat-SCD uses imagery collected from 1990 to 2020 at 30m ground sampling distance, with four land-cover classes and an explicit no-change class.The observed region is Tumushuke, Xinjiang, at the margin of the Taklimakan Desert.
  • Landsat-SCD: Landsat-SCD contains 8468 image pairs, including 2425 original pairs split into training, validation, and test sets in a 3:1:1 ratio.The original-data split contains 1455 training, 485 validation, and 485 test pairs.
  • Evaluation metrics: OA measures correctly classified pixels, while mIoU and SeK evaluate change discrimination and semantic segmentation, respectively.SeK excludes dominant true-positive no-change pixels, and Fscd evaluates segmentation accuracy in changed areas.

V. EXPERIMENTAL RESULTS

The experiments assess the proposed components through ablations, qualitative comparisons, and quantitative evaluation. The semantic learning scheme and SCanFormer improve change detection and semantic discrimination.

  • Experimental design: The experiments evaluate the TED framework, semantic learning scheme, and SCanNet through ablation and qualitative studies.The study also compares the proposed network with state-of-the-art methods on benchmark datasets.
  • Quantitative Results: Around 0.7% improvements in mIoU, SeK, and Fscd follow from applying the semantic learning scheme to TED.Compared with SSCDl, TED itself improves mIoU and Fscd by around 0.3%.
  • Effects of the Semantic Learning Scheme: Pseudo labels are generally correct and supervise semantic information in unchanged regions where bi-temporal prediction confidence is high.The labels cover pixels without semantic labels and include multiple semantic categories.
  • Effects of the Semantic Learning Scheme: The semantic learning scheme captures non-salient changes and improves semantic-class discrimination in qualitative comparisons.Examples include detecting a playground and reducing ground-versus-low-vegetation segmentation errors.
  • Conclusion: The ablation study concludes that TED and semantic learning improve change and semantic extraction, while SCanFormer improves temporal semantic information.These conclusions combine the framework, learning-scheme, and Transformer ablations.

B. Comparative Experiments

Comparative experiments show that the proposed methods perform strongly against accessible SCD and related methods on two benchmark datasets. SCanNet improves semantic-change discrimination, consistency, and fine-grained recognition, including with limited change samples.

  • Experimental setup: The comparison covers CD and SCD architectures, hybrid CNN-RNN methods, and other recent literature methods.Evaluation uses methods with accessible implementations and two benchmark datasets.
  • Quantitative Results: Plain TED outperforms most literature methods, with clearer advantages on Landsat-SCD attributed to its spatial-preserving designs.TED reuses semantic information from temporal-branch features and is more sensitive to semantic changes than HRSCD-str.4.
  • Quantitative Results: Around 0.7% higher SeK and around 1% higher Fscd than the second-best results demonstrate SCanNet’s advantage across metrics.The reported advantage is associated with deep spatio-temporal dependency modeling.
  • Qualitative Results: SCanNet improves recognition of non-salient changes and reduces contradictions between the two bi-temporal semantic results.Examples include playground emergence, small-building removal, and low-vegetation-versus-water discrimination.
  • Qualitative Results: TED-based methods capture fine-grained land-use changes on low-GSD Landsat-SCD imagery, while SCanNet better discriminates small-object categories.Examples include river drying and emergence of small farmlands.
  • Change Analysis: False changes account for 0.2% in SECOND and 0.26% in Landsat-SCD for SCDNet, excluding building→building changes affected by wrong annotations.The reported confusion-matrix analysis identifies ground→ground and farmland→farmland as false-change examples.

VI. CONCLUSIONS

The proposed SCanNet jointly models spatio-temporal dependencies through SCanFormer and task-specific semantic constraints. It improves SCD accuracy while leaving rare semantic changes with very few samples for future study.

  • SCanFormer jointly models spatio-temporal dependencies, while semantic consistency and pseudo-learning objectives encode prior SCD constraints.The hybrid architecture combines CNN spatial context with transformer-based temporal modeling and supervises no-change areas.
  • Over 1% improvement in Fscd over SOTA methods was achieved on an SCD benchmark dataset.
  • Rare semantic changes with very few samples remain an unresolved challenge for future studies.
Loading 2212.05245v4…