PaperlayerView exact revision on arXiv

Source-linked AI summary

Bi-Temporal Semantic Reasoning for the Semantic Change Detection in HR Remote Sensing Images

Lei Ding, Haitao Guo, Sicong Liu, Lichao Mou, Jing Zhang, Lorenzo Bruzzone

arXiv:2108.06103v4cs.CVeess.IV

TL;DR

Semantic change detection must identify both where changes occur and which land-cover/land-use categories appear before and after them, but existing architectures provide insufficient communication between temporal and change branches. The paper proposes a deeply fused CNN architecture and extends it with bi-temporal semantic reasoning blocks and a consistency loss. Experiments report improved semantic-category segmentation and changed-area detection over existing approaches.

  • Problem

    Semantic change detection requires joint semantic segmentation and binary change detection, while existing triple-branch CNNs insufficiently communicate between temporal and change branches.

  • Method

    The paper proposes SSCD-l with deeply fused semantic temporal features, then extends it into Bi-SRNet using Siam-SR blocks, a Cross-temporal SR block, and Semantic Consistency Loss.

  • Results

    Bi-SRNet’s auxiliary designs improve semantic-category segmentation and changed-area detection, while SSCD-l-based methods achieve the highest accuracy metrics on SECOND against SOTA methods.

  • Takeaways & Limitations

    Disentangling semantic segmentation and change detection while deeply integrating their features improves both change results and discrimination of land-cover classes in critical areas.

Abstract

from arXiv · show

Semantic change detection (SCD) extends the multi-class change detection (MCD) task to provide not only the change locations but also the detailed land-cover/land-use (LCLU) categories before and after the observation intervals. This fine-grained semantic change information is very useful in many applications. Recent studies indicate that the SCD can be modeled through a triple-branch Convolutional Neural Network (CNN), which contains two temporal branches and a change branch. However, in this architecture, the communications between the temporal branches and the change branch are insufficient. To overcome the limitations in existing methods, we propose a novel CNN architecture for the SCD, where the semantic temporal features are merged in a deep CD unit. Furthermore, we elaborate on this architecture to reason the bi-temporal semantic correlations. The resulting Bi-temporal Semantic Reasoning Network (Bi-SRNet) contains two types of semantic reasoning blocks to reason both single-temporal and cross-temporal semantic correlations, as well as a novel loss function to improve the semantic consistency of change detection results. Experimental results on a benchmark dataset show that the proposed architecture obtains significant accuracy improvements over the existing approaches, while the added designs in the Bi-SRNet further improves the segmentation of both semantic categories and the changed areas. The codes in this paper are accessible at: github.com/ggsDing/Bi-SRNet.

I. INTRODUCTION

Semantic change detection provides both the locations of changes and the detailed land-cover/land-use categories before and after them. The paper addresses limited CNN-based SCD research and insufficient integration between semantic segmentation and change detection by proposing deeply fused, semantically reasoned architectures.

  • Motivation: Semantic change detection identifies both changed areas and the detailed land-cover/land-use maps before and after change events.This provides richer semantic change information than binary change detection.
  • Motivation: CNN-based change detection learns to segment multi-temporal images and can robustly model complex changes without handcrafted weights or thresholds.Its hierarchical features are combined and transformed to model change information.
  • Research gap: CNN-based SCD remains rarely studied because it combines semantic segmentation of LCLU classes with binary detection of changed areas.Outputs may comprise two temporal LCLU maps and a change map, or two semantic change maps.
  • Research gap: Existing SCD architectures separately model the sub-tasks through triple-branch designs without considering their intrinsic correlations.The temporal branches and change branch therefore do not fully integrate semantic information.
  • Contributions: The paper exploits spatial and temporal semantic correlations and proposes a CNN architecture that disentangles SS and CD while sharing and deeply fusing their features.Separate loss functions supervise semantic segmentation and change detection, while the architecture reuses semantic features for change detection.
  • Contributions: Bi-SRNet adds Siam-SR blocks, a Cross-temporal SR block, and Semantic Consistency Loss to model temporal semantics and align semantic and change representations.The designs are evaluated through ablation studies and comparisons with state-of-the-art methods.

B. CNN-based Change Detection

CNN-based semantic change detection combines semantic segmentation of bi-temporal LCLU classes with binary change detection. Existing architectures differ in fusion and task disentanglement, while SSCD-l merges semantic temporal features in a change unit to address limited temporal communication.

  • Task formulation: SCD jointly requires bi-temporal LCLU semantic segmentation and binary detection of changed areas.Its outputs can be two temporal LCLU maps plus a change map, or two semantic change maps encoding both classes and changed regions.
  • Possible architectures: Post-classification comparison is sub-optimal because independently produced LCLU maps neglect temporal correlations and can accumulate errors.CNNs instead learn semantic changes directly, an approach associated with multidate direct classification in multi-class change detection.
  • Possible architectures: CNN approaches include direct early fusion, direct late fusion, and disentangled semantic-segmentation/change architectures with early or late fusion.DSCD-e concatenates temporal images, whereas DSCD-l separately encodes them before convolutional change modeling; SSCD-e separates semantic and change information before masking.
  • Architecture limitations: Direct early fusion underuses temporal LCLU information, while changed-area imbalance drives the encoder toward unchanged regions.This limitation is reported for architectures such as FC-EF and UNet++.
  • Architecture limitations: The disentangled early-fusion architecture separately models semantic and change information, then masks temporal semantic maps with a shared change map.Separate encoders produce P1 and P2, while a change encoder produces C; masking yields S1 and S2.
  • Proposed architecture: SSCD-l merges extracted temporal semantic features to train a change unit, explicitly modeling LCLU and change information while producing one consistent change map.The architecture is designed to address limited communication between temporal and change branches and is evaluated against alternative CNN architectures.

B. Bi-temporal Semantic Reasoning Network

Bi-SRNet builds on SSCD-l by reasoning about semantic correlations across two temporal images while separately modeling semantic and change information. It produces two semantic maps and one binary change map, then derives semantic change maps by masking the temporal predictions.

  • Motivation: Bi-SRNet addresses temporal correlation and coherence by adding semantic reasoning to the SSCD-l architecture.The network targets the predominantly unchanged regions, whose visual patterns are often similar across observation intervals.
  • Architecture: Two CNN encoders extract temporal semantic features, which are enhanced by Siam-SR blocks before semantic-map projection.Encoder and Siam-SR weights are shared when the temporal domains have no significant difference, reducing over-fitting risks.
  • Architecture: Bi-SRNet uses temporally aligned features for semantic maps and unaligned features for the change block.The semantic maps P1 and P2 and binary change map C are produced through separate 1×1 convolutional projections.
  • Outputs: The network outputs P1, P2, and C, then computes semantic change maps S1 and S2 by multiplying C with the corresponding semantic predictions.This preserves separate direct outputs for semantic segmentation and binary change detection while producing masked semantic change results.
  • Implementation: SR blocks operate at 1/8 of the input resolution, and the network outputs are directly enlarged to produce final results.The selected scale is described as balancing spatial accuracy with context modelling distance.

C. Semantic Reasoning Blocks

The semantic reasoning design combines within-image spatial reasoning with cross-temporal consistency modeling. Siam-SR enhances each temporal branch, while Cot-SR transfers attention across branches to exploit shared semantic context.

  • Design rationale: Non-local units provide the basis for semantic reasoning by modeling long-range dependencies in images.The paper uses them to capture spatial and cross-temporal relationships relevant to semantic segmentation and SCD.
  • Siam-SR: Siam-SR blocks use shared-weight non-local units to aggregate spatial information and embed semantic focuses within each temporal branch.An input feature is projected into query, key, and value vectors, followed by an attention matrix over spatial positions.
  • Siam-SR: The attention matrix records correlations between every pair of spatial positions and is used to obtain an enhanced feature.The matrix is normalized along the row dimension before operating on the value features.
  • Cot-SR: Cot-SR extends non-local reasoning to cross-temporal information by simultaneously enhancing the two temporal features.It generates attention maps for each branch and applies each map to the opposite temporal branch.
  • Combined reasoning: Together, Siam-SR and Cot-SR model spatial correlations within each image and semantic consistency between the bi-temporal images.Siam-SR supplies single-temporal semantic focusing, whereas Cot-SR enhances features in unchanged areas through cross-temporal consistency.

D. Loss Functions

Bi-SRNet uses separate semantic and change losses, linked by a semantic consistency loss. The combined supervision trains temporal semantic branches and the change block while encouraging temporally consistent predictions.

  • Loss design: Training uses semantic class loss Lsem, binary change loss Lchange, and semantic consistency loss Lsc.The first two supervise semantic segmentation and change detection separately; the third links the tasks.
  • Semantic loss: Lsem is multi-class cross entropy between temporal predictions P1 and P2 and the corresponding ground-truth semantic change maps.The no-change class is excluded so temporal branches focus on extracting semantic features.
  • Change loss: Lchange is binary cross entropy between the predicted change map C and a reference map Lc derived by converting non-zero semantic labels to the changed label 1.The reference map can be generated from either temporal ground-truth semantic map.
  • Consistency loss: SCLoss rewards similar semantic probability distributions in unchanged areas and penalizes them in changed areas.It aligns bi-temporal semantic representations and guides training of the Cot-SR block.
  • Joint supervision: The temporal branches receive direct semantic supervision and consistency assistance, while the change block receives direct supervision from Lc.The total loss combines supervision across the three outputs; SSCD-l uses no hyper-parameters to balance semantic and change losses.

IV. DATASET DESCRIPTION AND EXPERIMENTAL SETTINGS

This section introduces the dataset, evaluation metrics, and experimental settings used for the study.

  • The section covers the dataset, evaluation metrics, and experimental settings.

A. Dataset

SECOND is a benchmark dataset for semantic change detection, built from bitemporal high-resolution optical imagery collected across several Chinese cities.

  • SECOND contains bitemporal HR optical images with RGB channels collected by several aerial platforms and sensors.
  • The observed regions include Hangzhou, Chengdu, and Shanghai, China.
  • Each image measures 512 × 512 pixels, with spatial resolution ranging from 0.5m to 3m per pixel.

B. Evaluation Metrics

The evaluation uses OA, mIoU, and SeK to assess overall classification, changed-region discrimination, and LCLU semantic segmentation, while Fscd specifically evaluates LCLU segmentation in changed areas.

  • The study adopts overall accuracy (OA), mean Intersection over Union (mIoU), and Separated Kappa (SeK) to evaluate SCD accuracy.
  • OA is limited because no-change pixels dominate and incorrect LCLU labels within changed pixels are not counted adequately.
  • mIoU averages IoU for no-change and changed regions, directly evaluating discrimination of these two region types.
  • SeK excludes true-positive no-change pixels from its confusion matrix because their number is dominant.
  • Fscd measures LCLU-class segmentation accuracy specifically within changed areas using changed-area precision and recall variants.
  • Computational costs are reported with Params, FLOPs, and inference time for paired 512 × 512 input images.

C. Experimental settings

All CNN models use a common PyTorch training setup with SGD optimization, fixed training parameters, and random geometric augmentation.

  • The experiments use PyTorch CNN implementations trained with batch size 8, 50 epochs, and initial learning rate 0.1.
  • Optimization uses stochastic gradient descent with Nesterov momentum.
  • Image-pair augmentation applies random flipping and rotation during loading.
  • Experiments run on a workstation equipped with an NVIDIA Quadro P6000 GPU.

V. EXPERIMENTAL RESULTS

Experiments identify SSCD-l as the strongest base architecture and show that Bi-SRNet’s semantic reasoning and auxiliary loss further improve semantic and change detection performance.

  • V. EXPERIMENTAL RESULTS: The experiments evaluate SSCD-l and Bi-SRNet components, then compare the proposed methods with recent SCD and binary CD approaches.
  • A. Comparison of SCD Architectures: The architecture comparison uses four basic CNN designs with a shared ResNet34 encoder and ×1/8 down-sampling stride.
  • A. Comparison of SCD Architectures: SSCD-l surpasses SSCD-e by 4.81% in SeK and 4.26% in Fscd, with computational cost only slightly above DSCD-l.
  • A. Comparison of SCD Architectures: SSCD-l reduces inconsistencies between semantic maps and change maps by directly providing semantic features to the CD unit.
  • A. Comparison of SCD Architectures: SSCD-l provides the most accurate SCD results among the compared architectures, with particularly strong semantic embedding performance indicated by SeK.
  • B. Ablation Study: SCLoss improves SeK by around 0.82% and Fscd by 0.76%, while Siam-SR and Cot-SR provide additional semantic and change-detection gains.
  • B. Ablation Study: Compared with standard SSCD-l, Bi-SRNet improves mIoU by around 0.81%, SeK by 1.36%, and Fscd by 1.39%.
  • B. Ablation Study: Qualitative results show improved identification of ground, low vegetation, and tree classes after adding SCLoss and semantic reasoning blocks.

C. Comparative Experiments

Comparisons show that architectures separating semantic segmentation and change detection, while deeply merging temporal semantic features, outperform alternative SCD designs and literature methods. Bi-SRNet further improves semantic classification and change detection, with inference time remaining intermediate among compared methods.

  • Architecture comparisons: DSCD-e methods generally perform poorly because semantic and change information are entangled, whereas SSCD-l-based methods obtain the best accuracy.The reported comparison covers mIoU, SeK, and Fscd across proposed and literature methods.
  • Literature comparisons: Bi-SRNet outperforms state-of-the-art methods by 1.45% in mIoU, 3.06% in SeK, and 3% in Fscd.The comparison is reported for the SCD task on the SECOND dataset.
  • Computational comparison: SSCD-l and Bi-SRNet have inference times near some UNet-based methods and lower than the most computationally intensive decoder-based alternatives.Most proposed-model calculations occur on down-scaled feature maps.
  • Qualitative comparisons: Bi-SRNet captures change types that competing methods omit, including low vegetation-to-tree and tree-to-ground transitions.The qualitative comparison also reports fewer critical-area confusions than SSCD-e-based approaches.

VI. CONCLUSIONS

The paper proposes SSCD-l and extends it into Bi-SRNet to improve semantic change detection through feature reuse and bi-temporal semantic modelling. Experiments report improvements over standard architectures and state-of-the-art methods, while temporal LCLU transitions in changed areas remain incompletely exploited.

  • Conclusions: SSCD-l disentangles semantic segmentation and change detection while deeply integrating them through reused semantic features and separate loss functions.The architecture merges semantic temporal features in a deep convolutional change-detection block.
  • Conclusions: Bi-SRNet adds Siam-SR blocks, a Cot-SR block, and SCLoss to model temporal semantics and enhance temporal coherence.These designs target single-temporal and cross-temporal semantic correlations.
  • Conclusions: Bi-SRNet improves both land-cover segmentation and change detection over SSCD-l, including better discrimination of critical land-cover areas.The conclusion attributes this to semantic reasoning blocks and SCLoss modelling spatial and temporal correlations.
  • Limitations: Temporal correlations of LCLU classes, especially in changed areas, are not fully exploited by the proposed method.Learning LCLU transition types and adding more CD-unit connections to temporal branches are identified as future directions.
Loading 2108.06103v4…