PaperlayerView exact revision on arXiv

Source-linked AI summary

Remote Sensing Image Change Detection with Transformers

Hao Chen, Zipeng Qi, Zhenwei Shi

arXiv:2103.00208v3cs.CV

TL;DR

High-resolution remote-sensing change detection must relate long-range bitemporal context despite appearance variation and the cost of dense pixel-wise attention. BIT compresses bitemporal features into semantic tokens, models their space-time relations with a transformer, and feeds them back to refine pixel features. Across three datasets, it improves accuracy over a convolutional baseline while using substantially lower computational complexity and parameter count.

  • Problem

    High-resolution remote-sensing change detection is challenged by complex scenes, appearance variation across times and locations, and inefficient dense self-attention for long-range context.

  • Method

    BIT tokenizes bitemporal CNN features into compact semantic tokens, applies a transformer encoder to model token-based space-time context, and decodes tokens back to refine pixel features.

  • Results

    Across three datasets, BIT improves F1-score by 1.7/2.4/10.8 points on LEVIR-CD/WHU-CD/DSIFN-CD while using 3 times lower computational complexity and 3 times smaller model parameters.

  • Takeaways & Limitations

    BIT provides an efficient transformer-based alternative to purely convolutional change-detection modules for modeling global bitemporal semantic relations.

Abstract

from arXiv · show

Modern change detection (CD) has achieved remarkable success by the powerful discriminative ability of deep convolutions. However, high-resolution remote sensing CD remains challenging due to the complexity of objects in the scene. Objects with the same semantic concept may show distinct spectral characteristics at different times and spatial locations. Most recent CD pipelines using pure convolutions are still struggling to relate long-range concepts in space-time. Non-local self-attention approaches show promising performance via modeling dense relations among pixels, yet are computationally inefficient. Here, we propose a bitemporal image transformer (BIT) to efficiently and effectively model contexts within the spatial-temporal domain. Our intuition is that the high-level concepts of the change of interest can be represented by a few visual words, i.e., semantic tokens. To achieve this, we express the bitemporal image into a few tokens, and use a transformer encoder to model contexts in the compact token-based space-time. The learned context-rich tokens are then feedback to the pixel-space for refining the original features via a transformer decoder. We incorporate BIT in a deep feature differencing-based CD framework. Extensive experiments on three CD datasets demonstrate the effectiveness and efficiency of the proposed method. Notably, our BIT-based model significantly outperforms the purely convolutional baseline using only 3 times lower computational costs and model parameters. Based on a naive backbone (ResNet18) without sophisticated structures (e.g., FPN, UNet), our model surpasses several state-of-the-art CD methods, including better than four recent attention-based methods in terms of efficiency and accuracy. Our code is available at https://github.com/justchenhao/BIT\_CD.

I. INTRODUCTION

High-resolution remote-sensing change detection must distinguish real changes from appearance variation across times and locations. BIT addresses this by modeling bitemporal context with compact semantic tokens and reports improved accuracy and efficiency.

  • Motivation: High-resolution remote-sensing change detection is difficult because objects with the same semantic concept can have different spectral characteristics across times and locations.Scene complexity and changing imaging conditions complicate recognition of the change of interest.
  • Motivation: Purely convolutional models have limited receptive fields, while non-local self-attention models global pixel relations but are computationally inefficient.Existing attention methods also struggle to fully exploit time-related context.
  • Method: BIT represents bitemporal images with a few semantic tokens and models context in compact token-based space-time rather than dense pixel-space relations.This design targets efficient long-range context modeling for identifying relevant changes and excluding irrelevant ones.
  • Method: BIT uses a CNN backbone, spatial attention tokenization, a transformer encoder, a Siamese transformer decoder, and feature differencing for pixel-level predictions.The decoder reprojects context-rich tokens to refine the original pixel-level features before prediction.
  • Results: Experiments on three change-detection datasets show that the BIT-based model outperforms its purely convolutional counterpart with 3 times lower computational costs and model parameters.Using a naive CNN backbone without FPN or UNet, it also exceeds several recent attention-based methods in efficiency and accuracy.

A. Deep Learning based Remote Sensing Image Change detection

Remote-sensing change detection methods increasingly use attention to enlarge context, but existing approaches often handle temporal information incompletely or at high computational cost. BIT instead models global semantic relations between compact bitemporal tokens and feeds them back to pixel features.

  • Deep Learning based Remote Sensing Image Change detection: Supervised remote-sensing change detection includes two-stage methods that classify each temporal image separately and single-stage methods that directly predict change results.Two-stage methods require both change labels and bitemporal semantic labels to be practical.
  • Deep Learning based Remote Sensing Image Change detection: Single-stage pixel-level approaches directly generate high-resolution change maps from bitemporal inputs using fully convolutional networks.They are distinguished from patch-level similarity detection approaches.
  • Deep Learning based Remote Sensing Image Change detection: Existing methods improve feature discrimination through multi-level fusion, GAN-based objectives, larger receptive fields, and attention mechanisms.These strategies target context modeling across spatial and temporal scopes.
  • Deep Learning based Remote Sensing Image Change detection: Many attention-based methods do not fully exploit temporal context, while non-local self-attention is promising but computationally inefficient.Some methods apply attention separately to each temporal image or reweight fused bitemporal features.
  • Deep Learning based Remote Sensing Image Change detection: BIT extracts a few semantic tokens, models their global relations in bitemporal token-space, and uses the resulting representations to enhance pixel-space features.The method combines convolutional feature extraction with transformer-based context modeling in a change-detection pipeline.

A. Semantic Tokenizer

The semantic tokenizer converts each temporal feature map into a compact set of visual-word tokens. These tokens provide the representation that BIT uses for efficient bitemporal context modeling and subsequent feature refinement.

  • Semantic Tokenizer: BIT processes a pair of registered images by extracting high-level features with a CNN backbone before refining them with BIT.The inference pipeline then produces a pixel-level change mask through a prediction head.
  • Semantic Tokenizer: The two temporal token sets are concatenated and passed through a transformer encoder to model global semantic relations in token-based space-time.The encoder returns context-rich token representations for each temporal image.
  • Semantic Tokenizer: A Siamese transformer decoder projects context-rich tokens back to pixel-space to refine each temporal feature map before feature differencing.The refined features support the final change prediction stage.
  • Semantic Tokenizer: A Siamese tokenizer splits each temporal feature map into a few visual words, with each token vector representing a visual concept.Spatial attention maps pool the feature map into the token set.
  • Semantic Tokenizer: For each feature map, a point-wise convolution forms L semantic groups, softmax produces spatial attention maps, and weighted pixel averages yield L tokens.The tokenization compresses dense spatial features into a compact vocabulary set.

B. Transformer Encoder

The transformer encoder models context between compact semantic token sets from the two temporal images. It uses self-attention, MLP blocks, and positional embeddings to produce context-rich token representations.

  • Token sequence: The two temporal token sets are concatenated, jointly encoded, and then split back into temporal sets of context-rich tokens.The encoder operates on a combined token sequence T ∈ R^2L×C.
  • Encoder blocks: Each encoder layer contains multi-head self-attention and multilayer perceptron blocks.The MLP uses GELU activation, with input and output dimensionality C and inner dimensionality 2C.
  • Self-attention: Self-attention computes query, key, and value projections from the token sequence before applying multi-head attention.The projections use learnable linear parameters, with d denoting the channel dimension of each attention triple.
  • Positional embedding: Learnable positional embeddings are added before the transformer layers to encode relative or absolute position in token-based space-time.The paper reports that positional embeddings are necessary for the tokens and may provide temporal context information.

C. Transformer Decoder

The transformer decoder projects context-rich semantic tokens back into pixel space to refine each temporal image’s features. It uses pixel-to-token cross-attention while removing dense pixel self-attention to reduce computation.

  • Feature refinement: The decoder uses context-rich tokens to refine pixel-level features for each temporal image.Pixels serve as queries, while the corresponding token set supplies keys and values.
  • Feature refinement: Each pixel is represented through its relation to a compact set of semantic tokens.This design projects the learned concept representation from token space back into image-feature space.
  • Decoder blocks: The decoder consists of multi-head cross-attention and MLP blocks across ND layers.Unlike the original decoder implementation, it omits the multi-head self-attention block.
  • Efficiency: Removing pixel self-attention avoids the abundant computation required to model dense relations among pixels.Cross-attention instead derives queries from image features and keys and values from tokens.
  • Positional embedding: Positional embeddings are not added to decoder input queries because experiments showed no considerable gains.

D. Network Details

The network combines a modified ResNet18 backbone with BIT and a shallow change-prediction head. It predicts pixelwise change probabilities from the absolute difference between refined temporal feature maps.

  • CNN backbone: The modified ResNet18 backbone outputs feature maps at downsampling factor 4 to preserve more spatial detail.The last two stages use stride 1, and a point-wise convolution reduces the feature dimension to C = 32.
  • Bitemporal image transformer: BIT uses token length L = 4, one transformer encoder layer, and eight transformer decoder layers.The attention modules use eight heads with channel dimension d = 8 per head.
  • Prediction head: The prediction head applies a very shallow FCN to upsampled refined feature maps from the two temporal images.It produces pixelwise change probability maps with two output channels.
  • Prediction head: Change probabilities are generated by applying a classifier to the element-wise absolute difference between the two feature maps.The classifier contains two 3×3 convolutional layers with BatchNorm and output channels 32 and 2.
  • Inference: During inference, the predicted mask is obtained with pixelwise Argmax over the probability-map channel dimension.
  • Training objective: Training minimizes cross-entropy loss over the pixelwise predictions and labels.For each pixel, the loss is defined as −log(P_hwy).

A. Experimental setup

Experiments evaluate BIT against convolutional baselines on three change-detection datasets using standard change-category metrics. The setup varies dataset resolution, patching, splits, and backbone variants.

  • Datasets: Experiments use LEVIR-CD, WHU-CD, and DSIFN-CD, covering high-resolution building and multi-class land-cover change detection.LEVIR-CD contains building changes, WHU-CD contains building changes, and DSIFN-CD includes roads, buildings, croplands, and water bodies.
  • Datasets: LEVIR-CD provides 637 image pairs at 0.5m resolution, patched to 256 × 256 with 7120/1024/2048 training/validation/test patches.
  • Datasets: WHU-CD contains one 0.075m aerial-image pair patched to 256 × 256 and randomly split into 6096/762/762 training/validation/test patches.
  • Datasets: DSIFN-CD contains six pairs of 2m satellite images from major Chinese cities, with 3600/340/48 training/validation/test samples.
  • Compared models: Comparisons include the standard ResNet18 S5 baseline, lighter S4 and S3 baselines, and BIT variants using S4 or S3 backbones.
  • Implementation: Models are trained with data augmentation and stochastic gradient descent with momentum on a single NVIDIA Tesla V100 GPU.Augmentations include flipping, rescaling, cropping, and Gaussian blur.
  • Evaluation: F1-score for the change category is the main evaluation metric, with precision, recall, IoU, and OA also reported.TP, FP, and FN denote true positives, false positives, and false negatives, respectively.

B. Comparison to state-of-the-art

BIT consistently outperforms convolutional and attention-based change-detection methods across three datasets, despite using a simple CNN backbone.

  • BIT is compared with three purely convolutional methods and four attention-based methods on three CD test sets.
  • BIT consistently outperforms the compared methods across LEVIR-CD, WHU-CD, and DSIFN-CD with a significant margin.Its F1-score exceeds STANet by 2, 1.6, and 4.7 points on the three datasets, respectively.
  • BIT achieves superior performance using a pure ResNet backbone without sophisticated structures such as FPN or UNet.
  • Visualization results show that BIT reduces false positives from visually similar pools and roads and handles seasonal or appearance-related irrelevant changes.

C. Model efficiency and effectiveness

BIT improves change-detection accuracy and training behavior while reducing computational demands relative to deeper convolutional and attention-based alternatives.

  • BIT S4 improves F1-score over Base S5 by 1.7/2.4/10.8 points while using 3 times fewer parameters and computational costs across three test sets.
  • BIT S4 outperforms four attention-based counterparts in F1/IoU with substantially lower computational complexity and model parameters.BIT S3 remains superior on most datasets despite a backbone about 10 times smaller.
  • BIT provides more stable and effective validation accuracy than the Base model, although training accuracy is similar.
  • Removing the transformer encoder causes consistent and significant F1-score drops on all three datasets.
  • BIT outperforms pixel-based Non-local self-attention on all three test sets, supporting compact token-based context modeling.The paper attributes this to the higher information density of the token-based space.
  • Removing the tokenizer or transformer decoder causes performance declines, indicating that token extraction and feature refinement both contribute to BIT.
  • Adding position embeddings to transformer-encoder tokens consistently improves F1-score, whereas adding them to decoder queries yields no significant improvement.

E. Parameter analysis

Parameter analyses examine token length, transformer depth, and token attention to clarify how BIT represents and processes change-related information.

  • Token length: Reducing token length from 32 to 4 significantly improves F1-score, while reducing it from 4 to 2 causes a slight decline.The model therefore uses L = 4, balancing compactness with retained change-related information.
  • Transformer depth: Increasing transformer-encoder depth produces no significant F1/IoU improvement across the three datasets.The paper reports that a single encoder layer can learn relations between bitemporal tokens.
  • Token visualization: Tokenizer attention maps highlight semantic regions associated with the change of interest, including buildings, croplands, and water bodies.On building-change datasets, tokens mainly attend to building pixels; on DSIFN-CD, they attend to multiple semantic areas.
  • Token visualization: Tokenizer attention can also highlight pixels surrounding buildings, such as shadows, without explicit supervision for those areas.The paper identifies surrounding context as a cue for object recognition.

G. Network visualization

BIT transforms bitemporal feature maps into compact semantic tokens, models their space-time context, and projects context-rich tokens back to refine pixel features. The visualization indicates that refinement highlights semantic concepts related to change categories and supports change-probability prediction.

  • Visualization pipeline: The Siamese FCN first generates high-level feature maps, which the tokenizer spatially pools into several token vectors.These tokens provide the compact representation used for subsequent context modeling.
  • Visualization pipeline: A transformer encoder models context among the semantic tokens, and a transformer decoder projects the context-rich tokens back into pixel-space feature maps.The decoder produces refined features from the original temporal-image representations.
  • Activation maps: The refined features extract concepts related to the change of interest, including buildings and their edges, more clearly than the original features.The visualization compares representative maps before and after BIT refinement.
  • Activation maps: BIT further highlights regions of semantic concepts associated with the change category in the difference between refined and original features.The resulting prediction stage generates a change-probability map from the refined and original features.
  • Discussion: The BIT-base model produces more accurate predictions with fewer false alarms and higher recalls than the Base model.The paper attributes the improvement to context-rich token enhancement of the original features.
  • Conclusion: Replacing ResNet18’s last convolutional stage with BIT improves F1-score by 1.7, 2.4, and 10.8 points on LEVIR-CD, WHU-CD, and DSIFN-CD, respectively.The model achieves these gains with 3 times lower computational complexity and 3 times smaller model parameters.
Loading 2103.00208v3…