Remote Sensing Change Detection With Transformers Trained from Scratch

1. Introduction

• Change detection identifies relevant changes between co-registered satellite images; CNN-based approaches can struggle to capture long-range dependencies between image regions.The target includes changes in man-made facilities while ignoring noisy changes such as shadows and seasonal or environmental variations.Source: §1 · p. 1
• Existing transformer-based CD methods generally rely on ImageNet or another CD dataset for pre-training, while direct training on the target dataset performs poorly.The introduction attributes this challenge in part to dense self-attention’s quadratic token complexity, slower convergence, and overfitting tendency.Source: §1 · p. 1
• Common two-stream methods fuse features by difference, summation, or concatenation without explicit re-weighting, which may inadequately aggregate semantic changes.ScratchFormer’s CEFF addresses this by performing per-channel re-calibration.Source: §1 · pp. 1–2
• In DSIFN-CD examples, ScratchFormer reduces the false positives and false negatives shown for ChangeFormer [1].The figure marks false positives in purple and false negatives in red.Source: Figure 1 · p. 2
• ScratchFormer combines shuffled sparse attention (SSA), which attends to sparse informative regions, with CEFF, which re-weights channels to enhance semantic changes and suppress noisy ones.The architecture is designed for training directly on the target CD dataset.Source: §1 · p. 2
• On DSIFN-CD, ScratchFormer achieves 90.75% IoU, outperforming the best existing method [1] in the literature by 14.27%, and sets new state-of-the-art performance across all four evaluated datasets.Source: §1 · p. 2

2. Preliminaries

• Change detection predicts a binary mask of relevant semantic changes between paired images, excluding seasonal, illumination, shadow, and atmospheric variations.The input images are co-registered satellite images acquired at distinct times.Source: §2 · p. 2

2.1. Baseline Change Detection Framework

• The baseline shares transformer-encoder weights across pre- and post-change image streams, fuses multi-scale difference features, and decodes them into a semantic change map.The encoder uses standard self-attention; the decoder upsamples fused features before mask prediction.Source: §2.1 · pp. 2–3
• Standard self-attention over uniformly sampled dense patches requires large training data for optimal convergence; pre-training on another CD dataset adds training time.Figure 2 compares IoU across training epochs for scratch-trained, pre-trained, and proposed approaches; ScratchFormer reaches similar performance to the scratch baseline’s final result with 10% of its training time.Source: §2.1 · p. 3, Figure 2 · p. 3

3.1. Motivation

• The method targets two CD challenges: learning useful representations from small datasets and distinguishing relevant semantic changes from noisy ones.The motivation identifies sparse informative regions and explicit per-channel modeling between pre- and post-change features as design priorities.Source: §3.1 · p. 3

3.2. Overall Architecture

• ScratchFormer uses a shared-weight, four-stage Siamese encoder, CEFF to fuse paired features at each stage, and a decoder to predict a binary change map.The architecture computes features at different spatial resolutions; CEFF reweights paired features per channel to retain relevant changes while suppressing noisy ones.Source: §3.2 · p. 3

3.3. Shuffled Sparse Attention Layer

• SSA samples data-dependent feature subsets, performs token mixing on them, then returns attended features to the full-resolution feature map.Learned position offsets guide the sampling; sparse attention maintains a global receptive field while reducing computation by a factor reported as O(γ^2).Source: §3.3 · pp. 4–5

3.4. Change Enhanced Features Fusion Module

• CEFF combines paired features and reweights their channels to emphasize semantic changes and suppress noisy changes.It predicts channel weights for both streams from a pooled feature vector, normalizes them across the streams, and fuses the weighted features by addition.Source: §3.4 · p. 5

4.1. Experimental Setup

• The experiments use four public datasets, 256 × 256 input patches, and random initialization; training runs for 300 epochs on four NVIDIA A100 GPUs.The setup also describes separate dataset splits, data augmentation, and evaluation using F1, IoU, and overall accuracy.Source: §4.1 · pp. 5–6

4.2. State-of-the-art Comparison

• ScratchFormer sets state-of-the-art on four datasets without pre-training; on DSIFN-CD, it gains 8.48% F1, 2.8% OA, and 14.27% IoU over ChangeFormer [1].The comparison covers DSIFN-CD, LEVIR-CD, WHU-CD, and CDD-CD.Source: Table 1 · p. 6, §4.2 · pp. 6–7
• Qualitative comparisons show clearer semantic-change boundaries on DSIFN-CD and LEVIR-CD, including better localization of changes at multiple scales.The examples compare ScratchFormer with BIT [2] and ChangeFormer [1].Source: §4.2 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7

4.3. Ablation Study

• On DSIFN-CD, IoU rises from 78.11% for the scratch baseline to 83.94% after LEVIR-CD pre-training, 86.62% with SSA, and 90.75% with SSA plus CEFF trained from scratch.The ablation also reports consistent gains from adding SSA and CEFF to the baseline on LEVIR-CD.Source: §4.3 · pp. 7–8, Table 2 · p. 8
• γ = 4 provides the best reported sparsity performance on both DSIFN-CD and LEVIR-CD, so the experiments use that setting throughout.Source: §4.3 · p. 8, Table 3 · p. 8
• CEFF's per-channel feature re-weighting outperforms addition, subtraction, and concatenation on DSFIN-CD.The comparison evaluates the fusion techniques on all reported metrics.Source: §4.3 · p. 8

• CNN-based change detection uses approaches such as dual attention, channel attention, and feature pyramids to capture discriminative features or long-range dependencies.Source: §5 · p. 8
• Transformer-based prior work includes BIT, TransUNetCD, multi-scale Swin transformers, hybrid transformers, and hierarchical Siamese transformers [1].Source: §5 · p. 8
• Unlike the described prior approaches, ScratchFormer uses SSA for scratch training and CEFF for channel re-weighting; CEFF outperforms addition, subtraction, and concatenation on DSFIN-CD.Source: §5 · p. 8, Table 4 · p. 8

• ScratchFormer is a Siamese transformer architecture for remote-sensing change detection that uses SSA and CEFF.Source: §6 · p. 8
• The authors report state-of-the-art change-detection performance on all four evaluated datasets.Source: §6 · p. 8
• Future work could explore change detection in natural images and medical imaging.Source: §6 · p. 9