Source-linked AI summary
HANet: A Hierarchical Attention Network for Change Detection With Bitemporal Very-High-Resolution Remote Sensing Images
Chengxi Han, Chen Wu, Haonan Guo, Meiqi Hu, Hongruixuan Chen
TL;DR
Deep-learning CD methods face severe imbalance between changed and unchanged pixels, complicating learning of changed features. The paper proposes PFBS and the Siamese HANet to address this imbalance and refine multiscale features; experiments on CDD-CD and LEVIR-CD validate their effectiveness and efficiency.
Problem
Deep-learning-based CD methods are hindered by imbalance between changed and unchanged pixels, which limits learning of changed-pixel features during training.
Method
The paper combines PFBS, which progressively emphasizes foreground images without adding change information, with discriminative Siamese HANet for multiscale integration and detailed-feature refinement.
Results
Extensive experimental validation on CDD-CD and LEVIR-CD shows that PFBS and HANet outperform many existing models.
Takeaways & Limitations
The proposed methods are reported to improve change detection in datasets with extremely unbalanced labels while remaining effective and efficient.
Abstract
from arXiv · showhide
Benefiting from the developments in deep learning technology, deep-learning-based algorithms employing automatic feature extraction have achieved remarkable performance on the change detection (CD) task. However, the performance of existing deep-learning-based CD methods is hindered by the imbalance between changed and unchanged pixels. To tackle this problem, a progressive foreground-balanced sampling strategy on the basis of not adding change information is proposed in this article to help the model accurately learn the features of the changed pixels during the early training process and thereby improve detection performance.Furthermore, we design a discriminative Siamese network, hierarchical attention network (HANet), which can integrate multiscale features and refine detailed features. The main part of HANet is the HAN module, which is a lightweight and effective self-attention mechanism. Extensive experiments and ablation studies on two CDdatasets with extremely unbalanced labels validate the effectiveness and efficiency of the proposed method.
I. INTRODUCTION
VHR image change detection is challenged by severe changed-versus-unchanged pixel imbalance and temporal image differences. The paper addresses these issues with PFBS and HANet, which respectively emphasize changed pixels early and integrate multiscale, detailed features.
- Problem: VHR binary change-detection datasets contain extremely few changed pixels relative to unchanged pixels.The paper refers to this setting as extremely unbalanced binary CD.
- Problem: Existing methods often learn directly from imbalanced data, limiting their ability to sufficiently learn changed-pixel features.
- Problem: Attention methods may miss long-range space-time relationships, while some self-attention designs have unacceptably high computational complexity.
- Contributions: PFBS progressively foreground-balances training without adding change information or additional computation cost.It is designed to help the model learn foreground features accurately during early training.
- Contributions: HANet is a discriminative Siamese network that integrates multiscale features and refines detailed spatial and temporal change features.Its HAN module is described as lightweight and effective.
- Results: Extensive experiments and ablation studies on extremely unbalanced CD datasets validate the proposed methods' effectiveness and efficiency.The conclusion states that results on CDD-CD and LEVIR-CD show PFBS and HANet outperform many existing models.
II. RELATED WORK
This section introduces deep learning-based methods and attention mechanisms in the change-detection context.
- The section briefly introduces deep learning-based change-detection methods.
- The section also introduces attention mechanisms for change detection.
- Its stated scope covers both the methods and the attention mechanism used in the CD context.
A. Deep Learning-based CD Methods
Deep learning-based CD methods use learned hierarchical features and Siamese or encoder-decoder architectures to compare bitemporal images, but existing approaches may neglect extreme sample imbalance and end-to-end training.
- Deep Learning-based CD Methods: Hierarchical features provide high-level semantic information for complex backgrounds and low-level spatial details for precise boundaries.
- Deep Learning-based CD Methods: Deep learning CD methods use pixel-level prediction architectures that compare features from different depths to generate difference maps or change vectors.
- Deep Learning-based CD Methods: Siamese networks extract features from bitemporal image pairs using identical structures with shared weights.
- Deep Learning-based CD Methods: Some combined deep-learning and machine-learning approaches cannot achieve end-to-end training.
- Deep Learning-based CD Methods: Existing deep learning CD methods generally do not focus on extremely unbalanced samples.
B. Attention Mechanism
Attention mechanisms weight input regions to extract important information for CD, while the paper develops a lightweight hierarchical design alongside progressive foreground-balanced sampling.
- Attention Mechanism: Attention mechanisms assign different weights to input parts so models can extract more critical information.
- Attention Mechanism: Existing CD attention methods model relationships across channels, space, and bitemporal pixels to generate discriminative features.
- Attention Mechanism: Attention-based approaches have been used to improve boundary integrity, semantic consistency, super-resolution, and interval correlations between low- and high-level features.
- HANet: HANet's architecture integrates multiscale features and refines detailed features through the proposed hierarchical attention design.
- PFBS: PFBS trains foreground images first, then gradually introduces background images to address sample imbalance.
- PFBS: Fixed-X, Linear-Y, and Fixed-X Linear-Y progressively determine when and how background images enter training before the full dataset is used.
B. HANet details
HANet is a Siamese architecture that extracts four-scale building features and uses hierarchical processing to integrate semantic information and refine building details.
- Architecture: HANet uses a Siamese architecture with weight-shared feature extraction for two same-sized temporal input images.The VHR input pairs have size 256 × 256 × 3.
- Output: HANet produces one prediction image from two temporal images after multiscale feature extraction and hierarchical feature refinement.The two temporal images are denoted T1 and T2.
- Multiscale feature extraction: Four convolutional steps extract building features at multiple scales, with adaptive average pooling producing multi-scale feature blocks.The pooling layers use 128-, 64-, and 32-layer features.
- Hierarchical refinement: The HAN module receives extracted features to identify changed regional features and improve feature details.It progressively integrates and refines building semantic features.
- Network components: The architecture contains four convolutional blocks and three adaptive average pooling layers, with residual blocks composed of repeated convolution, batch normalization, and ReLU operations.Each convolutional block repeats the convolutional layer, BN, and ReLU sequence twice.
C. HAN module
The HAN module combines parallel convolutional multiscale fusion with channel and column-row attention to refine features while reducing computational cost.
- HAN pipeline: The HAN module first applies PCS for multiscale feature fusion, then jointly refines features with CAM and column-row attention.The CAM and Col.-Row-A outputs are added to produce the HAN module result.
- Attention refinement: The module contains spatial-spectral axial attention together with column and row attention for feature refinement.CAM and Col.-Row-A are the attention components described for the HAN module.
- Efficiency: Column-row attention reduces the computation required by self-attention and achieves higher computational efficiency.The passage contrasts its cost with second-order self-attention computation.
- Parallel convolution: PCS uses four group convolutions to integrate multiscale features and reduce the amount of input data processed.The group-convolution design supports lower computational cost.
- Loss function: Hybrid loss combines weighted cross-entropy loss and dice loss for the extremely unbalanced change-detection challenge.The loss is adopted to alleviate the impact of data imbalance.
A. Experimental setup
Experiments use the WHU-CD and LEVIR-CD building-change datasets, both with highly imbalanced labels, alongside fixed and progressive foreground-sampling settings.
- Datasets: WHU-CD is a public remote-sensing building-change dataset evaluated with 1260 training and 690 testing image pairs.The default split is reported for the image pairs.
- Data preparation: Image pairs are cropped into nonoverlapping 256 × 256 patches, with 10% of training images randomly selected for validation.The cropping choice reflects GPU memory limitations and supports comparison with other algorithms.
- Datasets: LEVIR-CD contains seasonal and lighting changes that make change detection more difficult.Changed pixels account for 4.65% and unchanged pixels for 95.35% of the dataset.
- Datasets: LEVIR-CD uses a default split of 445 training, 64 validation, and 128 testing image pairs.The dataset is described as an extremely unbalanced binary classification dataset.
- Sampling strategy: The sampling variants are called Fixed-X for fixed foreground images and Linear-X for linearly increasing foreground images.X denotes the number associated with the sampling setting.
(RKCR#CSKCS#)' +
HANet is evaluated against CNN-, attention-, and transformer-based change-detection methods using F1-score, OA, and KC, with quantitative and visual comparisons on WHU-CD and LEVIR-CD.
- Evaluation metrics: F1-score, OA, and KC are the principal metrics, and higher values indicate better change-detection performance.TP, TN, FP, and FN denote the standard confusion-matrix counts used in the metric definitions.
- Quantitative comparison: HANet achieves better performance than the compared methods on the key metrics of F1-score, OA, and KC.The comparison includes CNN-, attention-, and transformer-based methods.
- Quantitative comparison: 0.98/0.08 points: HANet exceeds Change Former in F1-score on WHU-CD and LEVIR-CD, respectively.Change Former is identified as a transformer-based method with more network parameters.
- Feature representation: HANet better integrates contextual semantic information at different scales, attributed in the passage to the HAN module’s semantic extraction ability.The claim concerns the reported comparison with Change Former.
- Visual comparison: Visualizations show fewer missed detections for HANet than several CNN- and attention-based methods on challenging building samples.Blue indicates missed detection and red indicates error detection in the visualization scheme.
- Visual comparison: HANet better avoids false positives from seasonal variation, land-use changes, and complex backgrounds in the reported examples.It also better extracts semantic changes in the region of interest for a changed U-shaped building.
- Visual comparison: HANet extracts detailed edge features from large, non-conventional quadrilateral buildings and small-building semantic features.These observations come from the reported challenging visualization cases.
C. Ablation Study
The ablation study evaluates HANet components, training behavior, and visualization outcomes. Results indicate that PFBS stabilizes convergence while the HAN design supports comparison across attention and sampling variants.
- Model configurations: The ablation study evaluates selected HAN module components and model variants against a CNN Siamese baseline with PCS and CAM.Variants include HANet, Fixed-X, Linear-X, and Fixed-X Linear-Y configurations.
- Visualization analysis: The visualizations compare prediction outcomes, with colors identifying true positives, false positives, true negatives, and false negatives.White denotes TP, red FP, black TN, and blue FN.
- Training stability: Around 20K iterations, the general change-detection method shows an unstable convergence process, whereas HANet with PFBS converges smoothly.The passage equates 20K iterations with 20 epochs and attributes the smoother convergence to PFBS.
- Comparison methods: Visualization comparisons on WHU-CD and LEVIR-CD include BIT, Change Former, and RSP-BIT among the evaluated methods.The comparisons are presented on the corresponding test sets.
T1 T2 GT
The ablations examine PFBS hyperparameters and axial-attention choices on WHU-CD. Fixed sampling and combined column-row attention repeatedly outperform corresponding alternatives and the baseline.
- PFBS settings: Fixed-15 achieves better performance than the baseline, indicating that foreground semantic features in WHU-CD can be learned in 15 epochs.This result is reported in the PFBS ablation visualization.
- PFBS settings: Across parameter choices, the proposed approach exceeds the baseline, although many combinations of X and Y are possible.The comparison concerns Linear-X, Fixed-X, and Fixed-X with Linear-Y settings.
- PFBS settings: Using more foreground images does not necessarily improve performance because results depend on foreground-image scale.Fixed-10 plus Linear-10 is reported to yield relatively good results.
- Axial attention: Combined column-row attention is superior to single axial attention in the reported WHU-CD experiments.The ablation compares column attention, row attention, and their combination.
- PFBS and HAN combination: Fixed-X with the HAN module outperforms Linear-X with the HAN module, and Fixed 25 plus column-row attention is reported as better than Linear-X.Both configurations also exceed the baseline in the PFBS–HAN ablation.
ABLATION STUDY ON THE FIXED-X (F-X) AND LINEAR-X
The ablation studies examine sampling strategies and loss functions for change detection, while the conclusion reports PFBS and HANet validation on two extremely unbalanced datasets. The authors also identify unresolved PFBS settings and broader-model evaluation as future work.
- Loss-function ablation: The ablation study compares Focal Loss and Hybrid Loss on WHU-CD to assess PFBS effectiveness.The cited passages identify the loss-function comparison and state that PFBS is evaluated in this context.
- Loss-function ablation: Focal Loss performs better than Hybrid Loss, while Hybrid Loss with Fixed-15 is reported as better in another comparison.The passages contain both statements, but do not provide the associated numerical values.
- Sampling-strategy ablation: PFBS is designed without adding change information to help models learn foreground features during early training.The strategy is described as progressive foreground-balanced sampling, with Fixed-X and Linear-Y among its forms.
- Network design: HANet is a discriminative Siamese network that integrates multiscale features and refines detailed features.Its HAN module is described as part of the network design, while the supplied passages also mention multi-scale feature blocks.
- Overall validation: Experiments on WHU-CD and LEVIR-CD report that the proposed PFBS and HANet methods outperform many existing models.The datasets are characterized as having extremely unbalanced binary change-detection labels.
- Future work: Future work includes finding optimal X and Y values, testing nonlinear PFBS variants, and evaluating PFBS and HAN modules on additional models.These statements define the current scope boundary of the reported ablations and validation.