PaperlayerView exact revision on arXiv

Source-linked AI summary

HANet: A Hierarchical Attention Network for Change Detection With Bitemporal Very-High-Resolution Remote Sensing Images

Chengxi Han, Chen Wu, Haonan Guo, Meiqi Hu, Hongruixuan Chen

arXiv:2404.09178v1cs.CV

TL;DR

Differences in light illumination can negatively affect extraction from temporal remote-sensing images. HANet combines progressive foreground-balanced sampling with a discriminative Siamese network, and experiments report that the proposed methods outperform many existing models.

  • Problem

    Differences in light illumination can negatively affect extraction from temporal remote-sensing images.

    Source: §I · p. 1

  • Method

    The paper proposes progressive foreground-balanced sampling and a discriminative Siamese HANet that integrates multiscale features and refines detailed features.

    Source: §V · p. 13

  • Results

    The proposed PFBS and HANet outperform many existing models.

    Source: §V · p. 13

  • Takeaways & Limitations

    The reported experiments support the effectiveness of the proposed methods on change-detection datasets.

    Source: §V · p. 13

Abstract

from arXiv · show

Benefiting from the developments in deep learning technology, deep-learning-based algorithms employing automatic feature extraction have achieved remarkable performance on the change detection (CD) task. However, the performance of existing deep-learning-based CD methods is hindered by the imbalance between changed and unchanged pixels. To tackle this problem, a progressive foreground-balanced sampling strategy on the basis of not adding change information is proposed in this article to help the model accurately learn the features of the changed pixels during the early training process and thereby improve detection performance.Furthermore, we design a discriminative Siamese network, hierarchical attention network (HANet), which can integrate multiscale features and refine detailed features. The main part of HANet is the HAN module, which is a lightweight and effective self-attention mechanism. Extensive experiments and ablation studies on two CDdatasets with extremely unbalanced labels validate the effectiveness and efficiency of the proposed method.

I. INTRODUCTION

The introduction describes VHR bitemporal change detection as binary labeling complicated by acquisition differences and severe changed/unchanged pixel imbalance. It presents PFBS for early foreground-feature learning without added change information and HANet for integrating multiscale features and refining spatial and temporal details.

I. INTRODUCTION

  • The excerpts describe change detection as assigning binary labels (“change” or “no change”) to identify changed regions, while background changes may be irrelevant and arise from seasonal or illumination variation.Source: §I · pp. 1–2
  • Temporal differences in illumination, contrast, quality, resolution, and noise hinder feature extraction in remote-sensing images.VHR imagery also contains more spatial detail, which can lead traditional methods to fragment changed regions and create salt-and-pepper noise in unchanged areas.Source: §I · pp. 1–2
  • The passages describe traditional methods as depending primarily on original image information, with handcrafted features also forming a primary basis for those approaches.Source: §I · p. 2
  • In [11], three fully convolutional neural network architectures are proposed: FC-EF, FC-Siam-conc, and FC-Siam-diff.Source: §I · p. 2
  • The introduction identifies extremely low changed-pixel prevalence as extreme binary change-detection imbalance, which existing methods often ignore, limiting feature learning.Source: §I · p. 3
  • It also notes that some attention methods struggle with long-range space-time relationships or have unacceptably high computational complexity, motivating lightweight attention.Source: §I · p. 3
  • PFBS addresses binary change-detection imbalance without adding change information or incurring additional computation cost.The authors describe it as a progressive foreground-balanced sampling strategy.Source: §I · p. 3
  • The progressive policy first centralizes minority foreground samples so the network learns the most significant characteristics of change.Source: §I · p. 3
  • HANet is a discriminative Siamese network that integrates multiscale features, refines spatial and temporal change details, and uses lightweight HAN to capture long-term dependencies across rows and columns.Source: §I · p. 3
  • Experiments and ablation studies on two extremely unbalanced binary change-detection datasets are reported to validate the method’s effectiveness and efficiency.Source: §I · p. 3

II. RELATED WORK

The related-work section briefly introduces deep-learning-based change-detection methods and attention mechanisms for change detection.

II. RELATED WORK

  • The section’s scope is deep-learning-based change-detection methods and attention mechanisms.It briefly introduces both topics.Source: §II · p. 3

A. Deep Learning-based CD Methods

Deep-learning change detection (CD) uses fully convolutional structures for pixel-level prediction, comparing bitemporal image features to produce change maps or vectors. Prior methods include Siamese networks and segmentation architectures, but some combine deep learning with clustering and do not train end to end, while existing methods may neglect sample imbalance.

A. Deep Learning-based CD Methods

  • Deep-learning CD uses fully convolutional structures for pixel-level prediction, comparing features at different depths to generate difference maps or change vectors.The passage notes that high-level features provide abstract semantic information and low-level features contain spatial detail.Source: A. Deep Learning-based CD Methods · p. 3
  • The Siamese CNN in used K-nearest neighbors to cluster similar feature-map pixels into changed regions, so the process was not end-to-end.The passage describes these approaches as combining deep neural networks with machine-learning algorithms.Source: A. Deep Learning-based CD Methods · p. 3
  • Prior architectures combined bitemporal images for modified U-Net [35] in, or applied UNet++ [37] to change detection in.Deeply supervised image-fusion strategies were also used in encoder structures to capture global and fine-boundary information for VHR remote-sensing CD.Source: A. Deep Learning-based CD Methods · pp. 3–4
  • The Siamese fully convolutional network in [11] extracted paired-image features with identical shared-weight branches, but these methods did not address extreme changed/unchanged sample imbalance.The passage links deeply supervised image fusion with improved VHR CD and identifies the imbalance and feature-learning rules as neglected issues.Source: A. Deep Learning-based CD Methods · p. 4

B. Attention Mechanism · III. PROPOSED HANET · A. Progressive Foreground-Balanced Sampling (PFBS)

The merged sections introduce HANet and its attention-based feature refinement, alongside PFBS, which addresses changed/unchanged pixel imbalance by prioritizing foreground learning early and then incorporating background data progressively.

B. Attention Mechanism

  • Attention assigns different weights to input parts to extract important information and support more accurate judgments.The passage states that attention does not introduce additional calculation or storage overhead.Source: B. Attention Mechanism · p. 4

III. PROPOSED HANET

  • The proposed HANet section introduces the method’s motivation and model details and points to Fig. 4 for its overall architecture.The supplied passages do not describe further architectural components here.Source: §III · p. 4

A. Progressive Foreground-Balanced Sampling (PFBS)

  • PFBS prioritizes foreground learning early, then progressively introduces background data to help the model learn changed-pixel features despite sample imbalance.Its stated aim is to improve the influence of changed pixels during training.Source: A. Progressive Foreground-Balanced… · p. 4
  • Fixed-X trains on the 1200 foreground images for the first X epochs before using the full set, while Normal Train uses all data each epoch.The WHU-CD example has 4536 training images: 1200 foreground and 3336 background.Source: A. Progressive Foreground-Balanced… · p. 4
  • Linear-Y starts with foreground images, gradually increases the background-to-foreground ratio over Y epochs, and then trains on the full dataset.In the WHU-CD example, it adds 222 background images per step.Source: A. Progressive Foreground-Balanced… · p. 4
  • Fixed-X Linear-Y combines the schedules by training on foreground images for X epochs, adding background linearly through Y epochs, then using the full dataset.The example adds 333 background images per step.Source: A. Progressive Foreground-Balanced… · pp. 4–5

B. HANet details · C. HAN module

HANet pairs a weight-shared Siamese multiscale feature extractor with a hierarchical attention module that integrates semantic information and refines building and change-region details. The HAN module combines multiscale feature fusion with channel and spatial-spectral axial attention.

B. HANet details

  • HANet is a weight-shared Siamese network combining multiscale feature extraction with HAN to integrate semantics and refine building and change-region features.The feature extractor supplies multiscale building semantics, while HAN progressively identifies changed-region features and improves their detail.Source: B. HANet details · p. 5
  • For 256 × 256 × 3 VHR image pairs, HANet extracts four-scale building features in four steps and pools 128-, 64-, and 32-layer features.Source: B. HANet details · p. 5
  • Four convolutional blocks use residual connections and repeated convolution, batch-normalization, and ReLU sequences, alongside three adaptive-average-pooling layers.The convolution, batch normalization, and ReLU sequence is repeated twice within a block.Source: B. HANet details · p. 5

C. HAN module

  • HAN comprises a parallel convolution structure (PCS) and spatial-spectral axial attention, including column and row attention [43], [44].Source: C. HAN module · p. 5
  • PCS uses four different group convolutions to integrate multiscale features, increase feature-extraction effectiveness, and reduce model computational cost.Source: C. HAN module · p. 5
  • PCS first fuses multiscale features; CAM and Col.-Row-A then refine them in parallel, and their outputs are added to form the HAN output.Source: C. HAN module · p. 5
  • CAM refines detailed features, whereas Col.-Row-A computes horizontal then vertical attention to reduce second-order computation and improve efficiency.Source: C. HAN module · p. 6
  • By combining PCS, CAM, and Col.-Row-A, HAN extracts multiscale and contextual information to better capture building context, interiors, and edges.Source: C. HAN module · p. 6

D. Loss Function

To mitigate the impact of data imbalance, the paper adopts hybrid loss, combining weighted cross-entropy and Dice losses.

D. Loss Function

IV. EXPERIMENT · A. Experimental setup

The experiment section introduces the datasets, environment, comparison models, and evaluation metrics, then discusses results and ablations. The supplied passages detail the dataset and training setup but provide no result comparisons.

IV. EXPERIMENT

  • The experimental section introduces comparison models and evaluation metrics, and states that it discusses experimental results and ablation studies.Source: §IV · p. 7

A. Experimental setup

  • WHU-CD [21] is a public building-change dataset with 4.26% changed pixels and 95.74% unchanged pixels, making it extremely imbalanced.It uses 0.2 m/pixel imagery; the described image pair covers 32,507 × 15,345 pixels.Source: A. Experimental setup · p. 7
  • WHU-CD uses the default 1,260/690 training/testing image-pair split; image pairs were cropped to nonoverlapping 256 × 256 patches, with 10% of training images reserved for validation.The resulting patch split is 4,536/504/2,760 training/validation/testing pairs.Source: A. Experimental setup · p. 7
  • LEVIR-CD is a public large-scale remote-sensing dataset with seasonal and light changes that make change detection more difficult; it has 31,066,643 changed pixels (4.65%) and 636,876,269 unchanged pixels (95.35%).Source: A. Experimental setup · p. 7
  • LEVIR-CD uses the default 445/64/128 training/validation/testing image-pair split, with nonoverlapping 256 × 256 crops yielding 7,120/1,024/2,048 patch pairs.Source: A. Experimental setup · pp. 7–8
  • Models were implemented in PyTorch and trained on a single NVIDIA RTX 3090 GPU using Adam, with weight decay 5e-4, learning rate 5e-4, and gamma 0.5 for updating the learning rate.The learning-rate update used a step size of 8; the passages do not identify a scheduler.Source: A. Experimental setup · p. 8
  • The setup uses evaluation metrics to quantitatively assess the proposed model’s effectiveness.Source: A. Experimental setup · p. 8

(RKCR#CSKCS#)' + · B. Comparison with state-of-the-art methods

The section compares HANet with CNN-, attention-, and transformer-based change-detection methods using quantitative results and qualitative examples on WHU-CD and LEVIR-CD. It also reports HANet’s extraction of detailed building features, including edges of large, non-conventional quadrilateral buildings.

(RKCR#CSKCS#)' +

B. Comparison with state-of-the-art methods

  • HANet's F1-score exceeds Change Former's by 0.98/0.08 points on WHU-CD/LEVIR-CD, respectively, and HANet performs better on F1-score, OA, and KC.The comparison covers both datasets, which the paper describes as having highly unbalanced samples.Source: B. Comparison with state-of-the-art… · p. 9
  • HANet integrates contextual semantic information at different scales, and its HAN module is described as a lightweight, effective self-attention mechanism.The paper attributes the result to the HAN module's semantic-information extraction ability.Source: B. Comparison with state-of-the-art… · p. 9
  • In visual examples, HANet avoids false negatives on complex building features, including cases where CNN and attention-based baselines miss detections.Compared with SNUNet and MSPSNet on one example, HANet has fewer blue pixels, which indicate missed detections.Source: B. Comparison with state-of-the-art… · p. 9
  • HANet better avoids false positives on complex backgrounds and extracts semantic changes in regions of interest, including a changed impermeable surface.The examples include irrelevant changes associated with seasonal variation and land-use/land-cover change.Source: B. Comparison with state-of-the-art… · p. 9
  • HANet better extracts detailed building features, including edges of large, non-conventional quadrilateral buildings (Fig. 6 (h)).The comparison specifically notes this advantage even when the quadrilateral building size is very large.Source: B. Comparison with state-of-the-art… · p. 9

C. Ablation Study

The ablation study reports smoother HANet convergence after PFBS, contrasting with instability in a general change-detection method. It defines a baseline and HANet variants to test the model and attention configurations.

Ablation Study

  • The Base model is a CNN Siamese Network with PCS and CAM, while HANet adds Row-Col.-A to Base.Source: C. Ablation Study · p. 9
  • The study also evaluates HANet-Fixed-X and HANet-Linear-X variants by adding Fixed-X or Linear-X, respectively.Source: C. Ablation Study · p. 9

Ablation on training stability

  • PFBS makes HANet convergence smooth, while the general change-detection method is unstable around 20K iterations (20 epochs).The passage attributes the smoother convergence to PFBS.Source: C. Ablation Study · p. 9

T1 T2 GT · ABLATION STUDY ON THE FIXED-X (F-X) AND LINEAR-X

The ablations show Fixed-15 outperforming the baseline, while the Fixed-X/Linear-X study reports gains across tested settings and finds that performance depends on foreground-image scale.

T1 T2 GT

  • Fixed-15 outperforms the baseline, indicating WHU-CD foreground building semantics can be learned within 15 epochs.The comparison is visualized in Fig. 7.Source: Figure 7 · p. 11, T1 T2 GT · p. 11

ABLATION STUDY ON THE FIXED-X (F-X) AND LINEAR-X

T1 T2 GT

ABLATION STUDY ON THE FIXED-X (F-X) AND LINEAR-X

ABLATION STUDY ON THE KEY PART OF HAN MODULE … V. CONCLUSION

The conclusion presents PFBS and HANet as complementary contributions and reports that they outperform many existing models on WHU-CD and LEVIR-CD. Ablations support combining row and column attention and show benefits from PFBS and the HAN module, while the authors identify open questions for future work.

ABLATION STUDY ON THE KEY PART OF HAN MODULE

  • Combining column and row attention outperforms using either single axial attention variant.The passage notes that scores differ between single-column and single-row attention because parameters are initialized randomly.Source: ABLATION STUDY ON THE KEY PART OF HAN… · p. 12

ABLATION STUDY ON THE FIXED-X (F-X) AND LINEAR-X

  • Both Fixed-X plus HAN module and Linear-X plus HAN module exceed the baseline, with Fixed-X plus HAN module performing better.The passage also reports that Fixed 25 plus Col.-Row-A is better than Linear-X plus Col.-Row-A.Source: ABLATION STUDY ON THE KEY PART OF HAN… · p. 12

FUNCTIONS INCLUDING HYBRID LOSS(HL) AND FOCAL

V. CONCLUSION

  • PFBS uses no added change information to help the model learn foreground features during early training.The conclusion describes PFBS as progressive foreground-balanced sampling.Source: §V · p. 13
  • HANet is a discriminative Siamese network designed to integrate multiscale features and refine detailed features.Source: §V · p. 13
  • PFBS and HANet outperform many existing models on the extremely unbalanced WHU-CD and LEVIR-CD datasets.The conclusion reports validation on two binary change-detection datasets.Source: §V · p. 13
  • Future work should examine optimal PFBS parameters, additional PFBS forms and models, and transfer of the HAN module to other models.The conclusion specifically mentions nonlinear increases as one possible PFBS form.Source: §V · p. 13

Related papers

Loading 2404.09178v1…