Source-linked AI summary
Multi-stage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images
Rui Li, Shunyi Zheng, Chenxi Duan, Jianlin Su, Ce Zhang
TL;DR
Dot-product attention becomes prohibitively expensive as input size grows because its memory and computation scale quadratically. The paper proposes a Linear Attention Mechanism and uses it to redesign U-Net skip connections in MAResU-Net; on Vaihingen, the model achieves strong segmentation accuracy and computational efficiency.
Problem
Dot-product attention’s memory and computational costs increase quadratically with input size, hindering its use for large-scale inputs.
Method
The paper uses a first-order Taylor-based Linear Attention Mechanism and integrates it into multi-stage ResNet-backed U-Net skip connections.
Results
MAResU-Net based on ResNet-34 achieves a mean F1-score of 90.277 %, OA of 90.860%, and mIoU of 83.301% on the Vaihingen dataset.
Takeaways & Limitations
The proposed approach demonstrates effectiveness in fine-resolution remote sensing segmentation while maintaining computational efficiency.
Abstract
from arXiv · showhide
The attention mechanism can refine the extracted feature maps and boost the classification performance of the deep network, which has become an essential technique in computer vision and natural language processing. However, the memory and computational costs of the dot-product attention mechanism increase quadratically with the spatio-temporal size of the input. Such growth hinders the usage of attention mechanisms considerably in application scenarios with large-scale inputs. In this Letter, we propose a Linear Attention Mechanism (LAM) to address this issue, which is approximately equivalent to dot-product attention with computational efficiency. Such a design makes the incorporation between attention mechanisms and deep networks much more flexible and versatile. Based on the proposed LAM, we re-factor the skip connections in the raw U-Net and design a Multi-stage Attention ResU-Net (MAResU-Net) for semantic segmentation from fine-resolution remote sensing images. Experiments conducted on the Vaihingen dataset demonstrated the effectiveness and efficiency of our MAResU-Net. Open-source code is available at https://github.com/lironui/Multistage-Attention-ResU-Net.
I. INTRODUCTION
The paper addresses the quadratic resource demands of dot-product attention on large inputs by proposing a linear attention mechanism and integrating it into a multi-stage ResU-Net for fine-resolution remote sensing segmentation.
- Dot-product attention has O(N^2) memory and computational complexity, hindering global-dependency modeling for large-scale videos, long sequences, and fine-resolution images.
- MAResU-Net uses a ResNet-34 encoder and replaces raw U-Net skip connections with attention blocks at multiple stages to refine multi-scale feature maps.
- The design targets insufficient utilization of abundant encoder-decoder information in U-Net segmentation.
- Experiments on the Vaihingen semantic segmentation dataset demonstrate the proposed network’s effectiveness.
- The proposed Linear Attention Mechanism reduces dot-product attention complexity from O(N^2) to O(N).
II. RELATED WORKS
Related work covers attention mechanisms for global dependencies and scaling attention for feature reweighting, while this Letter focuses on dot-product attention.
- Attention mechanisms have been applied broadly in computer vision and natural language processing, including non-local modules and Transformer architectures.
- Dot-product attention models global dependencies, whereas scaling attention reinforces informative features and suppresses information-lacking features through pooling.
- The Letter distinguishes scaling attention from dot-product attention and focuses on the latter.
C. Semantic Segmentation
The methodology frames attention as global feature aggregation, then replaces its quadratic position-pair computation with a linear formulation based on first-order Taylor approximation.
- Encoder-decoder segmentation networks use an encoder for multi-level feature maps and a decoder to incorporate them into final predictions.
- The proposed linear attention replaces conventional softmax attention with a first-order Taylor approximation to obtain linear time and memory complexity.
- Queries and keys are generated by projecting input features through learned matrices, with N equal to the spatial size H×W.
- Dot-product attention computes each position by aggregating value features from all positions using pairwise similarities as weights.
- The N×N product of queries and transposed keys causes O(N^2) memory and computational complexity, limiting large-input applications.
B. Generalization of Dot-Product Attention
The paper generalizes dot-product attention through feature-map similarities, enabling algebraic reordering that avoids constructing an N×N attention matrix.
- A general normalization function rewrites attention similarities as products of query and key feature maps.
- When the feature maps are exponential functions, the generalized formulation is equivalent to conventional softmax attention.
- The generalized equations provide the reformulation used to simplify dot-product attention.
- Reordering the computation produces a K V^T product of size D_k×D_v instead of an N×N matrix, substantially reducing complexity.
C. Linear Attention Mechanism
The proposed LAM approximates dot-product attention using a first-order Taylor expansion and achieves linear time and memory complexity. Normalization supports the approximation's nonnegativity requirement, while reuse across queries enables O(N) complexity.
- LAM is conceived from a first-order Taylor expansion approximation of dot-product attention.
- The approximation requires l2 normalization of q_i and k_j to ensure q_i^T k_j ≥ −1 and support nonnegativity.
- The resulting formulation is successively rewritten and vectorized for efficient computation.
- O(N) time and memory complexity is obtained by reusing the computed term for every query.
D. Multi-stage Attention ResU-Net
The attention block uses LAM to capture global context while applying dot-product attention across channels. MAResU-Net combines multi-scale encoder and decoder features through attention blocks at multiple stages using ResNet backbones.
- The attention block uses LAM for spatial processing and dot-product attention for channels because C is normally much smaller than N.
- MAResU-Net combines low-level and high-level feature maps through attention blocks at multiple stages.
- ResNet18 and ResNet-34 are selected as concise backbones instead of more complicated architectures such as ResNet-101.
A. Dataset
The Vaihingen dataset provides fine-resolution remote-sensing imagery with near-infrared, red, green, and DSM channels. The experiments use specified training, validation, and test tile splits, excluding DSM, and evaluate segmentation with OA, mIoU, and F1.
- The Vaihingen dataset contains 33 tiles averaging 2494 × 2064 pixels at 5 cm GSD.
- The dataset provides near-infrared, red, green, and DSM channels, but DSM is excluded from the experiments.
- The experiment assigns 15 images for training, one for validation, and 17 for testing.
- Overall accuracy, mean Intersection over Union, and F1-score are used as evaluation indexes.
- MAResU-Net with ResNet-34 achieves a mean F1-score of 90.277 %, OA of 90.860%, and mIoU of 83.301%.
1) Performance Comparison:
MAResU-Net shows stronger classification accuracy than the compared contextual and multi-scale feature aggregation methods on Vaihingen. Statistical testing and ablation analysis further examine the reported performance differences and attention blocks.
- Performance Comparison:: MAResU-Net prevails over PSPNet, DANet, ResUNet-a, and EaNet in the reported Vaihingen comparison.
- Performance Comparison:: The visual comparison reports clear advantages for MAResU-Net over other benchmarks.
- Statistical Significance:: The Kappa z-test evaluates pairwise statistical significance using Kappa coefficients and their variances.
- Statistical Significance:: MAResU-Net's classification accuracy is statistically higher than other comparative methods in the reported analysis.
- Ablation Study:: The ablation study measures the impact of individual multi-stage attention blocks on classification performance.
3) Ablation Study:
The ablation study presents MAResU-Net’s architecture and reports computational efficiency, while indicating stronger contributions from low-level than high-level attention blocks.
- Ablation findings: Low-level attention blocks contribute more than high-level attention blocks in the ablation study.
- Efficiency: MAResU-Net’s training time, complexity, and parameter counts demonstrate the computational efficiency of the design.
- Overall design: The proposed MAResU-Net reconstructs U-Net skip connections with linear attention and achieves classification accuracy and computational efficiency on Vaihingen.