PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Attention-Network for Semantic Segmentation of Fine Resolution Remote Sensing Images

Rui Li, Shunyi Zheng, Chenxi Duan, Ce Zhang, Jianlin Su, P. M. Atkinson

arXiv:2009.02130v4eess.IVcs.CV

TL;DR

Semantic segmentation of fine-resolution remote-sensing images is limited by underused multi-scale features, insufficient long-range dependencies, and the quadratic cost of dot-product attention. MANet combines linear-complexity kernel attention with channel attention and a ResNeXt-101 backbone, achieving higher benchmark accuracy across remote-sensing datasets.

  • Problem

    Standard segmentation models underuse multi-scale information, insufficiently model long-range dependencies, and make attention costly for large inputs.

  • Method

    MANet combines kernel attention, channel attention, multi-kernel contextual modeling, and a ResNeXt-101 backbone for fine-resolution remote-sensing segmentation.

  • Results

    MANet outperformed benchmark methods, with gains of 1.977% mIoU and 1.226% F1-score over DANet on Potsdam, and 3.535% mIoU and 3.374% F1-score over PSPNet on GID.

  • Takeaways & Limitations

    Kernel attention provides O(N) complexity while MANet integrates contextual dependencies with local features for improved accuracy and computational efficiency.

Abstract

from arXiv · show

Semantic segmentation of remote sensing images plays an important role in a wide range of applications including land resource management, biosphere monitoring and urban planning. Although the accuracy of semantic segmentation in remote sensing images has been increased significantly by deep convolutional neural networks, several limitations exist in standard models. First, for encoder-decoder architectures such as U-Net, the utilization of multi-scale features causes the underuse of information, where low-level features and high-level features are concatenated directly without any refinement. Second, long-range dependencies of feature maps are insufficiently explored, resulting in sub-optimal feature representations associated with each semantic class. Third, even though the dot-product attention mechanism has been introduced and utilized in semantic segmentation to model long-range dependencies, the large time and space demands of attention impede the actual usage of attention in application scenarios with large-scale input. This paper proposed a Multi-Attention-Network (MANet) to address these issues by extracting contextual dependencies through multiple efficient attention modules. A novel attention mechanism of kernel attention with linear complexity is proposed to alleviate the large computational demand in attention. Based on kernel attention and channel attention, we integrate local feature maps extracted by ResNeXt-101 with their corresponding global dependencies and reweight interdependent channel maps adaptively. Numerical experiments on three large-scale fine resolution remote sensing images captured by different satellite sensors demonstrate the superior performance of the proposed MANet, outperforming the DeepLab V3+, PSPNet, FastFCN, DANet, OCRNet, and other benchmark approaches.

I. INTRODUCTION

Remote-sensing segmentation models face underused multi-scale features, insufficient long-range dependencies, and costly attention on large inputs. MANet addresses these issues with linear-complexity kernel attention, a ResNeXt-101 backbone, and multi-kernel attention.

  • Encoder-decoder models can concatenate low-level details with high-level semantics without refinement, limiting feature exploitation and discrimination.
  • Multi-scale context methods improve feature discrimination but use homogeneous, non-adaptive dependencies and insufficiently leverage long-range relationships.
  • Dot-product attention models long-range dependencies but incurs memory and computational costs that grow quadratically with input size.
  • MANet treats attention as a kernel function to reduce dot-product attention complexity to O(N).
  • The proposed framework combines a ResNeXt-101 backbone with kernel attention and multi-kernel attention to extract refined contextual dependencies.

II. RELATED WORK

Related work develops attention, context-fusion, and encoder-decoder approaches for semantic segmentation. These methods respectively target long-range dependencies, multi-scale context, and integration of encoder and decoder features.

  • Attention mechanisms have achieved strong results across language and vision tasks, including semantic segmentation.
  • Scaling-attention methods reweight channels to reinforce informative features and suppress information-lacking features.
  • DilatedFCN: Dilated convolutions and pyramid modules embed contextual information at multiple scales for dense prediction.
  • DilatedFCN: OCRNet distinguishes contextual pixels associated with the same object class from those associated with different classes.
  • EncoderDecoder: Encoder-decoder methods use skip connections to combine high-level decoder features with corresponding low-level encoder features.

E. Attention-based Networks for Semantic Segmentation

Attention-based segmentation networks explore long-range dependencies, but dot-product attention creates a quadratic resource bottleneck for large feature maps. The paper formulates the attention computation and motivates more efficient alternatives.

  • Several segmentation networks use dot-product attention or its variants to model long-range dependencies.
  • Efficiency research targets attention-matrix generation, block pruning, low-rank reconstruction, and graph-convolutional formulations.
  • Given feature X, projected query, key, and value matrices Q, K, and V are generated for attention computation.
  • The normalization function ρ evaluates similarities between query and key features, whose weighted values are aggregated across positions.
  • The product ρ(QK^T) represents all pairwise positional similarities and requires O(N^2) memory and computation.

B. Generalization of Dot-Product Attention Based on Kernel

The paper generalizes dot-product attention by expressing similarity through kernel feature maps. Reordering the resulting products avoids explicitly forming the N×N attention matrix and reduces computational complexity.

  • Under softmax normalization, each output row is produced by aggregating value features according to query-key similarities.
  • The attention formulation can be generalized from softmax to an arbitrary normalization function.
  • Similarity can be represented as the inner product φ(q_i)^Tφ(k_j), with φ and φ considered kernel smoothers when identical.
  • Reordering kernel-feature and value products avoids constructing the full positional attention matrix.

C. Kernel Attention

The proposed kernel attention rewrites attention computation to avoid explicitly forming the full pairwise attention matrix. By reusing key-dependent terms across queries, the resulting mechanism has linear time and memory complexity.

  • Softplus-based similarity replaces the usual activation to preserve nonzero gradients for negative inputs.The paper states that softplus is selected instead of ReLU because its nonzero property avoids zero gradients when inputs are negative.
  • The rewritten attention expression factors query and key terms so shared key computations can be reused across queries.The derivation rewrites the original attention equation into an equivalent vectorized form.
  • O(N) complexity results because the summed softplus-transformed keys are calculated once and reused for each query.This reduces the time and memory complexity of the proposed linear attention mechanism to O(N).

D. The Structure of the ResNeXt

MANet uses ResNeXt rather than ResNet as its backbone. ResNeXt combines repeated blocks with grouped transformations to increase representational accuracy without increasing model complexity.

  • ResNeXt replaces ResNet as MANet’s backbone and uses repeated blocks that aggregate transformations with a shared topology.The structure is inspired by ResNet and Inception and defines the number of transformations as cardinality.
  • ResNeXt combines ResNet-style block stacking with Inception-style grouping and split-convert-merge transformations.This aggregated transformation strategy is intended to improve discrimination without increasing model complexity.
  • Increasing cardinality provides a measurable way to enhance network representation when deeper or wider designs yield diminishing returns.

E. Multi-Attention-Network

MANet combines kernel and channel attention across multi-level ResNeXt features to model spatial and channel dependencies. It is evaluated on three remote-sensing datasets with different image characteristics and experimental settings.

  • Kernel attention models spatial dependencies, while channel attention models dependencies across feature channels.The paper notes that channel count C is normally much smaller than the number of spatial positions N, making channel attention less costly.
  • The attention block enhances the discriminative ability of feature maps extracted at each network layer.
  • MANet extracts five differently scaled feature maps with ImageNet-pretrained ResNeXt-101, then upsamples, concatenates, and attention-refines them.The final attention output is upsampled to the input image’s spatial resolution.
  • The evaluation uses the ISPRS Potsdam, ISPRS Vaihingen, and fine-resolution Gaofen Image Dataset datasets.The datasets differ in sensors, spatial dimensions, channels, class counts, and train-validation-test partitions.
  • Table III compares methods using input shape, parameter count, and computational complexity.The table denotes parameters in millions and complexity in billions.

B. Evaluation Metrics

MANet is evaluated with six segmentation metrics covering pixel accuracy, class-wise accuracy, agreement, overlap, F1 performance, and frequency-weighted overlap. F1 and FWIoU supplement metrics with known sensitivities to class imbalance.

  • The evaluation reports PA, MPA, K, mIoU, F1, and FWIoU for semantic segmentation performance.These metrics provide multiple views of pixel-level accuracy, class-level performance, agreement, overlap, and frequency weighting.
  • F1 and FWIoU are added because PA can underrepresent minority categories and mIoU can overemphasize them in imbalanced datasets.
  • The metric equations use true positives, false positives, false negatives, and the number of classes as core quantities.

C. Experimental Setting

The experiments compare MANet and benchmark segmentation models under a common implementation setting, while evaluating kernel attention against dot-attention for computational efficiency across input sizes.

  • Experimental setup: MANet is evaluated against U-Net, DeepLab V3, DeepLab V3+, RefineNet, PSPNet, FastFCN, DANet, and OCRNet using PyTorch and Adam optimization.Experiments use a 0.0003 learning rate, batch size 16, cross-entropy loss, and one NVIDIA GeForce RTX 2080ti GPU with 11 GB RAM.
  • Attention efficiency: Dot-attention requires O(D_kN^2) time and O(N^2) space because it forms an N × N position-similarity matrix.This quadratic dependence creates substantial resource demands as spatial input size increases.
  • Attention efficiency: Kernel attention changes the operation order after replacing softmax with kernel smoothers, avoiding direct K–Q multiplication and reducing complexity to O(dD) time and space.The reformulation first computes softplus(K)^T V and then multiplies the result with Q.
  • Efficiency results: For a 64 × 256 × 256 input, kernel attention uses 1/340 the memory and 1/1417 the computation required by dot-attention.Dot-attention requires 17 GB and 829 GMACC, whereas kernel attention uses 51MB and 585 MMACC.

E. Results on ISPRS Datasets

MANet outperforms the compared segmentation methods on the ISPRS Potsdam and Vaihingen datasets, while its multi-level attention models long-range dependencies and supports refined feature representations.

  • Quantitative results: MANet exceeds the other evaluated algorithms across all quantitative evaluation indices on the ISPRS Potsdam and Vaihingen datasets.The comparison includes U-Net, DeepLab-based models, RefineNet, PSPNet, FastFCN, DANet, and OCRNet.
  • Method comparison: DANet and OCRNet perform relatively well because attention mechanisms capture long-range dependencies.Other compared methods obtain gains from atrous spatial pyramid pooling, pyramid pooling, joint pyramid upsampling, or chained residual pooling.
  • MANet design: MANet models long-range dependencies across four scales, supporting refined feature extraction from the input.The design directly models global dependencies across multiple levels rather than relying only on dilated convolutions, pooling, or individual attention.
  • ISPRS Potsdam: MANet’s 1.977% mIoU and 1.226% F1-score gains on Potsdam are measured against DANet.These are reported as observed accuracy increases for the ISPRS Potsdam dataset.
  • ISPRS Vaihingen: MANet’s accuracy increases on Vaihingen are 2.248% and 1.568% compared with OCRNet.The passage reports these increases without assigning metric labels to the two values.

G. Ablation Study

Ablation experiments attribute accuracy gains to MANet’s attention blocks, ResNeXt backbone, and kernel attention, with kernel attention improving multiple baseline architectures on GID.

  • Component ablations: Attention blocks and ResNeXt increase accuracy significantly compared with baseline settings on the ISPRS Potsdam and GID datasets.The ablations separately evaluate the proposed attention block and backbone choices.
  • Kernel-attention ablation: Kernel attention is tested by attaching it to six baseline architectures on the GID dataset.The placements include decoder, path, convolutional, and ResNet-101 backbone components.
  • Kernel-attention ablation: 3.750% mIoU and 2.853% F1-score improvements result when kernel attention is added to U-Net on GID.The passage identifies U-Net as receiving the largest stated improvement because its representation ability is limited.
  • Kernel-attention ablation: Kernel attention improves mIoU by more than 2% for DeepLab V3 and DeepLab V3+ and by around 1% for the other tested baselines.The reported gains are less distinct than U-Net’s but remain significant.
  • Overall conclusion: The MANet combines kernel attention, ResNeXt-101, multi-scale semantic integration, and hierarchical self-attention to improve accuracy and computational efficiency.Experiments on ISPRS and GID report the framework as outperforming eight baseline methods across all accuracy indices.
Loading 2009.02130v4…