PaperlayerView exact revision on arXiv

Source-linked AI summary

ABCNet: Attentive Bilateral Contextual Network for Efficient Semantic Segmentation of Fine-Resolution Remote Sensing Images

Rui Li, Chenxi Duan

arXiv:2102.02531v1cs.CV

TL;DR

Lightweight networks face a substantial accuracy gap because they have limited global-context extraction, while dot-product attention scales quadratically with input spatio-temporal size. ABCNet proposes an efficient semantic-segmentation approach, achieving 91.095% overall accuracy on Potsdam at 72.13 FPS on an 1660Ti graphics card.

  • Problem

    Lightweight networks have limited global-context extraction, creating a substantial accuracy gap relative to state-of-the-art models, while dot-product attention has quadratic input-size consumption.

  • Method

    ABCNet is proposed as a novel approach for efficient semantic segmentation of fine-resolution remote sensing imagery.

  • Results

    91.095% overall accuracy was obtained on the Potsdam test dataset at 72.13 FPS on a mid-range 1660Ti graphics card.

  • Takeaways & Limitations

    The reported results support the effectiveness and efficiency of ABCNet.

  • Takeaways & Limitations

    Dot-product attention consumption increases quadratically with the input spatio-temporal size, limiting its fit with lightweight-network goals.

Abstract

from arXiv · show

Semantic segmentation of remotely sensed images plays a crucial role in precision agriculture, environmental protection, and economic assessment. In recent years, substantial fine-resolution remote sensing images are available for semantic segmentation. However, due to the complicated information caused by the increased spatial resolution, state-of-the-art deep learning algorithms normally utilize complex network architectures for segmentation, which usually incurs high computational complexity. Specifically, the high-caliber performance of the convolutional neural network (CNN) heavily relies on fine-grained spatial details (fine resolution) and sufficient contextual information (large receptive fields), both of which trigger high computational costs. This crucially impedes their practicability and availability in real-world scenarios that require real-time processing. In this paper, we propose an Attentive Bilateral Contextual Network (ABCNet), a convolutional neural network (CNN) with double branches, with prominently lower computational consumptions compared to the cutting-edge algorithms, while maintaining a competitive accuracy. Code is available at https://github.com/lironui/ABCNet.

1. INTRODUCTION

Fine-resolution remote sensing segmentation requires both detailed spatial information and broad contextual information, but their extraction can be computationally expensive. ABCNet addresses this trade-off with a bilateral architecture that combines spatial detail preservation and linear-attention-based context modeling, achieving competitive accuracy and speed.

  • Fine-resolution remote sensing segmentation supports precision agriculture, environmental protection, economic assessment, and automatic Earth monitoring.
  • Global context and large-scale feature maps improve segmentation but are computationally expensive, while dot-product attention additionally has quadratic memory and computation with input spatio-temporal size.
  • Lightweight networks reduce computation, but limited global-context capacity creates an accuracy gap relative to state-of-the-art models on fine-resolution imagery.
  • ABCNet uses linear attention to reduce attention complexity from O(N^2) to O(N), targeting improved segmentation accuracy with maintained efficiency.
  • Its bilateral design separates a spatial path for fine details from a contextual path for global information, then fuses their features with a feature aggregation module.
  • ABCNet reports competitive results on the ISPRS Vaihingen and Potsdam datasets, including 91.095% overall accuracy and 72.13 FPS on Potsdam using an 1660Ti graphics card.

1) Context information extraction

Semantic segmentation methods seek contextual information through dilated convolution, encoder–decoder structures, spatial pyramid pooling, and attention. Dot-product attention captures global dependencies but its quadratic resource demand conflicts with lightweight processing.

  • Dilated convolution enlarges receptive fields without shrinking spatial resolution, while encoder–decoder architectures merge high- and low-level features through skip connections.
  • Spatial pyramid pooling uses multiscale convolutions, but large kernels increase parameters while small kernels can weaken adjacent-feature connections and cause gridding.These limitations are described for standard-convolution SPP and small-kernel variants such as ASPP.
  • The input feature is represented as N=H×W positions with C channels, then projected to query, key, and value matrices for attention processing.The query and key dimensions must match, and the normalization function ρ is applied within the attention formulation.
  • Attention aggregates value features from all positions using pairwise similarities as weights, with softmax commonly normalizing the query–key scores.
  • Dot-product attention models similarities between positions to extract global dependencies and has been adapted from machine translation for computer vision.The mechanism and related variants have been applied to semantic segmentation by multiple attention-based networks.
  • Dot-product attention forms an N×N query–key product, resulting in O(N^2) memory and computation complexity for large inputs.This resource demand hinders applying attention to large images and conflicts with lightweight-network goals.

3) Generalization and simplification of the dot-product attention mechanism

The paper generalizes dot-product attention by expressing similarity through feature mappings, then reorders matrix multiplication to avoid explicitly forming the N×N attention matrix. This reduces complexity while retaining competitive performance when suitable mappings are used.

  • Similarity between query and key vectors can be expressed as ϕ(q_i)^Tφ(k_j), allowing the attention formulation to be generalized beyond softmax.When both mappings are exponential, the generalized form is equivalent to the softmax-based formulation.
  • The generalized attention output is written in vectorized form using mapped queries, mapped keys, values, and a normalization term.
  • Replacing softmax similarity with mapped features permits commutative reordering, computing φ(K)^T V before multiplying by the query representation.This avoids multiplication between the reshaped key and query matrices.
  • The reordered computation has O(d_k d_v) time and O(d_k d_v) space complexity, rather than forming the N×N product.Suitable ϕ and φ mappings enable competitive performance with finite complexity.

4) Linear Attention Mechanism

The linear attention mechanism replaces softmax attention with a first-order Taylor approximation, reducing the dependence on input size while preserving attention computation. Its efficiency is analyzed against dot-product attention across different input sizes.

  • Validation: The proposed attention mechanism's validity and efficiency were supported by extensive ablation experiments and analysis.
  • Linear attention formulation: Linear attention replaces the softmax function with a first-order Taylor expansion approximation.The approximation is introduced as equation (9).
  • Linear attention formulation: L2 normalization of q_i and k_j guarantees that the approximation remains nonnegative.
  • Linear attention efficiency: The reformulated attention computation can calculate and reuse a term for each query, reducing repeated computation.The passage describes vectorization and reuse of the key-dependent term.
  • Efficiency analysis: The comparison evaluates computation and memory requirements for linear and dot-product attention under different input sizes.The calculation assumes C = D_v = 2D_k = 64, and the figure uses a log scale.

5) Efficient semantic segmentation

Efficient semantic segmentation must preserve fine spatial details while capturing large-receptive-field context. ABCNet addresses this tension with a bilateral architecture that separates spatial-detail extraction from contextual processing and uses efficient attention.

  • Motivation: Real-time semantic segmentation requires efficient models, but input downsampling reduces resolution and loses image details.The motivation specifically identifies real-time operation as at least 30 FPS in applications such as autonomous driving.
  • Design rationale: The architecture is motivated by the difficulty of reconciling abundant spatial details with a large receptive field at efficient computational cost.Large-kernel spatial pyramid pooling can increase computation and memory demand.
  • Bilateral architecture: ABCNet uses a bilateral architecture with a spatial path for low-level details and a contextual path for high-level semantic features.The spatial path preserves larger feature-map resolution, while the contextual path extracts global context.
  • Spatial path: The spatial path uses three stride-2 convolutional layers, producing feature maps at 1/8 of the original image size.These feature maps retain abundant spatial details because of their relatively large spatial size.
  • Contextual path: The contextual path uses a lightweight ResNet-18 backbone, attention enhancement modules, and linear attention to encode global context efficiently.The path downsamples feature maps, applies two AEMs, and uses linear attention to capture long-range dependencies.

3) Feature aggregation module

ABCNet's feature aggregation module combines complementary spatial and contextual representations that occupy different feature domains. The module is designed to balance segmentation accuracy and efficiency rather than relying on simple summation or concatenation.

  • Feature complementarity: Spatial and contextual paths produce complementary features, with spatial details from one path and semantic context from the other.
  • Fusion motivation: Because the two feature types belong to different domains, simple summation and concatenation are not considered appropriate fusion strategies.
  • FAM design: The feature aggregation module merges both representations with explicit consideration of accuracy and efficiency.FAM first concatenates the path outputs, then applies convolution, batch normalization, and ReLU to balance feature scales.
  • Loss function: The network uses a principal cross-entropy loss for the final output and two auxiliary focal losses on the contextual path.The focal-loss focusing parameter γ is set to 2 in the experiments.
  • Experimental setting: ABCNet's effectiveness is evaluated on the ISPRS Potsdam and Vaihingen datasets using only RGB channels for Potsdam.The datasets contain fine-resolution imagery with a 5 cm ground sampling distance.

2) Evaluation Metrics

The evaluation uses standard segmentation metrics, controlled training procedures, and comparisons with contextual, multi-scale, and lightweight segmentation networks. Ablations assess the attention enhancement and feature aggregation components on two datasets.

  • Metrics: Performance is measured using overall accuracy, mean Intersection over Union, and F1 score.The metrics are computed from the accumulated confusion matrix.
  • Comparative evaluation: Comparisons include contextual methods, remote-sensing multi-scale models, and lightweight networks, with test-time rotation and flipping applied to all methods.
  • Ablation study: AEM improves performance by more than 1.5% on both datasets relative to the contextual-path baseline.The baseline uses ResNet-18 without AEM, with its feature maps directly upsampled.
  • Ablation study: The performance gap between simple fusion schemes and FAM supports the validity of the proposed feature aggregation module.

5) The complexity and speed of the network

ABCNet is evaluated for computational complexity and inference speed, with experiments comparing network parameters, operations, and FPS under fixed input settings. It combines competitive Potsdam accuracy with fast inference and large-input handling.

  • The experiments compare parameters and computational complexity across networks, with G denoting billions of floating-point operations and M denoting millions of parameters.Inference speed is measured in frames per second on a midrange notebook graphics card.
  • ABCNet simultaneously balances speed and accuracy, achieving at least 1.79% higher Potsdam mIoU than comparative methods.
  • ABCNet handles a 4096×4096 input, whereas more than half of the comparative methods run out of memory at that resolution.

6) Results on the ISPRS Vaihingen dataset

On the ISPRS Vaihingen dataset, ABCNet delivers statistically strong and competitive segmentation performance while remaining faster than other compared methods. Its results include strong overall accuracy and F1 performance using efficient architectures and TOP imagery.

  • TOP-only ABCNet surpasses lightweight networks while achieving competitive performance against specially designed models.
  • ABCNet exceeds other lightweight networks in mean F1, overall accuracy, and mIoU by a considerable margin.The comparison concerns Vaihingen performance, including the difficult small-object car class.
  • ABCNet achieves an 85.299% F1 score on Vaihingen, at least 4% higher than other methods.
  • The Vaihingen evaluation includes quantitative test-set comparisons and qualitative visualizations of area 38 and enlarged results.
  • ABCNet's Vaihingen accuracy is reported as statistically higher than other comparative methods.The paper uses pairwise Kappa z-tests, treating z values greater than 1.96 as significant at the 95% confidence level.
  • ABCNet is reported as two to seven times faster than the compared state-of-the-art methods while maintaining competitive performance.

7) Results on the ISPRS Potsdam dataset

On ISPRS Potsdam, ABCNet achieves strong segmentation accuracy and favorable statistical comparisons while preserving the speed advantages expected of a lightweight network. The evaluation combines numerical, Kappa z-test, and visualization-based comparisons.

  • Potsdam results include numerical test-set comparisons, Kappa z-tests, and visualizations of area 3_13 with enlarged results.
  • 91.095% overall accuracy and 88.561% mIoU are achieved by ABCNet on the Potsdam dataset.
  • ABCNet's Kappa-z test strongly indicates superiority over other lightweight networks on Potsdam.
  • The ABCNet uses spatial and contextual paths to capture spatial details and global contextual information.An attention enhancement module models long-range dependencies, while feature aggregation merges spatial and contextual features.
  • Extensive experiments on ISPRS Vaihingen and Potsdam demonstrate the proposed network's effectiveness and efficiency.
Loading 2102.02531v1…