PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images

Libo Wang, Rui Li, Dongzhi Wang, Chenxi Duan, Teng Wang, Xiaoliang Meng

arXiv:2106.12413v2cs.CV

TL;DR

VFR urban scene segmentation must handle extensive detail and object-scale and appearance variation, while existing convolutional approaches have limited global-context modeling. BANet combines Transformer-based dependency features with convolutional texture features and fuses them using linear attention. It demonstrates effectiveness on three urban segmentation datasets, with reported gains on Vaihingen and Potsdam and a 64.6% mIoU on UAVid.

  • Problem

    VFR segmentation remains challenging because convolutional networks extract local patterns but lack the ability to model global context needed for confusing urban objects.

  • Method

    BANet uses a Transformer dependency path, a convolutional texture path, and a linear-attention feature aggregation module to fuse their features.

  • Results

    BANet achieves the highest OA of 90.48% and mIoU of 81.35% on Vaihingen, and 91.06% OA and 86.25% mIoU on Potsdam.

  • Takeaways & Limitations

    Experiments on ISPRS Vaihingen, ISPRS Potsdam, and UAVid demonstrate the effectiveness of combining Transformer and convolutional paths for VFR urban scene segmentation.

  • Takeaways & Limitations

    The paper identifies applying the convolution–Transformer hybrid structure to a wider range of urban applications as future work.

Abstract

from arXiv · show

Semantic segmentation from very fine resolution (VFR) urban scene images plays a significant role in several application scenarios including autonomous driving, land cover classification, and urban planning, etc. However, the tremendous details contained in the VFR image, especially the considerable variations in scale and appearance of objects, severely limit the potential of the existing deep learning approaches. Addressing such issues represents a promising research field in the remote sensing community, which paves the way for scene-level landscape pattern analysis and decision making. In this paper, we propose a Bilateral Awareness Network which contains a dependency path and a texture path to fully capture the long-range relationships and fine-grained details in VFR images. Specifically, the dependency path is conducted based on the ResT, a novel Transformer backbone with memory-efficient multi-head self-attention, while the texture path is built on the stacked convolution operation. Besides, using the linear attention mechanism, a feature aggregation module is designed to effectively fuse the dependency features and texture features. Extensive experiments conducted on the three large-scale urban scene image segmentation datasets, i.e., ISPRS Vaihingen dataset, ISPRS Potsdam dataset, and UAVid dataset, demonstrate the effectiveness of our BANet. Specifically, a 64.6% mIoU is achieved on the UAVid dataset. Code is available at https://github.com/WangLibo1995/GeoSeg.

1. Introduction

VFR urban scene segmentation is difficult because local-pattern methods struggle to model global context amid varied urban objects. BANet addresses this with parallel texture and dependency paths, fused through linear attention, and is evaluated across three large-scale datasets.

  • Motivation: FCN-based networks achieve strong urban scene-labeling performance but remain limited for VFR segmentation because DCNNs lack global-context modeling.Dilated convolutions and attention-based networks have been investigated to capture larger receptive fields and long-range relationships.
  • Transformer background: Transformers capture long-range dependencies by converting image patches into sequences processed by stacked transformer blocks.The standard block combines multi-head self-attention, layer normalization, multilayer perceptrons, and addition operations.
  • Proposed approach: BANet combines stacked convolutions for textural features with Transformer blocks for long-range dependent features in two feature-extraction paths.The bilateral structure is designed to capture fine-grained texture and long-range dependencies simultaneously.
  • Proposed approach: BANet uses a feature aggregation module with linear attention to fuse texture and dependency features while reducing fitting residuals.The module is intended to strengthen the generalization capability of the fused features.
  • Evaluation: The method is experimentally compared on the UAVid, ISPRS Vaihingen, and ISPRS Potsdam semantic-segmentation datasets.The paper organizes its evaluation around three large-scale urban scene image segmentation datasets.

2. Bilateral Awareness Network

BANet combines Transformer-based dependency extraction with convolutional texture extraction, then fuses both feature types through a feature aggregation module using linear attention.

  • Bilateral architecture: BANet feeds the input simultaneously into a dependency path and a texture path to capture long-range dependencies and textural information.The dependency path uses Transformer blocks, while the texture path uses stacked convolution layers.
  • Texture path: The texture path applies four convolutional layers with batch normalization and ReLU, downscaling spatial features by 8 while producing 128 channels.Its convolutions preserve textural information through lightweight local feature extraction.
  • Dependency path: The dependency path uses ResT-Lite with a stem block, patch embedding, and efficient Transformer blocks across four stages.Stages 2–4 additionally perform patch embedding, producing long-range dependent features LDF3 and LDF4.
  • Dependency path: EMSA reduces attention cost by applying depth-wise convolution to K and V before attention, with reduction ratios r of 8, 4, 2, and 1 across the four stages.This compresses the key and value resolutions to reduce computation and memory relative to standard multi-head self-attention.
  • Feature aggregation: The feature aggregation module merges LDF3 and LDF4 with attentional embedding, concatenates the upsampled result with TF, and applies linear attention.The resulting fused feature is passed to a segmentation head, while linear attention has O(N) memory and time complexity.
  • Feature aggregation: Feature fusion yields a segmentation representation containing both long-range dependency and textural information, while linear attention reduces fitting residual and strengthens generalization.The module is designed to combine complementary features for precise urban-scene segmentation.

3. Experiments

BANet is evaluated on three publicly available urban-scene segmentation datasets, comparing against remote-sensing and natural-image models under a parameter-matched backbone setting.

  • Experimental setup: Experiments cover the ISPRS Vaihingen, ISPRS Potsdam, and UAVid semantic segmentation datasets.The study compares BANet with state-of-the-art remote-sensing models and methods designed for natural images.
  • Experimental setup: ResT-Lite with 10.49 M parameters is compared against ResNet-18 with 11.7 M parameters as the comparative backbone.The backbone selection is described as a fair comparison setting.

3.1. Experiments on the ISPRS Vaihingen and Potsdam datasets

BANet is tested on the Vaihingen and Potsdam VFR urban datasets using standard segmentation metrics and visual comparisons, achieving the highest reported OA and mIoU on both.

  • Datasets: Vaihingen contains 33 images averaging 2494 × 2064 pixels at 9 cm GSD, while Potsdam contains 38 images of 6000 × 6000 pixels at 5 cm GSD.Vaihingen experiments use near-infrared, red, and green channels; Potsdam experiments use red, green, and blue channels.
  • Evaluation metrics: The Potsdam evaluation uses overall accuracy, mean Intersection over Union, and F1 score computed from the accumulated confusion matrix.OA is calculated across all categories, including the background.
  • Experimental results: 90.48% OA and 81.35% mIoU are achieved on Vaihingen, while Potsdam reaches 91.06% OA and 86.25% mIoU.These are the highest reported values in the comparison, exceeding the suboptimal methods by more than 0.4% and 1.7% on Vaihingen, and 1.1% and 1.8% on Potsdam.
  • Experimental results: 86.76% F1-score is achieved for the difficult Vaihingen Car category, exceeding the suboptimal method by more than 5.5%.The Car category is described as difficult because it contains relatively small objects.
  • Experimental results: BANet preserves complex Low vegetation contours completely and produces smoother Building outlines than the comparative methods in qualitative results.The dependency path captures long-range global relations, while the texture path uses spatial details in feature maps.

3.2. Experiments on the UAVid Dataset

BANet is evaluated on UAVid, a high-resolution and challenging urban-street benchmark, using IoU-based assessment and quantitative and qualitative comparisons.

  • Dataset: UAVid contains 420 urban-street images at 3840 × 2160 resolution, with 200 for training, 70 for validation, and 150 for testing.The dataset is challenging because of its large image resolution, scale variation, and scene complexity.
  • Dataset: UAVid training uses 1024 × 1024 patches and batch size 4, while retaining the other experimental hyperparameters and augmentations.The stated augmentations are those used for the ISPRS experiments.
  • Evaluation metrics: UAVid performance is assessed by the official server using the intersection-over-union metric.The accompanying formulation uses class-level true-positive, false-positive, true-negative, and false-negative quantities.
  • Experimental results: BANet obtains the best IoU on five of eight classes and the best mIoU, with a 3% gain over the suboptimal BiSeNet.Qualitative results are reported for both the validation and test sets.
  • Experimental results: Compared with MSD, BANet recognizes all moving cars in an example where MSD misidentifies a car crossing the street as static.The comparison is presented as evidence that BANet captures cues to scene semantics more effectively than MSD in that example.

4. Discussion

Ablation experiments on ISPRS Potsdam show that BANet benefits from its texture path, feature aggregation module, and ResT-Lite backbone, while retaining real-time inference speed. The paper positions the framework for remotely sensed urban-scene applications and identifies broader hybrid-architecture exploration as future work.

  • Texture path: Texture-path fusion improves OA by at least 0.2% over the dependency path alone.Both summation and concatenation of texture information improve performance.
  • Feature aggregation module: BANet improves OA by more than 0.5% over summation and concatenation fusion schemes.This ablation supports the proposed feature aggregation module for combining dependency and texture features.
  • ResT-Lite backbone: Replacing ResNet with ResT-Lite improves OA by more than 1%, while BANet exceeds BAResNet by 1.2% OA.The comparison also reports 15.44 million parameters and 33.2 FPS for BANet versus 14.77 million parameters and 73.2 FPS for BAResNet; both meet the stated real-time threshold of at least 30 FPS, although the authors note that the comparison is not completely fair because GPU optimization favors CNNs.
  • Application scenarios: The method targets urban-scene segmentation from satellite, aerial, and UAV imagery and is presented as applicable to semantic segmentation, object detection, and change detection.The authors cite illegal land-use detection, real-time traffic monitoring, and urban environmental assessment as potential applications.
  • Future work: The authors plan to study convolution–Transformer hybrid structures across a wider range of urban applications.

5. Conclusions

The paper concludes that BANet combines Transformer-based dependency modeling, convolutional texture extraction, and attentional feature aggregation for VFR urban-scene segmentation. Experiments on three datasets demonstrate its effectiveness, and the authors present the hybrid design as a basis for further remote-sensing research.

  • Conclusions: BANet uses a dependency path for long-range relationships and a texture path for fine-grained details in VFR images.The dependency path uses a Transformer backbone, while the texture path uses stacked convolution operations.
  • Conclusions: An attentional feature aggregation module fuses global relationship information with spatial texture information.
  • Conclusions: Experiments on the ISPRS Vaihingen, ISPRS Potsdam, and UAVid datasets demonstrate BANet's effectiveness.
  • Conclusions: The authors describe BANet as a novel exploration combining Transformer and convolution in a bilateral structure for remote sensing.They envisage that it could inspire further investigation of Transformer applications in the remote-sensing domain.
Loading 2106.12413v2…