Source-linked AI summary
UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery
Libo Wang, Rui Li, Ce Zhang, Shenghui Fang, Chenxi Duan, Xiaoliang Meng, Peter M. Atkinson
TL;DR
Remote-sensing urban segmentation must combine local detail with global context while remaining efficient for real-time use. The paper proposes UNetFormer, a UNet-like model with a lightweight CNN encoder and Transformer decoder using global-local attention. It reports effective and efficient real-time urban segmentation, with state-of-the-art performance also obtained using a Swin Transformer encoder on Vaihingen.
Problem
CNNs capture local information but are limited in modelling global context, while Transformer-based global modelling can be computationally expensive for real-time urban applications.
Method
UNetFormer combines a lightweight ResNet18 encoder with a Transformer-based decoder containing global-local attention and feature-refinement components.
Results
The method demonstrates effectiveness and efficiency for real-time urban segmentation, with state-of-the-art performance also achieved on Vaihingen using a Transformer-based encoder.
Takeaways & Limitations
A Transformer-based decoder can provide an efficient UNet-like structure for remotely sensed urban-scene segmentation while supporting strong benchmark performance.
Takeaways & Limitations
The paper identifies the potential and feasibility of Transformers for geospatial applications as an area for future research.
Abstract
from arXiv · showhide
Semantic segmentation of remotely sensed urban scene images is required in a wide range of practical applications, such as land cover mapping, urban change detection, environmental protection, and economic assessment.Driven by rapid developments in deep learning technologies, the convolutional neural network (CNN) has dominated semantic segmentation for many years. CNN adopts hierarchical feature representation, demonstrating strong capabilities for local information extraction. However, the local property of the convolution layer limits the network from capturing the global context. Recently, as a hot topic in the domain of computer vision, Transformer has demonstrated its great potential in global information modelling, boosting many vision-related tasks such as image classification, object detection, and particularly semantic segmentation. In this paper, we propose a Transformer-based decoder and construct a UNet-like Transformer (UNetFormer) for real-time urban scene segmentation. For efficient segmentation, the UNetFormer selects the lightweight ResNet18 as the encoder and develops an efficient global-local attention mechanism to model both global and local information in the decoder. Extensive experiments reveal that our method not only runs faster but also produces higher accuracy compared with state-of-the-art lightweight models. Specifically, the proposed UNetFormer achieved 67.8% and 52.4% mIoU on the UAVid and LoveDA datasets, respectively, while the inference speed can achieve up to 322.4 FPS with a 512x512 input on a single NVIDIA GTX 3090 GPU. In further exploration, the proposed Transformer-based decoder combined with a Swin Transformer encoder also achieves the state-of-the-art result (91.3% F1 and 84.1% mIoU) on the Vaihingen dataset. The source code will be freely available at https://github.com/WangLibo1995/GeoSeg.
1. Introduction
Urban-scene segmentation needs both fine local detail and global context, but existing CNN and Transformer designs impose complementary accuracy–efficiency limitations. UNetFormer addresses this with a lightweight CNN encoder and a Transformer-based decoder combining global-local attention and feature refinement.
- CNNs capture fine-grained local context but fixed receptive fields limit global-context and long-range-dependency modelling.
- Transformers can model global context, but their computational time and memory demands restrict efficient real-time urban applications.
- UNetFormer combines a CNN-based encoder with a specifically designed Transformer-based decoder for efficient urban-scene segmentation.
- The model uses ResNet18, a global-local Transformer block, and a feature refinement head to capture context and fuse spatial details with semantic information.The global-local block uses attention and convolutional branches, while the refinement head further refines feature maps.
- UNetFormer exceeds state-of-the-art lightweight networks on four public remotely sensed urban-scene datasets while targeting an accuracy–efficiency trade-off.
2. Related work
Related work progresses from CNN encoder–decoder models toward attention and Transformer-based methods, seeking stronger global-context modelling without sacrificing efficiency. UNetFormer instead places a lightweight Transformer decoder over a CNN encoder to address the computational cost of Transformer encoders while retaining global and local context.
- CNN-based semantic segmentation methods: UNet improves encoder–decoder segmentation through symmetric contracting and expanding paths that extract hierarchical features and restore spatial resolution.
- CNN-based semantic segmentation methods: CNN encoder–decoder methods became standard for remote-sensing segmentation but limited receptive fields hinder global information modelling.FCN's oversimplified decoder can also produce coarse-resolution segmentation, limiting fidelity and accuracy.
- Global context modelling: Attention modules improve contextual representation, but convolution-heavy designs can restrict global features and single modules cannot model global information across decoder semantic levels.
- Transformer-based methods: Transformer methods strengthen global-context extraction for vision and remote-sensing tasks, but Transformer encoders have higher complexity because self-attention scales quadratically.
- Transformer-based methods: UNetFormer uses a CNN encoder and Transformer decoder, selecting ResNet18 and a dual-branch global-local attention mechanism for efficient context modelling.The dual branches capture global and local contexts rather than only global contexts.
3. Methodology
UNetFormer combines a lightweight CNN encoder with a Transformer decoder that models global and local context while refining spatially detailed features. Its hierarchical design targets efficient, precise segmentation of complex urban scenes.
- Architecture: UNetFormer uses a CNN-based encoder and Transformer-based decoder with three global-local Transformer blocks and a feature refinement head.The decoder is designed to capture global and local information hierarchically while remaining lightweight.
- Design rationale: The method addresses the trade-off between accuracy and efficiency by combining global-local modeling with effective feature refinement.The design targets precise segmentation of complex urban scenes while maintaining high efficiency.
- Global-local Transformer block: Cross-shaped window context interaction captures cross-window relations with high computational efficiency without adding a shifted Transformer block.Horizontal and vertical pooling establish long-range dependencies among windows.
- Feature refinement head: The feature refinement head narrows the semantic gap between deep coarse features and shallow spatially detailed features.It first performs weighted fusion, then applies channel-wise and spatial-wise attention, post-processing, upsampling, and a residual connection.
4.1 Experimental settings
The experiments evaluate urban-scene segmentation across UAVid, Vaihingen, Potsdam, and LoveDA using accuracy and efficiency metrics. The settings cover diverse resolutions, modalities, classes, data splits, augmentations, and lightweight or Transformer-based baselines.
- Datasets: LoveDA contains 5987 0.3 m optical images across seven land-cover categories, urban and rural scenes, and three Chinese cities.Its multi-scale objects, complex backgrounds, and inconsistent class distributions create evaluation challenges.
- Evaluation: Experiments measure OA, mean F1, mIoU, Flops, FPS, memory footprint, and model parameters.These metrics jointly assess segmentation accuracy, computational complexity, speed, and memory requirements.
- Baselines: The comparison includes CNN-based lightweight, attentional, and general segmentation networks alongside hybrid and fully Transformer-based models.The benchmark set spans methods with CNN or Transformer encoders and decoders.
4.2 Ablation study
Ablations show that global-local attention, cross-shaped window interaction, feature refinement, and the hybrid encoder-decoder design improve segmentation accuracy and efficiency.
- Each component of UNetFormer: 3.4% mIoU is added on the UAVid validation set by deploying GLTB.The cross-shaped window context interaction module contributes 1.0%.
- Each component of UNetFormer: 1.0% or more mIoU is gained by adding the feature refinement head.This result supports the validity of the feature refinement module.
- Efficient global-local attention: 70.0% mIoU is achieved by the proposed global-local attention on the UAVid validation set, with advantages in complexity, memory, parameters, and speed.It is more accurate and faster than shifted window attention and efficient multi-head self-attention.
- Network stability: Less than 0.7% mIoU deviation is observed across different input sizes.The 1024×1024 input performs best, while 2048×2048 can reduce IoU for the small-object human class.
- Encoder-decoder combinations: 4.3% mIoU improvement over U-Net and 80% lower computational complexity than SwinUNet demonstrate the hybrid design’s efficiency.The UNetFormer also provides a competitive accuracy-speed trade-off against other encoder-decoder combinations.
4.3 Experiment results
Across UAVid, Vaihingen, and Potsdam, UNetFormer achieves strong segmentation accuracy while remaining competitive in speed and complexity against lightweight CNN, Transformer, and hybrid networks.
- UAVid: 2.0% higher mIoU than CoaT is achieved while running 10 times faster.The comparison evaluates accuracy and efficiency on the UAVid test set.
- UAVid: 67.8% mIoU is obtained on the official UAVid test set, with improved results against ABCNet, BANet, and BoTNet.The reported margins are 4.0%, 3.2%, and 4.6% in mIoU, respectively.
- Potsdam: 92.8% mean F1 and 86.8% mIoU are achieved on the Potsdam test set.The results exceed the compared CNN-based and Transformer-based lightweight networks.
10. Remarkably, the UNetFormer obtains the highest mIoU (52.4%) with the least complexity
UNetFormer achieves the highest mIoU on LoveDA while maintaining the lowest complexity and fastest speed among the compared networks.
- LoveDA: 52.4% mIoU is achieved with the least complexity and fastest speed on the LoveDA test set.The method handles both urban and rural scenes well.
5. Discussion
The discussion attributes UNetFormer’s performance to combining global and local context in a hybrid architecture, while noting trade-offs of local-window attention and Transformer encoders.
- Global-local context: Global-local context combines local detail with the spatial consistency of global context.Local context preserves abundant local features but lacks spatial consistency, whereas global context has the opposite trade-off.
- Global-local context: Cross-shaped window context interaction addresses jagged edges caused by self-attention within non-overlapping local windows.It enhances interaction between windows during context aggregation.
- Hybrid architecture: The hybrid CNN-encoder and Transformer-decoder structure provides a superior accuracy-efficiency trade-off.The decoder uses cross-shaped interaction instead of shift-window attention to capture cross-window relationships more efficiently.
- Hybrid architecture: Real-time UAV image processing for urban environmental perception and monitoring is identified as a potential application.This follows from the reported trade-off between segmentation accuracy and efficiency.
- Fully Transformer-based exploration: Transformer encoders are unsuitable for real-time applications despite advantages in pursuing high precision.Their speed limitation motivates retaining a lightweight CNN encoder for real-time segmentation.
- Fully Transformer-based exploration: 91.3% F1 and 84.1% mIoU are achieved by FT-UNetFormer on the Vaihingen test set.The model uses the proposed Transformer decoder with a Swin-Base encoder and exceeds other networks by at least 0.3% in F1.
6. Conclusion
The paper presents UNetFormer, a UNet-like Transformer for efficient remote-sensing urban-scene segmentation, combining a lightweight CNN encoder with a Transformer decoder that models global and local context. Benchmark experiments and further encoder exploration support its effectiveness and efficiency for real-time segmentation and its state-of-the-art performance on Vaihingen.
- Method: UNetFormer combines a lightweight CNN-based encoder with a Transformer-based decoder for efficient semantic segmentation of remotely sensed urban scenes.The architecture is designed as a UNet-like Transformer for real-time urban-scene segmentation.
- Method: The global-local Transformer block models both global and local contexts, while the feature refinement head optimizes the extracted context.These components form the decoder and refine its global-local representations.
- Results: Benchmark experiments and ablation studies on Vaihingen, Potsdam, UAVid, and LoveDA demonstrated the method’s effectiveness and efficiency for real-time urban segmentation.The experiments cover multiple remote-sensing datasets and include component ablations.
- Results: The Transformer-based decoder also performed well when paired with a fully Transformer-based encoder, achieving state-of-the-art performance on the Vaihingen dataset.This further explores the decoder’s compatibility beyond the lightweight CNN-based encoder setting.
- Future work: Future work will continue exploring the potential and feasibility of Transformers for geospatial applications.The conclusion identifies continued investigation of Transformer use in geospatial settings as future research.
Declaration of Competing Interest
The authors report no known competing financial interests or personal relationships that could have influenced the reported work.
- Declaration: The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.This statement covers both financial interests and personal relationships.