PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Segmentation using Vision Transformers: A survey

Hans Thisanke, Chamli Deshan, Kavindu Chamith, Sachith Seneviratne, Rajith Vidanaarachchi, Damayanthi Herath

arXiv:2305.03273v1cs.CVcs.AIcs.LG

TL;DR

Semantic segmentation needs pixel-level dense prediction, but plain ViTs are not directly suited to it and often require substantial data. This survey reviews adapted ViT architectures, compares their benchmark performance, and summarizes their strengths, limitations, and practical data strategies. It identifies high-performing variants across datasets while noting that some architectures remain task-specific and domain-specific data can limit transfer learning.

  • Problem

    Plain ViTs lack segmentation heads and suitable dense prediction outputs, while Transformer training often requires large labeled datasets and domain-matched pretraining.

  • Method

    The survey reviews ViT architectures for semantic segmentation, compares their variants across benchmark datasets, and discusses self-supervised learning for data-limited settings.

  • Results

    The survey identifies highest-performing ViT model variants for each benchmark dataset and reports their mIoU (%) results.

  • Takeaways & Limitations

    ViT architectures provide a powerful alternative to CNNs for semantic segmentation, with adaptations such as pyramidal backbones enabling finer-grained outputs at lower computational cost.

  • Takeaways & Limitations

    Mask2Former, despite being universal for segmentation, must still be trained separately for each specific task.

Abstract

from arXiv · show

Semantic segmentation has a broad range of applications in a variety of domains including land coverage analysis, autonomous driving, and medical image analysis. Convolutional neural networks (CNN) and Vision Transformers (ViTs) provide the architecture models for semantic segmentation. Even though ViTs have proven success in image classification, they cannot be directly applied to dense prediction tasks such as image segmentation and object detection since ViT is not a general purpose backbone due to its patch partitioning scheme. In this survey, we discuss some of the different ViT architectures that can be used for semantic segmentation and how their evolution managed the above-stated challenge. The rise of ViT and its performance with a high success rate motivated the community to slowly replace the traditional convolutional neural networks in various computer vision tasks. This survey aims to review and compare the performances of ViT architectures designed for semantic segmentation using benchmarking datasets. This will be worthwhile for the community to yield knowledge regarding the implementations carried out in semantic segmentation and to discover more efficient methodologies using ViTs.

1. Introduction

The introduction frames Vision Transformers as powerful but data-hungry alternatives to CNNs, while noting that plain ViTs require architectural adaptation for semantic segmentation. The survey addresses this gap by comparing segmentation-specific variants across benchmark datasets.

  • Motivation: Vision Transformers capture long-range dependencies with less inductive bias through multi-head self-attention, but typically require very large training datasets.Self-supervised approaches are identified as a way to alleviate some of these data requirements.
  • Challenge: Plain ViTs cannot directly perform semantic segmentation because they lack segmentation heads and dense prediction outputs.SETR and Swin Transformer architectures are presented as usable alternatives for segmentation.
  • Challenge: Dense prediction is difficult because of intra-class variation, context variation, occlusion ambiguities, and low image resolution.The survey notes that ViT developments continue to address these challenges and improve efficiency.
  • Survey gap: Existing surveys had not compared segmentation models across several benchmark datasets to identify the best-performing model.The survey organizes models by architecture and evaluates variants using common benchmark categories.
  • Survey contribution: The survey reports mIoU (%) across multiple semantic-segmentation benchmarks to identify overall evaluation results and the highest-performing model variant for each dataset.It also examines parameters such as patch size and iteration count for each model variant.
  • Organization: The paper progresses from applications, ViT challenges, datasets, and loss functions to existing ViT segmentation work, quantitative analysis, discussion, and future directions.These topics are covered across Sections 2 through 6.

2. Semantic Segmentation using Vision Transformers

This section reviews semantic segmentation applications, Vision Transformer architectures, and practical responses to dense-prediction and data limitations. It emphasizes patch-wise attention, self-supervised learning, and domain-specific segmentation challenges.

  • Semantic Segmentation: FCN-based methods improve segmentation through fully convolutional feature extraction but suffer from low-resolution outputs and limited long-range dependency modeling.These limitations motivated attention mechanisms and Transformer architectures for computer vision.
  • Vision Transformers: ViT divides images into patches treated as tokens, replacing pixel-wise attention with patch-wise attention to reduce self-attention complexity.Pixel-level self-attention has quadratic cost in the number of pixels because each pixel attends to every other pixel.
  • Vision Transformers: ViT achieved 88.55% accuracy on ImageNet, 90.72% on ImageNet-ReaL, and 94.55% on CIFAR-100, while requiring more training data.The cited experiments report improved performance with increasing data size.
  • Applications: Semantic segmentation supports remote sensing, medical imaging, and video processing, each presenting distinct spatial, annotation, or temporal challenges.Video segmentation must account for temporal context, while medical and remote-sensing segmentation require expert annotation or domain-specific handling.
  • Practical Approaches: Self-supervised learning trains models on pretext tasks and transfers the learned weights to downstream tasks such as semantic segmentation.This approach addresses the difficulty of obtaining large labeled datasets, especially when pretrained data domains differ from target applications.

3. Datasets

The survey organizes semantic-segmentation datasets by environmental conditions and describes representative datasets spanning autonomous driving, urban scenes, synthetic data, and diverse visual domains.

  • Dataset classification: Datasets are classified as cross-domain or no-cross-domain according to whether environmental conditions deviate from normal daytime settings.Rainy, cloudy, nighttime, and snowy conditions are examples of cross-domain data.
  • Dataset descriptions: PASCAL-Context provides manually labeled pixel-level semantic categories covering objects, stuff, and hybrid classes.Its data contain varied objects and regions such as sky and water.
  • Dataset descriptions: ADE20K contains continuously growing scene, object, and object-part annotations, while IDD focuses on Indian road scenes with weather, lighting, and local-class variation.IDD includes classes such as auto-rickshaws and roadside animals.
  • Dataset descriptions: KITTI contains 2D and 3D images from urban and rural traffic scenarios and supports robotics and autonomous-driving research.KITTI-2012 and KITTI-2015 are variants with differences in ground truth.
  • Dataset descriptions: Cityscapes contains large-scale pixel- and instance-level annotations from stereo video sequences recorded across 50 cities in Germany and neighboring countries.The survey characterizes its quality, data size, and annotations as strong relative to other datasets.
  • Synthetic datasets: Virtual KITTI provides automatically labeled synthetic data under varied weather and imaging conditions for pre-training and multiple vision tasks.IDDA extends synthetic autonomous-driving data to 1 million frames across seven CARLA city models and more than 100 visual domains.

4. Meta - analysis

The meta-analysis compares Vision Transformer segmentation models on common benchmark datasets and reviews architectures that adapt Transformers for dense prediction. It highlights SETR’s pure-Transformer encoder and Swin’s hierarchical, shifted-window design for efficient general-purpose vision backbones.

  • Comparison methodology: The survey compares selected ViT segmentation models on ADE20K, Cityscapes, and PASCAL-Context to establish a common benchmark basis.Benchmark results are summarized in Table 2.
  • SETR: SETR formulates semantic segmentation as sequence-to-sequence prediction using a pure Transformer encoder without convolution layers.The encoder replaces stacked convolutional layers while gradually reducing spatial resolution.
  • SETR: SETR variants differ in their decoders: SETR-PUP uses progressive up-sampling, whereas SETR-MLA uses multi-level feature aggregation.The standard SETR encoder processes patch and positional embeddings through Transformer layers without per-layer spatial down-sampling.
  • SETR results: SETR achieved state-of-the-art semantic-segmentation results on ADE20K and PASCAL-Context at submission and was also tested on Cityscapes.The reported Cityscapes outcome is described as impressive without a numerical value here.
  • Swin Transformer: Swin Transformer addresses the lack of a general-purpose Transformer backbone through hierarchical feature maps and shifted-window self-attention.Its hierarchical stages also reduce self-attention complexity from quadratic to linear, while shifted windows reduce latency relative to sliding-window approaches.
  • Swin results: Swin Transformer reports 53.5% mIoU on ADE20Kval, alongside 87.3% ImageNet-1K top-1 accuracy and 58.7 box AP and 51.1 mask AP on COCO test-dev.Its patch merging reduces token counts in deeper layers, producing feature-map resolutions similar to convolutional backbones such as ResNet.

4.3. Segmenter

Segmenter is a purely Transformer-based semantic-segmentation model that combines an ImageNet-pretrained ViT backbone with a mask-transformer decoder. Its design emphasizes global image context and can be adapted for panoptic segmentation.

  • Architecture: Segmenter combines an ImageNet-pretrained ViT backbone with a mask transformer decoder for semantic segmentation.The backbone is fine-tuned on moderate-sized segmentation datasets after classification pre-training.
  • Motivation: Segmenter is designed to capture global image context, addressing the limited global-information access of local convolution operations.The passage notes that modeling global interactions has quadratic complexity with image size.
  • Extensions: Segmenter can also support panoptic segmentation by replacing class embeddings with object embeddings.Panoptic segmentation combines semantic and instance segmentation.

4.4. SegFormer

SegFormer uses a hierarchical Transformer encoder and lightweight MLP decoder for semantic segmentation, with design choices intended to preserve local detail and avoid positional-encoding interpolation. Its largest variant surpasses SETR on ADE20K while running four times faster.

  • Architecture: SegFormer uses a hierarchical Transformer encoder for feature extraction and a lightweight MLP decoder to predict the final mask.The architecture uses 4 × 4 patches and overlapped patch merging to maintain local continuity.
  • Design choices: SegFormer introduces a positional-encoding-free design because fixed-resolution positional encodings require interpolation for different test resolutions.The survey associates this interpolation with reduced accuracy.
  • Results: SegFormer achieves competitive results on ADE20K, Cityscapes, and COCO-Stuff across variants B0 through B5.The variants trade off model size, accuracy, and runtime.
  • Results: 4× faster than SETR, SegFormer-B5 surpasses SETR on ADE20K while achieving the highest mIoU among the compared models described here.SegFormer-B5 is the largest variant in the series.

4.5. Pyramid Vision Transformer (PVT)

PVT adapts ViT for dense prediction with a progressive shrinking pyramid that reduces computation while producing fine-grained segmentation features. PVT v2 further addresses high-resolution processing, local continuity, and variable-sized inputs.

  • PVT v1: PVT uses a progressive shrinking pyramid backbone to reduce computational costs and produce more fine-grained segmentation outputs.Its stages progressively reduce output resolution from high to low.
  • PVT v1: 4 × 4 input patches improve high-resolution representation learning compared with ViT’s 16 × 16 patches.The pyramid also reduces the computational demand of traditional ViT.
  • PVT v2: PVT v2 uses linear spatial reduction attention, overlapping patch embedding, and convolutional feed-forward networks to address PVT v1’s design limitations.These changes target computational cost, local continuity, and variable input resolutions.
  • PVT v2: PVT v2 brings PVT v1’s complexity down to linear complexity through its three architectural improvements.
  • Twins: Twins introduces Twins-PCPVT and Twins-SVT, with spatially separable self-attention capturing local and global information while reducing high-resolution computation.Twins-PCPVT uses conditional position encoding through a positional encoding generator, while Twins-SVT uses locally grouped and global sub-sampled attention.
  • Twins: Twins architectures achieve semantic-segmentation results that are highly competitive with Swin Transformer and PVT.

4.7. Dense Prediction Transformer (DPT)

DPT combines a Transformer backbone with an encoder-decoder design that preserves spatial information and reconstructs multi-resolution feature maps for dense segmentation. Its experiments report improved benchmark accuracy, including DPT-Hybrid outperforming fully convolutional models on ADE20K.

  • DPT uses a Transformer backbone inside an encoder-decoder architecture to produce fine-grained segmentation predictions.The ViT-based encoder maintains spatial resolution across Transformer stages, supporting dense prediction.
  • DPT-Base and DPT-Large use non-overlapping patch embeddings with learnable position embeddings, whereas DPT-Hybrid uses a ResNet-50 feature extractor.The hybrid model supplies pixel-based feature maps as token inputs to the Transformer.
  • Transformer tokens are reassembled into image-like representations at multiple resolutions and fused through residual convolutional units for final dense prediction.
  • DPT reports improved accuracy across several benchmark comparisons, with DPT-Hybrid outperforming fully convolutional models on ADE20K semantic segmentation.The model also identifies precise object boundaries with less distortion.

4.8. High-Resolution Transformer (HRFormer)

HRFormer targets dense prediction by combining high-resolution multi-resolution streams with local-window self-attention and depth-wise convolutional feed-forward networks. Its design maintains high-resolution features while improving computational and memory efficiency relative to conventional ViT outputs.

  • HRFormer combines depth-wise convolutional FFNs, local-window self-attention, and a multi-resolution parallel Transformer module.It is developed for pose estimation and semantic segmentation.
  • HRFormer is more computationally and memory efficient while avoiding the low-resolution outputs of conventional ViT models.
  • HRFormer incorporates HRNet’s multi-scale parallel design to capture feature maps at varied resolutions while maintaining high resolution.
  • Non-overlapping windows perform separate self-attention, improving efficiency compared with earlier overlapping local-window mechanisms.A following depth-wise convolutional FFN exchanges information between local windows and increases receptive-field size.

4.9. Masked-attention Mask Transformer (Mask2Former)

Mask2Former is a universal Transformer architecture for panoptic, instance, and semantic segmentation, built from a backbone, pixel decoder, and Transformer decoder. Its masked attention localizes cross-attention to predicted foreground regions, while the architecture still requires separate training for each segmentation task.

  • Mask2Former supports panoptic, instance, and semantic segmentation within a universal architecture.It was reported to outperform specialized state-of-the-art architectures for each task at submission.
  • Masked attention restricts cross-attention to the foreground region of the predicted mask and extracts localized features instead of attending to the full feature map.This makes the attention mechanism more efficient.
  • Its main components are a backbone feature extractor, pixel decoder, and Transformer decoder.The backbone may be CNN-based or Transformer-based, and the pixel decoder uses multi-scale deformable attention.
  • Mask2Former achieves new state-of-the-art performance on panoptic, instance, and semantic segmentation datasets including COCO, ADE20K, and Cityscapes.Its semantic-segmentation results are compared on ADE20K and Cityscapes in Table 2.
  • Despite its universal design, Mask2Former must still be trained separately for each specific segmentation task.

5. Discussion

The survey reviews ViTs as alternatives to CNNs for semantic segmentation across remote sensing, medical, and video-processing domains. It summarizes datasets and ViT results to provide a high-level comparison.

  • ViTs are discussed as alternatives to classical CNNs across remote sensing, medical, and video-processing applications.
  • The survey includes selected CNN architectures for comparison, while excluding an in-depth CNN architecture discussion.
  • Dataset statistics and ViT segmentation results are summarized to give readers a high-level overview of the field.

6. Conclusions and Future Directions

The survey presents ViTs as still developing but competitive with mature CNNs, while identifying computational demand as a barrier to experimentation. It points to lightweight architectures and additional application areas as future directions.

  • ViTs remain in an early development stage but are described as powerful and competitive with CNN counterparts.
  • The survey identifies underexplored application areas, including big data analytics, as opportunities for future ViT research.
  • High computational demand makes ViTs challenging to experiment with and motivates lighter, more efficient architectures.
Loading 2305.03273v1…