PaperlayerView exact revision on arXiv

Source-linked AI summary

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Ping Luo

arXiv:2105.15203v3cs.CVcs.LG

TL;DR

Semantic segmentation needs Transformer frameworks that balance accuracy, efficiency, and robustness while addressing limitations in existing encoders and decoders. SegFormer introduces a hierarchical positional-encoding-free Transformer encoder with a lightweight All-MLP decoder, achieving state-of-the-art results across three datasets and strong zero-shot robustness.

  • Problem

    Existing Transformer segmentation methods mainly redesign encoders while retaining limitations in feature scales, computation cost, or decoder design.

  • Method

    SegFormer combines a positional-encoding-free hierarchical Transformer encoder with a lightweight All-MLP decoder for semantic segmentation.

  • Results

    SegFormer sets new state-of-the-art results in efficiency, accuracy, and robustness across ADE20K, Cityscapes, and COCO-Stuff, including 84.0% mIoU on Cityscapes while being 5× faster than SETR.

  • Takeaways & Limitations

    SegFormer provides a simple, efficient baseline that achieves strong semantic segmentation performance and zero-shot robustness without complex architectural designs.

  • Takeaways & Limitations

    It remains unclear whether the smallest 3.7M-parameter model can run effectively on an edge-device chip with only 100k memory.

Abstract

from arXiv · show

We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding, thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution differs from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from different layers, and thus combining both local attention and global attention to render powerful representations. We show that this simple and lightweight design is the key to efficient segmentation on Transformers. We scale our approach up to obtain a series of models from SegFormer-B0 to SegFormer-B5, reaching significantly better performance and efficiency than previous counterparts. For example, SegFormer-B4 achieves 50.3% mIoU on ADE20K with 64M parameters, being 5x smaller and 2.2% better than the previous best method. Our best model, SegFormer-B5, achieves 84.0% mIoU on Cityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C. Code will be released at: github.com/NVlabs/SegFormer.

1 Introduction

SegFormer is introduced as a Transformer framework for semantic segmentation that redesigns both encoder and decoder to jointly improve efficiency, accuracy, and robustness. Its positional-encoding-free hierarchical encoder and lightweight All-MLP decoder address limitations of prior vision Transformers while achieving strong dataset results.

  • Contributions: SegFormer redesigns both the encoder and decoder for efficient, accurate, and robust semantic segmentation.The framework is evaluated on ADE20K, Cityscapes, and COCO-Stuff.
  • Encoder: The encoder is positional-encoding-free and hierarchical, producing high-resolution fine features and low-resolution coarse features.It avoids interpolating positional codes when inference and training resolutions differ, unlike ViT’s single low-resolution feature maps.
  • Decoder: The lightweight All-MLP decoder produces powerful representations without complex, computationally demanding modules.It complements the encoder redesign with a simple decoder focused on efficiency.
  • Results: 71.9% mIoU at 48 FPS is achieved by SegFormer-B0 on Cityscapes without accelerated implementations such as TensorRT.Compared with ICNet, this represents relative improvements of 60% in latency and 4.2% in performance.
  • Results: 84.0% mIoU is achieved by the largest model, SegFormer-B5, on Cityscapes.The paper also reports new state-of-the-art efficiency, accuracy, and robustness across three publicly available semantic segmentation datasets.

2 Related Work

Prior semantic-segmentation methods improved FCN through receptive-field, context, boundary, attention, and AutoML techniques, but often became complicated and computationally demanding. Transformer research progressed from pure classification backbones to pyramid architectures and task-specific applications, including semantic segmentation.

  • Semantic Segmentation: FCN established end-to-end pixel-to-pixel semantic segmentation, while later methods expanded receptive fields and refined context, boundaries, attention, or automated design.These approaches significantly improved segmentation performance but introduced many empirical modules, making frameworks computationally demanding and complicated.
  • Transformer backbones: ViT showed that a pure Transformer can achieve state-of-the-art image classification by processing images as token sequences through multiple Transformer layers.DeiT subsequently explored data-efficient training and distillation, while later models introduced tailored architectural improvements.
  • Transformer backbones: PVT introduced pyramid-structured Transformers for dense prediction, demonstrating the potential of pure Transformer backbones relative to CNN counterparts.Swin, CvT, CoaT, LeViT, and Twins further enhanced local feature continuity and removed fixed-size positional embeddings for dense prediction.
  • Transformers for specific tasks: Transformers have been applied to detection, tracking, super-resolution, ReID, colorization, retrieval, and multimodal learning.DETR built an end-to-end object-detection framework without non-maximum suppression, while SETR used ViT as a semantic-segmentation backbone and achieved impressive performance.

3 Method

SegFormer combines a hierarchical Transformer encoder with a lightweight All-MLP decoder for efficient semantic segmentation. The encoder produces multiscale features using overlapping patch merging, efficient self-attention, and convolutional Mix-FFN blocks, while the decoder fuses them into a segmentation mask.

  • 3 Method: SegFormer consists of a hierarchical Transformer encoder that generates multilevel features and a lightweight All-MLP decoder that produces the final segmentation mask.The encoder provides high-resolution coarse and low-resolution fine features for decoding.
  • Mix Transformer Encoder: MiT-B0 through MiT-B5 share an architecture but vary in size, from fast lightweight inference to best performance.The MiT design is inspired by ViT but optimized for semantic segmentation.
  • Hierarchical Feature Representation: Overlapping patch merging uses K = 7, S = 4, and P = 3 to preserve local continuity while generating hierarchical feature maps.The encoder produces features at 1/4, 1/8, 1/16, and 1/32 of the original resolution from 4 × 4 input patches.
  • Efficient Self-Attention: Efficient self-attention reduces complexity from O(N^2) to O(N^2/R), using R = [64, 16, 4, 1] across stages 1–4.Sequence reduction shortens the key sequence before attention computation.
  • Mix-FFN: Mix-FFN replaces positional encoding with a 3 × 3 depth-wise convolution combined with an MLP, reducing parameters while providing positional information.This avoids interpolation of fixed positional codes when test and training resolutions differ.
  • All-MLP Decoder: The All-MLP decoder unifies channel dimensions, upsamples multilevel features to 1/4 resolution, concatenates and fuses them, then predicts the mask.It leverages complementary local and non-local attention from the Transformer encoder with few additional parameters.

4 Experiments

Experiments show that SegFormer’s lightweight MLP decoder works best with a Transformer encoder, Mix-FFN improves resolution robustness, and scaling the model delivers strong accuracy-efficiency results across datasets. SegFormer also achieves leading performance on ADE20K, Cityscapes, and COCO-Stuff while showing substantial corruption robustness.

  • Model size: Increasing encoder size improves segmentation performance while retaining a lightweight decoder, whose size is only 0.4M parameters for the lightweight model.The experiments analyze performance and model efficiency as encoder size increases across ADE20K, Cityscapes, and COCO-Stuff.
  • MLP decoder dimension: C = 256 provides a competitive balance of MLP-decoder performance and computational cost, while wider channel dimensions increase model size and eventually plateau in performance.Performance rises with C but becomes less efficient at larger dimensions.
  • Resolution robustness: 0.7%: Mix-FFN reduces the accuracy drop across test resolutions from 3.3% with positional encoding to 0.7%.Mix-FFN also clearly outperforms positional encoding at the evaluated Cityscapes resolutions.
  • State-of-the-art comparisons: 51.8% mIoU: SegFormer-B5 sets a new ADE20K state of the art, improving on SETR by 1.6% mIoU, while SegFormer-B0 reaches 37.4% mIoU with 3.8M parameters and 8.4G FLOPs.On ADE20K, B0 also outperforms real-time counterparts in parameters, FLOPs, and latency.
  • State-of-the-art comparisons: 84.0% mIoU: SegFormer-B5 achieves the best Cityscapes IoU, while B0 reaches 76.2% mIoU at 15.2 FPS and 71.9% mIoU at 47.6 FPS.B5 outperforms existing methods by at least 1.8% mIoU; B0 improves on DeeplabV3+ and ICNet under the reported settings.
  • Robustness: 588%: SegFormer provides up to 588% relative improvement on Gaussian Noise and up to 295% on snow weather.These results indicate strong robustness to common corruptions and perturbations.

5 Conclusion

SegFormer is presented as a simple, clean semantic segmentation method combining a positional-encoding-free hierarchical Transformer encoder with a lightweight All-MLP decoder. By avoiding complex designs, it delivers high efficiency and performance, achieves state-of-the-art results, and shows strong zero-shot robustness.

  • 5 Conclusion: SegFormer combines a positional-encoding-free hierarchical Transformer encoder with a lightweight All-MLP decoder.This design is described as simple, clean, and powerful.
  • 5 Conclusion: Avoiding complex designs in previous methods leads to high efficiency and performance.
  • 5 Conclusion: SegFormer achieves new state-of-the-art results on common datasets and shows strong zero-shot robustness.

A Details of MiT Series

The MiT encoder is scaled from B0 to B5 by varying key hyper-parameters. These variants range from B0, designed for real-time use, to B5, designed for high performance.

  • MiT Series: MiT encoders scale from B0 to B5 by changing selected hyper-parameters.The series uses code names B0 through B5.
  • MiT Hyper-parameters: Stage-wise embedding hyper-parameters include patch size Ki, stride Si, padding size Pi, and output channel number Ci.These parameters describe the overlapping patch embedding and stage outputs.
  • MiT Hyper-parameters: Stage-wise processing hyper-parameters include encoder-layer count Li, attention reduction ratio Ri, attention head count Ni, and feed-forward expansion ratio Ei.Ri and Ni belong to Efficient Self-Attention, while Ei belongs to the feed-forward layer.
  • MiT Series: B0 is the smallest MiT model designed for real-time use, whereas B5 is the largest model designed for high performance.The paper assigns code names B0 through B5 to the MiT encoders.

B More Qualitative Results on Mask Predictions

Qualitative results on Cityscapes, ADE20K, and COCO-Stuff compare SegFormer with SETR and DeepLabV3+. SegFormer produces finer boundary details than SETR and fewer long-range errors than DeepLabV3+ through its Transformer encoder.

  • Figure 5 presents qualitative mask predictions on Cityscapes, ADE20K, and COCO-Stuff compared with SETR and DeepLabV3+.
  • Compared with SETR, SegFormer predicts significantly finer details near object boundaries.Its Transformer encoder captures higher-resolution features and preserves more detailed texture information.
  • Compared with DeepLabV3+, SegFormer reduces long-range errors through the Transformer encoder’s larger effective receptive field.

C More Visualization on Effective Receptive Field

SegFormer exhibits a larger and more image-context-sensitive effective receptive field (ERF) than DeepLabV3+, with learned patterns corresponding to roads, cars, and buildings. These results indicate stronger feature extraction by SegFormer’s Transformer encoder than by ConvNets.

  • C More Visualization on Effective Receptive Field: SegFormer’s ERF is larger and more sensitive to image context than DeepLabV3+’s, learning patterns of roads, cars, and buildings instead of a relatively fixed pattern.The comparison is shown using representative images and ERFs in Figure 6.

D More Comparison of DeeplabV3+ and SegFormer on Cityscapes-C

On Cityscapes-C, SegFormer demonstrates stronger zero-shot robustness than DeepLabV3+ across corruption types and severities. As corruption severity increases, DeepLabV3+ degrades considerably, while SegFormer remains relatively stable.

  • Evaluation protocol: The Cityscapes-C evaluation covers four noise types at three severities and twelve other corruption or perturbation types at five severities.These testing settings follow the protocol in reference [77].
  • Zero-shot robustness: SegFormer has significant advantages over DeepLabV3+ across all Cityscapes-C corruptions, perturbations, and tested severities.The comparison evaluates three severities for four noise types and five severities for the remaining twelve corruption and perturbation types.
  • Zero-shot robustness: With increasing severity, DeepLabV3+ shows considerable performance degradation, whereas SegFormer’s performance remains relatively stable.Figure 7 plots mIoU against corruption severity for both architectures.
Loading 2105.15203v3…