Source-linked AI summary
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, Li Zhang
TL;DR
Semantic segmentation FCNs have limited long-range dependency modeling because receptive fields grow with depth and attention is typically restricted to higher layers. SETR reformulates segmentation as sequence-to-sequence prediction with a pure transformer encoder, achieving first place on the ADE20K test server leaderboard on the submission day.
Problem
FCN-based segmentation models have limited long-range dependency modeling, while attention is often restricted to higher layers, leaving lower-level dependency learning lacking.
Method
SETR reformulates semantic segmentation as sequence-to-sequence prediction by encoding image patches with a pure self-attention transformer and combining it with decoder designs.
Results
SETR achieved first place on the highly competitive ADE20K test server leaderboard on the day of submission.
Takeaways & Limitations
The paper demonstrates an alternative architectural perspective that eliminates reliance on FCNs while modeling global context throughout feature learning.
Takeaways & Limitations
Tokenizing every image pixel is impractical because transformer complexity is quadratic in the number of pixels, requiring patch-based tokenization.
Abstract
from arXiv · showhide
Most recent semantic segmentation methods adopt a fully-convolutional network (FCN) with an encoder-decoder architecture. The encoder progressively reduces the spatial resolution and learns more abstract/semantic visual concepts with larger receptive fields. Since context modeling is critical for segmentation, the latest efforts have been focused on increasing the receptive field, through either dilated/atrous convolutions or inserting attention modules. However, the encoder-decoder based FCN architecture remains unchanged. In this paper, we aim to provide an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task. Specifically, we deploy a pure transformer (ie, without convolution and resolution reduction) to encode an image as a sequence of patches. With the global context modeled in every layer of the transformer, this encoder can be combined with a simple decoder to provide a powerful segmentation model, termed SEgmentation TRansformer (SETR). Extensive experiments show that SETR achieves new state of the art on ADE20K (50.28% mIoU), Pascal Context (55.83% mIoU) and competitive results on Cityscapes. Particularly, we achieve the first position in the highly competitive ADE20K test server leaderboard on the day of submission.
1. Introduction
The paper reframes semantic segmentation as sequence-to-sequence prediction, replacing the progressively downsampling convolutional encoder of FCNs with a pure transformer over image patches. This design yields SETR, an alternative segmentation model with fully attentive feature representation learning.
- FCN limitations: Standard FCN segmentation uses an encoder-decoder architecture: the encoder learns feature representations, while the decoder performs pixel-level classification.The encoder progressively reduces feature-map resolution to learn increasingly abstract semantic concepts with larger receptive fields.
- FCN limitations: FCN encoders downsample spatial resolution to produce lower-resolution mappings, and decoders upsample them into full-resolution segmentation maps.
- SETR: SETR replaces the progressively downsampling convolutional encoder with a pure transformer that represents an input image as a sequence of learned patch embeddings.
- Proposed perspective: The paper reformulates semantic segmentation as sequence-to-sequence learning, offering an alternative to the dominant encoder-decoder FCN design.
- SETR: The transformer-based encoder provides fully attentive feature representation learning by sequentializing images.The motivation is that image features need not be learned progressively from local to global context through spatial-resolution reduction.
2. Related work
Related work traces semantic segmentation from FCN-based pixel prediction and context-enhancement methods to transformer and self-attention models. SETR is positioned against prior attention-based segmentation by using a pure-transformer, sequence-to-sequence design that preserves spatial resolution instead of progressively reducing it.
- Semantic segmentation: FCN enables pixel-wise predictions after removing fully connected layers, while CRF/MRF approaches refine its relatively coarse outputs.The FCN framework addresses semantic segmentation through fully convolutional prediction, with CRF/MRF methods improving coarse results.
- Semantic segmentation: Recent FCN-based methods enlarge receptive fields or model context using dilated convolutions, PPM, and ASPP with different dilation rates.DeepLab and Dilation introduce dilated convolution; PSPNet uses PPM for regional context, while DeepLabV2 uses pyramid dilated convolutions.
- Transformer: Transformer and self-attention models have expanded from machine translation and NLP into vision, including convolution-attention hybrids and local self-attention networks.Prior vision applications include Non-local networks, AANet, LRNet, and stand-alone networks, with local attention explored to reduce global computation.
- Transformer: ViT showed that a pure-transformer image-classification model can achieve state-of-the-art performance, directly inspiring a pure-transformer encoder for semantic segmentation.Other transformer applications mentioned include object detection, disparity estimation, and lane shape prediction.
- Closest related work: Compared with the most related attention-based segmentation work, SETR removes convolution while keeping spatial resolution constant rather than progressively reducing feature-map resolution.The cited prior model also removes convolution but retains the conventional FCN design, whereas SETR represents a step-change through sequence-to-sequence prediction.
3. Method
SETR reframes semantic segmentation as sequence prediction by converting an image into patch embeddings and encoding them with a pure transformer. Its encoder provides global receptive fields, while three decoder designs reshape and upsample transformer features into pixel-level segmentation outputs.
- Motivation: FCN locality limits receptive fields, while attention-based methods restrict long-range dependency learning to higher-resolution-reduced layers because of quadratic complexity.This leaves lower-level feature representations insufficiently modeled and can produce sub-optimal representation learning.
- Image to Sequence: SETR converts an image x ∈ R^(H×W×3) into a 1D feature sequence Z ∈ R^(L×C) to match the transformer’s sequence-processing formulation.Image sequentialization bridges the mismatch between the 2D image and 1D transformer input.
- Image to Sequence: Pixel-level tokenization is infeasible because a 480×480×3 image yields 691,200 values, and transformer complexity is quadratic in sequence length.SETR instead divides the image into a uniform H/16 × W/16 patch grid, projects each patch into C dimensions, and adds learned location-specific embeddings.
- Transformer Encoder: A pure transformer encoder processes the patch-embedding sequence with global receptive fields in every layer, eliminating the FCN encoder’s limited-receptive-field problem.The encoder is composed of multi-head self-attention and multilayer perceptron blocks.
- Decoder Designs: SETR evaluates three pixel-level decoders: Naive one-step bilinear upsampling, PUP progressive 2× upsampling, and MLA multi-level feature aggregation.All decoders reshape transformer features from HW/256 × C into an H/16 × W/16 × C feature map before producing full-resolution segmentation outputs.
4. Experiments
Experiments evaluate SETR across Cityscapes, ADE20K, and PASCAL Context using multiple decoder variants, transformer backbones, pre-training strategies, and matched FCN baselines. SETR achieves strong benchmark results, including 50.28% mIoU on ADE20K and 55.83% on PASCAL Context.
- Pre-training comparison: 48.64% mIoU: SETR-MLA with T-Large and ImageNet-21K pre-training outperforms FCN with the same pre-training, which achieves 42.17% on ADE20K.On Cityscapes, the corresponding values are 76.65% for SETR-MLA and 76.38% for FCN.
- Model variants: SETR compares three decoder variants—SETR-Naïve, SETR-PUP, and SETR-MLA—and two transformer encoders, T-Base and T-Large.T-Base has 12 layers and T-Large has 24 layers; T-Large is the default encoder for the three main variants.
- Cityscapes and ablations: SETR-PUP performs best among the variants on Cityscapes, while SETR-MLA outperforms SETR-Naïve on ADE20K validation.The passage attributes MLA’s relative weakness on Cityscapes possibly to lacking a resolution pyramid across transformer-layer outputs.
- ADE20K results: 50.28% mIoU: SETR-MLA establishes a new state of the art on ADE20K with multi-scale inference, versus 48.64% with single-scale inference.The model ranked first on the ADE20K test-server leaderboard when trained on the train-plus-validation set.
- PASCAL Context results: 55.83% mIoU: SETR-MLA with multi-scale inference improves over 54.87% with single-scale inference and surpasses APCNet.The ResNet-101 FCN baseline achieves 45.74% mIoU under the same training schedule.
5. Conclusion
The paper reframes semantic segmentation as sequence-to-sequence prediction, replacing FCN-based design with an architecture that eliminates reliance on FCNs and addresses limited receptive fields. The method ranked first on the ADE20K test server leaderboard on the submission day.
- 5. Conclusion: The work introduces a sequence-to-sequence prediction framework as an alternative perspective for semantic segmentation.This reframes the task at the architectural level rather than modifying individual FCN components.
- 5. Conclusion: The proposed architecture completely eliminates reliance on FCNs, addressing the limited receptive field challenge without component-level dilation or attention modifications.Existing FCN methods typically enlarge receptive fields using dilated convolutions and attention modules.
- 5. Conclusion: 1st place was achieved on the highly competitive ADE20K test server leaderboard on the day of submission.The ranking is reported specifically for the ADE20K test server leaderboard.
Appendix · A. Visualizations
The appendix visualizes SETR’s learned positional structure, transformer attention, and intermediate features. These visualizations show that position-embedding similarity encodes image distance and document representations across encoder and decoder stages.
- A. Visualizations: Learned position embeddings encode image distance through similarity between embeddings.Figure 7 visualizes cosine similarities between a selected patch’s position embedding and those of all other patches.
- A. Visualizations: Feature visualization tracks 24 encoder outputs, Z1−Z24, and 5 decoder outputs, U1−U5.The encoder features come from the 24 transformer layers, while decoder features are collected after each bilinear interpolation.
- A. Visualizations: Attention maps are examined across the transformer’s layers and heads.T-large contains 16 heads and 24 layers, and the analysis uses a recursion perspective with selected spatial points.
- A. Visualizations: Figure 8 illustrates attention maps for selected spatial points in Pascal Context images.The selected points are marked in red, while corresponding attention maps appear in alternating columns.
- A. Visualizations: Figure 7 visualizes positional similarity for SETR-PUP trained on Pascal Context.Tiles show cosine similarity between the indicated patch and every other patch.
- A. Visualizations: Figure 9 displays SETR-PUP features across encoder layers Z1−Z24 and decoder features U1−U5.The visualization is organized into input, early encoder, late encoder, and decoder rows.