Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Introduction

• FCN segmentation models dominate and use an encoder for feature learning and a decoder for pixel-level classification; progressive resolution reduction expands receptive fields but limits long-range dependency learning.The encoder’s translation equivariance and locality are described as favorable properties, but limited receptive fields make long-range context challenging.Source: §1 · p. 1
• Larger kernels, atrous convolutions, pyramids, and attention address context, but retain the FCN structure; attention-alone models still downsample in the encoder and upsample in the decoder.Source: §1 · pp. 1–2
• SETR replaces the downsampling convolutional encoder with a pure transformer that embeds fixed-size image patches and models global context at every layer, then decodes features to original resolution.Flattened patch pixels are linearly embedded into a sequence of feature vectors for the transformer.Source: §1 · p. 2
• 50.28% on ADE20K and 55.83% on Pascal Context, with competitive Cityscapes results; SETR also ranked first on the ADE20K test-server leaderboard.The introduction describes these as new state-of-the-art results on ADE20K and Pascal Context.Source: §1 · p. 2
• The paper reformulates segmentation as sequence-to-sequence learning, implements a fully attentive encoder, and introduces three decoder designs of varying complexity; adapting pure transformers to spatially sensitive segmentation is non-trivial.The approach is motivated by pure transformers’ success in NLP and image classification, while the paper reports empirical evidence of SETR’s effectiveness.Source: §1 · p. 2

Semantic segmentation

• FCNs enabled pixel-wise prediction [36], followed by CRF/MRF refinement [6] [35] and encoder-decoder fusion of coarse and fine features [38] [42].Source: §2 · p. 2
• FCN context methods enlarge receptive fields with dilated convolution, pyramid context, large kernels, attention, or graph message passing, but retain classical ConvNet backbones [7] [40] [18] [26] [43] [20].Source: §2 · p. 2

Transformer

• Image-transformer studies explored convolutional or hybrid and local or axial attention [48] [3] [24] [41] [47], transformer heads for detection, and disparity or lane prediction [32] [34].ViT [17] showed that a pure-transformer image classifier could achieve state-of-the-art performance, inspiring a pure-transformer encoder for segmentation.Source: §2 · pp. 2–3
• Compared with [47], SETR removes convolution while retaining spatial resolution, unlike its progressively downsampled FCN design, uses standard rather than axial attention [21], and reports higher segmentation accuracy.The paper describes the unchanged spatial resolution as a step-change in model design and says standard self-attention is more scalable to standard computing facilities.Source: §2 · p. 3

3.1. FCN-based semantic segmentation

• SETR addresses FCN context limits: convolutional receptive fields grow with depth, while attention in prior methods is restricted to higher layers, by using a pure self-attention encoder.The paper identifies limited receptive fields as an intrinsic FCN limitation and says restricting attention to higher layers leaves lower-level dependencies unmodeled.Source: §3.1 · p. 3

3.2. Segmentation transformers (SETR)

• SETR converts images into sequences by dividing them into patches, linearly embedding each patch, and adding learned location embeddings to preserve spatial information.Pixel-by-pixel tokenization is impractical because transformer complexity is quadratic in sequence length.Source: §3.2 · p. 4
• Each transformer layer has a global receptive field, enabling the encoder to model context beyond the limited receptive fields of FCNs.The encoder uses multi-head self-attention and MLP blocks.Source: §3.2 · p. 4

3.3. Decoder designs

• SETR offers Naïve one-step bilinear upsampling, PUP progressive 2× upsampling, and MLA aggregation of features from multiple transformer layers.Naïve projects features to category channels before upsampling; PUP alternates convolution and upsampling, while MLA uses multiple feature streams and fuses them through top-down addition and concatenation.Source: §3.3 · pp. 4–5

4.3. Comparison to state-of-the-art

• 50.28% mIoU is achieved by SETR with multi-scale inference on ADE20K, a new state of the art; single-scale SETR-MLA reaches 48.64%.Training one model on train+validation for 160,000 iterations ranked SETR first on the ADE20K test-server leaderboard.Source: Table 4 · p. 6, §4.3 · p. 8

4.1. Experimental setup

• SETR-Naïve, SETR-PUP, and SETR-MLA use different decoder designs; experiments generally pair them with the 24-layer T-Large encoder, with T-Base also tested.Dilated FCN and Semantic FPN serve as baselines, trained and tested in the same settings as SETR for fair comparison.Source: §4.1 · p. 6

4.3. Comparison to state-of-the-art

• 55.83% mIoU is achieved by SETR-MLA with multi-scale inference on Pascal Context, outperforming nearest rival APCNet; single-scale SETR-PUP and SETR-MLA reach 54.40% and 54.87%.The dilated FCN with ResNet-101 reaches 45.74% mIoU under the same training schedule.Source: Table 5 · p. 6, §4.3 · p. 8
• SETR-PUP outperforms FCN and attention-based FCN baselines on Cityscapes validation and is on par with the best reported results; with multi-scale inference, it surpasses Axial-DeepLab.On the test set using fine annotations only, SETR trained for 100k iterations outperforms Axial-DeepLab-XL by a clear margin.Source: Table 6 · p. 7, Table 7 · p. 7, §4.3 · p. 8

4.2. Ablation studies

• 78.02 mIoU for SETR-PUP-Base at 80k iterations exceeds Hybrid-Base’s 76.76, while randomly initialized SETR-PUP reaches 42.27% on Cityscapes; DeiT pre-training performs slightly better than ViT pre-training.SETR-PUP is the best variant on Cityscapes; on ADE20K, SETR-MLA slightly exceeds SETR-PUP and both outperform SETR-Naïve, while T-Large variants outperform T-Base counterparts. FCN also improves with ImageNet-21k pre-training, but SETR outperforms the FCN counterparts, supporting the contribution of sequence-to-sequence modeling beyond larger pre-training data.Source: Table 3 · p. 5, §4.2 · pp. 7–8

• SETR set new state of the art on ADE20 and Pascal Context, achieved competitive Cityscapes results, and ranked first on the ADE20K test-server leaderboard on submission day.Source: §5 · p. 8
• The sequence-to-sequence framework replaces FCN-based receptive-field expansion with a Transformer architecture that models global context at every feature-learning stage.The paper presents decoder designs of varying complexity and reports strong segmentation models without the bells and whistles used by recent methods.Source: §5 · p. 8

A. Visualizations

• Learned position embeddings encode within-image distance through their similarity, visualized as cosine similarities from an indicated patch to all others.Figure 7 shows this for SETR-PUP trained on Pascal Context.Source: A. Visualizations · p. 11, Figure 7 · p. 11
• SETR-PUP feature visualizations include 24 encoder outputs (Z1–Z24) and five decoder features (U1–U5) after bilinear interpolation.Figure 9 arranges the input, encoder layers Z1–Z12 and Z13–Z24, and decoder features in separate rows.Source: A. Visualizations · p. 11, Figure 9 · p. 12
• Figure 9 visualizes SETR-PUP trained on Pascal Context, with encoder features grouped into two layer ranges and decoder features shown separately.The rows show the input image, Z1–Z12, Z13–Z24, and U1–U5, respectively.Source: A. Visualizations · p. 11, Figure 9 · p. 12
• T-large attention maps span 24 transformer layers with 16 heads, and the analysis applies a recursion perspective similar to [1].The passage notes attention maps in each transformer layer.Source: A. Visualizations · p. 11
• Attention maps for selected red-marked spatial points are shown alongside Pascal Context images.Figure 8 places the images in the first and third columns and corresponding attention maps in the second and fourth.Source: A. Visualizations · p. 11, Figure 8 · p. 11
• Figure 10 provides additional examples of attention maps from SETR-PUP trained on Pascal Context.The accompanying text describes attention maps across transformer layers.Source: A. Visualizations · p. 11, Figure 10 · p. 12