Source-linked AI summary
Segmenter: Transformer for Semantic Segmentation
Robin Strudel, Ricardo Garcia, Ivan Laptev, Cordelia Schmid
TL;DR
Semantic segmentation requires contextual information because individual image patches can be ambiguous, while convolutional approaches are biased toward local interactions. Segmenter extends ViT with a fully transformer-based encoder and linear or mask-transformer decoders for pixel-level segmentation. It achieves state-of-the-art results on ADE20K and Pascal Context, is competitive on Cityscapes, and improves further with mask decoding, larger models, and smaller patches.
Problem
Semantic segmentation requires global context because local labeling can be ambiguous, while convolutional methods remain biased toward local interactions.
Method
Segmenter extends ViT into a convolution-free encoder-decoder that maps patch embeddings to pixel-level annotations using linear or mask-transformer decoding.
Results
Segmenter achieves state-of-the-art results on ADE20K and Pascal Context, is competitive on Cityscapes, and its mask transformer improves over the linear baseline.
Takeaways & Limitations
Global-context transformer encoding and mask-transformer decoding provide a strong approach to semantic segmentation across standard benchmarks.
Abstract
from arXiv · showhide
Image segmentation is often ambiguous at the level of individual image patches and requires contextual information to reach label consensus. In this paper we introduce Segmenter, a transformer model for semantic segmentation. In contrast to convolution-based methods, our approach allows to model global context already at the first layer and throughout the network. We build on the recent Vision Transformer (ViT) and extend it to semantic segmentation. To do so, we rely on the output embeddings corresponding to image patches and obtain class labels from these embeddings with a point-wise linear decoder or a mask transformer decoder. We leverage models pre-trained for image classification and show that we can fine-tune them on moderate sized datasets available for semantic segmentation. The linear decoder allows to obtain excellent results already, but the performance can be further improved by a mask transformer generating class masks. We conduct an extensive ablation study to show the impact of the different parameters, in particular the performance is better for large models and small patch sizes. Segmenter attains excellent results for semantic segmentation. It outperforms the state of the art on both ADE20K and Pascal Context datasets and is competitive on Cityscapes.
1. Introduction
Semantic segmentation remains difficult because local image patches can be ambiguous, while convolutional methods have limited access to global context. Segmenter addresses this with a convolution-free Vision Transformer approach and achieves state-of-the-art results on ADE20K and Pascal Context.
- Motivation: Semantic segmentation assigns every image pixel a category label while providing high-level representations for downstream tasks.Applications include autonomous driving, robotics, augmented reality, image editing, and medical imaging.
- Motivation: Rich intra-class variation, context variation, occlusions, and low resolution make image segmentation challenging.
- Motivation: Convolutional methods capture semantically rich information but their local operations limit access to global image information needed for ambiguous local labeling.
- Approach: Segmenter formulates semantic segmentation as sequence-to-sequence prediction with a transformer that models contextual information throughout the network.The approach avoids convolutions and leverages global interactions between scene elements.
- Results: Segmenter achieves state-of-the-art results on ADE20K and Pascal Context and competitive performance on Cityscapes.
2. Related work
Prior semantic-segmentation systems largely use convolutional encoder-decoder architectures, whose local operations can restrict global-context processing. Segmenter instead uses a fully transformer-based encoder-decoder with a ViT backbone and mask decoder.
- Convolutional approaches: FCN-based encoder-decoder systems dominate semantic segmentation by combining convolutional feature extraction, pooling, upsampling, and dense predictions.
- Limitations of prior work: The locality of convolutional operations may make global image-context processing inefficient and lead to suboptimal segmentation results.
- Vision Transformers: Vision Transformers process images as sequences of patch tokens, extending transformer self-attention from long-range language dependencies to vision.
- Related segmentation models: Related segmentation models combine ViT backbones or variants with CNN-based decoders, including SETR and Swin Transformer.
- Segmenter: Segmenter uses a convolution-free ViT encoder-decoder and introduces a DETR-inspired mask decoder to capture global image context by design.
3. Our approach: Segmenter
Segmenter uses a fully transformer-based encoder-decoder to map image-patch embeddings to pixel-level semantic labels. It offers linear and mask-transformer decoders, with the latter producing class masks from jointly processed patch and class embeddings.
- Encoder: The model encodes image patches with positional embeddings and transformer layers, producing contextualized patch representations for decoding.Each transformer layer combines multi-headed self-attention and a point-wise MLP with layer normalization and residual connections.
- Encoder: The encoder maps a sequence of embedded patches to semantic encodings that contain rich contextual information for the decoder.
- Linear decoder: A linear decoder maps each patch encoding to class logits, reshapes them into a low-resolution map, and upsamples them to pixel-level predictions.Softmax over classes produces the final segmentation map.
- Mask transformer: The mask transformer jointly processes patch encodings with K learnable class embeddings to generate class masks.The masks are reshaped, bilinearly upsampled, normalized, and converted into pixel-wise class scores.
- Mask transformer: Mask sequences are softly exclusive across classes, with per-pixel class probabilities summing to one.
- Mask transformer: Jointly processing patch and class embeddings produces input-dependent dynamic filters and supports adaptation from semantic to panoptic segmentation using object embeddings.
4. Experimental results
Segmenter is evaluated across datasets, model configurations, patch sizes, regularization schemes, decoders, object sizes, and inference settings. Larger models, smaller patches, and mask-transformer decoding improve segmentation, while results are state of the art on ADE20K and Pascal Context and competitive on Cityscapes.
- Experimental setup: Segmenter is evaluated on ADE20K, Pascal Context, and Cityscapes, using ViT and DeiT backbones with 8×8, 16×16, or 32×32 input patches.Models are initialized from ImageNet-pretrained vision transformers; inference uses sliding windows and optionally multi-scale averaging.
- Ablation study: Stochastic depth improves Seg-S/16 performance by 0.36% over no regularization, whereas dropout hurts performance alone or combined with stochastic depth.The selected training setup uses stochastic depth 0.1 without dropout.
- Ablation study: A 32-to-16 patch-size reduction improves Seg-B by 5%, while Seg-B/8 reaches 49.54% mIoU and narrows the gap to ViT-L/16 to 1.17%.Smaller patches improve resolution but increase attention cost, computation time, and memory usage.
- Ablation study: The mask transformer consistently improves over the linear decoder, with gains of 1.6% for Seg-B†/16, 1.1% for Seg-B-Mask/32, and 0.6% for Seg-L/16.Object-size analysis reports gains on both small and large objects; Seg-L-Mask/16 improves over DeepLabv3+ by 6.39% on large instances.
- Comparison with state of the art: On ADE20K, Seg-L-Mask/16 achieves 53.63% mIoU and outperforms DeepLabv3+ ResNeSt-200 and transformer-based competitors by 5.27% mIoU.On Pascal Context it reaches 59.04% mIoU, while on Cityscapes Seg-L-Mask/16 achieves 81.3% mIoU and remains competitive with other state-of-the-art methods.
5. Conclusion
Segmenter is a pure transformer approach for semantic segmentation that uses all image-patch encodings. A linear decoder already performs excellently, while a mask transformer further improves performance.
- Segmenter uses a pure transformer encoder built on Vision Transformer and relies on encodings of all image patches.
- A point-wise linear decoder applied to patch encodings already achieves excellent semantic-segmentation results.
- A mask transformer decoder further improves performance over the point-wise linear decoder.
Appendix
The appendix adds analyses of pretraining, attention maps, class embeddings, and qualitative comparisons with DeepLabv3+ across three datasets.
- The appendix studies ImageNet pretraining and analyzes attention maps and learned class embeddings for Segmenter.
- It also provides qualitative comparisons between Segmenter and DeepLabv3+ on ADE20K, Cityscapes, and Pascal Context.
A. ImageNet pre-training
The pretraining study compares ImageNet-pretrained Segmenter models with equivalent models trained from scratch using specified optimization procedures.
- The study compares ImageNet-pretrained Segmenter models with equivalent models trained from scratch.
- Models trained from scratch use randomly initialized truncated-normal weights, a 10^-3 base learning rate, and two training procedures.
B. Attention maps and class embeddings
Segmenter’s attention adapts its receptive field to instance size and gathers information progressively across layers. Its learned class embeddings implicitly cluster semantically related categories.
- Attention maps: Attention maps gather global information for large instances while focusing on local information for smaller instances.This adaptability differs from CNNs, whose field of view is constant independently of the data.
- Attention maps: Attention information is gathered progressively from bottom to top layers, moving from identifying an instance to identifying other instances of the same category.
- Attention maps: Already in the first layer, some attention heads attend to distant patches outside the initial receptive fields of ResNet/ResNeSt.
- Class embeddings: Two-dimensional projections of mask-transformer class embeddings implicitly cluster semantically related categories.Examples include means of transportation, household objects, and outdoor categories.
C. Qualitative results
Segmenter produces more coherent segmentation maps than DeepLabv3+ in several examples, while both methods confuse visually similar regions and struggle with small instances. DeepLabv3+ retains an advantage for boundaries between nearby people.
- Segmenter produces more coherent segmentation maps than DeepLabv3+ for examples such as wedding dresses and airplane signalmen’s helmets.
- DeepLabv3+ handles boundaries between different people entities better than Segmenter.
- Both methods confuse visually similar regions, including armchairs and couches, cushions and pillows, and trees, flowers, and plants.
- Both Segmenter and DeepLabv3+ have difficulty segmenting small instances such as lamps, people, flowers, cars, and signals.
- The qualitative analysis includes patch attention maps, attended-area measurements, and singular-value decompositions of mask-transformer class embeddings.