Source-linked AI summary
Vision Transformers for Dense Prediction
René Ranftl, Alexey Bochkovskiy, Vladlen Koltun
TL;DR
Dense prediction needs spatial detail and broad context, but convolutional backbones commonly lose resolution through downsampling. DPT uses a vision-transformer backbone, reassembles tokens into multiresolution features, and fuses them with a convolutional decoder; it improves depth estimation and achieves state-of-the-art results across depth and segmentation benchmarks.
Problem
Dense prediction requires fine-grained spatial information, while convolutional backbones lose feature resolution and granularity through downsampling.
Method
DPT replaces the convolutional backbone with a vision transformer, reassembles tokens from multiple stages into image-like multiresolution features, and progressively combines them with a convolutional decoder.
Results
DPT produces more fine-grained and globally coherent predictions, including more than 28% relative improvement over MiDaS for zero-shot monocular depth transfer and state-of-the-art results on depth and segmentation datasets.
Takeaways & Limitations
DPT is especially effective when trained with large-scale data and can also be fine-tuned on smaller depth and segmentation datasets.
Abstract
from arXiv · showhide
We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense vision transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network. When applied to semantic segmentation, dense vision transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art. Our models are available at https://github.com/intel-isl/DPT.
1. Introduction
Dense prediction models typically use convolutional encoder-decoders, but downsampling can discard the fine spatial information these tasks require. DPT replaces the convolutional backbone with a vision transformer to preserve higher-resolution representations and global context.
- Convolutional dense-prediction architectures commonly separate an ImageNet-pretrained encoder from a feature-aggregating decoder.
- Information lost in the encoder cannot be recovered by the decoder, making backbone choice central to overall model capability.
- Downsampling expands receptive fields and keeps computation tractable, but loses deep-stage feature resolution and granularity needed for dense prediction.
- Higher input resolution, dilated convolutions, skip connections, and parallel multiresolution features mitigate granularity loss but remain constrained by memory.
- DPT uses a vision-transformer backbone, reassembles tokens into multiresolution image-like features, and progressively combines them with a convolutional decoder.
- More than 28% performance improvement is reported for monocular depth estimation against the top-performing fully-convolutional network, while DPT also reaches state-of-the-art results on several segmentation and depth datasets.
2. Related Work
Prior dense-prediction systems overwhelmingly use convolution and subsampling to build multiscale context. The related literature explores improved aggregation, dilation, parallel multiscale processing, and attention-based alternatives.
- Fully-convolutional networks remain the prototypical dense-prediction architecture, using convolution and subsampling to learn multiscale representations with broad context.
- Existing convolutional variants progressively upsample pooled representations, use dilated convolutions, or aggregate features in parallel across multiple scales.
- Transformers are self-attention-based set-to-set models that have been especially successful as high-capacity architectures trained on very large datasets.
3. Architecture
DPT retains an encoder-decoder structure while using ViT tokens as its backbone representation. It converts tokens from multiple transformer stages into multiresolution feature maps and progressively fuses them into dense predictions.
- Architecture overview: DPT uses vision transformers as the encoder backbone and transforms their token representations into dense predictions within an encoder-decoder architecture.
- Transformer encoder: ViT represents embedded image patches or extracted image features as tokens and transforms them through sequential multi-headed self-attention blocks.
- Transformer encoder: The transformer preserves token count and spatial resolution while giving every token a global receptive field after the initial embedding.
- Fusion decoder: Fusion modules combine consecutive feature maps with residual convolutional units and progressively upsample them by a factor of two before a task-specific output head produces the prediction.The final representation has half the input resolution before the task-specific head.
- Transformer encoder: ViT embeds non-overlapping image patches with a linear projection, adds positional embeddings, and includes a special readout token for global image representation.
- Reassemble: The Reassemble operation maps Np + 1 tokens to Np spatial tokens, reshapes them into feature maps, and resamples their dimensions and spatial resolution.
- Implementation: DPT supports varying image sizes when the image dimensions are divisible by the patch size, producing a corresponding variable number of tokens.
4. Experiments
DPT is evaluated on monocular depth estimation and semantic segmentation, with comparisons against convolutional networks across large and smaller datasets. Results include strong cross-dataset depth transfer, competitive smaller-dataset performance, improved visual quality, and ablation findings on feature selection and backbones.
- Experimental setup: DPT is evaluated on monocular depth estimation and semantic segmentation, including experiments on large and smaller datasets.The experiments compare DPT with convolutional networks and examine multiple architectural choices.
- Monocular depth estimation: 28% average relative improvement over MiDaS is achieved by DPT-Large in zero-shot cross-dataset depth transfer.DPT-Hybrid exceeds 23% average relative improvement, while DPT-Large is about three times larger than MiDaS and has similar latency.
- Monocular depth estimation: DPT variants still strongly outperform the retrained fully-convolutional MiDaS network on the larger MIX 6 dataset.This indicates that DPT benefits more from increased training-set size than the compared fully-convolutional network.
- Monocular depth estimation: DPT improves fine-detail reconstruction and global coherence in challenging depth-estimation regions compared with the convolutional architecture.Visual comparisons highlight fine details and relative depth arrangement across images.
- Monocular depth estimation: DPT matches or improves state-of-the-art performance on NYUv2 and KITTI across all reported metrics.These results support applying DPT to smaller monocular depth-prediction datasets.
- Semantic segmentation: DPT-Hybrid outperforms existing fully-convolutional architectures on ADE20K, while DPT predictions show cleaner and finer-grained object boundaries.DPT-Large performs slightly worse on ADE20K, likely because the dataset is substantially smaller than those used in earlier experiments.
- Semantic segmentation: Fine-tuning DPT-Hybrid on Pascal Context provides strong performance on the smaller dataset.The model is fine-tuned for 50 epochs using the same other hyper-parameters as in the preceding experiments.
- Ablations: Ablations favor combining low-level and deep transformer features, using embedding-network low-level features in the hybrid model, and projecting the readout token.ViT-Large performs best but is nearly three times larger, while ViT-Hybrid offers a favorable accuracy-capacity trade-off; DeIT distillation also improves performance over ViT-Base.
5. Conclusion
DPT leverages vision transformers for dense prediction and produces more fine-grained, globally coherent predictions than fully-convolutional architectures, especially with large-scale training data.
- DPT effectively leverages vision transformers for monocular depth estimation and semantic segmentation.
- DPT produces more fine-grained and globally coherent predictions than fully-convolutional architectures.
- Large-scale datasets allow DPT to realize its full potential, consistent with prior work on transformers.
A. Architecture details
The supplementary architecture details describe the hybrid encoder, residual convolutional decoder units, and task-specific output heads for depth estimation and semantic segmentation.
- Hybrid encoder: The hybrid encoder uses a preactivation ResNet50 with group normalization and weight standardization, tapping skip connections after its first and second stages.
- Residual convolutional units: Residual convolutional units form the decoder, with batch normalization enabled for semantic segmentation but disabled for monocular depth estimation.
- Monocular depth estimation head: The monocular depth head projects features to a non-negative inverse-depth scalar for every pixel and upsamples using bilinear interpolation.
- Semantic segmentation head: The semantic segmentation head projects features to class logits, applies dropout at rate 0.1, and performs final bilinear upsampling.
B. Additional results
Additional experiments report finer-grained and more globally arranged depth predictions, broad per-class segmentation improvements, and increasingly global attention in deeper transformer layers.
- Monocular depth estimation: Dense, high-resolution evaluation datasets receive the largest zero-shot depth-transfer gains, plausibly because DPT produces more fine-grained predictions.
- Monocular depth estimation: DPT predictions show more details and better global depth arrangement than the fully-convolutional baseline at the same 384-pixel input resolution.
- Semantic segmentation: Semantic segmentation generally improves per-class IoU over the baseline, without a strong pattern across classes.
- Attention maps: Attention is more spatially localized in shallow layers, whereas deeper layers frequently attend across the whole image.