PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformers for Dense Prediction

René Ranftl, Alexey Bochkovskiy, Vladlen Koltun

arXiv:2103.13413v1cs.CV

TL;DR

Dense prediction needs spatial detail and broad context, but convolutional backbones commonly lose resolution through downsampling. DPT uses a vision-transformer backbone, reassembles tokens into multiresolution features, and fuses them with a convolutional decoder; it improves depth estimation and achieves state-of-the-art results across depth and segmentation benchmarks.

  • Problem

    Dense prediction requires fine-grained spatial information, while convolutional backbones lose feature resolution and granularity through downsampling.

    Source: §1 · p. 1

  • Method

    DPT replaces the convolutional backbone with a vision transformer, reassembles tokens from multiple stages into image-like multiresolution features, and progressively combines them with a convolutional decoder.

    Source: §1 · p. 2, §3 · p. 2

  • Results

    DPT produces more fine-grained and globally coherent predictions, including more than 28% relative improvement over MiDaS for zero-shot monocular depth transfer and state-of-the-art results on depth and segmentation datasets.

    Source: §1 · p. 2, §4.1 · p. 5, §5 · p. 8

  • Takeaways & Limitations

    DPT is especially effective when trained with large-scale data and can also be fine-tuned on smaller depth and segmentation datasets.

    Source: §1 · p. 2, §4.1 · p. 4, §5 · p. 8

Abstract

from arXiv · show

We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense vision transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network. When applied to semantic segmentation, dense vision transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art. Our models are available at https://github.com/intel-isl/DPT.

1. Introduction

Dense prediction models typically use convolutional encoder-decoders, but downsampling can discard the fine spatial information these tasks require. DPT replaces the convolutional backbone with a vision transformer to preserve higher-resolution representations and global context.

  • Convolutional dense-prediction architectures commonly separate an ImageNet-pretrained encoder from a feature-aggregating decoder.Source: §1 · p. 1
  • Information lost in the encoder cannot be recovered by the decoder, making backbone choice central to overall model capability.Source: §1 · p. 1
  • Downsampling expands receptive fields and keeps computation tractable, but loses deep-stage feature resolution and granularity needed for dense prediction.Source: §1 · pp. 1–2
  • Higher input resolution, dilated convolutions, skip connections, and parallel multiresolution features mitigate granularity loss but remain constrained by memory.Source: §1 · pp. 1–2
  • DPT uses a vision-transformer backbone, reassembles tokens into multiresolution image-like features, and progressively combines them with a convolutional decoder.Source: §1 · p. 2
  • More than 28% performance improvement is reported for monocular depth estimation against the top-performing fully-convolutional network, while DPT also reaches state-of-the-art results on several segmentation and depth datasets.Source: §1 · p. 2

2. Related Work

Prior dense-prediction systems overwhelmingly use convolution and subsampling to build multiscale context. The related literature explores improved aggregation, dilation, parallel multiscale processing, and attention-based alternatives.

  • Fully-convolutional networks remain the prototypical dense-prediction architecture, using convolution and subsampling to learn multiscale representations with broad context.Source: §2 · p. 2
  • Existing convolutional variants progressively upsample pooled representations, use dilated convolutions, or aggregate features in parallel across multiple scales.Source: §2 · p. 2
  • Transformers are self-attention-based set-to-set models that have been especially successful as high-capacity architectures trained on very large datasets.Source: §2 · p. 2

3. Architecture

DPT retains an encoder-decoder structure while using ViT tokens as its backbone representation. It converts tokens from multiple transformer stages into multiresolution feature maps and progressively fuses them into dense predictions.

Architecture overview

  • DPT uses vision transformers as the encoder backbone and transforms their token representations into dense predictions within an encoder-decoder architecture.Source: §3 · p. 2

Transformer encoder

  • ViT represents embedded image patches or extracted image features as tokens and transforms them through sequential multi-headed self-attention blocks.Source: §3 · p. 2
  • The transformer preserves token count and spatial resolution while giving every token a global receptive field after the initial embedding.Source: §3 · p. 2

Fusion decoder

  • Fusion modules combine consecutive feature maps with residual convolutional units and progressively upsample them by a factor of two before a task-specific output head produces the prediction.The final representation has half the input resolution before the task-specific head.Source: Figure 1 · p. 3, §3 · p. 4

Transformer encoder

  • ViT embeds non-overlapping image patches with a linear projection, adds positional embeddings, and includes a special readout token for global image representation.Source: §3 · p. 3

Reassemble

  • The Reassemble operation maps Np + 1 tokens to Np spatial tokens, reshapes them into feature maps, and resamples their dimensions and spatial resolution.Source: §3 · pp. 3–4

Implementation

  • DPT supports varying image sizes when the image dimensions are divisible by the patch size, producing a corresponding variable number of tokens.Source: §3 · p. 4

4. Experiments

DPT is evaluated on monocular depth estimation and semantic segmentation, with comparisons against convolutional networks across large and smaller datasets. Results include strong cross-dataset depth transfer, competitive smaller-dataset performance, improved visual quality, and ablation findings on feature selection and backbones.

Experimental setup

  • DPT is evaluated on monocular depth estimation and semantic segmentation, including experiments on large and smaller datasets.The experiments compare DPT with convolutional networks and examine multiple architectural choices.Source: §4 · p. 4, §4.1 · p. 4, §4.2 · p. 6

Monocular depth estimation

  • 28% average relative improvement over MiDaS is achieved by DPT-Large in zero-shot cross-dataset depth transfer.DPT-Hybrid exceeds 23% average relative improvement, while DPT-Large is about three times larger than MiDaS and has similar latency.Source: §4.1 · p. 5
  • DPT variants still strongly outperform the retrained fully-convolutional MiDaS network on the larger MIX 6 dataset.This indicates that DPT benefits more from increased training-set size than the compared fully-convolutional network.Source: §4.1 · p. 5
  • DPT improves fine-detail reconstruction and global coherence in challenging depth-estimation regions compared with the convolutional architecture.Visual comparisons highlight fine details and relative depth arrangement across images.Source: §4.1 · p. 5
  • DPT matches or improves state-of-the-art performance on NYUv2 and KITTI across all reported metrics.These results support applying DPT to smaller monocular depth-prediction datasets.Source: §4.1 · p. 6

Semantic segmentation

  • DPT-Hybrid outperforms existing fully-convolutional architectures on ADE20K, while DPT predictions show cleaner and finer-grained object boundaries.DPT-Large performs slightly worse on ADE20K, likely because the dataset is substantially smaller than those used in earlier experiments.Source: §4.2 · p. 6
  • Fine-tuning DPT-Hybrid on Pascal Context provides strong performance on the smaller dataset.The model is fine-tuned for 50 epochs using the same other hyper-parameters as in the preceding experiments.Source: §4.2 · p. 6

Ablations

  • Ablations favor combining low-level and deep transformer features, using embedding-network low-level features in the hybrid model, and projecting the readout token.ViT-Large performs best but is nearly three times larger, while ViT-Hybrid offers a favorable accuracy-capacity trade-off; DeIT distillation also improves performance over ViT-Base.Source: §4.3 · pp. 7–8

5. Conclusion

DPT leverages vision transformers for dense prediction and produces more fine-grained, globally coherent predictions than fully-convolutional architectures, especially with large-scale training data.

  • DPT effectively leverages vision transformers for monocular depth estimation and semantic segmentation.Source: §5 · p. 8
  • DPT produces more fine-grained and globally coherent predictions than fully-convolutional architectures.Source: §5 · p. 8
  • Large-scale datasets allow DPT to realize its full potential, consistent with prior work on transformers.Source: §5 · p. 8

A. Architecture details

The supplementary architecture details describe the hybrid encoder, residual convolutional decoder units, and task-specific output heads for depth estimation and semantic segmentation.

Hybrid encoder

  • The hybrid encoder uses a preactivation ResNet50 with group normalization and weight standardization, tapping skip connections after its first and second stages.Source: A. Architecture details · p. 11

Residual convolutional units

  • Residual convolutional units form the decoder, with batch normalization enabled for semantic segmentation but disabled for monocular depth estimation.Source: A. Architecture details · p. 11

Monocular depth estimation head

  • The monocular depth head projects features to a non-negative inverse-depth scalar for every pixel and upsamples using bilinear interpolation.Source: A. Architecture details · p. 11

Semantic segmentation head

  • The semantic segmentation head projects features to class logits, applies dropout at rate 0.1, and performs final bilinear upsampling.Source: A. Architecture details · p. 11

B. Additional results

Additional experiments report finer-grained and more globally arranged depth predictions, broad per-class segmentation improvements, and increasingly global attention in deeper transformer layers.

Monocular depth estimation

  • Dense, high-resolution evaluation datasets receive the largest zero-shot depth-transfer gains, plausibly because DPT produces more fine-grained predictions.Source: B. Additional results · p. 11
  • DPT predictions show more details and better global depth arrangement than the fully-convolutional baseline at the same 384-pixel input resolution.Source: B. Additional results · p. 11

Semantic segmentation

  • Semantic segmentation generally improves per-class IoU over the baseline, without a strong pattern across classes.Source: B. Additional results · p. 11

Attention maps

  • Attention is more spatially localized in shallow layers, whereas deeper layers frequently attend across the whole image.Source: B. Additional results · p. 11

Related papers

Loading 2103.13413v1…