PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformer Adapter for Dense Predictions

Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, Yu Qiao

arXiv:2205.08534v4cs.CV

TL;DR

Plain ViT has weak prior assumptions for dense prediction, motivating an adapter that adds image-related inductive biases without altering the backbone. ViT-Adapter uses spatial and multi-scale feature interactions, matches or exceeds vision-specific transformers across dense prediction tasks, and benefits from multi-modal pre-training.

  • Problem

    Plain ViT has weak prior assumptions and inferior performance on dense prediction tasks compared with vision-specific transformers.

    Source: §3.3 · p. 5, §5 · p. 9

  • Method

    ViT-Adapter is a pre-training-free additional network that injects spatial priors and reconstructs multi-scale features while leaving the plain ViT architecture unchanged.

    Source: §1 · p. 2

  • Results

    Across object detection, instance segmentation, and semantic segmentation, ViT-Adapter achieves comparable or better performance than vision-specific transformers and benefits from multi-modal pre-training.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    ViT-Adapter provides a flexible way to use plain ViT backbones for dense prediction while introducing vision-specific inductive biases during task transfer.

    Source: §1 · p. 2, §5 · p. 9

    ViTDet's box AP drops 4.0 points when regular ImageNet-22K pre-training replaces MAE pre-training.

    Source: §2 · p. 3

Abstract

from arXiv · show

This work investigates a simple yet powerful dense prediction task adapter for Vision Transformer (ViT). Unlike recently advanced variants that incorporate vision-specific inductive biases into their architectures, the plain ViT suffers inferior performance on dense predictions due to weak prior assumptions. To address this issue, we propose the ViT-Adapter, which allows plain ViT to achieve comparable performance to vision-specific transformers. Specifically, the backbone in our framework is a plain ViT that can learn powerful representations from large-scale multi-modal data. When transferring to downstream tasks, a pre-training-free adapter is used to introduce the image-related inductive biases into the model, making it suitable for these tasks. We verify ViT-Adapter on multiple dense prediction tasks, including object detection, instance segmentation, and semantic segmentation. Notably, without using extra detection data, our ViT-Adapter-L yields state-of-the-art 60.9 box AP and 53.0 mask AP on COCO test-dev. We hope that the ViT-Adapter could serve as an alternative for vision-specific transformers and facilitate future research. The code and models will be released at https://github.com/czczup/ViT-Adapter.

1 INTRODUCTION

ViT-Adapter adds image-related inductive biases to a plain ViT through a pre-training-free adapter without modifying the backbone architecture. Across dense prediction benchmarks, it reaches comparable or better performance than vision-specific transformers and benefits from multi-modal pre-training.

Contribution

  • ViT-Adapter is a pre-training-free network that adapts plain ViT backbones to downstream dense prediction tasks without changing their original architecture.It introduces image-related inductive biases through an additional adapter.Source: §1 · p. 2
  • The adapter uses a spatial prior module, spatial feature injector, and multi-scale feature extractor to support dense prediction.These modules capture local image semantics, inject spatial priors into ViT, and reconstruct task-required multi-scale features.Source: §1 · p. 2

Method

  • The framework combines a general-purpose plain ViT backbone with a randomly initialized adapter, allowing pre-training on images or multi-modal data.The adapter introduces image-related prior knowledge during transfer to dense prediction tasks.Source: §1 · p. 2

Results

  • ViT-Adapter-B reaches 49.6 box AP on COCO val with ImageNet-1K pre-training, outperforming Swin-B by 1.0 points.This comparison uses only ImageNet-1K pre-training.Source: §1 · p. 2
  • ViT-Adapter-L reaches 60.9 box AP on COCO test-dev without extra detection data, reported as the best record under that setting.The result benefits from multi-modal pre-training.Source: §1 · p. 2

2 RELATED WORK

Prior work adapts ViT to dense prediction through vision-specific architectures, task-specific decoders, or general adapter modules. These approaches address task needs but leave plain ViT's single-scale, low-resolution representation weakness or do not target this setting directly.

Transformers

  • Vision transformers such as PVT and Swin introduce CNN-inspired pyramid structures as vision-specific inductive biases.Other models combine convolutional and transformer components, while plain ViT remains a general-purpose architecture.Source: §2 · p. 3

Decoders for ViT

  • ViT dense-prediction decoders use CNN or transformer components to translate encoder features into task outputs.SETR, Segmenter, and DPT apply this encoder-decoder strategy across semantic segmentation and depth estimation.Source: §2 · p. 3
  • These task-specific decoders improve dense prediction but retain ViT's weakness of single-scale and low-resolution representations.The limitation is identified in the related-work discussion.Source: §2 · p. 3

Adapters

  • Adapters originated as added modules for task-specific fine-tuning in NLP and were later used in computer vision for incremental and domain adaptation.The cited vision adapters address settings different from dense prediction adaptation.Source: §2 · p. 3

Related comparison

  • Using regular ImageNet-22K pre-training instead of MAE causes ViTDet's box AP to drop by 4.0 points.This footnote marks a pre-training sensitivity in the related comparison.Source: §2 · p. 3

3 VISION TRANSFORMER ADAPTER

ViT-Adapter augments a plain ViT with image-specific spatial priors and feature interactions while preserving the original backbone architecture. Its modules inject spatial information and produce multi-scale features for dense prediction.

Overall architecture

  • The architecture combines a plain ViT with a spatial prior module, spatial feature injector, and multi-scale feature extractor.The adapter captures spatial features from the image, injects them into ViT, and extracts hierarchical features.Source: §3.1 · p. 4

Multi-scale features

  • After repeated feature interactions, the adapter reshapes features at 1/8, 1/16, and 1/32 scales and upsamples the 1/8 feature to construct a 1/4-scale map.Together these maps form a feature pyramid for dense prediction.Source: §3.1 · p. 4, §3.2 · p. 5

Spatial Prior Module

  • The Spatial Prior Module models local image contexts in parallel with patch embedding without altering ViT’s original architecture.It uses convolutional processing to obtain feature maps at 1/8, 1/16, and 1/32 resolutions.Source: §3.2 · p. 4

Feature interaction

  • Spatial priors are injected into ViT blocks through cross-attention, with sparse attention reducing computational cost.A learnable vector initialized to 0 balances the attention output and input feature, preserving the pre-trained feature distribution during adaptation.Source: §3.3 · p. 5
  • The multi-scale feature extractor uses cross-attention and an FFN to derive hierarchical features from ViT block outputs.The generated spatial feature is reused as input to the next spatial feature injector.Source: §3.3 · p. 5

Model variants

  • The adapter is implemented for ViT-T, ViT-S, ViT-B, and ViT-L with parameter counts of 2.5M, 5.8M, 14.0M, and 23.7M, respectively.The default sparse attention is deformable attention with four sampling points.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments evaluate ViT-Adapter across object detection, instance segmentation, and semantic segmentation under multiple pre-training settings and frameworks. The results show that its components improve plain ViT features and performance across dense prediction tasks.

Experimental settings

  • ViT-Adapter is evaluated on COCO and ADE20K using multiple detection and segmentation frameworks under controlled pre-training settings.The experiments include Mask R-CNN, Cascade Mask R-CNN, ATSS, GFL, Semantic FPN, and UperNet.Source: §4.1 · pp. 5–6, §4.2 · p. 7, §4.3 · p. 8

Semantic segmentation

  • 47.1 MS mIoU is achieved by ViT-Adapter-S with UperNet, while ViT-Adapter-B reaches 49.7 MS mIoU on ADE20K with ImageNet-1K pre-training.The ViT-Adapter-B result is 2.6 points higher than ViT-B and on par with Swin-B and Twins-SVT-L.Source: §4.2 · p. 8
  • Multi-modal pre-training improves ViT-Adapter-L semantic segmentation by 1.3 mIoU with Semantic FPN and 1.6 mIoU with UperNet.Source: §4.2 · p. 8

Ablation study

  • The spatial prior, feature injector, and multi-scale extractor collectively improve ViT-S by 4.5 APb and 2.8 APm over the baseline.Their individual gains are 1.4 APb and 0.9 APm, 1.0 APb and 0.8 APm, and 2.1 APb and 1.1 APm, respectively.Source: §4.4 · p. 8

5 CONCLUSION

ViT-Adapter bridges the performance gap between plain ViT and vision-specific transformers for dense prediction without modifying ViT’s architecture. It injects image-related inductive biases and reconstructs fine-grained multi-scale features.

  • ViT-Adapter injects image-related inductive biases into plain ViT without modifying its inherent architecture.Source: §5 · p. 9
  • The method reconstructs fine-grained multi-scale features required for dense prediction tasks.Source: §5 · p. 9
  • Extensive experiments cover object detection, instance segmentation, and semantic segmentation.Source: §5 · p. 9

A COMPARISON WITH PREVIOUS STATE-OF-THE-ARTS

Plain ViT is rarely used in current dense prediction state-of-the-art models, which are primarily vision-specific transformers. The experiments test whether ViT-Adapter enables plain ViT to reach leading performance.

A.1 OBJECT DETECTION AND INSTANCE SEGMENTATION

Object detection and instance segmentation experiments compare ViT-Adapter-L with prior methods on COCO test-dev under ImageNet-22K and advanced pre-training. Advanced pre-training produces the strongest reported results.

ImageNet-22K pre-training

  • 58.4 APb and 50.7 APm are achieved by ViT-Adapter-L on COCO test-dev with ImageNet-22K supervised pre-training.Source: §A.1 · p. 15

More advanced pre-training

  • 60.4 APb and 52.5 APm are obtained with BEiT pre-training, while BEiTv2 raises the results to 60.9 APb and 53.0 APm.Source: §A.1 · p. 16
  • The BEiTv2-based method attains similar performance to ViTDet using 36 training epochs rather than 100.The comparison is explicitly described as not perfectly controlled.Source: §A.1 · p. 16

A.2 SEMANTIC SEGMENTATION

The ViT-Adapter is evaluated for semantic segmentation across ImageNet-22K and more advanced pre-training settings, achieving performance comparable to or better than vision-specific models. Further transfer strategies and multimodal pre-training produce stronger ADE20K results.

Settings

  • Semantic-segmentation training uses AdamW with a 2×10^-5 learning rate, batch size 16, weight decay 0.05, layer-wise decay 0.9, and drop path 0.4.Other settings, including pre-training techniques, crop size, and iterations, are listed in Table 9.Source: §A.2 · p. 16

Results with ImageNet-22K Pre-training

  • 56.8 mIoU and 57.7 MS mIoU are achieved by ViT-Adapter-L on ADE20K val with ImageNet-22K pre-training.Using Mask2Former as the segmenter, these results are comparable to several recent vision-specific models.Source: §A.2 · p. 16

Results with More Advanced Pre-training

  • 58.4 MS mIoU is achieved by ViT-Adapter-L with BEiT, outperforming BEiT-L by 1.4 points with 10M additional parameters.The result uses UperNet for 160k iterations.Source: §A.2 · p. 16
  • 60.5 MS mIoU is achieved by ViT-Adapter-L with BEiT under COCO-Stuff pre-training followed by ADE20K fine-tuning.The total training schedule remains 160k iterations without additional training overhead.Source: §A.2 · p. 17
  • 61.5 MS mIoU is achieved by ViT-Adapter-L with BEiTv2, slightly exceeding FD-SwinV2-G while using 571M rather than 3.0B parameters.BEiT-3 with ViT-Adapter is also reported to reach 62.8 MS mIoU on ADE20K val.Source: §A.2 · p. 17

B ADDITIONAL ABLATION AND DISCUSSION

Additional analyses attribute ViT-Adapter’s dense-prediction gains to improved feature structure and image-related priors, while contrasting its task focus with other adapters and its training cost with ViTDet.

TIDE Error Type Analysis

  • The adapter reduces missed-detection and localization errors relative to the ViT baseline, while substantially affecting false-negative and false-positive errors.The analysis uses TIDE and compares models listed in Table 1.Source: B ADDITIONAL ABLATION AND DISCUSSION · p. 17

Feature Visualization

Comparison with SETR

  • ViT-Adapter differs from SETR by incorporating input-image information in addition to task-specific priors.The supplied comparison passage identifies this as one of three main differences.Source: B ADDITIONAL ABLATION AND DISCUSSION · p. 18

Comparison with other Adapters

  • VPT, Visual Prompt, AdaptFormer, NOAH, and Convpass target classification, whereas ViT-Adapter is designed for dense prediction tasks.The cited comparison distinguishes the task settings of these adapter families.Source: B ADDITIONAL ABLATION AND DISCUSSION · p. 18
  • These classification-oriented adapters perform below expectations when applied to dense prediction, with VPT showing a large gap to ViT-L on ADE20K.The comparison refers to Table 11.Source: B ADDITIONAL ABLATION AND DISCUSSION · p. 18, Table 11 · p. 18
  • ViT-Adapter and parameter-efficient adapters pursue different targets: dense-prediction performance and generality versus efficient transfer learning.The paper describes these technical lines as orthogonal and suggests combining them as future work.Source: B ADDITIONAL ABLATION AND DISCUSSION · p. 18

ViTDet Comparison

  • ViTDet’s stronger result uses MAE weights and Large Scale Jitter for 100 epochs, costing almost three times as much as the 36-epoch schedule.The comparison indicates that plain ViT’s missing image-related biases can slow dense-prediction convergence.Source: B ADDITIONAL ABLATION AND DISCUSSION · p. 18
  • ViT-Adapter outperforms ViTDet under the same 3×+MS schedule and ImageNet-1K initialization by injecting image-related priors.The authors connect this injection with faster convergence and improved performance.Source: B ADDITIONAL ABLATION AND DISCUSSION · p. 18

Related papers

Loading 2205.08534v4…