PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformer Adapter for Dense Predictions

Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, Yu Qiao

arXiv:2205.08534v4cs.CV

TL;DR

Plain ViT has weak prior assumptions for dense prediction, motivating an adapter that adds image-related inductive biases without altering the backbone. ViT-Adapter uses spatial and multi-scale feature interactions, matches or exceeds vision-specific transformers across dense prediction tasks, and benefits from multi-modal pre-training.

  • Problem

    Plain ViT has weak prior assumptions and inferior performance on dense prediction tasks compared with vision-specific transformers.

  • Method

    ViT-Adapter is a pre-training-free additional network that injects spatial priors and reconstructs multi-scale features while leaving the plain ViT architecture unchanged.

  • Results

    Across object detection, instance segmentation, and semantic segmentation, ViT-Adapter achieves comparable or better performance than vision-specific transformers and benefits from multi-modal pre-training.

  • Takeaways & Limitations

    ViT-Adapter provides a flexible way to use plain ViT backbones for dense prediction while introducing vision-specific inductive biases during task transfer.

  • Takeaways & Limitations

    ViTDet's box AP drops 4.0 points when regular ImageNet-22K pre-training replaces MAE pre-training.

Abstract

from arXiv · show

This work investigates a simple yet powerful dense prediction task adapter for Vision Transformer (ViT). Unlike recently advanced variants that incorporate vision-specific inductive biases into their architectures, the plain ViT suffers inferior performance on dense predictions due to weak prior assumptions. To address this issue, we propose the ViT-Adapter, which allows plain ViT to achieve comparable performance to vision-specific transformers. Specifically, the backbone in our framework is a plain ViT that can learn powerful representations from large-scale multi-modal data. When transferring to downstream tasks, a pre-training-free adapter is used to introduce the image-related inductive biases into the model, making it suitable for these tasks. We verify ViT-Adapter on multiple dense prediction tasks, including object detection, instance segmentation, and semantic segmentation. Notably, without using extra detection data, our ViT-Adapter-L yields state-of-the-art 60.9 box AP and 53.0 mask AP on COCO test-dev. We hope that the ViT-Adapter could serve as an alternative for vision-specific transformers and facilitate future research. The code and models will be released at https://github.com/czczup/ViT-Adapter.

1 INTRODUCTION

ViT-Adapter adds image-related inductive biases to a plain ViT through a pre-training-free adapter without modifying the backbone architecture. Across dense prediction benchmarks, it reaches comparable or better performance than vision-specific transformers and benefits from multi-modal pre-training.

  • Contribution: ViT-Adapter is a pre-training-free network that adapts plain ViT backbones to downstream dense prediction tasks without changing their original architecture.It introduces image-related inductive biases through an additional adapter.
  • Contribution: The adapter uses a spatial prior module, spatial feature injector, and multi-scale feature extractor to support dense prediction.These modules capture local image semantics, inject spatial priors into ViT, and reconstruct task-required multi-scale features.
  • Method: The framework combines a general-purpose plain ViT backbone with a randomly initialized adapter, allowing pre-training on images or multi-modal data.The adapter introduces image-related prior knowledge during transfer to dense prediction tasks.
  • Results: ViT-Adapter-B reaches 49.6 box AP on COCO val with ImageNet-1K pre-training, outperforming Swin-B by 1.0 points.This comparison uses only ImageNet-1K pre-training.
  • Results: ViT-Adapter-L reaches 60.9 box AP on COCO test-dev without extra detection data, reported as the best record under that setting.The result benefits from multi-modal pre-training.

2 RELATED WORK

Prior work adapts ViT to dense prediction through vision-specific architectures, task-specific decoders, or general adapter modules. These approaches address task needs but leave plain ViT's single-scale, low-resolution representation weakness or do not target this setting directly.

  • Transformers: Vision transformers such as PVT and Swin introduce CNN-inspired pyramid structures as vision-specific inductive biases.Other models combine convolutional and transformer components, while plain ViT remains a general-purpose architecture.
  • Decoders for ViT: ViT dense-prediction decoders use CNN or transformer components to translate encoder features into task outputs.SETR, Segmenter, and DPT apply this encoder-decoder strategy across semantic segmentation and depth estimation.
  • Decoders for ViT: These task-specific decoders improve dense prediction but retain ViT's weakness of single-scale and low-resolution representations.The limitation is identified in the related-work discussion.
  • Adapters: Adapters originated as added modules for task-specific fine-tuning in NLP and were later used in computer vision for incremental and domain adaptation.The cited vision adapters address settings different from dense prediction adaptation.
  • Related comparison: Using regular ImageNet-22K pre-training instead of MAE causes ViTDet's box AP to drop by 4.0 points.This footnote marks a pre-training sensitivity in the related comparison.

3 VISION TRANSFORMER ADAPTER

ViT-Adapter augments a plain ViT with image-specific spatial priors and feature interactions while preserving the original backbone architecture. Its modules inject spatial information and produce multi-scale features for dense prediction.

  • Overall architecture: The architecture combines a plain ViT with a spatial prior module, spatial feature injector, and multi-scale feature extractor.The adapter captures spatial features from the image, injects them into ViT, and extracts hierarchical features.
  • Multi-scale features: After repeated feature interactions, the adapter reshapes features at 1/8, 1/16, and 1/32 scales and upsamples the 1/8 feature to construct a 1/4-scale map.Together these maps form a feature pyramid for dense prediction.
  • Spatial Prior Module: The Spatial Prior Module models local image contexts in parallel with patch embedding without altering ViT’s original architecture.It uses convolutional processing to obtain feature maps at 1/8, 1/16, and 1/32 resolutions.
  • Feature interaction: Spatial priors are injected into ViT blocks through cross-attention, with sparse attention reducing computational cost.A learnable vector initialized to 0 balances the attention output and input feature, preserving the pre-trained feature distribution during adaptation.
  • Feature interaction: The multi-scale feature extractor uses cross-attention and an FFN to derive hierarchical features from ViT block outputs.The generated spatial feature is reused as input to the next spatial feature injector.
  • Model variants: The adapter is implemented for ViT-T, ViT-S, ViT-B, and ViT-L with parameter counts of 2.5M, 5.8M, 14.0M, and 23.7M, respectively.The default sparse attention is deformable attention with four sampling points.

4 EXPERIMENTS

Experiments evaluate ViT-Adapter across object detection, instance segmentation, and semantic segmentation under multiple pre-training settings and frameworks. The results show that its components improve plain ViT features and performance across dense prediction tasks.

  • Experimental settings: ViT-Adapter is evaluated on COCO and ADE20K using multiple detection and segmentation frameworks under controlled pre-training settings.The experiments include Mask R-CNN, Cascade Mask R-CNN, ATSS, GFL, Semantic FPN, and UperNet.
  • Semantic segmentation: 47.1 MS mIoU is achieved by ViT-Adapter-S with UperNet, while ViT-Adapter-B reaches 49.7 MS mIoU on ADE20K with ImageNet-1K pre-training.The ViT-Adapter-B result is 2.6 points higher than ViT-B and on par with Swin-B and Twins-SVT-L.
  • Semantic segmentation: Multi-modal pre-training improves ViT-Adapter-L semantic segmentation by 1.3 mIoU with Semantic FPN and 1.6 mIoU with UperNet.
  • Ablation study: The spatial prior, feature injector, and multi-scale extractor collectively improve ViT-S by 4.5 APb and 2.8 APm over the baseline.Their individual gains are 1.4 APb and 0.9 APm, 1.0 APb and 0.8 APm, and 2.1 APb and 1.1 APm, respectively.

5 CONCLUSION

ViT-Adapter bridges the performance gap between plain ViT and vision-specific transformers for dense prediction without modifying ViT’s architecture. It injects image-related inductive biases and reconstructs fine-grained multi-scale features.

  • ViT-Adapter injects image-related inductive biases into plain ViT without modifying its inherent architecture.
  • The method reconstructs fine-grained multi-scale features required for dense prediction tasks.
  • Extensive experiments cover object detection, instance segmentation, and semantic segmentation.

A COMPARISON WITH PREVIOUS STATE-OF-THE-ARTS

Plain ViT is rarely used in current dense prediction state-of-the-art models, which are primarily vision-specific transformers. The experiments test whether ViT-Adapter enables plain ViT to reach leading performance.

  • Current state-of-the-art dense prediction models are primarily vision-specific transformers such as Swin, Focal, MViTv2, and SwinV2.
  • Plain ViT is rarely found among state-of-the-art dense prediction models.
  • The study conducts extensive additional experiments to evaluate whether plain ViT can reach leading performance with ViT-Adapter.

A.1 OBJECT DETECTION AND INSTANCE SEGMENTATION

Object detection and instance segmentation experiments compare ViT-Adapter-L with prior methods on COCO test-dev under ImageNet-22K and advanced pre-training. Advanced pre-training produces the strongest reported results.

  • ImageNet-22K pre-training: 58.4 APb and 50.7 APm are achieved by ViT-Adapter-L on COCO test-dev with ImageNet-22K supervised pre-training.
  • More advanced pre-training: 60.4 APb and 52.5 APm are obtained with BEiT pre-training, while BEiTv2 raises the results to 60.9 APb and 53.0 APm.
  • More advanced pre-training: The BEiTv2-based method attains similar performance to ViTDet using 36 training epochs rather than 100.The comparison is explicitly described as not perfectly controlled.

A.2 SEMANTIC SEGMENTATION

The ViT-Adapter is evaluated for semantic segmentation across ImageNet-22K and more advanced pre-training settings, achieving performance comparable to or better than vision-specific models. Further transfer strategies and multimodal pre-training produce stronger ADE20K results.

  • Settings: Semantic-segmentation training uses AdamW with a 2×10^-5 learning rate, batch size 16, weight decay 0.05, layer-wise decay 0.9, and drop path 0.4.Other settings, including pre-training techniques, crop size, and iterations, are listed in Table 9.
  • Results with ImageNet-22K Pre-training: 56.8 mIoU and 57.7 MS mIoU are achieved by ViT-Adapter-L on ADE20K val with ImageNet-22K pre-training.Using Mask2Former as the segmenter, these results are comparable to several recent vision-specific models.
  • Results with More Advanced Pre-training: 58.4 MS mIoU is achieved by ViT-Adapter-L with BEiT, outperforming BEiT-L by 1.4 points with 10M additional parameters.The result uses UperNet for 160k iterations.
  • Results with More Advanced Pre-training: 60.5 MS mIoU is achieved by ViT-Adapter-L with BEiT under COCO-Stuff pre-training followed by ADE20K fine-tuning.The total training schedule remains 160k iterations without additional training overhead.
  • Results with More Advanced Pre-training: 61.5 MS mIoU is achieved by ViT-Adapter-L with BEiTv2, slightly exceeding FD-SwinV2-G while using 571M rather than 3.0B parameters.BEiT-3 with ViT-Adapter is also reported to reach 62.8 MS mIoU on ADE20K val.

B ADDITIONAL ABLATION AND DISCUSSION

Additional analyses attribute ViT-Adapter’s dense-prediction gains to improved feature structure and image-related priors, while contrasting its task focus with other adapters and its training cost with ViTDet.

  • TIDE Error Type Analysis: The adapter reduces missed-detection and localization errors relative to the ViT baseline, while substantially affecting false-negative and false-positive errors.The analysis uses TIDE and compares models listed in Table 1.
  • Feature Visualization: ViT-Adapter produces more refined feature maps with richer local edges and textures than ViT-B for detection and segmentation.The visualization agrees with the paper’s Fourier analysis.
  • Comparison with SETR: ViT-Adapter differs from SETR by incorporating input-image information in addition to task-specific priors.The supplied comparison passage identifies this as one of three main differences.
  • Comparison with other Adapters: VPT, Visual Prompt, AdaptFormer, NOAH, and Convpass target classification, whereas ViT-Adapter is designed for dense prediction tasks.The cited comparison distinguishes the task settings of these adapter families.
  • Comparison with other Adapters: These classification-oriented adapters perform below expectations when applied to dense prediction, with VPT showing a large gap to ViT-L on ADE20K.The comparison refers to Table 11.
  • Comparison with other Adapters: ViT-Adapter and parameter-efficient adapters pursue different targets: dense-prediction performance and generality versus efficient transfer learning.The paper describes these technical lines as orthogonal and suggests combining them as future work.
  • ViTDet Comparison: ViTDet’s stronger result uses MAE weights and Large Scale Jitter for 100 epochs, costing almost three times as much as the 36-epoch schedule.The comparison indicates that plain ViT’s missing image-related biases can slow dense-prediction convergence.
  • ViTDet Comparison: ViT-Adapter outperforms ViTDet under the same 3×+MS schedule and ImageNet-1K initialization by injecting image-related priors.The authors connect this injection with faster convergence and improved performance.
Loading 2205.08534v4…