PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual Vision Transformer

Ting Yao, Yehao Li, Yingwei Pan, Yu Wang, Xiao-Ping Zhang, Tao Mei

arXiv:2207.04976v2cs.CVcs.AI

TL;DR

Dense self-attention makes high-resolution Transformer training costly, while regional methods using down-sampling can show performance drops. Dual-ViT links compressed global semantics with pixel-level features, and reports competitive results with reduced computation.

  • Problem

    Dense self-attention makes high-resolution Transformer training slow, while down-sampling approaches can incur performance drops.

    Source: §1 · p. 1

  • Method

    Dual-ViT uses compressed semantic tokens as global prior information to guide finer feature extraction in an interacting pixel pathway.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    Across vision tasks, Dual-ViT reports superiority to state-of-the-art ViTs; on ImageNet it achieves 85.7% top-1 accuracy with 41.1% FLOPs and 37.8% parameters compared to VOLO [11].

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Dual-ViT is presented as a multi-scale backbone pursuing a better accuracy-latency trade-off.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Prior works have proposed several strategies to reduce the computational cost of self-attention mechanism. Many of these works consider decomposing the self-attention procedure into regional and local feature extraction procedures that each incurs a much smaller computational complexity. However, regional information is typically only achieved at the expense of undesirable information lost owing to down-sampling. In this paper, we propose a novel Transformer architecture that aims to mitigate the cost issue, named Dual Vision Transformer (Dual-ViT). The new architecture incorporates a critical semantic pathway that can more efficiently compress token vectors into global semantics with reduced order of complexity. Such compressed global semantics then serve as useful prior information in learning finer pixel level details, through another constructed pixel pathway. The semantic pathway and pixel pathway are then integrated together and are jointly trained, spreading the enhanced self-attention information in parallel through both of the pathways. Dual-ViT is henceforth able to reduce the computational complexity without compromising much accuracy. We empirically demonstrate that Dual-ViT provides superior accuracy than SOTA Transformer architectures with reduced training complexity. Source code is available at \url{https://github.com/YehLi/ImageNetModel}.

1 INTRODUCTION

Dense self-attention makes high-resolution Transformer training costly, while down-sampling-based regional methods can lose performance. Dual-ViT couples compressed global semantics with pixel-level features through two interacting pathways.

1 INTRODUCTION

  • Dense self-attention updates each token by attending to all tokens, making Transformer training slow for high-resolution inputs.Source: §1 · p. 1
  • Down-sampling-based regional methods save computation but have shown evident performance drops.Source: §1 · p. 1
  • Dual-ViT uses a semantic pathway to provide global prior information for finer pixel-level feature extraction, integrating and jointly training both pathways.Source: §1 · p. 1
  • 85.7% top-1 accuracy on ImageNet with 41.1% FLOPs and 37.8% parameters compared to VOLO [11]; on COCO, it improves PVT [8] mAP by more than 1.2% and 0.9%, with 48.0% fewer parameters.Source: §1 · p. 2

2 RELATED WORK

The related work traces CNN backbones through deeper, multibranch, and multiscale designs, then reviews Transformer strategies for managing resolution and attention cost. Dual-ViT is distinguished by modeling global semantics and finer features in interacting pathways.

2.1 Convolutional Neural Networks Backbones

  • CNN backbones developed through deeper networks, skip connections, multibranch and multiscale designs, with EfficientNet balancing width, depth, and resolution.Source: §2.1 · p. 2

2.2 Vision Transformers

  • Vision Transformer successors address resolution and attention cost through pooling, windows, down-sampling, and regional/local attention strategies.Source: §2.2 · p. 2
  • Unlike multi-scale ViTs relying on local attention or down-sampling, Dual-ViT models global semantics and finer features in two pathways to reduce attention tokens.Source: §2.2 · p. 3

3 METHOD

Dual-ViT splits high-resolution self-attention into semantic and pixel pathways, then merges them for later-stage attention. The design uses global semantics as a prior for pixel-level features while reducing attention cost.

3.1 Conventional Transformer

  • Pooling-based attention reduction can lose information, local windows limit global dependencies, and larger patches lower feature-map resolution.Conventional dense attention costs O(n^2d), making high-resolution inputs expensive.Source: §3.1 · p. 3

3.2 Dual Block

  • Dual blocks use parametric semantic queries to extract global tokens, which guide pixel-feature refinement through cross-attention at reduced cost.The stated complexity is O(nmd + m^2d), where m is the number of semantic tokens; both pathways are jointly trained to address information loss and finer-feature extraction.Source: §3.2 · p. 4

3.3 Merge Block

  • Merge blocks concatenate semantic and pixel tokens for self-attention in the final two, lower-resolution stages, with separate feed-forward layers for each pathway.Global average pooling over the output tokens produces the classification token.Source: §3.3 · p. 4

3.4 Dual Vision Transformer

  • Dual-ViT stacks Dual blocks in its first two stages and Merge blocks in its last two, with Small, Base, and Large variants at similar scale to Swin-T/S/B.Patch embedding increases channels and shrinks spatial resolution at each stage's start.Source: §3.4 · p. 4

3.5 Differences between Our Dual-ViT and Previous Vision Transformers

  • Unlike RegionViT, Dual-ViT uses flexible holistic semantic tokens as priors; unlike Twins, its semantic tokens summarize the whole feature map.Unlike CrossViT, Dual-ViT interacts between pathways within each block and throughout the process, rather than only integrating final features with a single CLS token.Source: §3.5 · pp. 4–5

4 EXPERIMENTS

Experiments evaluate Dual-ViT on ImageNet and downstream COCO and ADE20K tasks, including ablations and comparisons with other vision backbones. The reported results cover multiple model sizes, resolutions, and vision tasks.

4.2 Experimental Settings of Each Vision Task

  • ImageNet models are trained from scratch, while downstream detectors and segmenters use ImageNet-pretrained backbones fine-tuned on COCO or ADE20K.The downstream tasks include object detection, instance segmentation, and semantic segmentation.Source: §4 · p. 5, §4.2 · pp. 5–6

4.1 Datasets

  • The experiments use ImageNet with 1.28 million training and 50K validation images, COCO with about 118K training and 5K validation images, and ADE20K with 25K images.ADE20K contains 150 semantic categories.Source: §4.1 · p. 5

4.3 Ablation Study of Dual Block

  • In the Dual block ablation, removing semantic self-attention or feed-forward layers lowers performance, and self-attention before cross-attention outperforms the reverse order.The comparison fixes Dual-ViT-S and replaces blocks in the first two stages.Source: §4.3 · p. 6, Figure 3 · p. 6

4.4 Performance Comparisons with State-of-the-Art Vision Backbones

  • At typical resolution, Dual-ViT-L† reaches 85.7% top-1 accuracy using 41.1% GFLOPs and 37.8% parameters against VOLO.The authors report superior performance with reduced computational cost across most model sizes.Source: Table 2 · p. 7, §4.4 · p. 7
  • At 384 × 384, Dual-ViT backbones consistently outperform state-of-the-art vision backbones with reduced computational cost across most model sizes.The backbones are fine-tuned at higher resolution for this comparison.Source: Table 3 · p. 7, §4.4 · p. 8
  • On COCO, Dual-ViT-S/B lead all metrics at each size; Dual-ViT-S tops PVTv2-B2 APb by 1.3%/1.1%/1.1%/1.3% across four detectors.The four detectors are Cascade Mask R-CNN, ATSS, GFL, and Sparse RCNN.Source: Table 5 · p. 8, §4.4 · pp. 8–9, Table 4 · p. 8
  • On ADE20K with UPerNet, Dual-ViT-S/B perform more than 2.1%/2.2% better than the best competitors, Twins-SVT-S/Swin-S, respectively.The reported metric is mean IoU averaged over all classes.Source: Table 6 · p. 8, §4.4 · p. 9

4.5 Visualization Analysis

  • Score-CAM visualizations show Dual-ViT-S capturing more meaningful pixels for target-object recognition than RegionViT-S and PVTv2-B2.The figure compares saliency maps for four ImageNet images.Source: §4.5 · p. 9, Figure 4 · p. 9

5 CONCLUSIONS

Dual-ViT uses interactive semantic and pixel pathways, with semantic tokens serving as prior information for finer local feature extraction; experiments report superiority over state-of-the-art ViTs across vision tasks.

5 CONCLUSIONS

  • Dual-ViT models self-attention in two interactive pathways: a pixel pathway for finer details and a semantic pathway for holistic global information.The learned semantic tokens serve as high-level prior information for local feature extraction in the pixel pathway.Source: §5 · p. 9
  • Semantic tokens from the semantic pathway serve as high-level prior information to facilitate finer local feature extraction in the pixel pathway.Source: §5 · p. 9
  • Extensive empirical results across vision tasks demonstrate Dual-ViT's superiority over state-of-the-art ViTs while pursuing a better accuracy-latency trade-off.Source: §5 · p. 9

Related papers

Loading 2207.04976v2…