PaperlayerView exact revision on arXiv

Source-linked AI summary

FasterViT: Fast Vision Transformers with Hierarchical Attention

Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, Jose M. Alvarez, Jan Kautz, Pavlo Molchanov

arXiv:2306.06189v2cs.CVcs.AIcs.LG

TL;DR

Vision transformers offer long-range modeling but incur high costs at high resolution, while local windows limit cross-window communication. FasterViT combines early CNN processing with later transformer blocks and Hierarchical Attention, achieving a strong accuracy-throughput trade-off across vision tasks. The method also supports efficient cross-window modeling through dedicated carrier tokens.

  • Problem

    Vision-transformer self-attention is quadratically costly, and local-window methods make long-range cross-window interactions difficult for high-resolution images.

  • Method

    FasterViT combines convolutional blocks in early stages with transformer blocks using Hierarchical Attention and dedicated carrier tokens for efficient cross-window interaction.

  • Results

    FasterViT achieves a SOTA accuracy-throughput Pareto front and competitive performance across classification, detection, instance segmentation, and semantic segmentation.

  • Takeaways & Limitations

    Hierarchical Attention provides an efficient mechanism for modeling long-range dependencies while preserving throughput, including as a module for existing architectures.

  • Takeaways & Limitations

    The formulation assumes a square input feature map for simplicity, with height and width treated as equal.

Abstract

from arXiv · show

We design a new family of hybrid CNN-ViT neural networks, named FasterViT, with a focus on high image throughput for computer vision (CV) applications. FasterViT combines the benefits of fast local representation learning in CNNs and global modeling properties in ViT. Our newly introduced Hierarchical Attention (HAT) approach decomposes global self-attention with quadratic complexity into a multi-level attention with reduced computational costs. We benefit from efficient window-based self-attention. Each window has access to dedicated carrier tokens that participate in local and global representation learning. At a high level, global self-attentions enable the efficient cross-window communication at lower costs. FasterViT achieves a SOTA Pareto-front in terms of accuracy and image throughput. We have extensively validated its effectiveness on various CV tasks including classification, object detection and segmentation. We also show that HAT can be used as a plug-and-play module for existing networks and enhance them. We further demonstrate significantly faster and more accurate performance than competitive counterparts for images with high resolution. Code is available at https://github.com/NVlabs/FasterViT.

1 INTRODUCTION

FasterViT addresses the efficiency and cross-window limitations of vision transformers with a hybrid CNN-transformer architecture and Hierarchical Attention. It achieves a strong accuracy-throughput trade-off across classification and downstream vision tasks.

  • 1 INTRODUCTION: Self-attention has quadratic computational complexity, while local-window approaches make cross-window and long-range interactions difficult, especially for high-resolution inputs.The passage also notes that early-stage self-attention can reduce throughput because high-resolution features create many local windows.
  • 1 INTRODUCTION: FasterViT combines convolutional processing in early stages with transformer blocks and Hierarchical Attention in later stages to capture short- and long-range dependencies efficiently.The architecture reduces image resolution through strided convolutions, uses residual convolutional blocks in stages 1–2, and transformer blocks in stages 3–4.
  • 1 INTRODUCTION: The model is validated on ImageNet-1K, MS COCO, and ADE20K for classification, detection, instance segmentation, and semantic segmentation.The reported downstream results include competitive performance for dense prediction tasks.
  • 1 INTRODUCTION: FasterViT achieves a new SOTA Pareto front for image throughput and accuracy, with significantly higher speed than comparable vision-transformer architectures.The comparison concerns ImageNet-1K Top-1 accuracy and throughput measured on an A100 GPU with batch size 128.

2 RELATED WORK

Prior work improves vision-transformer efficiency through architectural, attention, compression, and inference strategies. Related global-attention methods use specialized tokens or hierarchical mechanisms to extend contextual modeling beyond local regions.

  • 2 RELATED WORK: Efficiency-oriented vision-transformer research includes efficient attention, network compression, dynamic inference, and operator adaptation.These approaches target reductions in network complexity or computation.
  • 2 RELATED WORK: Global self-attention methods such as BigBird, Longformer, EdgeViT, Twins, and Focal Transformer introduce specialized or hierarchical mechanisms for broader contextual modeling.The cited approaches include non-learnable global tokens and hierarchical-like attention designs.
  • 2 RELATED WORK: FasterViT is presented with a multi-scale architecture combining CNN-based blocks in stages 1–2 and transformer-based blocks in stages 3–4.This architecture overview connects the paper’s design to the broader effort to improve transformer efficiency.

3 FASTERVIT

FasterViT combines convolutional processing in high-resolution stages with hierarchical attention in later stages. Its carrier-token design separates local and global attention to reduce computational cost while preserving cross-window communication.

  • 3 FASTERVIT: FasterViT uses dense residual convolutions in early high-resolution stages and transformer blocks with hierarchical attention in later stages.The architecture reduces spatial resolution between stages while increasing feature channels.
  • 3 FASTERVIT: Carrier tokens summarize local windows, undergo full self-attention, and then interact only with the corresponding local window tokens.This separates global carrier-token attention from local window attention.
  • 3 FASTERVIT: The HAT block concatenates each local window with its dedicated carrier tokens, applies attention and an MLP, then splits the tokens for subsequent layers.The design also applies positional biases and performs global information propagation at the end of a stage.
  • 3 FASTERVIT: HAT decomposes full attention into local window attention and carrier-token attention, enabling cross-window information exchange at lower complexity.Full attention has complexity O(H^4d), whereas windowed attention has complexity O(k^2H^2d); HAT combines local and carrier-token attention.
  • 3 FASTERVIT: Twins provides subsampled global information inside local attention, whereas HAT separates local and carrier-token attention into two dense operations.The comparison is presented as an alternative multilevel-attention design.

4 RESULTS

FasterViT delivers favorable accuracy-throughput trade-offs across classification, detection, instance segmentation, and semantic segmentation benchmarks. Its dense-prediction results also include faster throughput and strong detection accuracy at higher-resolution settings.

  • 4 RESULTS: FasterViT models achieve higher accuracy under the same throughput than Conv-based architectures, including a 2.2% improvement over ConvNeXt-T.The classification comparison is reported on ImageNet-1K.
  • 4 RESULTS: FasterViT-4 outperforms ConvNeXt-B and Swin-B by +0.2 and +1.0 box AP, and +0.3 and +1.0 mask AP, respectively.It is also 15% and 30% faster in throughput, respectively.
  • 4 RESULTS: FasterViT-4 with ImageNet-21K pretraining achieves 58.7 box AP on MS COCO using the DINO model.The result is reported as an additional object-detection experiment.
  • 4 RESULTS: FasterViT models show better performance-throughput trade-offs than counterpart models on ADE20K semantic segmentation.The semantic-segmentation benchmarks use UPerNet.

5 ABLATION

The ablations show that HAT improves the accuracy-throughput balance and can be inserted into existing architectures. Larger carrier or window sizes improve accuracy but reduce throughput, especially at higher resolutions.

  • 5 ABLATION: HAT is more effective and efficient for modeling long-range dependencies than increasing carrier-token or window size without sacrificing throughput.Larger carrier-token and window sizes improve accuracy while decreasing throughput; window size scales poorly at higher resolution.
  • 5 ABLATION: Higher-resolution fine-tuning makes FasterViT 0.9% more accurate on average and 2x faster than SwinV2.The comparison is summarized in Table 6.
  • 5 ABLATION: Replacing Swin-T attention with HAT improves mIoU by +0.9%, ImageNet Top-1 accuracy by +0.4%, box AP by +0.5, and mask AP by +0.6.Throughput comparisons show that the module can be used with minimal overhead.

6 CONCLUSION

FasterViT achieves a state-of-the-art ImageNet accuracy-throughput Pareto front and performs competitively on downstream dense-prediction tasks. Its benchmarks report better accuracy-throughput trade-offs than ConvNeXt and Swin Transformer.

  • 6 CONCLUSION: FasterViT achieves a state-of-the-art Pareto front for ImageNet Top-1 accuracy and throughput while outperforming ConvNeXt and Swin Transformer in accuracy-throughput trade-offs.The model is also validated on object detection, instance segmentation, and semantic segmentation.

B TRAINING SETTINGS

The study evaluates FasterViT on ImageNet classification, MS COCO detection and instance segmentation, and ADE20K semantic segmentation using pretrained backbones and task-specific fine-tuning.

  • ImageNet-1K classification uses 1.2M training images, 50K validation images, 1,000 categories, and Top-1 accuracy; ImageNet-21K supplies 14M images across 21,841 classes for pretraining.
  • FasterViT models are trained for 300 epochs with LAMB, learning rate 5e-3, total batch size 4096, and 32 A100 GPUs.
  • MS COCO experiments fine-tune Cascade Mask R-CNN with FasterViT backbones using AdamW, learning rate 1e-4, a 3x schedule, weight decay 5e-2, and batch size 16.
  • ADE20K experiments fine-tune UperNet with pretrained FasterViT backbones using AdamW, learning rate 6e-5, weight decay 1e-2, and batch size 16.

C ROBUSTNESS ANALYSIS

Using ImageNet-1K pretrained weights without fine-tuning, FasterViT shows robustness across ImageNet-A, ImageNet-R, and ImageNetV2, with larger variants exceeding comparable models on all three benchmarks.

  • FasterViT-3 outperforms comparable ConvNeXt-B and Swin-B models on the evaluated robustness datasets, while similar trends hold for smaller variants.
  • The robustness analysis uses ImageNet-1K pretrained weights directly on ImageNet-A, ImageNet-R, and ImageNetV2 without additional fine-tuning.
  • FasterViT-4 outperforms ConvNeXt-L by +7.9%, +2.6%, and +1.5% on ImageNet-A, ImageNet-R, and ImageNetV2, respectively.

D.1 COMPONENT-WISE STUDY

The component study examines HAT and related design choices through ablations and attention visualizations, showing that HAT preserves a throughput-oriented alternative to larger global windows while enabling cross-window interaction.

  • HAT ablation: +0.1% accuracy from a 14x14 global window costs 10% throughput, while removing HAT drops accuracy by 0.24% after retraining and 1.49% post-training.
  • Component ablations: Removing CT attention causes a 3.85% post-training accuracy drop, while removing attention bias reduces retrained accuracy by 0.31%.
  • Component ablations: Removing CT propagation costs 7% of inference and lowers accuracy by 0.16%, whereas removing CT initialization lowers post-training accuracy by 0.48%.
  • High-resolution comparison: FasterViT maintains higher throughput than Swin Transformer V2 when fine-tuned at larger image resolutions, supporting HAT for high-resolution inputs.
  • Attention visualizations: FasterViT attention maps transition toward carrier-token attention and later exhibit global patterns indicating cross-interaction between regions.
  • Positional bias: The learned positional-bias kernels demonstrate position-dependent features while sharing patterns between pixels.

I DESIGN INSIGHTS

FasterViT’s design insights balance accuracy and throughput through normalization, tokenization, regularization, and carrier-token choices, with HAT assigned the largest profiled stage cost.

  • Normalization: Layer normalization is critical in transformer stages: replacing it with batch normalization reduces accuracy by 0.7%.
  • Tokenization and throughput: The architecture performs windowing only once in stages 3 and 4, producing a 5% throughput improvement by avoiding repeated windowing and de-windowing.
  • Profiling: Stage 3 containing HAT has the highest latency, FLOPs, and memory footprint because it contains considerably more layers than the other stages.
  • Regularization: A drop-path rate of 30% with weight decay 0.12 reaches 85.15%, compared with 84.91% for 50% drop-path and weight decay 0.05.
  • Optimization: MESA is enabled after 25% of training with coefficients ranging from 0.25 for FasterViT-0 to 3.0 for FasterViT-4.
  • Carrier-token and window size: Increasing window size from 7 to 14 raises Top-1 accuracy by 0.2% but latency by 10%, while a 2x2 carrier-token window offers a strong accuracy-latency trade-off.

L DOWNSTREAM EXPERIMENTS

FasterViT improves throughput and accuracy across downstream detection and segmentation experiments, including comparisons with Swin-L and PoolFormer backbones. Conv-based stages also provide efficiency and accuracy benefits for larger inputs.

  • Semantic segmentation: +1.1 mIoU and 10.05% faster throughput are achieved by FasterViT-2 versus PoolFormer-M36 on ADE20K semantic segmentation.FasterViT-1 also exceeds PoolFormer-S36 by +0.7 mIoU while being 8.38% faster.
  • Semantic segmentation: +0.7 mIoU and 8.38% faster throughput are achieved by FasterViT-1 versus PoolFormer-S36 on ADE20K semantic segmentation.The experiment uses an FPN network on the ADE20K dataset.
  • Conv-based stages: Replacing Conv-based blocks with Transformer-based blocks significantly reduces both throughput and accuracy, especially for larger input sizes.The ablation compares Conv-based and Transformer-based counterparts in FasterViT models.

N THROUGHPUT ON DIFFERENT PLATFORMS

FasterViT maintains a strong accuracy-throughput trade-off across GPUs, CPU hardware, and an embedded Jetson Nano platform. The reported comparisons generally place FasterViT on a Pareto front, with some EfficientNetV2 and RegNetY variants performing comparably on CPU or A6000 hardware.

  • Cross-platform comparison: FasterViT achieves a SOTA Pareto front for ImageNet Top-1 accuracy and throughput across multiple hardware platforms.The benchmarks cover V100, TITAN RTX, A6000, CPU, and Jetson Nano devices.
  • GPU platforms: FasterViT achieves a Pareto front on both NVIDIA V100 and TITAN RTX GPUs for ImageNet Top-1 accuracy and image throughput.The V100 comparison uses batch size 128, and the TITAN RTX comparison also uses batch size 128.
  • GPU platforms: FasterViT achieves a SOTA Pareto front on NVIDIA A6000, except for an EfficientNetV2 variant with comparable performance to FasterViT-2.The A6000 comparison uses batch size 64.
  • CPU and embedded platforms: FasterViT variants show strong performance on Jetson Nano, while EfficientNetV2 and RegNetY variants are comparable to some FasterViT models on CPU.The CPU comparison uses an Intel Xeon E5-2698 v4, and the Jetson Nano comparison uses batch size 1.
Loading 2306.06189v2…