PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, Jianfeng Gao

arXiv:2103.15358v2cs.CVcs.AIcs.LG

TL;DR

High-resolution vision tasks expose vanilla ViT’s quadratic self-attention cost and memory demands. The paper combines multi-scale Transformer stages with 2-D Vision Longformer attention, reporting strong performance across classification, detection, and segmentation while reducing efficiency costs.

  • Problem

    Vanilla ViT’s quadratic computational and memory complexity limits its use for high-resolution feature maps in vision tasks.

  • Method

    The paper combines a multi-scale Transformer structure with 2-D Vision Longformer attention using local attention and global memory.

  • Results

    The proposed ViT significantly outperforms strong ViT, ResNet, and concurrent baselines across image classification, object detection, and segmentation.

  • Takeaways & Limitations

    Vision Longformer’s local-attention-plus-global-memory design supports scalable high-resolution vision tasks while saving computation and memory.

  • Takeaways & Limitations

    Partial X-formers and multi-scale ViTs with full attention are not transferable to high-resolution tasks because of prohibitively large memory usage; Linformer is resolution-specific.

Abstract

from arXiv · show

This paper presents a new Vision Transformer (ViT) architecture Multi-Scale Vision Longformer, which significantly enhances the ViT of \cite{dosovitskiy2020image} for encoding high-resolution images using two techniques. The first is the multi-scale model structure, which provides image encodings at multiple scales with manageable computational cost. The second is the attention mechanism of vision Longformer, which is a variant of Longformer \cite{beltagy2020longformer}, originally developed for natural language processing, and achieves a linear complexity w.r.t. the number of input tokens. A comprehensive empirical study shows that the new ViT significantly outperforms several strong baselines, including the existing ViT models and their ResNet counterparts, and the Pyramid Vision Transformer from a concurrent work \cite{wang2021pyramid}, on a range of vision tasks, including image classification, object detection, and segmentation. The models and source code are released at \url{https://github.com/microsoft/vision-longformer}.

1. Introduction

The paper addresses ViT’s computational and memory barriers for high-resolution vision by combining a multi-scale structure with Vision Longformer attention. The resulting architecture is evaluated across major vision tasks against strong baselines.

  • Motivation: High-resolution vision tasks require feature maps that vanilla ViT struggles to compute because self-attention has quadratic token complexity.The problem is especially severe in vision, where complexity grows quartically with image resolution.
  • Proposed architecture: Multi-Scale Vision Longformer combines the multi-scale model structure and Vision Longformer attention for efficient high-resolution image encoding.These are the paper’s two stated techniques for enhancing baseline ViT.
  • Multi-scale architecture: Multi-scale ViT stages provide image encodings at multiple scales while keeping computation and memory manageable.The proposed design stacks multiple Transformer stages, beginning with high-resolution features and progressively reducing resolution.
  • Efficient attention: Vision Longformer uses 2-D local attention with global memory to achieve linear complexity with respect to the number of input tokens.Compared with baseline ViT, the mechanism reduces computational and memory cost without performance drop in the reported experiments.
  • Evaluation: The empirical study reports significant gains over previous ViTs, ResNet counterparts, and a concurrent model across classification, detection, and segmentation.The comparison spans multiple vision tasks rather than a single benchmark.

2. Related Work

Related work develops efficient and hierarchical Transformer designs for vision and language. The paper combines multi-scale processing with sparse local attention and global memory for high-resolution vision.

  • Vision Transformers: ViT applies Transformer encoders to image patches and has outperformed CNN models such as ResNet on multiple image-classification benchmarks.Its long-range context modeling is a key strength, while high-resolution processing remains costly.
  • Efficient attention: Efficient attention mechanisms include sparse attention and memory-based approaches that restrict token interactions or use extra global tokens.These categories motivate the paper’s combination of local sparsity and global memory.
  • Vision Longformer: Vision Longformer is a 2-D Longformer variant that combines conv-like sparse attention with memory mechanisms.Its sparsity mechanism is conceptually similar to that of Image Transformer.
  • Multi-scale Transformers: The proposed multi-scale Transformer reduces spatial size through patch embedding at the beginning of each stage, merging tokens from the previous stage.This creates a hierarchical sequence of Transformer stages.
  • Concurrent work: PVT, Swin Transformer, and HanoNet are concurrent multi-scale ViTs, but PVT still has quartic resolution complexity despite a smaller constant.The paper positions Vision Longformer as an alternative efficient-attention design.
  • Architecture overview: Figure 1 represents the multi-scale model as four stacked E-ViT modules, each defined by attention type, block count, and input patch size.The modules also add LayerNorm after patch embedding and use extra global tokens as memory.

3. Multi-Scale Stacked Vision Transformers

The multi-scale stacked ViT combines staged architecture choices with efficient attention to produce high-resolution feature maps at manageable cost. Vision Longformer preserves performance while reducing the memory and computation burden of full attention.

  • E-ViT extends vanilla ViT with LayerNorm, global and local tokens, efficient attention, and 2-D positional encoding choices.
  • Multi-Scale Model Architecture: Multi-scale models stack E-ViT stages, using smaller patches and hidden dimensions early, then downsampling while increasing hidden dimensions in later stages.The first stage generates the highest-resolution feature map; later stages use patch size 2 and double hidden dimension when resolution halves.
  • Multi-Scale Model Architecture: Global tokens are discarded between stages for classification, detection, and instance segmentation, while reshaped local tokens feed the next stage.In this configuration, global tokens provide efficient communication between distant local tokens, functioning as global memory.
  • Multi-Scale Model Architecture: Multi-scale models outperform flat models on low-resolution ImageNet classification, whereas full attention quickly incurs prohibitive memory growth with high-resolution blocks.The paper reports that four-stage multi-scale ViTs with full attention cannot be trained under the same batch-size and hardware setting as DeiT.
  • Vision Longformer: Vision Longformer uses local attention plus global memory to achieve linear complexity in token count, reducing computation and memory without performance drop versus full attention.Global tokens attend to all tokens, while local tokens attend to global tokens and nearby 2-D neighbors; relative positional bias further improves performance.
  • Other Efficient Attention Mechanisms: Among alternative efficient attention mechanisms, Vision Longformer outperforms the compared methods on ImageNet classification and transfers to high-resolution tasks.Linformer is not transferable because its projection weights are resolution-specific, while some full-attention alternatives require prohibitively large memory after transfer.

4. Experiments

Experiments evaluate ViL on ImageNet classification and COCO detection tasks under multiple pretraining, model-size, schedule, and efficient-attention settings. ViL consistently outperforms strong baselines, while ImageNet-21K pretraining substantially improves medium and base models.

  • ImageNet-21K Pretraining: Pure ImageNet-1K training yields only a marginal gain from ViL-Medium to ViL-Base.The authors relate this observation to the need for sufficient training data for large pure-transformer models.
  • ImageNet-21K Pretraining: ImageNet-21K pretraining significantly boosts ViL-Medium and ViL-Base performance.The resulting ViL-Medium surpasses ViT-Base/16, ViT-Large/16, and BiT-152x4-M, while ViL-Base performs better still.
  • Detection Tasks: ViL achieves significantly better RetinaNet detection performance than ResNet and PVT across model sizes and object scales.ViL-Tiny with the 3x+MS schedule already outperforms ResNeXt101-64x4d and PVT-Large.
  • Detection Tasks: ViL also surpasses ResNet and PVT on Mask R-CNN detection and instance segmentation, and ViL-Small reaches 47.1 AP b with 45M parameters versus Swin-Tiny’s 46.0 AP b with 48M.The comparison covers both 1× and 3×+MS training schedules.
  • Ablation Study for Detection Tasks: With efficient attention in all stages, ViL reaches 42.9 AP b while SRA, Global Transformer, and Performer are each around 36.0 AP b.The results indicate that local attention plus global memory retains full-attention performance better than the other evaluated efficient mechanisms.
  • Ablation Study for Detection Tasks: Adding global tokens beyond one does not improve performance in the reported ablation.The figure caption identifies the comparison as varying window size and the number of global tokens with the same ImageNet1K-pretrained checkpoint.

5. Conclusions

The paper concludes that Multi-Scale Vision Longformer addresses the computational and memory barriers limiting vanilla ViT on high-resolution vision tasks. Its multi-scale structure and linear-token-complexity attention support high-resolution encoding and strong results across classification and detection benchmarks.

  • Conclusions: Multi-Scale Vision Longformer combines multi-scale encoding with efficient 2-D Vision Longformer attention for high-resolution image processing.The multi-scale design provides multiple image scales with manageable cost, while attention has linear complexity with respect to input tokens.
  • Conclusions: The architecture effectively addresses vanilla ViT’s computational and memory-efficiency problem for high-resolution feature maps.The conclusion reports validation through comprehensive ablation studies.
  • Conclusions: ViL outperforms several strong baselines on image classification and object detection.The conclusion summarizes the reported experimental outcome rather than a single benchmark value.

A.1. Model configurations

The model-configuration appendix documents the architecture parameters and the common configuration used across attention mechanisms.

  • Model Configurations: Attention-specific mechanisms are omitted from the shared model configuration and specified separately in Table 15.The remaining model configuration is the same across attention mechanisms.
  • Model Configurations: Table 9 lists architecture parameters for multi-scale stacked ViTs, including attention blocks, patch size, heads, and hidden dimension.These parameters are specified for each E-ViT module.

A.2. Experimental settings

The experiments use task-specific training setups for ImageNet and COCO. For larger COCO models, over-fitting leads the authors to report the best checkpoint along the training trajectory.

  • ImageNet training mainly follows DeiT, using random path, DeiT augmentations, and repeated augmentation only for Medium and Base models.
  • Fine-tuning from ImageNet-21K uses 384 × 384 images, SGD with momentum 0.9, no weight decay, and random cropping only.
  • COCO detection and segmentation follow standard “1×” and “3×+MS” schedules, changing the optimizer from SGD to AdamW while tuning learning rate and weight decay.
  • Training ViL-Medium and ViL-Base on COCO shows obvious over-fitting because these large models are pretrained only on ImageNet.
  • The reported COCO performance uses the best checkpoint selected along the training trajectory, with one checkpoint evaluated per epoch.

B.1. Ablation study on the architecture design of multi-scale Vision Longformer

The architecture ablations examine LayerNorm, positional embeddings, attention configuration, and classification-feature extraction. They report benefits from the added normalization and positional design choices relative to baseline configurations.

  • Architecture ablation: The flat E-ViT model adds LayerNorm after patch embedding and 2-D positional embedding to standard ViT/DeiT.
  • Architecture ablation: The compared attention mechanisms are evaluated on ImageNet Top-1 accuracy and separately by parameter count and GFLOPs.

C. Implementations and Efficiency of Vision Longformer In Practice

Vision Longformer combines local sliding-window attention with implementation choices designed to reduce practical time and memory costs. Sliding chunks provide an efficient implementation, while boundary handling affects accuracy and translational invariance.

  • Implementations and complexity: Full quadratic attention followed by masking has quadratic complexity in tokens and quartic complexity in feature-map size, making it impractical.
  • Implementations and complexity: The implementations include PyTorch unfold variants, a customized CUDA kernel, and sliding chunks using autograd or a handwritten backward function.
  • Implementations and complexity: The CUDA-kernel implementation achieves theoretical optimal memory usage and linear token-time complexity, whereas sliding chunks use optimized CUDA matrix-multiplication libraries.
  • Sliding-chunk attention: The sliding-chunk approach partitions features into w × w chunks, with each chunk attending to itself and its 8 neighbors.
  • Sliding-chunk attention: Sliding chunks can exactly reproduce conv-like local attention by masking invalid tokens, while also supporting boundary handling without padding or with cyclic padding.
  • Efficiency evaluation: Different mathematically equivalent conv-like implementations are compared by forward-backward running time and memory usage.
  • Masking choices: The three masking methods have nearly identical time and memory usage, but cyclic padding is slightly worse for ImageNet Top-1 accuracy most of the time.
  • Masking choices: Sliding chunk without padding performs consistently better than exact sliding-window attention for object detection, despite sacrificing some translational invariance.

D. Random-shifting strategy to improve training efficiency

Random-shifting reduces training attention from all eight neighboring patches to one randomly selected neighbor, then switches back to the full neighborhood. The strategy accelerates training without harming final performance.

  • Strategy: Random-shifting training lets each patch attend only to itself and one random neighbor instead of all 8 neighbor patches.
  • Strategy: The strategy defines modes for self-only attention, full eight-neighbor attention, and self-plus-one specified neighbor.
  • Training schedule: During training, one mode from 1 to 8 is sampled randomly, and the model switches to the default 8-neighbor mode after a hyperparameterized fraction of iterations.
  • Training schedule: The default switch time is 75% of training iterations, functioning as fine-tuning to mitigate differences between training and inference behavior.
  • Results: The random-shifting strategy significantly accelerates Vision Longformer training without harming final model performance.

E. Other Efficient Attention Mechanisms utilized in this work

The paper compares several efficient attention alternatives, including spatial reduction attention and Performer, examining their complexity, scaling behavior, and implementation settings.

  • Pyramid Vision Transformer uses spatial reduction attention with stage-specific reduction ratios R1 = 8, R2 = 4, R3 = 2, R4 = 1.These settings reduce key and value feature-map resolution at successive stages.
  • SRA retains quartic computation and memory growth with feature-map resolution despite a smaller constant factor.Its memory complexity is O(n_l^2/R^2), while high-resolution transfer remains costly.
  • A 600×1000 image resolution is feasible with SRA/32, but memory usage exceeds ResNet counterparts at 800 × 1333.
  • SRA/64 doubles the downsizing relative to PVT, making memory usage comparable with other efficient attention methods but increasing parameters.Its ratios are R1 = 16, R2 = 8, R3 = 4, R4 = 2, and the larger convolutional projection kernel adds parameters.
  • Performer approximates Softmax attention with K = 256 orthogonal random features and has linear complexity in token count and feature count.Its stated memory complexity is O(Kd + n_ld + Kn_l), and computation complexity is O(Kn_ld).
  • Performer training redraws random features adaptively, with schedules differing between ImageNet classification and COCO detection/segmentation.The ImageNet schedule redraws every 1 + 5T iterations in epoch T; COCO uses a less frequent early schedule after pretrained initialization.
Loading 2103.15358v2…