PaperlayerView exact revision on arXiv

Source-linked AI summary

RegionViT: Regional-to-Local Attention for Vision Transformers

Chun-Fu Chen, Rameswar Panda, Quanfu Fan

arXiv:2106.02689v3cs.CV

TL;DR

Vision transformers inherit design choices that are not optimized for vision and incur high memory costs at fine-grained resolutions. RegionViT combines a pyramid architecture with regional-to-local attention over multiscale tokens, reducing attention memory while retaining global information. Across four vision applications, it outperforms or matches many concurrent vision-transformer variants.

  • Problem

    Vanilla ViT uses an isotropic architecture and quadratic self-attention, limiting efficient processing of high-resolution visual details and broader vision applications.

  • Method

    RegionViT uses a pyramid structure and regional-to-local attention, generating regional and local tokens with different patch sizes and exchanging information hierarchically.

  • Results

    Across image classification, object and keypoint detection, semantic segmentation, and action recognition, RegionViT outperforms or matches many concurrent vision-transformer variants.

  • Takeaways & Limitations

    Regional-to-local attention reduces the memory overhead of fine-grained self-attention while preserving the ability to explore global information.

  • Takeaways & Limitations

    The authors note that the approach shares pitfalls associated with deep classification models, including privacy implications for image and video classification systems.

Abstract

from arXiv · show

Vision transformer (ViT) has recently shown its strong capability in achieving comparable results to convolutional neural networks (CNNs) on image classification. However, vanilla ViT simply inherits the same architecture from the natural language processing directly, which is often not optimized for vision applications. Motivated by this, in this paper, we propose a new architecture that adopts the pyramid structure and employ a novel regional-to-local attention rather than global self-attention in vision transformers. More specifically, our model first generates regional tokens and local tokens from an image with different patch sizes, where each regional token is associated with a set of local tokens based on the spatial location. The regional-to-local attention includes two steps: first, the regional self-attention extract global information among all regional tokens and then the local self-attention exchanges the information among one regional token and the associated local tokens via self-attention. Therefore, even though local self-attention confines the scope in a local region but it can still receive global information. Extensive experiments on four vision tasks, including image classification, object and keypoint detection, semantics segmentation and action recognition, show that our approach outperforms or is on par with state-of-the-art ViT variants including many concurrent works. Our source codes and models are available at https://github.com/ibm/regionvit.

1 INTRODUCTION

RegionViT addresses vision-transformer limitations from isotropic architecture and quadratic self-attention by combining a pyramid structure with regional-to-local attention. The method reduces memory use while preserving global information and performs competitively across vision tasks.

  • Motivation: ViT’s fixed-scale architecture and quadratic self-attention make fine-grained, high-resolution vision processing computationally and memory intensive.The cost grows quadratically with token count and quadruple with image resolution, complicating object detection and semantic segmentation.
  • Approach: RegionViT combines a pyramid structure with regional and local tokens generated from different patch sizes.Regional tokens use larger patches, while local tokens use smaller patches associated by spatial location.
  • Efficiency: 73% memory savings are reported for R2L attention in the default setting compared with regular global self-attention.The reduced memory complexity is attributed to attention over fewer regional and local tokens.
  • Approach: Regional-to-local attention first extracts global information among regional tokens, then exchanges information between each regional token and its associated local tokens.This hierarchical mechanism limits local attention’s scope while retaining access to global information.
  • Results: RegionViT models outperform or match concurrent pyramid-based vision transformers for image classification and also perform well on object detection and action recognition.The paper reports experiments across several downstream vision tasks.

2 RELATED WORK

Related vision-transformer work combines attention with CNNs, restricts attention locally, or introduces pyramid structures. RegionViT is positioned among methods that use windowed or hierarchical token interactions to balance computation and information exchange.

  • CNNs with Attention: CNN-attention hybrids range from replacing convolutional layers with local self-attention to modifying or approximating attention to balance computation and accuracy.The cited methods differ in how extensively they replace convolution and how they manage attention complexity.
  • Comparison: The related-work comparison organizes methods around token grouping and mechanisms that enable information exchange among regions.The table caption emphasizes non-overlapped windows as a common grouping strategy.
  • Vision Transformer: ViT achieved CNN-comparable classification results, while DeiT enabled similar performance on ImageNet1K through an efficient training scheme.The related-work discussion also notes that ViT’s architecture was borrowed directly from natural language processing.
  • Differences from Related Works: PVT uses a pyramid structure but retains global attention, whereas ViL uses overlapping local windows and Swin, Twins, and RegionViT use non-overlapping windows.The methods therefore differ in attention scope, windowing, and mechanisms for exchanging information across regions.

3 METHOD

RegionViT adapts vision transformers with a pyramid architecture and regional-to-local attention, combining global context from regional tokens with fine-grained local features. Its hierarchical tokenization, attention, and downsampling reduce memory demands while supporting multi-resolution vision processing.

  • Architecture: RegionViT converts images into regional and local tokens using different patch sizes, with each regional token associated spatially with local tokens.Regional tokens use larger patches, while local tokens use smaller patches; later stages downsample both token sets and double their channel dimensions.
  • Regional-to-local attention: Regional self-attention first exchanges information among all regional tokens, after which local self-attention combines each regional token with its associated local tokens.The regional token supplies global information to the local attention operating within its spatial region.
  • Regional-to-local attention: The R2L encoder applies a feed-forward network after attention and uses layer normalization, residual shortcuts, and shared RSA and LSA weights.The resulting tokens are split back into regional and local streams for subsequent processing.
  • Efficiency: R2L attention reduces memory complexity to O((M + 1)^2 × (N/M) + (N/M)^2), compared with O(N^2) for global self-attention.Here M is the number of local tokens in a region and N is the total number of local tokens; in stage 3, M=49 and N=196 yield approximately 73% memory savings.
  • Position encoding: Relative position bias is added to local-token attention to encode spatial relationships without requiring vanilla ViT's absolute positional embeddings.The bias is learned from relative coordinate differences, while the content-based query-key term remains part of the attention value.
  • Implementation: The model uses non-overlapping regional windows, convolutional tokenization options, and depth-wise-convolution downsampling across stages.Local tokens use patch size 4, while regional tokens use patch size 4 × M; the default window size is 7, with additional models using 14.

4 EXPERIMENTS

Experiments evaluate RegionViT across classification, transfer learning, detection, segmentation, action recognition, and ablations. Results show favorable accuracy-efficiency trade-offs and support regional tokens, shared weights, 3×3 downsampling, relative position bias, and non-overlapped windows.

  • Image classification: RegionViT achieves a better accuracy-complexity trade-off for smaller ImageNet1K models and higher accuracy with fewer FLOPs and parameters for larger models.The reported efficiency is attributed to the regional-to-local transformer encoder.
  • Image classification: RegionViT-B+ uses 25% of the parameters and FLOPs of CvT-W24 and half those of Swin-L on ImageNet21K pretraining.
  • Transfer learning: RegionViT outperforms DeiT by 2∼3% on ChestXRay8 transfer learning.The authors associate this result with hierarchical features providing better generalization across the domain gap.
  • Downstream tasks: RegionViT provides competitive or improved results across object detection, semantic segmentation, keypoint detection, and action recognition, with efficiency advantages in several settings.RegionViT-M reduces more than 50% of TimeSformer’s FLOPs and parameters, while semantic segmentation improves with similar FLOPs.
  • Ablation study: Regional tokens improve image classification by around 0.4% and clearly improve detection and segmentation with less than 2% overhead in FLOPs and parameters.The passage attributes their value to multi-scale features with global contextual information for dense prediction.
  • Ablation study: Ablations select 3×3 depthwise convolution, shared RSA and LSA weights, associated regional tokens, relative position bias, and non-overlapped windows.Using all regional tokens improves only 0.1% at 13% more computation, while overlapped windows improve 0.2% at 3.5× FLOPs.

5 CONCLUSION

RegionViT combines a CNN-like pyramid structure with regional-to-local attention to provide multi-scale features while retaining global information with lower memory overhead. Across four vision applications, it outperforms or matches many concurrent ViT variants.

  • RegionViT uses a pyramid structure to provide multi-scale features and facilitate extension to applications such as object detection.
  • Regional-to-local attention limits fine-grained attention scope to reduce memory overhead while retaining the ability to explore global information.
  • Across image classification, object and keypoint detection, semantic segmentation, and action recognition, RegionViT outperforms or matches many concurrent ViT variants.

ETHICS STATEMENT

The authors note that the method may enable faster processing and lower computation costs, while visual recognition systems can create privacy risks.

  • The method may enable faster responses in applications such as medical-emergency detection and manufacturing-defect detection.
  • Lower computation costs may save energy and benefit the environment.
  • Visual classification systems can threaten privacy and may be misused by malicious actors or governments.
  • The authors identify private and ethical aspects of visual recognition as an important direction for future research.

REPRODUCIBILITY STATEMENT

The supplementary materials document training and evaluation details across the paper’s vision tasks and provide code and instructions intended to support reproduction.

  • The authors describe hyperparameters and provide image-classification code so users can rerun experiments with the default settings.
  • Appendix A separately describes training details for image classification, object and keypoint detection, semantic segmentation, and action recognition.
  • The supplementary materials include detailed object-detection results and ablation studies.
  • The attached code and README provide instructions for training and evaluating the networks.

A.1 IMAGE CLASSIFICATION

For ImageNet1K classification, the authors follow DeiT’s training procedure with specified optimization, scheduling, batch-size, and augmentation settings.

  • The models are trained on ImageNet1K following DeiT, with batch size 4,096 and base learning rate 0.004.
  • Training uses AdamW with a cosine learning-rate scheduler and 50 warm-up epochs.
  • The training pipeline applies Mixup, CutMix, RandomErasing, label smoothing, RandAugment, and instance repetition.

A.2 OBJECT DETECTION AND KEYPOINT DETECTION

For detection and keypoint tasks, RegionViT replaces the backbone in established detector frameworks, supplying local tokens as multi-scale features.

  • RegionViT replaces the backbone in RetinaNet and MaskRCNN for object detection and in KeypointRCNN for keypoint detection.
  • Local tokens from each RegionViT stage are output as multi-scale features for the detector.
  • Regional tokens are not used in detection, and new layer-normalization layers normalize features before detector input.
  • RegionViT-S and RegionViT-B are used for the detection experiments.

A.3 SEMANTIC SEGMENTATION

The semantic segmentation experiments use Semantic FPN with RegionViT replacing the backbone, pretrained on ImageNet1K and evaluated on resized whole images.

  • A.3 SEMANTIC SEGMENTATION: Semantic FPN replaces its backbone with RegionViT for semantic segmentation experiments.The setup uses ImageNet1K-pretrained weights.
  • A.3 SEMANTIC SEGMENTATION: Training resizes images so the shorter side is 448, limits the longer side to 1792, and crops a 448×448 region.
  • A.3 SEMANTIC SEGMENTATION: Evaluation uses the whole image after resizing rather than a training crop.

B MORE DETAILED RESULTS

Additional results cover keypoint detection and complexity-related analyses, while the appendix provides detailed object-detection tables and throughput comparisons.

  • B MORE DETAILED RESULTS: RegionViT outperforms ResNet-50 on person keypoint detection with moderate increases in FLOPs and parameters.The authors interpret this as evidence that RegionViT can model global context for keypoint detection.
  • B MORE DETAILED RESULTS: Appendix tables report object-detection performance on COCO val2017 under both 1× and 3× training schedules.For MaskRCNN, both bounding-box and mask AP are annotated, and bold values mark the best number within each section.
  • B MORE DETAILED RESULTS: Swin results are sourced from Twins because the original Swin paper does not report ImageNet1K-weighted results; one entry uses 896×1344 input resolution.
  • B MORE DETAILED RESULTS: The appendix includes a comparison of RegionViT-S with and without regional tokens.
  • B MORE DETAILED RESULTS: A separate appendix table reports throughput comparisons.
Loading 2106.02689v3…