PaperlayerView exact revision on arXiv

Source-linked AI summary

Global Context Vision Transformers

Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, Pavlo Molchanov

arXiv:2206.09959v5cs.CVcs.AIcs.LG

TL;DR

High-resolution vision transformers must capture long-range dependencies without the prohibitive cost of quadratic attention and limited receptive fields of local windows. GC ViT combines local and global self-attention with global query tokens and modified Fused-MBConv blocks, achieving state-of-the-art or competitive results across classification, detection, instance segmentation, and semantic segmentation.

  • Problem

    Quadratic self-attention and limited local-window receptive fields constrain efficient high-resolution vision-transformer modeling of long-range spatial information.

  • Method

    GC ViT uses hierarchical local and global self-attention with shared global query tokens, plus modified Fused-MBConv blocks that provide inductive bias and inter-channel modeling.

  • Results

    GC ViT achieves state-of-the-art or competitive performance across ImageNet classification, MS COCO detection and instance segmentation, and ADE20K semantic segmentation.

  • Takeaways & Limitations

    GC ViT provides a hierarchical vision-transformer framework that captures global context while maintaining computational efficiency comparable to Swin Transformer.

Abstract

from arXiv · show

We propose global context vision transformer (GC ViT), a novel architecture that enhances parameter and compute utilization for computer vision. Our method leverages global context self-attention modules, joint with standard local self-attention, to effectively and efficiently model both long and short-range spatial interactions, without the need for expensive operations such as computing attention masks or shifting local windows. In addition, we address the lack of the inductive bias in ViTs, and propose to leverage a modified fused inverted residual blocks in our architecture. Our proposed GC ViT achieves state-of-the-art results across image classification, object detection and semantic segmentation tasks. On ImageNet-1K dataset for classification, the variants of GC ViT with 51M, 90M and 201M parameters achieve 84.3%, 85.0% and 85.7% Top-1 accuracy, respectively, at 224 image resolution and without any pre-training, hence surpassing comparably-sized prior art such as CNN-based ConvNeXt and ViT-based MaxViT and Swin Transformer by a large margin. Pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation using MS COCO and ADE20K datasets outperform prior work consistently. Specifically, GC ViT with a 4-scale DINO detection head achieves a box AP of 58.3 on MS COCO dataset.

1. Introduction

GC ViT combines local and global self-attention with modified Fused-MBConv downsampling to model short- and long-range information efficiently. It achieves strong results across ImageNet classification and downstream detection, segmentation, and instance-segmentation tasks.

  • Motivation: Quadratic self-attention and ViT’s monolithic architecture hinder efficient application to high-resolution images where multi-scale long-range information matters.Local-window approaches improve scalability but limited receptive fields challenge long-range modeling.
  • Architecture: GC ViT uses hierarchical local and global self-attention, with shared global query tokens interacting with local key and value representations.Local attention models short-range information, while global queries incorporate context from different image regions.
  • Architecture: GC ViT introduces a parameter-efficient modified Fused-MBConv downsampling block that adds inductive bias and models inter-channel dependencies.The framework is designed to avoid sophisticated, computationally expensive operations while supporting efficient implementation.
  • Results: 84.3%, 85.0%, and 85.7% Top-1 accuracy are achieved by 51M, 90M, and 201M-parameter GC ViT variants on ImageNet-1K without extra data or pre-training.These models establish new classification benchmarks across model sizes.
  • Results: GC ViT consistently outperforms ConvNeXt, MaxViT, and Swin Transformer, with better or comparable computational efficiency across competing model sizes.The comparison concerns ImageNet-1K classification under matched evaluation settings.
  • Results: 58.3% box AP is achieved on MS COCO with a 4-scale DINO head, alongside 52.9 box mAP, 45.8 mask mAP, and 49.2 mIoU on downstream tasks.The reported downstream evaluations cover object detection, instance segmentation, and semantic segmentation.

2. GC ViT architecture

GC ViT is a hierarchical architecture that alternates local and global self-attention, using globally generated query tokens to capture long-range interactions while retaining local processing. CNN-like modified Fused-MBConv components provide downsampling, spatial processing, and inductive bias.

  • Architecture: GC ViT builds multi-resolution representations by reducing spatial dimensions and expanding embedding dimensions across hierarchical stages.Each stage changes both resolution and channel capacity, with downsampling between stages.
  • Attention: Each stage alternates local and global self-attention modules, with local windows modeling short-range features and global queries providing cross-region context.The global query generator extracts features from the entire image once per stage, and the resulting queries are shared across global-attention blocks.
  • Downsampling: Modified Fused-MBConv blocks and max pooling form the downsampling operator, supplying locality bias and inter-channel interaction during spatial contraction.The blocks include operations such as squeeze-and-excitation, GELU, and depth-wise 3 × 3 convolution, and are used to provide inductive bias.
  • Global query generator: The global query generator transforms stage features and repeatedly applies modified Fused-MBConv blocks with downsampling to match the local window dimensions.The generator then reshapes and repeats the resulting features to form global query tokens.
  • Attention: Global attention interacts globally extracted query tokens with local key and value representations, enlarging the receptive field without abandoning windowed computation.Global queries are repeated across windows and reshaped into multiple heads before interacting with locally computed keys and values.
  • Complexity: GC ViT’s global-query design maintains computational complexity similar to Swin Transformer while supporting long-range information and higher reported accuracy.The paper attributes this efficiency to the global query token generator and related components.

3. Experiments

GC ViT is evaluated across ImageNet-1K classification, MS COCO detection and instance segmentation, and ADE20K semantic segmentation under standard benchmark settings. It achieves strong performance across model scales and tasks, including efficient comparisons with competing architectures.

  • 3.1. Classification: GC ViT achieves better ImageNet-1K performance than established CNN and ViT-based benchmarks while offering better or comparable FLOPs.The comparison spans different model sizes and includes ConvNeXt.
  • 3.2. Detection and Instance Segmentation: GC ViT-T with Mask R-CNN reaches 47.9 box AP and 43.2 mask AP on MS COCO, exceeding pretrained ConvNeXt-T and Swin-T counterparts.The reported gains are +1.7/+1.5 over ConvNeXt-T and +1.9/+1.6 over Swin-T for box/mask AP.
  • 3.2. Detection and Instance Segmentation: 58.3% box AP is achieved on MS COCO using an ImageNet-21K-pretrained GC ViT-L backbone with a 4-scale DINO head.This setup uses the DINO detection benchmark configuration.
  • 3.3. Semantic Segmentation: GC ViT backbones achieve 47.0, 48.3, and 49.2 mIoU on ADE20K with UPerNet, outperforming corresponding Twins-SVT models.The comparisons use pretrained GC ViT-T, GC ViT-S, and GC ViT-B backbones.

4. Ablation

Ablations evaluate GC ViT components across classification, detection, instance segmentation, and semantic segmentation. The results support the roles of convolutional downsampling, cross-channel interaction, and global self-attention.

  • 4. Ablation: Removing window shifting causes significant performance degradation across classification, detection, instance segmentation, and semantic segmentation tasks.The ablation starts from Swin Transformer and progressively redesigns its components.
  • 4. Ablation: +0.9 accuracy, +0.8 box AP, +0.6 mask AP, and +1.2 mIoU result from leveraging global self-attention.The effect is highlighted for high-resolution tasks such as semantic segmentation, where long-range spatial dependencies matter.
  • 4.1. ImageNet-21K: +0.3% and +0.1% Top-1 accuracy separate pretrained GC ViT-L from Swin-L and CSwin-L on ImageNet-1K, while matching ConvNeXt-L.This comparison uses ImageNet-21K pretraining followed by ImageNet-1K fine-tuning.
  • 4.2. ImageNetV2: GC ViT demonstrates competitive performance on ImageNetV2 under Matched Frequency and Threshold-0.7 sampling strategies.These evaluations are used to measure robustness and generalizability.
  • 4.3. Downsampler Design: The modified Fused-MBConv downsampler outperforms convolution-plus-maxpooling and patch merging alternatives in ImageNet Top-1 accuracy.The convolution-plus-maxpooling alternative reduces accuracy by -0.8, while patch merging reduces it by -0.6.

5. Interpretability

Visualizations indicate that GC ViT’s global self-attention aligns with image semantics and provides contextual information for local attention. Grad-CAM maps also show accurate object localization.

  • 5. Interpretability: Global self-attention maps align with image semantics and act as an informative source for local attention modules.The visualization is paired with input images and corresponding Grad-CAM maps.
  • 5. Interpretability: Grad-CAM maps demonstrate accurate object localization with intricate details.

6. Related work

Related work traces the progression from CNNs and monolithic ViTs to hybrid and hierarchical architectures, including local-window attention and global-attention alternatives. GC ViT is positioned as a distinct approach to combining local and global interactions.

  • ConvNet: CNNs historically dominated computer vision benchmarks, while ConvNeXt modified ResNet to achieve competitive classification, detection, and segmentation results.
  • Transformer: ViT introduced enlarged receptive fields through self-attention but lacked CNN-like inductive biases and translation invariance and required large-scale training data.DeiT later used distillation-based training to improve data efficiency.
  • Hybrid: Hybrid and hierarchical models use redesigned attention, convolutional token processing, adjacent-token aggregation, patch embedding, or local windows to improve efficiency and spatial modeling.
  • Hybrid: Swin Transformer models local windows and uses cross-window connections such as shifting, but limited local receptive fields challenge long-range interaction modeling.
  • Global Attention: GC ViT’s global attention differs from EdgeViT by computing attention between global query representations rather than only sparse representative-token interactions.

7. Conclusion

GC ViT is a hierarchical vision transformer that captures global context through global query tokens interacting with local regions. Across image classification and high-resolution downstream tasks, it achieves state-of-the-art or competitive results.

  • GC ViT efficiently captures global context by using global query tokens that interact with local regions.
  • GC ViT achieves state-of-the-art ImageNet-1K classification results across various model sizes, surpassing CNN- and ViT-based counterparts.
  • GC ViT achieves state-of-the-art or competitive performance for detection and semantic segmentation on high-resolution images.

A.1. GC ViT Model Configurations

The supplementary configuration table specifies GC ViT architecture choices across variants and identifies the meanings of key layer abbreviations and model-size labels.

  • Table S.1 describes GC ViT architecture configurations and internal hyperparameter choices for models with varying compute loads and parameter counts.
  • LG-SA and Conv denote local or global self-attention and a 3 × 3 convolutional layer, respectively.
  • GC ViT-XT, GC ViT-T, GC ViT-S, and GC ViT-B denote the XTiny, Tiny, Small, and Base variants, respectively.

A.2.1. GLOBAL QUERY

GC ViT’s global query mechanism shares globally generated context across global self-attention modules while local attention models short-range information. Supplementary ablations and visualizations examine its operation, effectiveness, and training behavior.

  • Global query tokens generated at each stage contain contextual information from different image regions and are shared across global self-attention modules.
  • Replacing the global query with global key and value features, or global value features, significantly affects segmentation and downstream detection and segmentation performance.
  • Adding the GC module to Swin Transformers improves ImageNet Top-1 accuracy by +0.9% and removes the need for window shifting to model cross-region interactions.
  • The global attention block reuses a global query computed through Global Token Generation rather than computing a query vector itself.
  • Learned global query token maps illustrate long-range contextual representations captured from different image regions.
  • EMA provides slight ImageNet Top-1 improvement, while GC ViT-T performance remains stable across different batch sizes.
  • Classification models were trained on NVIDIA A100 GPUs with total batch sizes of 1024 or 4096, and each model required about 32 hours on average.
Loading 2206.09959v5…