Focal Self-attention for Local-Global Interactions in Vision Transformers

Jianwei Yang; Chunyuan Li; Pengchuan Zhang; Xiyang Dai; Bin Xiao; Lu Yuan; Jianfeng Gao

Source-linked AI summary

Focal Self-attention for Local-Global Interactions in Vision Transformers

Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, Jianfeng Gao

arXiv:2107.00641v1Jul 1, 2021cs.CVcs.AIcs.LG

TL;DR

High-resolution vision tasks make global fine-grained self-attention expensive, motivating a mechanism that preserves both local and global interactions efficiently. The paper introduces focal self-attention and Focal Transformers, which use fine-grained local attention and coarse-grained global attention. Across image classification, object detection, and segmentation, Focal Transformers consistently outperform state-of-the-art vision Transformers, with the largest model reaching 58.9 box mAP and 51.3 mask mAP on COCO test-dev and 55.4 mIoU on ADE20K.

Problem
Fine-grained global self-attention captures short- and long-range dependencies but has quadratic cost on high-resolution feature maps, while local or coarse alternatives lose part of this capability.
Method
Focal self-attention applies fine-grained attention to nearby tokens and coarse-grained attention to summarized distant tokens, and Focal Transformers incorporate it into a multi-scale architecture.
Results
Focal Transformers consistently outperform state-of-the-art vision Transformers across image classification, object detection, and segmentation benchmarks.
Takeaways & Limitations
Focal self-attention provides a generic approach for modeling local-global interactions across multiple vision tasks at a reasonable computational cost.
Takeaways & Limitations
Focal self-attention introduces extra computational and memory cost because each query attends to coarsened global tokens in addition to local tokens.

Abstract

from arXiv · show

Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability of capturing short- and long-range visual dependencies through self-attention is arguably the main source for the success. But it also brings challenges due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection). In this paper, we present focal self-attention, a new mechanism that incorporates both fine-grained local and coarse-grained global interactions. Using this new mechanism, each token attends the closest surrounding tokens at fine granularity but the tokens far away at coarse granularity, and thus can capture both short- and long-range visual dependencies efficiently and effectively. With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers on a range of public image classification and object detection benchmarks. In particular, our Focal Transformer models with a moderate size of 51.1M and a larger size of 89.8M achieve 83.5 and 83.8 Top-1 accuracy, respectively, on ImageNet classification at 224x224 resolution. Using Focal Transformers as the backbones, we obtain consistent and substantial improvements over the current state-of-the-art Swin Transformers for 6 different object detection methods trained with standard 1x and 3x schedules. Our largest Focal Transformer yields 58.7/58.9 box mAPs and 50.9/51.3 mask mAPs on COCO mini-val/test-dev, and 55.4 mIoU on ADE20K for semantic segmentation, creating new SoTA on three of the most challenging computer vision tasks.

1 Introduction

Vision Transformers use self-attention to model both local and global visual dependencies, but fine-grained global attention becomes costly for high-resolution inputs. Focal self-attention addresses this by combining fine-grained local attention with increasingly coarse global attention, and Focal Transformers improve results across major vision benchmarks.

Self-attention enables Vision Transformers to model short- and long-range interactions across image regions.
High-resolution dense prediction makes fine-grained global self-attention difficult because its computational cost grows quadratically with the number of feature-map grids.
Focal self-attention applies fine-grained attention locally and coarse-grained attention globally, with farther regions represented at coarser granularity.
83.5% and 83.8% Top-1 accuracy are achieved by 51.1M- and 89.8M-parameter Focal Transformers on ImageNet-1K.
Focal Transformers consistently outperform state-of-the-art Swin Transformers across six object detection methods and also achieve strong COCO and ADE20K results.The largest model reaches 58.9 box mAP and 51.3 mask mAP on COCO test-dev, plus 55.4 mIoU on ADE20K.

2 Method

Focal Transformer uses multi-scale stages with focal self-attention, combining fine-grained local tokens and coarse-grained distant tokens to scale interactions for high-resolution inputs. Its window-wise implementation pools sub-windows at multiple levels, gathers surrounding keys and values, and computes attention in parallel with reduced cost.

Model architecture: The architecture uses four focal Transformer stages, reducing spatial size by 2 and increasing feature dimension by 2 between stages.Images are patch-embedded from 4 × 4 patches before entering the stages.
Motivation: Standard fine-grained self-attention has quadratic cost in the number of feature-map grids, making high-resolution inputs expensive.For object detection, min(H, W) can reach 800 or larger.
Focal self-attention: Focal self-attention applies fine-grained attention locally and summarized attention globally, covering distant regions with fewer tokens.Using coarser granularity for farther surroundings produces larger receptive fields for the same number of attended visual tokens.
Window-wise attention: Window-wise focal attention partitions the feature map so all queries in a window share surrounding tokens, avoiding per-query duplication.The surrounding tokens are extracted for each query window rather than each individual query token.
Window-wise attention: The module defines focal levels, focal window sizes, and focal region sizes to specify the granularity and spatial extent of attended tokens.The levels extract tokens at multiple granularities, while region sizes count attended sub-windows horizontally and vertically.
Window-wise attention: Sub-window pooling creates fine- and coarse-grained feature maps, with the finest level retaining the input granularity and negligible extra pooling parameters.The focal window size is at most 7 in the described settings.
Window-wise attention: For each query window, keys and values from surrounding regions at all focal levels are gathered and concatenated before attention computation.A relative position bias is included, and independent window computations can run in parallel before the MLP block.
Complexity analysis: Pooling across all focal levels costs O(L(MN)d), while the attention cost depends on the query-window size, focal-region sizes, and feature-map dimensions.The pooling complexity is independent of the sub-window size at each focal level.

3 Related work

Vision Transformers extend Transformer encoders from NLP to visual tokens and have been applied across classification, detection, segmentation, and other vision tasks. Efficient attention methods trade between coarse global context and fine local detail, motivating their joint use.

Vision Transformer adapts the NLP Transformer encoder by splitting images into sequences of visual tokens.
Vision Transformers have demonstrated strong image-classification performance and expanded to self-supervised learning, object detection, and semantic segmentation.
Coarse-grained global attention preserves long-range interactions but loses detailed context surrounding query tokens.
Local fine-grained attention improves efficiency by attending to neighboring tokens within a constant-size window.
The paper argues that both coarse global and fine local attention are important for efficient visual modeling.

4 Experiments

Experiments across classification, detection, instance segmentation, and semantic segmentation show that Focal Transformers consistently outperform comparable baselines, including Swin Transformers, while ablations support the value of multiscale interactions and efficient model capacity.

Image classification on ImageNet-1K: 83.5% Top-1 accuracy is achieved by Focal-Small with 51.1M parameters, while Focal-Base reaches 83.8% with comparable parameters and FLOPs.On ImageNet-1K at 224 × 224 resolution, Focal-Tiny also improves over DeiT-Small/16 by 2.0% and Swin-Tiny by 1.0 point.
Object detection and instance segmentation: 0.7-1.7 points of mAP improvement over Swin Transformer are observed across comparable COCO detection settings, with gains also reported under 3× schedules.The evaluation covers RetinaNet and Mask R-CNN using box mAP and mask mAP.
Object detection and instance segmentation: 1.0-2.3 points over Swin-Tiny are obtained by Focal-Tiny across Cascade R-CNN, ATSS, RepPoints, and Sparse R-CNN.All four detectors use Focal-Tiny as backbone and a 3× training schedule.
System-level comparisons: 58.4 box mAP and 51.3 mask mAP are reached on COCO test-dev by Focal-Large with multi-scale testing, yielding new state-of-the-art results on both metrics.The corresponding mini-val results are 58.1 box mAP and 50.9 mask mAP.
Semantic segmentation: More than 1 point mIoU improvement over Swin-Large is achieved by Focal-Large on ADE20K in both single-scale and multi-scale evaluation.The comparison uses semantic segmentation with Focal-Large as the backbone.
Ablation studies: Focal-Tiny consistently outperforms Swin-Tiny at window sizes 7 and 14, including with the smaller window size.The authors associate this advantage with the model’s long-range interactions.
Ablation studies: Removing window shift severely degrades Swin Transformer, whereas Focal Transformer does not require window shift as a necessary ingredient.The ablation indicates that Focal Transformer can avoid Swin Transformer’s even-layer constraint for alternative window shifting.

5 Conclusion

The paper presents focal self-attention for efficient local-global interactions in vision Transformers and reports superiority across image classification, object detection, and segmentation. It also identifies extra computational and memory costs as a limitation requiring further reduction.

Focal self-attention performs fine-grained local attention and coarse-grained global attention to capture short- and long-range context at reasonable cost.
Focal Transformers demonstrate superiority over state-of-the-art methods on image classification, object detection, and segmentation.
Each query token attends coarsened global tokens in addition to local tokens, introducing extra computational and memory cost.
Reducing the number of Transformer layers is identified as one potential way to reduce this cost, but further study is needed.

A.1 Image classification

The comparison frames focal attention as an efficient combination of local and global operations within Transformer models. Unlike approaches using only one interaction type, it combines both and reports better performance, although with slower running speed than Swin at similar FLOPs.

Focal attention is presented as the first efficient combination of global and local self-attention in a Transformer.
Replacing either global or local self-attention with focal self-attention achieves better performance than using either interaction type alone.
Combining local and global interactions is reported as more effective than using either interaction alone.
Focal Transformer has slower running speed than Swin Transformer despite similar FLOPs.The paper attributes this to coarse-grained global attention and time-consuming extraction of surrounding and global tokens.

A.2 Object detection and segmentation

On COCO detection and segmentation with a 1x schedule, Focal Transformers consistently outperform previous methods, including state-of-the-art Swin Transformers, across all reported metrics.

Focal Transformers consistently outperform previous works, including state-of-the-art Swin Transformers, on all reported RetinaNet and Mask R-CNN metrics.
+1.2 and +1.0 box mAP gains over previous best models are reported for RetinaNet and Mask R-CNN with a 1x schedule, respectively.The corresponding 3x-schedule gains are +0.8 and +0.7 box mAP.

A.3 Model inspections

Model inspections examine learning speed, attention allocation, relative position bias, and benchmark comparisons. They indicate faster learning and a progression from stronger local emphasis in earlier stages toward stronger global emphasis in later stages.

Learning speed comparison: 75.7% top-1 accuracy at the 100-th epoch is achieved by Focal-Tiny, compared with 73.9% for Swin-Tiny.
The inspections cover model comparisons on ImageNet-1k and COCO, including classification, detection, and segmentation result tables.
Attention scores for different token types: Focal attention uses local tokens inside windows, surrounding local tokens, and pooled global tokens as three analyzed attention types.
Attention scores for different token types: Global attention strengthens toward upper layers while attention inside local windows gradually weakens.
Attention scores for different token types: Earlier stages emphasize local details, whereas later stages emphasize global context.
Local-to-global relative position bias: Relative position bias patterns differ between image classification and object detection, with neighboring layers cooperating to extract local and global information.