Source-linked AI summary
Twins: Revisiting the Design of Spatial Attention in Vision Transformers
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, Chunhua Shen
TL;DR
Vision Transformers require effective spatial attention for dense prediction, but standard and locally grouped designs face computational or receptive-field limitations. The paper proposes Twins-PCPVT and Twins-SVT, whose spatial designs achieve strong results across classification, detection, and segmentation while remaining efficient and simple to implement. Both models set new state-of-the-art results across the evaluated visual tasks.
Problem
Vision Transformers need spatial attention designs that address computational complexity and limited receptive fields in dense prediction tasks.
Method
The paper proposes Twins-PCPVT with global sub-sampled attention and Twins-SVT with spatially separable self-attention interleaving local and global attention.
Results
Both architectures perform favorably against state-of-the-art vision transformers across image classification, object detection, and semantic and instance segmentation.
Takeaways & Limitations
The proposed architectures provide efficient, easy-to-implement vision transformer backbones for a range of visual tasks.
Abstract
from arXiv · showhide
Very recently, a variety of vision transformer architectures for dense prediction tasks have been proposed and they show that the design of spatial attention is critical to their success in these tasks. In this work, we revisit the design of the spatial attention and demonstrate that a carefully-devised yet simple spatial attention mechanism performs favourably against the state-of-the-art schemes. As a result, we propose two vision transformer architectures, namely, Twins-PCPVT and Twins-SVT. Our proposed architectures are highly-efficient and easy to implement, only involving matrix multiplications that are highly optimized in modern deep learning frameworks. More importantly, the proposed architectures achieve excellent performance on a wide range of visual tasks, including image level classification as well as dense detection and segmentation. The simplicity and strong performance suggest that our proposed architectures may serve as stronger backbones for many vision tasks. Our code is released at https://github.com/Meituan-AutoML/Twins .
1 Introduction
Vision Transformers offer flexible long-range modeling but face costly spatial attention and limited receptive fields under local windows. The paper revisits spatial attention and proposes efficient architectures that perform favorably across visual tasks.
- Spatial self-attention has computational complexity that grows quadratically with the number of image pixels.
- Locally grouped attention reduces complexity but limits connections between windows and therefore the receptive field, especially for dense prediction.
- PVT reduces practical attention cost by having each query attend to a sub-sampled version of the input tokens.
- Twins revisits spatial attention through Twins-PCPVT and Twins-SVT, including global sub-sampled attention and spatially separable self-attention.SSSA combines locally grouped attention for fine-grained short-distance information with global sub-sampled attention for long-distance information.
- Across classification, detection, and segmentation, both architectures perform favorably against state-of-the-art vision transformers at similar or reduced computational complexity.
2 Related Work
Vision transformers extend transformer-based modeling to vision, but dense prediction remains challenging because detection and segmentation require suitable spatial features. Recent architectures address this through feature pyramids, localized attention, positional encodings, and efficient attention designs.
- CNNs remain standard vision models, with top-performing classification networks also serving as backbones for detection and segmentation.
- Vision transformers use patch embeddings and self-attention, yet applying them to dense prediction tasks such as detection and segmentation remains challenging.
- PVT produces CNN-like feature pyramids, while Swin Transformer restricts attention to local windows for linear token complexity and designs window partitions to exchange information across areas.
- Grouped and Separable Convolutions: Grouped and depthwise convolutions improve efficiency through channel grouping, and spatially separable self-attention shares similarities with this design.
- Positional Encodings: Vision transformers commonly use sinusoidal or learnable positional encodings, while CPVT introduces input-conditioned encodings that perform better than absolute and relative alternatives.
3 Our Method: Twins
Twins introduces two spatial-attention designs: Twins-PCPVT uses global sub-sampled attention with conditional positional encoding, while Twins-SVT interleaves local and global attention. These designs target efficient vision-transformer computation while preserving communication across spatial regions.
- Twins-SVT: Twins-SVT uses spatially separable self-attention by interleaving locally-grouped attention and global sub-sampled attention.LSA operates within sub-windows, while GSA exchanges information through representative keys from those sub-windows.
- Twins-PCPVT: With appropriate positional encodings, PVT-style global attention can achieve performance on par with or better than Swin.The paper attributes PVT’s weaker performance mainly to its absolute positional encodings.
- Twins-PCPVT: Twins-PCPVT replaces PVT’s absolute positional encoding with conditional positional encoding generated by a depth-wise convolutional PEG.The PEG is inserted after the first encoder block of each stage.
- Twins-SVT: For fixed sub-window sizes, locally-grouped attention grows linearly with the number of spatial positions, while GSA uses sub-sampled representatives to reduce global-attention cost.The implementation uses square windows of size 7 and regular strided convolutions for sub-sampling.
- Twins-SVT: Local attention reduces computation but restricts communication between non-overlapping sub-windows, so global sub-sampled attention supplies cross-window interaction.Using full global attention after local attention would retain quadratic complexity; GSA instead summarizes each sub-window with a representative.
- Design rationale: Both architectures are designed for favorable performance against comparable vision transformers with similar or reduced computational complexity.The evaluation covers image classification, object detection, and semantic and instance segmentation.
4 Experiments
Across ImageNet classification, ADE20K segmentation, and COCO detection and instance segmentation, Twins models generally match or outperform comparable vision transformers while using similar or fewer computational resources. Ablations indicate that combining local and global attention and choosing suitable sub-sampling and positional encoding designs are important for performance.
- ImageNet classification: Twins-PCPVT-S matches Swin-T on ImageNet classification with 18% fewer FLOPs, while Twins-SVT-S outperforms Swin-T with about 35% fewer FLOPs.Twins-PCPVT-S also outperforms PVT-small by 1.4%.
- Semantic segmentation: Twins-PCPVT-S outperforms PVT-Small by 4.5% mIoU on ADE20K with comparable FLOPs and also surpasses Swin-T.Twins-PCPVT-B and Twins-PCPVT-L exceed their PVT counterparts by 3.3% and 4.3% mIoU, respectively.
- Semantic segmentation: Twins-SVT-S exceeds Swin-T by 1.7% mIoU with Semantic FPN, while Twins-SVT-L exceeds Swin-B by 0.7% mIoU.With multi-scale testing, Twins-SVT-S exceeds Swin-T by 1.3% mIoU, and Twins-SVT-L reaches 50.2% mIoU under comparable FLOPs.
- Object detection: Twins-PCPVT-S exceeds PVT-Small by 2.6% mAP on COCO RetinaNet, while Twins-SVT-S exceeds Swin-T by 1.5% mAP with 12% fewer FLOPs.The models retain similar advantages in 3× experiments.
- Instance segmentation: In COCO Mask R-CNN experiments, Twins-PCPVT-S improves over PVT-Small by 2.5% mAP, while Twins-SVT variants perform better or slightly better than corresponding Swin models.Twins-PCPVT-B obtains 2.6% higher mAP than PVT-Medium and is on par with Swin.
- Ablation studies: Ablations show that locally grouped attention alone reaches 76.9%, whereas adding global attention in the last stage improves classification by 3.6%; LLG reaches 81.5%.The locally grouped-only setting is associated with a limited receptive field.
5 Conclusion
The paper presents Twins-PCPVT and Twins-SVT as vision transformer backbones for classification and dense prediction. Both achieve state-of-the-art results across the reported tasks.
- Twins-PCPVT and Twins-SVT are proposed as vision transformer backbones for image-level classification and downstream dense prediction tasks.
- Twins-PCPVT explores conditional positional encodings within the pyramid vision transformer design.
- Twins-SVT uses an efficient attention paradigm that interleaves local and global attention.
- Both transformer models set a new state of the art in image classification, object detection, and semantic/instance segmentation.
A Experiment
The section includes a table describing large-model object detection performance on the COCO val2017 split under a 1× schedule.
- The table reports object detection performance for large models.
- The evaluation uses the COCO val2017 split.
- The experiments use a 1× schedule.
B Algorithm
The algorithm implements grouped attention through a PyTorch module that projects inputs to queries, keys, and values, computes scaled attention, and reconstructs the output.
- The LSA algorithm is presented as a PyTorch implementation snippet.
- GroupAttention initializes multi-head projections, dropout layers, output projection, and two group-size parameters.
- The forward pass partitions the spatial dimensions into groups and reshapes the input for grouped attention.
- Attention is computed with scaled query-key products, softmax normalization, and value aggregation before reshaping and projection.
C Architecture Setting
The appendix provides configuration details for Twins-PCPVT and Twins-SVT.
- Table 9 contains configuration details for Twins-PCPVT.
- Together, the tables document configuration settings for both proposed architectures.
- Table 10 contains configuration details for Twins-SVT.