Source-linked AI summary
PVT v2: Improved Baselines with Pyramid Vision Transformer
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao
TL;DR
PVT v1 is limited by high-resolution attention cost, lost local continuity, and fixed-size positional encoding. PVT v2 addresses these issues with three architectural designs and reports stronger results across classification, detection, and segmentation, including comparisons favoring contemporary Transformer backbones.
Problem
PVT v1 has high computational complexity for high-resolution inputs, loses local continuity with nonoverlapping patches, and cannot flexibly process arbitrary image sizes because of fixed-size positional encoding.
Method
PVT v2 improves PVT v1 with linear complexity attention, overlapping patch embedding, and convolutional feed-forward networks.
Results
PVT v2 is stronger than PVT v1 and other state-of-the-art Transformer backbones across image classification, object detection, and semantic segmentation under comparable parameter counts.
Takeaways & Limitations
PVT v2 provides improved baselines for future vision Transformer research, combining linear complexity with stronger performance across fundamental vision tasks.
Takeaways & Limitations
PVT v1’s high-resolution computational cost, lost local continuity, and fixed-size positional encoding define the scope of limitations motivating PVT v2.
Abstract
from arXiv · showhide
Transformer recently has presented encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs, including (1) linear complexity attention layer, (2) overlapping patch embedding, and (3) convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linear and achieves significant improvements on fundamental vision tasks such as classification, detection, and segmentation. Notably, the proposed PVT v2 achieves comparable or better performances than recent works such as Swin Transformer. We hope this work will facilitate state-of-the-art Transformer researches in computer vision. Code is available at https://github.com/whai362/PVT.
1. Introduction
PVT v2 strengthens the PVT v1 framework with three orthogonal design improvements and reports better performance across classification, detection, and segmentation. It also achieves strong results against contemporary Transformer backbones with fewer parameters and GFLOPs in a representative comparison.
- PVT v2 introduces linear complexity attention, overlapping patch embedding, and convolutional feed-forward networks as orthogonal improvements to PVT v1.The improved framework is designed as a stronger and more feasible baseline.
- The three improvements produce better image classification, object detection, instance segmentation, and semantic segmentation performance when used with PVT v1.
- 83.8% top-1 error on ImageNet is reported for PVT v2-B51, with fewer parameters and GFLOPs than Swin-B and Twins-SVT-L.
- 50.2 AP on COCO val2017 is achieved by GFL with PVT-B2, exceeding GFL with Swin-T by 2.6 AP and with ResNet50 by 5.7 AP.
2. Related Work
Related work established pure Transformer and hierarchical Transformer backbones for classification and dense prediction, while later methods added locality, positional flexibility, and convolution-like operations. PVT v1 introduced a four-stage pyramid Transformer for versatile detection and segmentation.
- ViT treats images as fixed-length patch-token sequences and showed that pure Transformers can achieve state-of-the-art image classification with sufficient training data.
- DeiT extended ViT with data-efficient training and knowledge distillation approaches.
- Classification-focused methods modify tokenization, positional encoding, patch scales, or convolutional components to improve ViT.Examples include overlapping sliding-window tokens in T2T ViT, conditional position encodings in CPVT, dual-branch processing in CrossViT, and depth-wise convolution in LocalViT.
- PVT v1 introduced a four-stage hierarchical Transformer with a pyramid structure, showing that pure Transformer backbones can support detection and segmentation like CNN counterparts.
- Swin, CvT, CoaT, LeViT, and Twins improve Transformer backbones through shifted-window or relative-position mechanisms, convolution-like operations, or combined local and global attention.
3. Methodology
PVT v2 addresses three PVT v1 limitations—high attention cost, lost local continuity, and fixed-size positional encoding—with linear SRA, overlapping patch embedding, and convolutional feed-forward designs. Together, these changes support linear complexity, local continuity, and more flexible variable-resolution processing.
- 3.1. Limitations in PVT v1: PVT v1 has high computational complexity on high-resolution inputs, loses local continuity through nonoverlapping patches, and uses inflexible fixed-size positional encoding.These limitations constrain its performance on vision tasks.
- 3.1. Limitations in PVT v1: PVT v2 addresses the PVT v1 limitations through three designs described in the following methodology sections.
- 3.2. Linear Spatial Reduction Attention: Linear SRA uses average pooling to reduce h×w spatial dimensions to fixed P×P before attention, giving linear computational and memory costs like a convolutional layer.For linear SRA, the pooling size P is set to 7.
- 3.3. Overlapping Patch Embedding: Overlapping patch embedding enlarges patch windows so adjacent windows overlap by half their area, using zero-padded convolution with stride S and kernel size 2S −1.
- 3.4. Convolutional Feed-Forward: PVT v2 removes fixed-size positional encoding and inserts a zero-padding position encoding using a 3 × 3 depth-wise convolution inside the feed-forward network.
- PVT v2 Series: The PVT v2 series scales from B0 to B5 by changing hyperparameters, with variants and linear-SRA models documented in the model settings.The series follows a pyramid pattern in which channels increase and spatial resolution shrinks at deeper stages.
- PVT v2 Design Benefits: The combined improvements provide greater local continuity, more flexible variable-resolution processing, and the same linear complexity as CNNs.
4. Experiment
Experiments evaluate PVT v2 across image classification, object detection, instance segmentation, and semantic segmentation, showing improved accuracy and reduced computation relative to PVT v1 and recent counterparts. Ablations attribute gains to overlapping patch embedding, convolutional feed-forward networks, and linear SRA.
- Image Classification: PVT v2 improves ImageNet classification accuracy over PVT v1 and achieves 83.8% top-1 accuracy with fewer parameters and GFLOPs than comparable recent models.PVT v2-B5 is reported as 0.5% higher than Swin Transformer and Twins while using fewer parameters and FLOPs.
- Object Detection and Instance Segmentation: PVT v2-B4 reaches 46.1 AP with RetinaNet and 47.5 APb with Mask R-CNN, surpassing PVT v1 by 3.5 AP and 4.6 APb, respectively.The experiments cover one-stage and two-stage detectors on COCO val2017.
- Comparison with Swin Transformer: On ATSS, PVT v2 achieves 49.9 AP, 2.7 higher than Swin-T with similar parameters and FLOPs; PVT v2-Li reduces computation from 258 to 194 GFLOPs with little performance sacrifice.The comparison uses matched ImageNet pre-training and COCO fine-tuning settings.
- Semantic Segmentation: PVT v2 backbones improve semantic segmentation efficiency and accuracy, with PVT v2-Large reaching 48.7 mIoU versus 40.2 for ResNeXt101-64x4d at 12% lower GFLOPs.The evaluation uses Semantic FPN on ADE20K.
- Ablation Study: Overlapping patch embedding improves ImageNet top-1 accuracy from 79.8% to 81.1% and COCO AP from 40.4 to 42.2 by modeling local image continuity.The comparison is between original patch embedding and overlapping patch embedding.
- Ablation Study: CFFN adds 0.9 ImageNet points and 2.4 COCO points, while LSRA cuts GFLOPs by 22% with comparable ImageNet accuracy and only a 1-point lower COCO AP.LSRA achieves 82.1% versus 82.0% top-1 accuracy and 43.6 versus 44.6 COCO AP.
5. Conclusion
PVT v2 improves PVT v1 with three architectural designs and demonstrates stronger performance than its predecessor and other transformer-based backbones across major vision tasks.
- PVT v2 combines overlapping patch embedding, convolutional feed-forward networks, and linear spatial reduction attention.These three designs are presented as improvements to PVT v1.
- Extensive experiments cover image classification, object detection, and semantic segmentation.
- PVT v2 is stronger than PVT v1 and other state-of-the-art transformer-based backbones under comparable parameter counts.
- The authors position these improved baselines as a reference for future vision Transformer research.