Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Wenhai Wang; Enze Xie; Xiang Li; Deng-Ping Fan; Kaitao Song; Ding Liang; Tong Lu; Ping Luo; Ling Shao

Source-linked AI summary

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao

arXiv:2102.12122v2Feb 24, 2021cs.CV

TL;DR

Dense prediction tasks need a convolution-free backbone that retains high-resolution, multi-scale features without ViT’s adaptation and resource challenges. PVT addresses this with a progressive shrinking pyramid and spatial-reduction attention, improving performance across vision tasks, including 40.4 AP versus ResNet50’s 36.3 AP on COCO object detection.

Problem
ViT is difficult to adapt to dense prediction because it produces single-scale, low-resolution features while incurring relatively high computational and memory costs.
Method
PVT introduces a progressive shrinking pyramid and spatial-reduction attention to produce high-resolution, multi-scale features with reduced resource consumption.
Results
PVT consistently improves performance across classification, detection, and segmentation tasks; RetinaNet+PVT achieves 40.4 AP versus ResNet50’s 36.3 AP on COCO.
Takeaways & Limitations
PVT provides a versatile convolution-free Transformer backbone that can be plugged into diverse dense prediction pipelines and handle higher-resolution feature maps.
Takeaways & Limitations
PVT does not consider several CNN-specific modules and operations, while many heavily engineered CNN backbones remain available for comparison.

Abstract

from arXiv · show

Although using convolutional neural networks (CNNs) as backbones achieves great successes in computer vision, this work investigates a simple backbone network useful for many dense prediction tasks without convolutions. Unlike the recently-proposed Transformer model (e.g., ViT) that is specially designed for image classification, we propose Pyramid Vision Transformer~(PVT), which overcomes the difficulties of porting Transformer to various dense prediction tasks. PVT has several merits compared to prior arts. (1) Different from ViT that typically has low-resolution outputs and high computational and memory cost, PVT can be not only trained on dense partitions of the image to achieve high output resolution, which is important for dense predictions but also using a progressive shrinking pyramid to reduce computations of large feature maps. (2) PVT inherits the advantages from both CNN and Transformer, making it a unified backbone in various vision tasks without convolutions by simply replacing CNN backbones. (3) We validate PVT by conducting extensive experiments, showing that it boosts the performance of many downstream tasks, e.g., object detection, semantic, and instance segmentation. For example, with a comparable number of parameters, RetinaNet+PVT achieves 40.4 AP on the COCO dataset, surpassing RetinNet+ResNet50 (36.3 AP) by 4.1 absolute AP. We hope PVT could serve as an alternative and useful backbone for pixel-level predictions and facilitate future researches. Code is available at https://github.com/whai362/PVT.

1. Introduction

The paper proposes PVT, a pure Transformer backbone designed to overcome ViT’s limitations for high-resolution, pixel-level dense prediction. It introduces a progressive shrinking pyramid and spatial-reduction attention, and demonstrates gains across vision tasks, including object detection.

Limitations of prior approaches: ViT is difficult to adapt to dense prediction because resource constraints force coarse input patches, producing low-resolution output feature maps.The passage gives 32×32-pixel patches and an example 32-stride output resolution.
Motivation and contribution: PVT is proposed as an alternative, convolution-free backbone for image-level and pixel-level dense prediction tasks.The target tasks include object detection, semantic segmentation, and instance segmentation.
PVT design: PVT uses fine-grained 4×4-pixel patches, a progressive shrinking pyramid, and spatial-reduction attention to learn high-resolution, multi-scale features with lower resource consumption.These designs address the difficulty of porting Transformer architectures to dense prediction.
PVT advantages: PVT provides a global receptive field and its pyramid structure makes it easier to plug into representative dense prediction pipelines.The paper contrasts this with CNNs’ receptive fields increasing with network depth and ViT’s columnar structure.
Experimental results: 40.4 AP: RetinaNet+PVT-Small outperforms RetinaNet+ResNet50 by 4.1 points on COCO val2017 at a comparable parameter count.The reported comparison is 40.4 versus 36.3 AP.

2. Related Work

Related work establishes CNNs as the dominant vision backbone, while Vision Transformers remain less developed and face challenges for dense prediction. Dense prediction requires pixel-level outputs, motivating architectures that support high-resolution or multi-scale representations across detection and segmentation.

CNN Backbones: CNNs use shared convolutional kernels to capture visual context and provide translation equivariance across the image.The passages describe CNNs as foundational and dominant in visual recognition.
Vision Transformers: Vision Transformer backbones remain in an early development stage, motivating efforts to extend them into versatile backbones for most vision tasks.The stated goal is to design a Transformer backbone suitable for broad vision applications.
Dense Prediction: Dense prediction performs pixel-level classification or regression, with object detection and semantic segmentation as representative tasks.These tasks operate on feature maps and require spatially detailed outputs.
Object Detection: Object detectors commonly rely on high-resolution or multi-scale feature maps for accurate detection, spanning single-stage and multi-stage frameworks.Examples include RetinaNet and FCOS among single-stage detectors, and Faster R-CNN and Mask R-CNN among multi-stage detectors.
Semantic Segmentation: CNNs remain important in semantic segmentation, including fully convolutional and encoder–decoder approaches such as FCN and U-Net.FCN generates spatial segmentation maps, while U-Net targets medical image segmentation.
Attention-Based Alternatives: Fixed convolutional filters motivate dynamic filters and self-attention, but non-local operators incur high computational and memory costs.The passage identifies limited input adaptivity in fixed filters and efficiency concerns for non-local modeling.

3. Pyramid Vision Transformer (PVT)

PVT introduces a four-stage progressive-shrinking Transformer pyramid that produces multi-scale feature maps for dense prediction tasks. Its spatial-reduction attention lowers computational and memory costs, enabling processing of high-resolution feature maps.

Pyramid structure: PVT uses four stages combining patch embedding with Transformer encoder layers to generate feature maps at progressively different scales.The stages produce a feature pyramid with strides of 4, 8, 16, and 32 pixels relative to the input image.
Pyramid structure: The progressive-shrinking strategy adjusts feature-map scale through stage-specific patch embeddings, constructing a multi-scale Transformer feature pyramid.Each patch embedding reduces spatial height and width according to the stage’s patch size Pi.
Spatial-reduction attention: SRA reduces the spatial scale of keys and values before attention, substantially lowering computational and memory overhead compared with standard MHA.The attention costs are R_i^2 times lower than those of MHA, allowing larger input feature maps or sequences under limited resources.
Model variants: PVT models are provided in Tiny, Small, Medium, and Large scales, with parameter counts comparable to ResNet18, ResNet50, ResNet101, and ResNet152, respectively.The paper states that these variants support discussion of different model scales and that downstream-task usage is described later.
PVT versus ViT: Unlike ViT’s single-scale output, PVT generates multi-scale feature maps like a traditional CNN backbone while remaining a pure Transformer without convolutions.The progressive-shrinking pyramid is the primary architectural difference identified between PVT and ViT.

4. Application to Downstream Tasks

PVT is evaluated as a versatile backbone for image classification and dense prediction. The downstream experiments use scaled PVT variants, classification-token prediction, and established detection and segmentation frameworks.

Image classification: PVT-Tiny, -Small, -Medium, and -Large are designed with parameter counts similar to ResNet18, 50, 101, and 152, respectively.Detailed hyper-parameter settings are provided in the supplementary material.
Image classification: For image classification, PVT appends a learnable classification token at the last stage and uses a fully connected layer for prediction.This follows the classification approach used in ViT and DeiT.
Dense prediction: PVT is applied to RetinaNet, Mask R-CNN, and Semantic FPN to evaluate backbones for object detection, instance segmentation, and semantic segmentation.These represent single-stage detection, two-stage instance segmentation, and vanilla semantic segmentation, respectively.
Dense prediction: For dense prediction, PVT outputs the feature pyramid {F1, F2, F3, F4} to FPN, whose refined feature maps feed the detection or segmentation head.PVT layers remain unfrozen during downstream training, and the backbone is initialized with ImageNet-pretrained weights.
Dense prediction: Bilinear interpolation adapts ImageNet-pretrained position embeddings to arbitrary detection and segmentation input resolutions.The adaptation addresses the possibility that pretrained position embeddings are not meaningful for inputs with different shapes.

5. Experiments

Experiments show that PVT provides competitive or superior performance across image classification, object detection, instance segmentation, and semantic segmentation, while its pyramid structure enables high-resolution dense prediction at manageable cost. Ablations further show gains over ViT and CNN backbones, benefits from deeper designs, and a speed–accuracy tradeoff tied to input resolution.

Image classification: PVT-Small reaches 20.2 top-1 error versus ResNet50’s 21.5 under roughly similar GFLOPs, while PVT-Large matches ViT(DeiT)-Base/16 at 18.3.These results indicate competitive classification performance under similar or lower complexity.
Instance segmentation: 35.1 mask AP from PVT-Tiny exceeds ResNet18’s 31.2 and ResNet50’s 34.4, while PVT-Large reaches 40.7 versus ResNeXt101-64x4d’s 39.7 with 20% fewer parameters.The instance-segmentation results are reported with Mask R-CNN using the 1× training schedule.
Semantic segmentation: 42.1 mIoU from PVT-Large exceeds ResNeXt101-64x4d’s 40.2 despite 20% fewer parameters and GFLOPs, while longer training and multi-scale testing raise it to 44.8.PVT-Tiny/Small/Medium are also at least 2.8 points higher than similarly sized ResNet-18/50/101 backbones.
Pyramid structure: 40.4 AP from PVT exceeds ViT-Small/32’s 31.7 by 8.7 points on COCO val2017, while progressive shrinking processes high-resolution shallow features and low-resolution deep features.Using fine-grained 4×4-pixel patches, ViT exhausts 32G of GPU memory, whereas PVT avoids this problem through its pyramid.
Ablation studies: PVT-Medium consistently outperforms parameter-matched PVT-Small-Wide on ImageNet and COCO, showing that going deeper is more effective than going wider.The comparison scales PVT-Small hidden dimensions by 1.4 to match PVT-Medium’s parameter count.
Efficiency: At 640-pixel input resolution, PVT-Small RetinaNet runs at 51.7ms versus ResNet50’s 55.9ms and achieves 38.7 AP versus 36.3, reversing its speed disadvantage at 800 pixels.At the original 800-pixel shorter side, PVT-Small is slower than the ResNet50-based model.

6. Conclusions and Future Work

PVT is introduced as a pure Transformer backbone for dense prediction, using progressive shrinking and spatial-reduction attention to produce high-resolution, multi-scale features efficiently. Experiments show stronger performance than comparable CNN backbones, while future work includes adapting CNN-specific techniques and addressing competition from well-engineered CNNs.

Contributions: PVT is a pure Transformer backbone designed for dense prediction tasks including object detection and semantic segmentation.
Computational efficiency: PVT-Small and ResNet50 have similar GFLOPs when input scales are below 640 × 640.
Contributions: Progressive shrinking pyramids and spatial-reduction attention provide high-resolution, multi-scale feature maps under limited computation and memory.
Experimental validation: Extensive object detection and semantic segmentation experiments show PVT outperforming well-designed CNN backbones with comparable parameter counts.
Future work: Future work could incorporate CNN-specific modules and operations, including SE, SK, dilated convolution, model pruning, and NAS, while considering well-engineered CNN backbones.