PaperlayerView exact revision on arXiv

Source-linked AI summary

P2T: Pyramid Pooling Transformer for Scene Understanding

Yu-Huan Wu, Yun Liu, Xin Zhan, Ming-Ming Cheng

arXiv:2106.12011v6cs.CV

TL;DR

Vision transformers incur high cost from long image-token sequences, and single pooling can yield less powerful pooled features. P2T adapts pyramid pooling to MHSA to reduce sequence length while capturing contextual features, and experiments report superiority across image classification, semantic segmentation, object detection, and instance segmentation.

  • Problem

    Vision transformers face quadratic MHSA cost from long image-token sequences, while single pooling reduces length but can produce less powerful pooled features.

  • Method

    P2T adapts pyramid pooling to MHSA, using pooling-based attention to reduce sequence length and learn contextual representations.

  • Results

    P2T substantially outperforms previous CNN- and transformer-based backbones across image classification, semantic segmentation, object detection, and instance segmentation.

  • Takeaways & Limitations

    Pyramid pooling provides a vision-transformer backbone that combines shorter attention sequences with contextual feature learning across several scene-understanding tasks.

Abstract

from arXiv · show

Recently, the vision transformer has achieved great success by pushing the state-of-the-art of various vision tasks. One of the most challenging problems in the vision transformer is that the large sequence length of image tokens leads to high computational cost (quadratic complexity). A popular solution to this problem is to use a single pooling operation to reduce the sequence length. This paper considers how to improve existing vision transformers, where the pooled feature extracted by a single pooling operation seems less powerful. To this end, we note that pyramid pooling has been demonstrated to be effective in various vision tasks owing to its powerful ability in context abstraction. However, pyramid pooling has not been explored in backbone network design. To bridge this gap, we propose to adapt pyramid pooling to Multi-Head Self-Attention (MHSA) in the vision transformer, simultaneously reducing the sequence length and capturing powerful contextual features. Plugged with our pooling-based MHSA, we build a universal vision transformer backbone, dubbed Pyramid Pooling Transformer (P2T). Extensive experiments demonstrate that, when applied P2T as the backbone network, it shows substantial superiority in various vision tasks such as image classification, semantic segmentation, object detection, and instance segmentation, compared to previous CNN- and transformer-based networks. The code will be released at https://github.com/yuhuan-wu/P2T.

1 INTRODUCTION

Vision transformers face high MHSA cost from long image-token sequences, while single pooling can produce less powerful features. P2T adapts pyramid pooling to MHSA and reports stronger performance across major vision tasks.

  • Motivation: Image tokens can produce sequence lengths of 3136 for 224 × 224 images with 4 × 4 patches, making MHSA computationally and spatially expensive.MHSA complexity is quadratic in image size, unlike CNN complexity, creating a major cost for direct vision-transformer application.
  • Motivation: Single pooling reduces MHSA cost but models token-to-region relationships, and the resulting pooled feature can be less powerful.PVT and MViT use one pooling operation, whereas Swin Transformer restricts attention to local windows and expands receptive fields through shifting.
  • Results: Across image classification, semantic segmentation, object detection, and instance segmentation, P2T is reported to outperform previous CNN- and transformer-based backbones.The paper presents this as the outcome of extensive experiments across several fundamental vision tasks.
  • Proposed approach: P2T adapts pyramid pooling to MHSA, simultaneously shortening image-token sequences and extracting richer contextual representations.The approach uses multiple pooling operations with different receptive fields and strides to address the limitations of single pooling.
  • Proposed approach: P2T embeds pooling-based MHSA into a vision-transformer backbone designed to be flexible for visual recognition.The authors describe the backbone as applicable across image classification, semantic segmentation, object detection, and instance segmentation.

2 RELATED WORK

Prior vision systems evolved from CNN-based feature extraction toward transformers that model global dependencies, motivating transformer backbones for computer vision.

  • 2.1 Convolutional Neural Networks: CNN research developed deeper, residual, cardinality-based, densely connected, lightweight, and searched architectures for visual recognition.The related-work discussion covers VGG, GoogleNet, ResNet, ResNeXt, Res2Net, DenseNet, MobileNet, ShuffleNet, EfficientNet, and MnasNet.

Pyramid Pooling Transformer

P2T adapts pyramid pooling to vision-transformer attention, replacing traditional MHSA with pooling-based MHSA and producing multistage feature maps for scene understanding.

  • Architecture: Pooling-based MHSA replaces traditional MHSA in P2T, and the resulting feature maps {B1, B2, B3, B4} support downstream scene-understanding tasks.The architecture is organized into stages whose feature maps can be reused by task-specific heads.
  • Pooling-based MHSA: P2T adapts pyramid pooling to MHSA to reduce sequence length while learning powerful contextual representations with negligible additional computational cost.The paper also reports compatibility with patch embedding, positional encoding, and feed-forward-network techniques.
  • Pyramid pooling: Pyramid pooling applies multiple pooling operations to extract contextual representations across different receptive fields and strides.Its prior use spans image classification, object detection, and semantic segmentation, often with CNN backbones.
  • Pyramid Pooling Transformer: P2T transfers pyramid pooling from task-specific CNN applications into a broadly usable vision-transformer backbone.The proposed backbone is intended for feature representation learning across multiple vision tasks rather than one specific task.
  • Results: P2T is reported to achieve superior performance to existing CNN- and transformer-based networks across several vision tasks.The evaluated tasks include image classification, semantic segmentation, object detection, and instance segmentation.

3 METHODOLOGY

P2T integrates pyramid pooling into transformer self-attention to reduce token-sequence cost while retaining multi-scale contextual information. Its backbone combines pooling-based MHSA with transformer blocks, convolution-enhanced FFNs, and multi-stage feature representations.

  • Pyramid Pooling Transformer: P2T uses four stages with progressively reduced spatial resolutions, producing four feature representations for classification and downstream scene-understanding tasks.Only the final representation is used for image-classification prediction, while all pyramid features can support downstream tasks.
  • Pyramid Pooling Transformer: P2T adapts pyramid pooling to MHSA, reducing computational load while capturing rich contextual information within a transformer backbone.The design is presented as a generic backbone for scene understanding rather than task-specific pooling atop a CNN.
  • Pyramid Pooling Transformer: Each P2T block applies pooling-based MHSA with residual addition and LayerNorm, followed by an FFN with another residual connection and LayerNorm.The block is expressed as Xatt = LayerNorm(X + P-MHSA(X)) and Xout = LayerNorm(Xatt + FFN(Xatt)).
  • Pooling-based MHSA: Pooling-based MHSA reshapes input tokens into 2D feature maps and applies multiple average-pooling layers with different ratios to form pyramid features.The pooled maps are depthwise-convolved for relative positional encoding, then flattened and concatenated.
  • Pooling-based MHSA: The concatenated pooled representation can be shorter than the input sequence while providing contextual abstraction as a substitute during MHSA.Because keys and values have smaller sequence lengths than the input, P-MHSA is more efficient than traditional MHSA.
  • Pooling-based MHSA: P-MHSA uses default pooling ratios {12, 16, 20, 24}, yielding M ≈ N/66.3 ≈ N/8^2 for the concatenated pooled sequence.Here, M denotes the concatenated sequence length of all pooled features, and the pooling operation has negligible O(NC) complexity.

4 EXPERIMENTS

Experiments evaluate P2T as a backbone for image classification, semantic segmentation, object detection, and instance segmentation, using standard benchmark datasets and frameworks. Across these tasks, P2T generally outperforms CNN- and transformer-based competitors, while ablations support multiple pyramid pooling operations and average pooling.

  • Experimental setup: P2T is evaluated for image classification, semantic segmentation, object detection, and instance segmentation using ImageNet-1K, ADE20K, and MS-COCO benchmarks.The experiments use standard backbone-based setups, including Semantic FPN for segmentation and RetinaNet for object detection.
  • Image classification: 11.3%/3.9%/3.7%/2.4% higher top-1 accuracy than ResNet-18/50/101 and ResNeXt-101-64x4d, respectively, for P2T-Tiny/Small/Base/Large.The corresponding running times are 2.98/1.70/1.58/1.15 times those comparator models.
  • Semantic segmentation: P2T-Small/Base/Large improve semantic segmentation over Swin-T/S/B by 5.2%/3.5%/3.4%, respectively, on ADE20K.P2T-Tiny/Small/Base/Large also exceed the cited ResNet and ResNeXt backbones by 10.5%/10.0%/9.9%/9.2%, respectively, with fewer parameters and GFlops.
  • Object detection: P2T-Small achieves 2.9%, 1.4%, and 0.6% higher AP than Swin-T, Twins-SVT-S, and PVTv2-B2, respectively, for MS-COCO object detection.P2T achieves the best performance under all tiny, small, and large complexity settings, and P2T consistently outperforms PVTv2 with fewer parameters, less computational cost, and faster speed.
  • Ablation studies: Four parallel pooling operations produce the best ablation performance at a comparable squeezed ratio, whereas a single operation with a large pooling ratio performs poorly.Average pooling is selected as the default because it outperforms max pooling and depthwise convolution on ImageNet classification accuracy and ADE20K mIoU.

5 CONCLUSION

P2T introduces pyramid pooling into MHSA to reduce sequence length while learning contextual representations, forming a backbone evaluated across fundamental vision tasks.

  • Pyramid pooling adapts MHSA to reduce sequence length and learn powerful contextual representations simultaneously.
  • The resulting Pyramid Pooling Transformer (P2T) is a universal vision transformer backbone.
  • P2T is evaluated on image classification, semantic segmentation, object detection, and instance segmentation.
  • Experiments show that P2T significantly outperforms previous CNN- and transformer-based backbone networks.
Loading 2106.12011v6…