PaperlayerView exact revision on arXiv

Source-linked AI summary

MPViT: Multi-Path Vision Transformer for Dense Prediction

Youngwan Lee, Jonghee Kim, Jeff Willette, Sung Ju Hwang

arXiv:2112.11010v2cs.CV

TL;DR

Dense prediction requires multi-scale representations, while many Vision Transformer backbones rely on single-scale patches or mechanisms focused mainly on attention efficiency. MPViT combines multi-scale patch embedding, parallel Transformer paths, global-to-local feature interaction, and feature aggregation, achieving state-of-the-art results across classification, detection, and segmentation. Its main reported limitation is slower inference than some competing backbones.

  • Problem

    Dense prediction needs effective multi-scale feature representation for objects and regions of varying sizes, but Vision Transformer backbones commonly use single-scale patches in fine-to-coarse structures.

  • Method

    MPViT embeds different-sized patches at the same feature resolution, processes them through parallel Transformer encoders, aggregates their features, and combines Transformer global features with convolutional local features.

  • Results

    MPViT achieves state-of-the-art performance on ImageNet classification, COCO detection and instance segmentation, and ADE20K semantic segmentation.

  • Takeaways & Limitations

    MPViT serves as a versatile backbone for image classification, object detection, instance segmentation, and semantic segmentation.

  • Takeaways & Limitations

    MPViT-S has slower inference than Swin-T and XCiT-S12/16, possibly because its multi-path structure causes suboptimal GPU utilization.

Abstract

from arXiv · show

Dense computer vision tasks such as object detection and segmentation require effective multi-scale feature representation for detecting or classifying objects or regions with varying sizes. While Convolutional Neural Networks (CNNs) have been the dominant architectures for such tasks, recently introduced Vision Transformers (ViTs) aim to replace them as a backbone. Similar to CNNs, ViTs build a simple multi-stage structure (i.e., fine-to-coarse) for multi-scale representation with single-scale patches. In this work, with a different perspective from existing Transformers, we explore multi-scale patch embedding and multi-path structure, constructing the Multi-Path Vision Transformer (MPViT). MPViT embeds features of the same size~(i.e., sequence length) with patches of different scales simultaneously by using overlapping convolutional patch embedding. Tokens of different scales are then independently fed into the Transformer encoders via multiple paths and the resulting features are aggregated, enabling both fine and coarse feature representations at the same feature level. Thanks to the diverse, multi-scale feature representations, our MPViTs scaling from tiny~(5M) to base~(73M) consistently achieve superior performance over state-of-the-art Vision Transformers on ImageNet classification, object detection, instance segmentation, and semantic segmentation. These extensive results demonstrate that MPViT can serve as a versatile backbone network for various vision tasks. Code will be made publicly available at \url{https://git.io/MPViT}.

1. Introduction

Dense prediction needs multi-scale features, but existing Vision Transformer backbones commonly use single-scale patches within fine-to-coarse stages. MPViT addresses this with multi-scale patch embedding, parallel Transformer paths, feature interaction, and aggregation, achieving strong results across vision tasks.

  • Dense prediction requires multi-scale feature representations to distinguish objects or regions of varying sizes.
  • Existing Vision Transformer backbones often use single-scale patches in a simple fine-to-coarse multi-stage structure, paying less attention to effective multi-scale representations.
  • MPViT embeds different-sized patches at the same feature resolution, processes their tokens through independent Transformer encoders, and aggregates the resulting features.The approach uses overlapping convolution operations and introduces global-to-local feature interaction by concatenating convolutional local features with Transformer global features.
  • MPViT-Small uses 22M parameters and 4GFLOPs, while surpassing Focal-Base with 89M parameters and 16GFLOPs.
  • MPViT combines multi-scale embedding and a multi-path structure to represent fine and coarse features simultaneously for dense prediction tasks.
  • MPViT achieves state-of-the-art performance on ImageNet classification, COCO detection and instance segmentation, and ADE20K semantic segmentation.

2. Related works

Related Vision Transformers primarily address self-attention cost through local, sparse, pooled, channel-wise, or cross-layer mechanisms. Their structural choices range from fine-to-coarse pyramids to single-stage designs, while MPViT is presented as a four-stage architecture using factorized self-attention.

  • Recent Vision Transformers reduce dense-prediction self-attention costs using local attention, sliding windows, sparse global attention, spatial reduction, or channel-wise attention.
  • MPViT instead constructs a four-stage feature hierarchy and uses factorized self-attention throughout the model to address the computation of higher-resolution features.
  • Most cited Transformer backbones use a fine-to-coarse pyramid, whereas XCiT uses a single-stage structure and adds downsampling and upsampling for dense prediction.
  • CoaT combines a pyramid backbone with cross-layer attention, while CoaT-Lite uses a simpler pyramid structure.

3. Multi-Path Vision Transformer

MPViT builds a four-stage backbone that combines multi-scale patch embedding, multiple Transformer paths, and global-to-local feature interaction for dense prediction. Its design uses efficient factorized self-attention and scales from Tiny to Base models.

  • Architecture: MPViT uses a four-stage feature hierarchy, with a stem block followed by MS-PatchEmbed and MP-Transformer blocks from stages 2 to 5.The stem produces an H/4 × W/4 feature map, while factorized self-attention is used throughout to reduce complexity.
  • Multi-Scale Patch Embedding: MS-PatchEmbed applies parallel overlapping convolutions with different kernel sizes to produce fine- and coarse-grained tokens at the same resolution.Different patch sizes, such as 3 × 3, 5 × 5, and 7 × 7, can share sequence length through stride and padding adjustments.
  • Model Configuration: MPViT configurations use paths [2,3,3,3] across four stages, with eight Transformer heads in every model.FLOPs are measured with a 224 × 224 input image.
  • Global-to-Local Feature Interaction: Tokens from separate paths are processed by Transformer encoders, while local convolutional features and global Transformer features are concatenated and projected into the next-stage representation.A 1 × 1 convolution implements the interaction function that maps the concatenated feature to the next stage’s channel dimension.
  • Model Configuration: Factorized self-attention has time complexity O(LhNC^2) and memory complexity O(LhC^2 + LhNC), motivating path expansion through reduced channel dimension C.The models use two paths at stage 2 and three paths from stage 3 onward.
  • Model Configuration: Triple-path models are selected because they outperform dual-path models on dense prediction despite similar ImageNet classification accuracy.MPViT models scale from MPViT-Tiny (5M) to MPViT-Base (74M).

4. Experiments

MPViT is evaluated as a versatile backbone across ImageNet classification, COCO detection and instance segmentation, and ADE20K semantic segmentation. Ablations examine path dimension, multi-scale embedding, and global-to-local feature interaction.

  • Benchmark results: MPViT models consistently outperform similarly sized Vision Transformers on ImageNet classification and dense prediction benchmarks.Evaluations cover ImageNet-1K, COCO, and ADE20K using standard backbone integrations and fair-comparison settings.
  • ImageNet classification: 84.3%: MPViT-Base achieves this ImageNet-1K top-1 accuracy while surpassing larger Swin-Base and Focal-Base models.The comparison uses 224 × 224 inputs without distillation or 384 × 384 evaluation.
  • Object detection and instance segmentation: 47.6%: MPViT-S achieves this RetinaNet detection result, exceeding Swin-T and Focal-T by 2.1–2.6%.MPViT-S also outperforms substantially larger Swin and Focal models in the reported comparison.
  • Object detection and instance segmentation: 43.9%: MPViT-S reaches this mask AP, exceeding larger XCiT-M24/8 and Focal-B models with much less FLOPs.For Mask R-CNN, MPViT variants outperform single-path CoaT-Lite baselines; MPViT-S has comparable performance to CoaT-S with less computation.
  • Semantic segmentation: 48.3%: MPViT-S achieves this ADE20K semantic-segmentation performance, surpassing Swin-T, Focal-T, and XCiT-S12/16.MPViT-B also outperforms the larger Focal-B model.
  • Ablation study: Under similar model-size and FLOPs constraints, triple-path is the best tested choice, while deeper and thinner configurations outperform wider alternatives.Expanding paths can reduce memory usage, but the increased total layer count decreases inference speed relative to single-path designs.
  • Ablation study: Series multi-scale embedding improves over parallel embedding while reducing model size and FLOPs, whereas concatenation improves global-to-local feature interaction over addition.The series design provides equivalent receptive fields and includes nonlinear convolutional processing; concatenation preserves local and global features before 1 × 1 convolution.

5. Discussion

MPViT combines multi-path, multi-scale representations to capture objects at varying scales, while offering favorable efficiency in dense prediction. Its main limitation is slower inference than some alternatives.

  • Efficiency: MPViT-S consumes much less memory and runs 4× faster than CoaT Small with comparable detection performance.The comparison uses actual GPU throughput and memory measurements on the same Nvidia V100 GPU.
  • Efficiency: MPViT can scale to larger models than CoaT, whose exhaustive memory usage limits scaling.
  • Qualitative analysis: Fine patches attend to small objects, whereas coarse paths can attend to larger objects and boundaries.In the cited visualization, path-1 captures a smaller ball while path-3 attends to a larger person.
  • Qualitative analysis: Combining fine and coarse features enables MPViT to capture objects of varying scales in visual inputs.
  • Limitation and future work: MPViT has slower inference than Swin-T and XCiT-S12/16, possibly because its multi-path structure causes suboptimal GPU utilization.The paper hypothesizes effects such as GPU context switching and kernel synchronization.

A. Appendix

The appendix provides training details, additional analyses for classification and detection, and further attention-map and failure-case visualizations.

  • The appendix covers training details for ImageNet classification, COCO detection and instance segmentation, and ADE20K semantic segmentation.
  • It also presents further experimental analyses for ImageNet classification and COCO object detection.
  • Additional qualitative analysis examines learned attention maps and failure cases.

A.1. Detailed Experimental Settings

The appendix specifies datasets, training recipes, model-comparison settings, and evaluation protocols for classification, detection, instance segmentation, and semantic segmentation.

  • ImageNet classification: ImageNet-1K classification uses 224×224 inputs and excludes distilled models and models using 384×384 resolution for fair comparison.The training setup follows DeiT and CoaT recipes with extensive data augmentation and 300 training epochs.
  • Object detection and instance segmentation: COCO detection and instance segmentation use RetinaNet and Mask R-CNN with a 3× schedule and multi-scale inputs.Inputs resize the shorter side between 480 and 800 while limiting the longer side to 1333.
  • Semantic segmentation: ADE20K semantic segmentation uses UperNet, 160K iterations, batch size 16, and 512×512 inputs.Small and Base models use stochastic-depth drop rates of 0.2 and 0.4, respectively.

A.2. More Experimental Analysis

Additional experiments report consistent MPViT advantages across classification and dense prediction settings, while also documenting efficiency and comparison protocols.

  • ImageNet classification: MPViTs consistently achieve state-of-the-art ImageNet-1K classification performance against models with similar capacity.Smaller MPViT variants often outperform larger baseline counterparts despite having fewer parameters.
  • ImageNet classification: MPViT is presented as a more efficient and effective Vision Transformer in computation and model parameters.
  • Deformable DETR: Deformable DETR comparisons train MPViT, CoaT-Lite Small, and CoaT Small with the same code base.
  • COCO detection and instance segmentation: Table 10 evaluates COCO detection and instance segmentation with 1× single-scale training, using ImageNet-1K-pretrained backbones.GFLOPs are measured at 800 × 1280 resolution.
  • COCO detection and instance segmentation: MPViTs consistently outperform on RetinaNet and Mask R-CNN under the 3× schedule with multi-scale inputs, including improved PVTv2 models.

A.3. More Qualitative Results

The qualitative analysis visualizes how MPViT’s attention maps are formed and shows complementary scale specialization across its three paths. Compared with CoaT-Lite’s single path, MPViT better separates attention to objects and boundaries of varying sizes.

  • Spatial attention is obtained by averaging softmax(K) across channels, resizing it to the input image, normalizing it to [0,1], and multiplying it by the image.
  • The comparison uses fourth-stage attention maps, selecting the best qualitative map from each eight-head method and visualizing all three MPViT paths.
  • Path-1 emphasizes small objects or textures, whereas path-3 focuses on large objects or high-level semantic concepts.
  • MPViT’s paths independently handle different scales while acting complementarily, supporting dense prediction across varying object sizes.
  • Path-2 exhibits intermediate behavior, transitioning between path-1’s smaller-object attention and path-3’s emphasis on larger objects.
  • CoaT-Lite’s single path attends to large and small objects simultaneously but misses some objects and captures boundaries less precisely than MPViT’s path-3.
Loading 2112.11010v2…