PaperlayerView exact revision on arXiv

Source-linked AI summary

MViTv2: Improved Multiscale Vision Transformers for Classification and Detection

Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, Christoph Feichtenhofer

arXiv:2112.01526v2cs.CV

TL;DR

High-resolution image, detection, and video tasks make transformer attention costly, motivating a unified and efficient vision backbone. The paper improves MViT with decomposed positional embeddings and residual pooling connections, evaluates five sizes across three domains, and reports strong benchmark performance. MViTv2 achieves 88.8% ImageNet accuracy, 58.7 APbox on COCO, and 86.1% on Kinetics-400.

  • Problem

    High-resolution object detection and space-time video understanding remain challenging for ViTs because self-attention has quadratic compute and memory complexity.

  • Method

    MViTv2 improves MViT with decomposed location-distance positional embeddings, residual pooling connections, and five model sizes evaluated across image, detection, and video tasks.

  • Results

    88.8% accuracy on ImageNet-1K, 58.7 APbox on COCO object detection, and 86.1% accuracy on Kinetics-400 are reported for MViTv2.

  • Takeaways & Limitations

    MViT provides a general hierarchical architecture with strong performance across image classification, object detection, instance segmentation, and video recognition.

  • Takeaways & Limitations

    Q and (K, V) can reside at different scales because their pooling operations may differ, requiring a shared-scale index mapping.

Abstract

from arXiv · show

In this paper, we study Multiscale Vision Transformers (MViTv2) as a unified architecture for image and video classification, as well as object detection. We present an improved version of MViT that incorporates decomposed relative positional embeddings and residual pooling connections. We instantiate this architecture in five sizes and evaluate it for ImageNet classification, COCO detection and Kinetics video recognition where it outperforms prior work. We further compare MViTv2s' pooling attention to window attention mechanisms where it outperforms the latter in accuracy/compute. Without bells-and-whistles, MViTv2 has state-of-the-art performance in 3 domains: 88.8% accuracy on ImageNet classification, 58.7 boxAP on COCO object detection as well as 86.1% on Kinetics-400 video classification. Code and models are available at https://github.com/facebookresearch/mvit.

1. Introduction

MViTv2 improves pooling attention and applies a unified multiscale architecture across image classification, object detection, and video classification. It achieves strong benchmark results while comparing favorably with local window attention.

  • Motivation: ViT-based models face severe compute and memory challenges on high-resolution detection and space-time video tasks because self-attention scales quadratically.Window attention and pooling attention are introduced as strategies to reduce this burden.
  • Improvements: MViTv2 adds decomposed location-distance positional embeddings and residual pooling connections to improve pooling attention.These upgrades inject position information and compensate for pooling-stride effects.
  • Attention comparison: Pooling attention is more effective than local window attention mechanisms, while Hybrid window attention improves the accuracy/compute tradeoff.The paper combines pooling attention with a simple hybrid scheme to add complementary cross-window connections.
  • Unified architecture: MViTv2 is instantiated in five sizes and applied with minimal modification to image classification, object detection, and video classification.The study evaluates whether one model family can serve as a general vision architecture across spatial and spatiotemporal recognition.
  • Results: 88.8% accuracy is achieved on ImageNet-1K, 58.7 APbox on COCO detection, and 86.1% on Kinetics-400 video classification.The reported results span image classification, object detection, and video recognition.

2. Related Work

Related work develops vision transformers, efficient attention mechanisms, and multiscale architectures for image, detection, and video tasks. MViTv2 builds on MViT while studying pooling attention as a general alternative to local attention.

  • Vision backbones: CNNs remain primary backbones across image recognition, object detection, and video recognition.
  • Vision transformers: Vision transformers began with ViT and expanded through efficient training, multiscale structures, and advanced self-attention designs.This paper builds on MViT to study a general backbone for different vision tasks.
  • Object detection: High-resolution detection makes transformer self-attention expensive, motivating shifted-window, Longformer, and pooling-attention approaches.The paper studies MViT for detection and compares pooling attention with local attention mechanisms.
  • Video recognition: Video vision transformers often rely on large-scale external pretraining, whereas MViTv1 provides a training-from-scratch recipe on Kinetics data.MViTv2 uses that recipe while improving pooling attention and examining ImageNet pretraining effects.

3. Revisiting Multiscale Vision Transformers

MViT builds a hierarchical feature pyramid by progressively increasing channel width and reducing sequence resolution. Pooling attention performs this downsampling while reducing self-attention compute and memory.

  • Multiscale hierarchy: MViT uses multiple stages for low- and high-level visual modeling instead of single-scale transformer blocks.Across stages, channel width D increases while sequence length L decreases.
  • Pooling attention: Pooling attention projects an input sequence X ∈ R^L×D into query, key, and value tensors before applying pooling operators.The pooled tensors are then used for self-attention.
  • Pooling attention: Pooling factors can reduce the lengths of query, key, and value tensors independently, producing an output sequence with flexible length.The query length is denoted ˜L, and key/value pooling may differ from query pooling.
  • Pooling attention: Pooled self-attention computes Attn(Q, K, V) = Softmax(QK^T / √D) after the pooling operations.
  • Efficiency: Pooling the query enables resolution reduction between MViT stages, while pooling keys and values significantly reduces compute and memory complexity.

4. Improved Multiscale Vision Transformers

This section improves MViT pooling attention with decomposed relative positional embeddings and residual pooling, then applies the resulting backbone across detection and video settings. It also describes multiscale configurations and attention variants for managing computation while retaining global or cross-window information.

  • Improved pooling attention: Decomposed relative positional embeddings encode positional relationships along height, width, and optionally temporal axes to reduce learned embeddings to O(T + W + H).The decomposition addresses the O(TWH) cost of undivided embeddings and is especially useful for early high-resolution feature maps.
  • Improved pooling attention: Residual pooling adds the pooled query tensor to the attention output, increasing information flow while preserving low-complexity attention with large key and value strides.The output sequence has the same length as the pooled query tensor.
  • MViT for object detection: MViT’s four-stage hierarchical features integrate with FPN, whose top-down pyramid and lateral connections provide semantically strong feature maps at all scales for detection architectures such as Mask R-CNN.The backbone is applied to object detection and instance segmentation.
  • Attention mechanisms: Pooling attention downsamples features before global self-attention, whereas window attention preserves resolution but restricts self-attention to non-overlapping local windows.The two mechanisms reduce attention cost through different operations on query, key, and value tensors.
  • Attention mechanisms: Hybrid window attention adds cross-window connections by using local attention in most blocks and global-information blocks in the last three FPN-feeding stages.This design places global information in the feature maps passed to FPN.
  • Model variants and video recognition: The architecture is instantiated in five variants with different channel widths, block counts, head counts, parameters, and FLOPs, while video models extend pooling and positional embeddings to spatiotemporal feature maps.Video adaptation also changes the patchification stem to produce space-time cubes.

5. Experiments: Image Recognition

Experiments show that MViTv2 improves ImageNet classification and COCO detection while offering favorable accuracy–compute trade-offs across attention mechanisms and model scales.

  • Image Classification: 83.6% accuracy lets MViTv2-S improve +0.6% over MViTv1-B-16 with 10% fewer FLOPs, while MViTv2-B reaches 84.4% and improves +1.0% over MViTv1-B-24 with fewer FLOPs.MViTv2-B also surpasses DeiT-B and Swin-B by 2.6% and 1.1%, respectively, with over 33% fewer FLOPs and parameters.
  • Object Detection: 58.7 APbox is achieved on COCO using Cascade Mask R-CNN with SoftNMS and multiscale testing, exceeding Swin’s 58.0 APbox.With Mask R-CNN, MViTv2-B outperforms Swin-B by +2.5/+2.3 in APbox/APmask, while MViTv2-L with ImageNet-21K pretraining reaches 52.7 APbox.
  • Attention Ablations: Pooling attention achieves a similar ViT-B accuracy with ∼38% fewer FLOPs, providing the best accuracy/computation trade-off among the compared ImageNet attention mechanisms.Hybrid window attention outperforms Swin attention by +1.7%, while increasing MViTv2-S pooling stride from 4 to 8 gives the best accuracy/compute trade-off for that model.
  • Attention Ablations: Pooling and pooling plus Hybrid window attention achieve +0.6/+0.3 APbox over standard full attention with ∼2× test speedup on COCO.Combining pooling and Hybrid window attention provides the best reported trade-off in the COCO attention comparison.
  • Attention Ablations: Decomposed relative position embeddings train 3.9× faster than joint relative position embeddings on COCO.Relative positions improve pooling attention by introducing shift-invariance priors, whereas absolute positions provide only a slight improvement over no positional embedding.
  • Runtime Comparison: MViTv2-S surpasses Swin-B on ImageNet-1K by +0.3% and on COCO by +1.4% while achieving higher ImageNet throughput and lower memory cost.MViTv2-S reaches 341 im/s versus 276 im/s on ImageNet-1K and uses 5.2G versus 6.3G memory on COCO.
  • Object Detection: FPN improves MViTv2-S by +2.9 APbox, compared with +1.5 APbox for ViT-B.The reported gain is associated with MViTv2-S’s native hierarchical multi-scale design for dense object detection.

6. Experiments: Video Recognition

The experiments evaluate MViTv2 across Kinetics-400, Kinetics-600, Kinetics-700, and Something-Something-v2, including effects of pretraining and architectural improvements. MViTv2 achieves strong video-recognition accuracy, often with lower computation or fewer parameters than comparison models.

  • Experimental setup: MViTv2 is evaluated on Kinetics-400, Kinetics-600, Kinetics-700, and Something-Something-v2 using established training and inference protocols.Models are generally trained from scratch on Kinetics and fine-tuned from Kinetics models for Something-Something-v2.
  • Kinetics-400: 81.0% and 82.9% top-1 accuracy are achieved by MViTv2-S and MViTv2-B trained from scratch on Kinetics-400, improving over MViTv1 by 2.6% and 2.7%.The training recipe is identical, so the gains stem from the improvements in the architecture.
  • Kinetics-400: 86.1% top-1 accuracy is achieved on Kinetics-400 by MViTv2-L with a 40×3122 time×space input.The model is fine-tuned with a large spatiotemporal input size.
  • Kinetics-600/-700: 87.9% top-1 accuracy is achieved by MViTv2-L on Kinetics-600, while MViTv2-L reaches 79.4% on Kinetics-700, surpassing the previous best by 7.1%.On Kinetics-600, the larger model is reported as state of the art.
  • Something-Something-v2: 73.3% top-1 accuracy is achieved by MViTv2-L on Something-Something-v2, while MViTv2-B reaches 70.5% with fewer FLOPs and parameters than the comparison model.With ImageNet-21K pretraining, MViTv2-B reaches 72.1%.
  • Pretraining: ImageNet-1K or ImageNet-21K pretraining improves Kinetics-400 accuracy for MViTv2-S and MViTv2-B, while large models overfit when trained from scratch.MViTv2-S gains 1.0% with ImageNet-1K and 1.4% with ImageNet-21K pretraining.

7. Conclusion

The paper presents MViTv2 as an improved hierarchical Multiscale Vision Transformer for visual recognition. Across image classification, object detection, instance segmentation, and video recognition, it reports strong performance on widely used benchmarks.

  • MViTv2 is presented as a general hierarchical architecture for visual recognition.
  • The empirical evaluation covers image classification, object detection, instance segmentation, and video recognition.

A.1. Results: COCO Object Detection

The COCO experiments compare MViTv2 within standard and system-level detection frameworks. MViTv2-L achieves 58.7 APbox with multi-scale testing, exceeding the cited Swin-L system result.

  • 58.7 APbox is achieved by MViTv2-L with multi-scale testing, 0.7 AP above the best cited Swin-L result using HTC++.The comparison is system-level and uses different detection frameworks.

A.2. Results: AVA Action Detection

The AVA experiments evaluate MViTv2 for spatiotemporal human-action localization and compare it with MViTv1 under related pretraining and testing settings. MViTv2 improves over MViTv1 and reaches the reported state-of-the-art result with the largest model.

  • Experimental setup: AVA evaluates spatiotemporal localization of human actions using center-crop and full-resolution testing strategies.Center testing uses a 2242 crop, while full testing uses the complete resized image.
  • Results: 26.8 mAP is achieved by MViTv2-S 16×4, improving 2.3 over MViTv1-B 16×4 with fewer FLOPs and parameters.Both models use the same recipe and default K400 pretraining.
  • Results: 29.9 mAP is achieved by MViTv2-B 32×3 with K600 pretraining, improving 1.2 over MViTv1-B-24 32×3.
  • Results: 30.5 mAP is achieved by MViTv2-B with full-resolution testing, a further 0.6 improvement over its reported center-testing result.
  • Results: 34.4 mAP is achieved by MViTv2-L 40×3 with ImageNet-21K and K700 pretraining, reported as state of the art.

A.5. Ablations: Kinetics Action Classification

Ablations on Kinetics-400 show that decomposed relative positional embeddings retain accuracy while substantially improving training efficiency, and that residual pooling components work best together.

  • Positional embeddings: Relative positional embeddings surpass absolute positional embeddings by approximately 0.6% on Kinetics-400.
  • Positional embeddings: Decomposed space/time relative positional embeddings achieve nearly the same accuracy as joint space relative embeddings while training approximately 2× faster.
  • Positional embeddings: Decomposed space/time relative positional embeddings are approximately 8× faster and use approximately 2× fewer parameters than joint space/time relative embeddings.
  • Residual pooling connections: +1.7% accuracy on Kinetics-400 requires combining Q pooling blocks with residual paths; using either separately improves accuracy by only +0.4%.

B. Additional Implementation Details

The implementation applies MViTv2 across image, video, detection, segmentation, and action-recognition settings with task-specific standard frameworks and training recipes. Architectural changes include removing the class token, using average token pooling, and expanding channels in the first block of each stage.

  • Architecture: MViTv2 expands channel dimensions in the first transformer block of each stage, reducing parameters and FLOPs with similar accuracy to MViTv1.The reported accuracy difference is within ±0.1%.
  • Architecture: Removing the class token and averaging final-block output tokens reduces image-classification training time by approximately 8%.
  • Image detection and segmentation: Mask R-CNN with Feature Pyramid Networks is used for object detection and instance segmentation, alongside Mask R-CNN and Cascade Mask R-CNN frameworks.
  • Video classification: Video models are trained from scratch on Kinetics for 200 epochs with synchronized AdamW, repeated augmentation, and 128 GPUs.
  • Video action detection: For AVA action detection, RoIAlign extracts region features from MViTv2 spatiotemporal maps before max pooling and per-class sigmoid prediction.

C. Additional Discussions

The paper discusses MViTv2 as a general backbone with possible benefits across vision applications, while acknowledging deployment risks and limitations in hyperparameter tuning and model scaling.

  • Societal impact: MViTv2 is presented as a general backbone for image recognition, object detection, instance segmentation, video classification, and video detection.
  • Societal impact: Potential applications include visual inspection, cancer and tumor detection, vehicle re-identification, and pedestrian detection.
  • Societal impact: Vision technologies may have negative societal impacts when adopted in harmful or mismanaged applications such as privacy-violating surveillance.
  • Limitations: Hyperparameters for different MViTv2 variants may be suboptimal because experiments mainly follow standard task recipes with lightweight tuning.
  • Limitations: The five variants span tiny to huge models, while scaling toward smaller mobile models or larger models for large-scale data remains future work.
Loading 2112.01526v2…