PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiscale Vision Transformers

Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, Christoph Feichtenhofer

arXiv:2104.11227v1cs.CVcs.AIcs.LG

TL;DR

The paper addresses how transformer models can better represent dense visual signals without relying on large external pre-training. It introduces Multiscale Vision Transformers, which progressively increase channel capacity while reducing resolution, and reports strong video and image-recognition performance, including lower-cost video comparisons and temporal sensitivity.

  • Problem

    Transformer models need an architectural connection to multiscale feature hierarchies for modeling dense visual signals in video and images.

  • Method

    MViT uses channel-resolution scale stages that expand channel capacity while reducing spatial resolution, forming a multiscale feature pyramid.

  • Results

    MViT outperforms concurrent video transformers and single-scale image transformers, while showing significant accuracy decay when video frames are shuffled.

  • Takeaways & Limitations

    Multiscale feature hierarchies provide a transformer architecture applicable across video and image recognition tasks.

Abstract

from arXiv · show

We present Multiscale Vision Transformers (MViT) for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models. Multiscale Transformers have several channel-resolution scale stages. Starting from the input resolution and a small channel dimension, the stages hierarchically expand the channel capacity while reducing the spatial resolution. This creates a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information, and deeper layers at spatially coarse, but complex, high-dimensional features. We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10x more costly in computation and parameters. We further remove the temporal dimension and apply our model for image classification where it outperforms prior work on vision transformers. Code is available at: https://github.com/facebookresearch/SlowFast

1. Introduction

MViT connects multiscale feature hierarchies with transformers by progressively increasing channel capacity while reducing spatial or temporal resolution. This design supports high-resolution processing of simple visual information early and coarse, complex feature modeling later, yielding strong video and image-recognition results.

  • Architectural motivation: MViT connects multiscale feature hierarchies with transformer models through resolution-channel scale stages.The stages form a multiscale pyramid inside the transformer network.
  • Architectural motivation: Early layers retain high spatial resolution and lightweight channels for simple low-level information, while deeper layers model coarse, complex semantics.This allocation reflects the dense nature of visual signals, especially in video.
  • Video recognition: MViT video models use temporal information more strongly than conventional vision transformers under frame-shuffling tests.Conventional models showed no performance decay, whereas MViT models showed significant accuracy decay.
  • Video recognition: MViT provides significant performance gains over concurrent video transformers without external pre-training data.The evaluation covers Kinetics, Charades, SSv2, and AVA.
  • Video recognition: 6.8× higher FLOPs and 8.5× more parameters are required by ViViT-L than MViT at equal accuracy, alongside ImageNet-21K pre-training.The comparison is reported for video-level inference on Kinetics-400.
  • Image recognition: Removing the temporal dimension extends the architecture to image classification, where it shows significant gains over single-scale vision transformers.The image model is derived from the video architecture selected through Kinetics ablations.

2. Related Work

Related work spans convolutional networks, self-attention mechanisms, vision transformers, and efficient transformer designs. Concurrent video vision transformers rely on large external pre-training, whereas MViT introduces multiscale feature hierarchies to model dense visual inputs without it.

  • Vision architectures: ConvNets remain standard vision backbones, while self-attention has been applied to image, object, and multimodal understanding.The related work also includes transformer applications across many vision tasks.
  • Efficient transformers: Efficient Transformer research reduces quadratic attention complexity for language processing, complementing MViT’s multiscale architectural approach.The cited efficient-transformer line is presented as complementary rather than equivalent.
  • Video transformers: Concurrent video vision transformers use vanilla ViT adaptations and depend on pre-training on vast external datasets such as ImageNet-21K.MViT instead introduces multiscale feature hierarchies for dense visual input without large-scale external data.

3. Multiscale Vision Transformer (MViT)

MViT connects transformer blocks with multiscale feature hierarchies by progressively increasing channel capacity while reducing spatiotemporal resolution. Its pooling attention shortens sequences and enables stages that move from fine, low-dimensional representations to coarse, high-dimensional ones.

  • Multi Head Pooling Attention: Pooling attention reduces attended sequence length by independently pooling query, key, and value tensors.Pooling uses kernels, strides, and padding, with query pooling reducing output resolution and key/value pooling shortening the attended sequence.
  • Multi Head Pooling Attention: Pooling attention can reduce query resolution, key/value sequence length, or both within a transformer block.The mechanism supports separate query pooling and key/value pooling choices, subject to equal key and value strides.
  • Computational Analysis: Attention computation scales quadratically with sequence length, so pooling reduces the model’s compute and memory requirements.The reported per-head runtime is O(THWD/h(D + THW/fQfK)) and memory is O(THWh(D/h + THW/fQfK)).
  • Scale stages: Unlike ViT’s constant channel capacity and resolution, MViT uses scale stages with identical dimensions within each stage and transitions that downsample space-time features while increasing channels.Each stage contains multiple transformer blocks operating at a shared scale; stage transitions pool attention inputs and expand MLP dimensions.
  • Multiscale Vision Transformers (MViT): MViT progressively grows channel dimension while reducing spatiotemporal resolution across stages.Early layers retain fine space-time resolution with smaller channel capacity, while later layers use coarser resolution and larger channel capacity.

4. Experiments: Video Recognition

MViT delivers strong video-recognition results across datasets and tasks, often outperforming ViT or convolutional baselines without external pre-training. Its multiscale design also improves efficiency and yields stronger temporal modeling than ViT.

  • Kinetics-400: MViT-S reaches 76.0% accuracy with 26.1M parameters and 164.5G FLOPs, outperforming ViT-B by +7.5% at 5.5× less compute.Both models use identical train/validation settings.
  • Kinetics-400: MViT-B reaches 78.4% accuracy, a +9.9% boost over ViT-B, while using 2.6× fewer FLOPs and 2.4× fewer parameters.With 32×3 frame sampling, accuracy rises to 80.2%; longer-input fine-tuning reaches 81.2% with 3 spatial and 3 temporal views.
  • Kinetics-600: MViT achieves 83.4% top-1 accuracy on Kinetics-600, with 56.0× fewer FLOPs and 8.4× fewer parameters than ViT-L-ViViT.The result uses 5-clip center-crop testing and no external pre-training.
  • Temporal and long-range recognition: On SSv2, MViT-B reaches 64.7% top-1 accuracy with 16 frames, increasing to 67.7% with more frames and 68.7% for the deeper MViT-B-24.The results support MViT’s temporal-modeling capability on object-interaction videos.
  • Temporal and long-range recognition: On Charades, MViT-B 16×4 improves over SlowFast R50 by +2.0 mAP at similar FLOPs and parameter counts, while on AVA it is competitive with SlowFast and X3D.The AVA comparison uses the same pre-training and testing strategy.
  • Ablations and efficiency: Frame shuffling causes MViT-B accuracy to drop by -7.1%, whereas ViT-B remains robust, indicating that MViT uses temporal information while ViT largely relies on appearance.The reported MViT-B change is 77.2 →70.1; ViT-B’s temporal positional embedding appears effectively ignored.
  • Ablations and efficiency: Average pooling reduces accuracy from 77.2% to 75.4%, while learnable channelwise conv-pooling improves over max pooling by +1.2%.The convolutional pooling variant is used for the reported experiments in Sections 4.1 and 5.
  • Ablations and efficiency: MViT-S has 3.4× higher throughput, +5.8% accuracy, and 3.3× fewer parameters than ViT-B.The authors also report that replacing max-pooling with convolution reduces training speed by ∼20%.

5. Experiments: Image Recognition

MViT models adapted from video classification and trained from scratch on ImageNet outperform comparable vision-transformer baselines while improving accuracy–compute trade-offs. Deeper and wider variants further increase accuracy at competitive or lower computation.

  • Video-designed MViT models are adapted to images by removing the temporal dimension and trained from scratch on ImageNet with the DeiT recipe.
  • MViT-B-16 achieves 82.5% top-1 accuracy with 7.8 GFLOPs, outperforming DeiT-B by +0.7% with 2.3× fewer FLOPs and parameters.
  • MViT-B-24 provides a +0.6% accuracy gain at a slight increase in computation.
  • MViT-B-24-wide reaches 84.3%, a +1.2% gain over DeiT-B↑3842 at 1.7× fewer FLOPs.
  • These results suggest an architectural advantage for Multiscale Vision Transformers over Vision Transformers.

6. Conclusion

The paper presents MViT as a multiscale transformer that hierarchically expands feature complexity while reducing visual resolution. Across video and image recognition, it reports an advantage over single-scale vision transformers.

  • MViT connects multiscale feature hierarchies with transformers by expanding feature complexity while reducing visual resolution.
  • Empirical evaluation shows a fundamental advantage over single-scale vision transformers for video and image recognition.
  • The appendix provides additional ablations, computational analysis, qualitative observations, and implementation details across the evaluated tasks.
  • Figure A.4 evaluates accuracy–complexity trade-offs for varying numbers of inference clips per video using linear and logarithmic cost axes.

A.1. Ablations: Kinetics Action Classification

Kinetics ablations examine inference efficiency, stage-transition skip connections, and training recipes. MViT compares favorably with concurrent video transformers, while normalized skip connections are important for performance.

  • Inference cost: MViT matches concurrent video transformers with substantially lower cost; ViViT-L uses 6.8× higher FLOPs and 8.5× more parameters at equal accuracy.
  • Inference cost: MViT achieves 80.2% accuracy with 0.85 TFLOPs, compared with ViT-L-ViViT’s 80.3% using 5.78 TFLOPs.
  • Training recipe: Concurrent transformer methods require ImageNet-21K pretraining to be competitive, whereas MViT achieves its reported result without external data.
  • Inference cost: Smaller X3D-S and X3D-M models remain more efficient in multiply-add operations, while MViT-B has a compute–accuracy trade-off similar to X3D-XL.
  • Skip-connections: The normalized skip connection obtains 77.2% top-1 accuracy, compared with 74.6% for an un-normalized expansion and similar performance without skip connections.
  • Training recipe: Applying the MViT recipe directly to SlowFast does not improve its models, and higher-capacity R101 models perform worse with that recipe.

A.2. Ablations: ImageNet Image Classification

ImageNet ablations evaluate key–value pooling in MViT-B-16, showing that the default adaptive pooling provides a favorable accuracy–cost balance. More aggressive or absent pooling worsens this trade-off.

  • The experiment reports top-1 accuracy, GFLOPs, parameter count, and training GPU memory for MViT-B-16.
  • Key-Value pooling: A 8×8 pooling stride reduces accuracy from 82.5% to 81.6% while reducing FLOPs and memory by 0.6G and 2.9G.
  • Key-Value pooling: A 2×2 stride increases FLOPs and memory substantially but performs 0.7% worse than the default 4×4 stride.
  • Key-Value pooling: Removing key–value pooling increases FLOPs by 33% and memory consumption by 45% while producing lower accuracy than the default.
  • Key-Value pooling: The ablation concludes that key–value pooling increases accuracy while decreasing computational and memory cost for image classification.

B. Qualitative Experiments: Kinetics

MViT and ViT exhibit distinct attention-distance patterns after training despite identical initialization schemes. MViT maintains heterogeneous, non-monotonic attention across its hierarchy, whereas ViT becomes increasingly distant and more redundant in deeper layers.

  • Experimental setup: The attention-signature difference between MViT and ViT arises from their multiscale skeleton because both use the same weight-initialization scheme.Figure A.6 compares all heads across layers at initialization and convergence.
  • Attention patterns: After training, ViT attention distance increases consistently across layers, while MViT attention distance is non-monotonic.The comparison uses converged models shown in the bottom row of Figure A.6.
  • Attention patterns: MViT heads continue focusing on different features in higher layers, unlike ViT heads whose intra-head variation decreases with depth.The authors interpret the reduced variation in ViT as evidence that some model capacity may be spent on redundant computation.
  • Attention patterns: MViT attends to both close and distant features throughout its network hierarchy.Mean attention distance is normalized, with 1.0 representing the maximum possible distance.

C. Computational Analysis

Pooling reduces the sequence lengths used by attention, lowering MViT’s runtime and memory costs. The resulting channel-resolution trade-off motivates increasing channels and heads as spatio-temporal resolution decreases across stages.

  • Pooling attention: Pooling the key, query, and value vectors directly reduces the compute and memory requirements of the pooling attention operator.Their spatio-temporal resolutions are downsampled according to their respective strides, shortening the resulting sequences.
  • Complexity: The final runtime complexity of MHPA is O(THWD(D + THW/fqfk)).The pooling operation’s additional cost is negligible relative to quadratic attention computation.
  • Complexity: The total space complexity of MHPA is O(THWh(D/h + THW/fqfk)).This includes storage for the sequence and the pooled full attention matrix.
  • Architectural trade-offs: Both runtime and space complexity expose a trade-off between channel dimension D and the reduced sequence-length term THW/fqfk.This trade-off informs two architectural design choices in Multiscale Transformers.
  • Architectural trade-offs: MViT increases channel capacity as spatio-temporal resolution decreases across layers to keep computational time roughly constant across stages.The relevant sequence-length term diminishes as resolution decreases.
  • Architectural trade-offs: MViT starts with few heads and increases the head count as resolution decreases to control memory requirements.For fixed D, more heads increase the memory term involving D + h∗THW/fqfk.

D. Additional Implementation Details

The implementation specifies MViT architecture, optimization, augmentation, and task-specific training and inference procedures across video detection, action classification, and image classification experiments.

  • Implementation: The model is implemented with PySlowFast, with code and models released in the SlowFast repository.The implementation reference is provided directly by the authors.
  • Architecture: MViT uses pre-normalized residual Transformer blocks with Layer Normalization and two-layer GELU MLPs that expand D to 4D before restoring it.Channel dimensions are increased at scale-stage boundaries to match the next stage.
  • Video recognition: Kinetics training uses synchronized AdamW on 128 GPUs for 200 epochs, with a batch size of 512 clips and repeated augmentation.The learning rate follows a half-period cosine decay with linear warm-up.
  • Video recognition: Video inputs use temporally sampled clips and shared-across-frames augmentation, including random flips, Mixup, CutMix, Random Erasing, and Rand Augment.Spatial crops are resized with Inception-style sampling before producing 224×224 crops.
  • AVA detection: AVA detection reconstructs spatiotemporal feature maps from MViT outputs and applies a video-adapted Faster R-CNN-style detector with 3D RoIs.RoIAlign is applied frame-wise, followed by temporal global average pooling.
  • Additional datasets: Charades and Something-Something V2 are fine-tuned from Kinetics models with task-specific sigmoid or sampling procedures, while ImageNet experiments report top-1 and top-5 accuracy.Something-Something V2 disables random flipping to preserve directional information; ImageNet training follows the DeiT recipe.
Loading 2104.11227v1…