PaperlayerView exact revision on arXiv

Source-linked AI summary

UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, Yu Qiao

arXiv:2211.09552v1cs.CV

TL;DR

Video transformers must capture long-term dependencies while handling local redundancy and computational cost, but UniFormer requires costly image pretraining. UniFormerV2 equips pretrained image ViTs with new local and global relation aggregators, achieving state-of-the-art performance on eight video benchmarks and 90% top-1 accuracy on Kinetics-400.

  • Problem

    Video ViTs capture global dependencies but struggle with local video redundancy, while UniFormer requires a costly image-pretraining phase before video finetuning.

  • Method

    UniFormerV2 arms image-pretrained ViTs with local temporal and query-based global relation aggregators, organized through multi-stage fusion.

  • Results

    UniFormerV2 achieves state-of-the-art performance on 8 popular video benchmarks and reaches 90% top-1 accuracy on Kinetics-400.

  • Takeaways & Limitations

    The paradigm integrates pretrained ViT representations with UniFormer video designs for effective spatiotemporal modeling at tractable complexity.

Abstract

from arXiv · show

Learning discriminative spatiotemporal representation is the key problem of video understanding. Recently, Vision Transformers (ViTs) have shown their power in learning long-term video dependency with self-attention. Unfortunately, they exhibit limitations in tackling local video redundancy, due to the blind global comparison among tokens. UniFormer has successfully alleviated this issue, by unifying convolution and self-attention as a relation aggregator in the transformer format. However, this model has to require a tiresome and complicated image-pretraining phrase, before being finetuned on videos. This blocks its wide usage in practice. On the contrary, open-sourced ViTs are readily available and well-pretrained with rich image supervision. Based on these observations, we propose a generic paradigm to build a powerful family of video networks, by arming the pretrained ViTs with efficient UniFormer designs. We call this family UniFormerV2, since it inherits the concise style of the UniFormer block. But it contains brand-new local and global relation aggregators, which allow for preferable accuracy-computation balance by seamlessly integrating advantages from both ViTs and UniFormer. Without any bells and whistles, our UniFormerV2 gets the state-of-the-art recognition performance on 8 popular video benchmarks, including scene-related Kinetics-400/600/700 and Moments in Time, temporal-related Something-Something V1/V2, untrimmed ActivityNet and HACS. In particular, it is the first model to achieve 90% top-1 accuracy on Kinetics-400, to our best knowledge. Code will be available at https://github.com/OpenGVLab/UniFormerV2.

1 INTRODUCTION

Video ViTs capture global dependencies but struggle with local redundancy and efficiency, while UniFormer addresses this at the cost of image pretraining. UniFormerV2 equips pretrained image ViTs with new local and global relation aggregators and achieves strong performance across eight video benchmarks.

  • Video ViTs focus on global dependencies but inadequately address local redundancy, imposing computational burdens in shallow layers.
  • UniFormer unifies convolution and self-attention as a transformer-style Multi-Head Relation Aggregator.
  • UniFormer reduces computation by modeling local and global relations in shallow and deep layers, respectively, but requires costly image pretraining from scratch.
  • UniFormerV2 arms image-pretrained ViTs with new local and global UniBlocks for efficient local and global spatiotemporal modeling.The local block inserts temporal MHRA before the spatial ViT block, while the global block uses query-based cross MHRA to summarize tokens into a video token.
  • UniFormerV2 achieves state-of-the-art recognition performance on 8 popular video benchmarks spanning scene-related, temporal-related, and untrimmed datasets.

2 RELATED WORK

Video modeling has progressed from local 3D convolutions toward global attention and more efficient hierarchical or unified transformers. However, several efficient video transformers still depend on tiresome image pretraining, motivating comparison with readily available pretrained ViTs.

  • 3D convolutions are limited by local receptive fields when capturing long-range video dependencies.
  • Video transformers extend image-based ViTs with temporal attention to model long-range dependencies.
  • UniFormerV1 requires costly image pretraining, whereas direct temporal MHSA insertion into ViTs struggles with the accuracy-FLOPs balance.
  • Hierarchical, local, and unified attention designs improve video-transformer efficiency but rely on tiresome image pretraining.

3 METHOD

UniFormerV2 arms image-pretrained ViTs with efficient UniFormer video designs, combining local temporal modeling, global spatiotemporal aggregation, and multi-stage fusion. Its relation aggregators preserve pretrained spatial representations while reducing temporal redundancy and global-attention complexity.

  • Overall Framework: UniFormerV2 projects videos into spatiotemporal tokens, then applies local and global UniBlocks followed by multi-stage fusion.The architecture uses 3D convolution for tokenization and adaptively integrates global tokens from different stages.
  • Local UniBlock: The local UniBlock inserts local temporal MHRA before the pretrained ViT block to model temporal relations while preserving spatial representation.LT MHRA uses local affinity within a temporal tube, while the ViT block supplies global spatial self-attention.
  • Local UniBlock: Local UniBlock affinity decomposes spatiotemporal modeling into local temporal and global spatial relations, inheriting UniFormer efficiency and ViT image pretraining.The design avoids the original UniFormer’s jointly spatiotemporal local affinity, whose parameters must learn from scratch.
  • Global UniBlock: The global UniBlock uses DPE, cross-attention-style MHRA, and FFN to capture long-range spatiotemporal dependency through a learnable video token.Cross MHRA aggregates context from all spatiotemporal tokens and reduces complexity from O(L^2) to O(L).
  • Experiments: UniFormerV2 is evaluated against state-of-the-art methods across Kinetics-400/600/700, Moments in Time, and Something-Something benchmarks.The supplied comparison captions identify accuracy and/or efficiency comparisons, including 90.0% top-1 accuracy on Kinetics-400.
  • Multi-Stage Fusion Block: The multi-stage fusion block integrates video tokens from global blocks, with sequential fusion selected for efficacy and efficiency among four studied strategies.The final representation can also combine the final global video token with the local class token through a learnable weighted sum.

4 EXPERIMENTS

Experiments evaluate UniFormerV2 across eight video benchmarks, pretrained ViT variants, component ablations, and training recipes. Results show strong accuracy, efficient temporal modeling, and reduced training cost.

  • UniFormerV2 is evaluated on eight benchmarks spanning scene-related, temporal-related, and untrimmed video recognition.
  • 4.2% performance gain is achieved over ViViT-L on Moments in Time with 19% of its model parameters and 15% of its FLOPs.
  • 62.7% accuracy establishes new state-of-the-art performance on Something-Something V1.
  • 4.5% and 3.6% improvements over previous best results are reported on ActivityNet and HACS, respectively.
  • All tested pretrained ViT variants outperform TimeSformer, especially on Something-Something, demonstrating generality across supervision types.
  • Global UniBlocks benefit scene-related benchmarks, whereas local UniBlocks are critical for temporal-related benchmarks such as SSV2.
  • Kinetics-710 post-pretraining saves about 33% training cost while consistently improving classification performance over co-training.

5 CONCLUSION

The paper concludes that UniFormerV2 equips image-pretrained ViTs with efficient UniFormer designs for video learning. Its local and global aggregators, together with multi-scale token fusion, deliver state-of-the-art results across eight benchmarks.

  • UniFormerV2 combines image-pretrained ViTs with local and global UniFormer relation aggregators for tractable spatiotemporal modeling.
  • State-of-the-art performance is achieved on eight popular video benchmarks, including 90% top-1 accuracy on Kinetics-400.
  • Multi-scale token fusion further enhances the video representation.

A ADDITIONAL IMPLEMENTATION DETAILS

Additional implementation details describe datasets, sampling, architecture choices, and training-cost comparisons. Kinetics-710 reduces redundant training data and enables substantial cost savings.

  • Moments in Time contains 0.8M three-second clips across 339 classes, emphasizing dynamic-scene understanding.
  • Figure 4 samples Kinetics frames using different strategies; UniFormerV1 doubles frames and downsamples temporal resolution during patch embedding.
  • Scene-related datasets use only global UniBlocks in the last four ViT-B/L layers, while Something-Something uses broader temporal modeling designs.
  • Kinetics-710 reduces combined Kinetics training videos from 1.14M to 0.66M by removing duplicates and leaked test videos.
  • Training with Kinetics-710 saves almost 33% of training cost and permits eight-frame finetuning for models using 16, 32, or 64 frames.

B VISUALIZATIONS

Visualizations compare where TimeSformer, UniFormerV1, and UniFormerV2 focus across network depth. UniFormerV2 preserves local details while also modeling broader video content in deep layers.

  • CAM visualizations compare discriminative feature locations for TimeSformer, UniFormerV1, and UniFormerV2 across their network stages or layers.
  • UniFormerV1 and UniFormerV2 capture local details, but UniFormerV1 may lose deeper-layer information as resolution shrinks.
  • TimeSformer learns local features mainly in shallow layers and struggles to focus on meaningful areas.
  • UniFormerV2 maintains local details in deep layers while observing the whole video for broader representation.

C MORE ABLATION STUDIES

The ablations examine output-token combinations, additional Kinetics pretraining, query counts, and temporal modules. They identify design choices affecting accuracy, overfitting, and temporal modeling effectiveness.

  • Output token combination: 84.4% top-1 accuracy falls to 81.8% when only the global token is used for classification.Both local and global output tokens are essential for maintaining performance.
  • Kinetics pretraining for Something-Something: Extra Kinetics pretraining harms the representation inherited from CLIP, leading to lower performance.
  • Query number: Increasing the query number causes severe overfitting and reduces performance.
  • Different modules: Local MHRA achieves 2.0% to 22.6% higher top-1 accuracy than the compared temporal modules.The comparison includes mean pooling, space-time MHSA, temporal convolution, temporal shift, and temporal transformer modules.

D ADDITIONAL RESULTS

Additional results cover eight video benchmarks spanning Kinetics, Moments in Time, Something-Something, ActivityNet, and HACS.

  • More results are reported on eight video benchmarks: Kinetics-400/600/700, Moments in Time, Something-Something V1/V2, ActivityNet, and HACS.

E MORE DISCUSSIONS

The discussion contrasts UniFormerV2's local and global UniBlock designs with related temporal and cross-attention approaches. It also identifies dependence on pretraining-data scale as a limitation.

  • Local UniBlock vs. ST-Adapter: The Local UniBlock treats temporal depth-wise convolution as a local temporal relation aggregator and adds BatchNorm before the first linear projection.
  • Global UniBlock vs. Perceiver, DETR and Flamingo: The Global UniBlock replaces self-attention MHRA with cross-attention MHRA and combines DPE, cross MHRA, and FFN.
  • Limitations: Performance tends to depend on the scale of pretraining data, motivating evaluation with huge image foundation models pretrained on massive datasets.

F LABEL LIST OF KINETICS-710

The Kinetics-710 label appendix documents how labels from Kinetics-400/600/700 are aligned and represented for reproducible pretraining and finetuning.

  • Label construction: Kinetics-710 aligns labels by filtering symbols and replacing synonyms, with the final label list provided in Table 20.
  • Label coverage: Compared with Kinetics-700, Kinetics-400 and Kinetics-600 contribute 8 and 2 unique labels, respectively.
  • Finetuning labels: Finetuning Kinetics-710-pretrained models requires loading the classification-layer weights and mapping them according to the aligned label list.
Loading 2211.09552v1…