Source-linked AI summary
Multiview Transformers for Video Recognition
Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, Cordelia Schmid
TL;DR
Video transformers lacked explicit modeling of different spatiotemporal resolutions, despite video understanding requiring both fine-grained and long-duration reasoning. MTV addresses this gap with separate encoders for multiple video views and lateral cross-view fusion, achieving strong accuracy/computation trade-offs and state-of-the-art results across six datasets.
Problem
Transformer architectures had not explicitly modeled the multiple spatiotemporal resolutions needed for video understanding.
Method
MTV processes multiple tubelet-based video views with separate transformer encoders and lateral connections that fuse information across views.
Results
MTV consistently achieves superior accuracy/computation trade-offs across model sizes and state-of-the-art results on six video classification datasets, further improved by large-scale pretraining.
Takeaways & Limitations
Processing multiple views in parallel improves accuracy/computation trade-offs more effectively than increasing single-view transformer depth.
Takeaways & Limitations
The authors report substantial room for improvement on datasets other than Kinetics.
Abstract
from arXiv · showhide
Video understanding requires reasoning at multiple spatiotemporal resolutions -- from short fine-grained motions to events taking place over longer durations. Although transformer architectures have recently advanced the state-of-the-art, they have not explicitly modelled different spatiotemporal resolutions. To this end, we present Multiview Transformers for Video Recognition (MTV). Our model consists of separate encoders to represent different views of the input video with lateral connections to fuse information across views. We present thorough ablation studies of our model and show that MTV consistently performs better than single-view counterparts in terms of accuracy and computational cost across a range of model sizes. Furthermore, we achieve state-of-the-art results on six standard datasets, and improve even further with large-scale pretraining. Code and checkpoints are available at: https://github.com/google-research/scenic/tree/main/scenic/projects/mtv.
1. Introduction
MTV addresses multiresolution video understanding by processing multiple temporal views with separate encoders and lateral fusion. The design improves accuracy/computation trade-offs across model sizes and achieves state-of-the-art results on six datasets.
- Video recognition benefits from analyzing signals at multiple resolutions, including distinct temporal rates for modeling long-range and fine-grained information.
- Pyramidal pooling or subsampling can discard spatio-temporal information, motivating MTV’s use of multiple video views without input subsampling.
- MTV tokenizes the input over multiple temporal durations, with longer intervals capturing scene gist and shorter segments capturing fine-grained motion.
- Separate transformer encoders process each view, while lateral connections fuse information across views and a final global encoder aggregates the representations.
- Smaller encoders represent broader views while larger encoders capture details, producing better accuracy/computation trade-offs than pyramid-based designs.
- MTV generalizes to variable numbers of views, improves accuracy/computation trade-offs across Small-to-Huge models, and reaches state-of-the-art results on six datasets.
2. Related Work
Related video models evolved from handcrafted features and CNNs toward pure transformers, while multiscale processing remained a central strategy. MTV replaces conventional pyramidal processing with direct multiview encoding and cross-view fusion.
- Video understanding progressed from handcrafted motion and appearance features to CNNs, RNNs, and pure transformer architectures.
- Pyramid structures are a longstanding multiscale representation that progressively reduce spatial dimensions while increasing network depth.
- MTV directly processes multiple video views with cross-view encoders and consistently outperforms single-view counterparts in accuracy/FLOP trade-offs, including at billion-parameter scale.
3. Multiview Transformers for Video
MTV constructs multiple tokenized views of a video, processes each through a separate transformer stream, and fuses adjacent views efficiently before global aggregation. Its fusion methods avoid joint attention over all view tokens, whose quadratic cost is prohibitive for video.
- 3.1. Preliminaries: ViT and ViViT: The model forms each view by extracting non-overlapping spatio-temporal tubelets, linearly projecting them into tokens, and adding a class token with positional embeddings.
- 3.3.3 Global encoder: The global encoder processes classification tokens from each view to aggregate their information and maps the resulting token to the class outputs.
- 3.2. Multiview tokenization: A view is a video representation defined by fixed-sized tubelets; larger tubelets produce fewer tokens, while smaller tubelets produce more tokens and capture finer motions.
- 3.3. Multiview encoder: Separate transformer encoders process tokens from each view, using lateral connections to fuse cross-view information while restricting self-attention to tokens from the same temporal index.
- 3.3.2 Cross-view fusion: Joint self-attention over all view tokens is computationally prohibitive because self-attention has quadratic complexity.
- 3.3.2 Cross-view fusion: Cross-view attention updates larger-view tokens using queries from that view and keys and values from the adjacent smaller view, projecting hidden dimensions when necessary.
- 3.3.2 Cross-view fusion: Bottleneck fusion transfers information through a small learned token set, projecting bottleneck tokens between adjacent views to reduce communication cost.
- 3.3.2 Cross-view fusion: MLP fusion concatenates adjacent-view tokens along the hidden dimension before the MLP block, then projects the result back to the target depth.
4. Experiments
Ablations show that MTV benefits from assigning larger encoders to finer views, using CVA for cross-view fusion, and adding views rather than depth. Across model scales and datasets, MTV improves accuracy/computation trade-offs and achieves state-of-the-art results, especially with large-scale pretraining.
- Ablation study: 81.8% versus 78.5% shows that assigning larger encoders to smaller views outperforms assigning larger encoders to larger views.The authors attribute this to broader views capturing scene gist and smaller views capturing details that require greater capacity.
- Ablation study: CVA is the best cross-view fusion method, while late fusion improves the B/4 baseline from 78.3% to 80.6%.Ensemble, logit summation, majority voting, and MLP fusion perform worse than CVA.
- Ablation study: +2.5% Top-1 accuracy with two views and 2.8% with three views exceed the gains from adding layers at similar FLOPs.The deeper 14-layer and 17-layer Base variants remain similar to the single-view baseline.
- Ablation study: Fusion at the 5th and 11th layers performs best, while combining mid- and late-stage fusion outperforms adding more fusion layers within one stage.The selected fusion layers for larger variants are {11, 23} and {11, 23, 31}.
- Accuracy/computation trade-offs: MTV consistently achieves higher accuracy with fewer FLOPs than ViViT-FE, and its B/2 and L/2 models outperform deeper ViViT-L/2 and ViViT-H/2 models.The same efficiency trends hold when inference time is used and with an unfactorized backbone.
- Comparison to the state of the art: 89.9%, 90.3%, and 83.4% Top-1 accuracy on Kinetics-400, -600, and -700 improve previous highest results by 2.7%, 2.4%, and 3.6%, respectively.With WTS pretraining, MTV also reaches 50.5% action accuracy on Epic-Kitchens-100, improves Something-Something V2 over ViViT-L-FE by 2.6%, and achieves 47.2% on Moments in Time.
5. Conclusion
MTV captures multi-resolution temporal context by processing multiple video views in parallel, outperforming deeper single-view architectures on accuracy/computation trade-offs and achieving state-of-the-art results across six datasets. The authors note remaining limitations involving non-Kinetics performance, reliance on supervised pretraining, and possible dataset bias.
- MTV processes multiple video views in parallel to capture multi-resolution temporal context.
- MTV achieves better accuracy/computation trade-offs than increasing the depth of single-view architectures.
- MTV achieves state-of-the-art results on six popular video classification datasets, with further improvements from large-scale pretraining.
- Performance remains limited on datasets other than Kinetics, leaving substantial room for improvement.
- The approach relies on models pretrained on large image or video datasets, motivating research to reduce dependence on supervised pretraining.
- Training datasets may contain biases that make video classification models unsuitable for certain applications.
A. Additional experiments
The appendix provides additional experimental details covering efficiency comparisons, tubelet spatial resolution, training hyperparameters, and model configurations.
- The appendix adds accuracy-FLOPs and accuracy-throughput comparisons for ViViT and MTV variants.
- It evaluates the effect of tubelet spatial resolution on model performance.
- It documents training hyperparameters and model configurations used in the experiments.
A.1. Changing transformer encoder architecture
Additional experiments replace the factorized encoder with an unfactorized ViViT transformer, and MTV continues to outperform its single-view counterpart across model scales and efficiency measures.
- MTV with an unfactorized encoder consistently outperforms single-view unfactorized ViViT at every model scale.
- MTV unfactorized produces a better accuracy-throughput curve than single-view ViViT.
- MTV’s accuracy-throughput and accuracy-FLOPs improvements remain significant against the factorized ViViT-FE baseline.
- The unfactorized encoder is less efficient than the factorized architecture, but MTV yields larger relative improvements over its corresponding single-view baseline.
A.2. Spatial resolution of tubelets
Experiments with B/4+Ti/16 on Kinetics 400 examine how tubelet spatial resolution affects accuracy and computation.
- Tubelet spatial resolution primarily affects the accuracy of the large encoder.
- Processing more tokens and using more computation typically results in higher accuracies.
- Table 3 reports accuracies for B/4+Ti/16 on Kinetics 400 using 4 × 3 crops.
A.3. Hyperparameters for each datasets
Experiments use a common optimization setup on Kinetics datasets, with dataset-specific adjustments for smaller Epic Kitchens and Something-Something v2 datasets.
- All Kinetics experiments use synchronous SGD with momentum, cosine learning-rate decay, linear warmup, and batch size 64.
- Larger batch sizes and additional regularization help training on the smaller Epic Kitchens and Something-Something v2 datasets.
- The training setup distinguishes Kinetics experiments from the smaller-dataset settings rather than applying one configuration uniformly.
A.4. Model configurations
MTV configurations use ViT-sized backbones for individual views and a shared global encoder, while Figure 4 compares model accuracy, complexity, and throughput across scales.
- Model configurations: Each MTV view backbone uses one of five ViT variants: Tiny, Small, Base, Large, or Huge.These settings follow the architectures defined in BERT and ViT.
- Model configurations: All MTV variants use the same global encoder based on the Base architecture, with eight attention heads instead of twelve.
- Accuracy and complexity comparisons: Figure 4 compares ViViT and MTV across model scales using accuracy versus FLOPs and accuracy versus throughput.MTV is shown in red and ViViT or ViViT-FE in blue; comparisons use a single 32 × 224 × 224 × 3 input video and 4 × 3 view testing.
- Training configurations: Table 4 identifies training hyperparameters across K400, K600, K700, MiT, Epic Kitchens, and Something-Something v2, with dashes marking unused regularization.