Source-linked AI summary
Video Transformer Network
Daniel Neimark, Omri Bar, Maya Zohar, Dotan Asselmann
TL;DR
Video recognition commonly uses 3D ConvNets and often processes long videos as short clips, limiting whole-sequence analysis. VTN uses 2D spatial features followed by attention over frame sequences, achieving competitive accuracy with faster training and inference while supporting full-video processing.
Problem
Video recognition commonly relies on 3D ConvNets, while long videos are often processed as short clips because long sequences are difficult to handle.
Method
VTN uses a modular 2D spatial backbone and adds temporal information later through attention mechanisms over the resulting frame features.
Results
VTN achieves competitive accuracy while training 16.1× faster, supports full-video analysis at inference, and uses fewer computational resources.
Takeaways & Limitations
VTN provides a modular transformer-based framework for efficient video evaluation that can process entire videos during testing.
Takeaways & Limitations
Self-attention has O(n^2) complexity per layer, constraining Transformers’ ability to process long sequences.
Abstract
from arXiv · showhide
This paper presents VTN, a transformer-based framework for video recognition. Inspired by recent developments in vision transformers, we ditch the standard approach in video action recognition that relies on 3D ConvNets and introduce a method that classifies actions by attending to the entire video sequence information. Our approach is generic and builds on top of any given 2D spatial network. In terms of wall runtime, it trains $16.1\times$ faster and runs $5.1\times$ faster during inference while maintaining competitive accuracy compared to other state-of-the-art methods. It enables whole video analysis, via a single end-to-end pass, while requiring $1.5\times$ fewer GFLOPs. We report competitive results on Kinetics-400 and present an ablation study of VTN properties and the trade-off between accuracy and inference speed. We hope our approach will serve as a new baseline and start a fresh line of research in the video recognition domain. Code and models are available at: https://github.com/bomri/SlowFast/blob/master/projects/vtn/README.md
1. Introduction
VTN replaces clip-oriented 3D ConvNet processing with 2D spatial features followed by temporal attention over video sequences. It targets full-video context while improving the speed–accuracy trade-off and retaining modularity.
- Motivation and approach: VTN moves temporal modeling after spatial feature extraction, using attention mechanisms on features from state-of-the-art 2D architectures instead of 3D networks.The method uses RGB frames without optical flow or other additional inference components while achieving comparable results to state-of-the-art models.
- Motivation and approach: Short-clip processing limits access to long-term information, even though Kinetics-400 videos are about 10 seconds and longer real-world videos may span minutes or hours.The paper argues that full-video context can help avoid false predictions caused by clips in which the relevant action is obscured.
- Results and contributions: 16.1× faster training, 5.1× faster validation, and 1.5× fewer GFLOPs summarize VTN’s reported efficiency gains while maintaining similar accuracy.The comparison concerns end-to-end training wall runtime, validation wall runtime, and GFLOPs × Views, respectively.
- Results and contributions: VTN is modular: its 2D backbone, temporal Transformer encoder, and classification head can be replaced or adapted for other video-based tasks.The paper specifically mentions temporal action localization as a possible task supported by modifying the classification head.
2. Related Work
Prior video-recognition systems largely extend ConvNets into temporal architectures or process short clips, while Transformer research motivates sequence-based alternatives. Longformer supplies a more efficient mechanism for longer sequences, though Transformer processing remains constrained by sequence length.
- Spatial-temporal networks: Video recognition commonly uses 3D ConvNets, including two-stream, inflated 2D, SlowFast, and X3D variants that incorporate temporal information through network design.These approaches differ in input streams, frame rates, pathways, and expansion axes while remaining rooted in spatial-temporal convolutional architectures.
- Spatial-temporal networks: Earlier attention-based video methods model relationships within short clips or combine short-term features with a long-term feature bank.Non-local Networks are confined to short clips, whereas feature-bank methods maintain separate long-term video memory.
- Transformers in computer vision: Vision Transformers and related models show that Transformer architectures can perform strongly across computer-vision tasks, making video’s sequential frame structure a natural match.The related work connects Transformer success in NLP and vision with representing videos as sequences.
- Applying Transformers on long sequences: O(n^2) self-attention per layer limits Transformer processing of long sequences, where n is sequence length.Longformer addresses this constraint with sliding-window local attention and task-specific global attention, yielding O(n) complexity and sequence-wide integration.
3. Video Transformer Network
VTN combines a flexible 2D spatial backbone with a temporal attention encoder and classification head, enabling full-video inference or memory-aware alternatives. Its design addresses the limitations of short-clip, multi-view processing.
- Framework: VTN is a single-stream, generic video-recognition framework that classifies an input video into an action category.The architecture is intended to support video recognition tasks beyond the demonstrated action-classification setting.
- Framework: The modular pipeline contains a 2D spatial feature extractor, a temporal attention-based encoder, and a classification MLP head.The spatial backbone may be convolutional- or Transformer-based, pretrained or learned during training.
- Inference: VTN supports end-to-end full-video inference, chunked feature extraction, or precomputed features followed by temporal encoding when memory is limited.These alternatives trade implementation and memory requirements while preserving the same broad temporal-encoding design.
- Temporal encoder: Longformer receives the sequence of spatial feature vectors as tokens and uses sliding-window attention to enable linear-complexity temporal processing.A [CLS] token retains global attention and its final state becomes the video representation supplied to the classification head.
- Inference: Short-clip multi-view inference can miss relevant action segments, whereas full-video processing lets the model attend to the complete context; existing short-clip-trained methods may degrade on full videos.The paper identifies hardware, memory, and runtime constraints as practical limitations of full-video inference for prior methods.
4. Video Action Recognition with VTN
VTN evaluates multiple 2D spatial backbones within a transformer-based video framework, using Longformer temporal processing and configurable positional embeddings. The experiments compare learned and uniform attention, multi-view and full-video inference, and alternative training configurations.
- Spatial backbones: VTN is evaluated with ViT-Base, ResNet-50/101, and DeiT-Tiny/Base/Base-Distilled spatial backbones.ViT-Base uses ImageNet-21K pretraining, while the ResNet and DeiT variants use ImageNet-pretrained models.
- Experimental setup: The ablations use SGD with an initial learning rate of 10^-3, while wall-runtime measurements use an 8-V100-GPU machine.The ablation experiments were conducted on a 4-GPU machine with batch sizes determined by the backbone family.
- Inference and attention: The experiments compare learned attention with fixed uniform attention and evaluate both multi-view and full-video inference.Multi-view inference averages softmax scores from 10 clips and three crops per clip, whereas full-video inference uniformly aligns all frames to 250.
- Training: 25 frames from all clips are stacked into one batch so the 2D backbone processes them in a single forward-backward pass.The clip tensor is reshaped from B×C×T×H×W to (B·T)×C×H×W before propagation.
- Temporal processing: The temporal component uses Longformer attention, enabling global attention across long frame sequences.The Longformer configuration uses windowed attention layers plus global attention for selected tokens such as [CLS].
- Ablations: The study varies temporal positional information using learned embeddings, fixed absolute encoding, or no positional embedding.Learned embeddings encode a frame’s location in the original video sequence.
5. Experiments
Experiments on Kinetics-400 examine VTN’s backbone, temporal attention, positional embeddings, convergence, augmentation, and inference cost. VTN variants achieve competitive accuracy while converging faster and preserving performance under full-video inference.
- Spatial backbone variations: ViT-B-VTN reaches 78.6% top-1 and 93.7% top-5 accuracy, with pretraining choices affecting performance.Changing from a DeiT ImageNet-pretrained backbone to a ViT ImageNet-21K-pretrained backbone improves results.
- Longformer depth: Deeper Longformer attention produces only small accuracy differences on roughly 10-second Kinetics-400 videos.The authors suggest longer videos may benefit more from the larger receptive field of deeper Longformers.
- Longformer positional embedding: 78.9% top-1 accuracy is reached after randomly shuffling validation frames with no positional embedding, suggesting Kinetics-400 may emphasize appearance over motion.The authors also report that the no-positional-embedding version slightly outperforms fixed and learned alternatives.
- Backbone training: Fine-tuning the backbone improves Kinetics-400 top-1 accuracy by 7%.The comparison uses ViT-B-VTN with three attention layers, with and without backbone training.
- Runtime and convergence: 16.1× faster end-to-end training and 5.1× faster validation runtime are achieved while ViT-B-VTN retains nearly the same results as SlowFast.The faster validation is attributed to full-video inference, while faster training reflects earlier convergence and fewer required epochs.
- Data augmentation: 79.8% top-1 accuracy is obtained with extensive augmentation, a 1.2% improvement over the same model without augmentation.Augmentation adds 10 training epochs but does not affect training wall runtime.
- Inference complexity: ViT-B-VTN maintains performance between multi-view and full-video inference while requiring fewer inference GFLOPs end-to-end.By contrast, SlowFast-8X8-R50 loses about 8% when evaluated with the full-video approach.
- Moments in Time: On MiT-v2, ViT-B-VTN achieves the highest top-1 accuracy while using only RGB frames.The authors also train NL I3D on MiT-v2 and obtain results comparable to published I3D RGB+OF results on MiT-v1.
6. Conclusion
VTN is a modular transformer-based framework for video recognition that improves video evaluation efficiency in computational resources and wall runtime. It supports full-video processing during testing and may become more advantageous on future long-term video benchmarks.
- VTN is a modular transformer-based framework for video recognition tasks.
- VTN provides an efficient way to evaluate videos at scale in computational resources and wall runtime.
- Full-video processing during test time makes VTN more suitable for dealing with long videos.
A. Additional Qualitative Results
Figure 6 presents qualitative attention examples for VTN, covering both successful predictions and failure modes. The examples visualize first-layer [CLS] token weights to illustrate why attention matters.
- Figure 6 provides additional qualitative examples of why attention matters in VTN.
- The examples visualize [CLS] token weights from the first attention layer.
- The figure includes successful predictions and failure modes of the approach.