PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Transformers: A Survey

Javier Selva, Anders S. Johansen, Sergio Escalera, Kamal Nasrollahi, Thomas B. Moeslund, Albert Clapés

arXiv:2201.05991v3cs.CV

TL;DR

Video’s temporal dimension creates redundancy, high dimensionality, and motion-modeling challenges, while Transformers add quadratic complexity and lack inductive biases. The survey analyzes over 100 Video Transformer works across input processing, architecture, and training, then compares classification performance. It finds video-based pre-training and masked token modeling especially effective for video classification, while identifying practical limitations such as large negative-set requirements for contrastive learning.

  • Problem

    Video introduces high dimensionality, redundancy, and motion dynamics, while existing Transformer surveys do not deeply analyze video-specific designs.

  • Method

    The survey reviews over 100 Video Transformer works, organizes design choices across input, architecture, and training, and compares action-classification results.

  • Results

    Video-based pre-training outperforms image-based pre-training alone, while Masked Token Modeling achieves the strongest reported video-classification results without extra data or manual annotations.

  • Takeaways & Limitations

    Progressive hierarchical architectures, memory-based modeling, and masked token modeling are identified as prominent approaches for handling redundancy, long-range dynamics, and spatiotemporal learning.

  • Takeaways & Limitations

    Contrastive approaches require large negative sets, which can be difficult to maintain for full-video representations.

Abstract

from arXiv · show

Transformer models have shown great success handling long-range interactions, making them a promising tool for modeling video. However, they lack inductive biases and scale quadratically with input length. These limitations are further exacerbated when dealing with the high dimensionality introduced by the temporal dimension. While there are surveys analyzing the advances of Transformers for vision, none focus on an in-depth analysis of video-specific designs. In this survey, we analyze the main contributions and trends of works leveraging Transformers to model video. Specifically, we delve into how videos are handled at the input level first. Then, we study the architectural changes made to deal with video more efficiently, reduce redundancy, re-introduce useful inductive biases, and capture long-term temporal dynamics. In addition, we provide an overview of different training regimes and explore effective self-supervised learning strategies for video. Finally, we conduct a performance comparison on the most common benchmark for Video Transformers (i.e., action classification), finding them to outperform 3D ConvNets even with less computational complexity.

1 INTRODUCTION

Video combines visual content with temporal structure, creating high dimensionality, redundancy, and motion-modeling challenges. This survey addresses the need for a video-specific account of Transformer designs and trends.

  • Video combines continuous visual signals with sequence-like temporal structure, including motion and deformations.
  • The temporal dimension increases video dimensionality and redundancy while requiring models to capture motion dynamics.
  • Transformers model non-local interactions and process whole sequences in parallel, but lack inductive biases.
  • Existing surveys do not provide an in-depth account of video-specific Transformer designs despite video’s sequential and high-dimensional nature.
  • The survey reviews over 100 Video Transformer works, taxonomizes input, architecture, and training choices, and compares classification results on Kinetics 400 and Something-Something-v2.

2 THE TRANSFORMER

The original Transformer embeds token sequences, adds positional information, and uses attention-based encoder and decoder modules. Its flexibility comes with quadratic attention complexity and a lack of inductive biases, motivating video-specific adaptations.

  • Encoder module: The encoder stacks multi-head self-attention and position-wise feed-forward sub-layers to transform input representations.
  • Input preprocessing: Transformer inputs are projected from one-hot tokens into continuous embeddings, then augmented with positional encodings.
  • Self-attention: Self-attention maps token embeddings into queries, keys, and values before using pairwise similarities to contextualize each token.
  • Decoder module: The decoder uses masked self-attention, encoder-decoder cross-attention, and feed-forward layers to generate target-sequence probabilities.
  • Video adaptations: Video Transformers commonly use special tokens and encoder-only or decoder-only variants for fixed-size, sequence-level, or autoregressive outputs.
  • Transformer limitations: Self-attention has quadratic complexity O(N^2), while the absence of inductive biases can complicate learning and increase training costs.

3 INPUT PRE-PROCESSING

Video Transformer input pre-processing combines embedding, tokenization, and positional information to manage video’s dimensionality, redundancy, and spatiotemporal structure. Choices of embedding network and tokenization determine the granularity and temporal span of interactions available to the Transformer.

  • Overview: Video pre-processing comprises embedding, tokenization, and positioning, with embedding often preceding tokenization to create compact continuous representations.Embedding may directly produce tokens or representations that are further divided into more atomic units.
  • Embedding: Embedding strategies range from minimal linear layers to full CNN networks, which also reduce dimensionality and encode local inductive biases.Minimal embeddings follow tokenization-then-embedding, whereas large networks can embed full input sequences before tokenization.
  • Tokenization: Tokenization categories are defined by receptive field: patches model local spatial or spatiotemporal regions, while frame or clip tokens support longer temporal spans.3D patches include local motion within tokens; frame-wise tokenization improves long-video scalability but may limit fine-grained spatial interactions.
  • Positioning: Positional embeddings provide spatiotemporal order for permutation-invariant self-attention and may be fixed or learned, absolute or relative.Absolute encodings can represent temporal, 2D, or 3D positions; relative encodings add position-dependent biases within attention.
  • Discussion: Embedding and tokenization jointly determine the abstraction level and granularity at which spatial and temporal interactions are formed.CNN embeddings share information through local inductive biases, while 3D patches or clips can form fine-grained temporal interactions within tokens.
  • Discussion: Fixed absolute positional encodings use fewer parameters than learned counterparts, while relative encodings may better support translation equivariance and unseen lengths.Video’s inconsistent temporal lengths make length generalization particularly relevant, but rescaling temporal dimensions can harm fine-grained motion modeling.

4 ARCHITECTURE

The survey organizes Video Transformer architectures around reducing attention cost and preserving or extending temporal modeling. It contrasts restricted, aggregation-based, memory, recurrence, and multi-view strategies, emphasizing the trade-off between redundancy reduction, temporal resolution, and fine-grained motion.

  • Architectural overview: Video Transformer designs decompose full attention into smaller operations to reduce quadratic costs and introduce useful inductive biases.Restricted approaches limit each attention operation’s scope while preserving sequence length; aggregation approaches reduce sequence length.
  • Restricted approaches: Restricted approaches use local neighborhoods, axial token subsets, or sparse patterns, approximating global receptive fields by stacking smaller self-attention operations.Local attention uses neighborhoods or windows, axial attention follows spatial or temporal axes, and sparse attention reduces computation while retaining global access.
  • Aggregation approaches: Aggregation approaches compress tokens hierarchically or use query-driven condensation, but aggregation can discard fine-grained temporal cues.Hierarchical designs reduce sequence length for queries, keys, and values, whereas query-driven designs use a smaller query set to summarize the full input.
  • Aggregation approaches: Progressive hierarchies preserve spatiotemporal interactions across levels by aggregating local neighborhoods while increasing token dimensionality.This increases deeper-layer capacity, although the supplied passage truncates the discussion of its parameter cost.
  • Long-term temporal modeling: Memory and recurrence methods model long-term dynamics differently: recurrence compresses history into fixed-size states, whereas memory retains variable-size information for selective attention.Caching raw frames is prohibitive, so memory systems store features or embeddings and may aggregate or sparsify them during storage or access.
  • Discussion on architecture: Effective video architectures should reduce spatial redundancy while modeling temporal features at all levels in high-fidelity temporal contexts.Early removal of spatial or temporal information can hinder fine-grained motion modeling, so designs must balance efficiency with preservation of temporal detail.

5 TRAINING A TRANSFORMER

Video Transformer training must balance weak inductive biases and high computational demands against video’s dimensionality and redundancy. The survey reviews supervised, pre-training, contrastive, and masked-token strategies, highlighting both their benefits and unresolved limitations.

  • Pre-training: Self-supervised learning can reduce reliance on annotated data while providing more robust and general features, but remains less widespread for video than for language or images.The survey emphasizes large-scale unlabeled videos as a promising source for training Video Transformers.
  • Instance-based learning: Instance-based learning uses contrastive losses to align augmented views of a sequence while separating negative samples.Video augmentations are commonly applied consistently across time, but temporal views require care because arbitrary reversal or shuffling can remove temporal causality.
  • Instance-based learning: Contrastive approaches require large negative sets, motivating memory banks, hard-negative mining, or formulations that remove negatives altogether.Large negative sets can be difficult to maintain when full video representations limit batch size.
  • Masked Token Modeling: Masked Token Modeling adapts masked prediction to visual tokens, forcing contextualized representations even though video tokens lack a simple discrete vocabulary.Masking blocks consistently through time helps avoid shortcut learning, while quantized and feature-based targets appear preferable for video.
  • Training challenges: Video Transformer training balances limited inductive biases, computational budgets, large datasets, self-supervised learning, and the dimensionality of video inputs.Efficient designs, data augmentation, and self-supervised losses can help stand-alone Transformers learn without large CNN embeddings.
  • Training-strategy discussion: Instance-based contrastive learning can overlook fine-grained motion by becoming invariant to temporal changes, whereas MTM and carefully designed combinations may preserve broader temporal information.The survey identifies combined contrastive and MTM objectives as promising, while noting that their learned representations remain insufficiently understood.

6 PERFORMANCE ON VIDEO CLASSIFICATION

Video classification is the main evaluation task for Video Transformers, with performance compared across Kinetics-400 and Something-Something v2. The survey finds pre-training strategy is the largest performance factor, while masked-token modeling and video-based pre-training generally outperform image-only alternatives.

  • 6.1 Video classification: Video classification predicts the class of an input frame sequence from a high-level global representation and is commonly evaluated on Kinetics-400 and Something-Something v2.Kinetics-400 is appearance-biased, whereas Something-Something v2 better assesses complex temporal dynamics.
  • 6.3 Discussion on performance: The survey identifies pre-training strategy as the biggest factor influencing Video Transformer classification performance.The comparison considers training from scratch, image-based pre-training, supervised video pre-training, and self-supervised video pre-training.
  • 6.3 Discussion on performance: Small Video Transformers can train from scratch, with MViT and MViTv2 competing with or surpassing 3D ConvNets trained from scratch.MViTv2 also outperforms models pre-trained on very large weakly annotated video datasets; the authors attribute this success to infused locality bias.
  • 6.3 Discussion on performance: Video-based supervised or self-supervised pre-training is superior to image-based pre-training alone, while self-supervised video pre-training can surpass supervised video regimes.Image-based pre-training remains useful as initialization for subsequent video pre-training or temporal self-supervised fine-tuning.
  • 6.3 Discussion on performance: Masked Token Modeling stands out for spatiotemporal modeling: MaskFeat obtains the best Kinetics-400 results, while VideoMAE dominates Something-Something v2.Both methods can self-supervisedly pre-train on the evaluation dataset without extra data or manual annotations.
  • 6.3 Discussion on performance: Complex spatiotemporal dynamics benefit more from deeper models and temporal granularity than from extending temporal spans.The survey also reports that naive image- and NLP-model adaptations may underperform, while joint image-video self-supervised learning remains promising but immature.

7 FINAL DISCUSSION

The survey identifies computational complexity, redundancy, and inductive bias as central concerns in Video Transformers, while highlighting architectural, self-supervised, and generalization trends. It also notes unresolved needs for motion-sensitive representations, long-range temporal modeling, and explainability.

  • Complexity: Computational burden is addressed across the Video Transformer pipeline, often through frozen embedding networks that provide inductive biases and reduce input dimensionality.These embeddings let Transformers enhance representations through long-range interactions, but may limit learning of non-local low-level motion cues.
  • Architectural trends: Progressive hierarchical architectures reduce redundancy by modeling non-local temporal contexts before spatial aggregation, avoiding early loss of fine-grained motion features.Memory-based approaches with suitable sampling or aggregation may still be important for long-range interactions without sacrificing temporal fidelity.
  • Self-supervised learning: Masked token modeling uses high masking ratios and local token-wise prediction to encourage learning from global spatiotemporal semantic contexts, including motion and appearance cues.The survey also identifies instance-based contrastive sampling as a route toward less appearance-biased, more motion-specific features.
  • Generalization: Video Transformers have limited evidence on out-of-distribution generalization, while robustness to varied frame sampling rates remains an open research question.The survey hypothesizes that frame-rate robustness may require additional training or conditioning on sampling rates.
  • Generalization: Unseen sequence lengths create positional-encoding challenges, although fine-tuning, input-conditioned relative position encodings, and learned log-scale relative biases show promising extensions.These methods are intended to support longer sequences while preserving temporal fidelity.
  • Limitations: Video Transformers lack sufficiently developed explainability tools for assessing the spatiotemporal representations produced by different designs and self-supervised losses.Attention heat maps offer ad-hoc explanations but can require inspecting per-sample activations across multiple full videos.

8 SUPPLEMENTARY

The supplementary material provides a general overview of surveyed Video Transformers and application-oriented details on Transformer trends across video tasks.

  • Supplementary contents: The supplement contains a general table of relevant Video Transformers and details about task-specific Transformer trends.The general overview is in Section 8.1, while application-oriented trends are discussed in Section 8.2.

8.1 General table

The general table summarizes the most relevant Video Transformers surveyed, with its length requiring division into two subtables.

  • General table: The general overview of relevant Video Transformers is split between Tables 3 and 4 because of its length.Table 4 is presented as the continuation of Table 3.

8.2 Task-specific designs

The task-specific survey organizes Video Transformer designs across translation, classification, retrieval, object-centric, low-level, segmentation, summarization, and other tasks. Across these applications, architectures vary in tokenization, backbones, attention restrictions, aggregation, and temporal modeling.

  • Task coverage: Task-specific designs cover action classification, video translation, retrieval, object-centric tasks, low-level processing, segmentation, summarization, and other applications.The first four receive major subsections, followed by shorter summaries of the remaining tasks.
  • Action classification: Pure Transformer classification works largely focus on efficiency through space-time decomposition, tokenization choices, or restricted attention.One reported comparison found a pre-trained ViT encoding 2D patches with a temporal encoder on top performed best.
  • Action classification: Larger CNN-based backbones facilitate training on smaller datasets, while shallow Transformer encoders can function primarily as pooling operators.For detection backbones, Transformers can fuse information among detections or provide access to a larger visual context.
  • Video translation: Video translation commonly uses encoder-decoder architectures, with the video encoder supplying context to the decoder.Dense video captioning adds temporal proposal generation so the decoder can focus on selected locations in the sequence.
  • Video translation: Dense captioning methods differ in how proposals are used: they may re-encode clipped temporal segments, apply differentiable masks, or use local self-attention to avoid re-forwarding.These choices regulate how temporal proposals are connected to caption decoding and encoder computation.
  • Video translation: Translation decoders usually retain their canonical form, although some methods remove decoder self-attention, replace it with a moving average, or alter cross-attention inputs.These modifications are presented as ways to make models computationally lighter or adapt encoder information across depths.
  • Surveyed models: The tables include model-specific entries spanning classification, translation, retrieval, tracking, and other video tasks.They list designs such as TimeSformer, ViViT, MViT, VideoBERT, HERO, and multiple detection and tracking Transformers.
  • Taxonomy: The survey tables classify models by architecture, aggregation, attention restriction, long-term temporal modeling, backbone, embedding, tokenization, positional embedding, and self-supervised learning.Architectures include encoders, decoders, and encoder-decoder systems; aggregation includes hierarchical and query-driven compression.

8.2.3 Video retrieval

Video retrieval methods recover associated information across videos and modalities by learning representations whose similarities distinguish true pairs from negatives. Variants combine contrastive alignment with re-ranking or modality-aware weighting.

  • Video retrieval recovers associated information from video-video or cross-modal pairs, commonly involving language and sometimes audio.
  • Hinge-based and bidirectional max-margin losses enforce higher similarity for true pairs than negative pairs by a specified margin.
  • InfoNCE normalizes positive-pair similarity against negative samples to learn similar representations for matched samples and dissimilar representations for mismatches.
  • Some systems align siamese video-and-language CNN outputs for faster retrieval, then use a decoder-only Transformer to re-rank top-k results by decoding likelihood.
  • Multimodal similarities can be combined with caption-dependent weights before contrastive alignment, emphasizing the modality most salient to the caption.

8.2.4 Object-centric tasks: tracking and object detection

Object-centric video tasks exploit temporally coherent object representations to reduce redundant frame information and aggregate relevant context. Video Transformers implement this through messenger tokens, token shifting, bounding-box guidance, and object-specific branches.

  • Object-centric tracking and detection approaches use temporally coherent object representations because much frame information is redundant.
  • Messenger tokens relay contextual information between frames, either through an isolated encoder or by shifting tokens across frame sequences.
  • TeViT shifts tokens between frame sequences to aggregate object-specific information sequentially across multiple temporal steps.
  • ORViT injects auxiliary bounding-box information into each Transformer layer, while GroupFormer isolates objects for a separate object-specific action-classification branch.

8.2.5 Low-level tasks

Video generation remains challenging because of video’s high dimensionality, and relatively few Video Transformers address it. Most methods embed Transformers in convolutional auto-encoders, while one autoregressive approach uses local attention.

  • Video generation is challenging because of high-dimensional video data, and relatively few Video Transformers address generation tasks.
  • Existing methods target future-frame prediction, event-to-grayscale generation, or video inpainting.
  • Most generation methods embed a Transformer within a convolutional auto-encoder to evolve representations between encoder and decoder.
  • One exception uses local attention and autoregressively generates video one pixel channel at a time.
  • The inpainting method using overlapping patch tokenization outperforms the compared method across all tested inpainting benchmarks.

8.2.6 Segmentation

Video segmentation methods commonly use temporal relations to refine intermediate representations. A notable approach adds temporal matching of individual representations to encourage implicit object tracking and fine-grained temporal modeling.

  • Most video segmentation methods leverage temporal relations to refine intermediate feature representations.
  • One approach adds an auxiliary loss that temporally matches individual representations across the sequence.
  • This temporal matching trains the network to implicitly track objects while using fine-grained temporal information.

8.2.7 Summarization

Video Transformers have been used for video summarization, with early approaches combining recurrent processing or attention over aggregated video subsets. Other applications are listed, but the field is too sparse to establish specific trends.

  • Few works apply Transformers to video summarization by predicting frame-wise importance scores.
  • Two trends are using RNNs initially and having individual frames attend to aggregated video subsets.The subsets are aggregated through either a GRU or a masked Transformer.
  • Video Transformers have also been applied to tasks including action anticipation, sign-language translation, visual-question answering, and autonomous driving.Additional applications include navigation, personality recognition, lip reading, dynamic scene graph generation, and multimedia recommendation.
Loading 2201.05991v3…