Source-linked AI summary
Long Movie Clip Classification with State-Space Video Models
Md Mohaiminul Islam, Gedas Bertasius
TL;DR
Long movie understanding requires long-range temporal reasoning, but short-clip models and self-attention-based extensions are costly for such sequences. ViS4mer combines a Transformer encoder with a multi-scale temporal S4 decoder, achieving strong LVU performance while improving efficiency and generalizing to procedural activity datasets.
Problem
Short-clip video recognition models are not equipped for long-range movie understanding, while extending them with more frames is impractical because of computational and GPU-memory costs.
Method
ViS4mer combines a standard Transformer encoder for short-range spatial features with a multi-scale temporal S4 decoder for long-range reasoning and progressively reduces tokens and channel dimensions.
Results
ViS4mer outperforms previous approaches on 6 of 9 LVU tasks, is 2.63× faster and uses 8× less GPU memory than its self-attention counterpart, and achieves competitive results on Breakfast and COIN.
Takeaways & Limitations
ViS4mer provides an efficient framework for long-range movie video classification that also generalizes to procedural activity classification.
Abstract
from arXiv · showhide
Most modern video recognition models are designed to operate on short video clips (e.g., 5-10s in length). Thus, it is challenging to apply such models to long movie understanding tasks, which typically require sophisticated long-range temporal reasoning. The recently introduced video transformers partially address this issue by using long-range temporal self-attention. However, due to the quadratic cost of self-attention, such models are often costly and impractical to use. Instead, we propose ViS4mer, an efficient long-range video model that combines the strengths of self-attention and the recently introduced structured state-space sequence (S4) layer. Our model uses a standard Transformer encoder for short-range spatiotemporal feature extraction, and a multi-scale temporal S4 decoder for subsequent long-range temporal reasoning. By progressively reducing the spatiotemporal feature resolution and channel dimension at each decoder layer, ViS4mer learns complex long-range spatiotemporal dependencies in a video. Furthermore, ViS4mer is $2.63\times$ faster and requires $8\times$ less GPU memory than the corresponding pure self-attention-based model. Additionally, ViS4mer achieves state-of-the-art results in $6$ out of $9$ long-form movie video classification tasks on the Long Video Understanding (LVU) benchmark. Furthermore, we show that our approach successfully generalizes to other domains, achieving competitive results on the Breakfast and the COIN procedural activity datasets. The code is publicly available at: https://github.com/md-mohaiminul/ViS4mer.
1 Introduction
Long movie understanding requires temporal cues beyond the short clips used by most video models, but extending them is computationally and memory intensive. ViS4mer combines short-range self-attention with efficient long-range S4 reasoning and performs strongly across movie understanding tasks.
- Motivation: Long movie questions such as character relationships and genre require information from extended video context rather than only short clips.The motivating examples contrast several-second clips with viewing an entire movie.
- Motivation: Most modern video recognition models target short-range videos, typically 5–10 seconds long.
- Motivation: Extending short-range models by adding more frames is impractical because of excessive computational cost and GPU memory consumption.
- Motivation: Long-range Transformers face quadratic self-attention costs, while CNN-feature-based alternatives discard fine-grained spatiotemporal information.
- Approach: ViS4mer combines a Transformer encoder for spatial short-range features with a multi-scale temporal S4 decoder for long-range reasoning.The decoder progressively reduces token count and channel dimension to learn hierarchical representations while lowering computational cost.
- Results: ViS4mer outperforms previous approaches on 6 of 9 LVU tasks, is 2.63× faster, and requires 8× less GPU memory than its self-attention counterpart.It also generalizes to Breakfast and COIN procedural activity datasets with competitive results.
2 Related Work
Prior work established the difficulty of modeling long sequences and long-form videos: convolutional and Transformer-based video models face short-context or quadratic-cost limitations. ViS4mer builds on S4 layers and the LVU benchmark to address these constraints.
- Long Sequence Modeling: S4 research targets long-sequence modeling with lower complexity than standard self-attention.The related-work discussion motivates incorporating S4 ideas into a video architecture for long-range movie understanding.
- Video Recognition: Convolutional video models usually operate on short clips because their local receptive fields do not directly support long-range temporal reasoning.
- Video Recognition: Transformer video models improve video recognition but remain computationally costly because standard self-attention has quadratic complexity.
- Long-form Movie Videos: LVU provides nine publicly based, diverse long-form movie-understanding tasks for evaluating video-only models.The benchmark addresses limitations of earlier movie datasets, including restricted accessibility and multimodal language bias.
3 Background: Structured State-Space Sequence Model
The S4 framework maps input signals through a learned latent state and output projection while using structured state-space design to make long-sequence processing more efficient. Its linear dependence on sequence length motivates its use in ViS4mer.
- State-Space Model: A state-space model maps a one-dimensional input signal into an N-dimensional latent state and projects that state to a one-dimensional output.The state dynamics and output are parameterized by learned matrices A, B, C, and D.
- State-Space Model: Standard state-space computation requires successive matrix multiplications across the sequence, costing O(N^2L) operations and O(NL) space.The sequential computation also suffers from vanishing or exploding gradients.
- Structured State Spaces: HiPPO-based parameterization provides theoretical guarantees for capturing long-range dependencies in sequential data.
- Structured State Spaces: The S4 layer substantially reduces the computational cost of a basic state-space model.
- Complexity: State-space operations have linear time and space dependence on sequence length L, whereas self-attention has quadratic dependence.
- Connection to ViS4mer: ViS4mer applies multi-scale temporal S4 processing to whole-video patch tokens after short-range feature extraction.
4 The ViS4mer Model
ViS4mer combines a transformer encoder for short-range spatiotemporal feature extraction with a multi-scale temporal S4 decoder for long-range reasoning. The decoder progressively reduces spatiotemporal resolution and channel dimension to model long-range dependencies while lowering computation.
- Architecture: ViS4mer uses a transformer encoder for fine-grained patch-level feature extraction and a multi-scale temporal S4 decoder for long-range temporal reasoning.The encoder processes each frame, while the decoder operates on the resulting spatiotemporal tokens.
- Architecture: 11,760 output tokens arise from 60 frames at 224 × 224 resolution with 16 × 16 patches, making another self-attention stage costly.The token count is 14 × 14 × 60, and standard self-attention has quadratic complexity.
- Multi-scale Temporal S4 Decoder: The S4 decoder replaces self-attention for long sequences and has linear computation and memory dependence on sequence length.This design reduces the cost of processing large spatiotemporal token sequences.
- Multi-scale Temporal S4 Decoder: The decoder progressively decreases spatial resolution and channel dimension across blocks to learn dependencies at different scales.Deeper blocks operate on shorter sequences with smaller channel dimensions, which also reduces overfitting on the LVU benchmark.
- Multi-scale Temporal S4 Decoder: The S4 layer receives a flattened sequence of L = T × H′ × W′ vectors and outputs a feature tensor with the same sequence length and channel dimension.The decoder then applies pooling, an MLP, and a pooled linear skip connection before combining outputs.
- Multi-scale Temporal S4 Decoder: Each decoder block reduces spatiotemporal resolution through max-pooling, enabling multiscale representations while reducing long-sequence computation.Pooling uses temporal and spatial strides s_T × s_H × s_W.
- Loss Functions: Training uses cross-entropy loss for classification tasks and mean squared error for regression tasks.The classification model, regression model, labels, inputs, batch size, class count, and learnable parameters are defined in the loss formulation.
5 Experiments
Experiments evaluate ViS4mer on LVU and other long-range activity datasets, comparing it with prior models, self-attention baselines, efficient attention schemes, and architectural variants. Results show strong task performance, improved efficiency, and generalization beyond movie understanding.
- Main Results on the LVU benchmark: ViS4mer uses a Transformer encoder for short-range feature extraction and a multi-scale S4 decoder for long-range temporal reasoning.The decoder progressively reduces token count and channel dimension across layers.
- Main Results on the LVU benchmark: ViS4mer achieves state-of-the-art performance in most LVU tasks and outperforms prior long-range models on content understanding and metadata prediction.
- Ablation Studies: 2.63× faster and 8× less GPU memory than the corresponding Long Sequence Transformer at 11,760 spatiotemporal tokens.At this sequence length, the self-attention model requires 41.38GB, whereas ViS4mer requires 5.15GB.
- Ablation Studies: ViS4mer outperforms Performer and Orthoformer on most LVU tasks while using similar memory and computation cost.
- Ablation Studies: Multi-scale feature learning and progressive channel reduction both improve LVU performance and efficiency relative to vanilla S4.Compared with vanilla S4, ViS4mer is 2.2× faster and uses 1.41× less GPU memory.
- Evaluation on Other Datasets: ViS4mer generalizes to Breakfast and COIN, achieving competitive performance against prior approaches.It is competitive with a state-of-the-art method using several orders of magnitude more pretraining data.
6 Conclusion
ViS4mer combines self-attention with structured state-space models for efficient long-range movie video classification. The framework achieves strong results, uses relatively little computation and memory, and generalizes to procedural activity classification.
- ViS4mer combines self-attention and structured state-space models in an efficient framework for long-range movie video classification.
- The method achieves state-of-the-art results on several complex movie understanding tasks.
- ViS4mer has low memory requirements and computation cost.
- The framework successfully generalizes to procedural activity classification.