Source-linked AI summary
Towards Long-Form Video Understanding
Chao-Yuan Wu, Philipp Krähenbühl
TL;DR
Short-term vision systems recognize patterns within a few seconds but fail to contextualize the present across past and future events. The paper introduces an object-centric transformer framework and large-scale evaluations, finding that Object Transformers significantly outperform existing methods across long-form tasks and AVA.
Problem
Current vision systems focus on short video clips and struggle to connect present objects and events with past or future context.
Method
The paper detects and tracks objects throughout long videos and uses Object Transformers to model their complex interactions across time.
Results
Object Transformers outperform existing state-of-the-art methods on most long-form tasks and significantly outperform comparable methods on AVA 2.2.
Takeaways & Limitations
Long-term and object-centric design is important for a wide range of long-form video understanding tasks.
Takeaways & Limitations
Preliminary experiments found no advantages from longer training schedules or spans longer than 60 seconds.
Abstract
from arXiv · showhide
Our world offers a never-ending stream of visual stimuli, yet today's vision systems only accurately recognize patterns within a few seconds. These systems understand the present, but fail to contextualize it in past or future events. In this paper, we study long-form video understanding. We introduce a framework for modeling long-form videos and develop evaluation protocols on large-scale datasets. We show that existing state-of-the-art short-term models are limited for long-form tasks. A novel object-centric transformer-based video recognition architecture performs significantly better on 7 diverse tasks. It also outperforms comparable state-of-the-art on the AVA dataset.
1. Introduction
The paper frames long-form video understanding as connecting events, objects, and interactions across time to capture a video's full picture. It introduces an object-centric transformer, a large-scale benchmark, and evaluations showing short-term models are limited while Object Transformers perform better.
- Long-form understanding connects the present to past and future events rather than modeling only local properties.
- Object Transformers detect and track objects throughout videos, then model complex synergies among them across time.
- The benchmark contains 9 diverse tasks spanning more than 1,000 hours of video.
- Current short-term approaches perform poorly on long-form tasks even with strong pre-training and aggregation methods.
- Object Transformers outperform existing state-of-the-art methods on most long-form tasks and significantly outperform comparable methods on AVA 2.2.
2. Related Work
Prior work largely emphasizes short-term or adjacent-frame video patterns, while long-form understanding remains less explored and often lacks accessible vision-only benchmarks. This paper addresses that gap with a publicly accessible nine-task benchmark and object-centric long-range modeling.
- Most state-of-the-art video models operate on clips shorter than five seconds and target short-form understanding.
- Existing long-form datasets may rely heavily on language or use videos unavailable because of copyright restrictions.
- The new benchmark provides 9 vision-only tasks on more than 30K freely accessible videos.
- Some efficient or pooling-based methods process long videos but primarily model interactions between adjacent frames.
- Object-centric modeling instead captures long-range interactions between tracked objects.
3. Preliminaries
The preliminaries establish short-term detection, tracking, and shot segmentation as local building blocks. The paper then motivates connecting these parsed elements into a more complete long-form video representation.
- Short-term recognition modules detect objects, track boxes, and provide local understanding used by Object Transformers.
- Action and Object Detection: Action detection recognizes atomic human actions, while object detection finds objects and their categories.
- Tracking: Tracking associates repeated appearances across frames with an instance identity.
- Shot Transition Detection: Shot transitions divide videos into shots that serve as natural semantic boundaries.
- These local parsing methods do not connect video properties into a complete picture of the whole video.
- An instance is defined as one appearance in a space-time region that may span multiple frames.
4. Long-Form Video Understanding
Object Transformers model long-form videos around tracked objects, their temporal evolution, and inter-instance interactions, with self-supervised pretraining designed for long-range semantics. The section contrasts this approach with frame- and pixel-centric modeling and adapts compatibility prediction to flexible video structure.
- 4.1. Object-Centric Design: The object-centric view replaces pixel-volume modeling with structured representations intended to preserve semantic object relationships despite changing viewpoints.Pixel-volume modeling is described as costly for long-form videos and insensitive to invariance in the viewed world.
- 4.1. Object-Centric Design: Object Transformers represent videos through tracked instances and model how each instance evolves and interacts with other instances over time.Each instance is represented using box-level features together with spatial, link, and shot information.
- 4.1. Object-Centric Design: Object Transformers use projected feature and spatial embeddings, temporal and shot position embeddings, and learned instance embeddings to organize inputs by identity and location.The instance embedding is intended to distinguish inputs belonging to the same tracked instance, while the exact specification is deferred to the Appendix.
- 4.1. Object-Centric Design: The architecture uses a video-level [CLS] representation for task-specific heads and supports both direct end-task training and self-supervised pretraining.The self-supervised setup adds masked-instance and compatibility heads with associated losses.
- 4.2. Self-Supervision: The pretraining methods use unlabeled videos, but the full approach is not self-supervised because short-term features come from labeled data.The paper motivates pretraining as a response to weaker supervision per pixel in long-form videos and reports generalization challenges when training from scratch.
- 4.2. Self-Supervision: Masked-Instance Prediction hides selected instance features while retaining temporal, spatial, instance, and shot context to predict object or action semantics.The masked feature is replaced by a learned embedding with 80% probability, a random feature with 10%, and left unchanged with 10%.
- 4.2. Self-Supervision: Span Compatibility Prediction classifies whether two video spans belong together, encouraging high-level semantic relationships rather than immediate-frame continuity.Compatibility is framed as a relaxed alternative to next-sentence prediction because video events can vary in duration, participant count, and shot structure.
- 4.3. Implementation Details: For MovieClips, two spans are defined as compatible when they originate from the same one-to-three-minute segment.Compatibility training forms positive span pairs and uses other examples in the minibatch as negatives.
5. The Long-Form Video Understanding (LVU) Benchmark
The LVU Benchmark evaluates long-form video understanding across diverse content, engagement, and movie-metadata tasks using publicly available MovieClips videos. Its protocols use classification accuracy or regression MSE, while comparisons report that Object Transformers outperform baselines overall.
- Benchmark construction: LVU contains 9 long-form video-understanding tasks built from approximately 30K MovieClips videos drawn from approximately 3K movies.The videos are typically one-to-three minutes long and are resized to 480-pixel height.
- Tasks: The benchmark covers content understanding, user engagement prediction, and movie metadata prediction.Examples include relationships, speaking style, scene/place, YouTube statistics, director, genre, writer, and release year.
- Benchmark construction: MovieClips videos were accessed on February 26, 2020, with animations excluded and outros removed.
- Results: Object Transformers outperform both baselines by a clear margin in overall ranking, while short-term models remain effective for year prediction.The paper attributes year-prediction performance to cues such as picture resolution or quality.
- Evaluation Protocol: Content and metadata tasks use top-1 classification accuracy, whereas user-engagement tasks use mean-squared error.The benchmark uses relatively simple output spaces and evaluation protocols to make results easier to interpret.
6. Experiments
Experiments show that Object Transformers improve long-form video understanding across diverse tasks and also transfer long-term context to short-form action recognition. Ablations support self-supervised pre-training and object-level interaction modeling, while the model remains computationally efficient.
- Main Results: Object Transformers outperform both short-term and frame-based baselines in overall ranking across the long-form tasks.Short-term performance remains weak despite a large backbone and Kinetics-600 and AVA pre-training; VideoBERT performs well for writer prediction, while short-term models suffice for year prediction.
- Ablation Experiments: Self-supervised pre-training improves performance on all evaluated tasks, while Compatibility Pre-Training adds benefits on 4 out of 9 tasks.Masked Instance Pre-Training alone works well in almost all tasks, and the best validation-selected pre-training method is used by default for each task.
- Ablation Experiments: 8 out of 9 tasks favor the Object Transformer over pooling-based or late-fusion long-term modules.All compared methods use the same input features, isolating the long-term module; movie writer prediction is an exception where average pooling performs at least as well.
- Ablation Experiments: Adding non-person objects brings only mild improvement on three tasks, indicating that human behavior understanding is strong for most long-form tasks.Pre-training on more data helps on most tasks, and masked-instance case studies show plausible predictions from long-term context without viewing the masked content.
- Ablation Experiments: 2.2× smaller and 0.7% of the FLOPs compared to Action Detection, making the Object Transformer small and efficient.The paper connects this complexity profile to accessible future research on long-form video understanding.
7. Conclusion
The paper presents a benchmark for long-form video understanding and finds that Object Transformers perform significantly better across most evaluated tasks.
- The benchmark contains 9 tasks built from publicly available large datasets to evaluate diverse aspects of long-form video understanding.
- Existing short-term and frame-based long-term models are limited on most long-form tasks.
- Object Transformers model synergies among people and objects and work significantly better on most evaluated tasks.
A. Supplementary Implementation Details
The supplementary implementation details describe the Object Transformer architecture, training choices, baselines, and a controlled comparison with VideoBERT.
- The default Object Transformer uses 12 attention heads per layer in a 3-layer architecture with 64-dimensional heads, 768-dimensional hidden layers, and 3072-dimensional feed-forward networks.
- Attention masking accommodates examples containing different numbers of instances with different lengths.
- End-task fine-tuning selects batch size and training epochs for each task using validation-set grid search.
- The R101-SlowFast+NL baseline uses the open-source PySlowFast implementation with Kinetics-600 and AVA pre-training weights.
- VideoBERT is compared with Object Transformers using the same positional embedding formulation to control minor implementation differences.
- Three layers suffice for Object Transformers to achieve good performance, although the original BERTBASE architecture has 12 layers.
B. Supplementary Dataset Details
The supplementary dataset details define the benchmark’s classification and regression tasks, their labels, data sources, and selected dataset sizes.
- Relationship prediction is a 4-way classification task covering friends, wife-and-husband, boyfriend-and-girlfriend, and ex-boyfriend-and-ex-girlfriend, with 226 videos.
- Way of speaking prediction is a 5-way classification task with 1,345 videos, while scene/place prediction is a 6-way task with 723 videos.
- Director prediction is an 8-way task with 950 videos, and writer prediction is a 7-way task with 1,111 videos.
- Genre prediction uses four IMDb-derived categories and excludes videos belonging to more than one selected genre.
- Year prediction is a 9-way classification task over movie-release decades from the 1930s through the 2010s, containing 1,078 videos.
- YouTube like-ratio prediction uses videos with at least 30,000 votes and contains 940 videos.
C. Qualitative Evaluation Full Sequences
The qualitative evaluation supplements subsampled examples with complete 60-second frame sequences for three Masked Instance Prediction examples.
- The paper presents full 60-second sequences for the three examples shown in Figure 5.
- Each sequence is arranged from left to right and then top to bottom within its group.