Source-linked AI summary
VideoBERT: A Joint Model for Video and Language Representation Learning
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, Cordelia Schmid
TL;DR
Existing self-supervised video methods often learn low-level, short-term features despite the abundance of unlabeled video. VideoBERT addresses this by jointly modeling quantized visual tokens and ASR-derived words with BERT, achieving high-level semantic representations, open-vocabulary classification, and state-of-the-art video captioning on YouCook II.
Problem
Existing self-supervised video methods commonly focus on low-level features and short temporal scales, motivating high-level representations for longer-duration actions and events.
Method
VideoBERT applies BERT to joint sequences of vector-quantized visual tokens and ASR-derived spoken words, using cross-modal alignment and modality-specific training objectives.
Results
VideoBERT learns high-level semantic representations, supports open-vocabulary classification, and outperforms the state of the art for video captioning on YouCook II.
Takeaways & Limitations
The model provides a joint visual-linguistic representation that captures semantically meaningful and temporally long-range video structure.
Takeaways & Limitations
The current approach uses frame skipping with a single visual vocabulary and lacks spatially fine-grained representations and explicit multi-scale temporal modeling.
Abstract
from arXiv · showhide
Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, we propose a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, inspired by its recent success in language modeling, we build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. We use VideoBERT in numerous tasks, including action classification and video captioning. We show that it can be applied directly to open-vocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, we outperform the state-of-the-art on video captioning, and quantitative results verify that the model learns high-level semantic features.
1. Introduction
VideoBERT uses language as self-supervision to learn high-level, temporally extended video representations by jointly modeling visual and spoken-language tokens. The model supports text-to-video prediction, long-range forecasting, video captioning, and open-vocabulary classification.
- Motivation: Existing self-supervised video methods largely learn low-level features and short-term motion, whereas this work targets semantic actions and events unfolding over longer periods.The motivation is to obtain representations useful for downstream video understanding.
- Approach: VideoBERT combines ASR-derived text, vector-quantized visual features, and BERT to learn joint distributions over visual and linguistic token sequences.The visual tokens come from low-level spatio-temporal features, while spoken words provide linguistic tokens.
- Applications: The joint model enables text-to-video prediction for illustrating instructions and video-to-text dense captioning, which significantly outperforms prior state of the art on YouCook II.These applications use the learned relationship between visual and spoken-language sequences.
- Applications: VideoBERT’s visual-word marginal distribution supports long-range forecasting at a higher semantic abstraction than video generators that mainly predict small low-level scene changes.The model generates plausible future guesses despite uncertainty about the future.
- Contribution: The paper’s contribution is a simple method for learning semantically meaningful video representations with temporally long-range structure.The introduction identifies activity recognition and video captioning as evaluation tasks.
2. Related Work
Related work spans supervised video representation learning, generative and self-supervised video modeling, cross-modal supervision, natural-language modeling, and captioning. VideoBERT differs by jointly modeling discretized visual signals and speech-derived language without manual labeling.
- Supervised learning: Supervised video methods require expensive labels, use relatively small action vocabularies, and typically represent short clips.Their vocabularies may miss distinctions such as sipping, drinking, and gulping.
- Unsupervised learning: Prior unsupervised approaches model video with stochastic latent variables and VAE- or GAN-style objectives, whereas VideoBERT uses BERT without an explicit stochastic latent variable.The cited approaches include models with static or temporal stochastic variables.
- Self-supervised learning: Self-supervised methods often predict one signal block from another, while VideoBERT uses quantized visual words and learns a proper joint generative model.The paper describes its approach as similar in spirit but different in representation and probabilistic structure.
- Cross-modal learning: Cross-modal video methods use synchronized audio and visual signals for supervision; VideoBERT specifically uses ASR speech rather than low-level sounds.Speech is used as a source of cross-modal supervision.
- Natural language models: VideoBERT builds on BERT to capture structure in both linguistic and visual domains, extending language-modeling ideas beyond text.The related-work discussion situates BERT among large-scale language models.
- Image and video captioning: Unlike captioning systems that model p(y|x) from manually provided captions, VideoBERT applies a joint p(x, y) model and achieves state-of-the-art video captioning results.The paper applies this model to video captioning.
- Instructional videos: For instructional-video analysis, VideoBERT avoids manual labeling and learns a large-scale generative model of words and discretized visual signals.This distinguishes it from cited instructional-video work.
3. Models
VideoBERT extends BERT to jointly model linguistic and visual token sequences using masked prediction and alignment objectives. It converts video into discrete visual words, combines them with ASR text, and supports multimodal and unimodal downstream uses.
- 3.1. The BERT model: BERT learns bidirectional token distributions by masking tokens and maximizing their pseudo log-likelihood with a bidirectional transformer.The model embeds token identities and positions, replaces a token with MASK, and predicts the masked token from the remaining sequence.
- 3.1. The BERT model: For paired sequences, BERT concatenates inputs with [CLS] and [SEP] tokens and uses the [CLS] representation for classification.The joint model represents two sentences and a label indicating whether they are consecutive or separate in the source document.
- 3.2. The VideoBERT model: VideoBERT represents raw video as discrete visual words produced by hierarchical vector quantization of features from a pretrained video model.This design makes minimal changes to BERT while shifting video representation toward discrete, higher-level modeling.
- 3.2. The VideoBERT model: VideoBERT combines ASR-derived linguistic tokens and visual tokens in one sequence, using a special [>] token to join the modalities.A typical sequence masks both linguistic and visual tokens for joint prediction.
- 3.2. The VideoBERT model: A linguistic-visual alignment objective predicts whether the linguistic and visual sentences are temporally aligned, but this signal can be noisy when speech refers to unseen content.Neighboring sentences are randomly concatenated to encourage semantic correspondence beyond strict temporal alignment.
- 3.2. The VideoBERT model: VideoBERT is trained with text-only, video-only, and video-text regimes, then used for masked-token prediction and [CLS]-based representations in downstream tasks.Random video subsampling from 1 to 5 steps improves robustness to speed variation and supports longer-term state transitions.
4. Experiments and Analysis
Experiments evaluate VideoBERT on YouCook II using large-scale YouTube pretraining, zero-shot action classification, and feature transfer for captioning. Results show that video inputs, more pretraining data, and the model’s open vocabulary support competitive recognition performance.
- Experimental setup: 312K YouTube cooking videos provide 23,186 hours of pretraining data, over two orders of magnitude larger than YouCook II.The pretraining collection contains videos shorter than 15 minutes; YouCook II contains 2K videos totaling 176 hours.
- Experimental setup: VideoBERT converts 1.5-second video clips into quantized visual tokens using pretrained S3D features and hierarchical k-means.The visual vocabulary contains 20,736 clusters from four hierarchy levels with 12 clusters per level.
- Zero-shot action classification: Zero-shot classification predicts YouCook II verbs and nouns by filling masked slots in the fixed sentence “now let me show you how to [MASK] the [MASK].”The first and second masked slots provide the verb and noun labels, respectively, without training on YouCook II data or its label ontology.
- Zero-shot action classification: VideoBERT significantly outperforms language-only baselines, while its cooking-adapted language prior exceeds vanilla BERT.The comparison removes video inputs and evaluates both VideoBERT’s language model and a text-only BERT prior.
- Zero-shot action classification: VideoBERT achieves comparable top-5 accuracy to a fully supervised S3D classifier, although the supervised model leads on top-1 verb accuracy.The comparison highlights the trade-off between VideoBERT’s effectively open vocabulary and the supervised classifier’s fixed label setting.
- Benefits of large training sets: Accuracy grows monotonically across 10K, 50K, and 100K-video pretraining subsets, with no observed saturation.The authors report that this pattern indicates potential benefit from even larger pretraining datasets.
5. Discussion and conclusion
VideoBERT demonstrates high-level joint visual-linguistic representations, while identifying spatial and temporal granularity as important directions for future work.
- VideoBERT learns high-level semantic representations and outperforms the state-of-the-art on video captioning.
- The model supports direct open-vocabulary classification, with performance growing monotonically as the training set increases.
- Fine-grained spatial representations are needed to distinguish individual objects and their attributes, especially in cooking applications.
- The current approach skips frames but uses a single vocabulary, leaving explicit modeling of multiple temporal scales for future work.
- Future evaluation should cover additional video-understanding tasks and domains beyond cooking.