Source-linked AI summary
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer
TL;DR
Video-language pre-training must support both cross-modal reasoning and tasks that use modalities separately, but existing approaches specialize in one architecture or objective. The paper introduces a single task-agnostic encoder with masking schemes for cross-modal fusion and separability, achieving strong results across diverse tasks with fewer parameters. Its reported error analysis identifies persistent difficulties recognizing visually similar objects, attributes, and subtle actions.
Problem
Existing video-language pre-training uses architectures specialized for cross-modal reasoning or separately embedded modalities, limiting broad task support.
Method
VLM uses a single encoder that accepts video, text, or both, combining masked modality modeling with masked token loss for cross-modal fusion.
Results
VLM performs well across a wider range of tasks, outperforming task-specific baselines by more than 2% on retrieval and 1% on video captioning despite using a smaller model.
Takeaways & Limitations
Task-agnostic pre-training can support retrieval and text generation while maintaining competitive performance with significantly fewer parameters.
Takeaways & Limitations
Error analysis reports recurring difficulties with recognizing similar objects, matching object attributes, and distinguishing subtle actions.
Abstract
from arXiv · showhide
We present a simplified, task-agnostic multi-modal pre-training approach that can accept either video or text input, or both for a variety of end tasks. Existing pre-training are task-specific by adopting either a single cross-modal encoder that requires both modalities, limiting their use for retrieval-style end tasks or more complex multitask learning with two unimodal encoders, limiting early cross-modal fusion. We instead introduce new pretraining masking schemes that better mix across modalities (e.g. by forcing masks for text to predict the closest video embeddings) while also maintaining separability (e.g. unimodal predictions are sometimes required, without using all the input). Experimental results show strong performance across a wider range of tasks than any previous methods, often outperforming task-specific pre-training. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.
1 Introduction
The paper proposes a task-agnostic video-language model that accepts video, text, or both, addressing limitations of modality-specific and task-specific pre-training. Its single-encoder design uses new masking objectives to improve cross-modal fusion while preserving separability and achieves strong performance across diverse tasks.
- Existing video-language pre-training is task-specific, using either cross-modal encoders for joint reasoning or separate unimodal encoders for retrieval-style tasks.
- VLM accepts text, video, or both as input through a task-agnostic single encoder.
- The approach reduces pre-training complexity, supports feature sharing without sacrificing separability, and is parameter efficient with BERTBASE-sized models.
- Masked modality model randomly masks an entire modality so the remaining modality predicts the masked tokens, encouraging cross-modal fusion.
- Masked token loss combines video and text embeddings to learn joint hidden states instead of using separate modality-specific losses.
- VLM outperforms similarly configured unimodal-encoder baselines by more than 2% on retrieval and 1% on video captioning, using a smaller model.
2 Related Work
Prior video-text pre-training uses shared encoders or task-specific retrieval and generation objectives, each imposing constraints on downstream use. These approaches motivate a simpler model that supports multiple modalities and task types without relying on task-specific pre-training.
- Shared-encoder approaches concatenate vision and text sequences but generally require both modalities simultaneously for downstream tasks.
- Some shared-encoder systems add decoders or pairwise retrieval objectives, increasing task-specific requirements and computational cost.
- Two-stream and decoder-based methods introduce retrieval, multi-task, or text-generation pre-training to accommodate different downstream tasks.
- Image-text pre-training commonly combines masked language modeling with pairwise image-text matching, and related video methods extend multimodal pre-training toward generation.
3 Pre-training
VLM uses one BERT-based encoder with modality-specific token preparation and shared prediction heads. Its pre-training combines conventional token masking with whole-modality masking and a unified masked-token objective to promote cross-modal representations while preserving modality separability.
- Video frames are converted into tokens with a frozen video encoder and trainable MLP, while text tokens use BERT embeddings.
- Video and text tokens are concatenated into one BERT sequence, masked, and processed by a single transformer.
- A shared linear prediction head maps BERT hidden states to video and text token embeddings in a joint space.
- MFM-MLM: MFM-MLM masks video and text tokens separately, predicts masked video embeddings with NCE, and sums the MFM and MLM losses.
- MMM and masked token loss: MMM masks all video or all text tokens in an example, forcing one modality to recover the other; experiments use 50% MMM and 50% MFM-MLM examples.
- MMM and masked token loss: The masked token loss uses video and text embeddings as shared contrastive candidates, and outperforms the separate-loss baseline in ablations.
4 Fine-tuning
Fine-tuning adapts the single encoder to retrieval, frame-level labeling, localization, question answering, and generation through task-specific attention masks and input arrangements. Isolated masks support separate modality representations, while autoregressive masks and language-model heads support caption generation.
- Task-specific attention masks adapt VLM to downstream tasks requiring video, text, or both modalities.
- Text-Video Retrieval: For joint-space retrieval, diagonally placed isolated masks prevent video and text tokens from attending to each other while sharing self-attention layers.
- Text-Video Retrieval: Retrieval averages video and text hidden states separately and applies contrastive loss to distinguish the matching clip from other videos.
- Action Segmentation: Action segmentation feeds the whole video with a dummy text token and classifies the final-layer hidden state of each video token.
- Action Step Localization: Action step localization separately processes long videos and fixed step labels with isolated masks and dummy tokens to avoid large pairwise matching.
- Video Captioning: Video captioning uses shifted text, a lower-triangle mask, and BERT language-model heads to provide autoregressive generation without an explicit decoder.
5.1 Dataset
The experiments use instructional and open-domain video datasets spanning retrieval, VideoQA, localization, and captioning. Howto100M supplies pre-training data, while MSR-VTT, YouCook2, and related benchmarks support downstream evaluation.
- Howto100M: Howto100M videos average about 6.5 minutes and 110 clip-text pairs, yielding approximately 7.7+ GB of captions after repeated-text removal.
- MSR-VTT: MSR-VTT contains 200K clip-text pairs from 10K open-domain videos across 20 categories and supports retrieval and VideoQA evaluation.
- YouCook2: YouCook2 contains 2,000 cooking videos, 14K clips, and 89 recipes, with retrieval and multimodal captioning evaluations after overlap filtering.
5.2 Hyper-parameters
The implementation uses compact video-token preprocessing and BERTBASE with fixed-length multimodal inputs. Training is conducted with a batch-oriented masking setup and shared optimization settings across pre-training and fine-tuning.
- Video representation: Video frames are encoded by S3D at 30 fps, producing one 512-dimensional video token per second that an MLP maps to BERTBASE’s 768-dimensional hidden size.
- Model and input: BERTBASE uses a maximum sequence length of 96, with 32 positions allocated to video and the remainder to text and special tokens.
- Model comparison: VLM is a substantially smaller BERTBASE model that remains self-supervised, task-agnostic, and compatible with joint retrieval and text generation.
- Model and input: Training examples use randomly sized text clips of 8–64 tokens, corresponding video clips, 32 sampled pairs per video, and batches of 256 examples.
- Optimization: Pre-training runs for 15 epochs on 8 NVIDIA Tesla V100 GPUs using fp16, Adam, a 5e-5 learning rate, warm-up, polynomial decay, and gradient clipping.
5.3 Model Comparison
VLM is a substantially smaller BERTBASE-based multimodal pre-training model that remains fully self-supervised and task-agnostic while supporting joint retrieval and text generation.
- Model Comparison: VLM uses a BERTBASE-sized model, making it significantly smaller than other transformer-based multimodal pre-training baselines.The comparison collects training paradigms and model sizes from papers or source code.
- Model Comparison: Despite its smaller size, VLM remains fully self-supervised and task-agnostic, without retrieval or autoregressive-style pre-training.It also supports both joint retrieval and text generation.
5.4 Quantitative Analysis
VLM performs competitively across retrieval, segmentation, localization, question answering, and captioning, while ablations show that masked modality training and masked token loss are important to its behavior.
- Text-Video Retrieval: VLM achieves good text-video retrieval on MSR-VTT and Youcook2, outperforming retrieval-style pre-training baselines despite being task-agnostic.The authors attribute the retrieval result to combining MMM with isolated self-attention masks.
- Action Segmentation: VLM outperforms other baselines on COIN action segmentation, indicating strong token-level video representations without compromising unimodal encoding.This evaluation uses hidden states for video tokens.
- Action Step Localization: VLM transfers to CrossTask action step localization without explicit alignment pre-training and performs better than some baselines, approaching one supervised method.The method directly compares video-frame and text-description representations in joint space.
- Video Question Answering: VLM reaches 91.64 accuracy on multiple-choice VideoQA, compared with 85.7 for ActBERT and 83.4 for JSFusion.The table compares pairwise matching and VLM on the VideoQA task.
- Video Captioning: VLM’s compact decoder performs surprisingly well on video captioning compared with fine-tuning baselines using external decoders.The captioning setup uses an autoregressive attention mask and BERT language-model heads.
- Ablation Study: Removing MMM substantially reduces retrieval and captioning performance, while 50% MMM performs best among the tested masking probabilities.The ablation contrasts traditional MFM-MLM-only masking with MMM rates of 30%, 50%, and 70%.
- Ablation Study: Using longer clips with at least 16 text tokens slightly lowers performance, suggesting that pre-training on longer clips may not cover short-clip fine-tuning tasks.The ablation also reports that masked token loss can outperform the multi-task-style LMFM-MLM loss.
5.5 Qualitative Analysis
The qualitative analysis identifies recurring video-language understanding errors and examines how VLM attention differs across modalities. It also suggests that video representations are more spatially extensive and diverse than text representations.
- Error Analysis: Video retrieval errors include difficult object recognition, attribute matching, subtle action distinctions, and overly general or specific query-video pairings.The authors note that some general-versus-specific mismatches may reflect limitations of existing annotations or evaluations rather than genuine errors.
- Error Analysis: Video captioning errors commonly arise when objects have similar shapes and colors, such as onion rings and shrimp.
- Attention Analysis: Video tokens occupy most of the representational space, whereas text tokens are relatively clustered.The authors attribute this pattern to greater diversity and sparsity in physical-world videos than in text from a fixed vocabulary.
- Attention Analysis: Unlike language models, VLM shows no recurrent shifted position-wise attention pattern for video tokens.
- Attention Analysis: Self-attention is more diverse in the first layer than in later layers, suggesting existing video encoders may be too deep for transformers.
- Attention Analysis: Some attention heads map video and text across modalities, including word-level co-reference between pouring soy sauce and the token “soy.”
6 Conclusions
The paper presents a task-agnostic single masked language model for video-text understanding that accepts either modality or both. It supports varied downstream tasks through attention-mask choices while maintaining competitive performance with substantially fewer parameters.
- Contributions: The proposed masking schemes train one masked language model to accept video input, text input, or both.
- Downstream Adaptation: Different attention masks allow the same VLM to support downstream tasks including text-video retrieval and video captioning.
- Results: The methods maintain competitive performance while requiring significantly fewer parameters than competing methods.