Source-linked AI summary
Align and Prompt: Video-and-Language Pre-training with Entity Prompts
Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, Steven C. H. Hoi
TL;DR
Video-language pre-training must better align unimodal video and text features and provide fine-grained regional supervision without costly, limited-vocabulary detectors. ALPRO uses contrastive alignment and prompt-generated entity pseudo-labels, achieving substantial gains across retrieval and videoQA.
Problem
Existing methods leave unimodal video-text misalignment and detector-dependent fine-grained visual supervision insufficiently addressed.
Method
ALPRO combines video-text contrastive loss for instance-level alignment with prompting entity modeling for fine-grained region-entity alignment.
Results
ALPRO achieves substantial improvements over existing models on text-video retrieval and videoQA, including 3.0% and 5.4% absolute recall lifts on MSRVTT finetuning and zero-shot retrieval.
Takeaways & Limitations
Prompt-generated semantic pseudo-labels provide a detector-free route to region-entity alignment and can use a larger entity vocabulary.
Takeaways & Limitations
The paper identifies prompt engineering, prompt tuning, and temporally informed region selection as potential improvements to entity pseudo-label quality and crop selection.
Abstract
from arXiv · showhide
Video-and-language pre-training has shown promising improvements on various downstream tasks. Most previous methods capture cross-modal interactions with a transformer-based multimodal encoder, not fully addressing the misalignment between unimodal video and text features. Besides, learning fine-grained visual-language alignment usually requires off-the-shelf object detectors to provide object information, which is bottlenecked by the detector's limited vocabulary and expensive computation cost. We propose Align and Prompt: an efficient and effective video-and-language pre-training framework with better cross-modal alignment. First, we introduce a video-text contrastive (VTC) loss to align unimodal video-text features at the instance level, which eases the modeling of cross-modal interactions. Then, we propose a new visually-grounded pre-training task, prompting entity modeling (PEM), which aims to learn fine-grained region-entity alignment. To achieve this, we first introduce an entity prompter module, which is trained with VTC to produce the similarity between a video crop and text prompts instantiated with entity names. The PEM task then asks the model to predict the entity pseudo-labels (i.e~normalized similarity scores) for randomly-selected video crops. The resulting pre-trained model achieves state-of-the-art performance on both text-video retrieval and videoQA, outperforming prior work by a substantial margin. Our code and pre-trained models are available at https://github.com/salesforce/ALPRO.
1. Introduction
ALPRO addresses inefficient sparse-video pre-training, unimodal video-text misalignment, and limited fine-grained visual supervision. It combines contrastive instance alignment with prompt-generated entity supervision and improves retrieval performance.
- Motivation: Sparse frame sampling enables end-to-end video-backbone training while reducing the redundancy-related computation challenge in videos.The framework adopts sparse video-text pre-training rather than fixed offline video features.
- Challenges: Existing models often struggle because unimodal video and text features occupy different embedding spaces, while detector-based regional supervision has limited categories and costly computation.Detectors trained on MSCOCO recognize fewer than a hundred categories.
- Approach: ALPRO applies video-text contrastive loss to unimodal features, encouraging paired video-text instances to have similar representations before multimodal interaction modeling.Its architecture uses transformer-based video, text, and multimodal encoders.
- Approach: Prompting entity modeling predicts entities in randomly selected video crops from video and text inputs to learn fine-grained region-entity alignment.A standalone entity prompter generates pseudo-labels from normalized similarities between crops and entity prompts.
- Results: 3.0% and 5.4% absolute recall lifts are reported for finetuning and zero-shot text-video retrieval on MSRVTT, respectively.The comparison is reported as recall-score improvement over previous state-of-the-art methods.
2. Related Work
Related work contrasts dense offline video representations with sparse end-to-end training and describes prompt-based supervision for region-entity alignment. ALPRO combines sparse video processing with text-derived entity pseudo-labels.
- Dense versus Sparse Video Representation: Dense offline video features are computationally expensive and difficult to adapt because their visual backbones are not fine-tuned on target tasks.Consecutive frames often contain visually similar information, motivating sparse sampling.
- Dense versus Sparse Video Representation: ClipBERT and FiT use sparsely sampled frames with end-to-end visual-backbone fine-tuning, but ClipBERT is less effective across frames and FiT is retrieval-specific.These differences limit their natural generalization to videoQA or broad video-language modeling.
- Prompt-Based Supervision: ALPRO’s prompter generates soft entity labels by comparing video crops with textual prompts, providing region-entity supervision without relying on detector categories.The prompt-based labels exploit video-text alignment and a large vocabulary.
3. Video-Language Pre-training with ALPRO
ALPRO combines unimodal video-text alignment with fine-grained region-entity supervision in a video-language pre-training architecture. Its objectives use contrastive alignment, prompting entity modeling, masked language modeling, and video-text matching.
- Architecture: ALPRO contains a video-language pre-training model and a separate prompter, with unimodal encoders in both and an additional multimodal encoder in the pre-training model.The multimodal encoder captures interactions between video and text, while the prompter generates soft entity labels.
- Architecture: The model uses sparse video frames and independently encodes video and text before multimodal fusion.Video and text features are concatenated and processed by a multimodal transformer.
- Video-Text Contrastive Alignment: VTC aligns unimodal video and text [CLS] embeddings in a common normalized low-dimensional space before multimodal interaction.Its symmetric terms classify matched video-text pairs against other in-batch pairs, using a learnable temperature.
- Prompting Entity Modeling: PEM provides diverse regional supervision without extra human annotations, supporting efficient and scalable region-entity alignment.The task targets limitations in visually grounded modeling of local information relevant to visual reasoning.
- Prompting Entity Modeling: PEM trains the model to predict soft entity pseudo-labels for randomly selected video crops using joint video-text inputs.A prompter pretrained with VTC produces labels from similarities between crop features and text prompts instantiated with entity names.
4. Experiments
ALPRO is evaluated through ablations, retrieval comparisons, videoQA comparisons, prompt studies, and frame-count analyses. Across these experiments, its VTC and PEM objectives improve downstream performance, while sparse end-to-end training provides an efficiency-aware alternative to dense or offline video representations.
- Evaluation setup: ALPRO is evaluated on text-video retrieval and video question answering across four commonly used datasets.The evaluation covers proposed-objective ablations, comparisons with prior methods, and implementation analyses.
- Proposed objectives: PEM and VTC substantially improve performance over MLM+VTM across datasets, with VTC especially useful for retrieval and PEM especially beneficial for videoQA.The reported explanation links VTC to instance-level retrieval similarity and PEM to finer-grained regional features.
- Text-video retrieval: More than 6% lift in R10 is obtained on both MSRVTT and DiDeMo retrieval compared with previous methods.These comparisons include finetuning and zero-shot text-to-video retrieval, while using orders of magnitude fewer video-text pairs and no human-written texts.
- Video question answering: 2.6% and 3.3% accuracy lifts are reported on MSRVTT-QA and MSVD-QA, respectively, while ALPRO uses a generic architecture and 5.5M web video-text pairs.ALPRO is reported to obtain on-par results with VQA-T, which uses 69M QA-specific pre-training examples.
- Ablations and analysis: Prompt ensembling uses 12 templates each for video and image inputs, averaging prompt embeddings instantiated with the same entity.The study reports that prompt design and ensembling affect pre-training performance.
- Ablations and analysis: More frames generally improve performance, but retrieval benefits saturate beyond 8 frames; 8–16 frames provide a good metric-computation trade-off.Sparse frame sampling with end-to-end visual-backbone training is reported to outperform approaches using fixed offline features.
5. Conclusion
ALPRO is a sparse-frame video-language pretraining framework that combines instance-level video-text alignment with fine-grained region-entity alignment. It improves performance across multiple downstream datasets while highlighting prompt-based pseudo-labeling and deployment risks from web-collected data.
- ALPRO uses sparse video frames, video-text contrastive learning, and prompting entity modeling for video-language pretraining.
- The authors identify prompt engineering, prompt-guided temporal region selection, and extensions beyond video as directions for improving or expanding ALPRO.
- Web-collected pretraining data may contain unsuitable videos, harmful texts, and private information that could leak into pretrained models.
A.1. Prompt Templates
ALPRO ensembles multiple prompt templates by averaging entity prompt embeddings, improving pseudo-label quality without increasing computation relative to a single prompt. The dataloader alternates between video and image inputs using dedicated templates.
- Multiple prompt templates are ensembled in embedding space by averaging the embeddings for each entity.
- Embedding-space ensembling supports scalable addition of prompt templates at the same computation cost as one prompt.
- The dataloader switches between video and image inputs during pretraining and applies prompt templates for both modalities.
A.2. Finetuning Setups
Fine-tuning uses resized frames, dataset-specific frame sampling and augmentation settings, and uniform sampling without augmentation during inference. The configurations are reported for retrieval and videoQA datasets.
- Video frames are resized to 224 × 224 for all downstream datasets during fine-tuning.
- Fine-tuning randomly samples frames, applies consistent augmentation across frames within a video, and uses dataset-specific configurations.
- Inference disables augmentation and samples frames uniformly, while experiments use random seed 42 for reproducibility.
- Tables 9 and 10 provide end-to-end fine-tuning configurations for text-video retrieval and videoQA datasets.
B. Additional Examples of Pseudo-labels
The pseudo-label examples show that ALPRO generates diverse entity categories from highlighted video regions, including categories often absent from object-detector vocabularies. Entity labels can also provide corpus-level supervision when absent from descriptions.
- Pseudo-label examples include diverse categories such as towers, summit, and yoga that object detectors do not usually cover.
- Entity labels need not appear in text descriptions, allowing pseudo-labels to serve as a source of corpus-level supervision.