Source-linked AI summary
UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, Ming Zhou
TL;DR
Existing video-language pre-training largely emphasizes understanding, leaving a gap for generation-oriented downstream tasks. UniVL addresses this with a unified Transformer model trained using multiple video-text objectives and strategies, achieving improved performance and state-of-the-art results across five multimodal tasks.
Problem
Most existing multimodal models are pre-trained for understanding tasks, creating a pretrain-finetune discrepancy for generation tasks.
Method
UniVL uses two single-modal encoders, a cross encoder, and a decoder trained with five objectives and two pre-training strategies on instructional video and ASR transcript pairs.
Results
UniVL improves performance over baseline models and achieves state-of-the-art results on five typical multimodal video-language tasks.
Takeaways & Limitations
The model provides a flexible video-language representation framework for downstream understanding and generation tasks.
Takeaways & Limitations
End-to-end training from raw videos remains constrained by enormous time and memory costs, motivating lightweight schemes such as keyframe training and smaller feature dimensions.
Abstract
from arXiv · showhide
With the recent success of the pre-training technique for NLP and image-linguistic tasks, some video-linguistic pre-training works are gradually developed to improve video-text related downstream tasks. However, most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks. This paper proposes UniVL: a Unified Video and Language pre-training model for both multimodal understanding and generation. It comprises four components, including two single-modal encoders, a cross encoder, and a decoder with the Transformer backbone. Five objectives, including video-text joint, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction, are designed to train each of the components. We further develop two pre-training strategies, stage by stage pre-training (StagedP) and enhanced video representation (EnhancedV), to make the training process of the UniVL more effective. The pre-train is carried out on a sizeable instructional video dataset HowTo100M. Experimental results demonstrate that the UniVL can learn strong video-text representation and achieves state-of-the-art results on five downstream tasks.
1 Introduction
UniVL addresses the limited focus of prior video-language pre-training on understanding by unifying video-language representation learning for both understanding and generation. It combines multiple objectives and staged strategies, then evaluates the model across five downstream tasks.
- Prior video-language models mainly pre-trained for understanding, while UniVL targets both understanding and generation through an encoder-decoder paradigm.
- UniVL uses two single-modal encoders, a cross encoder, and a decoder with Transformer components.
- StagedP first trains the text and video encoders jointly, then pre-trains all modules under the complete objectives; EnhancedV masks entire text inputs to strengthen video representation.
- UniVL is trained on large-scale instructional videos and fine-tuned for retrieval, captioning, action segmentation, action step localization, and sentiment analysis.
- The pre-training combines five objectives: video-text joint, CMLM, CMFM, video-text alignment, and language reconstruction.
2 Related Works
Prior multimodal pre-training research spans language, vision-language, and video-language settings with different encoder architectures. UniVL adopts a joint-type two-stream design to support deeper cross-modal interaction.
- 2.1 Single Modal Pre-Training: Language pre-training evolved from understanding-focused BERT toward unified models such as UniLM and BART that address understanding and generation.
- 2.1 Single Modal Pre-Training: Video representation pre-training has used reconstruction, future-frame prediction, patch ranking, and latent feature prediction objectives.
- 2.2 Multimodal Pre-Training: Vision-language and video-language models use share-type, cross-type, or joint-type encoder architectures.
- 2.2 Multimodal Pre-Training: Two-stream architectures accommodate modality-specific processing and interaction at varying representation depths compared with single-stream inputs.
- 2.2 Multimodal Pre-Training: UniVL adopts the joint-type structure, which uses a cross-modal encoder for full interaction between the two streams.
3 Method
The method formulates video-language pre-training as learning joint representations from paired videos and ASR transcripts, followed by downstream task fine-tuning.
- Given paired videos and ASR transcripts, UniVL uses self-supervision to learn joint video-text representations before fine-tuning downstream tasks.
3.1 Model Architecture
UniVL uses separate text and video encoders, a cross encoder for multimodal interaction, and a Transformer decoder for reconstruction and generation. Its two-stream design supports modality-specific processing and scalable retrieval, while the cross encoder combines their representations.
- Overall architecture: UniVL is an encoder-decoder architecture with text and video encoders, a cross encoder, and a Transformer decoder.The text encoder embeds tokens, the video encoder embeds frame sequences, the cross encoder models their interaction, and the decoder reconstructs input text.
- Single-modal encoders: The text and video encoders separately process their respective modalities before cross-modal fusion.The video encoder uses a Transformer over extracted frame features, while the text encoder uses BERT-based representations.
- Design rationale: The two-stream design enables reuse of pretrained text modules and scalable retrieval through indexed representations with linear complexity in the number of videos.Separate modality encodings can be computed before matching candidate videos with text queries.
- Cross encoder: The cross encoder concatenates text and video encodings along the sequence dimension and applies a Transformer to produce attended multimodal features.Sequence-wise combination accommodates different text and video lengths and imperfect temporal-semantic alignment.
- Decoder: The decoder receives attended text-video encoding and reconstructs or generates token sequences autoregressively.During pre-training it reconstructs input text; during fine-tuning it generates outputs such as video captions.
3.2 Pre-training Objectives
UniVL trains its multimodal architecture with objectives covering joint representation learning, masked language and frame prediction, alignment, and text reconstruction. These objectives combine contrastive multimodal learning with an autoregressive decoder objective.
- Objective overview: UniVL uses five objectives: video-text joint learning, conditioned masked language modeling, conditioned masked frame modeling, video-text alignment, and language reconstruction.The objectives train modality representations, cross-modal interactions, and generation capability.
- Video-text joint: The video-text joint objective aligns text and video representation spaces using positive temporal neighbors and negative pairs from other batch instances.Positive pairs include the matched transcript and nearby transcripts, while negatives come from other videos or transcripts.
- CMLM: CMLM randomly masks 15% of text tokens and predicts them using the video and the remaining contextual tokens.The objective conditions masked-token reconstruction on both known text and video input.
- CMFM: CMFM randomly masks 15% of video feature vectors and identifies the correct masked frame against negative distractors using contrastive learning.The model predicts masked-frame representations from surrounding frames and input text rather than reconstructing RGB pixels.
- Video-text alignment: Video-text alignment scores fused multimodal representations and distinguishes positive pairs from negative clips with NCE loss.Negative sampling includes both random cases and clips from the same video, whose frames are more similar than those from different videos.
- Language reconstruction: Language reconstruction trains an autoregressive decoder to reconstruct masked input text, supporting generation during downstream fine-tuning.The decoder predicts tokens sequentially from prior generated tokens and the text-video inputs.
3.3 Pre-training Strategies
UniVL introduces staged pre-training and EnhancedV to coordinate its objectives and strengthen video representations. StagedP begins with modality-specific joint learning before full-objective training, while EnhancedV removes text input for some examples.
- StagedP: StagedP first trains the text and video encoders with the video-text joint objective, then continues training all modules with the five objectives.The second stage uses a reduced learning rate after the initial encoder-focused stage.
- EnhancedV: EnhancedV masks all text tokens for 15% of text-video pairs, requiring the model to generate transcripts from video information alone.The strategy creates a harder generation task intended to improve video representation.
4 Experiments
The experiments pre-train UniVL on HowTo100M and fine-tune it across five multimodal tasks spanning retrieval, captioning, action understanding, localization, and sentiment analysis. The evaluation uses diverse instructional and open-domain video datasets.
- Pre-training data: UniVL is pre-trained on 1.2M filtered HowTo100M videos with ASR transcripts.Each video averages 6.5 minutes and contains 110 clip-text pairs.
- Downstream tasks: The model is fine-tuned on five downstream tasks: text-based video retrieval, multimodal video captioning, action segmentation, action step localization, and multimodal sentiment analysis.These tasks are evaluated using five corresponding datasets.
- Retrieval and captioning: YouCook2 supports text-based retrieval and multimodal captioning using cooking videos with annotated caption sentences.The dataset contains 2,000 videos across 89 recipes and 14K video clips.
- Retrieval and captioning: MSR-VTT provides open-domain video retrieval data with 200K clip-text pairs from 10K videos across 20 categories.Evaluation follows a setting using 1,000 sampled clip-text pairs as test data.
- Action understanding: COIN and CrossTask evaluate action segmentation and action step localization, respectively, using annotated instructional videos and task-step descriptions.COIN contains 180 tasks and 11,827 videos, while CrossTask contains 83 tasks and 4.7K videos.
- Sentiment analysis: CMU-MOSI evaluates multimodal sentiment analysis on 2,199 opinion video clips with real-valued sentiment intensity annotations.Sentiment intensity is annotated on a range from -3 to +3.
4.2 Experimental Details
UniVL uses BERT- and S3D-based Transformer encoders with fixed-length text and video inputs, plus cross-encoder and decoder modules. Training uses staged optimization on multiple GPUs.
- Text encoding uses BERT-base with 12 Transformer layers, 12 attention heads per block, and hidden size 768.
- Video encoding extracts 1,024-dimensional S3D features at 16 fps, then models temporal information with a six-layer Transformer encoder.
- Inputs contain at most 32 text tokens and 48 video features; the cross encoder has two layers and the decoder has three layers.
- Pre-training runs on 8 NVIDIA Tesla V100 GPUs with separate first- and second-stage hyperparameters.
- The experiments report retrieval results on Youcook2 and MSR-VTT datasets using tables of text-based video retrieval results.
4.3 Main Results
UniVL is evaluated across retrieval, captioning, action segmentation, action-step localization, and sentiment analysis. It achieves strong reported performance while exposing a performance–efficiency trade-off between alignment and joint retrieval variants.
- Text-based video retrieval: UniVL retrieval encodes queries and videos separately, then scores them either by dot product or with a cross-encoder alignment module.
- Text-based video retrieval: UniVL achieves the best retrieval performance over baselines on Youcook2 and MSR-VTT, outperforming HowTo100M and VideoAsMT on all metrics.
- Text-based video retrieval: UniVL (FT-Align) performs better, whereas UniVL (FT-Joint) is 50 times faster on Youcook2 and 10 times faster on MSR-VTT.
- Multimodal video captioning: For captioning, video and transcripts pass through the encoders and cross encoder before the decoder generates a token sequence.
- Multimodal video captioning: UniVL achieves state-of-the-art captioning results and outperforms existing pre-trained models even when using video alone.
- Action segmentation: UniVL significantly outperforms action-segmentation baselines by more than 14% and learns visual representations without linguistic descriptions.
- Action step localization: On CrossTask, UniVL surpasses the supervised baseline for action-step localization without fine-tuning on that dataset.
- Multimodal sentiment analysis: On CMU-MOSI, video-and-language UniVL achieves the best results despite omitting audio compared with a video, transcript, and audio baseline.
4.4 Ablation Studies
Ablations examine objectives, pre-training strategies, and visual features across retrieval and captioning. Results favor cross-encoder and decoder components, while fixed visual features remain a practical bottleneck.
- Modules and Strategies: Ablations evaluate objectives, strategies, and visual features on text-based retrieval and multimodal captioning.
- Modules and Strategies: StagedP and EnhancedV consistently improve retrieval results, while cross encoder and decoder modules promote joint video-text representation.
- Modules and Strategies: The decoder adds more than 3 points on the BLUE-4 captioning metric, whereas Joint loss slightly decreases generation performance.
- Visual Features: The ablations compare S3D features pretrained on HowTo100M with ResNet-152 and ResNeXt-101 features pretrained on ImageNet and Kinetics.
- Visual Features: Visual features are important for both pre-training and downstream tasks, making end-to-end raw-video training a future direction.
- Visual Features: Raw-video end-to-end training is constrained by enormous time and memory costs, with visual representation identified as the key bottleneck.
5 Conclusion and Discussion
UniVL uses self-supervised learning to jointly model video and language across multimodal understanding and generation tasks. Experiments on five typical downstream tasks show substantial gains over baselines and state-of-the-art performance.
- UniVL combines four modules and five objectives to support video-language understanding and generation.The model is designed as a flexible framework for multimodal downstream tasks, balancing efficiency and effectiveness.
- The model is evaluated on five downstream tasks, including text-based video retrieval and multimodal video captioning.
- UniVL improves performance substantially over baseline models and achieves state-of-the-art results on five typical multimodal tasks.