Source-linked AI summary
End-to-end Generative Pretraining for Multimodal Video Captioning
Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, Cordelia Schmid
TL;DR
Existing video-language pretraining methods often lack sentence-generation ability, while unlabelled videos lack manually annotated caption targets. MV-GPT jointly trains multimodal encoders and a sentence decoder with bidirectional generation from present and future utterances, achieving state-of-the-art results across video captioning and other video-understanding tasks.
Problem
Video-language pretraining often lacks sentence decoders, while unlabelled videos provide no manually annotated caption targets for jointly training multimodal encoders and decoders.
Method
MV-GPT uses future utterances as an additional text source and trains a multimodal encoder and sentence decoder jointly with a bidirectional generation objective.
Results
MV-GPT achieves state-of-the-art performance on four video-captioning benchmarks and on VideoQA, video retrieval, and action classification.
Takeaways & Limitations
The framework transfers end-to-end generative pretraining from unlabelled videos to multimodal captioning and other video-understanding tasks.
Takeaways & Limitations
The approach is not always successful when pretraining data and downstream applications have a significant domain shift.
Abstract
from arXiv · showhide
Recent video and language pretraining frameworks lack the ability to generate sentences. We present Multimodal Video Generative Pretraining (MV-GPT), a new pretraining framework for learning from unlabelled videos which can be effectively used for generative tasks such as multimodal video captioning. Unlike recent video-language pretraining frameworks, our framework trains both a multimodal video encoder and a sentence decoder jointly. To overcome the lack of captions in unlabelled videos, we leverage the future utterance as an additional text source and propose a bidirectional generation objective -- we generate future utterances given the present mulitmodal context, and also the present utterance given future observations. With this objective, we train an encoder-decoder model end-to-end to generate a caption from raw pixels and transcribed speech directly. Our model achieves state-of-the-art performance for multimodal video captioning on four standard benchmarks, as well as for other video understanding tasks such as VideoQA, video retrieval and action classification.
1. Introduction
Multimodal video captioning requires models that understand frames and speech while generating coherent descriptions, but unlabelled videos lack caption targets and prior pretraining often omits decoders. MV-GPT addresses this with bidirectional generation from utterances sampled at different times, jointly training the encoder and decoder.
- Multimodal video captioning must encode video frames and speech while generating coherent natural-language descriptions.
- Large-scale caption annotation is time intensive, expensive, subjective, and scarce compared with fully annotated image-classification data.
- Prior video-language pretraining commonly learns embeddings or multimodal encoders from instructional videos but often lacks a sentence decoder.
- Unlabelled videos provide speech for encoder inputs but no second textual stream serving as the decoder’s captioning target.
- MV-GPT uses future utterances as an additional text source, generating future speech from present multimodal context and present speech from future observations.
- The bidirectional objective requires no manually annotated captions and jointly trains the entire encoder-decoder model from utterances sampled at different times.
- MV-GPT achieves state-of-the-art results on four video-captioning benchmarks and other tasks including VideoQA, retrieval, and action classification.
2. Related Work
Earlier video-captioning systems evolved from templates to encoder-decoder models, while video-language pretraining on unlabelled instructional videos primarily targeted discriminative representation learning. These approaches improved representations but generally lacked sentence-generation capability.
- Early video-captioning methods combined detected subjects, verbs, and objects into sentence templates before adopting encoder-decoder formulations.
- Unlabelled instructional videos provide frames and ASR transcripts for proxy objectives such as masked modeling, video-text matching, and segment ordering.
- These pretraining approaches improve visual or multimodal representations but are designed for discriminative tasks and lack generation capability.
3. Method
MV-GPT jointly encodes video frames and utterances and decodes sentences through end-to-end transformer components. Its bidirectional objective uses present and future utterances, supplemented by masked language modeling, to train generation from unlabelled videos.
- MV-GPT pretrains an encoder-decoder model to encode multimodal videos and decode natural-language sentences from unlabelled videos.
- The framework samples aligned frames F, present utterances U, and an immediate future utterance W for each unlabelled video clip.
- Bi-directional Utterance Generation: Forward generation predicts future utterance W from frames F and present utterances U using autoregressive negative log-likelihood.
- Bi-directional Utterance Generation: Backward generation predicts present utterances U from aligned frames F and future utterance W, keeping visual inputs temporally aligned with decoder outputs.
- Bi-directional Utterance Generation: Masked language modeling is applied independently to both forward and backward input utterances as an auxiliary loss.
- Model Architecture: The model uses modality-specific BERT and ViViT encoders, a co-attentional multimodal transformer, and a modified GPT-2 decoder trained end-to-end from pixels and word tokens.
- Model Architecture: The visual encoder operates on coarsely sampled 1 fps frames rather than consecutive 30 fps frames, reducing compute and supporting end-to-end training.
- Sentence Decoder: The decoder autoregressively predicts each next token from multimodal context and previous tokens until an EOS token is generated.
4. Experiments
Experiments evaluate MV-GPT across four multimodal video-captioning benchmarks and additional video-understanding tasks. Ablations show that joint encoder-decoder pretraining, bidirectional generation, end-to-end visual training, and multimodal inputs are important to performance.
- Datasets and evaluation: MV-GPT uses one HowTo100M-pretrained model, fine-tuned on four captioning benchmarks with established captioning metrics.HowTo100M provides 1.2M instructional videos and 53M frame–utterance triplets for pretraining.
- Pretraining losses: Jointly pretraining the encoder and decoder improves performance substantially over encoder-only pretraining, while MV-GPT outperforms existing joint losses.The comparison varies only the pretraining loss while keeping the model architecture and training strategies fixed.
- Loss ablations: Backward generation boosts all metrics, while decoder-side masked language modeling and weight decay provide additional gains.Forward generation already supplies strong supervision; decoder-side masking slightly improves performance through additional input contextualization.
- Visual encoder and end-to-end training: End-to-end visual-encoder training yields larger gains with ViViT, and delayed end-to-end fine-tuning avoids YouCook2 degradation while improving MSR-VTT.The paper attributes the larger MSR-VTT gain to its greater domain gap and finds end-to-end fine-tuning less critical for YouCook2.
- Initialization: Random initialization still performs very well and can outperform initialization with public BERT, GPT-2, and ViViT weights.Pretraining from scratch approaches the setting in which all components use public pretrained weights.
- Multimodal inputs: Combining visual and textual inputs performs best, whereas text-only pretraining causes a 25% relative drop across all four captioning metrics.On YouCook2, CIDEr falls from 2.14 for the pretrained multimodal model to 1.43 when the text-only-pretrained model is fine-tuned with both modalities, and to 1.20 with text-only fine-tuning.
- Comparison with the state of the art: MV-GPT outperforms prior methods on all four captioning datasets, including a 15% absolute advance over M-MASS on ViTT B-1 and METEOR.It also achieves over 31% relative improvement over UniVL on MSR-VTT despite the domain gap between pretraining and evaluation videos.
- Other video-understanding tasks: The learned encoder transfers to VideoQA, video retrieval, and action classification, slightly exceeding VQA-T on both evaluated VideoQA datasets.For action classification, an MV-GPT-initialized ViViT improves Kinetics-400 accuracy by 24% absolute over training ViViT from scratch.
5. Conclusion
MV-GPT is a generative pretraining framework that jointly trains multimodal encoders and a caption-generating decoder from utterances sampled at different times in unlabelled videos. It achieves state-of-the-art results across video captioning and other video understanding tasks.
- MV-GPT jointly trains an encoder for multimodal inputs and a decoder for generating captions using utterances sampled at different times in unlabelled videos.
- The bidirectional generative objective supports end-to-end training during both pretraining and finetuning.
- MV-GPT achieves state-of-the-art results on multiple video captioning benchmarks and on VideoQA, video retrieval, and action classification.
A.1. Ablations on MSR-VTT
On MSR-VTT, MV-GPT shows similar ablation trends to YouCook2, although the performance gaps are smaller, which the authors attribute to a larger domain gap.
- MV-GPT exhibits similar ablation trends on MSR-VTT and YouCook2, but with smaller performance gaps on MSR-VTT.The authors believe the smaller gaps result from the larger domain gap between HowTo100M and MSR-VTT.
A.2. Impact of Pretraining Dataset Size
Increasing the pretraining dataset size improves all four captioning metrics, while pretrained MV-GPT also improves performance on open-ended long-form VideoQA.
- All four captioning metrics improve almost linearly when the pretraining dataset size is doubled.The result suggests that collecting more unlabelled videos could further improve the model.
- On OL-VideoQA, the pretrained MV-GPT model boosts performance further in all reported metrics compared with the no-pretraining model.The training set contains 26K examples, fewer than the 53K examples reported in the referenced work.
A.4. Impact of Decoder as a Part of Encoder
Using the pretrained decoder as part of the VideoQA encoder improves accuracy on both MSRVTT-QA and ActivityNet-QA.
- 1.0% and 0.8% accuracy gains result from including the pretrained decoder in VideoQA on MSRVTT-QA and ActivityNet-QA, respectively.
B.1. Pretraining Dataset Preparation
The pretraining data are built from HowTo100M videos by extracting frame–utterance–future-utterance triplets and segmenting clips around complete ASR sentences. The experiments also use established retrieval, VideoQA, and action-classification benchmarks.
- Pretraining data: HowTo100M videos are converted into triplets of frames, present utterances, and future utterances using noisy YouTube ASR transcripts.Removed YouTube videos are excluded to respect licensing terms.
- Clip construction: Video clips are expanded backward from one ASR sentence until they exceed 5 seconds, while preserving complete sentences.No sentence is cut off mid-way.
- Downstream benchmarks: MSR-VTT supplies the standard 9K-train and 1K-test retrieval split.
- Downstream benchmarks: MSRVTT-QA contains 243K question–answer pairs, while ActivityNet-QA contains 58K pairs across train, validation, and test sets.
- Downstream benchmarks: Kinetics evaluation covers Kinetics 400 and Kinetics 600, with approximately 267K and 446K clips, respectively.
C. Implementation Details
MV-GPT pretrains a visual encoder, text encoder, multimodal encoder, and decoder with a bidirectional generation objective, then transfers components or the full model to downstream tasks. Captioning uses the entire model, while other tasks adapt the architecture to their prediction format.
- Pretraining: The bidirectional pretraining loss jointly trains a visual encoder, text encoder, multimodal encoder, and decoder end-to-end.Different component subsets are transferred and fine-tuned depending on the downstream task.
- Captioning: The full pretrained MV-GPT is transferred for multimodal video captioning, changing the pretraining input–output configuration to predict captions from present utterances.
- VideoQA: Generative VideoQA concatenates the question with the present utterance, allowing the original MV-GPT model to generate open-ended answers without architectural changes.
- VideoQA: VideoQA classification also concatenates the question with clip utterances and uses the decoder as an additional multimodal encoder despite producing no textual output.
- Action classification: Action classification discards the other components and appends a randomly initialized classification layer to the pretrained visual encoder.
- Video retrieval: Bidirectional NCE applies matching losses to both (F,U,W) and (F,W,U) triplets to test whether generative pretraining improves retrieval.
- Qualitative results: MV-GPT qualitative examples capture textual cues such as “parsely” and visual actions such as “spreading sauce,” unlike the model without multimodal pretraining.
E. Limitations and Broader Impact
The paper identifies domain shift as a limitation and warns that large, uncurated web datasets can contain biases that pretrained models may amplify. It recommends careful dataset examination and filtering before applications beyond the academic setting.
- Limitations: MV-GPT is not always successful when there is significant domain shift between pretraining data and the downstream application.The paper proposes curated pretraining data as one direction for addressing this limitation.
- Broader impact: Large, uncurated web datasets may contain unintended biases that pretrained models inadvertently amplify.
- Broader impact: Applications beyond the presented academic setting should first examine and filter the pretraining dataset for potentially harmful biases.