VindLU: A Recipe for Effective Video-and-Language Pretraining

Motivation

• Complex, specialized architectures and protocols make recent VidL frameworks difficult to reproduce, analyze, and compare, while obscuring which components matter.The paper also notes that existing frameworks differ substantially across design dimensions, leaving the importance of those differences unclear.Source: §1 · p. 2

Empirical findings

• Temporal modeling improves averaged video-retrieval accuracy by 6%, video-to-text fusion by 3.6%, MLM by 6.2%, MVM by 1%, and joint image-video pretraining by 2.7%.The authors also report that four-frame pretraining is sufficient and that multi-stage training does not significantly boost performance.Source: §1 · p. 3

Main results

• VINDLU reaches 61.2% R@1 on DiDeMo and 55.0% R@1 on ActivityNet, outperforming state of the art by 7.8% and 6.1%, respectively.It also reports state-of-the-art video question-answering results on ActivityNet-QA, MSRVTT-QA, MSRVTT-MC, and TVQA.Source: §1 · p. 3

Contribution

• Rather than claim novelty for individual design choices or superiority over prior methods, the paper collectively validates components and presents the VINDLU recipe.The authors say fair and complete comparisons are difficult because of implementation complexity, and release code and pretrained models.Source: §1 · pp. 2–3

Image-Language Pretraining

• Image-language analysis such as METER does not establish whether its findings generalize to video or address video-specific choices such as temporal modeling and frame counts.VindLU studies these video-specific factors alongside multimodal fusion, objectives, data, and finetuning and inference protocols.Source: §2 · p. 4

Video-and-Language Pretraining

• VidL methods combine different architectures, objectives, data, and protocols, making it difficult to identify which components are needed for good performance.The paper contrasts its broad empirical study with prior work that focuses on masked video modeling or spatial biases.Source: §2 · p. 4

3. A Recipe for Video-Language Pretraining

• The recipe progressively expands a standard image transformer while testing how architecture, fusion, objectives, data, frame counts, and scaling affect VidL performance.The order of steps is primarily set by implementation cost; the authors could not ablate the order itself.Source: §3 · p. 4, Step 0: Starting Ingredients · p. 5

Step 0: Starting Ingredients

• The recipe starts from a ViT-B/16 image transformer and BERT text encoder trained on single frames with video-text contrastive learning; this baseline averages 50.4% retrieval accuracy.WebVid-2M is the initial pretraining data, and the subsequent recipe expands the baseline with temporal modeling, fusion, and additional objectives.Source: Step 0: Starting Ingredients · pp. 4–5

Step 1: Temporal Modeling

• Temporal attention improves retrieval accuracy by 6.3% over the spatial-only baseline and outperforms temporal convolution by 2.1%, while late temporal attention has nearly no effect.The authors adopt temporal attention and four-frame inputs for subsequent experiments.Source: Step 1: Temporal Modeling · p. 5

Step 2: Multimodal Fusion Encoder

• Video-to-text fusion performs best, improving performance by 3.6%; text-to-video fusion decreases it by 1.3%, and bidirectional fusion adds no improvement over video-to-text fusion.The authors use video-to-text fusion in subsequent experiments.Source: Step 2: Multimodal Fusion Encoder · p. 6

Step 3: Pretraining Objectives

• MLM pretraining yields a 6.2% performance boost; adding MVM loss further improves accuracy by 1%, but slows training by about 40%, so it is omitted from remaining experiments.For remaining experiments, the pretraining objectives are VTC, VTM, and MLM.Source: Step 3: Pretraining Objectives · p. 6

Step 4: Pretraining Data

• Joint image-video pretraining adds a 2.7% performance boost over video-only training, and four-frame pretraining improves 1.7% over single-frame training.Performance saturates at four frames; compared with 16-frame pretraining, four frames provide a 2.5× speedup. Multi-stage pretraining adds no significant performance boost.Source: Step 4: Pretraining Data · p. 7

Step 5: Finetuning & Inference

• Finetuning with 12 frames is 2.6× faster than with 32 frames at only 0.3% lower accuracy; most models use 12-frame inputs.Single-frame finetuning is 22.4× faster than 32-frame finetuning but has 5% lower accuracy; accuracy saturates at about 12 frames.Source: Step 5: Finetuning & Inference · pp. 7–8
• On longer videos, inference accuracy generally benefits from more frames but saturates as speed slows: 12 frames gives 73.4/70.4 D/A accuracy at 10.6× speedup, versus 73.8/72.8 at 1× for 64 frames.Source: Step 5: Finetuning & Inference · p. 8

Step 6: Scaling Up

• Scaling the corpus from 5M to 17M raises accuracy from 70.2% to 72.4% (+2.2%), and scaling to 25M reaches 73.6% (+1.2%).Source: Step 6: Scaling Up · p. 8
• On the 5M corpus, ViT_base→ViT_large raises accuracy from 70.2% to 73.2% (+3.0%), versus 71.2% (+1.0%) with BERT_large.The authors report that computational cost limited these model-size experiments to the 5M corpus.Source: Step 6: Scaling Up · p. 8
• The final scaled-up VINDLU model improves the initial image-transformer baseline by 23.2%.Source: Step 6: Scaling Up · p. 8

Other Useful Empirical Tips

• On MSR-VTT, DiDeMo, and ActivityNet-Captions, ViT outperforms VideoSwin by 1.6% on averaged R@{1,5,10}.Source: Other Useful Empirical Tips · p. 8, Table 2 · p. 8
• Linear learning-rate scaling gives similar or worse results, so the experiments use a fixed learning rate of 1e-4 across batch sizes.Source: Other Useful Empirical Tips · p. 8, Table 3 · p. 8
• MLM and MVM work better when initialized with encoders pretrained on those objectives; otherwise averaged R@1,5,10 drops by about 5%.The examples given are BERT for text and BEIT for video.Source: Other Useful Empirical Tips · p. 8

Text-to-Video Retrieval

• 79.3% (+5.6%) averaged accuracy on DiDeMo, 75.4% (+4.7%) on ActivityNet-Captions, and 84.6% (+4.6%) on SSv2 are reported for VINDLU.Source: §4 · p. 9, Table 4 · p. 9, Table 5 · p. 9
• On MSR-VTT, VINDLU scores 66.5% versus OmniVL's 68.6%, while pretraining takes 82 versus 169 V100 GPU days.Source: §4 · p. 9
• VINDLU pretraining takes 82 V100 GPU days, versus 640 for LAVENDER, 448 for All-in-one, and 984 for CLIP-ViP; a 15-day variant remains competitive.The framework is also described as simpler than multi-stage alternatives because it can be trained in a single stage.Source: §4 · p. 10

Video Question-Answering

• VINDLU exceeds prior state-of-the-art video-QA results by 0.6% on ActivityNet-QA, 0.3% on MSRVTT-QA, 3.4% on MSRVTT-MC, and 0.3% on TVQA.Source: Table 6 · p. 9, §4 · p. 10
• FrozenBiLM is de-emphasized in video-QA comparisons because it has 1.2B parameters and uses 400M pretraining examples, versus VINDLU's 201M parameters and 25M examples.Source: §4 · p. 10

Action Recognition

• On Kinetics-400, VINDLU's video encoder outperforms TimeSformer by 2.1% and OmniVL by 1.0% when all models use the same architecture.Source: §4 · p. 10, Table 7 · p. 10

• Temporal modeling, multimodal fusion, masked modeling objectives, and joint image-video training are critical for downstream VidL performance.Source: §5 · p. 10
• The empirical insights inform a step-by-step recipe for effective VidL pretraining, yielding the VINDLU model.Source: §5 · p. 10
• VINDLU achieves competitive or better results on nine VidL benchmarks while being simpler and more efficient than existing approaches.Source: §5 · p. 10

• Temporal embeddings are learnable and absolute, while spatial embeddings are relative; the implementation pads temporal embeddings or interpolates spatial embeddings when adapting to downstream inputs.Pretraining takes about 1–3 days depending on corpus size, while downstream finetuning takes 2–40 hours.Source: A. Implementation Details · pp. 10–11
• Video retrieval finetuning uses VTC and VTM losses, then reranks the top 128 candidates using pairwise VTM scores.Candidates are first selected by unimodal video-text similarity.Source: A. Implementation Details · p. 10
• Open-ended QA generates answers with a decoder that attends to multimodal encoder outputs and is initialized from pretrained multimodal encoder weights.Inference restricts generation to the 3,128 most common answers.Source: A. Implementation Details · pp. 10–11, Figure 4 · p. 11
• Multiple-choice QA converts each question and candidate answer into retrieval sentences, ranking them with a weighted combination of similarity and VTM scores.The weights are 0.3 for similarity and 0.7 for VTM.Source: A. Implementation Details · p. 11
• For inference with more frames than finetuning, temporal positional embeddings are linearly interpolated before visual tokens enter the multimodal encoder.Source: A. Implementation Details · p. 11
• The final VINDLU model uses a ViT video encoder initialized from ImageNet-21k-pretrained BEITbase, a nine-layer BERTbase text encoder, and a three-layer BERTbase multimodal fusion encoder.Its pretraining objective sums VTC, VTM, and MLM losses.Source: A. Implementation Details · p. 11

• C5M contains 5.44 million image/video-text pairs from WebVid-2M and CC3M, while C17M adds COCO, Visual Genome, SBU Captions, and CC12M.C25M adds WebVid-10M to C17M, excluding the two million WebVid-2M videos already included.Source: B. Dataset Descriptions · p. 11
• Text-to-video retrieval evaluation covers MSRVTT, DiDeMo, ActivityNet-Captions, SSv2-Template, and SSv2-Label.MSRVTT uses a 9K-video training split and 1K-A test set; DiDeMo evaluation uses paragraph-to-video retrieval on the first 30 seconds.Source: B. Dataset Descriptions · p. 11
• Video QA evaluation includes ActivityNet-QA and MSRVTT-QA open-ended datasets, plus MSRVTT-MC and TVQA multiple-choice datasets.MSRVTT-MC has five answer candidates per question; TVQA focuses on popular TV shows.Source: B. Dataset Descriptions · p. 12

• The section presents additional quantitative results on temporal modeling.Source: C. Additional Quantitative Results · p. 12
• Table 9 studies the MLM masking ratio with four frames, the final model, and the 5M corpus.Source: Table 9 · p. 12

C.1. Additional Ablation Studies

• A 50% MLM masking ratio is more helpful for VidL pretraining than BERT’s 15% ratio, with complementary video cues offered as the conjectured reason.The passage reports the ratio comparison as a finding and frames the explanation as a conjecture.Source: §C.1 · p. 12
• Temporal modeling, multimodal fusion, and joint image-and-video pretraining remain important on MSRVTT-QA and SSv2-Label and SSv2-Template retrieval.These results are reported as further support that the paper’s conclusions also hold on these tasks and datasets.Source: §C.1 · p. 12
• MSRVTT-QA results without a multimodal fusion encoder are unavailable because the task needs that encoder to generate answers.Source: §C.1 · p. 12

C.2. Additional Temporal Modeling Baselines

• The baselines are evaluated with four-frame inputs pretrained on WebVid-2M; the reported table experiments omit multimodal fusion and use the VTC loss.The compared schemes include mean pooling, late temporal attention, temporal convolution, temporal attention, prompt-based temporal attention, and Window Attention.Source: §C.2 · p. 13, Table 11 · p. 14
• Window Attention applies attention within T × k × k cuboids and is inserted before spatial attention; larger k leads to an out-of-memory error.Source: §C.2 · p. 13
• Temporal Attention is selected for subsequent experiments because k = 7 Window Attention offers only slight benefits with substantially higher GPU memory consumption.The table caption reports a +0.6% benefit for k = 7 compared with Temporal Attention.Source: Table 11 · p. 14
• Temporal Attention beats Temporal Convolution by 2.1% and Temporal Attention via Prompts by 6.8% on averaged top-{1,5,10} accuracy.Window Attention with k = 2 and k = 7 exceeds Temporal Attention by 0.2% and 0.7%, respectively.Source: §C.2 · p. 14