VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners

1. Introduction

• CoCa combines contrastive and captioning learning and supports image classification, image-text retrieval, and visual question answering.The introduction presents this existing breadth as the starting point for extending image-text modeling to video.Source: §1 · p. 1
• The paper asks whether transferring a well-pretrained image-text model requires heavy video-specific adaptors or extensive video-specific data.Prior video-transfer work had added modules for temporal aggregation or curated higher-relevance text-video pairs for contrastive training.Source: §1 · p. 1
• VideoCoCa reuses pretrained CoCa weights, applying its image encoder independently to sampled frames and its attentional poolers to the flattened frame tokens.The figure describes offline frame-token computation from a frozen image encoder; continued pretraining optimizes the poolers and text decoders while keeping that encoder frozen.Source: §1 · p. 1, Figure 1 · p. 2
• Without video-specific parameters, early temporal fusion, or video data, VideoCoCa achieves state-of-the-art zero-shot video classification and text-to-video retrieval results.The reported classification datasets include Kinetics 400/600/700, UCF101, HMDB51, and Charades; retrieval datasets include MSR-VTT, ActivityNet Captions, and VATEX.Source: §1 · pp. 1–2

Efficient video learners from image-text models

• Efficient video learners transfer image-text models using approaches such as cross-frame attention or text-conditioned weighted averaging of image embeddings; VideoCoCa instead adapts attentional poolers.X-CLIP uses cross-frame attention and video-specific prompting, while CLIP-Hitchhiker uses text queries to weight image embeddings for retrieval.Source: §2 · p. 2

Adapting image transformers to videos

• Video-transformer adaptations commonly extend image-model self-attention into space and time, using initializations such as divided attention or center-frame tubelet weights.Prior examples also initialize each view of a multiview encoder from image-model weights.Source: §2 · p. 2

Zero-shot transfer for video understanding

• Zero-shot video methods include frozen pretrained models with lightweight learned modules; VideoCoCa uses CoCa’s generative pooler as a parameter-free downsampler for fixed spacetime tokens.The passage compares this role with the perceiver resampler in Flamingo and the query learner in PaLI.Source: §2 · pp. 2–3

3.1. Preliminaries: CoCa

• CoCa combines contrastive and generative pretraining in an encoder-decoder with unimodal text encoding, multimodal cross-attention, and joint contrastive and captioning losses.Its [CLS] text representation supports contrastive loss, while captioning loss is applied per text token.Source: §3.1 · p. 3
• CoCa’s generative pooler produces 256 captioning embeddings and its contrastive pooler one contrastive embedding, regardless of image resolution.These poolers provide fixed-size outputs and act as lightweight adaptors while the pretrained ViT backbone remains frozen.Source: §3.1 · p. 3

3.2. Transferring CoCa to Video-Text Tasks

• Flattened frame tokens fed to CoCa’s generative and contrastive poolers provide late temporal fusion without new learnable layers, enabling zero-shot transfer.The frame tokens are produced by applying the CoCa image encoder to video frames independently.Source: §3.2 · p. 3
• The factorized encoder adds a four-layer Transformer over frame-level spatial embeddings, using its outputs for captioning and their temporal average for contrastive loss.The video representations are first produced by the generative and contrastive poolers.Source: §3.2 · p. 3
• The joint space-time alternative repeats pretrained image positional embeddings across time, allowing the image encoder to model spatial-temporal token interactions from its first layer.It adds no learnable layers but increases encoder self-attention computation with the longer token sequence.Source: §3.2 · pp. 3–4
• Mean pooling separately averages frame-level representations after the poolers, ignores temporal information, and serves as the adaptor baseline.CoCa also uses this method for zero-shot text-to-video retrieval metrics.Source: §3.2 · p. 4

3.3. Finetuning VideoCoCa on Video-Text Data

• Tuning options include updating all pretrained and added parameters, freezing encoder and decoder while tuning poolers, or tuning poolers before full finetuning.The two-step approach first trains the small pooler parameter set, which may make subsequent finetuning more stable.Source: §3.3 · p. 4
• LiT freezes the pretrained CoCa image encoder while tuning poolers and decoder, allowing frame embeddings to be precomputed and reducing image-encoder computation.Figure 2 compares component parameters with learnable parameters across tuning methods and identifies LiT as the adopted tuning approach.Source: §3.3 · p. 4, Figure 2 · p. 4

4.1. Experimental setup

• Evaluation spans zero-shot classification, retrieval, and captioning, plus finetuned open-ended VideoQA; VideoQA requires a new task-specific attentional pooler and has finetuned results only.Classification uses aligned video/text embeddings and label names; retrieval reports Recall at K, captioning reports BLEU-4, CIDEr, and ROUGE, and VideoQA reports top-1 test accuracy.Source: §4.1 · pp. 4–5
• Experiments use one epoch of continued pretraining, uniformly sample eight frames per clip, and evaluate with a single view; the main results use 576 × 576 resolution.VideoCC3M contains 8.7M video clip-text pairs from a more diverse online-video corpus, while HowTo100M is restricted to instructional videos.Source: §4.1 · pp. 4–5

4.2. Ablation study

• The Attentional Pooler, a late-fusion adaptor without new video-specific parameters, outperforms the other tested adaptors across classification, retrieval, and captioning.The comparison uses CoCa-Small continued-pretrained on VideoCC3M; early-fusion Joint Space-Time Encoder is the worst of the four methods.Source: §4.2 · pp. 5–6, Table 1 · p. 5
• Frozen Encoder Tuning (LiT) is comparable to full finetuning, while achieving the best retrieval and captioning results among the four tuning methods.LiT freezes the image encoder, tunes the text decoder and attentional poolers, and permits precomputed frame embeddings that save most TPU memory during training.Source: §4.2 · p. 6, Table 2 · p. 6
• Increasing HowTo100M’s pretraining mix improves YouCook2 retrieval but reduces MSR-VTT and ActivityNet Captions results, with no ratio best on all three datasets.Compared with 100% VideoCC3M, 100% HowTo100M changes YouCook2 by +1.7%, MSR-VTT by −2.5%, and ActivityNet Captions by −6.0%.Source: §4.2 · p. 6, Table 3 · p. 6
• Across model scales, VideoCoCa outperforms CoCa with the same parameter count; the 24-layer Large model outperforms 40-layer CoCa using 3.9× less TFLOPs.The 2.1B-parameter model advances on CoCa’s best results with less than 4% extra TFLOPs.Source: §4.2 · p. 6, Table 5 · p. 7

4.3. Main Results

• VideoCoCa achieves state-of-the-art zero-shot classification results and the highest reported retrieval metrics on MSR-VTT, ActivityNet Captions, and VATEX; YouCook2 is an exception.The classification comparison covers Kinetics-400/600/700, UCF101, HMDB51, and Charades.Source: Table 4 · p. 7, §4.3 · p. 7, Table 6 · p. 8
• VideoCoCa significantly outperforms other models in zero-shot captioning on MSR-VTT, ActivityNet Captions, and VATEX, and achieves competitive finetuned captioning and strong VideoQA results.The paper reports no single video-text model as significantly better than the others on all four tasks.Source: Table 7 · p. 8, §4.3 · p. 8, Table 8 · p. 8

• VideoCoCa transfers pretrained image-text CoCa models to video-text tasks in a simplified zero-shot approach.Source: §5 · p. 8
• Without video-only or video-text data, its zero-shot baseline achieves state-of-the-art results in video classification and text-to-video retrieval.Source: §5 · p. 8
• Efficient finetuning on video-text data achieves strong results on video captioning and video question answering.Source: §5 · p. 8
• The authors hope the approach and empirical results establish a strong baseline for leveraging pretrained image-text foundation models on video-text tasks.Source: §5 · pp. 8–9

• The frame-count ablation evaluates zero-shot classification and retrieval using VideoCoCa-Small and VideoCoCa-Base without VideoCC3M continued pretraining.Source: A. Varying the number of input frames · p. 12
• Performance gains on classification and retrieval become smaller starting from 8 frames, consistently across VideoCoCa-Small and VideoCoCa-Base.Evaluations use Kinetics-400 classification accuracy and MSR-VTT text-to-video retrieval R@K without continued pretraining on VideoCC3M.Source: A. Varying the number of input frames · p. 12
• The authors use 8 frames per clip to train the larger model faster and save resources.Source: A. Varying the number of input frames · p. 12

• Finetuning uses Adafactor with β1 = 0.9, β2 = 0.99, decoupled weight decay of 1e-4, cosine scheduling with linear warmup, and batch sizes of 64 or 128.Source: B. Finetuning hyperparameters · p. 12
• The experiments precompute frame embeddings using center crops and omit other data augmentation and regularization methods.The omitted methods include random flip, color jitter, mixup, label smoothing, and stochastic layer drop.Source: B. Finetuning hyperparameters · p. 12, Table 10 · p. 13
• Video question answering additionally learns an identity-activation fully connected classifier with softmax loss, using the base learning rate.Source: B. Finetuning hyperparameters · p. 12