Source-linked AI summary
Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?
Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, Wanli Ouyang
TL;DR
Text-video retrieval commonly matches textual queries with video content, while videos may also have associated text. Cap4Video uses generated captions for training, video-caption interaction, and query-caption score fusion, and reports state-of-the-art results on four benchmarks.
Problem
Text-video retrieval typically matches query sentences to video visuals, while the potential of automatically generated auxiliary captions as additional retrieval information remains to be explored.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
Using captions across the framework’s stages yields consistent improvements in both matching mechanisms.
Source: §3.3 · p. 8
Abstract
from arXiv · showhide
Most existing text-video retrieval methods focus on cross-modal matching between the visual content of videos and textual query sentences. However, in real-world scenarios, online videos are often accompanied by relevant text information such as titles, tags, and even subtitles, which can be utilized to match textual queries. This insight has motivated us to propose a novel approach to text-video retrieval, where we directly generate associated captions from videos using zero-shot video captioning with knowledge from web-scale pre-trained models (e.g., CLIP and GPT-2). Given the generated captions, a natural question arises: what benefits do they bring to text-video retrieval? To answer this, we introduce Cap4Video, a new framework that leverages captions in three ways: i) Input data: video-caption pairs can augment the training data. ii) Intermediate feature interaction: we perform cross-modal feature interaction between the video and caption to produce enhanced video representations. iii) Output score: the Query-Caption matching branch can complement the original Query-Video matching branch for text-video retrieval. We conduct comprehensive ablation studies to demonstrate the effectiveness of our approach. Without any post-processing, Cap4Video achieves state-of-the-art performance on four standard text-video retrieval benchmarks: MSR-VTT (51.4%), VATEX (66.6%), MSVD (51.8%), and DiDeMo (52.0%). The code is available at https://github.com/whwu95/Cap4Video .
1. Introduction
Cap4Video uses automatically generated auxiliary captions to improve text-video retrieval through training-data augmentation, video-caption feature interaction, and query-caption score fusion. The introduction motivates these captions as a way to use associated video text without relying on manually crawled titles.
1. Introduction
- Prior methods expanded image-language pre-training models, especially CLIP [30], from global video-sentence matching to finer-grained alignments, fine-tuning pre-trained encoders end to end.The introduction attributes progress to pre-aligned visual and textual representations and end-to-end encoder fine-tuning on sparsely sampled frames.Source: §1 · pp. 1–2
- The captions are generated for videos without further training by extending ZeroCap’s frozen CLIP [30] and GPT-2 approach to the video domain.The paper presents this as an alternative to crawling video titles, which depends on annotations and risks invalid URLs.Source: §1 · p. 2
- Cap4Video uses captions as extra positive training pairs, for video-caption feature interaction, and for query-caption matching alongside query-video matching.The interaction is intended to reduce redundant video features and produce more discriminative representations; a two-stream architecture can reduce model bias and improve robustness.Source: Figure 1 · p. 1, §1 · p. 2
- Cap4Video achieves state-of-the-art performance on MSR-VTT (51.4%), VATEX (66.6%), MSVD [43] (51.8%), and DiDeMo [1] (52.0%).Source: §1 · p. 2
2. Methodology
Cap4Video uses auxiliary captions to augment training data, enhance video features through caption interaction, and complement video retrieval with query-caption matching.
2.1. Background: Text-Video Matching
- Global matching compares pooled video and query embeddings, while fine-grained matching models frame-word alignment with learned weighted pooling.The paper adopts the attention-based alignment mechanism as its enhanced baseline.Source: §2.1 · p. 2
2.2. Preprocessing: Caption Generation
- Captions come from crawled video titles or zero-shot GPT-2 generation guided by calibrated CLIP, without downstream training.Videos with expired links are skipped when crawling titles; the captioning approach uses an initial prompt such as “Video shows.”Source: §2.2 · p. 3, Figure 2 · p. 3
2.3. Data Augmentation with Auxiliary Captions
- At least N additional positive video-caption pairs can augment training for a dataset of N videos.When multiple generated captions are available, the method selects the caption most similar to the video's ground-truth query, using that query for filtering only during training.Source: §2.3 · p. 3
2.4. Video-Caption Cross-Modal Interaction
- Caption and frame embeddings interact through sum, MLP, cross-transformer, or co-attention strategies to produce enhanced frame representations.Interaction is restricted to final caption and frame embeddings; the resulting frames are pooled for global matching or retained individually for fine-grained matching.Source: §2.4 · pp. 3–4, Figure 3 · p. 4
2.5. Complementary Query-Caption Matching
- The query-caption branch aggregates caption embeddings and matches them to query embeddings, complementing query-video matching with a symmetric contrastive objective.The total training loss sums the query-caption and query-video losses.Source: §2.5 · pp. 4–5
3. Experiments: Text-Video Retrieval
Cap4Video is evaluated against state-of-the-art methods on four retrieval benchmarks, with ablations testing its three uses of captions and examples illustrating retrieval changes.
3.1. Setups
- Experiments use CLIP encoders for video, captions, and queries, train the query-video branch before the query-caption branch, and generate 30 captions per video with untuned CLIP and GPT-2.The caption and query encoders share parameters; text length is 32 and video length is 12 except for DiDeMo, which uses up to 64 words and frames.Source: §3.1 · p. 5
3.2. Comparison with State-of-the-Arts
- On DiDeMo, Cap4Video beats CLIP4Clip by 9.2% in R@1 and DRL by 3.0%; on MSR-VTT, it beats CLIP4Clip by 4.8% in R@1 with ViT-B/32.On MSR-VTT, it also exceeds TS2-Net by 2.3% with ViT-B/32 and 2.0% with ViT-B/16.Source: §3.2 · p. 5
- On MSVD, Cap4Video achieves 51.8% R@1, exceeding CLIP4Clip by 6.6% and X-Pool by 4.6%; on VATEX, it improves R@1 over TS2-Net by 7.5%.Both comparisons use ViT-B/16.Source: §3.2 · p. 5
- Cap4Video achieves consistent state-of-the-art performance across four benchmarks using standard retrieval without post-processing.The reported results do not use inference-time procedures such as DSL or QB-Norm.Source: Table 1 · p. 5, §3.2 · p. 6, Table 2 · p. 6
3.3. Ablation Study
- Generated captions used for training augmentation improve R@1 by 1.4% and 0.6% across two matching mechanisms; video titles improve global matching by 1%.The study also reports that one generated caption is sufficient for augmentation and that captions improve over the global-matching baseline for both online and offline videos.Source: §3.3 · p. 7
- For video-caption feature interaction, Sum improves fine-grained matching by 0.9% R@1, Cross Transformer gains 0.4% and 1.6%, and Co-attention Transformer gains 1.1% and 2.2% across the two matching settings.The MLP approach performs poorly in both settings; the authors attribute the transformer gains to modeling video-caption relationships.Source: §3.3 · p. 7
- The Query-Caption branch reaches 30.3% R@1, above ClipBERT’s 22.0% and MMT’s 26.6%; score fusion adds 0.8%, and all three caption uses yield gains of 2.8% and 3.6%.These final gains are reported for the two matching mechanisms.Source: §3.3 · pp. 7–8
3.4. Visualization
- In the examples, Cap4Video retrieves the ground-truth video with caption assistance, while the video-only model returns multiple relevant but imprecise videos.Figure 4 contrasts the video-only query-video ranking with Cap4Video on the MSR-VTT 1K-A test set.Source: §3.4 · p. 8, Figure 4 · p. 8
4. Related Works
The related work traces progress in zero-shot captioning and text-video retrieval, positioning auxiliary video captions as a training-free complement to existing retrieval methods.
Zero-shot Image Captioning
- GPT models generate text from prompts, while CLIP [30], trained on 400 million image-text pairs, shows zero-shot vision-task performance; transferring web-scale models to video captioning remains limited.Source: §4 · p. 8
- The method uses a video-domain extension of ZeroCap to generate auxiliary captions without additional training, following image-captioning approaches that combine CLIP with GPT-2.ZeroCap needs no retraining or fine-tuning, while MAGIC [32] requires fine-tuning on MS-COCO caption text; a later study [33] extended ZeroCap to video.Source: §4 · p. 8
Text-Video Retrieval
- Early retrieval methods [6] [10] [21] used expert-model knowledge and pre-extracted features, but their lack of end-to-end optimization limited performance.Source: §4 · p. 8
- Retrieval work shifted from large-scale text-video pretraining and downstream transfer [2] [26] [27] toward training-efficient VLM expansion, including CLIPBERT [19] and methods transferring CLIP knowledge [3] [4] [8] [9] [11] [13] [22] [48].CLIP-based matching developed from global video-sentence alignment [11] to fine-grained frame-word [36], video-word [13], and multi-hierarchical alignment [9] [28]. The proposed auxiliary-caption approach is compatible with both global and fine-grained matching.Source: §4 · p. 8
5. Conclusion
Cap4Video uses captions generated by web-scale language models to enhance text-video matching. It reports consistent gains over state-of-the-art methods on four standard benchmarks.
5. Conclusion
- Cap4Video consistently outperforms state-of-the-art methods on four standard text-video retrieval benchmarks.The conclusion attributes these gains to the framework’s use of generated captions.Source: §5 · p. 8
- Cap4Video uses captions for training-data augmentation, intermediate video-caption feature interaction, and output-score fusion.These mechanisms respectively augment training data, produce compact video representations, and improve text-video retrieval.Source: §5 · p. 8
A. Caption Generation
Cap4Video obtains auxiliary captions either by crawling video titles or by zero-shot video captioning with a CLIP- and GPT-based video extension, without additional training [33].
A. Caption Generation
- 2,555 of MSRVTT’s 10,000 videos have invalid source links, so those videos use only video-query matching rather than crawled titles.For videos with valid links, the website title is used as the caption.Source: A. Caption Generation · p. 11
- The captioner uses the video extension [33] of ZeroCap for zero-shot video captioning and can be replaced by other methods.ZeroCap uses GPT-2 to predict words from a prompt, while CLIP [30] provides visual guidance and an additional loss preserves the language model’s token distribution; the video extension optimizes pseudo-tokens through iterative sentence generation to improve coherence.Source: A. Caption Generation · p. 11, Figure A.1 · p. 11
- Generation requires no additional training; each sentence is capped at 20 tokens, frames are sampled at 3 FPS, and 30 iterations produce 30 captions per video.Source: A. Caption Generation · p. 11
B. More Results · B.1. Computation Efficiency
The section reports computational cost and efficiency measured as throughput on a single NVIDIA 3090 GPU with batch size 16.
B.1. Computation Efficiency
- Throughput for computational cost and efficiency is measured on a single NVIDIA 3090 GPU with batch size 16 (Table A.1).The passage does not provide throughput values.Source: §B.1 · p. 12
B.2. Additional Baselines
The additional baselines compare Cap4Video with a Video+Zero-Shot CLIP ensemble and examine the effects of fine-tuning CLIP for caption generation.
B.2. Additional Baselines
- 43.8% for Cap4Video exceeds 42.9% for the Video+Zero-Shot CLIP ensemble on MSRVTT 1k-A.The authors present the gap as evidence of GPT-2’s advantage.Source: Table A.2 · p. 12, §B.2 · p. 12
- Fine-tuned CLIP captions can further improve results, but reduce flexibility compared with the original CLIP and GPT-2 zero-shot captioner.The paper studies fine-tuned CLIP captions generated and filtered using the fine-tuned model; its standard captioner uses original CLIP and GPT-2 without fine-tuning to caption any video.Source: §B.2 · p. 12, Table A.3 · p. 12
B.3. Qualitative Results
The section presents visualizations of auxiliary captions and retrieval results to further examine the motivation for using auxiliary captions in text-video retrieval.
B.3. Qualitative Results
- The qualitative results include visualizations of auxiliary captions and retrieval results to further examine the motivation for using auxiliary captions in text-video retrieval.The visualizations are in Figures A.2 and A.3, respectively.Source: §B.3 · p. 13