PaperlayerView exact revision on arXiv

Source-linked AI summary

Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?

Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, Wanli Ouyang

arXiv:2301.00184v3cs.CV

TL;DR

Existing text-video retrieval often focuses on visual-text matching despite videos carrying useful associated text. Cap4Video generates captions with zero-shot video captioning and integrates them through data augmentation, feature interaction, and score fusion, achieving state-of-the-art results across four benchmarks.

  • Problem

    Text-video retrieval methods largely focus on visual content and query sentences, leaving the potential of associated video text for retrieval underexplored.

  • Method

    Cap4Video generates captions with zero-shot video captioning and uses them for training-data augmentation, video-caption feature interaction, and Query-Caption/Query-Video score fusion.

  • Results

    Cap4Video achieves state-of-the-art performance on MSR-VTT, VATEX, MSVD, and DiDeMo, reaching 51.4% on MSR-VTT.

  • Takeaways & Limitations

    Generated captions consistently improve text-video matching when incorporated as data, intermediate representations, and complementary output scores.

Abstract

from arXiv · show

Most existing text-video retrieval methods focus on cross-modal matching between the visual content of videos and textual query sentences. However, in real-world scenarios, online videos are often accompanied by relevant text information such as titles, tags, and even subtitles, which can be utilized to match textual queries. This insight has motivated us to propose a novel approach to text-video retrieval, where we directly generate associated captions from videos using zero-shot video captioning with knowledge from web-scale pre-trained models (e.g., CLIP and GPT-2). Given the generated captions, a natural question arises: what benefits do they bring to text-video retrieval? To answer this, we introduce Cap4Video, a new framework that leverages captions in three ways: i) Input data: video-caption pairs can augment the training data. ii) Intermediate feature interaction: we perform cross-modal feature interaction between the video and caption to produce enhanced video representations. iii) Output score: the Query-Caption matching branch can complement the original Query-Video matching branch for text-video retrieval. We conduct comprehensive ablation studies to demonstrate the effectiveness of our approach. Without any post-processing, Cap4Video achieves state-of-the-art performance on four standard text-video retrieval benchmarks: MSR-VTT (51.4%), VATEX (66.6%), MSVD (51.8%), and DiDeMo (52.0%). The code is available at https://github.com/whwu95/Cap4Video .

1. Introduction

Cap4Video addresses the use of automatically generated auxiliary captions to enhance text-video retrieval. It uses these captions for training-data augmentation, video-caption feature interaction, and output-score fusion, achieving strong benchmark performance.

  • Motivation: Text-video retrieval has evolved from global video-sentence alignment toward fine-grained cross-modal matching, supported by pre-trained image-language representations.CLIP provides pre-aligned visual and textual representations that reduce the challenge of cross-modal learning.
  • Motivation: Online videos often include titles or tags whose textual information can help describe video content and match queries.The paper therefore considers associated text alongside visual video signals.
  • Caption generation: Cap4Video generates video captions without additional training by extending zero-shot captioning with frozen CLIP and GPT-2.This avoids relying on manually crawled annotations or potentially invalid video URLs.
  • Cap4Video framework: The framework uses generated captions for input data augmentation, intermediate feature interaction, and output score fusion.These mechanisms supplement existing query-video matching mechanisms, including global and fine-grained matching.
  • Results: 51.4%, 66.6%, 51.8%, and 52.0% are Cap4Video’s reported state-of-the-art results on MSR-VTT, VATEX, MSVD, and DiDeMo, respectively.The results come from extensive experiments on four standard video benchmarks.

2. Methodology

Cap4Video augments text-video retrieval with automatically generated captions, using them to expand training data, enhance video representations, and add query-caption matching. Its interaction designs support both global and fine-grained matching.

  • Caption Generation: Cap4Video generates auxiliary captions with zero-shot video captioning based on frozen CLIP and GPT-2 knowledge.The framework considers automatic captioning alongside manual crawling of video titles.
  • Input Data: Generated video-caption pairs augment training beyond the original query-video pairs.For a dataset with N videos, selecting one caption per video adds at least N positive pairs.
  • Feature Interaction: Video-caption interaction combines frame and caption embeddings to produce enhanced video representations.The framework investigates sum, MLP, cross-transformer, and co-attention interaction strategies.
  • Feature Interaction: Enhanced frame embeddings are mean-pooled for global matching or retained for fine-grained matching.This preserves compatibility with both matching mechanisms.
  • Output Score: The query-caption branch compares query and aggregated caption embeddings to complement query-video matching.Caption embeddings are aggregated into a global representation before cosine similarity with the query embedding is computed.

3. Experiments: Text-Video Retrieval

Across four benchmarks, Cap4Video consistently improves text-to-video retrieval using generated captions, with gains from caption augmentation, video-caption interaction, and query-caption score fusion.

  • 3.2. Comparison with State-of-the-Arts: 9.2% R@1 improvement over CLIP4Clip is reported on DiDeMo, while Cap4Video exceeds DRL by 3.0%.
  • 3.2. Comparison with State-of-the-Arts: +4.8% R@1 over CLIP4Clip with ViT-B/32 is reported on MSR-VTT, with additional gains over TS2-Net for both tested backbones.
  • 3.2. Comparison with State-of-the-Arts: MSVD reaches 51.8% R@1, exceeding CLIP4Clip and X-Pool by 6.6% and 4.6%, while VATEX improves over TS2-Net by 7.5% R@1.
  • 3.2. Comparison with State-of-the-Arts: The reported results follow standard retrieval logic without post-processing based on relationships among other queries and videos.
  • 3.3. Ablation Study: Generated captions used as training augmentation improve R@1 by +1.4% and +0.6% under two matching mechanisms, while video titles improve global matching by 1%.
  • 3.3. Ablation Study: Intermediate video-caption interaction and query-caption score fusion further improve retrieval, with overall gains of +2.8% and +3.6% across the two matching mechanisms.

4. Related Works

Related work connects Cap4Video to zero-shot captioning methods that combine web-scale vision-language and language models to generate textual descriptions without retraining.

  • 4. Related Works: GPT-based models generate text from prompts, while CLIP provides zero-shot vision-language capabilities from large-scale image-text pretraining.
  • 4. Related Works: ZeroCap combines CLIP and GPT-2 to generate image descriptions without retraining or fine-tuning model parameters, motivating related video-captioning research.

5. Conclusion

Cap4Video uses web-scale generated captions to improve text-video matching through data augmentation, feature interaction, and score fusion, with consistent gains across four benchmarks.

  • 5. Conclusion: Cap4Video applies generated captions through input augmentation, intermediate video-caption interaction, and output score fusion.
  • 5. Conclusion: The framework reports consistent performance gains over state-of-the-art methods on four standard text-video retrieval benchmarks.

Supplementary Material

The supplementary material lists the paper’s authors.

  • Supplementary Material: The paper is authored by Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang.

A. Caption Generation

The appendix describes two ways to obtain auxiliary captions: crawling video titles or generating captions automatically with zero-shot video captioning. The captioner uses CLIP and GPT-2 to produce video descriptions without additional training.

  • Caption Generation: Auxiliary captions are obtained either by crawling video website titles or by automatic zero-shot video captioning.The crawling approach uses original video links, while automatic captioning generates descriptions directly from downstream videos.
  • Caption Generation: 2555 of 10,000 MSRVTT videos have invalid links, preventing title extraction for those videos.Those videos therefore use only video-query matching rather than title-based auxiliary information.
  • Caption Generation: ZeroCap uses GPT-2 to generate sentences guided by calibrated CLIP loss and regularized toward the original language model distribution.CLIP provides video-related guidance while the language-model objective preserves linguistic attributes.
  • Caption Generation: The captioner generates video captions without additional training using CLIP and GPT-2 knowledge.Generation uses a 20-token sentence limit, 3 FPS frame sampling, 30 optimization iterations, and produces 30 captions per video.
  • Caption Generation: Examples show generated captions describing videos such as a fashion runway scene and helping correct an original Top-1 retrieval prediction.The appendix also presents auxiliary-caption examples on the MSRVTT 1K-A test set.

B.1. Computation Efficiency

The paper reports computational cost and efficiency using a single NVIDIA 3090 GPU and batch size 16 to measure throughput.

  • Computational cost and efficiency are reported in Table A.1.The passage identifies Table A.1 as the location of these measurements.
  • Throughput is measured on a single NVIDIA 3090 GPU.
  • The throughput measurement uses a batch size of 16.

B.2. Additional Baselines

Additional baselines examine ensemble scoring and caption generation with a fine-tuned CLIP model. The reported comparison favors Cap4Video over the zero-shot CLIP ensemble, while fine-tuned captions can improve results at the cost of flexibility.

  • B.2. Additional Baselines: 42.9% vs 43.8%: “Video+Zero-Shot CLIP” scores lower than Cap4Video on MSRVTT 1k-A.The comparison is presented as evidence for the contribution of GPT-2 beyond the ensemble baseline.
  • B.2. Additional Baselines: The appendix includes a captioner-setting comparison table and an ensemble-baseline table for the additional baseline analysis.The tables are labeled Table A.3 and Table A.2, respectively.
  • B.2. Additional Baselines: Fine-tuned CLIP captions can bring further improvement compared with the original CLIP and GPT-2 captioner.The paper studies this setting as an additional captioner baseline in Table A.3.
  • B.2. Additional Baselines: Fine-tuned CLIP reduces Cap4Video’s flexibility despite improving the generated-caption setting.The original captioner uses unfine-tuned CLIP and GPT-2 to caption arbitrary videos without additional training.
  • B.2. Additional Baselines: Qualitative retrieval examples compare Cap4Video with a model that does not involve captions.The Cap4Video results are shown alongside the caption-free model’s results.

B.3. Qualitative Results

The qualitative-results appendix presents visualizations intended to further validate the motivation for auxiliary captions in text-video retrieval. It includes both caption examples and retrieval-result comparisons.

  • B.3. Qualitative Results: These visualizations are presented to further validate the motivation for using auxiliary captions for text-video retrieval.The appendix frames the figures as qualitative evidence related to the paper’s motivation.
Loading 2301.00184v3…