PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models

Junting Pan, Ziyi Lin, Yuying Ge, Xiatian Zhu, Renrui Zhang, Yi Wang, Yu Qiao, Hongsheng Li

arXiv:2306.11732v1cs.CV

TL;DR

VideoQA lacks comprehensive manual annotation, and existing approaches face costly cross-modal training or captioning models with limited domain generalization. R2A retrieves semantically similar text from a generic corpus to condition a pretrained language model, reporting state-of-the-art results across benchmarks and outperforming a much larger Flamingo model without fine-tuning.

  • Problem

    VideoQA's open-ended nature makes large, comprehensive manual annotation practically impossible, while alignment methods incur costly training and captioning methods have limited domain generalization.

    Source: §1 · p. 1

  • Method

    R2A retrieves semantically similar texts from a generic corpus for a video and uses them with the question to condition a pretrained language model, without cross-modal fine-tuning.

    Source: §1 · p. 2

  • Results

    With 1.3B parameters and no additional tuning, R2A achieves new state-of-the-art performance on multiple benchmarks and outperforms the cross-modal training-based Flamingo [1] with 80B parameters.

    Source: §1 · p. 2

  • Takeaways & Limitations

    R2A's modular design allows pretrained models to be used or changed for VideoQA without fine-tuning.

    Source: §1 · p. 2

    R2A's retrieved-caption quality depends heavily on retrieval-model performance and text-corpus diversity, and CLIP may struggle with some videos or text.

    Source: §6 · p. 8

Abstract

from arXiv · show

Video Question Answering (VideoQA) has been significantly advanced from the scaling of recent Large Language Models (LLMs). The key idea is to convert the visual information into the language feature space so that the capacity of LLMs can be fully exploited. Existing VideoQA methods typically take two paradigms: (1) learning cross-modal alignment, and (2) using an off-the-shelf captioning model to describe the visual data. However, the first design needs costly training on many extra multi-modal data, whilst the second is further limited by limited domain generalization. To address these limitations, a simple yet effective Retrieving-to-Answer (R2A) framework is proposed.Given an input video, R2A first retrieves a set of semantically similar texts from a generic text corpus using a pre-trained multi-modal model (e.g., CLIP). With both the question and the retrieved texts, a LLM (e.g., DeBERTa) can be directly used to yield a desired answer. Without the need for cross-modal fine-tuning, R2A allows for all the key components (e.g., LLM, retrieval model, and text corpus) to plug-and-play. Extensive experiments on several VideoQA benchmarks show that despite with 1.3B parameters and no fine-tuning, our R2A can outperform the 61 times larger Flamingo-80B model even additionally trained on nearly 2.1B multi-modal data.

1. Introduction

VideoQA's open-ended questions make comprehensive manual annotation practically impossible, while existing alignment and captioning approaches face training-cost, flexibility, or generalization limits. R2A retrieves relevant text from a generic corpus for a pretrained language model, and the paper reports state-of-the-art results across multiple benchmarks without fine-tuning.

1. Introduction

  • VideoQA's open-ended nature makes manual annotation of a large, comprehensive dataset practically impossible, motivating zero-shot approaches.The paper frames zero-shot learning as an alternative to task-specific training.Source: §1 · p. 1
  • Cross-modal alignment requires costly training and retraining when components change, while captioning relies on target-domain-finetuned models that may generalize less well.The cited alignment approach also requires large models and additional multimodal data.Source: §1 · p. 1
  • R2A replaces costly video captioning with retrieval of semantically similar text from a generic corpus, then supplies that text and the question to a pretrained language model.The retrieved pool serves as semantic video summarization, and the modular design allows components to be changed without fine-tuning.Source: Figure 1 · p. 1, §1 · p. 2
  • With 1.3B parameters and no additional tuning, R2A outperforms the cross-modal training-based Flamingo [1] with 80B parameters and achieves new state-of-the-art performance on multiple benchmarks.The framework uses a pretrained multimodal model to retrieve text and a pretrained language model to answer questions.Source: §1 · p. 2

2. Related Work

Prior VideoQA methods include supervised learning and large-scale pretraining followed by task-specific fine-tuning, while zero-shot work seeks generalization to unseen samples. Related approaches bridge video and language through cross-modal training or generated captions; this paper situates retrieval from open-world text knowledge as a distinct direction.

2.1. Recent advances of VideoQA

  • Zero-shot VideoQA targets previously unseen samples without supervision, alongside supervised and pretrained-then-fine-tuned approaches that remain limited by labeled-data size or task-specific domains.Reserve learns from web videos and transcripts; Flamingo and FrozenBiLM use frozen pretrained models with cross-modal training.Source: §2.1 · p. 2

2.2. LLMs for Vision and Language Tasks

  • Vision-language methods either train alignment modules with frozen language models or generate captions, while both routes retain resource or domain limitations.Flamingo [1] was trained for 500K TPU hours on billions of image/video-text pairs; FrozenBiLM [61] can require hundreds of GPU hours and millions of video-text pairs for adaptation.Source: §2.2 · pp. 2–3

2.3. Retrieval augmentation

  • The paper places its approach in retrieval augmentation and describes it as the first use of cross-modal retrieval from open-world knowledge for VideoQA.Prior examples include retrieval augmentation for training or inference across vision-related applications.Source: §2.3 · p. 3

3. Methodology

R2A bridges pretrained vision and language models for zero-shot VideoQA by retrieving video-relevant text from an external corpus, then applying a language model to answer.

3. Methodology

  • R2A retrieves video-relevant text from an external corpus to transfer video information into text for a language model to answer VideoQA questions.The framework is designed to use pretrained vision and language models.Source: §3 · p. 3

3.1. Text Corpus Encoding

  • Text corpus encoding precomputes an embedding for every corpus text sample once using the vision-language model’s text encoder.The embeddings are used for efficient retrieval.Source: §3.1 · p. 4

3.2. Video Encoding

  • Video encoding extracts a visual feature embedding for each frame with the pretrained vision encoder.The implementation uses CLIP ViT-L/14, while the passage notes that similar models such as ALIGN can also be considered.Source: §3.2 · p. 4

3.3. Video-Text Retrieval

  • Video-text retrieval scores frame features against text features, ranks corpus samples, and selects the top k matches for each frame.Dot product is the default choice in the main experiments.Source: §3.3 · p. 4

3.4. Temporal Aware Prompting

  • Temporal-aware prompting inserts natural-language cues such as “Firstly,” “Then,” and “Finally” to indicate the retrieved captions’ temporal order.Source: §3.4 · p. 4

3.5. Answer Generation

  • Answer generation prompts a pretrained masked-language model with the question, an answer mask, and the retrieved contextual text, then uses its masked-token prediction as the answer.The template places the question first, followed by “Answer:[MASK]” and the contextual hints.Source: §3.5 · p. 4

3.6. R2A with Learnable Visual-Text Projection

  • The optional R2A-FT variant adds a learnable visual-to-text projection while freezing the language model, but this training reduces component flexibility.The projection is the only part optimized and is randomly initialized.Source: §3.6 · p. 4
  • R2A-FT trains the projection using masked-language modeling on retrieved contextual text, video-conditioned prompts, and training captions with randomly masked tokens.At inference, it prepends the learned video-conditioned prompts to the answer-generation input; the mask ratio is 50%.Source: §3.6 · pp. 4–5

4. Experiments

Experiments evaluate zero-shot VideoQA across diverse open-ended benchmarks, comparing R2A with existing methods and testing its implementation choices. Results report broad state-of-the-art competitiveness, efficient inference, and gains associated with retrieval quality, quantity, and corpus construction.

4.1. Datasets and Evaluation

  • Evaluation covers six open-ended VideoQA benchmarks spanning YouTube videos, sports videos, and GIFs, using top-1 exact-match accuracy.The benchmarks are MSRVTT-QA, MSVD-QA, ActivityNet-QA, TGIF-QA, iVQA, and NextQA.Source: §4.1 · p. 5

4.2. Implementation details

  • The implementation uses CLIP ViT-L/14 features from 10 uniformly sampled frames, DeBERTa-XL by default, and naive top-k retrieval with duplicates removed.The maximum language-model input length is 500; cross-modal fine-tuning comparisons use one epoch and a batch size of 64.Source: §4.2 · p. 5

4.3. Comparison with the State-of-the-arts

  • R2A outperforms methods trained on million- to billion-scale vision-language data except on iVQA, and improves over VidIL with the same language model.Flamingo is the exception on iVQA; Table 1 compares methods across six benchmarks.Source: §4.3 · pp. 5–6, Table 1 · p. 6
  • At 10 retrieved sentences per video, R2A outperforms FrozenBiLM on all tasks; retrieval takes 3.5ms and preparation uses retrieval-set feature extraction.R2A’s inference time is 0.11s per video, while FrozenBiLM’s is 0.07s; R2A’s longer language-model latency reflects its longer input.Source: §4.3 · pp. 5–7, Table 2 · p. 7

4.4. Ablation Studies

  • Larger retrieval corpora consistently improve performance, while readable human-written COCO captions yield stronger gains than noisy CC captions despite smaller size.A 10k-sample corpus already improves over using no retrieved captions; the default is WebVid-10M.Source: §4.4 · p. 7, Table 5 · p. 7
  • Accuracy rises across all four datasets as more captions are retrieved, and R2A beats random text sampling across datasets and retrieval sources.Random text can still improve over the question-only baseline on MSRVTT and MSVD.Source: §4.4 · pp. 7–8, Table 6 · p. 7
  • Video-text retrieval outperforms video-video retrieval on all four datasets; retrieved contexts help alternative language models except BERT-base on ActivityNet-QA.Prompt choices show no significant variation, although “Hints:” and “Contexts:” perform slightly better than “Subtitles:” and “Captions:”.Source: Table 3 · p. 7, Table 4 · p. 7, §4.4 · p. 8
  • Cross-modal fine-tuning on 500K examples improves MSRVTT and TGIF, while 10k or 50k examples can hamper performance in the original setup.The experiments use one training epoch on downsampled WebVid-10M subsets.Source: §4.4 · p. 8, Table 8 · p. 8

5. Conclusions

R2A performs zero-shot VideoQA by using cross-modal retrieval to summarize videos in text, then answering questions with a pretrained language model. Experiments show state-of-the-art performance on multiple benchmarks, and components can be updated without extra training.

  • R2A achieves new state-of-the-art performance on multiple VideoQA benchmarks.Source: §5 · p. 8
  • R2A uses fast cross-modal retrieval from an external text corpus to summarize video content, then probes a pretrained language model with the retrieved text and question.Source: §5 · p. 8
  • R2A requires no task-specific training or explicit video-language alignment learning, and its components can be updated without extra training.Source: §5 · p. 8

6. Limitations

R2A’s performance depends on the retrieval model and the diversity of its text corpus. Some videos or text may remain difficult for CLIP, and existing data may not contain the desired answers.

  • R2A’s retrieved-caption quality depends heavily on retrieval-model performance and text-corpus diversity.Source: §6 · p. 8
  • Even CLIP may struggle with certain video or text types, constraining retrieval quality in those cases.Source: §6 · p. 8
  • Existing data may not contain desired answers, so retrieval cannot always find relevant content in the corpus.Source: §6 · p. 8
Loading 2306.11732v1…