Source-linked AI summary
VideoRAG: Retrieval-Augmented Generation over Video Corpus
Soyeong Jeong, Kangsan Kim, Jinheon Baek, Sung Ju Hwang
TL;DR
Existing RAG systems largely use text or images and either omit video retrieval or lose visual information by converting videos to text. VideoRAG dynamically retrieves videos and jointly uses their visual and textual content with LVLMs, while selecting informative frames and generating auxiliary text when subtitles are unavailable. The framework outperforms relevant baselines, although evaluation relies on constructed and synthetic datasets because suitable query-video-answer benchmarks are limited.
Problem
Existing RAG approaches primarily retrieve text or images, while video-based methods either assume relevant videos are known or discard multimodal information through textual conversion.
Method
VideoRAG dynamically retrieves query-relevant videos and jointly incorporates visual and textual information using LVLMs, adaptive frame selection, and auxiliary text extraction.
Results
VideoRAG achieves the best performance, significantly surpassing conventional textual, text-image, and text-video RAG baselines.
Takeaways & Limitations
VideoRAG demonstrates that video can serve as a useful external knowledge source for RAG by combining visual and textual features.
Takeaways & Limitations
Suitable query-video-answer datasets are limited, so evaluation uses datasets constructed by associating WikiHowQA with HowTo100M and automatically collecting synthetic data.
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) is a powerful strategy for improving the factual accuracy of models by retrieving external knowledge relevant to queries and incorporating it into the generation process. However, existing approaches primarily focus on text, with some recent advancements considering images, and they largely overlook videos, a rich source of multimodal knowledge capable of representing contextual details more effectively than any other modality. While very recent studies explore the use of videos in response generation, they either predefine query-associated videos without retrieval or convert videos into textual descriptions losing multimodal richness. To tackle these, we introduce VideoRAG, a framework that not only dynamically retrieves videos based on their relevance with queries but also utilizes both visual and textual information. The operation of VideoRAG is powered by recent Large Video Language Models (LVLMs), which enable the direct processing of video content to represent it for retrieval and the seamless integration of retrieved videos jointly with queries for response generation. Also, inspired by that the context size of LVLMs may not be sufficient to process all frames in extremely long videos and not all frames are equally important, we introduce a video frame selection mechanism to extract the most informative subset of frames, along with a strategy to extract textual information from videos (as it can aid the understanding of video content) when their subtitles are not available. We experimentally validate the effectiveness of VideoRAG, showcasing that it is superior to relevant baselines. Code is available at https://github.com/starsuzi/VideoRAG.
1 Introduction
Existing RAG systems mainly retrieve text and, more recently, images, while videos remain underused despite their temporal, spatial, and multimodal information. VideoRAG addresses this gap by dynamically retrieving videos and incorporating their visual and textual content into generation.
- Large vision-language models can generate factually incorrect outputs because their parametric knowledge may be inaccurate or outdated.
- Videos offer temporal dynamics, spatial details, and multimodal cues for representing complex processes, contextual interactions, and non-verbal signals.
- Prior video-query methods either assume relevant videos are already known or convert videos into text, discarding visual information such as temporal dynamics.
- Visual cues such as bared teeth, raised hackles, and narrowed eyes can be necessary for interpreting a dog's emotional state beyond barking or growling transcriptions.
- VideoRAG dynamically retrieves relevant videos and integrates their visual and textual elements into answer generation using Large Video Language Models.
- The framework introduces informative frame selection because long videos can exceed LVLM context capacity and contain redundant frames.
- Experiments using WikiHowQA and HowTo100M show significant improvement over relevant baselines, demonstrating the efficacy of leveraging videos for RAG.
2 Method
VideoRAG extends RAG to video corpora by retrieving query-relevant videos, jointly processing visual and textual information with LVLMs, and selecting informative frames. Its pipeline also reduces frame-search complexity and supplies auxiliary text when subtitles are unavailable.
- LVLMs jointly process video frames and associated text through vision and text encoders to form multimodal video representations.
- VideoRAG retrieves query-relevant videos from a large corpus by comparing query and video representations.
- Retrieved videos are concatenated with their textual data and the query before being forwarded to an LVLM for grounded response generation.
- Frame Selection: Adaptive frame selection extracts an informative, computationally feasible subset, using different selections for retrieval and query-guided generation.
- Frame Selection: k-means++ clustering reduces the frame-subset search space by selecting representative samples from frame groups.
- For videos without subtitles, audio is converted into auxiliary text usable during both retrieval and generation.
3 Experiment
VideoRAG is evaluated on automatically constructed video-RAG datasets using WikiHowQA and HowTo100M, with comparisons against naïve, text, text-image, and text-video baselines. Results show stronger performance from jointly leveraging retrieved videos and multimodal features, while adaptive frame selection, model scaling, and retrieval quality further affect outcomes.
- Datasets: VideoRAG is evaluated using WikiHowQA queries and HowTo100M videos, including automatically generated query-answer pairs for video-RAG assessment.The evaluation follows question-answering conventions for RAG and uses both existing and automatically collected data.
- Main Results: VideoRAG achieves the best performance among conventional textual, text-image, and text-video RAG baselines, while external knowledge improves over the NAÏVE baseline.The comparison includes BM25 and DPR text retrieval, text-image retrieval, and text-only video representations.
- Retrieval Quality: Oracle retrieval achieves the highest performance, indicating that improving video retrieval remains an opportunity for further gains.The oracle setting uses the ground-truth video associated with each query and is therefore not directly comparable to ordinary retrieval settings.
- Feature Analysis: Combining textual and visual features achieves the highest retrieval performance, with an optimal interpolation ratio around 0.5 to 0.7.Textual features alone generally outperform visual features, but their combination is complementary; a slight textual emphasis may help because of the modality gap.
- Frame Selection: Adaptive frame selection outperforms uniform sampling in both retrieval and generation by selecting more useful rather than redundant or less relevant frames.The advantage is reported quantitatively in Table 3 and qualitatively in retrieval and generation examples.
- Robustness and Scaling: VideoRAG performance improves with larger InternVL2.5 models and exceeds all baselines across 10 query categories except one.The category analysis highlights particularly notable gains for Food & Entertaining queries that benefit from visual details.
4 Related Work
Related work extends RAG beyond text to multimodal sources, while LVLMs provide a unified framework for incorporating video features. VideoRAG builds on these directions by treating videos as external knowledge.
- RAG retrieves relevant knowledge and uses it to generate answers grounded in external information.
- Multimodal RAG has expanded beyond text to images, code, tables, and audio.
- Videos offer temporal dynamics, spatial details, and multimodal cues that distinguish them from other RAG modalities.
- Large Video Language Models extend language models to process video content within a unified token-processing framework.
5 Conclusion
VideoRAG extends RAG by retrieving relevant videos and incorporating their visual and textual information, using adaptive frame selection for efficiency. Experiments support its superiority over existing RAG methods.
- VideoRAG uses a video corpus as external knowledge and retrieves videos relevant to each query.
- The framework integrates visual and textual video information into the RAG pipeline.
- Adaptive frame selection reduces full videos to informative subsets for more efficient and effective processing.
- Automatic speech recognition generates transcripts when textual video information is absent.
- Experiments validate VideoRAG’s superiority over existing RAG methods.
Limitations
VideoRAG’s evaluation depends on datasets linking queries, relevant videos, and ground-truth answers, but such benchmarks are currently limited. The authors construct synthetic evaluation data and identify benchmark development as future work.
- Evaluation of full video RAG requires triples containing queries, relevant videos, and ground-truth answers.
- Existing datasets containing these evaluation triples are limited.
- The authors associate WikiHowQA with HowTo100M and automatically collect synthetic data to enable evaluation.
- Developing and releasing a benchmark dataset remains future work.
Ethics Statement
VideoRAG’s retrieved corpus and LVLM-based generation can expose responses to biases in source videos and model training data. The authors recommend evaluating and mitigating these risks.
- Problematic examples in the retrieval corpus may lead to responses reflecting those issues.
- LVLM responses may inherit and amplify biases present in their multimodal training data.
- Practitioners should evaluate these risks and consider mitigation strategies such as bias detection and filtering.
A Additional Implementation Details
VideoRAG selects LVLMs according to the distinct demands of retrieval and generation. Retrieval requires precise text-video alignment, while generation requires strong response formulation grounded in retrieved content.
- Retrieval uses LVLMs whose specialized encoders precisely align textual and video features for contextually relevant video retrieval.
- Generation benefits from LVLMs capable of accurately formulating responses and grounding them in retrieved content.
A.2 Details on Synthetic Data Generation
The synthetic evaluation data is generated from individual videos using LVLM prompting, with questions designed to support retrieval across multiple relevant videos rather than target one video-specific detail.
- Synthetic question-answer pairs are automatically generated from individual videos by prompting LVLMs, alongside a real-world benchmark dataset.
- Questions are intentionally general enough to facilitate retrieval of multiple relevant videos instead of focusing on details unique to one video.
- A structured prompt instructs the LLM to create three diverse, well-formed question-answer pairs that leverage video content without being overly specific.
- GPT-4o is used as the LVLM for synthetic data creation.
A.3 Additional Details on Frame Selection
VideoRAG reduces video-frame search spaces differently for retrieval and generation, using clustering and adaptive frame scoring to balance efficiency with content coverage. Query-relevant retrieval improves generation quality over random video selection, while oracle retrieval performs best.
- Scoring: The frame-selection scoring function assigns scores to frame subsets for retrieval and generation and is trained using an automatically collected dataset.
- Retrieval: Retrieval selects 4 frames after 1-fps sampling, CLIP feature extraction, k-means++ clustering, and reduction to 8 candidate frames, yielding a search space of 8C4.
- Generation: Generation selects 32 frames from 64 clustered candidates and randomly samples 40 subsets because the full 64C32 combination space is too large.
- Video selection: Query-retrieved videos significantly improve answer quality over randomly selected videos, while the Oracle setting achieves the highest quality.
- Frame reduction: k-means++ is compared with random and uniform frame-reduction methods for producing diverse, representative frame subsets.
D Qualitative Results
Qualitative cases compare textual retrieval and alternative frame sampling with VideoRAG's adaptive selection. The examples show that video-based evidence can provide more relevant procedural guidance, while the reported tables organize comparisons across retrieval and generation settings.
- Video versus text retrieval: For procedural queries such as making a clay rose, TEXTRAG can retrieve an irrelevant document, whereas video-based retrieval supplies more relevant guidance.
- Comparisons: Tables 10 and 11 compare uniform sampling with the proposed frame selection on generation, while Table 12 compares TEXTRAG (BM25) with VIDEORAG-V.
- Acorn squash: Adaptive frame selection produces a 25.50 ROUGE-L score for the acorn-squash example, compared with 31.11 for the alternative generated answer.
- Ground truth: The qualitative section includes ground-truth procedures for cutting acorn squash and making coconut candy to contextualize generated answers.
- Coconut candy: For coconut candy, adaptive frame selection produces a 30.26 ROUGE-L score, compared with 33.55 for the alternative generated answer.
TEXTRAG (BM25)
For the clay-rose query, the retrieved text includes biographical material unrelated to the task alongside procedural answers. The generated procedures vary in detail, while the ground-truth passage specifies a spiral assembly process using seven pieces of clay.
- The retrieved document includes biographical information about Vally Wiselthier and Rose rather than instructions for making a clay rose.
- One generated answer recommends clay, a potter’s wheel, a wire cutter, a sponge, and optional decorative materials before shaping petals around a central clay ball.
- The guide cautions that its steps may vary with the clay type and techniques used and advises following safety guidelines when working with clay and kilns.
- Another generated answer rolls clay into a sheet, cuts petal shapes, spirals them into a rose, adds texture, and assembles them around a central stem.
- The ground-truth procedure divides a clay ball into seven pieces, forms guitar-pick-shaped petals, and attaches progressively larger petals in a spiral.