Source-linked AI summary
Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
Yongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, Rongrong Ji
TL;DR
Long-video comprehension remains difficult for current LVLMs because of limited context. Video-RAG retrieves visually aligned auxiliary texts from video and adds them to any LVLM, improving performance across long-video benchmarks and reaching proprietary-level performance with a 72B open-source model.
Problem
Current LVLMs perform promisingly on short videos but still struggle to comprehend extremely long videos effectively.
Method
Video-RAG retrieves OCR, ASR, and object-detection texts extracted with open-source tools and integrates them with video frames and queries in any LVLM.
Results
Video-RAG delivers consistent gains across long-video benchmarks, including a 2.8% average Video-MME improvement and proprietary-level performance with a 72B open-source LVLM.
Takeaways & Limitations
Video-RAG provides a resource-efficient, training-free, plug-and-play route to stronger long-video understanding using open-source tools.
Takeaways & Limitations
Increasing sampled frames can introduce information redundancy and additional challenges for complex reasoning.
Abstract
from arXiv · showhide
Existing large video-language models (LVLMs) struggle to comprehend long videos correctly due to limited context. To address this problem, fine-tuning long-context LVLMs and employing GPT-based agents have emerged as promising solutions. However, fine-tuning LVLMs would require extensive high-quality data and substantial GPU resources, while GPT-based agents would rely on proprietary models (e.g., GPT-4o). In this paper, we propose Video Retrieval-Augmented Generation (Video-RAG), a training-free and cost-effective pipeline that employs visually-aligned auxiliary texts to help facilitate cross-modality alignment while providing additional information beyond the visual content. Specifically, we leverage open-source external tools to extract visually-aligned information from pure video data (e.g., audio, optical character, and object detection), and incorporate the extracted information into an existing LVLM as auxiliary texts, alongside video frames and queries, in a plug-and-play manner. Our Video-RAG offers several key advantages: (i) lightweight with low computing overhead due to single-turn retrieval; (ii) easy implementation and compatibility with any LVLM; and (iii) significant, consistent performance gains across long video understanding benchmarks, including Video-MME, MLVU, and LongVideoBench. Notably, our model demonstrates superior performance over proprietary models like Gemini-1.5-Pro and GPT-4o when utilized with a 72B model.
1 Introduction
Video-RAG addresses the difficulty of comprehending extremely long videos with a training-free, plug-and-play pipeline that retrieves visually aligned auxiliary texts for LVLMs. It improves long-video understanding across benchmarks while maintaining low additional resource requirements and achieving proprietary-level performance with an open-source 72B model.
- Motivation: Current LVLMs perform promisingly on short videos but still struggle with extremely long videos, while increasing context length requires extensive pretraining and can introduce deployment distribution shifts.Simply sampling more frames also creates information redundancy and makes complex reasoning harder.
- Results: Video-RAG achieves a 2.8% average improvement on Video-MME across seven open-source LVLMs with only 2.0K additional text tokens per case, while beating proprietary LVLMs with a 72B model.The comparison covers Video-MME, MLVU, and LongVideoBench, with the proprietary-model comparison shown in Figure 1.
- Method: Video-RAG integrates OCR, ASR, and object-detection texts extracted by open-source tools and retrieved alongside video inputs to enhance any LVLM without training.The pipeline replaces some extended visual tokens with auxiliary texts that provide information beyond sampled frames.
- Efficiency: Applying Video-RAG to a 7B LVLM requires only 8GB of additional inference GPU memory and approximately 5 seconds of inference time per case.These costs are reported for inference and demonstrate the pipeline’s lightweight design.
2 Related Work
Related work advances video-language understanding through frame-level modeling, longer context windows, and tool-assisted extraction of structured multimodal information. These approaches motivate Video-RAG’s use of auxiliary textual representations for long-video comprehension.
- Generalist Video Models: Generalist video models combine frame features, temporal or spatial pooling, textual descriptions, and aligned image-video encoders for video understanding.Video-ChatGPT aggregates individual-frame features; VideoChat uses textual descriptions and appearance embeddings; Video-LLaVA aligns image and video encoders during preprocessing.
- Long-Context Video Understanding: Long-video methods expand context through continued training on extended text or training-free video-token rearrangement, but face challenges in processing long visual sequences.LongVA and Long-LLaVA transfer long-text comprehension to video, while INTP rearranges video tokens without training.
- Tool-Assisted Video Understanding: Tool-assisted approaches use LLMs and multimodal models to convert video frames, visual content, and audio into structured text or documents for question answering.MM-VID aligns frames with text descriptions, VLog summarizes visual and audio information, and other systems integrate multimodal inputs through tool interaction.
3 Method
VideoRAG is a training-free, plug-and-play pipeline for integrating visually aligned auxiliary texts into any LVLM. It decouples user queries into retrieval requests, generates and retrieves auxiliary information from video, and combines that information with sampled frames and the query for final generation.
- Overview: VideoRAG provides a training-free pipeline that can integrate with any LVLM through query decoupling, auxiliary-text generation and retrieval, and unified final inference.The pipeline is designed as a lightweight alternative that adds retrieved information without modifying the LVLM.
- Video Representation: The frame-based baseline samples N video frames, extracts visual features with an image encoder such as CLIP-L, and feeds them with the query into the LVLM.This establishes the visual input used alongside VideoRAG’s auxiliary texts.
- Query Decouple: During query decoupling, the LVLM processes only the textual query and emits JSON-formatted retrieval requests for auxiliary information such as speech recognition.The requests guide which auxiliary databases and information types are queried later.
- Auxiliary Text Generation & Retrieval: VideoRAG constructs OCR, ASR, and detection databases using EasyOCR, Whisper, and visual grounding, respectively, to supplement visual cues with text, audio, and object information.ASR transcripts are chunked and encoded, while detection captures object categories and positions for retrieval.
- Auxiliary Text Generation & Retrieval: Detection auxiliary texts represent object location, counting, and relative position information, making spatial relationships more interpretable to LVLMs.The object information is reformatted through scene-graph preprocessing before retrieval.
- Final Inference: The retrieved OCR, ASR, and detection texts are chronologically concatenated with the user query and sampled frames before final LVLM generation.The unified auxiliary input is denoted as Am = Concat(Aocr, Aasr, Adet).
4 Experiments
Video-RAG consistently improves long-video understanding across Video-MME, MLVU, and LongVideoBench while adding only lightweight auxiliary text. Gains extend from 7B to 72B models and include state-of-the-art or competitive performance against proprietary systems.
- Video-MME: 2.8% average improvement is obtained across seven LVLMs on Video-MME using approximately 2.0K auxiliary texts per sample versus approximately 3.0K full-subtitled tokens.Video-MME spans videos from 11 seconds to 1 hour, while the evaluation covers five 7B and two 72B open-source LVLMs.
- MLVU: Video-RAG improves performance across Video-MME, MLVU, and LongVideoBench, including a 1.6% gain at 7B and a 0.7% gain at 72B on MLVU, with the latter setting a new state-of-the-art.On MLVU, the 7B result also exceeds the 32B Qryx-1.5 by 0.1%.
- LongVideoBench: 65.4% on LongVideoBench is achieved by 72B LLaVA-Video with Video-RAG, surpassing Gemini-1.5-Pro by 1.4% and reaching second place, 1.3% behind GPT-4o.The result is reported on the validation set without LongVideoBench’s interleaved input format.
- Ablation and qualitative analysis: Video-RAG consistently improves performance across sampling rates, with larger gains possible at fewer frames, while a threshold of t = 0.3 balances performance, information density, and processing time.The selected configuration typically adds approximately 1.9K tokens, and qualitative analyses show reduced visual hallucinations and improved cross-modality alignment.
5 Conclusion
Video-RAG integrates retrieved, visually aligned auxiliary texts with LVLMs for effective long-video understanding, achieving proprietary-level performance with a 72B open-source LVLM. It is resource-efficient and plug-and-play, but may be limited by the selected visual tools.
- 5 Conclusion: Video-RAG achieves proprietary-level performance with a 72B open-source LVLM for long-video understanding.
- 5 Conclusion: Video-RAG provides a resource-efficient, plug-and-play solution by integrating retrieved auxiliary texts extracted with open-source tools.
- 5 Conclusion: Video-RAG’s effectiveness may be limited by the visual tools selected for extracting auxiliary information.
Supplemental Material · A Decouple Query
The initial Video-RAG phase uses a decoupled prompt to guide the LVLM in generating retrieval requests, with a multiple-choice example provided.
- A Decouple Query: Video-RAG uses a decoupled prompt P to guide LVLM retrieval-request generation, illustrated by a multiple-choice prompt example in Figure 8.The example is presented as supplemental material for multiple-choice questions.
B Sub-set of Video-MME
The study evaluates a computationally intensive agent-based method on a representative 10% Video-MME subset, verifying that its method ranking matches the full dataset.
- B Sub-set of Video-MME: The subset contains 90 videos—30 each of short, medium-length, and long duration—with three multiple-choice questions per video.It represents 10% of the full Video-MME dataset.
- B Sub-set of Video-MME: Method performance rankings on the subset mirror those on the full Video-MME dataset.This supports using the subset to evaluate the computationally resource-intensive agent-based method with long-context LVLMs.
C Results on Video-MME Sub-Set
On a Video-MME subset, the study compares Video-RAG with VideoAgent and LongVA on inference time, GPU requirements, and overall performance, highlighting VideoAgent’s costly and suboptimal processing.
- Results on Video-MME Sub-Set: VideoAgent requires substantial processing time while delivering suboptimal performance on the Video-MME subset.The comparison uses a GPT-based agent as a representative resource-intensive method.
- Results on Video-MME Sub-Set: The evaluation compares Video-RAG with VideoAgent and LongVA using overall performance, database-construction cost, and average per-query inference time.The reported inference-time measure includes database construction.
- Results on Video-MME Sub-Set: Figure 6 summarizes the comparison, with bubble sizes representing total inference time on the Video-MME subset.The subset was sampled because GPT-based agent methods are resource-intensive.
D Details of Similarity Score Calculation
Video-RAG retrieves auxiliary OCR and ASR texts with FAISS similarity and selects keyframes with CLIP similarity using a threshold of 0.3. Object-detection requests are linguistically filtered before CLIP-based frame scoring and thresholded keyframe selection.
- D Details of Similarity Score Calculation: The system sets the similarity threshold t to 0.3, using FAISS for OCR/ASR retrieval and CLIP for keyframe selection.FAISS efficiently searches auxiliary texts, while CLIP measures relevance between video frames and object-detection requests.
- D Details of Similarity Score Calculation: Object-detection requests are filtered with part-of-speech rules and Spacy, then prefixed with “A picture of” before CLIP matching.Single-word requests undergo direct filtering, while compound words are checked for adjective–noun or noun–noun patterns.
- D Details of Similarity Score Calculation: For each frame, CLIP computes image–text dot-product similarities across detection requests and averages them into a frame similarity vector.The resulting vector contains one mean similarity score per frame across all text descriptions.
- D Details of Similarity Score Calculation: The frame scores are scaled for different LVLM video sampling rates, normalized to sum to 1, and thresholded to select keyframes.The scaling uses the number of frames m and a base frame number b, with b set to 16 and 4.0, respectively, as reported.
E More Ablation Studies
The ablations show that Video-RAG’s auxiliary modalities provide complementary benefits across tasks, while combining all components yields the strongest results across MLVU and LongVideoBench at both model scales. A further VNBench study isolates DET and OCR for synthetic long-context retrieval, ordering, and counting challenges.
- Component analysis: Object detection improves spatial perception and object counting, OCR improves text recognition, and ASR generally improves inference tasks across Video-MME subtasks.These findings demonstrate complementary roles for the three auxiliary-text sources.
- Cross-benchmark ablation: Combining all Video-RAG components achieves optimal performance on both MLVU and LongVideoBench for 7B and 72B models.Without the RAG system, MLVU performance declines.
- VNBench analysis: A VNBench experiment with Long-LLaVA-7B evaluates DET and OCR on synthetic retrieval, ordering, and counting tasks involving irrelevant inserted stickers or text.VNBench is designed to test long-context needle-in-a-haystack ability.
F More Qualitative Results
Figure 9 presents qualitative Video-MME examples showing that Video-RAG’s retrieved auxiliary texts enhance LLaVA-Video-7B’s detailed video comprehension. The augmentation reduces visual hallucinations and enables more accurate, confident responses.
- More Qualitative Results: Video-RAG significantly reduces visual hallucinations in representative Video-MME examples, enabling LLaVA-Video-7B to answer user queries more accurately and confidently.The examples apply external tools to process and retrieve auxiliary texts from videos alongside LLaVA-Video.