PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking RAG in Long Videos: What to Retrieve and How to Use It?

Yuho Lee, Jisu Shin, Nicole Hee-Yeon Kim, Jihwan Bang, Juntae Lee, Kyuwoong Hwang, Fatih Porikli, Hwanjun Song

arXiv:2606.13141v1cs.AI

TL;DR

VideoRAG lacks benchmarks that separately measure retrieval and generation, while existing methods use one modality–granularity configuration per query. The paper introduces V-RAGBench and CARVE, whose chunk-level configuration selection outperforms eight recent baselines on both stages.

  • Problem

    Existing VideoRAG benchmarks do not separately evaluate retrieval and generation, while prior methods use one modality–granularity choice per query.

  • Method

    V-RAGBench provides query–evidence–answer triplets, while CARVE parallelizes four configuration-specific retrievers and adaptively reranks chunks.

  • Results

    CARVE outperforms eight recent VideoRAG baselines on both retrieval and generation, while surpassing trained query-level routers without additional training.

  • Takeaways & Limitations

    VideoRAG configuration selection can vary by chunk, with each winning configuration carried from retrieval into generation.

  • Takeaways & Limitations

    CARVE focuses on content-based representations because other retrieval signals are not consistently available.

Abstract

from arXiv · show

Retrieval-augmented generation is moving beyond text into long, egocentric video, where systems must select query-relevant chunks across multiple modalities and temporal granularities. Yet progress in VideoRAG is limited by two gaps: existing benchmarks allow queries to be answered without the video, obscuring retrieval errors, and prior methods apply a single modality-granularity configuration per query, ignoring chunk-level variability. We address both by introducing V-RAGBench, a benchmark of $\langle$query, evidence chunk, answer$\rangle$ triplets that enables faithful, decoupled evaluation of retrieval and generation, and CARVE, a simple method that runs parallel retrievers across configurations and employs chunk-adaptive reranking to identify the winning configuration for each chunk. Each chunk then enters the generator under its winning configuration selected during retrieval, yielding an interleaved evidence form where the chunk-level decision propagates across both stages. CARVE outperforms eight recent VideoRAG baselines, with the chunks supplied to the generator interleaving multiple configurations rather than sharing a single one, a behavior unattainable by query-level methods.

1 Introduction

VideoRAG lacks benchmarks that isolate retrieval from generation, while existing methods typically commit to one modality–granularity configuration per query. V-RAGBench enables faithful stage-wise evaluation, and CARVE uses chunk-adaptive selection across four configurations for retrieval and generation.

  • Motivation: VideoRAG benchmarks can obscure retrieval errors because over half of widely used video QA samples can be answered without the video.Such queries may rely on linguistic priors, world knowledge, or static cues.
  • Motivation: Retrieval is central to VideoRAG, yet prior work mainly validates it through downstream QA accuracy rather than stage-wise query–evidence metrics.This contrasts with text-centric RAG, where retrieval is audited as a first-class object of study.
  • V-RAGBench: V-RAGBench contains 2,100 query–evidence chunk–answer triplets from 216 Ego4D and EgoLife videos spanning 1–9 hours.The benchmark is designed to measure retrieval and generation both in isolation and in their interaction.
  • CARVE: Each video chunk has four candidate configurations formed by crossing visual or textual-summary modality with frame- or clip-level temporal granularity.Existing VideoRAG retrieval strategies apply a single configuration rather than adapting to chunk-level variability.
  • CARVE: CARVE runs four parallel retrievers, pools their top-k chunks, and reranks candidates under their retrieving configurations before passing interleaved evidence to the generator.A multimodal cross-encoder uses configuration-specific query–text or query-visual relevance during reranking.
  • Results: CARVE significantly outperforms eight baseline methods on both retrieval and generation, while ablations find no single modality–granularity configuration uniformly optimal.These findings motivate chunk-level rather than query-level configuration decisions.

2 Related Work

Related work shows that VideoRAG must retrieve multimodal, temporally granular evidence rather than rely on query-independent whole-video representations. Existing methods and benchmarks nevertheless largely fix modality–granularity choices per query and do not cleanly separate retrieval from generation, motivating chunk-level evaluation and selection.

  • Retrieval-Augmented Generation in Text: Text RAG improves generation by retrieving query-relevant evidence, with progress spanning sparse, dense, hybrid, and query-rewriting approaches.Retrieval quality largely dictates final-answer quality.
  • Retrieval-Augmented Generation in Long Video: VideoRAG retrieves query-relevant evidence because query-independent representations dilute relevant content as video length increases.Prior long-video systems use sparse frame sampling, visual-token compression, or full-video textualization.
  • Retrieval-Augmented Generation in Long Video: Video chunks require joint handling of continuous time, multiple modalities, and granularities ranging from individual frames to segments.Existing work explores modality and temporal-granularity axes from complementary angles.
  • VideoRAG Limitations: Existing VideoRAG methods commit to one modality–granularity choice or fixed fusion per query, while evaluation emphasizes downstream QA accuracy over retrieval quality.These limitations motivate treating modality and granularity as explicit chunk-level design axes.
  • Video Benchmark Datasets for QA: Video QA benchmarks have shifted from short third-person clips toward longer, increasingly egocentric videos, matching VideoRAG’s personal-video retrieval setting.Wearable devices accumulate hour-scale first-person video, and personal agents increasingly operate over it.
  • Video Benchmark Datasets for QA: Existing benchmarks do not target VideoRAG evaluation, and the proposed benchmark jointly enforces all three required properties on hour-scale egocentric video.Ego4D and EgoLife provide hour-scale source video but have scarce or differently targeted queries; CG-Bench and MM-Lifelong address only part of the gap.

3 Formulating VideoRAG along Two Design Axes

VideoRAG is formulated as retrieve-then-generate over fixed temporal chunks whose representations vary along modality and granularity. The formulation motivates choosing these representations per chunk and carrying each choice through retrieval and generation.

  • Chunking: VideoRAG partitions a long video into non-overlapping, fixed-interval chunks, defaulting to 2-minute retrieval units.The 2-minute default matches the reported empirical 2–3 minute span of bounded semantic content in long-form video.
  • Two design axes: Each video chunk has representations defined by modality m ∈ {vis, text} and granularity g ∈ {frame, clip}, yielding four configurations.The visual and textual modalities provide complementary channels, while frame and clip granularities expose fine-grained and whole-chunk temporal scopes.
  • Retrieve-then-generate: Retrieval and generation independently choose configurations (m_r, g_r) and (m_g, g_g), and these choices need not coincide.The objective is for retrieved chunks to support the answer and generated output to match the reference answer.
  • Chunk-level decisions: Query-level methods fix one configuration uniformly across chunks, whereas this formulation decides the representation per chunk for both retrieval and generation.Agentic variants may issue multiple retrievals per query, but each call still commits to a single configuration.
  • Chunk-level decisions: Frame-level visual embeddings suit objects visible in one keyframe, while clip-level textual summaries better express activities unfolding across a chunk.The proposed approach carries each chunk’s configuration consistently into retrieval and generation while retaining the standard pipeline.

4 Construction of V-RAGBench

V-RAGBench is constructed as a faithful benchmark of ⟨query, evidence chunk, answer⟩ triplets, enabling retrieval and generation to be evaluated separately and jointly through uniquely sufficient evidence chunks. Its pipeline selects egocentric sources, extracts and deduplicates event chunks, generates localized queries, applies post-hoc filters, and produces a balanced 2,100-query dataset.

  • Benchmark design: V-RAGBench comprises ⟨query, evidence chunk, answer⟩ triplets designed to make retrieval and generation measurable in isolation and together.Each query is tied to a uniquely sufficient evidence chunk.
  • Source Video Selection: The benchmark draws source videos from the Ego4D and EgoLife egocentric corpora, targeting continuous, weakly structured daily activity.Such first-person streams involve recurring scenes and routines, making personal-memory retrieval more realistic and harder.
  • Event Extraction and Deduplication: Each video is converted into distinct, non-redundant event chunks by encoding sampled frames with EVA02-E-14-plus and applying kernel temporal segmentation.The resulting segments are semantically coherent and time-contiguous, with corpus-specific caps described for Ego4D and EgoLife.
  • Query Generation: Gemini-3-flash-preview generates up to three candidate queries per event chunk across three categories, while requiring each query to include source-event anchor information.The localization constraint makes the query alone descriptive enough to match the correct chunk.
  • Post-hoc Filtering: Five post-hoc filters enforce query distinctness, answerability, visual grounding, evidence localization, and freedom from shortcut bias.Filtering removes semantically redundant, unanswerable, shortcut-solvable, or insufficiently localized queries.
  • Post-hoc Filtering: 2,100 queries form the balanced dataset after category subsampling, with 700 queries per category, 1,800 for training, and 300 for testing.The categories are subsampled to match the smallest surviving category.

5 CARVE: Chunk-Aware Reranking for Video Evidence

CARVE selects modality–granularity configurations per chunk through parallel retrieval and chunk-adaptive reranking, then propagates each winning configuration into generation. This produces interleaved evidence representations and emphasizes retrieval-to-generation propagation as its strongest improvement.

  • Method overview: CARVE makes modality–granularity decisions at the chunk level and propagates each decision consistently from retrieval into generation.It uses parallel candidate pooling followed by chunk-adaptive reranking.
  • Parallel Candidate Pooling: Four parallel retrievals cover visual or textual modalities at frame or clip granularity, forming a configuration-tagged candidate pool.Chunk embeddings are precomputed offline, while inference performs query encoding and nearest-neighbor lookup.
  • Chunk-Adaptive Reranking: A multimodal cross-encoder reranks tagged candidates, retains each chunk’s highest-scoring tag, and ranks chunks together by comparable winning scores.Each chunk is rescored only under the configuration that retrieved it, rather than under all four configurations.
  • Modality-Interleaved Generation: The final generator context interleaves chunks rendered under their winning configurations instead of using one query-level configuration or uniform representation fusion.The paper identifies propagation from retrieval to generation—not retrieval gain alone—as CARVE’s strongest improvement.

6 Evaluation on V-RAGBench

Evaluation on V-RAGBench shows that CARVE improves both retrieval and generation over VideoRAG baselines while selecting modality–granularity configurations at the chunk level. Its interleaved representations provide genuine chunk-level diversity with favorable latency.

  • Evaluation Setup: CARVE’s V-RAGBench evaluation covers retrieval and generation comparisons, configuration-selection ablations, and analyses of latency and chunk-level decisions.Retrieval and generation are evaluated separately using nDCG@k, Recall@k, and aggregate pass rate.
  • Overall Results: 0.603 Recall@5 and 0.433 nDCG@5 are CARVE’s best retrieval results, exceeding VideoRAG-A’s 0.510 Recall@5 and GQR’s 0.340 nDCG@5.The retrieval gains consistently translate into stronger generation across all three generator backbones.
  • Efficiency: 4.6s per query is CARVE’s full-setting latency, modestly above text-only configurations but faster than visual-only configurations.CARVE passes visual evidence only for chunks whose winning configurations benefit from it.
  • Chunk-Level Analysis: All four configurations win with non-trivial frequency at every top-5 rank, confirming that CARVE’s interleaving does not collapse onto one dominant configuration.{vis, clip} and {text, frame} win most often, while {text, clip} and {vis, frame} contribute consistently.
  • Reranking Analysis: 3.4s is CARVE’s reranking latency, while visual-only configurations take 4.1s and 8.0s and Concatenation takes 8.2s.CARVE retains the best retrieval performance by passing each chunk only under its winning configuration; the same trend holds with LamRA-Rank-7B.

7 Conclusion

The paper addresses two gaps in VideoRAG by introducing V-RAGBench for decoupled retrieval-generation evaluation and CARVE for chunk-level modality and temporal-granularity selection. V-RAGBench uses query–evidence chunk–answer triplets, while CARVE combines parallel candidate pooling with chunk-adaptive reranking.

  • Benchmark: V-RAGBench is a collection of query–evidence chunk–answer triplets over hour-scale egocentric videos.The benchmark supports evaluation across long, egocentric video content.
  • Benchmark: The benchmark jointly enforces non-recurring evidence, visual grounding, and evidence localization for decoupled retrieval and generation evaluation.These constraints target separate assessment of whether evidence is retrieved and whether answers are generated from it.
  • Method: CARVE reframes modality and temporal granularity as chunk-level decisions using parallel candidate pooling and chunk-adaptive reranking.This methodological design addresses configuration variability at the chunk level rather than treating it as a single query-level choice.

A Limitations … C.4 Event selection and de-duplication details

The paper identifies scope, pipeline, and evaluation-domain limitations, while detailing V-RAGBench’s construction, coverage, and event-chunk deduplication. The benchmark combines egocentric sources into uniquely grounded queries whose retained evidence remains contiguous and less redundant.

  • A Limitations: CARVE evaluates four modality–granularity configurations, retains a one-shot retrieve-then-generate pipeline, and focuses V-RAGBench on egocentric video rather than instructional, film, or broadcast domains.The configurations span visual versus textual modality and frame versus clip granularity; iterative or agentic retrieval is left for future work.
  • B Societal Impact: Egocentric VideoRAG deployment should address privacy, consent, and responsible use because personal daily-life activities are inherently captured.The paper reports no specific societal risks beyond those generally associated with multimodal retrieval and video understanding research.
  • C Dataset: 2,100 queries grounded in 216 uncut source videos form V-RAGBench, constructed from complementary EgoLife daily-life recordings and Ego4D activity-centric videos.The benchmark’s pipeline selects source videos, extracts and deduplicates events, generates candidate questions and answers, and filters for redundancy, answerability, shortcut bias, empirical answerability, and evidence uniqueness.
  • C.2 Definitions of the source video domains: Ego4D coverage is organized into Domestic, Social, Stationary, Manual, and Outdoor domains, while EgoLife contributes uniformly daily-life videos.Outdoor is held out for testing, and EgoLife test days are non-overlapping across participants to prevent leakage, enabling in-distribution and out-of-distribution evaluation.
  • C.3 Definitions of the query categories: The benchmark’s queries target Action Flow & Movement, Object-Centric Visual Understanding, and Interaction reasoning capabilities.These categories respectively probe temporal and sequential reasoning, fine-grained object attributes and states, and physical manipulation or social engagement.
  • C.4 Event selection and de-duplication details: Temporal segmentation defines contiguous event candidates, whereas semantic deduplication selects among them without changing temporal boundaries, so every retained chunk remains an original contiguous interval.This separates candidate formation from representative selection in the event-construction process.
  • C.4 Event selection and de-duplication details: Deduplication makes the final evidence pool smaller and less redundant by representing semantically similar segment groups with selected chunks.Repeated daily routines are counted through representatives, reducing the chance that a query can be answered from multiple near-duplicate events in one video.
  • C.4 Event selection and de-duplication details: Retained representatives are not fixed-length, so deduplicated event chunks span a range of durations reported in Tables 9 and 10.The reported distributions cover both descriptive dataset statistics and the final subset associated with 2,100 queries.

C.5 QA Filter Details

The QA filtering pipeline reduces 67,370 initial candidates to 5,907 retained queries by enforcing semantic distinctness, evidence-based answerability, shortcut resistance, empirical solvability, and evidence uniqueness. Human verification supports the retained set’s reliability, with 96.9% judged answerable by both annotators and substantial agreement.

  • Filtering pipeline: Only 5,907 of 67,370 candidates survive all five filtering stages, retaining 8.77% of the original pool.The stages enforce semantic non-redundancy, answerability from annotated evidence, shortcut-bias resistance, empirical answerability by a strong VLLM, and non-recoverability from non-evidence chunks.
  • Filtering pipeline: Query-level semantic filtering removes near-duplicate questions by comparing query embeddings against already retained queries.This prevents repeated phrasings of similar actions from inflating the benchmark.
  • Filtering pipeline: The answerability filters discard candidates unanswerable from their source clip or incorrectly answered by Gemini 2.5flash using only the source evidence chunk.GPT-5.2-chat evaluates visual answerability, while Gemini 2.5flash is assessed with a binary LLM-as-a-judge.
  • Filtering pipeline: Shortcut-biased queries are removed when GPT-5.2-chat produces the reference answer from the query alone without visual context.The filter targets answers recoverable from language priors, commonsense knowledge, or contextual cues rather than video evidence.
  • Filtering pipeline: Evidence uniqueness is tested by retrieving top-10 non-evidence 2-minute chunks with Qwen3-VL-8B and removing queries answerable from those chunks.This ensures the annotated evidence is uniquely sufficient within the source video.
  • Human verification: 96.9% of 577 sampled QA pairs were judged answerable by both annotators, with Cohen’s Kappa of 0.63 indicating substantial agreement.The sample corresponds to a 95% confidence interval with a margin of ±4 over the full set of 2,100 queries.

D Implementation details · D.1 CARVE Implementations.

CARVE’s implementation uses indexed multi-modal representations built from 120-second video segments, then retrieves and reranks five chunks before generation. Experiments use specified open-source and proprietary models, H200 GPUs, and a consistent binary judging setup.

  • D Implementation details: The experiments use the open-source models listed in Table 13 and the proprietary models summarized in Table 14.These tables document the pretrained models and proprietary APIs used in the pipeline.
  • D Implementation details: The total API cost is approximately $1,200, including QA generation and QA filtering.The cost covers both stages of the stated quality-assurance process.
  • D Implementation details: All experiments run on 8× NVIDIA H200 GPUs with 141 GB of memory using CUDA 13.0.The reported hardware and software configuration applies to all experiments.
  • D.1 CARVE Implementations.: Videos are split into non-overlapping 120-second segments, with four identically indexed representations constructed for each segment.The representations include visual and textual views at keyframe and per-second temporal granularities.
  • D.1 CARVE Implementations.: The VF representation encodes 5 keyframes per segment, while VC encodes up to 64 frames sampled at 1 fps.VC frames have a maximum resolution of 1,843,200 pixels per frame.
  • D.1 CARVE Implementations.: TF captions the 5 keyframes with Qwen3-VL-8B-Instruct using up to 512 tokens, while TC captions per-second frames sampled at 1 fps.Both textual views are built from the segment-level visual inputs described in the implementation.
  • D.1 CARVE Implementations.: At inference, the system retrieves top-K=5 chunks per query and reranks them with the default Qwen3-VL-Reranker-2B.The retrieved chunks are then passed to Qwen3-VL-8B-Instruct for answer generation.
  • D.1 CARVE Implementations.: Generation uses Qwen3-VL-8B-Instruct with a maximum of 512 new tokens under its default decoding setup.Every generated answer is evaluated against its reference by a binary Qwen3.6-35B-A3B judge using identical weights and prompts across methods and baselines.

D.2 Baseline Implementations … F.1 Baseline with Reranking

The paper specifies diverse system-, query-, and router-level baselines, separates retrieval from generation evaluation, and details CARVE’s validation and additional analyses. Reranking improves baselines only marginally, while CARVE significantly outperforms reranked alternatives, indicating its advantage is not merely standalone reranking.

  • D.2 Baseline Implementations: System-level baselines combine textual and visual representations, optimize query embeddings, or repurpose an MLLM for retrieval.VideoRAG-A uses α = 0.5, GQR uses text and video retrievers with learning rate 5e−4, and FreeRet uses Qwen2-VL-2B-Instruct.
  • D.2 Baseline Implementations: Query-level baselines decompose queries into memory-specific sub-queries or modality-targeted paraphrases before merging retrieved results.VideoRAG-B uses one visual and two text retrievers, while RRF targets text-clip and text-frame retrieval.
  • D.2 Baseline Implementations: Router comparisons assess query-level systems that select one configuration per query against CARVE’s chunk-level configuration decisions.The LLM-based Router fine-tunes Qwen3-VL-2B-Instruct as a four-configuration multi-label classifier using LoRA with r = 32 for 5 epochs at 2e−5 learning rate.
  • D.3 Evaluation Metrics: Retrieval and generation are evaluated separately using Recall@K, nDCG@K, and pass rate, with K = 5 in the main experiments.Because each query has a uniquely sufficient evidence chunk, Recall@K checks whether that chunk appears among the top-K results; nDCG@K also accounts for ranking.
  • D.3 Evaluation Metrics: The evaluation also reports online latency, distinguishing retrieval-only costs from end-to-end costs that include generation.Latency may include query encoding, retrieval, reranking, evidence rendering, and generation, while excluding offline preprocessing.
  • D.4 LLM-as-a-Judge Details: Open-ended answer correctness uses Qwen3.6-35B-A3B as the LLM-as-a-judge backbone, with human validation on 600 of 2,100 predictions.The sampled validation corresponds to a 95% confidence interval with a ±4 margin of error.
  • E CARVE Algorithm: Algorithm 1 formalizes CARVE, and additional analyses examine reranked baselines, source-video robustness, and generation errors.The analyses target configuration-selection effects, performance across different video scales, and generation behavior.
  • F.1 Baseline with Reranking: Reranking generally improves baselines only marginally, yet CARVE significantly outperforms reranked baselines, so its advantage is not merely the reranking mechanism.All baselines receive the same reranker used by CARVE to isolate the contribution of chunk-level configuration selection.

F.2 Performance Breakdown by Source Dataset

CARVE is evaluated across Ego4D’s hour-scale videos and EgoLife’s more challenging day-long recordings, outperforming all baselines on both datasets. Its advantage is especially pronounced on EgoLife, where CARVE reaches 0.394 nDCG@5 versus 0.302 for the strongest baseline.

  • Dataset scale: Ego4D contains videos exceeding one hour, while EgoLife extends the retrieval challenge to day-long recordings.The source-dataset breakdown examines CARVE across these differing video scales.
  • Overall performance: CARVE outperforms all baselines on both EgoLife and Ego4D.The performance gap is reported as more pronounced for EgoLife’s super-long recordings.
  • EgoLife results: 0.394 nDCG@5 is achieved by CARVE on EgoLife, versus 0.302 nDCG@5 for the strongest baseline.This wider margin on EgoLife exceeds CARVE’s relative advantage on hour-scale Ego4D clips.

F.3 Error Analysis on Retrieval and Generation … G Prompts

The analysis separates retrieval failures from generation failures, shows CARVE remains robust across retrieval depths, and documents the prompts used for QA creation, filtering, memory generation, and answering. Together, these sections clarify both system behavior and the procedures used to construct and evaluate video-grounded QA.

  • F.3 Error Analysis on Retrieval and Generation: 92 Case 1 and 108 Case 2 failures occurred with Qwen3-VL-8B, decreasing to 86 and 104 with Qwen3-VL-32B.Case 1 denotes retrieval and generation failure; Case 2 denotes successful retrieval followed by generation failure.
  • F.4 Effect of Top-K: Across tested k values, CARVE remains stable and competitive, indicating gains come from better-ranked evidence rather than a larger retrieval budget.Increasing k can improve recall but may also introduce distractor chunks that make generation harder.
  • F.5 Reranker Backbone Robustness: Table 18 evaluates retrieval performance for each representation configuration using the LamRA-Rank-7B reranker backbone.The cited passage identifies the table’s scope but provides no numerical results.
  • G.1 QA Generation Prompt: QA-generation prompts anchor each question to a unique observable event in egocentric video frames.The prompt instructs the model to ground up to four QA pairs strictly in visible evidence and avoid prior knowledge or common sense.
  • G.2 QA Filtering Prompt: Filtering prompts verify frame-based answerability and correctness, then check whether questions can be answered from common sense without visual input.These procedures are represented by the GPT Answerability Filtering and GPT Blind Check Filtering prompts.
  • G.3 LLM Judge Prompt: The LLM-judge prompt evaluates whether a candidate answer matches the reference answer’s key factual content.This evaluation procedure is represented in Figure 8.
  • G.4 Memory Generation Prompt: Memory-generation prompts produce first-person text summaries for video clips and detailed descriptions of visual experience at specific keyframes.The prompts emphasize what the camera wearer sees, does, and experiences.
  • G.5 QA Answering Prompt: Answering prompts cover retrieved text memories, combined text memories, visual clips, keyframes, and multimodal combinations.Figures 11–15 document the corresponding text-based, visual, and combined answering configurations.
Loading 2606.13141v1…