PaperlayerView exact revision on arXiv

Source-linked AI summary

DrVideo: Document Retrieval Based Long Video Understanding

Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, Jianfei Cai

arXiv:2406.12846v2cs.CV

TL;DR

Long-video understanding remains less explored than short-video understanding, despite the challenge of locating key information and reasoning across long videos. DrVideo converts videos into documents for retrieval and iterative augmentation, and reports gains over existing LLM-based state-of-the-art methods across long-video benchmarks.

  • Problem

    Long-video understanding remains unclear compared with work on videos lasting tens of seconds, although it requires spatio-temporal processing and advanced reasoning.

    Source: §1 · p. 1

  • Method

    DrVideo converts long videos into documents, retrieves key frames, augments their information, and iteratively searches for missing question-related details before answering.

    Source: §1 · p. 2

  • Results

    DrVideo exceeds existing LLM-based state-of-the-art methods by 3.8% on EgoSchema subset, 8.3% on MovieChat-1K global mode, 24.8% on MovieChat-1K breakpoint mode, and 6.3% on long-split Video-MME.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Comparative experiments and ablations on EgoSchema, MovieChat-1K, and Video-MME demonstrate the effectiveness of DrVideo.

    Source: §5 · p. 8

    The maximum video length depends on the LLM’s maximum token length, and document generation could better balance sufficient against irrelevant information.

    Source: §6 · p. 9

Abstract

from arXiv · show

Most of the existing methods for video understanding primarily focus on videos only lasting tens of seconds, with limited exploration of techniques for handling long videos. The increased number of frames in long videos poses two main challenges: difficulty in locating key information and performing long-range reasoning. Thus, we propose DrVideo, a document-retrieval-based system designed for long video understanding. Our key idea is to convert the long-video understanding problem into a long-document understanding task so as to effectively leverage the power of large language models. Specifically, DrVideo first transforms a long video into a coarse text-based long document to initially retrieve key frames and then updates the documents with the augmented key frame information. It then employs an agent-based iterative loop to continuously search for missing information and augment the document until sufficient question-related information is gathered for making the final predictions in a chain-of-thought manner. Extensive experiments on long video benchmarks confirm the effectiveness of our method. DrVideo significantly outperforms existing LLM-based state-of-the-art methods on EgoSchema benchmark (3 minutes), MovieChat-1K benchmark (10 minutes), and the long split of Video-MME benchmark (average of 44 minutes).

1. Introduction

Long-video understanding remains unclear despite progress on short clips, while existing video-language and caption-agent approaches can miss key information or make it hard to locate. DrVideo reframes the task as long-document retrieval and understanding, and reports gains over existing LLM-based state-of-the-art methods across long-video benchmarks.

1. Introduction

  • Video-language models face input-length limits, content-agnostic frame sampling, and difficulty locating question-related visual information in long token sequences.The passage notes that CLIP-L-14 produces 24×24 tokens per image, while LLaVA-NeXT-Video has an 8192-token maximum.Source: §1 · p. 1
  • Caption-based agent methods can miss question-related frames and omit essential visual details, leaving answers dependent on inference from incomplete descriptions.The passage contrasts a generic mirror-related caption with a question about what a woman is wearing.Source: §1 · p. 2
  • DrVideo converts long videos into documents, retrieves key frames, augments their information, and uses a multi-stage agent loop to search for missing details before answering.The approach is presented as a document-retrieval system for leveraging LLM key-information location and long-range reasoning.Source: §1 · p. 2, Figure 1 · p. 2
  • DrVideo exceeds existing LLM-based state-of-the-art methods by 3.8% on EgoSchema subset, 8.3% on MovieChat-1K global mode, 24.8% on MovieChat-1K breakpoint mode, and 6.3% on long-split Video-MME.The benchmarks span videos from 3 minutes to longer than 1 hour.Source: §1 · p. 2
  • DrVideo is described as training-free and reproducible on an RTX 4090 with a reasonable number of GPT accesses.The work frames the system as researcher-friendly.Source: §1 · p. 3

2. Related Work

Prior long-video work uses temporal modeling mechanisms, while LLM-based work includes video representation and agent approaches. DrVideo instead treats long-video understanding as document retrieval, augmentation, and understanding.

2. Related Work

  • Long-video systems use temporal attention, memory, space-time graphs, relational modules, or S4 layers, whereas DrVideo uses an LLM-based retrieval system for zero-shot understanding.Examples include LF-VILA, MeMViT and MovieChat [66] [49], and S4ND, ViS4mer, and S5 [23] [58].Source: §2 · p. 3
  • DrVideo distinguishes its approach from prior vision agents by recasting long-video understanding as document retrieval, augmentation, and understanding.The related work describes earlier agents as interacting with external tools or adding functionalities [14] [17] [61] [71].Source: §2 · p. 3

3. Methodology

DrVideo converts a long video into a text document, retrieves and augments question-related frame information, then iteratively searches for missing information before answering from the resulting document.

3.1. Video-Document Conversion Module

  • Each frame or short clip becomes a short description, forming an initial video document whose entries correspond to video positions.A vision-language captioning model generates the descriptions.Source: §3.1 · pp. 3–4

3.2. Document (Frames) Retrieval Module

  • The retrieval module embeds the question and initial document, then selects the top K question-related frames using cosine similarity.It uses the OpenAI embedding model for the representations.Source: §3.2 · p. 4

3.3. Document Augmentation Module

  • For each retrieved frame, LLaVA-NeXT uses an augmented prompt to generate a detailed description that is added to the document.Because K is much smaller than T, the additional descriptions are not expected to substantially increase document length.Source: §3.3 · p. 4

3.4. Multi-Stage Agent Interaction Loop

  • The planning agent checks whether the updated document supports a confident answer; if not, the interaction agent finds missing-detail frames, chooses information types, and requests augmentation.The loop repeats until the planner finds the information sufficient or the maximum iteration is reached.Source: §3 · p. 3, §3.4 · p. 4

3.5. Answering Module

  • After the loop, an answering module uses the final document and question to produce an answer, confidence score, and chain-of-thought reasoning.The authors state that this reasoning makes inference steps traceable.Source: §3.5 · pp. 4–5

4. Experiments

Experiments assess DrVideo on three long-video benchmarks and report gains over existing LLM-based methods. Ablations examine its retrieval and interaction components, information augmentation, iterative rounds, and model choices.

4.1. Datasets and Metrics

  • Evaluation covers zero-shot EgoSchema’s 500-question labeled subset, MovieChat-1K global and breakpoint modes, and Video-MME’s long split, using accuracy or GPT-assisted evaluation.The Video-MME long split averages 44 minutes; MovieChat-1K videos last about 10 minutes, while EgoSchema videos last three minutes.Source: §4.1 · p. 5

4.2. Implementation Details

  • LaViLa captions EgoSchema videos and LLaVA-NeXT captions MovieChat-1K and Video-MME frames; sampling rates are 0.5 FPS and 0.2 FPS, respectively, for Video-MME.GPT-4 agents are used for EgoSchema and MovieChat-1K comparisons, while DeepSeek V2.5 agents are used for Video-MME.Source: §4.2 · p. 5

4.3. Main Results

  • DrVideo outperforms two referenced methods by 5.2% and 6.2%, respectively; on MovieChat-1K, it improves global and breakpoint modes by 8.3% and 24.8%, respectively.On Video-MME, it achieves at least 6.3% improvement without subtitles and 71.7% accuracy with subtitles.Source: §4.3 · p. 6

4.4. Ablation Studies

  • Adding retrieval and then the agent loop raises EgoSchema performance from 57.4% to 60.6% and 62.6%; CoT improves it from 62.2% to 62.6%.Using only VQA or caption augmentation lowers performance, and more initial key frames than the default K=5 worsens it, with added noise cited as a likely reason.Source: §4.4 · pp. 6–7, Table 4 · p. 6, Table 5 · p. 7
  • DrVideo peaks at two interaction rounds while VideoAgent peaks at three; LaViLa performs best among tested captioners, and proprietary LLMs significantly outperform the tested open-source LLM.DrVideo performance declines beyond two rounds, and the authors report that it depends on LLM choice.Source: §4.4 · pp. 7–8, Figure 3 · p. 7, Table 6 · p. 7
  • In the EgoSchema case study, DrVideo retrieves and augments initial key frames, requests additional frames after planning, then answers once the planner deems the document sufficient.The example’s first retrieval includes frames 15, 46, 61, 84, and 86; a later interaction retrieves frames 4, 5, 6, 14, and 19.Source: §4.4 · p. 8, Figure 4 · p. 8

5. Conclusion

DrVideo recasts long-video understanding as document retrieval and reasoning, and the authors position it as a baseline for further work.

5. Conclusion

  • DrVideo adapts long-video understanding to long-document understanding, searching for missing information through document retrieval and a multi-stage agent loop.Source: §5 · p. 8
  • Comparative experiments and ablations on EgoSchema, MovieChat-1K, and Video-MME demonstrate the method’s effectiveness.Source: §5 · p. 8
  • The authors propose DrVideo as a strong baseline to stimulate further development in long-video understanding.Source: §5 · p. 9

6. Limitation

DrVideo’s usable video duration is bounded by the LLM’s maximum token length, and the authors identify information quality as another area for improvement.

6. Limitation

  • The LLM’s maximum token length bottlenecks DrVideo on videos much longer than existing benchmarks, with 10-hour videos given as an example.Source: §6 · p. 9
  • The authors identify further room to generate sufficient information while minimizing irrelevant information in the video document.Source: §6 · p. 9
  • The stated duration constraint concerns videos much longer than the existing benchmarks, rather than all long videos.Source: §6 · p. 9

7. More Implementation Details

The implementation uses different captioners and reproduction settings across benchmarks, with prompts specifying how the EgoSchema agents assess sufficiency, request frame information, and answer questions.

7. More Implementation Details

  • Experiments used one NVIDIA RTX 4090 GPU with a 24GB minimum memory requirement and temperature 0 for GPT-3.5, GPT-4, and DeepSeek.Source: §7 · p. 10
  • EgoSchema prompts have a planning agent judge caption sufficiency, an interaction agent request missing-frame captions or VQA, and an answering agent select an option.Source: §7 · pp. 10–11
  • EgoSchema videos were captioned with retrained LaViLa to avoid overlap with evaluation videos and an unfair comparison.Source: §7 · p. 11
  • MovieChat-1K and Video-MME use 7B-parameter LLaVA-NeXT as captioner, including for key-frame augmentation.Source: §7 · p. 11
  • VideoAgent reproduction used LLaVA-NeXT and EVA-CLIP with GPT-4 on MovieChat-1K, and DeepSeek with 0.2 FPS sampling on Video-MME.Source: §7 · p. 11
  • The LLoVi Video-MME reproduction used LLaVA-NeXT, DeepSeek, 0.2 FPS sampling, and a 500-word summary setting.Source: §7 · p. 11

8. More Ablation Studies

The EgoSchema sampling-rate ablation favors 0.5 FPS, while the described pipeline retrieves and augments key-frame information through iterative agent interactions.

8. More Ablation Studies

  • The EgoSchema comparison used LaViLa, tested 1, 0.5, and 0.25 FPS sampling, and used GPT-3.5 for planning, interaction, and answering.Source: §8 · p. 12
  • The pipeline converts video to a document, retrieves top-K content for the question, augments it, and iterates through sufficiency checks and missing-information requests.Source: §8 · p. 12
  • 0.5 FPS produced the best EgoSchema performance; 1 FPS performed worse amid redundancy, while 0.25 FPS lost significant information.Source: §8 · p. 12
  • The Video-MME case-study figure describes a 33-minute video.Source: Figure 5 · p. 13

9. Detailed Algorithm

The section points readers to Algorithm 1 for DrVideo’s algorithm and notes that its symbols are defined in the main text.

9. Detailed Algorithm

  • Algorithm 1 presents DrVideo’s algorithm to give readers a clearer understanding.The symbol definitions are provided in the main text.Source: §9 · p. 13
  • The algorithm’s symbols are defined in the main text.Source: §9 · p. 13

10. More Case Studies

The case studies show DrVideo solving an hour-long Video-MME question requiring broad video understanding and local detail, alongside a failure caused by inaccurate VLM descriptions.

10. More Case Studies

  • DrVideo correctly answers an hour-long Video-MME question about glaze colors, outperforming VideoAgent in the case study.The question requires both comprehensive video understanding and clear local-detail understanding.Source: §10 · p. 13
  • The glaze-color case combines comprehensive video understanding with attention to local details.Source: §10 · p. 13
  • In the failure case, an inaccurate VLM description leads to incorrect judgments and a wrong answer despite retrieval enhancements.The authors say DrVideo relies heavily on LLM and VLM capabilities; Figure 6 presents the failure case.Source: §10 · p. 13, Figure 6 · p. 14
Loading 2406.12846v2…