PaperlayerView exact revision on arXiv

Source-linked AI summary

A Simple LLM Framework for Long-Range Video Question-Answering

Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, Gedas Bertasius

arXiv:2312.17235v3cs.CV

TL;DR

Long-range video QA requires temporal reasoning beyond the short-video focus of many existing methods. LLoVi addresses this by captioning short clips and aggregating their descriptions with an LLM, achieving strong results across LVQA benchmarks while remaining simple and training-free.

  • Problem

    Extending short-video models to long videos requires sophisticated long-range temporal reasoning, while existing approaches often use costly specialized modeling schemes.

  • Method

    LLoVi densely captions short video clips, concatenates the descriptions, and uses an LLM for long-range reasoning, optionally summarizing noisy captions first.

  • Results

    LLoVi outperforms prior approaches across EgoSchema, NExT-QA, IntentQA, and NeXT-GQA, with LaViLa achieving 55.2% accuracy in the reported EgoSchema analysis.

  • Takeaways & Limitations

    The framework provides a simple, training-free, and flexible baseline that can incorporate different visual captioners and LLMs for LVQA.

  • Takeaways & Limitations

    Performance may be suboptimal when visual captions hallucinate or miss fine-grained visual details, and many LLMs are not designed for long-context modeling.

Abstract

from arXiv · show

We present LLoVi, a language-based framework for long-range video question-answering (LVQA). Unlike prior long-range video understanding methods, which are often costly and require specialized long-range video modeling design (e.g., memory queues, state-space layers, etc.), our approach uses a frame/clip-level visual captioner (e.g., BLIP2, LaViLa, LLaVA) coupled with a Large Language Model (GPT-3.5, GPT-4) leading to a simple yet surprisingly effective LVQA framework. Specifically, we decompose short and long-range modeling aspects of LVQA into two stages. First, we use a short-term visual captioner to generate textual descriptions of short video clips (0.5-8s in length) densely sampled from a long input video. Afterward, an LLM aggregates the densely extracted short-term captions to perform long-range temporal reasoning needed to understand the whole video and answer a question. To analyze what makes our simple framework so effective, we thoroughly evaluate various components of our system. Our empirical analysis reveals that the choice of the visual captioner and LLM is critical for good LVQA performance. Furthermore, we show that a specialized prompt that asks the LLM first to summarize the noisy short-term visual captions and then answer a given input question leads to a significant LVQA performance boost. On EgoSchema, which is best known as a very long-form video question-answering benchmark, our method achieves 50.3% accuracy, outperforming the previous best-performing approach by 18.1% (absolute gain). In addition, our approach outperforms the previous state-of-the-art by 4.1% and 3.1% on NeXT-QA and IntentQA. We also extend LLoVi to grounded LVQA and show that it outperforms all prior methods on the NeXT-GQA dataset. We will release our code at https://github.com/CeeZh/LLoVi.

1 Introduction

LLoVi addresses long-range video QA by replacing costly specialized temporal modules with a two-stage captioning-and-LLM framework. Its empirical study identifies prompting, model selection, and video processing choices that affect performance.

  • Long-range video understanding is difficult because extending short-video models to minutes- or hours-long videos requires sophisticated temporal reasoning.
  • Existing long-range video models commonly rely on costly mechanisms such as memory queues, feature banks, space-time graphs, and state-space layers.
  • LLoVi decomposes LVQA into short-term visual captioning and long-range reasoning over temporally ordered captions with an LLM.
  • The study evaluates visual captioner choice, LLM choice, prompt design, and video-processing configurations on EgoSchema.
  • 55.2% accuracy is achieved with LaViLa captions, while GPT-4 performs best and sparse sampling provides 8x improved efficiency with a 2.0% accuracy drop.
  • LLoVi is training-free and outperforms prior approaches on EgoSchema, NExT-QA, IntentQA, and NeXT-GQA.

2 Related Work

Related work spans complex long-range video models, short-term video QA, and LLM prompting. LLoVi differs by using multi-round summarization to handle long, noisy, and redundant video descriptions.

  • Long-range Video Understanding: Prior long-range video methods use temporal attention, memory-based designs, space-time graphs, and relational modeling to process extended videos.
  • Long-range Video Understanding: Earlier LLM-based video studies include GPT-4 long-range modeling without quantitative evaluation and movie-focused methods relying substantially on speech or subtitles.
  • Video Question Answering: Most video question-answering methods focus on short-term analysis of videos lasting approximately 5–30 seconds.
  • LLM Prompt Design: Prompt-design research includes few-shot, zero-shot, and automatically refined prompting strategies.
  • LLM Prompt Design: LLoVi proposes multi-round summarization for long, noisy, and redundant textual inputs describing video content.

3 Method

LLoVi converts a long video into short-clip captions, concatenates them in temporal order, and uses an LLM for long-range QA. A multi-round prompt first summarizes captions and then answers the question from that summary.

  • Method Overview: LLoVi decomposes LVQA into short-term video clip captioning and long-range text-based video understanding.
  • Short-term Video Clip Captioning: A long video is segmented into non-overlapping short clips, each processed by a pretrained visual captioner to produce textual descriptions.
  • Short-term Video Clip Captioning: The framework accepts multiple video and image captioners rather than restricting the visual-captioning component to one model.
  • Long-range Reasoning with an LLM: Caption outputs are concatenated in their original temporal order and supplied to an LLM with the question and answer candidates for LVQA.
  • Long-range Reasoning with an LLM: Because long caption sequences may be noisy or redundant, the multi-round prompt first generates a summary and then uses that summary for answer selection.
  • Experimental Configuration: The method uses LaViLa with consecutive 1-second clips and GPT-3.5 on EgoSchema, while other benchmarks use CogAgent and GPT-4.

4 Experiments

Experiments show that LLoVi’s performance depends strongly on its captioner, LLM, prompt, and video-sampling configuration. The framework achieves strong accuracy across LVQA benchmarks while allowing an accuracy–efficiency trade-off through sparse sampling.

  • Visual Captioning Model: 55.2% EgoSchema accuracy is achieved with LaViLa captions, ahead of BLIP-2 at 50.6% and EgoVLP at 46.6%.
  • Visual Captioning Model: The Oracle captioning baseline reaches 66.0% accuracy, 10.8 percentage points above LaViLa, indicating headroom from improved captioning models.
  • Large Language Model: 61.2% EgoSchema accuracy is achieved with GPT-4, while GPT-3.5 offers the reported best accuracy–cost trade-off.
  • Video Processing Configurations: Consecutive 1-second clips yield the best accuracy, while sampling every 8 seconds provides 8x efficiency with only a 2.0% accuracy drop.
  • LLM Prompt Analysis: +3.6% is the largest prompt improvement, produced by summarizing captions conditioned on the question before answering.
  • Main Results on EgoSchema: On EgoSchema, the multi-round prompt reaches 52.2% accuracy, exceeding Vamos by 3.9% and the standard-prompt variant by 9.4%.
  • Results on Other Datasets: In zero-shot evaluation, LLoVi exceeds SeViLA by 10.2% on NExT-QA and outperforms prior methods on IntentQA and NExT-GQA.

5 Conclusion

LLoVi is a simple, effective LLM-based framework for long-range video question-answering that outperforms prior models across multiple benchmarks and extends to grounded LVQA.

  • LLoVi outperforms all prior models on EgoSchema and generalizes to NExT-QA, IntentQA, and grounded LVQA tasks.The authors also evaluate design choices and analyze factors behind the method’s performance.

Limitations

LLoVi can produce suboptimal results when visual captions are inaccurate, and long-context limitations in modern LLMs constrain the framework.

  • Inaccurate visual captions can produce suboptimal LLoVi results because captioners may hallucinate or miss fine-grained visual details.The authors identify fine-grained human-object interactions as one example of difficult visual content.
  • Many modern LLMs are not designed for the long-context modeling required by LVQA.The authors expect future LLM designs to address this limitation.

A Additional Datasets and Metrics

This section describes the datasets and metrics used to evaluate long-range video question-answering and temporal grounding.

  • Datasets: NExT-QA contains 5,440 videos averaging 44 seconds, with 48K multiple-choice and 52K open-ended questions.Its question types are Temporal, Causal, and Descriptive; evaluation uses a validation subset of 570 videos and 5K multiple-choice questions.
  • Datasets: IntentQA contains 4,303 videos and 16K multiple-choice question-answer pairs focused on reasoning about people’s intent.The paper evaluates it zero-shot on a test set containing 2K questions.
  • Datasets: NExT-GQA extends NExT-QA with 10.5K temporal grounding annotations linked to the original question-answer pairs.It evaluates whether LVQA models can localize video segments needed to answer questions.
  • Metrics: Intersection over Prediction measures whether the predicted temporal window lies inside the ground-truth temporal segment.

B Qualitative Analysis

The qualitative analysis shows that captioning choices affect which actions and people are represented, while multi-round summarization can filter irrelevant or repetitive captions but remains vulnerable to incorrect inferences.

  • Captioner comparison: LaViLa captions are generally more concise and action-focused, whereas BLIP2 captions emphasize objects.
  • Captioner comparison: LaViLa better differentiates the camera wearer from other people and captures other people’s actions.
  • Standard prompt: Standard-prompt examples show that LLoVi uses short-term captions for LLM-based long-range modeling, with success depending on the described actions.
  • Multi-round summarization: Multi-round summarization can remove unrelated descriptions before answering questions about a tool’s primary function.
  • Multi-round summarization: A summarization-stage wrong inference can misdirect the subsequent question-answering stage and produce an incorrect answer.
  • Multi-round summarization: Repetitive actions in short-term captions create redundancy that multi-round summarization is intended to address.

C Additional Implementation Details

The implementation uses pretrained visual captioners and LLMs with dataset-specific configurations, sampling choices, and prompt-processing procedures. Additional figures illustrate successful and failed examples of the framework and its prompting variants.

  • Captioners: LaViLa is the primary EgoSchema captioner, while BLIP-2 and LLaVA use specified off-the-shelf pretrained variants.For NExT-QA, IntentQA, and NExT-GQA, the captioner choices include LLaVA-1.5 and CogAgent.
  • Training setup: LaViLa was retrained on non-overlapping Ego4D videos to avoid overlap between its training data and EgoSchema evaluation videos.The original LaViLa training set included EgoSchema videos, motivating removal of overlapping videos before retraining.
  • Qualitative examples: Figures provide qualitative examples of successful and failed standard-prompt answers, multi-round summarization cases, and their contrasting processes.Figure 5 also visualizes character identification behavior in EgoSchema.
  • Captioners: Narrator combines a TimeSformer visual encoder, a resampler producing 256 visual tokens, and a GPT-2 text decoder with cross-attention.The visual encoder and text decoder are frozen, while the cross-attention layers and resampler are trained.
  • Caption generation: Nucleus sampling uses p = 0.95 and returns K = 5 candidate narrations, after which the highest-confidence narration becomes the clip caption.This procedure selects one final caption from multiple generated outputs.
  • LLMs and output processing: EgoSchema experiments primarily use GPT-3.5 with temperature 0, while other datasets use GPT-4 and Llama-3 models with specified sampling settings.Multiple-choice outputs are post-processed to obtain a single-character answer.

D Additional Analysis

This section introduces additional analysis of the EgoSchema Subset using the standard prompt.

  • Scope: The additional analysis focuses on the EgoSchema Subset.The section uses the standard prompt for this analysis.
  • Evaluation setting: The standard prompt is the evaluation setting for the reported additional analysis.
  • Purpose: The section extends the paper’s empirical examination beyond its main experiments.

D.1 Additional Ablations on NExT-QA

Additional NExT-QA ablations examine how visual captioner and LLM choices affect the framework. CogAgent with GPT-4 performs best, while both component choices show consistent performance patterns.

  • Captioner and LLM combinations: CogAgent combined with GPT-4 achieves the best results among the tested captioner–LLM combinations.The tested captioners are BLIP-2, LLaVA-1.5, and CogAgent; tested LLMs are Llama-3-70B, GPT-3.5, and GPT-4.
  • Captioners: CogAgent consistently outperforms LLaVA-1.5, which consistently outperforms BLIP-2, across the tested LLMs.
  • Interpretation: The results suggest that stronger visual captioners and LLMs improve performance within this framework.

D.2 Accuracy on Different Question Types

The EgoSchema Subset is analyzed by question type, with character interaction and purpose/goal identification receiving especially strong performance. Purpose/goal questions are also among the most common categories.

  • Question categories: The analysis groups questions into purpose/goal identification, tools and materials usage, key action/moment detection, action sequence analysis, and character interaction.Questions may belong to more than one category.
  • Category distribution: Almost half of the questions concern purpose/goal identification, while tool usage, key action detection, and action sequence analysis form substantial portions.Character interaction is the smallest reported category fraction.
  • Performance by category: 63.8% accuracy is highest for Character Interaction questions, while the LaViLa captioner is described as suited to character interactions.
  • Performance by category: 56.5% accuracy is reported for Purpose/Goal Identification, and the remaining categories each exceed 50% accuracy.The paper highlights purpose and goal inference as requiring very long-form video analysis.
  • Prompt settings: The evaluated prompts require single-letter multiple-choice answers, with variants including standard, chain-of-thought, plan-and-solve, and multi-round summarization prompts.The multi-round prompt summarizes captions before answering the question.
Loading 2312.17235v3…