PaperlayerView exact revision on arXiv

Source-linked AI summary

LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos

Ying Wang, Yanlai Yang, Mengye Ren

arXiv:2312.05269v3cs.CVcs.LG

TL;DR

Long-form egocentric video QA is challenging because videos contain multiple scenes and long-range dependencies, while question answering requires reasoning that may not generalize from limited supervision. LifelongMemory compresses video into captions for pretrained LLM reasoning, achieving state-of-the-art EgoSchema results and competitive Ego4D NLQ performance.

  • Problem

    Long-form egocentric videos contain multiple scenes and long-range dependencies, and question answering requires reasoning that end-to-end models may not generalize across question types from limited supervision.

    Source: §1 · p. 1

  • Method

    LifelongMemory condenses video into query-relevant captions and uses pretrained LLMs to reason over the context, producing answers or temporal windows with confidence and explanations.

    Source: §1 · p. 1, §3.1 · p. 4, §3.2 · p. 4

  • Results

    LifelongMemory achieves state-of-the-art EgoSchema performance, beating concurrent methods by over 10%, and remains highly competitive on Ego4D NLQ.

    Source: §4.3 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    The framework showcases the potential of LLM-based video understanding and opens opportunities for personalized AIs that answer daily queries for people requiring assistance.

    Source: §5 · p. 10

    Captioning inevitably loses information, and the approach handles uncertain NLQ and EgoSchema predictions differently.

    Source: §4.4 · p. 10

Abstract

from arXiv · show

In this paper we introduce LifelongMemory, a new framework for accessing long-form egocentric videographic memory through natural language question answering and retrieval. LifelongMemory generates concise video activity descriptions of the camera wearer and leverages the zero-shot capabilities of pretrained large language models to perform reasoning over long-form video context. Furthermore, LifelongMemory uses a confidence and explanation module to produce confident, high-quality, and interpretable answers. Our approach achieves state-of-the-art performance on the EgoSchema benchmark for question answering and is highly competitive on the natural language query (NLQ) challenge of Ego4D. Code is available at https://github.com/agentic-learning-ai-lab/lifelong-memory.

1 Introduction

Long-form egocentric video QA is difficult because videos span many scenes and require reasoning across details, while supervision may not generalize across question types. LifelongMemory turns video into concise textual memory for LLM reasoning and reports strong results on EgoSchema and Ego4D NLQ.

1 Introduction

  • Long videos contain many scenes, tasks, people, and objects, making retrieval difficult; question answering also requires reasoning that end-to-end models may not learn from limited supervision.Earlier methods improve low-level action and object features but fall short on long-form video understanding.Source: §1 · p. 1
  • LifelongMemory compresses long video inputs into concise text descriptions that an LLM uses to answer questions and locate relevant time windows.Its natural-language episodic memory supports retrieving specific moments in response to queries.Source: §1 · p. 1, Figure 1 · p. 1
  • The LLM produces confidence levels and textual explanations to help refine predictions and make outputs more interpretable.The framework integrates pretrained MLLMs and leverages LLM reasoning for long-range temporal understanding.Source: §1 · p. 2
  • LifelongMemory achieves state-of-the-art EgoSchema performance and remains highly competitive on Ego4D NLQ.On EgoSchema it doubles the accuracy of pretrained video QA models and significantly outperforms other LLM-based methods; on Ego4D NLQ, coarse candidate windows increase pretrained-model precision in zero shot.Source: §1 · p. 2

2 Related Work

Related work covers multimodal language models, their use in video question answering, and methods for localizing answers in egocentric videos. LifelongMemory uses pretrained models for long-form video understanding and complements existing NLQ models with coarse predictions.

Multimodal Large Language Models

  • LLaVA [24] [23] and LaViLa [55] exemplify MLLMs that connect visual inputs with language models, while many open models accept only an image or short video.LifelongMemory integrates pretrained MLLMs for long-form video understanding.Source: §2 · pp. 2–3

Video Question Answering with Multimodal LLMs

  • Prior video-QA approaches use retrieved descriptions or caption-derived labels, but fixed vocabularies and noisy details limit some methods on long videos.LifelongMemory instead uses distilled captions without a fixed keyword vocabulary and outperforms concurrent LLoVi [48] and Vamos [35] on EgoSchema [27].Source: §2 · p. 3

Natural Language Queries in Egocentric Videos

  • NLQ methods address sparse annotations and long videos through architectures, pretraining, and data augmentation, with GroundNLQ [15] identified as the benchmark’s current state of the art.Those models can serve in LifelongMemory’s final stage to produce finer-grained predictions from frozen-LLM predictions.Source: §2 · p. 3

3 LifelongMemory

LifelongMemory converts long egocentric videos into concise textual logs, uses an LLM to reason over them, and refines predictions according to the task. Its stages include caption digestion, confidence-based QA selection, and fine-grained NLQ localization.

3 LifelongMemory

  • The framework condenses egocentric video into a textual log, then uses an LLM to predict answers or coarse temporal windows with confidence and explanations before task-specific refinement.For QA it selects among repeated predictions; for NLQ it refines candidate intervals.Source: §3 · p. 3

3.1 Egocentric Video Captioning

  • Pretrained MLLMs caption sampled frames or short clips, creating a compressed log of daily activities as episodic memory.Each sampled clip receives a line of caption.Source: §3.1 · p. 4
  • Caption Digest removes uninformative and query-irrelevant captions, then merges adjacent similar captions with an LLM into a concise context.The condensed captions augment the LLM’s context for subsequent reasoning.Source: §3.1 · p. 4

3.2 LLM Reasoning

  • The LLM combines captions and queries, aggregates information, and reasons about the visual scene to select likely answers or relevant intervals.It is instructed to account for information lost during captioning and can use contextual clues when captions omit an explicit detail.Source: §3.2 · p. 4
  • For interpretability and later precision control, the LLM explains each prediction and rates its confidence using three levels.The verbalized confidence strategy is used in later stages.Source: §3.2 · p. 5

3.3 Vote by Confidence (Video QA)

  • For Video QA, repeated LLM predictions are ensembled by selecting the answer with the highest confidence score.Tied highest-confidence answers are selected randomly.Source: §3.3 · p. 5

3.4 Fine-grained Interval Refinement (NLQ)

  • For NLQ, a pretrained model refines LLM candidate windows by classifying clips extracted from padded intervals; queries without candidates use the full video.Localization within a coarse window makes the task easier than searching the original full-length video.Source: §3.4 · p. 5

4 Experiments

The experiments evaluate LifelongMemory on EgoSchema QA and Ego4D NLQ, reporting strong benchmark performance and examining captioning, prompting, and error sources.

4.1 Experiment Setup

  • Evaluation uses EgoSchema’s 500-pair ablation subset and full benchmark, plus Ego4D NLQ’s 19,200 queries; NLQ candidate windows are selected and passed to pretrained localizers for refinement.EgoSchema has five-choice questions about three-minute videos; Ego4D NLQ answers are temporal windows.Source: §4.1 · pp. 5–6

4.2 Qualitative Results

  • Examples show captions capturing objects and actions for correct EgoSchema answers and many NLQ windows overlapping ground truth before interval refinement; confidence explanations make outputs interpretable.The Figure 4 frames were manually selected to show query-related key events.Source: §4.2 · p. 6, Figure 4 · p. 7

4.3 Quantitative Results

  • LifelongMemory is state of the art on EgoSchema, beating GPT-4-based LLoVi and Vamos by over 10%, and leads Ego4D NLQ validation with ground-truth narrations and test with GroundNLQ refinement.The reported EgoSchema comparison is against concurrent GPT-4-based approaches; the NLQ results differ by split and configuration.Source: §4.3 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8
  • LaViLa outperforms longer query-conditioned LLaVA captions, while ground-truth narrations perform best with fewer captions; GPT-4 and GPT-4o beat weaker LLMs, especially on NLQ instruction following.Weaker models may misunderstand the NLQ prompt and return an answer rather than temporal intervals.Source: §4.3 · p. 8
  • Caption Digest raises both NLQ metrics by around 10% but not EgoSchema; sparse sampling hurts both tasks, while explanations improve both and higher confidence scores track accuracy.Digesting is adopted for both datasets because shorter contexts save computation, even though EgoSchema's undigested context is already under 100 captions.Source: §4.3 · pp. 9–10

4.4 Error Analysis

  • Most errors originate in captioning information loss; NLQ uses null predictions pending refinement when context is insufficient, while EgoSchema selects the likeliest option, and some NLQ queries are ambiguous.The two tasks use different uncertainty strategies because EgoSchema has no pretrained QA refinement stage.Source: §4.4 · p. 10

5 Conclusion

LifelongMemory uses pretrained MLLMs and a concise textual log to answer natural-language queries in long-form egocentric videos. It achieves state-of-the-art performance on EgoSchema and remains highly competitive on Ego4D NLQ, with interpretability enhanced by verbalized confidence and explanation.

5 Conclusion

  • LifelongMemory condenses long-form egocentric videos into a concise textual log that an LLM uses to understand context and answer natural-language queries.Source: §5 · p. 10
  • The framework achieves state-of-the-art performance on EgoSchema and remains highly competitive on Ego4D NLQ.Source: §5 · p. 10
  • Verbalized confidence and explanation enhance the interpretability of its answers, while the paper identifies potential for personalized AIs to answer individuals’ daily queries.Source: §5 · p. 10

A Prompting

The prompts provide instructions for answering EgoSchema Video QA and Ego4D NLQ using captions and questions or queries. They use structured responses, group NLQ queries from the same clip, and handle uncertain answers differently across tasks.

A Prompting

  • NLQ prompts group all queries from one video clip to avoid repeatedly passing the same caption list and reduce API-call costs.Source: A Prompting · p. 15
  • The prompts request structured responses and encourage abstention on uninformative NLQ context for later refinement, but require a likely answer for EgoSchema QA.Source: A Prompting · p. 15
  • The prompts ask the LLM to use captions and the corresponding question-answer pair for EgoSchema QA, and queries with captions for NLQ.Source: Figure 6 · p. 15, A Prompting · p. 15, Figure 7 · p. 16

B Additional Qualitative Results

Qualitative examples show GPT-4 reasoning over LaViLa captions for EgoSchema and NLQ, including raw NLQ interval predictions without refinement. The examples also illustrate that the LLM can answer well despite caption errors or omissions.

B Additional Qualitative Results

  • The LLM can produce high-quality zero-shot answers even when machine-generated captions contain false objects or omit query-relevant information.Source: B Additional Qualitative Results · p. 17
  • The NLQ examples show raw GPT-4 interval predictions using LaViLa captions, without refinement, over two-second, 30fps clips.Source: Figure 8 · p. 17
  • The EgoSchema examples show GPT-4 predictions and ground truth over two-second, 30fps clips, while the paper suggests better captioning could further improve performance.Source: B Additional Qualitative Results · p. 17, Figure 9 · p. 18

C Ambiguity in NLQ Annotations

For some NLQ queries, the LLM proposes multiple plausible temporal windows although Ego4D provides only one ground-truth answer. The paper treats this as annotation ambiguity that warrants filtering or modification of queries.

C Ambiguity in NLQ Annotations

Loading 2312.05269v3…