PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Online Textual Memory for Episodic Memory Video Question Answering

Raffaele Cali, Giuseppe Lando, Rosario Forte, Antonino Furnari

arXiv:2610.31284v1cs.CV

TL;DR

Long egocentric video QA must find useful evidence in textual memory without exceeding context budgets or adding plausible distractors. This study isolates retrieval over fixed generated memories and finds that PARENT achieves 56.4% accuracy with compact context, while broader retrieval can hurt answers despite improving temporal coverage.

  • Problem

    Long egocentric videos create a challenge of selecting useful evidence from textual memory, since reading the full memory can exceed practical context budgets and introduce plausible distractors.

    Source: §I · pp. 1–2

  • Method

    The study holds generated textual memory fixed and compares retrieval strategies, with PARENT matching fine-grained caption subchunks and providing their parent captions to the reasoner.

    Source: §I · p. 2

  • Results

    PARENT reaches 56.4% accuracy on QAEgo4D-Closed using 3,600 tokens per query on average, while higher retrieval budgets improve temporal Hit@K and Recall@K but can lower accuracy.

    Source: §I · p. 2

  • Takeaways & Limitations

    Evidence-aware episodic memories should preserve temporally grounded events, uncertainty, and answer-relevant support rather than treating captions only as independent text units.

    Source: §V · p. 6

Abstract

from arXiv · show

Long egocentric videos pose a central challenge for multimodal language models: the visual stream must be compressed into a memory representation that can be searched efficiently while still preserving the evidence needed for question answering. Building on an online textual-memory framework in which first-person videos are converted into temporally grounded captions, this work systematically studies the role of retrieval in episodic-memory video question answering. We evaluate how different retrieval units, query formulations, context budgets, and reranking strategies affect multiple-choice answering on QAEgo4D-Closed, isolating retrieval as the main variable while keeping the underlying generated memory fixed. Our best configuration, Parent, matches fine-grained caption subchunks, then provides their full parent captions to the reasoner. It achieves 56.4 % accuracy, matching strong full-memory and retrieval baselines while exposing only a small fraction of the textual memory at question time. More importantly, our analysis shows that better temporal coverage does not necessarily translate into better answers: increasing the retrieval budget improves temporal Hit@K and Recall@K, but can reduce accuracy by introducing visually plausible distractors and uncertain caption evidence. These findings suggest that retrieval-augmented egocentric memory should not be optimized only for evidence recall, but also for the reliability, specificity, and answer-discriminative value of the retrieved context. The study motivates more structured episodic memories that move beyond raw caption retrieval toward evidence-aware representations for long-form egocentric reasoning.

I. INTRODUCTION

Long egocentric video QA must select useful evidence from textual memory without exceeding context budgets or adding plausible distractors. This study isolates retrieval over fixed generated memories, finding that PARENT achieves strong accuracy with compact context while broader retrieval can hurt answers despite improving temporal coverage.

I. INTRODUCTION

  • The paper builds on textual memory by asking which parts should reach the reasoner at question time, retrieving relevant subchunks and expanding them to parent captions.Prior textual-memory work is described as matching dedicated state-of-the-art systems on QAEgo4D-Closed with substantially less memory than visual-feature or large-video-representation approaches.Source: §I · pp. 1–2
  • Retrieval must select evidence that recovers the relevant event, preserves context for disambiguation, and avoids plausible but incorrect evidence from other moments.Reading the full textual memory can exceed practical context budgets and introduce plausible distractors.Source: §I · pp. 1–2
  • PARENT reaches 56.4% accuracy on QAEgo4D-Closed, matching strong state-of-the-art baselines while using 3,600 tokens per query on average.It matches fine-grained caption subchunks, then gives their full parent captions to the reasoner.Source: §I · p. 2
  • The study isolates retrieval by holding generated textual memory fixed while comparing how entries are indexed, retrieved, transformed, and passed to the reasoner.Compared strategies include flat caption, subchunk, parent, hybrid dense–lexical retrieval, and reranking.Source: §I · p. 2
  • Increasing the retrieval budget raises temporal Hit@K and Recall@K, but accuracy peaks earlier and can decline as plausible distractors enter the context.Repeated objects, similar locations, uncertain descriptions, and semantically close actions can correspond to the wrong moment.Source: §I · p. 2

II. RELATED WORK

Prior work frames egocentric QA as recalling temporally grounded evidence from long first-person videos, while video-RAG and memory research offer related approaches to retrieval and context organization. This paper focuses on selecting evidence from an already generated textual memory for downstream answering.

A. Egocentric Video Question Answering

  • Episodic-memory QA addresses questions about past first-person observations; NLQ requires temporal localization, while Ground VQA includes the QAEgo4D-Closed multiple-choice setting.Multiple-choice formulation reduces free-form evaluation ambiguity but still requires finding the relevant moment and distinguishing the answer from plausible distractors.Source: A. Egocentric Video Question Answering · p. 2

B. Video Retrieval-Augmented Generation

  • Unlike video-RAG systems that select visual-textual evidence or document-like representations, this work isolates retrieval in a pipeline whose egocentric captions are already generated.The study compares retrieval units, query formulation, context budget, and memory format for multiple-choice answering.Source: B. Video Retrieval-Augmented Generation · p. 2

C. Memory Structures for Long Context Reasoning

  • Long-context memory research explores organization beyond independent chunks, including hippocampal-inspired memory and graph-based associations.Textual memory is described as compact, human-readable, and reusable across queries.Source: C. Memory Structures for Long Context… · pp. 2–3
  • Egocentric questions often require temporal and relational context, so useful caption memories should preserve entities with temporal anchors, actions, and locations while remaining compact.The paper argues retrieval should consider token efficiency, context specificity, and answer-discriminative value as well as evidence recall.Source: C. Memory Structures for Long Context… · p. 3

III. METHOD

The method incrementally converts egocentric video clips into temporally grounded captions and indexes smaller text chunks for question-time retrieval. The question and candidate answers guide retrieval, and a reasoner uses the retrieved context to predict the answer.

III. METHOD

  • The pipeline extends online textual memory with query-specific retrieval before multiple-choice answering, separating online memory construction, memory management, and question answering.The question and candidate answers form the retrieval query; the reasoner receives them with retrieved context.Source: §III · p. 3

A. Online Memory Constructor

  • For each incoming clip, a multimodal descriptor generates a natural-language caption, which a chunking module decomposes into smaller textual units.Each chunk is embedded and stored with temporal metadata in a vector database.Source: §III · p. 3
  • Each chunk inherits the temporal interval of its source clip, preserving temporal grounding at finer textual granularity.The constructor outputs temporally grounded textual chunks.Source: A. Online Memory Constructor · p. 3

B. Memory Manager

The Memory Manager embeds textual chunks and stores them as indexed memory entries with their temporal intervals. Its memory accumulates as clips are processed, without a predefined upper bound on entries.

B. Memory Manager

  • The Memory Manager embeds each textual chunk, whose dense vector representation supports indexing and similarity search.Source: B. Memory Manager · p. 3
  • Each indexed memory entry associates a textual chunk with its clip’s start and end timestamps and the embedding used for similarity search.Source: B. Memory Manager · p. 3
  • After processing clips, the vector database contains the indexed memory entries accumulated so far.Source: B. Memory Manager · p. 4
  • The memory grows incrementally as the video stream is processed, with no predefined upper bound on the number of entries.Source: B. Memory Manager · p. 4

C. QA Handler

The QA Handler forms a retrieval query from the question and candidate answers, searches the indexed memory, and gives retrieved evidence to a reasoner to predict an answer.

C. QA Handler

  • The retrieval query concatenates the question with all candidate answers, then is embedded using the same model as the memory chunks for similarity search.Source: C. QA Handler · p. 4
  • The retriever selects the top-K memory entries, each carrying textual evidence and its associated temporal interval.Source: C. QA Handler · p. 4
  • The reasoner receives the question, candidate answers, and retrieved context, then predicts the final answer.Source: C. QA Handler · p. 4
  • The paper considers different retrieval strategies for supplying the reasoner with context.Source: C. QA Handler · p. 4

D. Flat Caption Retrieval

Flat caption retrieval treats each full caption as one searchable memory unit and selects captions by similarity, preserving each retrieved caption’s local context.

D. Flat Caption Retrieval

E. Subchunk Retrieval

Subchunk retrieval decomposes captions into independently embedded and indexed chunks, then supplies the retrieved chunks directly to the reasoner as focused evidence.

E. Subchunk Retrieval

F. Parent Retrieval

PARENT separates the unit used for retrieval from the context given to the reasoner: it matches fine-grained subchunks, then expands them to full parent captions. HYBRID combines dense and lexical retrieval rankings before similarly expanding retrieved subchunks.

F. Parent Retrieval

  • PARENT retrieves chunk-level entries and maps each match back to its full parent caption for reasoning.This uses fine-grained chunks for matching while preserving the broader caption context.Source: F. Parent Retrieval · p. 4

G. Hybrid Retrieval

  • HYBRID combines dense embedding-similarity rankings with lexical BM25 rankings using reciprocal rank fusion.The fused top-K subchunks are expanded to parent captions before the reasoner predicts an answer.Source: G. Hybrid Retrieval · pp. 4–5
  • The hybrid retrieval budget curve shows temporal coverage increasing with K while answer accuracy peaks at an intermediate budget.The figure describes the PARENT configuration.Source: Figure 4 · p. 5

H. Evaluation

Evaluation distinguishes whether retrieved entries overlap the annotated evidence region from how much of that region they cover, and separately measures final answer accuracy.

  • Answer accuracy is the percentage of questions whose predicted answer matches the ground truth.Source: H. Evaluation · p. 5
  • Hit@K measures whether at least one of the top-K retrieved entries overlaps the annotated evidence region, while Recall@K measures the fraction of that region covered.By construction, Hit@K ≥ Recall@K.Source: H. Evaluation · p. 5
  • Temporal Hit@K and Recall@K assess retrieval quality, whereas answer accuracy measures the full retrieval-augmented reasoning pipeline.Source: H. Evaluation · p. 5

IV. EXPERIMENTAL RESULTS AND ANALYSIS

Across retrieval configurations, PARENT provides the best reported accuracy, while larger retrieval budgets improve temporal coverage without guaranteeing better answers.

A. Analysis of Retrieval Strategies

  • 56.4% accuracy is achieved by PARENT, compared with 55.6% for FLAT at K = 12 retrieved units.PARENT has Hit@K of 79.0% and Recall@K of 70.7%; FLAT has slightly higher retrieval scores.Source: A. Analysis of Retrieval Strategies · p. 5
  • SUB-RERANK has the strongest Hit@K and Recall@K, but its higher retrieval scores do not translate into the best final accuracy.Fine-grained subchunks can fragment context, whereas parent captions preserve temporal and semantic context around matched evidence.Source: A. Analysis of Retrieval Strategies · p. 5
  • The most effective strategy couples retrieval precision with readable context rather than simply maximizing retrieval metrics.Source: A. Analysis of Retrieval Strategies · p. 6

B. Retrieval Budget and Temporal Coverage

  • As PARENT’s K increases, Hit@K rises from 53.4% to 98.2% and Recall@K from 42.7% to 97.9%, while accuracy peaks at 56.4% for K = 12 and then decreases.Source: B. Retrieval Budget and Temporal… · p. 6
  • Larger contexts can cover more annotated evidence while introducing plausible hard negatives from repeated rooms, objects, screens, tables, and actions, making option selection harder.Source: B. Retrieval Budget and Temporal… · p. 6

C. Comparison with State-of-the-Art Methods

PARENT matches a strong QAEgo4D-Closed result with substantially lower query-time token use; qualitative cases show its answers depend on the specificity of retrieved caption evidence.

C. Comparison with State-of-the-Art Methods

  • PARENT reaches 56.4% accuracy, on par with the 56.0% Gemini-based result, using 3,600 query-time tokens versus 10,880 for full-memory reasoning.Its input is roughly 33% of the full-memory input; the textual-memory footprint is 66.07 KB/min, though memory measurements are not directly equivalent across methods.Source: C. Comparison with State-of-the-Art… · p. 6

D. Qualitative Examples

  • Retrieved captions support a correct scissors-location answer when evidence is explicit, but coarse evidence about a knife near the sink leads to an incorrect, less precise location.The failure predicts “in the sink” rather than the ground-truth “in the strainer.”Source: D. Qualitative Examples · p. 6

V. CONCLUSION

  • PARENT retrieves fine-grained caption subchunks and expands them to parent captions, achieving 56.4% accuracy with a substantially smaller query-time context than full-memory reasoning.Source: §V · p. 6
  • As retrieval budget increases, Hit@K and Recall@K improve while accuracy peaks at an intermediate budget because additional captions can introduce plausible distractors.Source: §V · p. 6
  • The conclusion calls for evidence-aware episodic memories that preserve temporally grounded events, uncertainty, and answer-relevant support.Source: §V · p. 6
Loading 2610.31284v1…