PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding

Ziyang Wang, Yue Zhang, Shoubin Yu, Ce Zhang, Zengqi Zhao, Jaehong Yoon, Hyunji Lee, Gedas Bertasius, Mohit Bansal

arXiv:2605.09874v2cs.CVcs.AIcs.CL

TL;DR

Week-long video benchmarks often emphasize perception or localized evidence, although visual assistants must reason across sparse observations spanning hours or days. EgoMemReason evaluates three kinds of memory using questions built from week-long egocentric recordings. Across 17 systems, the best overall accuracy is 39.6%, and the memory types show distinct bottlenecks.

  • Problem

    Existing week-long video benchmarks primarily test perception and recognition rather than reasoning that integrates evidence across multiple days, a key demand for visual assistants processing continuous experience.

    Source: §1 · pp. 1–2

  • Method

    EgoMemReason builds 500 human-verified questions across six challenges from week-long egocentric recordings, evaluating entity, event, and behavior memory.

    Source: §1 · pp. 2–3

  • Results

    39.6% is the overall accuracy of the best evaluated system, Gemini-3-Flash, among 17 systems.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Entity, event, and behavior memory have distinct bottlenecks, indicating that long-horizon memory remains a substantial open challenge.

    Source: §1 · p. 3, §5 · p. 10

    The benchmark uses recordings from six EgoLife participants, limiting individual, environmental, and behavior-routine diversity, and controlled generation may not cover all real-world long-horizon scenarios.

    Source: D Limitation · p. 23

Abstract

from arXiv · show

Next-generation visual assistants, such as smart glasses, embodied agents, and always-on life-logging systems, must reason over an entire day or more of continuous visual experience. In ultra-long videos, relevant information is sparsely distributed across hours or days, making memory a fundamental challenge: models must accumulate information over time, recall prior states, track temporal order, and abstract recurring patterns. However, existing week-long video benchmarks are primarily designed for perception and recognition, such as moment localization or global summarization, rather than reasoning that requires integrating evidence across multiple days. To address this gap, we introduce EgoMemReason, a comprehensive benchmark for week-long egocentric video understanding through memory-driven reasoning. EgoMemReason evaluates three complementary memory types: entity memory, tracking how object states evolve and change across days; event memory, recalling and ordering activities separated by hours or days; and behavior memory, abstracting recurring patterns from sparse, repeated observations over the whole week period. EgoMemReason comprises 500 questions across three memory types and six core challenges, with an average of 5.1 video segments of evidence per question and 25.9 hours of memory backtracking. We evaluate EgoMemReason on 17 methods across MLLMs and agentic frameworks, revealing that even the best model achieves only 39.6% overall accuracy. Further analysis shows that the three memory types fail for distinct reasons and that performance degrades as evidence spans longer temporal horizons, revealing that long-horizon memory remains far from solved. We believe EgoMemReason establishes a strong foundation for evaluating and advancing long-context, memory-aware multimodal systems.

1 Introduction

Week-long egocentric video requires memory across sparse observations over hours or days, but existing benchmarks largely test recognition or localized reasoning. EgoMemReason evaluates three kinds of memory with multi-day evidence and exposes substantial limits in current systems.

1 Introduction

  • Existing benchmarks often ask about a single moment or a window under ten minutes, whereas week-long videos disperse relevant information across hours or days.Dense sampling is impractical under context-length limits, and irrelevant content can overwhelm models.Source: §1 · pp. 1–2
  • 5.1 video segments and 25.9 hours of backtracking are required on average per question, each exceeding the strongest prior week-long benchmark by 2×.These figures characterize the evidence aggregation demand of EgoMemReason.Source: §1 · p. 2, Figure 2 · p. 2
  • Entity memory tracks object changes, event memory relates or orders distant events, and behavior memory abstracts patterns from repeated observations.Together, these target aggregation, relational, and inductive reasoning over accumulated experience.Source: Figure 1 · p. 2, §1 · p. 3
  • The benchmark contains 500 questions across six core challenges, generated from week-long EgoLife recordings using model-based processing and human verification.Candidates are constrained to a query timestamp so only past observations are accessible; human reviewers assess clarity, correctness, and option quality.Source: §1 · p. 3
  • 39.6% is the overall accuracy of the best of 17 evaluated systems, Gemini-3-Flash.The analysis also finds distinct bottlenecks across entity, event, and behavior memory, while denser frame sampling and auxiliary text yield no consistent improvement.Source: §1 · p. 3

2 Related Work

Prior long-video and memory benchmarks extend evaluation beyond short clips, but often focus on localized evidence, retrieval, or task-specific histories. EgoMemReason instead isolates three functional types of long-horizon video memory.

Long Video Understanding (LVU)

  • Recent long-video benchmarks extend temporal horizons and reasoning, while methods explore external memory, agentic pipelines, and attention optimization.Examples include EgoLife, MM-Lifelong, and MA-EgoQA, which differ in their treatment of memory.Source: §2 · p. 3
  • EgoLife uses short-interval visual cues, MM-Lifelong primarily evaluates retrieval, and MA-EgoQA focuses on multi-agent interaction and shared context.The passage characterizes most long-video understanding benchmarks as retrieval-centric rather than systematic studies of memory mechanisms.Source: §2 · p. 4

Memory Benchmarks in Text and Multimodal Domains

  • Text and multimodal memory benchmarks track extended histories, but do not explicitly isolate what information is stored, how it is updated, or which memory types support success.EgoMemReason distinguishes entity, event, and behavior memory as object-state trajectories, temporally grounded episodes, and patterns from repeated observations.Source: §2 · p. 4

3 Benchmark Construction

EGOMEMREASON frames week-long egocentric video understanding as memory reasoning over accumulated evidence, organizing the benchmark around three memory types and six challenges. Its construction pipeline turns week-long recordings into verified multiple-choice questions.

3 Benchmark Construction

  • The benchmark formulates week-long video understanding as reasoning over accumulated temporal evidence, rather than single-clip retrieval or surface-level pattern matching.Source: §3 · p. 4

3.1 EGOMEMREASON Design Principle

  • Entity, event, and behavior memory capture distinct functions, while multimodal evaluations rarely distinguish memory types or integrate them over long temporal horizons.Source: §3.1 · pp. 4–5

3.2 EGOMEMREASON Task Definition

  • Entity tasks track object states and count instances over time, while event tasks order and link activities across hours or days.Source: §3.2 · pp. 5–6
  • Behavior tasks infer recurring spatial preferences and activity patterns from repeated observations, including likely next states.Source: §3.2 · p. 6

3.3 Benchmark Construction Pipeline and Statistics

  • The benchmark contains 500 multiple-choice questions, built from structured video evidence through query generation, automated filtering, and human review and revision.Only 15% of initial candidates survive combined model-based filtering and human verification.Source: §3.3 · p. 7

4 Experimental Results

Across 17 systems, long-horizon memory reasoning remains difficult, and performance depends on memory type and temporal span. Ablations show that neither longer evidence horizons nor simply increasing frame input resolves the distinct bottlenecks.

4.1 Experimental Setup

  • The evaluation covers 17 general MLLMs, video-specific MLLMs, and agentic frameworks, reporting multiple-choice accuracy by capability, memory type, and overall.General and video MLLMs uniformly sample 1024 frames from the video start to the query time, downscaling when needed for context limits.Source: §4.1 · p. 8

4.2 Main Results

  • 39.6% is Gemini-3-Flash’s best overall accuracy, and no model reaches 51% on any single capability.No paradigm consistently leads; general MLLMs lead on Event and Behavior, video-specific models are competitive on Entity, and agentic frameworks generally underperform general MLLMs.Source: §4.2 · pp. 8–9
  • Entity memory is bottlenecked by fine-grained visual grounding and long-context modeling, event memory by temporal coherence, and behavior memory by abstraction over sparse repeated evidence.Even the strongest models reach only 44% on Event Ordering and Event Linking, and 50.0% on Spatial Preference Inference and 48.0% on Activity Pattern Inference.Source: §4.2 · p. 9

4.3 Ablation Studies and Analysis

  • Accuracy decreases as temporal certification length grows, with event memory declining most sharply and behavior memory declining moderately over its longer-span range.The type-specific patterns show why temporal dynamics should be considered separately rather than reported only as one aggregate trend.Source: §4.3 · pp. 9–10, Table 2 · p. 10
  • Entity accuracy improves with more frames before saturating around 256, while event memory responds least and behavior memory remains unstable across frame budgets.No single frame budget is optimal across memory types; scaling frame count alone cannot address long-horizon memory.Source: Figure 6 · p. 10, §4.3 · p. 10
  • In 100 manually inspected Gemini-3-Flash failures, missing visual information and logical mistakes each account for 32%, while incorrect visual details account for 28%.Incorrect predictions despite correct reasoning account for 8%.Source: §4.3 · p. 10

5 Conclusion

EgoMemReason evaluates week-long egocentric memory reasoning across three memory types and six challenges. The conclusion identifies distinct bottlenecks that are not solved simply by scaling input or context.

5 Conclusion

  • EgoMemReason evaluates entity, event, and behavior memory across six challenges requiring multi-hop reasoning over temporally distributed evidence.The conclusion identifies fine-grained visual grounding, long-range temporal coherence, and abstraction over sparse repeated evidence as distinct bottlenecks.Source: §5 · p. 10

Appendix

  • The appendix describes data construction, implementation details, additional analysis, and limitations and future directions.Source: Appendix · p. 17

A Data Construction Details

The benchmark is built from multi-day EgoLife recordings through a four-stage process. The pipeline prepares structured evidence, generates memory-focused questions, filters them automatically, and verifies retained items with human review.

A Data Construction Details

  • EgoLife provides continuous multi-day recordings from six participants, capturing recurring activities, evolving object states, and extended social interactions.The recordings support evaluation of temporal dynamics unfolding across days rather than minutes.Source: A Data Construction Details · p. 17

A.1 Stage 1: Evidence Preparation

  • Stage 1 creates object-centric captions for non-overlapping 30-second clips sampled at 1 FPS, then summarizes evidence at four temporal scales.Captions track object presence, attributes, location, interactions, and counts; summaries span clips, 10-minute windows, 2-hour windows, and full days.Source: §A.1 · p. 17

A.2 Stage 2: Query Generation

  • Stage 2 extracts memory-specific statements and generates multiple-choice questions with distractors drawn from the same participant’s recording.Candidate questions target entity, event, or behavior memory and are associated with a query time that limits accessible observations to the past.Source: §A.2 · pp. 17–18

A.3 Stage 3: Automatic Filtering

  • Stage 3 rejects questions answerable without video and checks that cited evidence supports the answer and predates the query time.For event-ordering questions, the evidence clips’ temporal order is also checked against the ground-truth sequence.Source: §A.3 · p. 18

A.4 Stage 4: Human Verification

  • Stage 4 combines annotator review of query context and evidence clips with quality ratings, revisions, and a full-recording expert audit.The interface collects judgments on query, correct-choice, and option quality; the expert audit checks for alternative answers, misleading distractors, and temporal violations across the multi-day recording.Source: §A.4 · pp. 18–20, Figure 7 · p. 19

A.5 Benchmark Details and Statistics

The construction pipeline uses different models for evidence preparation, question generation, automatic filtering, and human verification.

A.5 Benchmark Details and Statistics

  • GPT-5 produces dense clip-level captions and hierarchical event summaries from the raw egocentric video.Source: §A.5 · p. 20
  • GPT-5.2 generates candidate questions, while three models filter for text-only leakage and check answer correctness, distractor plausibility, and multi-timestamp grounding.The filtering models are Gemini-3.1-Pro, GPT-5.2, and Qwen-3-VL-32B.Source: §A.5 · p. 20
  • Six annotators reviewed each surviving question for about 20 minutes, and only 15% of candidates passed the final human-verification stage.Source: §A.5 · p. 20

B Additional Implementation Details

Evaluation exposes models only to video observed before each question’s designated timestamp, with required evidence often more than a day earlier. Model input settings vary by system type.

B Additional Implementation Details

  • Only video observed before the designated query timestamp tq is available, preventing use of future information.The timestamp ranges from Day 1 to Day 7, and the gap from the earliest required evidence to tq routinely exceeds one day.Source: B Additional Implementation Details · p. 20
  • MLLM experiments use the highest possible frame settings; results for Qwen-3-VL-8B and Molmo2-8B report their best overall settings.Agentic approaches use a caption model with 1FPS sampling on 30-second clips.Source: B Additional Implementation Details · p. 20

C.1 Detailed Error Analysis

Manual inspection of 100 Gemini-3-Flash examples categorizes errors and finds reasoning and visual omission among the dominant failure modes. The findings point to multiple distinct challenges in long-horizon video understanding.

C.1 Detailed Error Analysis

  • The error analysis manually inspects 100 benchmark examples and groups Gemini-3-Flash failures into four categories.The categorization is inspired by Video-Holmes (Cheng et al., 2025).Source: §C.1 · p. 20
  • The categories distinguish wrong visual details (VPE), missed information (VOE), faulty synthesis (RE), and incorrect answers despite sound reasoning (TRAW).VOE commonly reflects context-window limits on long-range reasoning; RE includes mistakes synthesizing correctly tracked information across timestamps.Source: §C.1 · p. 21
  • Reasoning and Visual Omission errors together account for 32% of errors, followed by Visual Perception Error at 28%.Think-Right-Answer-Wrong errors account for 8%; the authors say the near-equal split among the three major error types calls for advances in perception, temporal coverage, and reasoning.Source: §C.1 · p. 21

C.2 Robustness Analysis

Robustness analyses find temporal-span degradation across model scales and diminishing returns from larger frame budgets. The performance trend remains stable under an alternative filtering configuration.

Robustness to Filtering Model

  • 93 of 100 candidates are retained with an alternative filtering ensemble, and the authors report that the overall performance trend remains stable.On the corresponding items, Qwen-3-VL-32B and Gemini-3-Flash achieve 37.6% and 39.8% accuracy, versus 36.8% and 39.6% on the full benchmark.Source: §C.2 · p. 21

Cross-Model Consistency

  • Performance degrades as required temporal span increases across all three tested models, with the lowest performance on the longest-range questions.The cross-model analysis covers Qwen-3-VL-8B, Qwen-3-VL-32B, and Gemini-3-Flash.Source: Table 3 · p. 21, §C.2 · p. 21

Robustness to Input Scaling

  • Increasing input frames generally improves performance for Qwen-3-VL-8B, Qwen-3-VL-32B, and Gemini-3-Flash, but gains plateau or slightly decrease at larger budgets.The table reports cross-model performance under different input frame budgets.Source: §C.2 · p. 21, Table 4 · p. 22

C.3 Additional Benchmark Analysis

Additional analyses probe how auxiliary text, prompting, evidence access, human verification, and agentic systems affect benchmark performance. Together, they show distinct effects across memory types and multiple sources of difficulty.

Prompting strategy

  • CoT prompting cuts overall accuracy by roughly a third and reduces every per-type score; ICL is comparable to direct QA overall.ICL gains slightly on Event memory but drops modestly on Entity memory; direct QA remains near-optimal.Source: Figure 8 · p. 22, §C.3 · p. 22

Auxiliary input information

  • Auxiliary text has memory-type-specific effects: captions hurt Event Memory, transcripts benefit Behavior Memory most, and Entity Memory is largely insensitive.Overall, transcripts yield a 0.4% improvement while captions provide no benefit; the table compares transcript and caption inputs.Source: §C.3 · p. 22

Oracle-Evidence Analysis

  • With oracle evidence, Gemini-3-Flash accuracy rises from 39.6% in the full-video setting to 67.2%, while reasoning over retrieved multi-event evidence remains challenging.The analysis provides only the ground-truth evidence clips required for each question.Source: §C.3 · p. 22

Human Verification Outcomes

  • 41.8% of candidates are rejected during human review, and only 11.0% are retained unchanged.Among candidates retained after review, 70.4% have machine-generated answer supervision substantially revised.Source: §C.3 · p. 23

Query-Only Evaluation

  • With question and options only, Qwen-3-VL-32B and Gemini-3-Flash achieve 18.8% and 20.4% accuracy, near the 16.8% random baseline and below full-video results.Their full-video accuracies are 36.8% and 39.6%, respectively.Source: §C.3 · p. 23

Agentic Failure Analysis

  • Among 100 failed SiLVR examples, 35% involve retrieval misses, 30% perception errors, 32% reasoning errors, and 3% answer mismatches.The analysis concludes that agentic systems fail at multiple stages despite structured external memory.Source: §C.3 · p. 23

Caption Quality

  • Dense captions average 190.7 words, 10.2 distinct objects or entities, and 15.7 action or event verbs, versus 10.8, 1.4, and 1.2 in original annotations.The authors will release the intermediate captions, hierarchical summaries, and evidence annotations with the benchmark.Source: §C.3 · p. 23

D Limitation

The benchmark’s scope is limited by its six-participant source, controlled construction, and sampled-frame evaluation. These constraints leave broader population generalization and the full diversity of real-world scenarios outside its coverage.

D Limitation

  • Six EgoLife participants limit the diversity of individuals, environments, and behavior routines, particularly for Behavior Memory questions.The authors frame the benchmark as evaluating long-horizon evidence accumulation and reasoning, not population-level Behavior generalization.Source: D Limitation · p. 23
  • Controlled generation and filtering may leave the benchmark short of the full diversity of real-world long-horizon scenarios.Source: D Limitation · p. 23
  • Sampled frames and optional transcripts or captions may not capture all temporal dynamics in raw video streams.Source: D Limitation · p. 23
  • The authors identify broader participants and environments, richer video representations, and streaming-based evaluation as future directions.Source: D Limitation · p. 23
Loading 2605.09874v2…