Source-linked AI summary
EM^2Mem: Event-Centric Multimodal Memory for Large Language Models
Yijun Chen, Yaqi Zheng, Yanya Li, Boyi Xiao, Buqiang Xu, Shuofei Qiao, Jizhan Fang, Xinle Deng, Yunzhi Yao, Xuehai Wang, Liuxin Zhang, Hui Li, Huajun Chen, Shumin Deng
TL;DR
Long-video QA needs grounded access to sparse, distributed multimodal evidence, while fragment-based memories leave LLMs to reconstruct alignments during generation. EM2Mem instead binds heterogeneous evidence to event anchors during construction and retrieves compact event-level evidence. Across three benchmarks, it improves accuracy while also improving evidence recall and reducing inference cost.
Problem
Fragment-centric memories leave multimodal and temporal evidence disconnected, requiring LLMs to reconstruct alignments during generation when context and attribution are limited.
Method
EM2Mem binds captions, transcripts, keyframes, metadata, temporal context, graph relations, semantic facts, and provenance to event anchors before retrieval.
Results
EM2Mem improves average accuracy over the strongest memory baseline on three long-video QA benchmarks and reduces per-query latency and total inference tokens.
Takeaways & Limitations
Event-indexed memory provides a grounded, compact interface for retrieving aligned multimodal events rather than isolated fragments.
Takeaways & Limitations
Structured textual fields may lose fine-grained visual details, and final visually detailed reasoning still relies on selected keyframes, leaving some alignment to inference time.
Abstract
from arXiv · showhide
Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, summaries, or graph facts as isolated fragments. Although searchable, such fragments are not generation-ready: language models must reconstruct cross-modal and temporal alignments at inference time, when context is limited and attribution is difficult. We propose EM^2Mem, an event-centric multimodal memory framework that binds heterogeneous evidence to event anchors during memory construction. Each event-indexed memory cell aligns multimodal records, temporal context, graph-linked relations, semantic facts, and provenance, enabling compact evidence readout over grounded multimodal events rather than modality-specific fragments. Across three long-video QA benchmarks, EM^2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points, improves strict event-level Top-5 evidence recall by 7.0 points, and reduces per-query latency by 4.67 times and total inference tokens by 63.66% (The code will be integrated into https://github.com/zjunlp/LightMem).
1 Introduction
EM2Mem addresses the difficulty of organizing heterogeneous, temporally distributed video evidence into units that support grounded long-video QA. It binds multimodal evidence to event anchors before retrieval, yielding gains in accuracy, efficiency, and attribution.
- Long-video QA requires external memory because supporting evidence may be sparse across visual observations, speech, OCR, scenes, objects, and recurring behaviors.
- Existing systems retrieve disconnected modality-specific fragments, leaving LLMs to reconstruct cross-modal and temporal bindings during generation.
- EM2Mem uses event anchors as language-addressable indices that bind captions, transcripts, keyframes, metadata, temporal context, and provenance during memory construction.
- EM2Mem also builds episodic and semantic graphs to ground cross-event relations, temporal transitions, long-term facts, habits, preferences, and stable relations.
- 2.0%, 2.4%, and 3.7% average accuracy gains over the strongest memory baseline occur on EgoLifeQA, Ego-R1 Bench, and Video-MME (L), respectively.Per-query latency decreases 4.67 times, total inference tokens decrease 63.66%, and strict event-level Top-5 recall improves by 7.0 points over WorldMM’s five-round retrieval.
2 Background
Long-video QA involves sparse, distant, cross-modal evidence that is inefficient to process directly. Existing memory systems make such evidence searchable through modality-typed units, but these units remain fragmented rather than event-coherent.
- Long-video QA evidence is often sparse, temporally distant, and distributed across visual and textual streams, making full-video input inefficient.
- Existing multimodal memory systems convert long videos or agent experiences into searchable modality-typed units.
- These units include summaries, embeddings, source evidence, timestamps, links, clip captions, transcript context, OCR or subtitles, and normalized textual cues.
- Despite a unified storage interface, modality-specific units remain organized around individual signals or compressed observations rather than coherent multimodal events.
3 Method
EM2Mem constructs event-indexed multimodal memory cells that unify local evidence, multi-scale temporal context, and graph-linked relations before lightweight query-time readout. Retrieval selects and expands relevant cells, then compiles compact evidence for answer generation.
- Overview: EM2Mem parses long videos into short temporal events and uses event anchors to align heterogeneous evidence during memory construction.
- Overview: Figure 2’s pipeline builds memory cells with local records Ri and multi-scale context views Ci, linked by episodic and semantic graphs.
- Event-Centric Multimodal Memory Schema: An event anchor is a temporal address and shared indexing key for heterogeneous evidence and higher-level abstractions.
- Event-Centric Multimodal Memory Schema: Each memory cell stores local multimodal evidence and temporal context, while GE captures episodic relations and transitions and GS captures long-term semantic relations.
- Event-Centric Multimodal Memory Schema: A cell can bind dialogue, keyframe captions, actions, objects, and topics under one event anchor, with context views and graphs connecting related events.
- Multimodal Memory Construction: Temporal context views summarize consecutive event anchors at scales such as 3 minutes, 10 minutes, and 1 hour.
- Inference-Time Readout: At inference time, retrieval selects relevant event cells, expands them through temporal and graph-linked evidence, and compiles a query-specific view for answer generation.
4 Experiment
EM2Mem is evaluated across three long-video QA benchmarks against diverse baselines, with controlled comparisons to WorldMM and analyses of efficiency, ablations, representation, fusion, and evidence localization. It achieves higher accuracy and more efficient, event-grounded retrieval by organizing heterogeneous evidence around event-indexed memory units.
- Main Results: EM2Mem improves over WorldMM† by 2.0 points on EgoLifeQA and 3.7 points on Video-MME (L), while surpassing WorldMM by 2.4 points on Ego-R1 Bench.The reported average accuracies are 66.0 vs. 64.0, 76.8 vs. 73.1, and 67.7 vs. 65.3, respectively.
- Inference Efficiency and Scalability: EM2Mem reduces per-query latency by 4.67 times and total inference tokens by 63.66% compared with WorldMM.It reduces wall-clock evaluation time from 229,502s to 6,138s and token cost from 42.03M to 15.27M.
- Ablation Study: Removing temporal context views lowers accuracy from 66.0% to 60.4%, while removing semantic facts or episodic graph support lowers it to 61.4% or 61.6%.These ablations support combining local event records, temporal context, episodic graph relations, and semantic facts in event-indexed memory.
- Representation Analysis: Structured visual evidence with construction-time unification achieves 71.2% accuracy on the first 250 EgoLifeQA questions.Structured fields outperform raw and caption-level visual memories across both fusion stages.
- Fusion Analysis: Construction-time unification aligns captions, transcripts, and structured visual fields into shared event-level retrieval units before indexing.This design preserves cross-modal evidence whose relevance emerges only after binding and yields a largest gain of +3.2% for structured evidence.
- Evidence Localization: EM2Mem achieves 30.8% strict event-level Top-5 recall, exceeding WorldMM after five iterative retrieval rounds by 7.0 points.Its Top-1 recall reaches 23.0%, close to WorldMM 5R at 23.8%, using single-pass event-anchor retrieval.
5 Related Work
Long-video research has expanded video-language models and multimodal memory systems, but increasingly sparse, distant, cross-modal evidence motivates explicit organization and retrieval mechanisms.
- Existing approaches therefore address long-video reasoning through progressively longer contexts, structured memories, and retrieval-augmented pipelines.
- Video-language models extend long-context understanding through token compression, sparse or hierarchical memory, long-context adaptation, and temporal reasoning.
- As videos scale from minutes to hours or days, relevant evidence becomes sparse, temporally distant, and distributed across modalities.These properties motivate explicit mechanisms for organizing and retrieving video information.
- Multimodal memory systems retrieve captions, transcripts, keyframes, OCR, embeddings, or graph indices before generation.Related egocentric and agentic systems also use hierarchical memories, episode summaries, and adaptive retrieval.
6 Conclusion
EM2Mem organizes heterogeneous multimodal evidence under shared event anchors and retrieves aligned events rather than isolated fragments. The framework improves long-video QA accuracy, evidence localization, and inference efficiency, while its limitations define important deployment boundaries.
- EM2Mem organizes heterogeneous evidence under shared event anchors and retrieves aligned multimodal events rather than isolated fragments.This design targets scalable long-video question answering.
- EM2Mem improves accuracy, evidence localization, and inference efficiency for scalable long-video question answering.
- The structured memory may lose fine-grained pixel details, including small objects, colors, layouts, and subtle visual states.The authors suggest coarse-to-fine memory as a possible direction for preserving finer visual evidence.
- Final visual reasoning still relies on selected keyframes, leaving part of cross-modal alignment to inference time and potentially introducing modality bias.
- The framework is more suitable for videos processed once and queried repeatedly than for real-time scenarios.It shifts computation from inference to memory construction.
A Dataset Details
The evaluation uses three long-video QA benchmarks spanning egocentric daily-life reasoning, ultra-long egocentric reasoning, and open-domain long-video understanding. The datasets use multiple-choice questions and benchmark-specific protocols.
- The three benchmarks cover ultra-long egocentric daily-life videos and general long-form videos across distinct long-video QA settings.EgoLifeQA and Ego-R1 Bench emphasize first-person daily-life reasoning, while Video-MME (L) evaluates general long-video understanding.
- A.1 EgoLifeQA: EgoLifeQA uses week-long first-person recordings and evaluates sparse-evidence retrieval and reasoning over long video histories.The A1_JAKE stream contains 44.3 hours of video and 500 multiple-choice questions.
- A.1 EgoLifeQA: EgoLifeQA queries cover entity tracking, event recall, recurring behaviors, social relations, and ongoing tasks.
- A.2 Ego-R1 Bench: Ego-R1 Bench evaluates ultra-long egocentric reasoning with 300 multiple-choice questions and emphasizes multi-step evidence gathering.Its query types are mapped to the five-category EgoLifeQA taxonomy.
- A.3 Video-MME: Video-MME (L) evaluates open-domain long-form video comprehension using videos longer than 30 minutes and 900 multiple-choice questions.It covers diverse domains, temporal durations, and multimodal inputs.
- The experiments use publicly released benchmark datasets under their corresponding licenses and research-use terms.
B Implementation Details
EM2Mem is evaluated against broad multimodal, long-video, retrieval-augmented, and memory-based baselines using event-centered memory construction and retrieval. Its design shifts alignment work offline, producing lower repeated-query inference cost.
- B.1 Baseline Setup: The baseline suite spans base MLLMs, long-video MLLMs, RAG-based methods, and memory-based long-video reasoning systems.
- B.1 Baseline Setup: WorldMM† is reproduced under the same evaluation setting as EM2Mem to provide the most controlled comparison.Other baseline results use previously reported WorldMM results on overlapping benchmarks.
- B.2 Memory Construction: EM2Mem constructs multimodal event records from event anchors, including captions, transcripts, structured metadata, visual fields, timestamps, and provenance.It also builds multi-scale temporal context views and episodic and semantic graph evidence.
- B.2 Memory Construction: The framework uses training-free memory construction and performs inference with GPT-5-based selection and answer generation.
- B.3 Inference: During inference, EM2Mem scores event candidates using local records, temporal views, episodic graph evidence, and semantic facts before selecting five event anchors.Up to three keyframes are attached per selected anchor for final answer generation.
- B.4 Efficiency: Offline construction tokens are incurred once and amortized across repeated queries, while event-indexed memory reduces per-query inference cost.
C.2 End-to-End Cost and Amortization Analysis
EM2Mem shifts computation to offline memory construction, increasing build cost but substantially reducing inference cost. The added construction overhead is amortized for repeated questioning, with wall-clock benefits appearing before token savings.
- 3.03× end-to-end wall-clock speedup results from higher offline construction cost but substantially lower inference cost.Construction rises from 88,708s to 99,022s, while inference falls from 229,502s to 6,138s.
- Approximately 29 queries amortize the additional construction cost using average latency, or approximately 24 queries using practical evaluation throughput.EM2Mem saves 360.79s per query under the latency calculation.
- Approximately 536 queries are required to break even on token usage under an unweighted token-count calculation.Memory construction uses 28.68M more tokens but inference saves 26.76M tokens on the 500-query evaluation.
C.3 Detailed Ablation Study
The full EM2Mem model outperforms variants that remove temporal context views, semantic memory, episodic graph evidence, or fine-grained event records. The ablations indicate that different memory components support different temporal and relational reasoning demands.
- 2.0–5.6 points separate full EM2Mem from all four ablated variants in overall accuracy.The compared variants remove temporal context views, semantic memory, episodic graph evidence, or all but 30-second event records.
- Removing temporal context views causes the largest overall drop, highlighting their role in connecting sparse evidence across long videos.
- Removing semantic memory or episodic graph evidence degrades performance, indicating useful long-term semantic support and event-level relational links.
- The 30s-only variant performs best on EntityLog and EventRecall but is weaker on categories requiring long-range context.The latter categories require temporal context, episodic graph structure, and semantic memory.
C.4 Analysis of Construction-Time Alignment
Construction-time alignment improves evidence localization and coverage when compared under matched retrieval conditions. The controlled comparison separates this alignment benefit from auxiliary memory components shared with prior systems.
- 5.1 and 3.4 points are the R@1 and R@5 gains from construction-time aligned retrieval over independent four-channel retrieval.The comparison uses the same underlying records, evidence, retrieval and answer models, prompts, filtering, and temporal budgets.
- 5.4 points higher coverage at the primary 300-second budget is statistically supported by a 95% CI of [+2.6, +8.3] and McNemar exact p = 0.0003.
- Final QA accuracy increases from 60.0% to 63.0%, but the difference is not statistically significant.The reported 95% CI is [-0.8, +6.8] with p = 0.1505.
- Binding heterogeneous evidence into a common retrieval unit before indexing improves evidence retrieval under matched conditions.
C.5 Category-wise Evidence Unification Analysis
Category-wise analyses show that evidence unification benefits long-range, relational, and task-level questions most clearly, while localized recall can rely on fine-grained records or iterative retrieval. Fixed 30-second anchors provide a robust localization–budget trade-off on EgoLifeQA without being universal semantic event boundaries.
- Category-wise Evidence Unification Analysis: Construction-time unification yields the largest gains on RelationMap and TaskMaster, which require connecting people, actions, tasks, and temporally distant evidence.
- Category-wise Evidence Unification Analysis: Structured event records perform strongly on EntityLog, whereas raw visual evidence performs best on HabitInsight.The design therefore uses structured records for primary retrieval and raw keyframes for targeted visual verification.
- Selector Budget: 30.8% strict Top-5 event-level recall exceeds WorldMM’s five-round result by 7.0 points, while EM2Mem’s 23.0% Top-1 recall is close to 23.8%.EventRecall is the only category where EM2Mem is slightly lower, suggesting localized questions may benefit from iterative caption-level retrieval.
- Anchor Granularity and Boundary Sensitivity: Fixed 30-second anchors achieve the highest coverage at all budgets and outperform alternatives by 3.4–12.8 points at 300 seconds.Identity-based adaptive boundaries produced frequent fragmentation under egocentric motion, occlusion, and re-entry.
- Anchor Granularity and Boundary Sensitivity: 30-second anchors offer a robust localization–budget trade-off on EgoLifeQA without implying a universal semantic event boundary.
- Qualitative Case Study: A TaskMaster case study shows event-centered records preserve person–plan relations and provenance that loosely connected flower-related snippets fail to bind.The retained evidence identifies Katrina through cultivation context and an explicit planting statement.