PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning

Woongyeong Yeo, Kangsan Kim, Jaehong Yoon, Sung Ju Hwang

arXiv:2512.02425v2cs.CVcs.AIcs.CLcs.IRcs.LG

TL;DR

WorldMM tackles long-video reasoning limits from textual abstraction, fixed temporal retrieval, and incomplete use of visual evidence. It builds complementary textual and visual memories and adaptively retrieves across modalities and timescales, achieving strong results across five long-video benchmarks. The method also supports persistent reasoning, although deployment requires privacy and security safeguards.

  • Problem

    Existing methods struggle to fully integrate multimodal information and dynamically retrieve knowledge across varying temporal scales in complex long-video scenarios.

  • Method

    WorldMM builds episodic, semantic, and visual memories and uses an iterative retrieval agent to select relevant memory sources and temporal scopes until sufficient information is gathered.

  • Results

    WorldMM consistently outperforms strong baselines across five hour- to week-long benchmarks, with WorldMM-GPT reaching 69.5% average accuracy and exceeding the strongest baseline by 8.4%.

  • Takeaways & Limitations

    The results support multimodal, multiscale memory with adaptive retrieval as a promising approach for robust long-video reasoning.

  • Takeaways & Limitations

    Continuous accumulation of structured knowledge raises privacy and security concerns, requiring strict access controls, secure data handling, and privacy protections in deployment.

Abstract

from arXiv · show

Recent advances in video large language models have demonstrated strong capabilities in understanding short clips. However, scaling them to hours- or days-long videos remains highly challenging due to limited context capacity and the loss of critical visual details during abstraction. Existing memory-augmented methods mitigate this by leveraging textual summaries of video segments, yet they heavily rely on text and fail to utilize visual evidence when reasoning over complex scenes. Moreover, retrieving from fixed temporal scales further limits their flexibility in capturing events that span variable durations. To address this, we introduce WorldMM, a novel multimodal memory agent that constructs and retrieves from multiple complementary memories, encompassing both textual and visual representations. WorldMM comprises three types of memory: episodic memory indexes factual events across multiple temporal scales, semantic memory continuously updates high-level conceptual knowledge, and visual memory preserves detailed information about scenes. During inference, an adaptive retrieval agent iteratively selects the most relevant memory source and leverages multiple temporal granularities based on the query, continuing until it determines that sufficient information has been gathered. WorldMM significantly outperforms existing baselines across five long video question-answering benchmarks, achieving an average 8.4% performance gain over previous state-of-the-art methods, showing its effectiveness on long video reasoning.

1. Introduction

WorldMM addresses long-video reasoning challenges caused by textual memory dependence, distracting visual context, and fixed temporal retrieval. It combines complementary memories with adaptive, multiscale retrieval and performs strongly across long-video benchmarks.

  • Motivation: Existing memory-based methods rely heavily on captions or summaries, which can underrepresent visual information needed for attributes, spatial relations, and precise scene understanding.Prior multimodal approaches may use visual inputs during memory construction without fully exploiting them during inference.
  • Motivation: Fixed clip counts and durations cannot flexibly answer questions spanning seconds to hours, motivating retrieval across multiple temporal scales.WorldMM combines short- and long-range context as needed rather than always retrieving predetermined segments.
  • Contribution: WorldMM constructs separate textual and visual memories, then adaptively selects the relevant modality and temporal granularity for each query.Episodic memory stores multiscale events, semantic memory captures long-term knowledge, and visual memory preserves scene details.

2. Related Work

Long-video understanding methods improve efficiency through compression, selection, retrieval, and memory, but ultra-long videos still strain context, temporal grounding, and multimodal integration. WorldMM targets these limitations with structured memories and adaptive retrieval across temporal scales.

  • Long Video Understanding: Day- and week-long benchmarks expose scalability and long-term dependency problems that remain difficult for existing long-video understanding strategies.Massive frame counts and extended temporal dependencies motivate more efficient, context-aware approaches.
  • Retrieval-Augmented Methods: Retrieval-augmented methods focus models on relevant frames or clips, but textual representations and simple similarity scores limit their multimodal reasoning.Some approaches encode multimodal interactions with graph structures, yet the cited limitations remain.
  • Memory-Based Video LLMs: Memory-based systems organize structured knowledge from video streams, including hierarchical events, vision-centric reasoning, semantic summaries, and entity-centric long-term memory.These approaches extend retrieval toward persistent knowledge over day-long activities.
  • Memory-Based Video LLMs: Existing methods still struggle to integrate multimodal information and dynamically retrieve knowledge across varying temporal scales for complex long-video scenarios.This limitation motivates WorldMM’s multimodal, multiscale memory design.

3. WorldMM

WorldMM builds episodic, semantic, and visual memories, then uses an iterative retrieval agent to gather sufficient query-relevant information across modalities and temporal scopes. The resulting system supports fine-grained events, persistent relations, visual details, and grounded response generation.

  • Multimodal Memory Construction: WorldMM constructs episodic, semantic, and visual memories that encode complementary event, relational, and visual knowledge.The three memories are designed to address adaptive visual use and diverse temporal ranges.
  • Episodic Memory: Episodic memory represents events as knowledge graphs across ordered temporal scales, enabling reasoning from fine-grained details to long-range narratives.Each scale partitions the video into segments, captions them, and converts them into factual triplets.
  • Semantic Memory: Semantic memory continuously consolidates conceptual triplets into an evolving knowledge graph that preserves long-term relationships and habits.The consolidation process identifies overlapping or conflicting triplets before updating the graph.
  • Visual Memory: Visual memory combines feature-based segment embeddings with timestamp-indexed frames to preserve appearance, scene dynamics, and spatial context.The two modes support keyword-based scene search and direct inspection of identified timestamps.
  • Adaptive Memory Retrieval: The retrieval agent iteratively selects a memory and query, or stops when sufficient information has been collected, before response generation uses all retrieved results.It can access episodic, semantic, or visual memory and conditions later retrieval on the accumulated history.

4. Experiment Results

WorldMM consistently outperforms strong long-video baselines across five benchmarks, while ablations show complementary contributions from episodic, visual, and semantic memories and iterative retrieval.

  • Main Results: 69.5% average accuracy lets WorldMM-GPT exceed the strongest baseline by 8.4% across long video QA benchmarks.WorldMM variants also surpass corresponding base-model baselines by more than 8% on average.
  • Main Results: WorldMM achieves higher average accuracy than HippoRAG and HippoMM, at 69.5% versus 57.0% and 51.8%.The results support combining textual and visual memory with adaptive temporal-scope selection.
  • Efficacy of Multimodal Memory: Adding memories consistently improves performance, indicating that episodic, semantic, and visual memories capture complementary knowledge.The ablation study varies memory composition across benchmarks, with full-memory results following the strongest configuration trend.
  • Efficacy of Multimodal Memory: Visual memory improves the full E+S configuration by 4.2% on average, especially for object-, action-, and perceptual-understanding categories.It preserves spatial and perceptual details that text does not fully represent.
  • Efficacy of Multimodal Memory: Semantic memory raises HabitInsight accuracy to 76.9%, a 23% improvement over the E+V setting for long-term relational reasoning.It captures relational and habitual knowledge accumulated over time.
  • Efficacy of Multi-turn Retrieval: Five retrieval steps improve EgoLifeQA performance by 9.3% over single-step retrieval, while multi-scale episodic structure and semantic consolidation also improve accuracy.Using a fixed episodic timescale causes a 6.1% drop, embeddings instead of graphs cause a 4.4% drop, and removing semantic consolidation causes an approximately 7% drop in the relevant category.

5. Conclusion

WorldMM combines multimodal, multi-scale memories with adaptive iterative retrieval to reason over long videos. Evaluation across hour- to week-long videos demonstrates robust performance across long-video reasoning tasks.

  • 5. Conclusion: WorldMM integrates textual and visual memories across modalities and timescales, with an iterative retrieval agent selecting relevant information.The framework includes episodic, semantic, and visual memory components.
  • 5. Conclusion: WorldMM demonstrates robust performance across benchmarks spanning hour- to week-long videos.The conclusion presents the approach as a promising solution for long video reasoning.

Supplementary Material

The supplementary material provides additional dataset, implementation, experimental, qualitative-analysis, limitations, and broader-impact details.

  • Supplementary Material: The supplement adds dataset and implementation details, experiment descriptions and results, qualitative analyses, and discussion of limitations and broader impacts.These materials are organized across Sections A through G.

A. Additional Details on Dataset

The evaluation uses five long-video benchmarks spanning week-long egocentric recordings, daily vlogs, hour-plus videos, and varied long-video reasoning tasks. Their questions cover entities, events, habits, relationships, tasks, modalities, summarization, and temporal scopes.

  • Benchmark Overview: Five benchmarks evaluate long-video understanding across week-long, daily-vlog, hour-plus, and varied-duration recordings.The datasets include EgoLifeQA, Ego-R1 Bench, HippoVlog, LVBench, and the long subset of Video-MME.
  • EgoLifeQA: EgoLifeQA uses 44.3-hour egocentric video and tests entity recall, event recall, habits, social relations, and ongoing tasks.Its questions address recalling past events, tracking object locations, and reasoning over long-term activities.
  • Ego-R1 Bench: Ego-R1 Bench complements EgoLifeQA by targeting multistep, tool-augmented reasoning over the same week-long egocentric video.Its query types are reorganized into the EgoLifeQA category scheme.
  • HippoVlog: HippoVlog contains 25 daily vlogs and 1,000 questions spanning auditory, visual, audiovisual, and summarization requirements.The benchmark evaluates continuous audiovisual event understanding.
  • LVBench: LVBench contains 103 videos and 1,549 questions, categorized as short, medium, or long according to the video duration needed for an answer.After excluding 15 questions without segment tags, 1,534 questions remain for evaluation.
  • Video-MME: The long Video-MME subset contains 900 questions from videos longer than 30 minutes across action, attribute, object, spatial, temporal, OCR, counting, and synopsis categories.Video-MME provides 2,700 questions overall, with this experiment using only the long subset.

B. Additional Implementation Details

The implementation constructs multimodal memories from sampled frames, transcripts, captions, graphs, and retrieval prompts. Temporal resolutions and evaluation settings are adapted to the duration and structure of each benchmark.

  • Baseline Setup: Base and long-video LLMs uniformly sample video at 0.5 fps and cap inputs at 768 frames, using visual frames without captions or speech transcripts.This setting reflects context-length limitations.
  • Baseline Setup: Text-based RAG baselines build retrieval pools from captions generated for 30-second video chunks.LightRAG retrieves graph information at fine or abstract levels, whereas HippoRAG retrieves ranked raw captions.
  • WorldMM Configuration: WorldMM builds episodic memory by captioning sampled frames with Distil-Whisper transcripts at dataset-specific temporal resolutions.Week-long datasets use 30-second, 3-minute, 10-minute, and 1-hour scales.
  • Prompt Configuration: The system relies on optimized prompts for captioning, episodic and semantic extraction, memory consolidation, multiscale retrieval, and iterative reasoning.These prompts support both memory construction and final response generation.

C. Additional Description on Experiments

The experiments test dynamic temporal retrieval and isolate the contributions of WorldMM’s memory components. Evaluation compares temporal grounding, embedding, hierarchical, and keyframe-based alternatives using temporal overlap and answer accuracy.

  • Dynamic Temporal Reasoning: Dynamic temporal reasoning is evaluated with temporal grounding, embedding-based retrieval, hierarchical retrieval, and keyframe-selection approaches.Temporal overlap is measured using tIoU from returned timestamps or selected content timestamps.
  • Evaluation Coverage: The reported category-wise comparisons cover WorldMM and baselines across EgoLifeQA, Ego-R1 Bench, HippoVlog, LVBench, and Video-MME.The corresponding breakdowns are presented in Tables 7 and 8.
  • Dynamic Temporal Reasoning: The temporal grounding baseline uses Time-R1 with a modified prompt that returns evidence timestamps and grounded responses.Videos are sampled at 0.5 fps with up to 768 frames.
  • Component Ablations: Ablations disable fixed temporal scales, graph-based retrieval, semantic consolidation, or parts of visual dual retrieval to measure component contributions.Embedding retrieval replaces graph-based episodic retrieval, while visual variants use keyword-only or timestamp-only retrieval.

D. Detailed Experimental Results

WorldMM improves long-video performance across benchmark categories, with especially large gains on visually dependent questions. Its memory usage varies by modality and reasoning demand, while dynamic episodic retrieval is strongest for long temporal spans.

  • Main Results: 11.2%: WorldMM exceeds Ego-R1 on EgoLifeQA EntityRecall, while HippoVlog Aud. and A+V categories each improve by 4%.The gains are larger in categories relying on visual information than in categories without explicit visual dependence.
  • Memory Utilization: Visual, auditory, audiovisual, and summarization categories use different memory types, with semantic memory most used for long-term summarization.Figure 8 reports memory-type utilization on HippoVlog, illustrating complementary roles across reasoning demands.
  • Dynamic Temporal Scope Retrieval: WorldMM achieves its largest relative advantage on LVBench’s Long category, where answers require reasoning over more than five minutes of video.This result highlights flexible retrieval and integration across diverse temporal spans.

E. Additional Experimental Results

Additional experiments show that WorldMM remains robust across backbone choices, temporal-scale perturbations, and iterative retrieval settings.

  • E.1. Generalization to Different Backbones: WorldMM generalizes across backbone architectures, with the Gemini-based variant outperforming other configurations on EgoLifeQA.The system integrates with multiple state-of-the-art backbones without architecture-specific modifications.
  • E.2. Impact of Temporal Scales: Perturbing the episodic temporal scales preserves WorldMM’s performance on EgoLifeQA, supporting the value of multiscale design beyond specific scale choices.The tested scales were selected from empirical statistics of real-world event durations.
  • F.1. Memory Construction: Episodic and semantic triplet extraction use distinct prompts, with semantic extraction targeting abstract relationships and long-term dependencies across segments.The examples illustrate separate construction processes for episodic and semantic knowledge.
  • F.2. Multi-turn Refinement: WorldMM progressively refines retrieval across turns, expanding keywords and switching from textual evidence to timestamped video frames when visual details are needed.This process moves from narrow episodic retrieval to broader textual search and then visual retrieval.

G. Limitation and Broader Impact

WorldMM requires substantial preprocessing to construct its memories, while its persistent structured knowledge creates privacy and security considerations for deployment.

  • Limitations: WorldMM requires careful preprocessing for video captioning, triplet extraction, and semantic consolidation before long-video reasoning.These requirements are described as a broader constraint shared by existing memory-based video LLMs.
  • Broader Impact: Persistent accumulation of structured knowledge creates privacy and security concerns, requiring access controls, secure data handling, and privacy protections in deployment.The paper connects these concerns to streaming scenarios such as egocentric assistants and embodied agents.
  • System Pipeline: WorldMM’s pipeline constructs multimodal memories through captioning, named-entity extraction, episodic and semantic triplets, and adaptive memory retrieval.The supplied prompts describe caption generation, graph construction, semantic consolidation, retrieval decisions, and response generation.
Loading 2512.02425v2…