PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline

Guo Chen, Lidong Lu, Yicheng Liu, Liangrui Dong, Lidong Zou, Jixin Lv, Zhenquan Li, Xinyi Mao, Baoqi Pei, Shihao Wang, Zhiqi Li, Karan Sapra, Fuxiao Liu, Yin-Dong Zheng, Yifei Huang, Limin Wang, Zhiding Yu, Andrew Tao, Guilin Liu, Tong Lu

arXiv:2603.05484v1cs.CV

TL;DR

Existing video datasets often use densely concatenated clips, leaving sparse, natural timelines across days to months insufficiently represented. The paper introduces MM-Lifelong and ReMA, then reports that dynamic memory management outperforms current paradigms while dataset splits isolate temporal and domain biases.

  • Problem

    Existing video benchmarks often densely concatenate clips, whereas lifelong understanding requires modeling sparse observations across extended physical timelines with unobserved gaps.

  • Method

    The paper introduces MM-Lifelong, a multi-scale dataset and protocol, and ReMA, an agent that recursively manages memory over lifelong multimodal streams.

  • Results

    ReMA achieves the highest reported accuracy across sets, including 18.62% on Val@Month, and a Ref@300 score of 16.37%.

  • Takeaways & Limitations

    The results support dynamic memory management and active knowledge-base treatment as a route for handling lifelong multimodal timelines.

  • Takeaways & Limitations

    MM-Lifelong follows only one primary subject at each benchmark scale, limiting subject variety compared with multi-person datasets.

Abstract

from arXiv · show

While datasets for video understanding have scaled to hour-long durations, they typically consist of densely concatenated clips that differ from natural, unscripted daily life. To bridge this gap, we introduce MM-Lifelong, a dataset designed for Multimodal Lifelong Understanding. Comprising 181.1 hours of footage, it is structured across Day, Week, and Month scales to capture varying temporal densities. Extensive evaluations reveal two critical failure modes in current paradigms: end-to-end MLLMs suffer from a Working Memory Bottleneck due to context saturation, while representative agentic baselines experience Global Localization Collapse when navigating sparse, month-long timelines. To address this, we propose the Recursive Multimodal Agent (ReMA), which employs dynamic memory management to iteratively update a recursive belief state, significantly outperforming existing methods. Finally, we establish dataset splits designed to isolate temporal and domain biases, providing a rigorous foundation for future research in supervised learning and out-of-distribution generalization.

1. Introduction

MM-Lifelong targets lifelong multimodal understanding by evaluating sparse timelines across days to months rather than densely concatenated clips. The paper identifies context saturation in end-to-end MLLMs and introduces ReMA to manage memory recursively.

  • Motivation: Existing multimodal systems must be evaluated beyond hour-long horizons because real-world timelines extend across days or months.The paper asks how current systems perform when temporal horizons stretch beyond hours.
  • Dataset: Lifelong data differs from standard long-video datasets through high temporal sparsity, requiring models to bridge unobserved gaps between disconnected observations.The Lifelong Regime is characterized by Tspan ≫Tdur.
  • Dataset: MM-Lifelong contains 181.1 hours of footage across Day-, Week-, and Month-scale settings with temporal and domain-bias-isolating splits.Its design targets evolving narratives, concept drift, and long-term generalization.
  • Findings: End-to-end MLLMs exhibit a Working Memory Bottleneck: adding video data eventually reaches saturation and causes performance decay from noise and computational overhead.The paper describes a saturation point rather than a fundamental limit of MLLM reasoning.
  • Approach: ReMA augments MLLMs with recursive memory management, treating lifelong streams as an active knowledge base to overcome the context ceiling.The approach lets the underlying MLLM focus on deep reasoning and cross-modal alignment.

2. Related Work

Related benchmarks have expanded multimodal understanding toward longer videos and persistent memory, but remain limited in simulating continuous, sparse, lifelong streams. The paper positions MM-Lifelong as addressing this gap across multimodal temporal reasoning and agentic memory.

  • Multimodal Understanding Benchmarks: Recent video benchmarks extend understanding to hour-long durations, while cross-video reasoning remains challenging.The cited progression runs from short-term recognition toward longer and multi-video settings.
  • Benchmarks for Long-Context Memory: Text memory benchmarks support state tracking over massive contexts but lack visual dimensions, while multimodal benchmarks often use discrete images or short-term streams.These settings do not simulate continuous lifelong multimodal entropy.
  • Benchmarks for Long-Context Memory: MM-Lifelong introduces lifelong evaluations such as Needle-in-a-Lifestream and multi-hop settings for maintaining a coherent belief state over weeks of unobserved gaps.The supplied passages identify these evaluations as responses to limitations in existing benchmarks.
  • Working Memory and Architectural Compression: Long-context video research explores token compression, eviction, linear attention, hybrid architectures, and conditional memory to reduce pressure on the KV cache.These approaches seek to decouple memory footprint from sequence length or alter memory architecture.
  • Agentic Systems and Persistent Memory: Agentic systems address finite context windows through recursive reasoning, external tools, memory orchestration, and advanced retrieval mechanisms.The passage situates persistent memory and multimodal retrieval as requirements for sustained interactions.

3. Multimodal Lifelong Understanding

Multimodal Lifelong Understanding models state accumulation over continuous physical timelines represented by sparse audiovisual observations. Its defining distinction is between observed playback time and the broader chronological span containing unobserved gaps.

  • Problem Formulation: Lifelong understanding concerns accumulated state over a massive continuous physical timeline, unlike traditional short-term multimodal perception.The task is formulated over a latent visual-audio stream extending over time.
  • Problem Formulation: The dataset represents the continuous stream as clips c_i containing raw sensory data, playback duration l_i, and real-world start timestamp τ_i.The timestamp preserves each clip’s position on the physical timeline.
  • Temporal Metrics: Observational Duration Tdur sums playback lengths, whereas Physical Temporal Span Tspan measures the chronological horizon from the first start to the final end.The two metrics separate processed information from covered physical time.
  • Temporal Metrics: Artificially stitched datasets typically satisfy Tspan ≈Tdur, while Lifelong Horizon tasks satisfy Tspan ≫Tdur because unobserved gaps represent real-world time passing.Temporal sparsity is therefore a defining property of the lifelong setting.
  • Definition of the Lifelong Horizon: The Lifelong Horizon requires at least 12 hours of daily active observation, a 24-hour cross-day span, and subject-centric evolution.These constraints ensure complete daily routines, disconnected-period associations, and long-term state changes are represented.

4. The MM-Lifelong Dataset

MM-Lifelong is a multi-scale proxy dataset for lifelong multimodal understanding, combining dense and sparse temporal regimes with diverse domains and clue-grounded evaluation. Its split protocol and metrics target temporal and domain generalization while testing precise localization and reasoning across long gaps.

  • Dataset Design: MM-Lifelong approximates lifelong multimodal understanding through a multi-scale proxy dataset designed around accumulated state over a continuous physical timeline.It addresses the impracticality of collecting an entire human life by approximating lifelong properties instead.
  • Multi-Scale Design: Day and Week scales provide dense observation, whereas the Month scale introduces substantial temporal sparsity and unobserved gaps.The design uses complementary ratios of Observational Duration (Tdur) to Physical Span (Tspan) to simulate lifespan entropy.
  • Video Diversity: The dataset spans specialized gaming and egocentric settings alongside high-entropy livestreams that shift across many indoor and outdoor activities.The livestream domain includes chatting, gaming, reactions, vlogs, sports, chaotic events, and singing performances.
  • Annotation and Tasks: Clue-grounded annotations mark the video intervals containing evidence, enabling automated evaluation and scalable supervision for lifelong reasoning.The benchmark includes Needle-in-a-Lifestream tasks for fleeting details and Multi-Hop Reasoning tasks requiring aggregation across separated intervals.
  • Dataset Statistics: 181.1 hours of footage support 1289 questions and 1810 clue intervals, including 267 questions spanning 1-10 hours and 127 exceeding 10 hours.The content covers 11 question categories and 8 video clip domains.
  • Evaluation Splits and Metrics: The split protocol reserves Day- and Week-scale subsets for testing and assigns the first 30% of chronologically ordered Month-scale triplets to training.The remaining 70% are used for validation, maximizing temporal gaps and limiting temporal leakage; evaluation combines answer accuracy with Ref@N grounding.

5. Baseline: Recursive Multimodal Agent

ReMA addresses context saturation and weak localization in lifelong video by combining a global memory bank with recursive perception and control. It achieves the strongest reported results, while ablations show that recursive depth, fine-grained updates, and multimodal backbones materially affect performance.

  • Architecture: ReMA uses an offline two-phase architecture with a global Memory Bank, multimodal tools, and foundation models for iterative query answering.The architecture separates passive perception and memory consolidation from controller-driven inspection, retrieval, and answer generation.
  • Perception Phase: The perception phase segments video into clips, summarizes each clip, and incrementally consolidates summaries into a compact global representation.MMInspect produces multimodal summaries, while MemManage updates the Memory Bank.
  • Control Phase: The control phase recursively selects Answer, MMInspect, or MemSearch actions and integrates their outcomes into the Memory Bank.The controller reasons over the query, accumulated memory, and prior action history.
  • Main Results: 18.62% on Val@Month and 16.37% Ref@300 are reported for ReMA, which achieves the highest accuracy across sets and dominant grounding performance.End-to-end models reached 15.25% and 15.63% accuracy but had minimal Ref@300 grounding scores.
  • Recursive Depth: 4.86% accuracy occurs without tools, accuracy saturates at approximately 9.40% beyond three rounds, and grounding precision peaks around Round 3.Later rounds slightly reduce grounding as the agent verifies negative intervals and broadens its search.
  • Perception Granularity: 12.83% accuracy and 7.82 Ref@60 are achieved with 2-minute updates, whereas Full Video updates reduce accuracy to 3.72% and Ref@60 to approximately 0.18.Average reasoning rounds also rise from approximately 4.9 to 6.81 with the coarsest granularity.
  • Backbones: GPT-5 perception improves accuracy from 9.40% to 10.57%, while Qwen3VL-A3B reaches 7.12% versus 2.30% for text-only Qwen3-A3B.Text-only controllers terminate prematurely, with average rounds below 2.8.
  • Evaluation: GPT-5 achieves an average F1 score of 99.39 as an LLM-as-a-Judge evaluator.The result supports its reliability for the evaluated reasoning tasks.

6. Conclusion

The paper formalizes Multimodal Lifelong Understanding around sparse temporal horizons and introduces MM-Lifelong to evaluate them. Its experiments identify limitations of context scaling and standard agents, while ReMA supports dynamic memory management as an alternative.

  • Conclusion: The Lifelong Horizon is defined by distinguishing Observational Duration from Physical Temporal Span.This distinction captures temporal sparsity and disconnected gaps in long-term real-world timelines.
  • Conclusion: MM-Lifelong is a multi-scale dataset designed to challenge models with temporal sparsity and concept drift.The dataset operationalizes lifelong understanding through timelines unlike densely sampled long-video collections.
  • Conclusion: End-to-end context scaling produces a Working Memory Bottleneck, while standard agentic baselines falter on long-term disconnected gaps.The conclusion contrasts passive context extension with the demands of sparse lifelong timelines.
  • Conclusion: ReMA treats video as an active knowledge base and uses Dynamic Memory Management to bridge perception and reasoning.The paper presents this shift toward persistent memory agents as important for extended interaction.

A. Dataset

The dataset appendix documents clip metadata, annotation procedures, contamination analysis, and evaluation-oriented labeling. These details specify how temporal records and external-knowledge risks are handled.

  • Metadata: Table 8 records each clip’s start time, end time, and duration using UTC or relative timestamps by subset.Live Stream uses absolute broadcast times, while Gamer’s Journey is treated as continuous gameplay within one day.
  • QA Types: The appendix identifies definition and example tables for QA types but does not provide their contents in the cited passage.The passage points readers to Tables 12 and 13.
  • Annotation: Eight annotators labeled the three subsets, with average annotation times of 19, 33, and 51 minutes for day-, week-, and month-scale samples.Annotation cost increases with temporal length.
  • Contamination Analysis: The contamination analysis examines year-scale data and compares Gemini3-Pro-preview with and without web search on a livestream subset.The study isolates the effect of external search access during evaluation.

A.3.1. WEB SEARCH IMPACT

The web-search experiment compares offline and web-enabled Gemini3-Pro-preview on livestream evaluation. Web access substantially raises the score, but the gain exposes contamination concerns that motivate controlled closed-book and open-book settings.

  • Prompting Strategy: The prompting strategy batches multiple questions into one deep-research request and requires structured JSON answers.The example format contains indexed questions and indexed answers.
  • Results: Gemini3-Pro-preview scores 5.54 without web access and 11.79 with web search.The comparison measures performance on a livestream subset under offline and web-enabled configurations.
  • Caveat: Web-search gains create a dataset-contamination risk because livestream content may be documented online.External exposure can make genuine long-context reasoning difficult to distinguish from indirect access to ground-truth information.

A.3.2. YEAR-SCALE DATA EXPLORATION

The year-scale exploration found that annotation was prohibitively costly and that external metadata introduced contamination risks, so the year-scale was excluded from MM-Lifelong.

  • 582 GB and 236.42 hours of footage were curated from 140 match videos spanning a professional athlete’s 19-year career.
  • Annotation was nearly prohibitive because each temporal QA pair required exhaustive manual verification of clue intervals, even with semi-automatic tools.
  • External metadata accelerated annotation but created contamination risks because models could answer some questions from internalized textual knowledge or web search.
  • The year-scale was excluded because evaluation risked measuring textual retrieval rather than pure visual reasoning.

B.2.1. MEMORY IMPLEMENTATION

The memory implementation combines multimodal perception, timestamp correction, dynamic memory consolidation, and retrieval-based hierarchical summarization to support iterative video understanding.

  • Perception: MMInspect supports passive captioning and query-focused visual question answering over selected time ranges.
  • Perception: The system extracts fine-grained multimodal information from video clips, then temporally corrects timestamps to align evidence with the full-video timeline.
  • Memory consolidation: MemoryManage merges new observations with overlapping memory nodes into unified summaries, keeping the memory bank compact while retaining high-entropy updates.
  • Retrieval: MemorySearch retrieves relevant memory nodes, groups them by temporal interval, and hierarchically summarizes them for queries spanning disjoint events.
  • Control: The agent operates through an iterative THINK–ACT–OBSERVE loop that can search memory, inspect video, and finish with an answer.

C.1. Detailed Results

Detailed results report split-dependent grounding performance, a strong effect of the controller backbone, and Ref@N trends showing ReMA’s leading localization performance across most subsets.

  • Split performance: 9.91% versus 15.46%: ReMA’s inference-only grounding score is lower on Train@Month than Val@Month.
  • Controller ablation: 6.01%: replacing GPT-5 with Qwen3-VL-235B lowers the Train grounding score, highlighting weaker tool-use reasoning for complex long multimodal streams.
  • Evaluation: Ref@N converts predicted and reference intervals into fixed temporal buckets before computing overlap-based grounding.
  • Ref@N: Ref@N generally increases as N grows, and ReMA achieves the highest scores on the Day-scale, Month-scale, and Full Dataset subsets.

D.2. Examples

The examples illustrate both successful long-video reasoning and failure cases where the system overrules evidence or omits key events, with judge scores reflecting these differences.

  • GPT-5 correctly recovered the Split or Steal rules, including equal sharing, unilateral stealing, and no prize when both steal, earning score 5.
  • The whistle answer earned score 4 because it identified the crowd but added an unsupported gender detail and omitted the timing.
  • The transportation example correctly identified car and ship as the evening modes during the Chongqing trip, earning score 5.
  • The hiking example predicted 3-1-2 and denied frog jumping, but the ground truth was 3-1-4-2, yielding score 2.
  • The follower example describes recurring combat roles including dealing damage, drawing aggro, and providing heals or buffs.

E. Limitations

MM-Lifelong has two stated limitations: limited subject diversity and incomplete evaluation of how unobserved periods influence observed events.

  • Each benchmark scale follows only one primary subject, limiting subject variety compared with datasets featuring many different people.The tracked subjects are one game character, one camera wearer, and one live streamer.
  • Despite limited subject variety, experiments show poor model performance even when following a single person, supporting long-term memory as the main difficulty.The authors distinguish this challenge from simply increasing the number of subjects.
  • Current QA pairs test connections across unobserved gaps but do not specifically measure interactions between recorded and missed events.The paper identifies this interaction as an important direction for more complex future tasks.
Loading 2603.05484v1…