Source-linked AI summary
ReflectWorld-MM: An Entity-Oriented Multimodal Memory System for Open-Ended Video Streams
Xiaokang Ma, Yifan Sun, Zhihong Jin, Jie Gu, Yudong Luo, Shenyi Shao, Chu Tang, Jingmin Chen, Li Pu
TL;DR
Assistants that continuously watch the world need memory for recalling people, objects, and changes across open-ended audiovisual streams. ReflectWorld-MM builds an entity-oriented hierarchical memory system and achieves the best accuracy across six long-video and lifelong-memory benchmarks, including a 10.4-point gain over M3-Agent on M3-bench’s web split.
Problem
Assistants must accumulate continuous multimodal observations into long-term memory to answer later questions about recurring people, objects, and changing situations.
Method
ReflectWorld-MM converts streams into entity-resolved observations and organizes them in hierarchical episodic, semantic, and procedural memories.
Results
ReflectWorld-MM achieves the best accuracy on all six benchmarks, improving over M3-Agent by 10.4 points on M3-bench’s web split.
Takeaways & Limitations
The system provides persistent, revisable memory that downstream agents can access for open-ended video-stream reasoning.
Takeaways & Limitations
The system assumes an unbounded stream while requiring per-step cost that does not grow with stream duration.
Abstract
from arXiv · showhide
Building assistants that can continually watch the world, remember what they see, and reason over their accumulated experience is a long-standing goal, and recently multimodal agents equipped with long-term memory over video streams have attracted increasing interest. Unfortunately, existing systems either keep their memory inside the model context or in a flat feature store, and organize it around frames rather than around the persistent entities a stream is really about, which confines them to bounded videos and weakens their ability to track who and what reappears over time. In this paper, we propose ReflectWorld-MM, an entity-oriented multimodal memory system for open-ended video streams. It consists of three parts. The first is a perception front-end that turns an audiovisual stream into entity-resolved observations under a bounded short-term memory. The second is a hierarchical long-term memory, grounded in human memory theory, that couples a multi-scale episodic memory, an evolving entity-centric semantic memory, and a procedural memory. The third is a complete realization, built for real-world operation, that ingests arbitrary streams and plugs into off-the-shelf assistants. Across six long-video and lifelong-memory benchmarks, ReflectWorld-MM achieves the best accuracy on all six, outperforming strong memory agents and a frontier model.
1 Introduction
ReflectWorld-MM addresses the need for assistants that continuously perceive multimodal streams, accumulate long-term memory, and answer or act on that memory later. It organizes memory around persistent entities through hierarchical episodic, semantic, and procedural stores, while using past context during perception and supporting real-world deployment.
- Motivation and limitations: Existing implicit memories are typically single-scale and content-agnostic, organized by frames or tokens, which limits performance as streams extend beyond tuned video lengths.Explicit-memory systems addressed in the introduction are also described as text-only and conversation-driven, lacking video perception and persistent visual-entity resolution.
- System overview: ReflectWorld-MM converts continuous audiovisual streams into entity-resolved observations and organizes them in hierarchical episodic, semantic, and procedural memory.Episodic and semantic memory are organized around persistent entities, while procedural memory stores user rules and supports immediate response.
- System overview: Its perception front-end injects working memory, scene context, entity history, and agent-provided scene understanding before interpreting each segment.This lets past information participate in seeing the present and gives perception narrative continuity before memory is written.
- Evaluation and deployment: Across six long-video benchmarks, ReflectWorld-MM outperforms prior memory competitors and maintains rare video fallback while achieving high accuracy.The system is implemented as a database-backed service with video-source adapters, an interactive dashboard, and agent-facing APIs for arbitrary streams and agentic systems.
- Novelty: The authors identify ReflectWorld-MM as the first system combining entity-oriented multimodal memory, open-ended streaming perception, cognitively grounded hierarchy, and real-world deployment.The system code has been open-sourced.
2 Related Work
Prior video-memory systems extend models to long inputs using compressed feature, token, cache, frame, or per-video-tree memories, but these remain content-agnostic and frame- or token-organized. Explicit-memory language and multimodal agents introduce retrieval, reflection, entity-centric graphs, and multi-scale memories, establishing foundations for ReflectWorld-MM.
- Video memory: Long-video systems such as MovieChat and MA-LMM compress dense frames or tokens into sparse memories or memory banks to bypass context limits.Flash-VStream and VideoLLM-online target real-time streams, while ReKV retrieves offloaded key–value caches for streaming question answering.
- Video memory: These video-memory methods remain content-agnostic and organize memory by frames or tokens, placing it inside the model or in main memory.Related agentic methods select query-relevant frames or construct per-video trees at inference time.
- Explicit agent memory: Language-agent research develops explicit memory through context–external-tier paging, retrieval and reflection, and mechanisms for memory writing, updating, and forgetting.Examples include MemGPT, Generative Agents, A-MEM, Mem0, and MemoryBank.
- Multimodal and entity-oriented memory: Multimodal long-term-memory systems include M3-Agent’s entity-centric graph and WorldMM’s multi-scale episodic, semantic, and visual memories for long-video reasoning.Related entity-oriented approaches include HippoRAG, Zep, GraphRAG, and egocentric assistants and benchmarks.
3 ReflectWorld-MM
ReflectWorld-MM is an entity-oriented system for open-ended audiovisual streams that converts observations into a hierarchical external memory. Its design combines evidence-grounded perception, evolving episodic and semantic memories, procedural rules, and tool-based retrieval.
- System overview: The system ingests timestamped audiovisual streams online, resolves detected people and objects across time, and organizes memory around persistent entities.A natural-language query can be answered using everything observed up to the current time T.
- Perception front-end: The perception front-end segments arbitrary live sources, uses detection and re-identification evidence to ground a vision–language model, and emits entity-resolved observations.A bounded working memory preserves within-event narrative continuity, while scene and entity history let past context participate in present perception.
- Design principles: ReflectWorld-MM separates scored evidence from identity and policy decisions, assigns identity resolution to a single resolver, and retrieves memory at question-appropriate time scales and abstraction levels.This keeps decisions local and auditable while avoiding a flat log.
- Long-term memory: Episodic memory stores entity observations, trace summaries, and schema-level chapters, enabling coarse memories to index fine-grained events.The hierarchy follows the autobiographical structure of human memory and connects memory levels through shared entity identifiers.
- Long-term memory: Semantic memory consolidates durable entity knowledge with Add, Update, Delete, or no-change decisions, while importance scores strengthen facts after repeated confirmation.This makes memory evolving rather than append-only, allowing stale facts to be revised or deleted.
- Retrieval and deployment: Procedural rules guide perception through prompts, and externalized memory is queried through tools that support live-source ingestion, capture, notifications, and context management.The same interface is shared by the benchmark agent and the OpenClaw deployment.
4 Experiments
ReflectWorld-MM is evaluated across six long-video and lifelong-memory benchmarks, achieving the best accuracy on all six and especially large gains on entity-centric tasks. Additional efficiency, ablation, case-study, and deployment results show that its memory supports accurate answers with limited video fallback and depends on entity-aware organization.
- Benchmark results: ReflectWorld-MM achieves the best accuracy on all six benchmarks, including general, egocentric, and entity-centric settings.The benchmarks are VideoMME-Long, LVBench, HippoVlog, EgoLife-QA, M3-bench-robot, and M3-bench-web.
- Benchmark results: 10.4 points on the M3-bench web split and 9.1 points on the robot split separate ReflectWorld-MM from M3-Agent.On general long-video question answering, the improvement grows from 2.6 points on VideoMME to 9.0 points on LVBench.
- Memory quality: 4.6% fallback on EgoLife-QA and 6.8% on VideoMME-L accompany the best accuracy, indicating that nearly all questions are answered from extracted memory.M3-Agent has 0.0% fallback but lower accuracy, while WorldMM requires video fallback for 34.0% of VideoMME-L questions.
- Ablation study: 37.4 →33.6 on M3-bench-robot is the largest ablation drop when schema memory is removed, while entity association and semantic memory also improve both datasets.The fixed-memory answer-time ablation treats each drop as a lower bound because overlapping evidence can still be retrieved through other paths.
- Memory formation: Case studies show longitudinal entity linking consolidating durable facts and contextual perception interpreting coffee-making segments as stages of one workflow.The system also provides an interface for general agents to recall and act over long-running visual streams.
- System realization: The complete service accepts arbitrary video sources, processes open-ended streams without fixed duration, and persists memory in an indexed database for downstream agents.Supported inputs include network cameras, webcams, local files, HTTP streams, and smartphone cameras.
5 Conclusion · Appendix
ReflectWorld-MM is an entity-oriented multimodal memory system for open-ended video streams that converts audiovisual input into entity-resolved observations and stores them externally across episodic, semantic, and procedural memories. This design supports persistent, revisable, agent-accessible memory grounded in current visual evidence and accumulated context.
- 5 Conclusion: ReflectWorld-MM addresses open-ended video streams with an entity-oriented multimodal memory system.The system is presented as a memory architecture for streaming audiovisual input.
- 5 Conclusion: The system converts streaming audiovisual input into entity-resolved observations.Perception is organized around resolved entities rather than unspecified stream content.
- 5 Conclusion: It writes observations into an externalized memory organized into episodic, semantic, and procedural stores.The memory is explicitly divided into three complementary store types.
- 5 Conclusion: The externalized memory is persistent and revisable.These properties describe how stored experience can remain available and be updated over time.
- 5 Conclusion: The memory remains accessible to downstream agents.Stored information is designed for use by subsequent assistant components.
- 5 Conclusion: Perception is grounded in both current visual evidence and accumulated context.The system combines immediate visual input with information retained in memory.
A Provenance of Reported Numbers
The reported evaluation numbers combine official-code re-runs, numbers quoted from prior papers, and new reference-model runs. M3-bench results use a GPT-5-mini judge for the evaluated systems, while remaining baselines retain the original GPT-4o-judge scores.
- M3-Agent head-to-head results are re-run from its official source code under ReflectWorld-MM’s evaluation protocol.
- WorldMM’s VideoMME-Long result is re-run with official code, whereas its LVBench and HippoVlog results are quoted from the original paper.
- The GPT-5 reference row reports new runs on EgoLife-QA and M3-bench, alongside numbers quoted from prior work.
- M3-bench evaluates GPT-5, M3-Agent, and ReflectWorld-MM with a GPT-5-mini judge because the benchmark’s original GPT-4o judge is unavailable.
- The remaining M3-bench baselines retain quoted scores produced under the original GPT-4o judge and are marked accordingly.
B Detailed Benchmark Breakdowns
The evaluation spans six benchmarks covering general long-video understanding, audiovisual memory, identity-dependent lifelong recall, and open-ended questioning. Detailed breakdowns are provided for M3-bench and EgoLife-QA, with baseline and current-system scores produced under distinct sourcing protocols.
- Benchmark Coverage: Six benchmarks cover complementary video-memory settings, including long videos up to about one and two hours in VideoMME and LVBench.These general multiple-choice benchmarks test whether a memory system preserves ordinary long-video understanding.
- Benchmark Coverage: HippoVlog contains roughly one thousand audiovisual-vlog questions probing multimodal memory formation and associative recall.
- Benchmark Coverage: EgoLife-QA uses weeklong egocentric recordings for identity-dependent questions spanning entity logging, event recall, and habit insight.
- Detailed Breakdowns: Table 5 reports M3-bench results by category, while Table 6 reports EgoLife-QA results by sub-task.
- Evaluation Protocol: Earlier-system baselines come from original papers, whereas GPT-5 and ReflectWorld-MM results use the current evaluation protocol.
C Additional Implementation Details
ReflectWorld-MM uses configurable vision-language models and a vector store organized around episodic, semantic, procedural, and re-identification memories. Its deployment stack normalizes and segments streams, skips uninformative segments under hard guards, retrieves entity history on re-identification, and supports constrained agent steering.
- Models and store: Perception and consolidation use GPT-5-mini in the experiments, while text-embedding-3-small produces 1,536-dimensional memory embeddings.The system uses configurable models and supports locally installed alternatives.
- Models and store: The vector database separates collections for three episodic levels, entity-centric and per-camera semantic memories, procedural rules, and face and body re-identification galleries.The capture mode and locally installed models are configurable.
- Stream processing: After gateway normalization, streams are split at natural boundaries using visual motion and voice activity, with frames, transcripts, and speaker diarization when available.A gate skips static or silent segments, while hard guards always analyze the first segment of the stream.
- Entity history: On a re-identification hit, the entity-history layer retrieves recent episodic and semantic records and adds them to the prompt as continuity hints.This connects newly detected observations with the matched entity’s prior memory records.
- Agent steering: The vision controller combines deterministic frame-level and semantic segment-level policies, consulting the host agent for risk-relevant segments under a cooldown and per-event budget.Replies are restricted to scene descriptions, focus targets, and security rules, which are injected into subsequent perception prompts.
D Agent and Memory Access
ReflectWorld-MM answers questions through an agent that accesses its memory via tools, while the perception and memory subsystem remains independent of the agent choice. For benchmarking, a GPT-5 tool-calling question-answering agent uses three memory-store tools.
- Agent and Memory Access: The read side is an agent that answers questions by calling memory tools.The perception and memory subsystem is independent of which agent is used.
- Agent and Memory Access: For benchmarks, ReflectWorld-MM uses a tool-calling question-answering agent driven by GPT-5.
- Agent and Memory Access: The benchmark agent accesses the memory store through similarity search, item retrieval with time-ordered neighbors, and visual-evidence fetching.Similarity search operates over a chosen memory level; the get tool returns an item together with its time-ordered neighbors.
E Connecting Arbitrary Video Streams
ReflectWorld-MM connects arbitrary live or prerecorded video sources through a common gateway that normalizes them into a uniform stream for downstream processing.
- E Connecting Arbitrary Video Streams: The system supports RTSP/RTSPS network cameras, local video files, USB webcams, HTTP streams, and smartphone cameras.These inputs are handled through a common media gateway.
- E Connecting Arbitrary Video Streams: A live media gateway converts each source into a uniform stream, making the capture stage independent of the video source.This separates source-specific handling from the rest of the pipeline.
- E Connecting Arbitrary Video Streams: For smartphones, a lightweight bridge exposes the browser camera as a local WebRTC stream that the gateway can ingest.The bridge enables smartphone-camera input through the same gateway path.
F The ReflectWorld-MM Dashboard
The ReflectWorld-MM dashboard provides a live interface for viewing the video, querying the agent’s memory, inspecting timestamped entity-resolved activities, and examining evolving entity memories. It exposes the same memory used by the benchmark agent rather than a separate system.
- Dashboard interface: The dashboard combines a video source with a chat panel where the agent answers questions from memory.It was demonstrated during a live interview.
- Dashboard interface: Its middle panel shows timestamped, entity-resolved activities for each video segment.These per-segment moments organize observed activities by time and resolved entity.
- Dashboard interface: Its right panel displays each entity’s evolving semantic memory, sightings, and recent activity.The entity panel provides an entity-centric view of accumulated observations.
- Dashboard interface: The dashboard is a window onto the same memory queried by the benchmark agent, not a separate system.This links the visualization directly to the system’s operational memory.
G Qualitative Benchmark Traces
The qualitative traces show ReflectWorld-MM answering benchmark questions from memories written during perception rather than by re-watching videos. Across four tasks, it retrieves entity- and event-centered memories, expands or verifies context when needed, and grounds answers in visual or spoken evidence.
- VideoMME (long): On VideoMME (long), cockpit name–role bindings and landing evidence are retrieved to answer that Barry, the First Officer, controlled the return landing.The trace includes the operational context stored during perception and grounds the answer in the landing event.
- HippoVlog: On HippoVlog, the agent retrieves a spoken moment containing the phrase and place Lover’s Leap, then verifies the place using its panoramic background.The quoted phrase, location, and visual background were all stored during perception.
- M3-bench-robot: On M3-bench-robot, the agent expands candidate tape-use memories with adjacent context, verifies frames, and counts three distinct tape-use episodes.The trace demonstrates that coarse summaries alone are insufficient for the counting question.
- EgoLife-QA: On EgoLife-QA, the agent retrieves cart-pushing memories with a companion cue, selects the latest matching event, and grounds the location at Hema Supermarket using visual evidence.This trace tests cross-time location recall from an egocentric life log.