PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory

Lin Long, Yichen He, Wentao Ye, Yiyuan Pan, Yuan Lin, Hang Li, Junbo Zhao, Wei Li

arXiv:2508.09736v4cs.CV

TL;DR

Multimodal agents need to continuously perceive, remember, and reason over long-term experience, but existing approaches face scalability and consistency gaps. M3-Agent addresses these gaps with entity-centric episodic and semantic memory plus iterative reasoning and retrieval, and it is evaluated using M3-Bench. It outperforms the strongest prompted baseline across the reported benchmarks.

  • Problem

    Existing multimodal-agent and long-video methods struggle to scale to arbitrarily long streams while maintaining consistent entities, events, and world knowledge for memory-based reasoning.

  • Method

    M3-Agent continuously processes video and audio to build entity-centric episodic and semantic memories, then uses reinforcement-learned multi-turn reasoning with iterative memory retrieval to execute instructions.

  • Results

    M3-Agent trained via reinforcement learning outperforms Gemini-GPT4o-Hybrid by 6.7%, 7.7%, and 5.3% accuracy on M3-Bench-robot, M3-Bench-web, and VideoMME-long, respectively.

  • Takeaways & Limitations

    M3-Bench provides a long-video evaluation of memory effectiveness and memory-based reasoning across robot and web videos involving person understanding, knowledge extraction, and cross-modal reasoning.

  • Takeaways & Limitations

    Spatial reasoning remains constrained because verbal memory is less effective than visual memory, motivating richer visual content such as snapshots.

Abstract

from arXiv · show

We introduce M3-Agent, a novel multimodal agent framework equipped with long-term memory. Like humans, M3-Agent can process real-time visual and auditory inputs to build and update episodic and semantic memories, gradually accumulating world knowledge. Its memory is organized in an entity-centric, multimodal manner, enabling deeper and more consistent understanding of the environment. Given an instruction, M3-Agent autonomously performs multi-turn reasoning and retrieves relevant memories to complete tasks. To evaluate memory effectiveness and memory-based reasoning in multimodal agents, we develop M3-Bench, a long-video question answering benchmark comprising 100 newly recorded robot-perspective videos (M3-Bench-robot) and 920 diverse web-sourced videos (M3-Bench-web). We annotate QA pairs designed to test capabilities essential for agent applications, such as person understanding, general knowledge extraction, and cross-modal reasoning. Experimental results show that M3-Agent, trained via reinforcement learning, outperforms the strongest baseline, a prompting agent using Gemini-1.5-pro and GPT-4o, achieving 6.7%, 7.7%, and 5.3% higher accuracy on M3-Bench-robot, M3-Bench-web and VideoMME-long, respectively. Our work advances multimodal agents toward more human-like long-term memory and provides insights for their practical design. Model, code and data are available at https://github.com/bytedance-seed/m3-agent.

1 Introduction

M3-Agent combines continuous multimodal memorization with instruction-driven reasoning and retrieval over long-term memory. The paper introduces M3-Bench to evaluate memory-based reasoning and reports consistent gains over prompted baselines.

  • M3-Agent: Episodic memory stores concrete observed events, whereas semantic memory captures generalized knowledge extracted from video.Examples include Alice taking coffee and preferring coffee in the morning.
  • M3-Agent: During control, M3-Agent uses reinforcement learning for multi-turn reasoning and iterative retrieval instead of single-turn RAG.It retrieves relevant information across dimensions such as events or characters.
  • Motivation and novelty: The framework addresses arbitrarily long multimodal streams and constructs entity-centric world knowledge to improve long-term coherence.This extends conventional long-video description beyond low-level visual details.
  • M3-Bench: M3-Bench contains 100 robot-perspective videos and 920 web videos, with 1,276 and 3,214 annotated QA pairs respectively.The benchmark targets person understanding, general knowledge extraction, and cross-modal reasoning.
  • Results: 6.7%, 7.7%, and 5.3% higher accuracy is achieved over Gemini-GPT4o-Hybrid on M3-Bench-robot, M3-Bench-web, and VideoMME-long, respectively.The reinforcement-learning-trained M3-Agent outperforms all baselines on all three benchmarks.
  • M3-Agent: M3-Agent continuously processes real-time video and audio, builds episodic and semantic memories, and reasons over them to complete complex instructions.Its long-term memory is organized around entities and multimodal information.

2 Related Work

Prior multimodal-agent memory systems must handle richer inputs while preserving entity and event consistency over long horizons. Existing online video approaches improve scalability but remain limited by finite context, visual-feature storage, or persistent consistency challenges.

  • Long-Term Memory of AI Agents: Multimodal agents store richer multimodal content than LLM agents and must maintain consistency while forming world knowledge from experience.The goal is more than storing descriptions of experience.
  • Online Video Understanding: Context-window extension and visual-token compression do not scale effectively to infinitely long video streams.Online video understanding requires real-time processing and decisions based on past observations.
  • Online Video Understanding: Memory modules that store encoded visual features improve online-processing scalability but struggle to track human identities and evolving events coherently.Their fundamental limitation is maintaining long-term consistency.
  • Online Video Understanding: Language-based video descriptions improve scalability, yet Socratic Models still face consistency challenges across complex, evolving video content.These methods use multimodal models to generate descriptions as memory.

3 Datasets

M3-Bench evaluates multimodal agents’ long-term-memory reasoning through robot-perspective and diverse web videos, with questions targeting higher-level cognitive abilities. Its collection and annotation procedures emphasize realistic, varied, objective, and temporally grounded QA.

  • Benchmark composition: M3-Bench contains 100 robot-perspective videos and 920 web-sourced videos covering diverse content and scenarios.The robot subset uses real-world first-person recordings, while the web subset broadens content and scenario diversity.
  • Evaluation scope: The benchmark targets person understanding, general knowledge extraction, cross-modal reasoning, and reasoning over long-term memory.These capabilities extend beyond the visual action, spatial, and temporal understanding emphasized by many existing long-video QA benchmarks.
  • M3-Bench-robot: M3-Bench-robot scripts span seven everyday scenarios and involve one robot interacting with two to four humans.Scenarios include living room, kitchen, bedroom, study, office, meeting room, and gym settings.
  • M3-Bench-robot: Each robot script contains at least 70 events and at least 15 questions labeled with reference events and question types.Reference events provide information needed to answer later questions and support videos lasting at least 30 minutes.
  • Video acquisition: Robot-perspective videos are simulated by human actors using head-mounted cameras, with recordings collected from 67 actors across 51 locations.The setup preserves egocentric visual and auditory input while reducing the operational costs and complexities of filming with actual robots.
  • Annotation: Robot annotations include QA timestamps preceding the relevant response or action, plus subtitles with dialogue timing, speaker identity, and transcription.Annotators review, revise, or discard scripted questions when filmed content differs from the script, and add questions to reach at least 12 QA pairs per video.
  • M3-Bench-web: Web videos are selected through a question-driven process and annotated with specific, objective QA pairs designed to have single unambiguous answers.The final web collection covers 46 video types, and its question timestamps are set to the end of each video.

4 Approach

M3-Agent combines continuous multimodal memorization with instruction-driven control over an entity-centric long-term memory. It stores episodic and semantic information, maintains cross-modal entity consistency, and uses iterative retrieval during multi-turn reasoning.

  • Architecture: M3-Agent runs parallel memorization and control processes: one builds lifelong memory from arbitrarily long video streams, while the other executes instructions using that memory.Memorization continuously processes incoming multimodal inputs; control reasons over stored information to complete tasks.
  • Long-Term Memory: Long-term memory is an entity-centric multimodal graph whose nodes store memory items with modalities, raw content, weights, embeddings, timestamps, and metadata.Undirected edges represent logical relationships, including links between items sharing an entity ID.
  • Long-Term Memory: M3-Agent incrementally adds or reactivates text, image, and audio nodes and edges, increasing weights for repeated entries to resolve conflicting information.During inference, frequently activated entries override conflicting entries with lower weights.
  • Memorization: Clip-by-clip memorization produces episodic memories of observed audiovisual events and semantic memories containing identities, attributes, relationships, and world knowledge.Semantic memories also provide retrieval cues for the control process.
  • Consistent Entity Representation: Persistent face and voice identifiers connect character information across clips, while cross-modal reasoning can link face and voice nodes belonging to the same person.The globally maintained memory graph supports coherent identity mapping across local memories.
  • Control: During control, the policy alternates reasoning, search actions, and answers for up to H rounds, retrieving memories from multiple perspectives such as people or events.Search results are appended to context for subsequent rounds, enabling iterative memory retrieval rather than single-turn RAG.
  • Training: Reinforcement learning optimizes separate policy models for memorization and control, initialized with Qwen2.5-Omni for multimodal processing and Qwen3 for reasoning.Training data comprise 500 long videos, 26,943 30-second clips, and 2,736 question-answer pairs; memory synthesis additionally uses 10,952 samples.

5 Experiments

M3-Agent is evaluated against online video understanding systems, prompted multimodal agents, and retrieval-based baselines across three long-video benchmarks. It consistently outperforms baselines, with results emphasizing long-term memory quality, reinforcement-learned control, and challenges in fine-grained and spatial reasoning.

  • Baselines: M3-Agent is compared with online video understanding frameworks, prompted Gemini/GPT-4o agents, and Socratic Models using retrieval-augmented generation.The evaluation includes MovieChat, MA-LMM, Flash-VStream, Gemini-Agent, Gemini-GPT4o-Hybrid, and multimodal memory-generation variants.
  • Main Results: M3-Agent outperforms all baselines on M3-Bench-robot, M3-Bench-web, and VideoMME-long.Compared with the strongest baseline, improvements are 6.3% on M3-Bench-robot, 7.7% on M3-Bench-web, and 5.3% on VideoMME-long.
  • Question-Type Results: M3-Agent improves human understanding by 4.2% and cross-modal reasoning by 8.5% on M3-Bench-robot over MA-LMM.On M3-Bench-web, its gains over Gemini-GPT4o-Hybrid are 15.5% and 6.7% in the respective categories.
  • Ablation Study: DAPO-trained control-32b-rl improves over control-32b-prompt by 10.0%, 8.0%, and 9.3% across the three benchmarks.Removing inter-instruction or reasoning produces further accuracy decreases, showing that both designs are important in the evaluated control process.
  • Case Study: Fine-grained-detail and spatial-reasoning questions remain hard because exhaustive memory is impractical and verbal memory is less effective for spatial information.The paper suggests selective memorization, task-specific world knowledge, and richer visual content such as snapshots as directions for addressing these cases.

6 Conclusion and Future Work

The paper concludes that M3-Agent combines multimodal perception, episodic and semantic memory, and autonomous retrieval-based reasoning, while M3-Bench evaluates these abilities in realistic long-video settings. The benchmark construction uses robot-perspective videos, scripted events, and multi-stage annotation and quality control.

  • Conclusion: M3-Agent processes real-time video and audio to build episodic and semantic memories, accumulate world knowledge, and retrieve information for instruction completion.Its evaluation covers human understanding, knowledge extraction, and cross-modal reasoning on M3-Bench-robot, M3-Bench-web, and VideoMME-long.
  • Benchmark Construction: M3-Bench-robot uses robot-perspective videos in practical environments with events and questions designed to evaluate memory.The robot is played by a human actor, with scenarios including household and other indoor settings.
  • Annotation: Scripts contain at least 70 events and videos are typically 30–45 minutes long, with questions linked to reference events and timestamps.Questions are reviewed for answerability using the video content up to the relevant point and may be revised when necessary.
  • Annotation: Questions are designed to avoid dialogue-only, commonsense-only, weakly related, or unverifiable answers.The guidelines require clear, objective answers and prohibit questions that cannot be meaningfully answered from the video.
  • Quality Control: Annotation quality is checked through trial tasks, reviewed batches, revision, and qualified-rate thresholds below which additional inspection occurs.Two authors participate in quality control, and annotators proceed through staged review procedures.
  • Quality Control: Five annotators answer each M3-Bench-robot question, yielding a final human accuracy of 90.7%.The most common human errors are counting-related problems.

A.5 Data Examples

The data examples illustrate scripted interactions, object placement, and questions grounded in reference events. Annotation guidelines emphasize objective, multimodal, nonredundant questions whose answers require sufficient video context.

  • Script Examples: Script examples describe interpersonal interactions, gifts, reminders, and household object-placement tasks.Examples include Rose photographing tulips, explaining a teddy bear gift, and asking the robot to place clothes on a wardrobe shelf.
  • Question Design: Questions must be objectively verifiable and avoid open-ended, compound, or multiply answerable formulations.Each video must also include questions targeting character attributes and commonsense reasoning.
  • Question Design: Questions must remain answerable at resolutions of 720p or below and cannot all be answerable using audio alone.The guidelines require a reasonable proportion of vision-centric questions.
  • Question Design: Questions should require more than 10 seconds of video context and must not be redundant within a video.These constraints are intended to prevent short-clip and repetitive questions.
  • Quality Control: Quality control uses three stages of trials, batch review, feedback, revision, and 90% qualified-rate thresholds.Two authors oversee the quality-control process, with annotators drawn from a commercial data annotation company.

C Implementation Details of Tools

The implementation extracts face, voice, and textual representations before organizing and retrieving them through modality-specific memory operations. Identity matching updates entity representations over time, while textual and clip-level retrieval select relevant memory.

  • Representation Extraction: Facial recognition samples frames at 5 frames per second, extracts facial attributes, filters low-quality detections, and clusters embeddings.The extracted attributes include bounding boxes, identity embeddings, and detection or quality scores.
  • Representation Extraction: Voice identification segments speakers, performs ASR, filters segments shorter than 2 seconds, and clusters voice embeddings by character.The process uses Gemini-1.5-Pro for audio segmentation and ERes2NetV2 for voice embeddings.
  • Search: Memory retrieval uses Maximum Inner Product Search with modality-specific adaptations.Face and voice nodes are matched using cosine-similarity thresholds of 0.3 for images and 0.6 for voice.
  • Search: Textual retrieval applies top-k similarity search over text nodes and ranks clips by their highest-scoring memory entry.The text-embedding-3-large model is used for textual memory embeddings.

D Demonstration Data Synthesis for Memorization

The memorization data pipeline synthesizes episodic and semantic memories from multimodal video clips, including cross-modal face–voice identity relationships. It combines GPT-4o and Gemini-1.5-Pro with progressive meta-clip annotation to create training data.

  • Data synthesis: Memorization inputs include video, audio, facial identifications, and voice identities, producing episodic and semantic memory outputs.Training videos are segmented into 30-second clips, yielding 10,752 synthetic training samples and 200 validation samples.
  • Episodic memory synthesis: GPT-4o first generates detailed visual descriptions from frames sampled at 0.5 fps, after which Gemini-1.5-Pro synthesizes episodic memory using multimodal context.The hybrid design combines GPT-4o’s fine-grained visual detail with Gemini-1.5-Pro’s audio support and event-level descriptions.
  • Episodic memory synthesis: The episodic-memory prompt requires descriptions of characters, actions, dialogue, behavior, and environmental or temporal cues, while correcting reference errors and incorporating video, face, and audio evidence.Outputs use feature IDs when available and concise descriptors otherwise, and are formatted as lists of English sentences.
  • Entity ID relationship detection: Cross-modal identity annotation identifies meta-clips containing exactly one face ID and one voice ID, treating them as high-confidence evidence that the identities belong to the same person.Short clips are limited to five seconds and selected for visual stability before face and voice recognition are applied.
  • Entity ID relationship detection: A voting-based meta-dictionary maps voice IDs to face IDs across the long video, then annotates valid 30-second clips with equivalence relations.The resulting dataset contains 10,952 valid clips; manual review of 48 mappings found 95.83% accuracy.
  • Semantic memory synthesis: Semantic memory is synthesized by prompting GPT-4o for preliminary content from frames and episodic memory, then asking Gemini-1.5-Pro to produce the final version.This hybrid process targets general knowledge and entity relationships beyond the raw event description.

D.4 Quality of the Synthetic Data

The synthetic memorization data is reported as detailed and accurate, while speaker-recognition errors remain the main identified source of mistakes.

  • Accuracy: 95.5% of 353 manually reviewed memory items were accurate across 10 randomly sampled clips from different videos.The review covered synthetic memory generated for the memorization data.
  • Detail: Synthetic episodic and semantic memories averaged 245.7 and 276.2 words, exceeding Gemini-1.5-Pro’s corresponding averages of 151.3 and 81.4 words.The authors report these differences as evidence that the synthetic memories capture more detail.
  • Error sources: Most observed errors originated from the speaker-recognition tool, with background noise and overlapping speech identified as contributing conditions.The passage introduces this as the main error source in the manual review.

E Evaluation of Memorization

Memorization models are evaluated on held-out data using separate measures for description quality and cross-modal identity equivalence.

  • Evaluation setup: Evaluation uses a held-out validation set of 200 samples, with the best checkpoint selected during training.The evaluation separates memory description quality from identity-equivalence performance.
  • Description quality: AutoDQ assesses episodic and semantic memory description quality by comparing generated outputs with reference descriptions, excluding identity equivalence.This metric evaluates the descriptive components of memorization.
  • Identity equivalence: Identity equivalence is evaluated against validation ground truth using precision, recall, and F1 score.The table defines P, R, and F1 as precision, recall, and F1 score.

F.1 Details of DAPO Training

DAPO training uses the reported hyperparameters and exhibits steadily improving reinforcement-learning scores over training steps.

  • Training curves: Figure 4 shows a steady increase in reinforcement-learning score as training steps progress.The figure is described as the training-curve summary for DAPO.
  • Hyperparameters: Table 14 reports the hyperparameters used during DAPO training.The passage identifies the table as the source of training-process hyperparameters.

F.2 GRPO Training

GRPO is used to optimize the policy model in the ablation study, with specified clipping and KL-penalty hyperparameters.

  • The ablation study uses Group Relative Policy Optimization (GRPO) to optimize its policy model.
  • GRPO optimizes the policy model πθ by maximizing a defined training objective.
  • ϵ and β are set to 0.2 and 0.01, respectively, while other hyperparameters match DAPO training.

G Case Study

The case studies illustrate multimodal memory generation and iterative control across household interactions and a technology demonstration, including visual, auditory, episodic, and semantic information.

  • Control and training: Table 17 presents a complete control-generation trajectory, while Figure 4 plots training-set scores and dev-set accuracy during DAPO training.
  • Family scene: The case study describes a family setting with outdoor activities, sibling dynamics, children, ducklings, and a father spending time with his family.
  • Memory generation: The memory examples combine faces, voices, settings, actions, relationships, and inferred characteristics from multimodal video content.
  • Household interaction: The household interaction captures boredom, disagreement over puzzles, alternative activity suggestions, and a familiar relationship between participants.
  • Technology demonstration: The technology demonstration records Tomasz’s engineering role, experimental approach, interest in personal aerial flight, and an optimistic narrative about successful testing.
Loading 2508.09736v4…