Source-linked AI summary
M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactions
Junyu Feng, Binxiao Xu, Jiayi Chen, Mengyu Dai, Cenyang Wu, Haodong Li, Bohan Zeng, Yunliu Xie, Hao Liang, Ming Lu, Wentao Zhang
TL;DR
Long-term multimodal personalization must handle evolving user concepts while conversations exceed context windows and static methods cannot absorb incremental changes. M2A uses cooperating agents and dual-layer hybrid memory for online updates, and experiments report substantial improvements over existing methods. The paper concludes that co-evolving memory supports individualized responses in long-term multimodal interactions.
Problem
Static personalization cannot effectively absorb users’ incremental concepts, aliases, and preferences, while long-term conversations require external memory and selective retrieval.
Method
M2A uses ChatAgent and MemoryManager with dual-layer hybrid memory linking semantic observations to immutable raw conversational evidence.
Results
44.64% average accuracy on GPT-4o-mini, surpassing RAG at 33.27%, Mem0 at 34.73%, and A-MEM at 36.26%.
Takeaways & Limitations
M2A demonstrates that editable, evidence-linked multimodal memory can support incremental personalization in long-term interactions.
Takeaways & Limitations
The user’s latent state is unobservable and can only be inferred through user queries and feedback.
Abstract
from arXiv · showhide
This work addresses the challenge of personalized question answering in long-term human-machine interactions: when conversational history spans weeks or months and exceeds the context window, existing personalization mechanisms struggle to continuously absorb and leverage users' incremental concepts, aliases, and preferences. Current personalized multimodal models are predominantly static-concepts are fixed at initialization and cannot evolve during interactions. We propose M2A, an agentic dual-layer hybrid memory system that maintains personalized multimodal information through online updates. The system employs two collaborative agents: ChatAgent manages user interactions and autonomously decides when to query or update memory, while MemoryManager breaks down memory requests from ChatAgent into detailed operations on the dual-layer memory bank, which couples a RawMessageStore (immutable conversation log) with a SemanticMemoryStore (high-level observations), providing memories at different granularities. In addition, we develop a reusable data synthesis pipeline that injects concept-grounded sessions from Yo'LLaVA and MC-LLaVA into LoCoMo long conversations while preserving temporal coherence. Experiments show that M2A significantly outperforms baselines, demonstrating that transforming personalization from one-shot configuration to a co-evolving memory mechanism provides a viable path for high-quality individualized responses in long-term multimodal interactions. The code is available at https://github.com/Little-Fridge/M2A.
1. Introduction
M2A addresses static personalization and long-context limitations by enabling incremental multimodal memory updates during interaction. Its agentic architecture combines collaborative memory management with dual-layer retrieval and a scalable multimodal data-synthesis pipeline.
- Existing personalization methods generally assume static concepts, although users continually refine concepts with new attributes, aliases, and preferences.
- Long-term conversations exceed context windows, while existing memory systems provide limited support for multimodal concepts, fine-grained updates, and editable memory structures.
- ChatAgent manages dialogue and memory access decisions, while MemoryManager performs reasoning-driven retrieval and updates across the memory bank.
- M2A introduces an agentic multimodal memory framework that supports incremental concept updates during interaction.
- Its two-tier memory architecture uses progressive narrowing for efficient and precise retrieval.
- M2A’s synthesis pipeline injects multimodal sub-sessions into long conversations for training and evaluation.
2. Related Work
Related work spans concept internalization, retrieval-augmented personalization, long-context memory, and agentic memory management. M2A combines editable multimodal updates with dual-layer, evidence-linked retrieval to address limitations in these approaches.
- Concept internalization encodes personalized visual entities into model representations, ranging from single-concept Yo’LLaVA to multi-concept and few-shot methods.
- Retrieval-augmented personalization stores profiles, aliases, and concept descriptions externally, but typically assumes a static concept set and lacks online refinement.
- M2A introduces autonomous memory updates that accumulate, refine, and correct user-specific knowledge across multi-session conversations.
- Existing long-context memory systems use external retrieval to reintroduce relevant dialogue, but commonly rely on fixed retrieval strategies.
- Existing agentic memory systems add dynamic control and persistence but are mostly text-based, single-pass, and single-granularity.
- M2A addresses these limitations with multimodal concepts and iterative reasoning-driven retrieval from semantic summaries to raw conversational evidence.
3. Problem Formulation
M2A formulates long-term personalized interaction as a POMDP with an evolving latent user state that cannot be directly observed. It approximates this state through a memory bank updated and queried by cooperating agents.
- The POMDP formulation models the user profile as a latent state that evolves through interaction.
- The latent state includes private concepts, visual preferences, and interaction styles that change over time.
- The system cannot directly observe the latent user state and must infer it from user queries and feedback.
- A multimodal observation is the user’s input query, potentially containing text and images.
- The memory bank approximates the latent state using historical observations and actions as a belief state.
- Memory evolution updates Mt from the previous memory, new observation xt, and system action at.
- Dual-layer memory stores complete observation history below and high-level semantic inferences above, while ChatAgent triggers updates and MemoryManager determines their contents.
4. M2A: Multimodal Memory Agent
M2A combines autonomous dialogue and memory-management agents with a dual-layer store that links semantic memories to raw evidence. Its hybrid retrieval and multimodal storage support progressive narrowing, cross-modal recall, and iterative memory updates, while the dataset pipeline injects concept-grounded content into long dialogues.
- Agent Collaboration: M2A separates interaction and memory management between ChatAgent and MemoryManager, enabling autonomous memory queries, response generation, and post-response updates.ChatAgent follows a three-stage Query → Generate → Update workflow, while MemoryManager performs retrieval and memory modifications.
- Dual-Layer Hybrid Memory: The dual-layer memory stores chronological raw messages alongside refined semantic memories linked to supporting evidence ids.The Raw Message Store preserves the complete conversation, while the Semantic Memory Store supports high-level knowledge and evidence-based narrowing.
- Hybrid Retrieval: Tri-path retrieval combines dense semantic matching, sparse keyword matching, and cross-modal similarity before fusing results with Reciprocal Rank Fusion.Cross-modal retrieval can use a query image or text-to-image similarity through stored visual representations and captions.
- Hybrid Retrieval: Generated visual captions make image memories retrievable through text queries, including cases where the semantic entry lacks the queried object name.The paper gives “my Corgi photo” as an example of text-to-image recall through caption similarity.
- Memory Operations: MemoryManager progressively narrows retrieval from semantic candidates to linked raw conversational segments and updates memories by adding, removing, or replacing entries.The same reasoning-driven process supports both query refinement and consistency-oriented memory maintenance.
- Multimodal Dataset Construction: The dataset pipeline groups images by concept, generates concept-grounded dialogues and QA pairs, and interpolates them into original LoCoMo sessions.This construction makes visual inputs narrative drivers within hybrid long-dialogue evaluations.
5. Experiment
M2A is evaluated on an enhanced multimodal LoCoMo benchmark against retrieval and memory baselines, with results showing broad gains and component-specific benefits.
- Experimental Setup: The enhanced LoCoMo dataset contains 10 long conversations averaging 621 turns, approximately 10k tokens, and 214 injected images.Visual-Centric questions test recall and reasoning over visual content introduced during concept sessions.
- Main Results: 44.64% average accuracy on GPT-4o-mini surpasses RAG (33.27%), Mem0 (34.73%), and A-MEM (36.26%).On Single-Hop questions, M2A improves from 44.71% for the best baseline to 56.48%.
- Main Results: 43.27% accuracy on Visual-Centric questions surpasses RAG (30.69%), while M2A remains ahead across open-source models and question categories.On Qwen3-VL-8B, M2A reaches 54.69% average accuracy versus 43.95% for the strongest baseline; on GLM-4.6V-Flash, the comparison is 56.48% versus 47.46%.
- Main Results: A-MEM underperforms Pure RAG in several categories, which the authors attribute to its text-only design and single-layer memory structure.These limitations are especially relevant to multimodal concepts and fine-grained temporal retrieval in the visually enriched benchmark.
- Ablation Study: Removing dual-layer memory, iterative retrieval, or tri-path retrieval reduces accuracy by 13.31, 16.02, and 4.10 percentage points, respectively.The ablations support evidence-linked raw-context verification, progressive narrowing, and combined semantic, lexical, and visual retrieval.
- Ablation Study: Providing no recent context severely degrades performance, while 5 turns produce substantial gains and longer windows yield only marginal improvements.The results indicate that short recent context can support effective memory operations with less additional computational overhead.
6. Conclusion
The conclusion presents M2A as an agentic multimodal memory system for incremental personalization in long-term interactions, combining editable memory, evidence-linked narrowing, and hybrid retrieval.
- 6. Conclusion: M2A addresses incremental personalization through dual-layer hybrid memory with evidence linking between semantic observations and fine-grained conversational context.ChatAgent and MemoryManager perform autonomous, reasoning-driven memory operations that adapt to conversation context.
- 6. Conclusion: Tri-path retrieval combines dense text embeddings, BM25 sparse retrieval, and cross-modal image embeddings to recall semantic, lexical, and visual cues.The conclusion reports substantial improvements over existing methods, particularly on temporally complex and visual-centric questions.
- 6. Conclusion: The paper frames co-evolving memory as a viable path toward more adaptive and individualized AI systems.
Impact Statement
The impact statement notes potential societal consequences but does not identify any that require specific highlighting.
- Impact Statement: The authors state that the work has potential societal consequences without specifically highlighting any of them.
A. Additional Related Work
Additional related work describes controllable generation and multimodal editing as complementary routes to user-specific multimodal experiences alongside memory-centric personalization.
- Controllable Generation: Controllable generation methods personalize visual synthesis through user-specified constraints and conditional generative models.Cat-DM is presented as an accelerated virtual try-on system that renders garments under user-defined constraints.
- Multimodal Editing: Multimodal editing methods support user-driven customization by using natural specifications and diverse modalities to modify or enhance garment designs.
- Cross-View Consistency: Robust multi-view virtual try-on addresses personalization across poses and viewpoints through cross-pose feature alignment and fusion.
- Complementary Paradigms: Generation-based methods encode personalization into outputs through conditional controls, whereas memory-based methods maintain explicit user representations that guide responses through retrieval.The paper characterizes the two paradigms as complementary rather than identical.
B.1. A-MEM
The baselines use retrieval-oriented memory systems that convert multimodal turns into searchable text or embeddings, without explicit long-term memory editing in the pure retrieval setup.
- A-MEM: A-MEM converts each conversation turn into a timestamped, speaker-aware memory note and captions images before appending them as text.Its memory notes include the timestamp, speaker name, and message content.
- Mem0: Mem0 ingests dialog turns with optional image captions, retrieves a fixed k = 10 memories, and generates answers from a retrieval-augmented prompt.Image captions are appended as system notes so visual information becomes searchable text.
- RAG: The pure retrieval baseline embeds all conversation turns, selects the top-k contexts by cosine similarity, and passes retrieved images as additional visual inputs.It uses default k = 5 and does not perform long-term memory updates.
C. Dataset Construction Details
The dataset pipeline generates concept-grounded multimodal sessions, distributes questions across several reasoning types, inserts sessions with interpolated timestamps, and operationalizes agent memory and judge workflows.
- Dataset synthesis: The pipeline samples 3–4 concepts and 2–3 images per concept to maintain visual consistency across generated dialogue turns.Concept groups are sampled for each target conversation from source datasets.
- Dataset synthesis: GPT-4 generates 5–6 sessions of 5–15 turns in one call, while the first message explicitly references every sampled concept.The one-call strategy is intended to maximize coherence and establishes entities through angle-bracket notation.
- QA taxonomy: The QA distribution uses a 2:3:1:4 ratio across generated categories, including multi-hop, temporal, open-domain, and single-hop questions.The supplied breakdown assigns 20% to multi-hop, 30% to temporal, 10% to open-domain, and 40% to single-hop questions.
- Temporal integration: Generated sessions are inserted between host timestamps using linear interpolation, preserving strictly increasing temporal order.For each insertion interval, the generated timestamps satisfy tstart < τ1 < ... < τNsess < tend.
- Agent workflows: ChatAgent selectively queries or updates memory for past information, long-established concepts, temporal reasoning, and information worth remembering.Its workflow combines recent context with retrieved memory before generating a response.
- Agent workflows: MemoryManager searches semantic memories, follows evidence ids to raw messages, and supports add, delete, and verification operations for updates.Progressive narrowing proceeds from broad retrieval to clusters and evidence-linked detail; contradictory information triggers deletion, replacement, and an update record.