PaperlayerView exact revision on arXiv

Source-linked AI summary

Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents

Yuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao, Zhining Liu, Xiao Lin, Yada Zhu, Hendrik Hamann, Jingrui He, Hanghang Tong

arXiv:2601.03515v1cs.CLcs.AI

TL;DR

Existing benchmarks do not adequately evaluate how multimodal memory is preserved, reasoned over, and updated across long-term conversations. Mem-Gallery addresses this gap with a multi-session vision–language benchmark and a three-dimensional evaluation framework, while benchmarking reveals the importance of explicit visual retention and memory organization alongside reasoning, knowledge-management, and efficiency limitations.

  • Problem

    Existing benchmarks either evaluate multi-session memory without visual information or assess multimodal understanding only in localized contexts, leaving long-term multimodal memory underexplored.

  • Method

    Mem-Gallery builds multi-session conversations grounded in visual and textual information and evaluates extraction and adaptation, reasoning, and knowledge management.

  • Results

    Benchmarking thirteen memory mechanisms finds that explicit visual retention and principled organization are beneficial, while reasoning, knowledge updates, conflict handling, and efficiency remain limited.

  • Takeaways & Limitations

    Future multimodal memory design needs principled organization, selective retrieval, and robust maintenance.

  • Takeaways & Limitations

    The benchmark focuses on vision–language conversational settings and does not explicitly cover audio, embodied signals, planning, or tool use.

Abstract

from arXiv · show

Long-term memory is a critical capability for multimodal large language model (MLLM) agents, particularly in conversational settings where information accumulates and evolves over time. However, existing benchmarks either evaluate multi-session memory in text-only conversations or assess multimodal understanding within localized contexts, failing to evaluate how multimodal memory is preserved, organized, and evolved across long-term conversational trajectories. Thus, we introduce Mem-Gallery, a new benchmark for evaluating multimodal long-term conversational memory in MLLM agents. Mem-Gallery features high-quality multi-session conversations grounded in both visual and textual information, with long interaction horizons and rich multimodal dependencies. Building on this dataset, we propose a systematic evaluation framework that assesses key memory capabilities along three functional dimensions: memory extraction and test-time adaptation, memory reasoning, and memory knowledge management. Extensive benchmarking across thirteen memory systems reveals several key findings, highlighting the necessity of explicit multimodal information retention and memory organization, the persistent limitations in memory reasoning and knowledge management, as well as the efficiency bottleneck of current models.

1 Introduction

Mem-Gallery addresses the underexplored problem of evaluating multimodal long-term conversational memory, where visual and textual information evolve across sessions. It introduces a multi-session multimodal dataset and evaluates memory extraction and adaptation, reasoning, and knowledge management, revealing persistent reasoning, organization, and efficiency challenges.

  • Motivation: MLLM agents need to store, retrieve, integrate, and update visual and textual information across extended conversations.Conversational settings provide a natural testbed because agents acquire, refine, and use memory over multiple sessions.
  • Research gap: Existing benchmarks separate text-only multi-session memory from short-horizon multimodal understanding, leaving multimodal memory evolution and management insufficiently evaluated.These benchmarks generally omit visual information or lack cross-session structure and information evolution.
  • Benchmark: Mem-Gallery introduces multi-session conversations grounded in images and text, with information incrementally introduced, referenced, and updated over long interaction horizons.The benchmark is designed to reflect both daily and domain-specific knowledge.
  • Findings: Benchmarking thirteen memory mechanisms shows benefits from explicit visual retention and principled organization, alongside limitations in reasoning, knowledge updates, conflict handling, and efficiency.The reported findings identify larger storage and retrieval overhead as a potential deployment bottleneck.
  • Evaluation framework: The evaluation framework covers memory extraction and adaptation, memory reasoning, and memory knowledge management as three functional dimensions.These dimensions correspond to acquiring usable memory, reasoning over evolving multimodal evidence, and regulating dynamic or inconsistent memory states.

2 Related Works

Prior conversational-memory benchmarks are structurally misaligned with multimodal long-term memory evaluation. They typically omit visual information, lack multi-session evolution, or provide multimodal contexts too localized to test persistent memory.

  • Existing benchmarks: Most established memory benchmarks, including LoCoMo, LongMemEval, and MemoryAgentBench, are text-only and do not evaluate multimodal capabilities.LoCoMo combines visual information with multi-session structure but offers restricted multimodal memory functionality.
  • Existing benchmarks: Localized multimodal benchmarks introduce vision but generally focus on single-session or short-context understanding rather than long-term conversational memory.This creates a mismatch with cross-session memory requirements.
  • Benchmark gap: Prior benchmarks often overlook visual information, lack multi-session structure, or fail to support multimodal memory-function evaluation.Table 1 summarizes these structural gaps across representative conversational benchmarks.
  • Multimodal effectiveness: In LoCoMo, adding visual information yields marginal or inconsistent gains, suggesting its tasks can often be solved without visual clues.The comparison motivates benchmarks with genuinely necessary visual information.
  • Long-term agent memory: Long-video multimodal memory benchmarks differ fundamentally from conversational memory because they do not model multi-round interactions and incrementally regulated information across sessions.The missing setting is especially relevant to personalized assistants.

3 Mem-Gallery Benchmark

Mem-Gallery constructs coherent multi-session, vision–language conversations and generates taxonomy-driven evaluation questions under a unified memory protocol. Its task families test extraction and adaptation, multimodal reasoning, and knowledge management over evolving conversational histories.

  • Benchmark design: Mem-Gallery is organized around benchmark construction, a unified conversational environment, and an evaluation framework with task taxonomies.The overall pipeline presents these three components as the benchmark’s core design.
  • Conversation data: The conversation data uses coherent multi-session interactions with consistent user personas and tightly coupled visual and textual content.The source materials emphasize diverse everyday and domain-specific scenarios with visual detail and semantic richness.
  • Conversation data: Conversations are synthesized through story-based generation and topic-based organization of existing multimodal dialogues, followed by automated and human quality assurance.The two-stage review checks coherence, fluency, factual consistency, answer correctness, and question clarity.
  • Conversational environment: Each conversation spans multiple sessions with temporal gaps, while information can be referenced, updated, or contradicted across sessions.Agents must associate visual content with textual context and integrate clues distributed over extended timelines.
  • Evaluation tasks: Memory extraction and adaptation includes factual retrieval, visual-centric search, and test-time learning over accumulating multimodal histories.These subtasks assess factual recall, retrieval of visual instances, and adaptation to unseen multimodal examples.
  • Evaluation tasks: Memory reasoning evaluates temporal, visual-centric, and multi-entity reasoning over multimodal memory clues.The tasks require synthesizing temporally dependent questions and using visual information as reasoning cues.
  • Evaluation tasks: Memory knowledge management tests knowledge resolution, conflict detection, and refusal under outdated, contradictory, or incomplete information.The framework evaluates whether agents regulate memory under dynamic states and maintain consistency over time.

4 Benchmarking Analysis

Mem-Gallery benchmarking examines multimodal memory designs, organization, reasoning, knowledge management, efficiency, and retrieval size across thirteen memory approaches. Results show that explicit multimodal preservation and principled organization matter, while reasoning, updating, conflict handling, and efficiency remain challenging.

  • RQ1: Memory storage designs: Explicit multimodal preservation is critical, but greater architectural complexity does not necessarily improve performance.MuRAG, a simple multimodal method, achieves 11.85%, 7.69%, 12.29%, and 29.06% F1 improvements over the best textual memory on overall, FR, VS, and TTL tasks, respectively.
  • RQ2: Memory organization: Without principled organization, Full Memory (MM) performs 8.08% and 51.85% worse F1 than Full Memory (Text) and MuRAG, respectively.The analysis attributes this to visual content consuming more tokens, introducing irrelevant noise, and crowding out informative text under context limits.
  • RQ3: Memory reasoning: Existing multimodal memory methods struggle with reasoning-intensive tasks, especially multi-entity and temporal reasoning.Even on visual-centric reasoning, MemGPT can achieve near-optimal performance, while multimodal approaches remain focused primarily on storage and retrieval.
  • RQ3: Knowledge management: Knowledge management remains limited: A-Mem and MuRAG show 7.22% and 5.82% worse F1 than FIFO on refusal performance, while conflict resolution and detection remain unsatisfactory.The findings identify a trade-off between retrieving relevant information and distinguishing outdated or conflicting knowledge.
  • RQ4–RQ5: Efficiency and retrieval: Multimodal memory generally incurs higher overhead, and increasing retrieval size raises recall but can reduce precision and degrade downstream performance through noise.MuRAG can approach the computational cost of sophisticated textual systems such as A-Mem and MemoryOS; structured methods show more stable precision as K grows.

5 Conclusion

Mem-Gallery introduces a benchmark for multimodal long-term conversational memory using multi-session conversations with tightly coupled visual and textual contexts. Its evaluation highlights principled organization, selective retrieval, and robust maintenance as priorities for future multimodal memory design.

  • Benchmark contribution: Mem-Gallery evaluates multimodal long-term conversational memory beyond prior text-centric and localized multimodal benchmarks.Its conversations are grounded in tightly coupled visual and textual contexts across multiple sessions.
  • Evaluation scope: The benchmark enables systematic assessment of memory capabilities in extended multimodal conversational settings.The dataset connects visual and textual information across multi-session interactions.
  • Conclusion: The findings highlight principled memory organization, selective retrieval, and robust maintenance as priorities for future multimodal memory design.These priorities are stated as takeaways from the extensive evaluation.

Limitations

Mem-Gallery evaluates multimodal long-term conversational memory within a vision–language setting, while leaving other modalities and broader agent behaviors outside its scope.

  • The benchmark does not explicitly cover modalities such as audio or embodied signals.
  • The evaluation focuses on memory-centric capabilities in long-horizon conversations rather than exhaustively assessing planning or tool use.
  • These broader multimodal and agent-behavior extensions are left for future work.

A.1 Problem Definition

The paper models an MLLM agent whose multimodal memory is continuously updated, selectively retrieved, and fused with current context and queries to generate responses.

  • The agent receives a stream of multimodal observations, with each observation potentially containing visual and textual information.
  • External multimodal memory consists of unbounded atomic units containing raw assets, cross-modal descriptions, and joint latent embeddings.
  • A multimodal encoder projects heterogeneous data into a unified d-dimensional latent space.
  • Memory Construction and Maintenance: The update operator continuously consolidates observations into memory through policies such as adding, merging, or deleting entries.
  • Memory Retrieval: At a task timestamp, retrieval surfaces memory items relevant to a multimodal query using similarity scoring and a retrieval-size parameter K.
  • Memory-Augmented Generation: The agent generates its response by integrating working context, retrieved knowledge, and the task query, supporting long-term coherence beyond the model’s context window.

A.2.1 Data Statistics

Mem-Gallery combines diverse, complementary multimodal conversations with annotated questions to evaluate long-term memory capabilities across visual and textual information.

  • The dataset spans 20 conversation scenarios containing 240 multi-session dialogues and 3,962 conversational rounds.
  • It includes 1,003 input images naturally grounded in dialogue context, with visual and textual information designed to be complementary.
  • The benchmark provides 1,711 human-annotated question–answer pairs, including 487 questions explicitly associated with visual inputs.
  • Its evaluation covers extraction, test-time adaptation, reasoning, and knowledge management across long multimodal conversations.

A.2.2 Data Source

The dataset combines permissively licensed image resources with manually designed multimodal conversation materials and reconstructed multimodal dialogues from MMRC.

  • Image Data Source: Image materials include fine-grained categories and images related to manually designed dialogue topics, selected from permissive knowledge-sharing sources.
  • Data Statistics: The dataset’s multi-session scenarios contain multiple related dialogue topics.
  • Image Data Source: Collected images are manually selected and inserted at information-relevant positions within generated textual conversations.
  • Single-Session Dialogue Data Source: Existing single-session dialogues with multimodal information are reconstructed from the prior MMRC dataset.
  • Single-Session Dialogue Data Source: The reconstruction process uses LLMs to extract representative keywords and perform session-level completion, followed by manual selection of higher-quality outputs.

A.2.3 Conversation Data Synthesis

Mem-Gallery conversations are synthesized through human-guided story generation and reconstruction of clustered single-session dialogues, with added multimodal content and extended session structure. The resulting dataset increases visual-information density and task difficulty relative to LoCoMo.

  • Human-guided story generation: Human annotators define protagonist profiles, conversation themes, session outlines, and transitions before LLMs generate textual dialogue.GPT-5.1 and Gemini-2.5-Pro are used in parallel for dialogue generation.
  • Dataset reconstruction: Existing single-session dialogues are clustered by topic, then reviewed and refined into multi-session conversations with summarized user profiles and topics.LLM-extracted keywords support clustering, while annotators verify the results.
  • Dataset reconstruction: The reconstructed conversations are extended with additional LLM-generated sessions, manual verification, and image augmentation while preserving logical continuity.These steps address weak transitions, limited fluency, and insufficient conversation length.
  • Multimodal density: Most Mem-Gallery sessions contain at least two images, shifting the image-count distribution rightward compared with LoCoMo.The dataset reduces sessions with no images or only one image, increasing visual-information density and dialogue difficulty.
  • Evaluation data: The evaluation statistics include QA counts and average clues by category, with AR designed to test refusal when requested information is absent from the dialogue.Unsupported clues are expected for the AR task because the queried information does not exist in the conversation.

A.2.4 Evaluation Data Synthesis

Evaluation results show that stronger backbones improve memory performance but do not remove the difficulty of reasoning-intensive and knowledge-management tasks. Retrieval studies further indicate that moderate, selective retrieval balances coverage and usability better than indiscriminately increasing retrieval size.

  • MLLM backbone analysis: Stronger MLLM backbones improve absolute memory performance across most tasks and methods, while reasoning and knowledge-management tasks remain comparatively difficult.The trend appears across Qwen-2.5-VL-3B to Qwen-2.5-VL-7B and GPT-4.1-Nano to Gemini-2.5-Flash-Lite.
  • MLLM backbone analysis: Relative memory-method rankings remain largely stable across backbone settings, with organized systems outperforming naïve or unstructured baselines.Backbone scaling amplifies memory-design differences rather than compensating for poor organization.
  • MLLM backbone analysis: Stronger backbones yield smaller gains on reasoning-oriented tasks than on extraction-focused tasks, and textual methods remain competitive in several reasoning subtasks.The pattern holds across open-source and closed-source backbones.
  • MLLM backbone analysis: Knowledge resolution, conflict detection, and answer refusal show relatively limited improvement from stronger backbones across evaluated MLLMs.Both open-source and closed-source backbones exhibit similar qualitative behavior.
  • Retrieval-size analysis: K=10 often provides the best trade-off between retrieval coverage and memory usability, whereas larger values frequently saturate, fluctuate, or degrade performance.The effect is particularly pronounced for multimodal systems because additional visual-textual entries can be redundant or weakly aligned.
  • Retrieval-size analysis: Reasoning-oriented tasks are relatively insensitive to retrieval quantity, while knowledge-management tasks remain stable across retrieval sizes.These patterns indicate that retrieval quantity alone is insufficient when memory structure and integration are limiting factors.
  • Retrieval-size analysis: Recall@K and Hit@K increase as K grows, especially for multimodal retrievers, but higher retrieval coverage does not consistently improve downstream task performance.Expanded retrieval can reduce Precision@K by introducing weakly relevant or redundant items.
  • Retrieval-size analysis: Retrieval quality, rather than retrieval quantity, governs downstream memory performance, favoring selective strategies that prioritize precision and cross-modal alignment.Beyond moderate retrieval sizes, recall gains are offset by retrieval noise.
Loading 2601.03515v1…