PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing, Maintaining, and Learning: The Evolution of Multimodal Agent Memory

Minghao Guo, Qingyue Jiao, Xiujin Liu, Jianhong Pang, Ruimeng Yang, Naihao Xu, Liwei Che, Haochen Xue, Ziqi Wang, Xinying Cai, Jiaxuan Luo, Jiaxin Deng, Xuesong Ye, Jintao Qu, Wenyao Gao, Jingwen Gu, Chenyang Zhao, Shichun Liu, Guibin Zhang, Zefang Liu, Siyin Wang, Wujiang Xu, Danrui Li, Zeru Shi, Woongyeong Yeo, Shuang Liu, Wei-Chieh Huang, Weizhi Zhang, Tianyu Yang, Kangsan Kim, Yebowen Hu, Bowen Wen, Bowen Jiang, Haohao Su, Ming Yin, Sheldon Yu, Mian Zhang, Yuyan Chen, Hanrong Zhang, Kai Mei, Zihao Zhang, Mingyu Jin, Xi Zhu, Hengyi Wang, Tianxin Wei, Linyong Nan, Zhenwen Liang, Zhendong Chu, Ziqi Huang, Yongye Su, Sixun Dong, Shujian Liu, Kaiqu Liang, Jiaojiao Han, Zhaoye Fei, Kun Wang, Shihan Dou, Zhichun Guo, Xingjian Diao, Zeqi Gu, Heng Wang, Tao Feng, Cheng Qian, Hamid Palangi, Silei Xu, Xi Yun, Yi Zhu, Shilong Liu, Qingsong Wen, Sung Ju Hwang, Zuxuan Wu, Yu-Gang Jiang, Yiyu Shi, Dimitris N. Metaxas, Jiaxuan You, Julian McAuley, Mengdi Wang, Jiawei Han, Philip S. Yu, Ziyi Ye

arXiv:2610.19366v1cs.AIcs.CLcs.RO

TL;DR

Multimodal agents must retain heterogeneous experience in forms that remain useful across long horizons, while keeping memory valid as observations and environments change. The paper presents a two-axis framework for representation and memory management, and its synthesis finds hybrid representations and dynamic updates widespread but deployment-time policy evolution scarce.

  • Problem

    Multimodal experience is heterogeneous and difficult to compress without loss, while memory must remain valid as new observations and environments change.

    Source: §1 · p. 3

  • Method

    The paper analyzes multimodal agent memory through formation and representation choices and three management tiers: static, dynamic, and self-evolving memory.

    Source: §1 · p. 4

  • Results

    The synthesis finds hybrid representations and dynamic updates widespread, while deployment-time memory-policy evolution remains relatively scarce.

    Source: §1 · p. 5

  • Takeaways & Limitations

    The analysis identifies recoverable, provenance-aware memory, temporal validity, cross-representation consistency, bounded lifelong memory, and governed forgetting as directions for future work.

    Source: §9 · p. 62

    Current systems often retain source observations, derived records, or structured state without an explicit policy for verification or for deciding what evidence to preserve, compress, or version.

    Source: §8.3.2 · p. 56

Abstract

from arXiv · show

Multimodal models and agents have emerged to enable AI systems to learn from and interact with the real world more like humans by integrating information across multiple modalities. However, enabling multimodal agents to continually acquire useful information from complex, heterogeneous multimodal data remains a significant challenge as they move toward long-horizon tasks and lifelong settings. Addressing this challenge requires effective memory mechanisms that can maintain continuity across sessions, adapt to changing users and environments, and refine memory-management policies through accumulated experience. We introduce a two-axis analytical framework for multimodal agent memory structured around two key questions: how multimodal information is formed and represented (Axis I, Formation and Representation), and how its management adapts with accumulated experience (Axis II, Memory Management). Axis I, Formation and Representation, characterizes what information is retained and how it is encoded (e.g., modality-specific embeddings, textual and structured records, latent and parametric representations) to support subsequent reasoning and action. Axis II, Memory Management, distinguishes three tiers: T1 Static Memory reuses fixed memory; T2 Dynamic Memory updates memory through a stable policy; and T3 Self-Evolving Memory updates memory through an evolving policy. Based on this framework, we synthesize recurring design patterns and derive principles for balancing source fidelity with abstraction, maintaining temporal validity and provenance, grounding memory use in recoverable evidence, and evolving memory policies over time. Furthermore, we discuss the challenges and potential future directions for the memory mechanism of multimodal agents.

1. Introduction

Multimodal agents must retain heterogeneous experience in forms that remain useful as observations and environments change. The paper frames this challenge through coupled questions of what memory preserves and how its state and management policy adapt.

1. Introduction

  • Multimodal memory must decide what to retain and how to represent it despite the cost of exhaustive storage and the risk that compression obscures perceptual, temporal, spatial, or acoustic information.Memory units can range from images, frames, and events to interface states, trajectories, or episodes; representation choices trade fidelity, efficiency, editability, and source recoverability.Source: §1 · p. 3
  • Management must reconcile partial, heterogeneous new evidence with persistent state, while systems differ in whether they reuse fixed memory, update it under a stable policy, or evolve that policy.New evidence may extend or correct records, invalidate stale state, link events, or provide feedback on prior memory decisions.Source: §1 · pp. 3–4
  • The framework organizes multimodal agent memory around formation and representation, which determine what experience persists and how it reaches downstream computation, and management, which determines persistent change.It distinguishes source- and modality-specific, textual and structured, latent and parametric representations, alongside T1 Static, T2 Dynamic, and T3 Self-Evolving Memory.Source: §1 · p. 4
  • The study synthesizes 297 memory architectures and 82 benchmark records, finding widespread hybrid representations and dynamic updates but relatively scarce deployment-time memory-policy evolution.It also derives directions including recoverable, provenance-aware memory, temporal validity, bounded lifelong memory, governed forgetting, and lifecycle-aware evaluation.Source: §1 · p. 5
  • The paper shifts analysis from memory components to the operational lifecycle, examining how experience becomes persistent memory and how representation affects later retrieval, revision, and verification.The study defines memory as experience-derived state that persists beyond its formation step and later affects reasoning, generation, planning, or action; at least one non-text modality must contribute to its formation or use.Source: §1 · p. 5

2. Background

The background situates multimodal agent memory within the development of memory concepts and modern agent systems. It distinguishes persistent memory from context, then introduces a lifecycle in which representation choices constrain how memory can be managed and used.

2. Background

  • Multimodal observations vary across visual, temporal, spatial, and actionable information, so retaining a frame, caption, map, token cache, or parameter update changes what an agent can later recover.Native preservation retains more perceptual detail at greater storage and retrieval cost, while translation creates compact, editable units that can omit unanticipated information.Source: §2 · p. 7

2.1. Classical Memory in AI & Cognitive Science

  • Classical cognitive and AI accounts treat memory as selection, organization, and use of information, while modern neural systems raise the question of what persists beyond computation and how it is accessed or changed.The background connects these concerns to formation and representation, and to management through access, maintenance, and adaptation.Source: §2.1 · pp. 7–8

2.2. Foundations of Modern Agentic Memory

  • Standard retrieval-augmented generation retrieves external information at inference time but does not itself specify how agents form, maintain, or revise memory from their own experience.Long-context processing uses information across extended sequences; context management selects what enters active context; memory persists beyond the current inference step and can influence later computation.Source: §2.2 · p. 8
  • Persistent agent memory extends beyond factual retrieval to retaining, transforming, and reusing prior interaction and experience across tasks or sessions.It can complement long-context processing and context management, and provide a substrate for inference-time adaptation and longer-term agent evolution.Source: §2.2 · p. 8

2.3. The Multimodal Memory Lifecycle

  • The lifecycle couples formation and representation with management: management is constrained by what formation preserves, while memory use can prompt further formation, revision, consolidation, or forgetting.Memory units and representations range from sensory evidence and modality-native features to textual or structured records, graphs, maps, latent states, and model parameters.Source: §2.3 · p. 9
  • The three management tiers distinguish fixed memory, updates under a stable policy, and memory-state updates that also revise the policy governing writing, maintenance, or access.Access and utilization remain distinct: a system may fail to retrieve evidence, misuse retrieved evidence, or produce an output without using memory; discarded or corrupted information cannot be recovered by access mechanisms.Source: §2.3 · pp. 9, 11

3. Axis I: Formation and Representation

Axis I classifies multimodal memory by the representation ultimately supplied to downstream reasoning or action. Its representation families trade off recoverability, interpretability, compactness, and integration with the agent policy.

3. Axis I: Formation and Representation

  • Memory paths are classified by what downstream computation receives, not merely by storage or indexing formats.The same experience may remain source evidence, become modality-specific or explicit representations, or be internalized as latent or parametric memory.Source: §3 · pp. 11–12, Figure 6 · p. 11
  • Source- and modality-specific representations preserve rich detail or perceptual structure, trading storage and processing costs against lossy, model-specific encoding.Retained source evidence supports future re-encoding, while modality-specific encodings can reduce costs.Source: §3 · p. 12
  • Explicit abstractions improve interpretability and editability but discard perceptual detail.Source: §3 · p. 12
  • Latent or parametric memory integrates tightly with inference but offers less direct access, provenance, and control.Source: §3 · p. 12
  • Hybrid architectures combine representations, often linking compact abstractions to recoverable source evidence.The paper treats these as system-level compositions, not a fourth representation family.Source: Figure 6 · p. 11, §3 · p. 12

3.1. Source- and Modality-Specific Representations

Source- and modality-specific representations retain observations in their originating modality or preserve modality-derived structure. They prioritize either direct verification and recoverability or compact, task-aligned access, with corresponding costs and losses.

3.1.1. Retained Perceptual Source Evidence

  • Source evidence remains accessible in its native modality, unlike memories available only through captions, embeddings, or symbolic descriptions.Systems may retain complete images, crops, video frames, audiovisual segments, or sensor observations.Source: §3.1.1 · p. 12
  • Retained source evidence preserves fine-grained details, temporal structure, acoustic cues, and context for later retrieval, grounding, and checking.Source media may be organized at frame, event, or entity granularity to suit later retrieval.Source: §3.1.1 · p. 13

3.1.2. Modality-Specific Encodings and Geometric State

  • Persistent encoded audio memory remains underexplored among the analyzed systems.OMAC illustrates an encoding mechanism but compresses current audiovisual context rather than preserving information across interactions.Source: §3.1.2 · p. 14
  • Modality-specific encodings and geometric state preserve task-relevant perceptual or spatial structure while reducing storage, context, and repeated encoding costs.Compression is generally non-invertible, so removed information cannot be recovered from encoded state alone.Source: §3.1.2 · pp. 13–14
  • Learned feature memories are model-specific and usually require a compatible projector, Q-Former, attention module, or policy interface.Geometric state is more explicitly structured, but discretization and feature aggregation can also remove perceptual detail.Source: §3.1.2 · p. 15

3.2. Cross-Represented Explicit Memory

Cross-represented explicit memory transforms multimodal experience into readable text or addressable structured records. These forms support inspection, retrieval, and targeted updates, but can lose perceptual information or incur construction and maintenance costs.

3.2. Cross-Represented Explicit Memory

  • Textual and structured representations make multimodal content readable or addressable, enabling direct reasoning, inspection, search, editing, and entry-level invalidation.A vector database of captions is textual memory, while a scene graph is structured memory when its organization supports retrieval, reasoning, or action.Source: §3.2 · p. 15

3.2.1. Perception-to-Text Records and Summaries

  • In video systems, event-level textual memories let temporally distant evidence be addressed without repeatedly processing the complete video.Systems may localize moments, group them into coherent events, and retrieve a minimal set of records for a query.Source: §3.2.1 · pp. 15–16
  • Textualization offers a compact, interpretable interface for semantic retrieval, model-context insertion, human editing, and cross-modal consolidation.Its semantic bottleneck can omit visual, temporal, spatial, or acoustic details and introduce unsupported interpretations that retrieval cannot restore.Source: §3.2.1 · p. 16

3.2.2. Spatial, Temporal, and Cross-Modal Structured Memory

  • Structured spatial memory can serve as the current world state for planning and guide agents toward locations where further evidence may be gathered.Scene graphs can support navigation, grasping, placement, and active exploration.Source: §3.2.2 · p. 17
  • Graphs and structured records support compositional queries, localized updates, temporal reasoning, and relation-aware planning.They require accurate entity and relation extraction, schema handling, and reconciliation; errors can propagate through retrieval and planning.Source: §3.2.2 · p. 18

3.3. Internalized Latent and Parametric Memory

Internalized memory represents past experience in continuous latent states or learned parameters that contribute directly to model computation. These forms can reduce repeated processing but make stored information harder to inspect and trace, motivating combinations with explicit memory.

3.3. Internalized Latent and Parametric Memory

  • Latent and parametric memory integrate experience into model computation, reducing repeated encoding, retrieval costs, and context growth while limiting inspection, editing, and traceability.Latent compression may discard unrecoverable information, while repeated parameter updates may cause interference or forgetting.Source: §3.3 · p. 18

3.3.1. Persistent Multimodal Latent State

  • Persistent multimodal latent states keep continuous representations separate from backbone parameters and may be recurrent, shared, or both.They can use hidden states, memory tokens, or key–value caches integrated through attention, residual connections, or projection and fusion layers.Source: §3.3.1 · pp. 18, 20
  • Persistent latent memory remains less developed for general audio experience than for visual reasoning, GUI interaction, and embodied control.Audio systems more commonly retain transcripts and separately accessible acoustic features such as speaker identity and prosody.Source: §3.3.1 · pp. 19–20

3.3.2. Experience-Conditioned Parametric Memory

  • Experience-conditioned parametric memory encodes past experience in learned weights or adapters so it can shape inference without retrieving original observations.Examples include modality-specific knowledge edits, video-specific adapters, and consolidation of interaction experience into a GUI policy.Source: §3.3.2 · p. 20
  • Parametric memory also stores task-specific action adapters for retrieval and composition, but makes underlying evidence less accessible and raises attribution, interference, updating, and deletion challenges.Many multimodal agents therefore pair parametric internalization with explicit memory that preserves source evidence and controllable access.Source: §3.3.2 · pp. 20–21

3.4. Hybrid Representations

Hybrid memory combines representations rather than introducing a new elementary encoding type. It coordinates compact abstractions with recoverable perceptual evidence so agents can use efficient retrieval while restoring detail for verification or changing information needs.

3.4. Hybrid Representations

  • Hybrid memory combines representation types because no single form suits every operation: records support compact retrieval and explicit reasoning, while latent or parametric forms contribute directly to computation.Source-linked memory retains a reference to the original observation or selected evidence so an agent can recover it when needed.Source: §3.4 · pp. 21, 23
  • VimRAG organizes multimodal retrieval and reasoning in a dependency graph, pairing textual summaries with visual tokens that preserve information potentially lost in summarization.It adjusts retained visual tokens under a bounded context budget according to relevance, recency, and position in the reasoning graph.Source: §3.4 · pp. 21–22
  • FOLIO links timestamped video records to cached keyframes, allowing retrieval of compact semantic records and restoration of visual evidence when grounding is needed.MementoGUI similarly pairs event summaries with interface crops and visual-region identifiers that can be resolved after older entries are compressed.Source: §3.4 · p. 22
  • Embodied hybrid memories organize visual evidence through spatial scene hierarchies, supporting bounded retrieval and decisions about where to search or explore next.ObsGraph links object evidence to views and rooms, while HiMe combines working memory, image–text records, and procedural plans for long-horizon manipulation.Source: §3.4 · p. 22
  • Coordinating representations requires valid links and consistency as memories change, plus a way to determine when abstractions suffice and when source evidence must be restored.Compact abstractions support routine retrieval; higher-fidelity evidence can be recovered for verification, reinterpretation, or details lost during abstraction.Source: §3.4 · p. 23

4. Axis II: Memory Management

Axis II classifies memory management by what changes during inference: fixed memory, memory updates under a stable policy, or memory and its policy evolving through feedback. Multimodal systems must also keep linked representations consistent and grounded as experience accumulates.

4. Axis II: Memory Management

  • Memory management includes writing, updating, consolidation, reorganization, retention, forgetting, and retrieval, governed by rules or policies ranging from heuristics to deployment-adaptive feedback.Possible mechanisms include fixed prompts, frozen learned modules, optimization procedures, and policies that adapt from deployment feedback.Source: §4 · p. 23
  • High-dimensional multimodal inputs create a storage and retrieval trade-off between efficiency and recoverable source evidence, requiring consistency across linked representations.The framework models representations as graph nodes linked by transformation, alignment, fusion, and grounding relations, with operations acting on both states and relations.Source: §4 · pp. 23–24
  • T2 remains dynamic when memory content, organization, indices, or metadata change under stable selection and update mechanisms; T3 requires persistent feedback-driven revision of reusable management rules or policies.Changes to retrieval keys or usage statistics alone alter memory state without evolving the management policy.Source: §4 · p. 24
  • The framework distinguishes three scopes of persistent deployment-time change: reusing fixed memory, updating memory through a stable policy, and updating memory through an evolving policy.The framework also notes that state or policy changes may help adaptation but can introduce instability and make updates harder to control.Source: §4 · p. 24

4.1. T1: Static Memory

T1 Static Memory is formed before downstream reasoning or action and then reused without modifying its content, organization, indices, or metadata. It suits slowly changing information reused across tasks, but can preserve formation errors and become outdated.

4.1. T1: Static Memory

  • T1 Static Memory is constructed before downstream reasoning or action and reused without modification, though each query may access a different subset.Its content, organization, access indices, and metadata remain fixed.Source: §4.1 · p. 24
  • Static memory suits large static corpora, stable knowledge bases, and invariant environment or task information reused across subsequent reasoning steps or tasks.Applications include one-time ingestion of long videos or GUI environments and retention of procedural knowledge for repeated planning and execution.Source: §4.1 · pp. 24–25
  • Because retrieval cannot modify or reorganize persistent memory, indexing and retrieval mechanisms are the principal design variables and remain fixed during inference.Fixed memories may use sequential, vector-indexed, hierarchical, relational, latent-state, or parametric organization.Source: §4.1 · p. 25
  • Without updates, T1 cannot incorporate later corrections, environmental changes, or procedural knowledge from subsequent successes and failures, so stored representations may become incomplete or outdated.This limitation applies as the source, environment, or task distribution changes.Source: §4.1 · p. 25

4.2. T2: Dynamic Memory

T2 Dynamic Memory changes stored memory during continued interaction while keeping its management policy stable. Its operations span update, consolidation, revision, reorganization, and forgetting across heterogeneous representations.

4.2. T2: Dynamic Memory

  • T2 changes memory through a stable policy, using fixed rules or learned-before-deployment mechanisms to operate across representations.Operations are characterized by how they handle representations and by whether they use heuristics, prompted models, trained modules, or fixed-objective optimization.Source: §4.2 · pp. 25–26

4.2.1. Memory Update

  • Updates incorporate new evidence through source-preserving logs, modality-specific matching, geometric or structured edits, and semantic comparisons with existing records.Latent modules can transform existing states or add entries, while parametric editors optimize designated model parameters; hybrid systems coordinate linked representations.Source: §4.2.1 · pp. 26–27
  • Hybrid updates link complementary representations, such as editable records and source evidence, but must keep linked components consistent without collapsing their distinctions.Source-preserving logs retain raw or minimally processed observations, while feature updates match and aggregate representations within a modality.Source: §4.2.1 · pp. 26–27

4.2.2. Memory Consolidation

  • Consolidation compresses accumulated memories through feature reduction, semantic abstraction, structured merging, or coordinated representations linked to source evidence.Compression can remove brief actions or fine details; incorrect graph alignment can collapse distinct entities or actions. ReMA's memory changes during ingestion and query answering, but its management policy remains stable.Source: §4.2.2 · pp. 27–28

4.2.3. Memory Revision

  • Revision replaces or corrects existing records when evidence indicates they are inaccurate, inconsistent, incomplete, or outdated, unlike independent updates or capacity-driven forgetting.Corrections can target affected spatial regions or semantic records; linked source evidence supports verification, while apparent conflicts may reflect temporal context or unreliable observations.Source: §4.2.3 · p. 28

4.2.4. Memory Reorganization

  • Reorganization changes where retained information resides or how it is linked and retrieved, including tier migration, graph relinking, and cross-representation reindexing.Migration moves records between temporal or functional roles; relinking and reindexing can support entity continuity and shared retrieval paths, but incorrect associations or index drift can impair access.Source: §4.2.4 · p. 29

4.2.5. Memory Forgetting

  • Forgetting can evict active evidence, reduce representational detail, lower retrieval priority, or selectively remove one representation while retaining another.Eviction from an active buffer does not necessarily erase derived long-term information; gradual accessibility decay leaves records stored but less likely to enter working context.Source: §4.2.5 · pp. 29–30

4.3. T3: Self-Evolving Memory

T3 Self-Evolving Memory uses interaction feedback to update memory-management policies, not just memory states. The paper organizes policy evolution around writing, maintenance, and retrieval.

4.3. T3: Self-Evolving Memory

  • T3 updates memory-management policies using task outcomes, verifier judgments, or user corrections, and revised policies persist into later interactions.Unlike T2, which changes stored memory while keeping its policy stable, T3 uses accumulated inference-time experience to evolve management decisions.Source: §4.3 · p. 30

4.3.1. Write-Policy Evolution

  • Write-policy evolution adapts which experiences are retained and how selected multimodal evidence is transformed into memory records.Feedback can improve selection for subsequent reasoning, but a policy optimized on prior experience may discard important information after substantial task or environment changes.Source: §4.3.1 · pp. 30–31

4.3.2. Maintenance-Policy Evolution

  • Maintenance-policy evolution uses user feedback or downstream utility evidence to adapt what persists and how much information is retained during memory operations.Feedback can extend retention after a failed query, but cannot recover removed information unless another linked representation remains available; abstraction criteria can also adapt to preserve useful information.Source: §4.3.2 · p. 31

4.3.3. Retrieval-Policy Evolution

  • Retrieval-policy evolution adapts query formulation, repair of incomplete searches, and prioritization of candidate memories; policy improvement must account for failures originating earlier in the lifecycle.The tier comparison adds policy evaluation and evolution to T1's formation and access and T2's controlled memory-state change; omitted or lost information cannot be recovered by retrieval improvements alone.Source: §4.3.3 · p. 31, Table 1 · p. 32

5. Cross-Axis Analysis

The cross-axis analysis shows that memory representation shapes what each management tier can reliably do, while higher tiers require increasingly controlled changes to memory state and policy. Across the corpus, T2 Dynamic Memory predominates, hybrid representations are common, and T3 policy evolution is concentrated in systems with explicit memory artifacts.

5. Cross-Axis Analysis

  • The two axes interact: each tier inherits prior representational bottlenecks and adds uncertainty as persistent change expands from memory state to policy.T1 depends on preserving and accessing relevant information; T2 adds correct state changes and coordination, and T3 adds evaluation and feedback attribution.Source: §5 · p. 32

5.1. Representation Requirements for Static Memory

  • T1 Static Memory requires retained information to remain accessible and reusable for downstream reasoning or action, without persistent memory change.Retrieval indices should identify context relevant to the current query.Source: §5.1 · p. 32

5.2. Representation Requirements for Dynamic Memory

  • T2 requires controlled edits: explicit records expose edit targets but need dependency tracking, while latent or parametric updates are difficult to scope and verify.Source-linked and hybrid representations can support correction and source-based verification, but dependent representations must stay synchronized.Source: §5.2 · pp. 33–34

5.3. Representation Requirements for Self-Evolving Memory

  • T3 adds policy addressability, feedback attribution, and persistent, validated policy revision to T2’s memory-state requirements.Traces must connect outcomes to management decisions and policy components; validation should test future, replayed, or held-out cases for regressions.Source: §5.3 · pp. 34–35

5.4. Corpus-Level Patterns Across the Two Axes

  • Figure 8 assigns each architecture to its highest supported tier and one representation category, unlike Figures 9 and 10, which count each representation family present.The mosaic plot uses rectangle area and labels for counts, and colors for Pearson residuals.Source: §5.4 · p. 35, Figure 8 · p. 35
  • 221 of 297 architectures (74.4%) update memory during operation: 204 (68.7%) use stable-policy T2 updates, while 17 (5.7%) use T3 policy evolution.Figures 9 and 10 show T2 predominance across modalities and the overall corpus, with T3 relatively rare.Source: §5.4 · p. 35, Figure 9 · p. 36, Figure 10 · p. 37
  • Hybrid systems are the largest representation category at 139 of 297 architectures (46.8%), although many narrower problems use a single task-suited representation.The paper suggests hybrid systems balance fidelity, modality-specific detail, retrieval efficiency, and storage cost.Source: §5.4 · pp. 35–36
  • All 17 T3 systems use cross-represented or hybrid memory with explicit artifacts; none are classified as source-preserving or latent/parametric under the paper’s definition.Explicit rules, workflows, retrieval descriptions, and programs provide identifiable targets for feedback-driven revision and reuse.Source: §5.4 · p. 36

6. Systems Infrastructure for Multimodal Agent Memory

The infrastructure discussion distinguishes logical memory from its physical storage and runtime form, then examines the mechanisms that make retained experience available and keep updates coordinated. These mechanisms operate under finite resources, and their guarantees remain scoped to the artifacts they manage.

6.1. Scope and System Abstractions

  • Logical memory is experience-derived state classified by its operative representation; physical realization stores or exposes it, while materialized execution state is the runtime form used in current computation.Persistence concerns the intended reuse horizon, not necessarily cross-session storage or durability after process or machine failure.Source: §6.1 · pp. 37–38

6.2. Physical Storage and Residency

  • Residency management places, promotes, demotes, replicates, or evicts artifacts across storage tiers, balancing reuse time, size, reconstruction cost, demand, and locality.HBM, host DRAM, and SSDs offer different capacity and access-cost trade-offs; physical cache replacement alone does not establish memory-state adaptation.Source: §6.1 · p. 38, §6.2 · p. 39

6.3. Data Movement, Compression, and Reconstruction

  • Transfer, decompression, conversion, compression, and reconstruction determine whether a retrieved artifact is ready for execution and how much restoration cost remains exposed.Pipelining and opportunistic overlap can reduce exposed latency, but buffering consumes memory and contention or short intervals can leave transfers on the critical path.Source: §6.3 · pp. 39–40

6.4. Indexing, Reuse, and Scheduling

  • Request-time control separates which evidence is relevant, whether cached execution state is compatible, and whether reuse is worthwhile given resource costs.A logical-memory hit does not imply reusable computation; execution-state reuse requires matching inputs and dependencies, and scheduling weighs reuse against load and displaced work.Source: §6.4 · pp. 40–41

6.5. Materialization and Write-back

  • Materialization turns selected memory into model-consumable inputs, while write-back turns observations or execution history into persistent state for later memory use.Systems differ in whether they retain compressed latent context, source-linked episodic records, recurrent state, or procedural experience, with distinct preparation and consumption costs.Source: §6.5 · pp. 41–42

6.6. Consistency and Coordination

  • Consistency and coordination support updates by linking affected objects to physical realizations, evidence, versions, and dependent artifacts, then controlling visibility and reuse eligibility.The analyzed mechanisms coordinate particular records, retrieval views, or execution blocks; guarantees do not automatically compose across separately maintained artifacts.Source: §6.6 · pp. 42–43

7. Evaluation and Applications

This section surveys multimodal memory evaluation across shared and modality-specific capabilities, including 82 benchmark resources spanning image, video, audio, GUI, and embodied memory. It examines how benchmarks test agents’ retention and use of history for state tracking, interaction, and action, and identifies gaps in robustness, lifecycle failure attribution, and lifelong real-world updating.

7.1. Evaluation Dimensions for Multimodal Agent Memory

  • Benchmarks assess evidence access (CE, LE), memory validity (SC, CF, UQ), and memory-grounded action (MA), supplemented by modality-specific retention and state criteria.Capability labels alone do not establish T3 Self-Evolving Memory, which requires deployment feedback to produce persistent changes to a reusable memory-management policy.Source: §7.1 · pp. 43–44

7.2. Benchmark Landscape Across Modalities

  • The 82 selected benchmark resources include dedicated memory tests and broader tasks, and are grouped by evaluation setting with annotations for directly, partially, or unevaluated capabilities.Inclusion indicates evaluation relevance, while annotations describe what each protocol measures directly or indirectly.Source: §7.2 · pp. 44, 46, Table 2 · p. 44

7.2.1. Image Memory

  • Image benchmarks cover fine-grained evidence, entity continuity, and visual state change more thoroughly than extended interaction or adaptive memory use; provenance and behavioral verification remain underdeveloped.Updates are usually defined by curated histories rather than an evolving memory-management policy.Source: §7.2.1 · pp. 47–48

7.2.2. Video Memory

  • Video evaluation spans completed-record recall, causal streaming, and future-oriented assistance, but remains readout-heavy and rarely tests feedback-driven persistent policy improvement.Temporal localization, long-range event relations, and multi-event integration receive more coverage than conflict resolution, unanswerable queries, grounded action, or provenance.Source: §7.2.2 · pp. 48–49

7.2.3. Audio and Speech Memory

  • Audio benchmarks separately cover acoustic retention, long-range retrieval, speaker association, and repair, but rarely combine cross-session acoustic memory, conflicting evidence, revision, and memory-grounded action.ChatMem-Bench uses 316 questions over 53 hours of dialogue; fixed-recording tasks mainly test access and reuse, whereas interactive ASR tests repair or carried state.Source: §7.2.3 · pp. 49–50

7.2.4. GUI Memory

  • GUI benchmarks evaluate state retention, action-outcome associations, and experience reuse through task progress, regions, and action supervision.MemGUI-Bench separates within-session retention from cross-session improvement, while MementoGUI-Bench frames long-horizon execution as memory control.Source: §7.2.4 · p. 50
  • GUI benchmarks span personalized history use, controlled transient-state reuse, cross-app workflows, and visual-cue retrieval that requires source provenance.KnowU-Bench tests preference inference and proactive assistance; Memory-World controls transient interface-state reuse; DataScope evaluates completed operations across 96 workflows; and DMV-Bench requires retaining visual evidence that textual descriptions cannot recover. Synthetic states and endpoint success still limit realism and error attribution.Source: §7.2.4 · p. 50
  • GUI evaluation remains weaker on conflicting histories, unanswerable or unsafe requests, interface changes, evolving memory policies, and identifying which lifecycle stage caused failure.Terminal task success may not distinguish errors in memory formation, retrieval, or action selection.Source: §7.2.4 · p. 50

7.2.5. Embodied Memory

  • Embodied benchmarks cover navigation and mapping, memory-dependent manipulation, and long-horizon stateful interaction, where actions affect later observations and environment state.Spatial structure, object and task states, and action consequences are central to this evaluation; spatial regions and maps provide supervision beyond terminal success.Source: §7.2.5 · pp. 50–51
  • Navigation benchmarks test spatial persistence and revision, while planning and active-perception benchmarks assess recall, uncertainty, viewpoint selection, and spatial reasoning.FindingDory, 3DMem-Bench, and SpaMEM also assess object histories or updates; MEMORA-Bench directly tests information-not-available controls, while SIS-Bench planning answers do not establish executed control or online task-progress tracking.Source: §7.2.5 · pp. 50–51
  • Manipulation benchmarks use state- and action-level supervision to evaluate memory-grounded action, but endpoint success cannot isolate memory failures from perception, planning, or control errors.Long-horizon and cross-session protocols extend evaluation toward dynamic-memory maintenance and revision, while RoboMME-Interference isolates retrieval robustness as unrelated sessions accumulate; most rely on simulated or hybrid settings, with limited tests of conflicting memories, unanswerable queries, lifelong physical-world updates, or lifecycle failure attribution.Source: §7.2.5 · p. 51

7.3. Application Regimes

Application regimes group multimodal memory by the function it serves, rather than by input modality. These regimes stress different memory properties, from recoverable evidence to continuity, action relevance, and consistency in generated artifacts.

7.3.1. Recall and Evidence-Based Reasoning

  • Recall and evidence-based reasoning depend on recoverable provenance and temporal validity, not recall volume alone.Personal archives and long-document assistants require cross-modal retrieval and source grounding; enterprise analysis also needs evidence attributable across files, reporting periods, and tool outputs.Source: §7.3.1 · pp. 51–52

7.3.2. Personalization and Cross-Session Continuity

  • Personalization maintains user-specific state across encounters, requiring correct identity scoping and revision of preferences without obsolete or cross-user information.This regime includes preferences, prior dialogue, viewed media, corrections, and recurring activity patterns.Source: §7.3.2 · p. 52

7.3.3. Control and Task Execution

  • Control applications require memory to change an agent’s actions, with stored experience remaining revisable as interfaces, environments, or conditions evolve.Applications include GUI automation, navigation, embodied tasks, and transfer of procedures; action outcomes and environment states can provide evidence of memory utility.Source: §7.3.3 · p. 52

7.3.4. Generation and World Consistency

  • Generative applications use memory to preserve constraints, entities, style, and causal state across iterative production, but dedicated memory benchmarks remain limited.Versioned state and recoverable dependencies are needed so changes to underlying facts or analyses can propagate coherently through multimodal artifacts.Source: §7.3.4 · p. 52

7.4. Application–Evaluation Alignment and Gaps

Evaluation coverage depends on task: retrieval-oriented benchmarks more consistently test recovery than whether memories remain valid, sufficient, and actionable. Coverage of conflict handling, policy adaptation, and memory-grounded action remains limited.

7.4. Application–Evaluation Alignment and Gaps

  • Image, video, and audio evaluations mainly use question answering, retrieval, or localization, whereas GUI and embodied benchmarks test action or planning through environment states and task success.The task determines which aspects of memory can be observed.Source: §7.4 · p. 53
  • Retrieval-centric applications are better represented than persistent state maintenance, conflict handling, or memory-grounded action, so current evaluation tests recovery more consistently than validity, sufficiency, and actionability.Long-range and cross-modal evidence access dominates evaluation; conflict resolution and evidence insufficiency remain sparsely evaluated.Source: §7.4 · p. 53
  • Image, video, and audio benchmarks emphasize modality-native structures, while GUI and embodied benchmarks focus on interface, spatial, object, and task states.Image tests visual details and state changes; video tests temporal localization and multi-event reasoning; audio tests long-form retrieval, speaker and acoustic retention, and stateful interaction.Source: §7.4 · p. 53
  • Audio evaluation remains predominantly recall- or transcription-oriented, with limited testing of persistent correction, personalized response, speaker conflict, abstention, and memory-grounded action.Lifecycle coverage also remains concentrated in T1 Static Memory; T2 state-update and experience-reuse tasks usually use stable policies, and a Learn label alone does not establish T3 policy evolution.Source: §7.4 · p. 53

8. Challenges and Outlook

This section examines challenges in forming, maintaining, and managing multimodal memory, including how representation choices affect inspection, updating, verification, and forgetting. It outlines directions for more recoverable, temporally valid, evidence-grounded memory and evaluation, as well as infrastructure guarantees for coordinating changing memory under bounded resources.

8.1.1. Recovering Supporting Evidence

  • Formed memories need deliberate provenance links and explicit rules for which supporting evidence remains reachable, balancing later verification against retention cost.A minimal provenance bundle should record where a memory came from, how it was produced, and what evidence supports it; recoverability does not require retaining every source observation indefinitely.Source: §8.1.1 · p. 54

8.1.2. Finding and Updating Specific Memory

  • Semantic addressability remains limited in latent and parametric memories, leaving policy decisions difficult to attribute, inspect, and revise at the units they target.Every architecture evolving its management policy at deployment maintains an explicit cross-represented component, while none using only latent or only parametric memory does; distributed memory needs units that can be independently revised and checked.Source: §8.1.2 · pp. 54–55

8.2.1. Maintaining Up-to-Date Memory

  • Memory maintenance should track whether states remain current, historical, superseded, uncertain, or concurrently valid, using evidence reliability and event timing rather than recency alone.Timestamps capture when observations were made, not how long their assertions remain true; systems still need rules for when evidence replaces an older state, coexists with it, or preserves uncertainty.Source: §8.2.1 · p. 55

8.2.2. Self-Improving Memory Management

  • Self-improving management should connect delayed, cross-modal outcomes to the earlier write, merge, revision, or forgetting decision that likely caused them.Retrieval receives more direct feedback than writing or maintenance, whose effects may emerge much later and in another modality; assigning credit must separate those decisions from intervening operations.Source: §8.2.2 · p. 55

8.3.1. Editable Representations without Loss of Detail

  • Memory designs must preserve perceptual detail while keeping updates editable, combining compact state with reachable source evidence and rules for reopening it when needed.Routine updates can operate on textual or structured state, while uncertain or high-impact changes can be checked against supporting images, video, audio, or trajectories.Source: §8.3.1 · p. 56

8.3.2. Selective Evidence Retention for Verifiable Memory

  • Verifiable memory needs selective evidence retention, because evidence and checkpoints have different costs and current systems rarely specify what to keep, compress, or preserve as earlier versions.Lightweight provenance can remain broadly available while expensive evidence is reserved for uncertain memories, conflicting modalities, or high-impact revisions; predicting future verification needs remains open.Source: §8.3.2 · p. 56

8.3.3. Policy Adaptation with Preserved Evidence Grounding; 8.3.4. Sustaining Memory without Losing the Ability to Forget

  • Policy adaptation should be evaluated for whether it weakens supporting evidence paths, while bounded memory requires representation-aware budgeting and verifiable forgetting across derived copies.Removing one record may leave the same information in captions, embeddings, summaries, latent states, or parameter updates; verification and version history can also conflict with deletion requirements.Source: §8.3.3 · p. 57, §8.3.4 · p. 57

8.4. Deployment-Oriented Memory Evaluation

  • Evaluation should diagnose memory across formation, maintenance, access, and utilization, then measure quality and resource costs as experience accumulates under real deployment conditions.Long-horizon tests should trace memory quality as a function of time, experience, and budget; deployment tests should account for deadlines, imperfect sensing, feedback, and changing users or worlds.Source: §8.4.1 · p. 58, §8.4.2 · p. 59, §8.4.3 · p. 59

8.5.1. Workflow-Responsive Residency and Data Movement

  • Runtime data movement should adapt to changing workflows while bounding wasted transfers, resumption delay, and lifetime cost-quality trade-offs.Contracts should cover cancellation, overlapping readers, reprioritization, and transfer granularity; compression decisions should include encoding, restoration, temporary storage, layout conversion, and expected reuse.Source: §8.5.1 · p. 60

8.5.2. Dependency-Aware Reuse and Scheduling

  • Reuse and scheduling should jointly account for dependency validity, evidence quality, restoration cost, and expected future use rather than maximizing cache hits.Compact dependency descriptions can support exact reuse, validated approximation, and selective invalidation, but fine-grained tracking may cost more than the recomputation it avoids.Source: §8.5.2 · p. 60

8.5.3. Coordinating Materialization and Write-back

  • Materialization and write-back should be coordinated with foreground inference while preserving representation-specific, policy-compliant fallbacks.Systems should balance eager materialization, lazy reconstruction, and incremental publication against responsiveness, groundedness, and recoverability, while preventing write backlogs from hiding new experience.Source: §8.5.3 · p. 61

8.5.4. Cross-Layer Consistency, Recovery, and Revocation

  • Infrastructure needs composable contracts for consistency, durability, recovery, revocation, and interoperability across dependent memory artifacts.Contracts should distinguish visibility and staleness guarantees, recovery of an exact artifact from generation of an alternative, and removal from preventing retrieval versus retiring derivatives and reclaiming storage.Source: §8.5.4 · p. 61

9. Conclusion

The conclusion frames multimodal agent memory along representation and management axes, linking what is retained and how it is encoded to how memory persists and adapts. It finds fixed memory and stable-policy updating dominant, while policy evolution, bounded lifelong memory, and lifecycle-aware evaluation remain underdeveloped.

9. Conclusion

  • The framework analyzes multimodal agent memory through Formation and Representation and Memory Management, distinguishing T1 Static, T2 Dynamic, and T3 Self-Evolving Memory.Axis I classifies operative representations available to downstream computation; Axis II classifies management by whether memory is fixed, updated through a stable policy, or updated through an evolving policy.Source: §9 · p. 61
  • Representation constrains memory accessibility, revisability, verifiability, and recoverability; multiple representations require provenance links, temporal validity, and cross-representation consistency.Updating memory state is not equivalent to evolving the management policy.Source: §9 · p. 62
  • Current systems remain dominated by fixed memory and updates through stable policies, while online policy evolution, bounded lifelong memory, governed forgetting, and lifecycle-aware evaluation remain underdeveloped.Benchmarks are insufficient to establish whether memories remain valid, sufficient, and actionable throughout the full lifecycle.Source: §9 · p. 62
  • The conclusion points toward systems that preserve verifiable, recoverable multimodal evidence under bounded resources, maintain temporally evolving and cross-representation-consistent states, and improve management policies through continual feedback.Current infrastructure lacks unified mechanisms to store, restore, and reuse memory efficiently and consistently under finite resource constraints.Source: §9 · p. 62

Related papers

Loading 2610.19366v1…