Remembering Through Change: Provenance-Preserving Hypergraph Memory for Multimodal Agents

• Multimodal memory must preserve visual details while tracking the same entity’s state across sessions despite changing observations.Abstraction-based memories may omit discriminative visual details, while pixel-preserving memories do not reliably connect observations to persistent identities or state histories.Source: §1 · pp. 1–2
• Occurrence hyperedges group identity-consistent mentions and associated facts, while evolution nodes summarize aspect-specific changes and link them to supporting evidence.Facts retain provenance to original dialogue and images, keeping inferred states traceable to sources.Source: §1 · p. 2
• Hypergraph-based Agentic Search adapts its entry layer and traverses typed links, inspecting source observations when fine-grained verification is needed.Compact evolution records guide cross-session reasoning, while grounded facts and images provide verifiable evidence.Source: §1 · p. 2
• MHG-Mem improves average performance by 8.20–9.98% over the strongest baseline across three benchmarks and three backbones.Source: §1 · p. 3

Long-Term Multimodal Agent Memory

• Existing systems compress observations, organize experiences around entities or events, or retrieve evidence across modalities, but these capabilities remain difficult to reconcile.Semantic abstractions may discard visual cues, while raw pixels alone do not establish identity or attribute evolution.Source: §2 · p. 3

Hypergraph-based Agent Memory

• Hypergraph memory methods make higher-order membership explicit, but the cited approaches remain largely text-centric and are limited for dynamically evolving multimodal interactions.MHG-Mem instead organizes multimodal occurrences, entity-specific evolutions, and provenance with typed hyperedges.Source: §2 · p. 3

3.1 Problem Formulation

• The formulation maps an ordered multimodal interaction history into persistent memory before a query, then retrieves evidence to condition the agent’s answer.The design objective is to choose memory construction and retrieval operators that support accurate multimodal answering.Source: §3.1 · p. 4

3.2 Multimodal Hypergraph Memory Structure

• Three node types and three n-ary hyperedge types represent facts, entity mentions, and evolution records, linking them across layers and preserving provenance.Occurrence hyperedges bind referent mentions to facts; evolutionary and evidential hyperedges connect evolution records to entities and supporting facts.Source: §3.2 · p. 4

3.3 MHG-Mem Construction

• Construction extracts grounded atomic facts from dialogue and images, then extracts entity mentions and links them into referent groups with occurrence hyperedges.Accepted mentions remain distinct EntityNodes, retaining their visual region when available; facts are attached by mention-name matching.Source: §3.3 · p. 5
• For each referent group, the method synthesizes state histories from associated facts, retaining changes in the same attribute or relation in observation order.Evolution records include an update type and ordered state sequence; evidential links preserve state–fact correspondence.Source: §3.3 · p. 5

3.4 Hypergraph-based Agentic Search

• Agentic search adaptively selects a memory layer, traverses hyperedges, and follows provenance links to inspect original dialogue or images when needed.Its controller selects among four tool types; retrieved evidence and image references are assembled into a query-specific evidence chain.Source: §3.4 · pp. 5–6

4.1 Experimental Setup

• The evaluation covers three multimodal memory benchmarks, with EM, token-level F1, BLEU-1, Judge scores, and memory-quality measures.Baselines include four unimodal and four multimodal memory or RAG methods; results are averaged over three runs.Source: §4.1 · pp. 6–7

4.2 Main Results

• 57.98% (+8.20%), 67.95% (+9.89%), and 73.28% (+9.98%) Avg. J. scores are reported for Qwen3-VL-8B, GPT-4.1-mini, and Qwen3.8-27B, respectively.MHG-Mem achieves the highest Judge score in every benchmark–backbone setting.Source: §4.2 · p. 7, Table 1 · p. 7
• On MemEye, MHG-Mem with Qwen3.8-27B reaches 78.76% EM versus 67.85% for UniversalRAG and 66.95% for SRAG.Figure 3(b) also reports 82% versus 60% on Mem-Gallery visual reasoning and 87% versus 83% on WorldMemArena-S visual fact recall.Source: §4.2 · p. 8, Figure 3 · p. 8
• MHG-Mem achieves the best result in 17 of 19 task categories, including a WorldMemArena-S temporal-reasoning increase from 48% to 87%.The passage also reports gains on dynamic update, test-time learning, cross-modal reasoning, and Mem-Gallery visual and temporal reasoning.Source: §4.2 · p. 8, Figure 3 · p. 8

4.3 Ablation Study

• Memory-side ablations report gains of 5.48% for the entity layer over fact-only memory, 3.37% for the evolution layer, and 1.56% for evidential hyperedges.Occurrence hyperedges account for 4.78% of the entity-layer gain; evidential hyperedges link evolution states to supporting facts.Source: §4.3 · pp. 8–9, Table 2 · p. 9
• Replacing agentic search with three-layer top-k retrieval lowers performance by 5.10%; removing inspection, cross-layer traversal, or the visual tool lowers it by 3.44%, 3.09%, or 1.37%.The inspection tool provides access to original dialogue and images for checking fine-grained evidence.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 Case Analysis

• In the renovation case, MHG-Mem tracks paint mentions and color changes from sage green to terracotta, then follows evidential links to source images to verify the answer.The described alternatives either miss initial color information or preserve pixels without tracking the evolving schemes.Source: §4.4 · p. 9, Figure 4 · p. 9

Conclusion

• MHG-Mem’s typed n-ary hyperedges connect entity mentions, facts, and evolution records, keeping individual states traceable to supporting observations.Hypergraph-based agentic search navigates these memory levels and can inspect original dialogue and images to verify evidence.Source: Conclusion · p. 10

B.1 Detailed Hypergraph Memory Schema

• The memory uses FactNodes for atomic multimodal observations, EntityNodes for distinct entity mentions, and EvolutionNodes for changes to a referent’s attributes or relations.Entity mentions remain distinct observations even when linked to the same underlying entity.Source: §B.1 · p. 16
• Occurrence hyperedges bind a referent group’s mentions and associated facts, while evolutionary and evidential hyperedges connect evolution records to mentions and supporting facts.Image identifiers, visual regions, and source-round references preserve observation-level grounding.Source: §B.1 · p. 16

B.2 Detailed Memory Construction

• Construction archives original rounds and images before extraction, and stores source references with extracted facts and entity mentions.For groups receiving newly anchored facts, it regenerates evolution records while retaining underlying facts, mentions, and original sources.Source: §B.2 · pp. 16–17
• Evolution validation removes invalid or repeated fact references, orders remaining state–fact pairs by source round, and requires nonempty, distinct state values.These checks validate the recorded sequence; identifying a meaningful change remains the synthesizer’s task.Source: §B.2 · p. 17

B.3 Agentic Search and Tool Specification

• Search scores combine text, text-image, and image-image components; visual scores are floored at zero, and unavailable terms contribute zero without renormalizing weights.The Visual Tool searches stored captions for text lookups and image embeddings for query-image lookups.Source: §B.3 · p. 18
• Evolution lookup ranks records using their attribute and summary, then reads supporting facts in the order retained by the state trace.Source inspection follows source-round references to recover original observations.Source: §B.3 · p. 18
• Retrieved facts and facts associated with accessed source rounds are registered for citation, and invalid references or image-to-source mismatches are removed.The checks validate identifier associations, not the semantic content of an optional controller summary.Source: §B.3 · p. 18

B.4 Provenance Preservation

• Each retained state cites facts in its subject referent’s fact set, and each cited fact must have an available source-round record.Expanding the evidential hyperedge by source-round references recovers exactly the source rounds recorded for the evolution.Source: §B.4 · pp. 18–19
• The recovered source set deduplicates shared rounds, while the ordered state trace retains the association between each state and its cited fact and source.Thus source-set equality does not discard the index-specific state-to-fact-to-round association.Source: §B.4 · p. 19
• For surviving evolutions, unchanged state references and source bindings preserve evidential membership across updates.Validated new evolutions satisfy the same assumptions, and induction over supported update operations preserves the property for successfully constructed snapshots.Source: §B.4 · p. 19

C.1 Benchmarks and Metrics

• MemEye evaluates visual evidence from scene-level to pixel-level and memory reasoning from atomic retrieval to evolutionary synthesis, using eight scenarios and 371 questions.Its tasks include visually grounded fact recovery, distinguishing similar instances, and reasoning across sessions.Source: §C.1 · p. 19
• Mem-Gallery evaluates persona-driven multimodal conversational memory across 20 scenarios, with 240 sessions, 3,962 dialogue rounds, 1,490 images, and 1,711 open-ended questions.It probes memory extraction, reasoning, and knowledge management over extended text–image histories.Source: §C.1 · p. 19
• WorldMemArena-S uses a released 150-sample subset with 7,906 questions across 11 types, including factual recall, dynamic updates, conflict resolution, and cross-modal reasoning.Memory is constructed only from sessions covered by each checkpoint; non-prefix session windows are constructed independently to prevent access to out-of-scope observations.Source: §C.1 · p. 20

C.2 Implementation Details

• Experiments use three backbones, fixed encoders, three-run averages, and up to three prior turns and eight images during construction.Occurrence linking uses K=3; evolution records require at least two facts and observations from at least two sessions, with refresh timing differing by benchmark.Source: §C.2 · p. 21

C.3 Baselines

• Baselines include linked textual notes, tiered conversational memory, compressed textual units, and topic–episode–fact retrieval.These are A-Mem, MemoryOS, SimpleMem, and HyperMem, respectively.Source: §C.3 · pp. 22–23
• Other baselines retain raw multimodal evidence, score reliability, retrieve by text-image similarity, or route across modality-specific corpora.These are M2A, MMA, SRAG, and UniversalRAG, respectively.Source: §C.3 · p. 23

D.1 Retrieval Tool Analysis

• Figure 5 shows frequent fact resolution and source inspection, alongside substantial entity- and evolution-layer retrieval in several task types.Fact and source-inspection tools are used most; entity and evolution retrieval is substantial in TTL, TR, KR, DU, and VU tasks.Source: Figure 5 · p. 23, §D.1 · pp. 23–24

D.2 Cost Analysis

• 8.20–9.98 points over the strongest baseline per backbone accompany approximately 2x token consumption for MHG-Mem.The token measure averages LLM input and output across construction, retrieval, and QA, including image input tokens for multimodal methods.Source: §D.2 · p. 24, Figure 6 · p. 24
• Unimodal caption-based memories use fewer tokens but achieve lower QA performance, while RAG methods use fewer tokens than memory-based approaches.The passage attributes the caption-based gap to omitted fine-grained visual details and the RAG cost difference to avoiding explicit construction and iterative search.Source: §D.2 · p. 24
• Without agentic search, MHG-Mem uses only approximately 1–2k more tokens than multimodal baselines while retaining 5%–15% performance gains.The paper suggests top-k hypergraph retrieval when resources are constrained and agentic search when prioritizing higher QA performance.Source: §D.2 · p. 24, Figure 6 · p. 24

D.3 Memory Quality and Fine-grained Analysis on More Backbones

• Across Qwen3-VL-8B and GPT-4.1-mini, MHG-Mem has the highest memory correctness and update handling, with retrieval coverage among the best.These memory-quality and task-level analyses cover WorldMemArena-S and Mem-Gallery.Source: §D.3 · p. 25
• On WorldMemArena-S, MHG-Mem outperforms all compared baselines in cross-modal, temporal, dynamic-update, and visual-update reasoning with both backbones.Cross-modal reasoning gains over the strongest baseline are 20% with Qwen3-VL-8B and 6% with GPT-4.1-mini.Source: §D.3 · p. 25
• On Mem-Gallery, visual-centric reasoning improves 4% and 6%, while knowledge resolution improves 5% under both backbones.Conflict detection also benefits in both settings.Source: §D.3 · p. 25
• On Mem-Gallery, temporal reasoning improves 3% with Qwen3-VL-8B but remains 6% below the strongest baseline with GPT-4.1-mini.MHG-Mem also trails the strongest factual-recall baseline on both benchmarks under both backbones.Source: §D.3 · p. 25

D.4 Hyperparameter Analysis

• Table 6 finds K=3 near the best tested occurrence-linking score and retrieval budgets that trade higher QA scores against token use.K=3 averages 73.28 versus 73.33 at K=5; raising budgets from (4, 4) to (16, 8) raises average QA from 67.29 to 72.48 and lowers forced submissions from 37.3% to 17.9%. Raising them further to (24, 16) adds only 0.24 points over (16, 10).Source: §D.4 · pp. 25–26, Table 6 · p. 26

D.5 More Case Studies

• The case studies recover the faucet’s off–running–off history and the oak-log count sequence 1 → 2 → 5, where two baselines recover only 1 → 2.The latest oak-log count is verified from the inventory image.Source: §D.5 · p. 26, Figure 9 · p. 27, Figure 10 · p. 27

E Prompt Templates / E.1 Atomic fact construction

• The prompt templates include atomic-fact construction, which retains temporal and visual details in standalone facts.The appendix preserves template wording and uses benchmark-provided prompts for judge and QA tasks.Source: E Prompt Templates · p. 28, §E.1 · p. 28

E.2 Multimodal entity extraction

• The multimodal entity-extraction prompt grounds nameable visual mentions in image regions.Source: §E.2 · p. 29

E.3 Occurrence linking

• Occurrence linking can associate a mention with a stored referent, create a new referent, or reject an invalid extraction.Source: §E.3 · p. 29

E.4 State evolution synthesis

• State-evolution synthesis identifies changing aspects and binds each reported state to its supporting fact identifier.Source: §E.4 · p. 30

E.5 Hypergraph-based agentic search

• Agentic search guides sequential tool use to assemble evidence for a separate answer model, inspecting images when visual details matter.The prompt specifies gathering tools, one action per turn, and submission once evidence is sufficient.Source: §E.5 · p. 31, ## Format — EXACTLY ONE action per… · p. 31

## Format — EXACTLY ONE action per reply, nothing after it

• The action format requires one tool action and its input per reply, followed by stopping until the observation arrives.Source: ## Format — EXACTLY ONE action per… · p. 31