PaperlayerView exact revision on arXiv

Source-linked AI summary

MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory

Minghao Guo, Qingyue Jiao, Zeru Shi, Yihao Quan, Boxuan Zhang, Danrui Li, Liwei Che, Wujiang Xu, Shilong Liu, Zirui Liu, Mubbasir Kapadia, Vladimir Pavlovic, Jiang Liu, Mengdi Wang, Yiyu Shi, Dimitris N. Metaxas, Ruixiang Tang

arXiv:2605.15128v1cs.CVcs.CLcs.IR

TL;DR

Existing evaluations provide limited evidence about whether multimodal agents preserve and reason over visual information across long-term interactions, especially when visual states change over time. MemEye evaluates memory along visual-evidence granularity and reasoning depth, and its evaluation finds that current systems still lose fine-grained details, retrieve stale evidence, or fail to synthesize the currently valid state.

  • Problem

    Existing evaluations provide limited evidence about whether agents preserve decisive visual details and reason over visual states that change across long-term interactions.

    Source: §1 · p. 1, §2.2 · p. 4

  • Method

    MemEye evaluates multimodal agent memory across visual-evidence granularity and memory-reasoning depth, using a benchmark of 371 questions across eight life-scenario tasks.

    Source: §1 · p. 2

  • Results

    Across 13 memory architectures and four VLM backbones, current systems remain far from saturation and recurrently lose fine-grained evidence, retrieve stale evidence, or fail to synthesize the currently valid state.

    Source: §6 · p. 12

  • Takeaways & Limitations

    Future multimodal memory systems should combine image evidence, text or structured state records, and mechanisms for selecting temporally valid evidence over long histories.

    Source: §6 · p. 12

    MemEye is a diagnostic rather than exhaustive benchmark, covering curated life-scenario tasks and representative architectures, with comparisons affected by method-specific encoders and implementations.

    Source: §D.7 · p. 34

Abstract

from arXiv · show

Long-term agent memory is increasingly multimodal, yet existing evaluations rarely test whether agents preserve the visual evidence needed for later reasoning. In prior work, many visually grounded questions can be answered using only captions or textual traces, allowing answers to be inferred without preserving the fine-grained visual evidence. Meanwhile, harder cases that require reasoning over changing visual states are largely absent. Therefore, we introduce MemEye, a framework that evaluates memory capabilities from two dimensions: one measures the granularity of decisive visual evidence (from scene-level to pixel-level evidence), and the other measures how retrieved evidence must be used (from single evidence to evolutionary synthesis). Under this framework, we construct a new benchmark across 8 life-scenario tasks, with ablation-driven validation gates for assessing answerability, shortcut resistance, visual necessity, and reasoning structure. By evaluating 13 memory methods across 4 VLM backbones, we show that current architectures still struggle to preserve fine-grained visual details and reason about state changes over time. Our findings show that long-term multimodal memory depends on evidence routing, temporal tracking, and detail extraction.

1. Introduction

MemEye evaluates whether multimodal agent memory preserves decisive visual evidence and uses it to reason across evolving observations. Its benchmark exposes persistent trade-offs between retaining visual detail and tracking temporal validity.

Introduction

  • Prior long-term memory evaluations often allow visually grounded answers from captions or dialogue and rarely test visual state changes through evolving evidence.Captions can also discard region layouts, object identities, and fine-grained textures, while state updates require linking sessions and allowing new observations to override earlier ones.Source: §1 · pp. 1–2
  • MemEye introduces 371 mirrored multiple-choice and open-ended questions across eight life-scenario tasks, annotated by evidence granularity and reasoning depth and filtered to reduce textual shortcuts.The benchmark tests whether agents preserve and reason over visual evidence that cannot be replaced by captions.Source: §1 · pp. 2–3
  • MemEye’s two evaluation dimensions are visual evidence granularity, from scene-level to pixel-level, and memory reasoning depth, from atomic retrieval to evolutionary synthesis.They measure whether systems preserve the visual details needed to answer questions and reason over retrieved evidence.Source: §1 · pp. 2–3
  • Across 13 methods and four VLM backbones, text memory organizes updates but loses fine detail, image memory preserves evidence but struggles with temporal validity, and specialized mechanisms matter more as histories diversify.The findings point to preserving visual details, tracking temporal validity, and selecting evidence over long histories as requirements for effective multimodal memory.Source: §1 · p. 3

2. Related Work

Related work shows that existing agent-memory methods and long-horizon benchmarks leave visual evidence preservation and evolving visual states insufficiently tested. MemEye centers evaluation on the visual evidence that must be retained and how it must be used over time.

2.1. Agent Memory Systems

  • Agent-memory research includes textual systems for writing, updating, and maintaining memories [42] [17] [7] [43], which mainly use text or textual abstractions of past experience.Prior work also explores memory mechanisms for computer-use and interactive agents [34] [30] [13] [4] [21].Source: §2.1 · p. 3
  • Existing multimodal memory evaluations often report end-task performance without isolating detail loss, stale retrieval, or failure to synthesize a valid state.MemEye evaluates these mechanisms, including whether an architecture loses visual evidence, selects stale evidence, or fails to recover the valid memory state.Source: §2.1 · pp. 3–4
  • Recent multimodal memory systems retain or retrieve visual experience, with differing trade-offs in coverage, selectivity, abstraction, and revision, including MIRIX [37], MMA [27], M2A [10], and FluxMem [40].MemEye’s taxonomy distinguishes the granularity of decisive visual evidence from the reasoning operation required over memory.Source: §2.1 · p. 3, Figure 3 · p. 4

2.2. Memory Benchmarks for Long-Horizon Multimodal Agents

  • Long-horizon benchmarks span text-centric recovery tasks and multimodal dialogue or multi-session settings, including LoCoMo [28], MMDU [26], and Mem-Gallery [2].Text-centric benchmarks primarily test linguistic facts across many turns; multimodal benchmarks add images to dialogue, and Mem-Gallery includes images throughout multi-session conversations.Source: §2.2 · p. 4
  • Prior benchmarks rarely test whether decisive image content survives caption shortcuts, requires instance- or pixel-level preservation, or changes over time.MemEye frames these as coupled challenges of visual evidence compression and state-evolving memory use; Mem-Gallery’s conflicts are primarily textual rather than visual-state updates.Source: §2.2 · p. 4

3. MemEye Framework and Benchmark

MemEye evaluates memory along two dimensions: how fine-grained the decisive visual evidence is and how deeply retrieved evidence must be reasoned over. Its benchmark uses filtering checks to retain answerable questions that require visual evidence for memory evaluation.

3.1. The Two-Dimensional Evaluation Framework

  • Each question receives an (X, Y) coordinate: X1–X4 span scene-level to pixel-level evidence, while Y1–Y3 span atomic retrieval to evolutionary synthesis.X3 tests distinguishing object or person instances; X4 tests details such as color, texture, or small text. Y2 requires associating evidence across sessions and modalities; Y3 requires handling updates, conflicts, and overrides.Source: §3.1 · pp. 4–5
  • The benchmark contains 371 questions across 221 sessions, 848 dialogue rounds, and 438 images.It covers eight tasks in four domains: Leisure, Domestic, Professional, and Personal; each question has multiple-choice and open-ended forms, and MCQ items have four answer rotations.Source: §3.1 · p. 5
  • Three filtering checks remove questions answerable without images, replaceable by minimal captions, or too difficult for the underlying foundation model.The checks use four answer rotations; the final check supplies images with answer-relevant context to isolate answerability rather than memory.Source: §3.1 · p. 6, Figure 4 · p. 6

4. Experiments and Analysis

The experiments validate MemEye’s evidence-granularity and reasoning-depth axes, then map system failures and diagnose visual-evidence loss and evolving-state tracking. Current systems struggle to preserve fine visual details and prioritize the currently valid state across updates.

4.1. Evaluation Setup

  • The evaluation covers 13 text-based and multimodal memory methods across four backbones, using EM for multiple choice and LLM-as-a-Judge for open-ended answers.Images become dense captions for text-based methods, while multimodal methods use original visual inputs; judge agreement with human labels is Cohen’s κ = 0.94. Comparisons are system-level because recommended retrieval stacks are used where available.Source: §4.1 · pp. 6–7

4.2. Validation of MemEye

  • With gold clue rounds and original images, GPT-5.4-mini’s LLM-Judge score drops from 0.673 at Y1 to 0.601 at Y2 and 0.558 at Y3, supporting the reasoning-depth axis.System-level results also show retrieval methods doing well in Y1, while full-context or state-aware methods become more competitive in Y3.Source: §4.2 · pp. 7–8

4.3. RQ1: Where Do Current Memory Systems Fail in the MemEye Matrix?

  • SRAG(V) leads aggregate results with LLM-Judge = 0.4937 and EM = 0.6177, but current systems remain far from saturating MemEye.Table 2 reports cell-level EM for multiple-choice and LLM-Judge for open-ended questions; results on other backbones are in Appendix D.Source: Table 2 · p. 8, §4.3 · p. 8
  • At high X, SRAG(V) reaches 0.6554 versus A-Mem’s 0.4459 at (X3, Y1), while Y3 shifts the bottleneck from evidence access to selecting valid states.At (X4, Y1), MMA and SRAG(V) both score 0.6389, versus 0.3889 for the best text-based method; no method solves visual-detail preservation and evolving-state selection together.Source: §4.3 · pp. 8–9, Figure 6 · p. 9

4.4. RQ2: Why Do Memory Systems Lose Visual Information?

  • Text-based methods match image-based methods on coarse X1–X2 questions, but native-image memory better preserves fine-grained X3–X4 evidence.Caption-Proof gains are small in scene-level regions and become positive in fine-grained cells, consistent with captions losing decisive instance- and pixel-level evidence.Source: Figure 6 · p. 9, §4.4 · p. 10

4.5. RQ3: Why Do Memory Systems Lose Evolving Visual States?

  • Providing only the latest valid visual state yields performance close to the full oracle chain, indicating that Y3 failures center on recovering and prioritizing the current state.SRAG(V) can retrieve topic-related but stale or incomplete evidence; as histories grow more diverse, memory mechanisms become increasingly important for filtering unrelated evidence.Source: Figure 6 · p. 9, §4.5 · pp. 10–11

5. Implications for Memory Architecture Design

MemEye’s findings suggest multimodal memory should combine image evidence, text or structured state records, and mechanisms for selecting temporally valid evidence. The design must preserve fine-grained visual details while tracking how valid states change across sessions.

5. Implications for Memory Architecture Design

  • MemEye’s two axes call for preserving decisive visual evidence at the right granularity and selecting temporally valid evidence as memory evolves.The paper argues against designing multimodal memory as a single retrieval module.Source: §5 · p. 11
  • For high-Y questions, preserving images alone is insufficient: memory must identify the currently valid visual state while using images to verify text-based or structured updates, conflicts, and overrides.The proposed direction combines image evidence with structured state records and recency-aware selection.Source: §5 · p. 11
  • As histories grow, full-context methods become more sensitive to unrelated histories, while retrieval-based or structured memory stays more stable; filtering helps, but semantically relevant evidence can still be stale.Recency re-ranking reduces stale-over-latest errors but does not fully resolve errors arising during answer generation.Source: §5 · p. 11
  • Image-based memory is needed for high-X questions because text summaries can miss fine-grained visual information, while text or structured memory can record state updates across sessions.The paper recommends retaining both image-based and text-based memory: images preserve visual detail, while text or structured records help track state changes.Source: §5 · p. 11

6. Conclusion · A. Benchmark Construction and Dataset Details

MemEye evaluates whether multimodal agent memory preserves visual evidence and supports reasoning over it across time. Its benchmark and evaluation expose persistent memory failures while providing a diagnostic, curated assessment.

A. Benchmark Construction and Dataset Details

  • MemEye organizes evaluation with a two-axis taxonomy separating visual-evidence granularity from memory-reasoning depth.Source: §6 · p. 11
  • The benchmark contains 371 questions across eight life-scenario tasks.Source: §6 · p. 11
  • Clue-centered construction and validation gates target answerability, shortcut resistance, visual grounding, and reasoning-structure alignment.Source: §6 · p. 12

6. Conclusion

  • Across 13 memory architectures and 4 VLM backbones, current systems remain far from saturation, often losing fine-grained evidence, retrieving stale evidence, or failing to synthesize the currently valid state.These are the three recurring failure modes revealed by the MemEye matrix.Source: §6 · p. 12
  • Future multimodal memory systems should combine image evidence with text or structured state records and select temporally valid evidence over long histories.Source: §6 · p. 12
  • MemEye is diagnostic rather than exhaustive, focusing on curated life-scenario tasks, representative memory architectures, and system-level comparisons.Broader human baselines and deployment-scale studies are left to future work discussed in Appendix D.7.Source: §6 · p. 12

A.1. Task Statistics … B. Benchmark Validation

MemEye’s benchmark spans eight life-scenario tasks and labels questions by the visual evidence and memory operation needed to answer them. Its construction uses item-level audits, human review, and diagnostics to assess taxonomy reproducibility and whether the axes capture visual and temporal reasoning requirements.

A.1. Task Statistics

  • The eight tasks mix archival/public and generated/rendered visual evidence, including public comics and driving footage alongside synthetic faces and AI-generated scenes.Sources include Pitt Image Ads [16], Alley Oop and Seed-Story [45], Cardiverse [22], StyleGAN [18], and Japan Open Driving Dataset. Each task has multiple-choice questions and an equal number of mirrored open-ended questions.Source: §A.1 · p. 16, Table 4 · p. 17

A.3. Detailed Taxonomy Definitions

  • A highest-bottleneck rule assigns X by the finest decisive visual evidence required and Y by the deepest memory operation needed once that evidence is available.The taxonomy is intended to reflect evidence requirements and memory operations, rather than task names.Source: §A.3 · p. 17, Table 6 · p. 19

A.4. Item-Level Filtering and Taxonomy Audit Details

  • Shortcut audits flag candidates robustly answered across four option rotations (EM = 1.0) using only options, clue text, or minimal captions.The two-model panel is gpt-5.4-mini and gpt-5.2; the minimal-caption gate applies to X1–X4 candidates and does not determine their X labels.Source: §A.4 · pp. 17–18
  • Oracle screening flags candidates still at or below chance (EM ≤0.25) with gold clues and original images, while taxonomy audits require Y1 atomic, Y2 relational, and Y3 temporally updated evidence.Flagged candidates may be revised, removed, or relabeled; the oracle gate is meant to exclude visually ambiguous, underspecified, or overly difficult questions.Source: §A.4 · pp. 17–18

A.2. Taxonomy Cell Distribution

  • The benchmark reports question counts across the two-axis (Xi, Yj) taxonomy matrix, complementing its per-task statistics.The supplied passages identify the scope of these tables but do not provide their cell or task-level values.Source: §A.2 · p. 16, §A.1 · p. 16, Table 3 · p. 17

B. Benchmark Validation

  • Benchmark validation pairs caption substitution to test X-axis visual irreplaceability with oracle evidence to test Y-axis reasoning over retrieved memory.Caption-Proof compares matched dense-caption and native-image streams; the supplied figure and table captions describe this diagnostic but provide no outcome values.Source: Figure 7 · p. 20, Figure 8 · p. 21, Table 7 · p. 21

A.5. Annotation Agreement

  • κX = 0.66 and κY = 0.63; annotators agreed on 76 of 100 X labels and 77 of 100 Y labels, with X disagreements concentrated at the X2/X3 boundary.The 100-question subset was stratified across all (X, Y) cells and labeled independently by GPT-5.4 and Gemini-2.5-Pro from textual descriptions, not original images.Source: §A.5 · pp. 19–20
  • Human annotators reviewed all 371 final questions against the images, dialogue context, and clue structure; LLM agreement served only as a reproducibility check, not a replacement for human review.Disagreements were checked against human-adjudicated labels, which were revised only when review revealed an annotation error rather than a boundary judgment call.Source: §A.5 · pp. 18–20

B.1. Caption-Proof Validation Details … C. Evaluation Protocol and Implementation

The validation details test whether MemEye questions require native visual evidence, whether its evidence and reasoning axes reflect their intended demands, and whether sampled questions remain answerable by humans with oracle evidence. The human study is a sanity check, not a ceiling estimate.

B.1. Caption-Proof Validation Details

  • Caption-Proof validation compares matched textual and multimodal memories under the same architecture, replacing images with dense captions in the textual condition and retaining originals in the multimodal condition.Visual gain is defined as ∆= scoreV −scoreT; positive values indicate that original images preserve critical evidence captions do not.Source: §B.1 · p. 21
  • Table 7 and Figure 8 aggregate Caption-Proof gaps for Full Context, Semantic RAG, and SimpleMem on gpt-5.4-mini across Y1–Y3 within each visual-granularity level.The diagnostic highlights that captions often retain scene- and region-level evidence, while instance identity, exact attributes, small text, and temporally valid visual states are more likely to be lost.Source: §B.1 · p. 21

B.2. Oracle-Evidence Validation Details

  • Oracle-evidence validation gives each question its gold-clue rounds and original images while controlling for full-history search and preserving answer-relevant evidence.This removes retrieval difficulty to test whether the annotated axes reflect their intended memory requirements.Source: §B.2 · p. 22
  • LLM-Judge scores fall from 0.673 at Y1 to 0.601 at Y2 and 0.558 at Y3 with gold evidence, consistent with increasing reasoning demands.The levels represent atomic retrieval, relational association, and evolutionary synthesis over updates, overrides, or conflicts.Source: §B.2 · p. 22, Table 8 · p. 22
  • +0.122 at X1 versus +0.298 at X4, showing that native visual evidence matters more as decisive evidence shifts toward instance- and pixel-level details.The comparison uses text-only and multimodal gold-evidence settings.Source: §B.2 · p. 22, Table 8 · p. 22

B.3. Human Validation Under Oracle Evidence

  • Three annotators independently answer a stratified MCQ subsample using only gold-clue rounds and original images, with Y3 clues ordered chronologically and instructions to use the latest valid state.They do not see the full conversation history; majority-vote and mean individual accuracy are reported by Y level.Source: §B.3 · p. 22, Table 9 · p. 23
  • Human annotators are near-perfect on Y1 but less accurate on Y3, a sanity check that does not establish a human ceiling or imply visual unanswerability.Y3 errors can arise from resolving updates, conflicts, or overrides even when retrieval is removed; separate adjudication and oracle-answerability gates remove ambiguous or under-specified candidates.Source: §B.3 · p. 22, Table 9 · p. 23

C.1. Benchmark Runner and Method Settings … D.1. Complete Result Matrices

These sections specify MemEye’s evaluation and memory-method protocols, validate the open-ended judge against human ratings, and document its prompts, uncertainty estimates, implementations, and result matrices.

C.1. Benchmark Runner and Method Settings

  • The benchmark evaluates 13 memory methods across four VLM backbones, with text-based methods receiving image captions and supported multimodal methods retaining native images.Methods use a shared runner and evaluation pipeline, while retrieval-based methods have method-specific implementations and budgets.Source: §C.1 · p. 23, §C.5 · p. 27

C.2. LLM-as-a-Judge Validation

  • Human agreement with GPT-5.2 judging was 97.2% (69/71, excluding one borderline case), with Cohen’s κ = 0.94.The two disagreements concerned a navigation answer expressed through perceptual attributes rather than landmarks; agreement was 97.2% for FC(V) and 97.1% for SRAG(V).Source: §C.2 · p. 24

C.3. Prompt Examples

  • Question prompts target visual-evidence levels X1–X4 and reasoning levels Y1–Y3, requiring answerability from evidence and, for Y3, a later clue that resolves a stale clue.Annotation instructions select the finest decisive-evidence level and deepest required reasoning operation.Source: §C.3 · pp. 25–26

C.4. Bootstrap Confidence Intervals

  • Aggregate uncertainty is estimated with 10,000 question-level bootstrap resamples and percentile 95% confidence intervals, while paired comparisons preserve within-question score dependence.The intervals capture sampling uncertainty over questions, not uncertainty from the judge model or prompt; small taxonomy cells are treated diagnostically rather than as independent significance tests.Source: §C.4 · pp. 26–27

C.5. Memory Method Implementations

  • Implementations share benchmark prompts and decoding settings, but differ in memory construction and retrieval, including full-history context, top-10 Semantic RAG retrieval, and M2A’s two-phase memory-and-query process.Full Context uses native multimodal history for FC(V) and captions for FC(T); Semantic RAG (V) combines text and image similarities, while M2A expands retrieved evidence identifiers to source rounds.Source: §C.5 · pp. 27–28

D.1. Complete Result Matrices

  • The complete result matrices report EM, BLEU-1, and LLM-as-a-Judge scores for gpt-5.4-mini and provide corresponding matrices for three additional backbones.The additional backbones are gpt-4.1-nano, Qwen3-VL-8B-Instruct, and gemini-2.5-flash-lite; the supplied passages do not include matrix cell values.Source: §D.1 · p. 29

D.2. Caption Robustness Ablation

Task-aware captions reduce the visual–textual performance gap, closing it at low X but leaving a smaller positive gap at high X, where fine-grained visual evidence remains harder to recover textually.

D.2. Caption Robustness Ablation

  • The ablation uses gpt-5.4-mini with a task-aware prompt, producing captions typically 2–3× longer than generic captions.The prompt targets OCR content, exact colors, instance identities, spatial relations, and fine-grained visual attributes.Source: §D.2 · p. 29
  • Evaluation covers 120 stratified open-ended questions: 40 low-X questions from X1–X2 and 80 high-X questions from X3–X4, sampled across all eight tasks.FC(T) and SRAG(T) with task-aware captions are compared with original generic-caption and visual-stream runs on the same questions.Source: §D.2 · p. 29
  • At low X, task-aware-caption SRAG(T) matches or exceeds the visual stream (∆= −0.005); at high X, the gap shrinks from +0.194 to +0.041, and at X4 from +0.215 to +0.094.The remaining high-X gap is consistent with instance- and pixel-level evidence—such as identity bindings, fine textures, small text, and exact colors—being harder to recover through textual mediation.Source: §D.2 · p. 29
  • Task-aware captions raise SRAG(T) from 0.425 to 0.595 at low X and from 0.235 to 0.387 at high X.The captions use a stronger captioner and an exhaustive prompt targeting OCR, exact colors, instance identities, spatial relations, and fine-grained attributes.Source: §D.2 · p. 29

D.3. Effective Visual Information Analysis · D.4. Retrieval Diagnostics · D.5. Evolving Visual State Probe

MemEye’s analyses show that benchmark questions depend on visual evidence that captions may not preserve, while retrieval systems struggle to find and prioritize the latest clues in evolving visual states. Recency helps diagnose stale-evidence ranking but cannot replace retrieving missing clues or reasoning across the full update chain.

D.3. Effective Visual Information Analysis

  • The visual-information analysis holds question text fixed and compares no visual input, dense captions, and original images using the same answering model.For each benchmark, it uses annotated or constructed gold-clue contexts and evaluates open-ended answers with the same LLM-as-a-Judge protocol.Source: §D.3 · pp. 29–30
  • MemEye shows a larger caption-only-to-multimodal performance gain than prior benchmarks, indicating its questions contain more irreplaceable visual evidence.The comparison uses gold clue contexts to focus on answer-relevant image information beyond textual context or captions, rather than on long-context retrieval.Source: §D.3 · pp. 29–30

D.4. Retrieval Diagnostics

  • On Y3, SRAG(V) has Full-Clue Recall@10 of 0.367 and Latest-Clue Recall@10 of 0.533, while MMA retrieves clue evidence slightly more often but has Stale-Dominance of 0.750 versus SRAG(V)’s 0.767.Native visual retrieval improves Any-Clue Recall@10 over SRAG(T), especially at Y1 and Y3; the diagnostics distinguish missing evidence, incomplete clue chains, and stale evidence ranked above the latest clue.Source: §D.4 · pp. 30–31, Table 12 · p. 31
  • α=0.7 yields the larger Y3 answer-quality point estimate (+0.067 Judge), while α=0.5 yields the larger Stale-Dominance reduction (−0.183 versus −0.083).Together, the settings show recency is a diagnostic signal for temporal-authority failures, not a complete solution; systems must also retrieve missing latest evidence and reason over the update chain.Source: §D.4 · p. 31, Table 14 · p. 33
  • The recency probe reduces Stale-Dominance and Rank-Inversion but does not improve Latest-Miss; Y3 answer-quality point estimates improve, but confidence intervals overlap zero.The probe reranks SRAG(V)’s fixed candidate pool using retrieval similarity and exponential recency, so it can correct stale-over-latest ranking but cannot recover an unretieved latest clue.Source: §D.4 · p. 31, Table 13 · p. 32

D.5. Evolving Visual State Probe

  • Evolving-state questions require systems to recover image-based clues and determine which state remains valid; case studies show failures from missed latest clues, stale evidence, or incomplete temporal chains.These visual updates can be difficult to expose in benchmarks where updates are primarily textual.Source: §D.5 · p. 32
  • The all-clue oracle scores highest, latest-only is slightly lower, stale-only is lower still, and all memory systems remain far below these controlled evidence settings.The comparison indicates that full visual update chains improve answerability, earlier states can be needed to compare changes, and stale evidence can support outdated or incomplete answers.Source: §D.5 · p. 32, Table 15 · p. 33

D.6. Cross-topic Dialogue Scaling Ablation

The cross-topic scaling ablation tests whether memory systems can retrieve fixed answer-relevant evidence as unrelated domains are added to long histories. Full Context becomes more sensitive to interference, while retrieval-based and structured methods are more stable, underscoring the need to route relevant evidence while preserving visual details and state information.

D.6. Cross-topic Dialogue Scaling Ablation

  • The ablation compares three scales over fixed questions, increasing from clean single-domain histories to pairwise combinations and full four-domain histories.Two controlled combinations cover Brand Memory, Social Chat, Cartoon Entertainment, and Card Playlog, or Health Care, Home Renovation, CrossScene Memory, and Outdoor Navigation; this design isolates cross-domain interference.Source: §D.6 · p. 33, Figure 9 · p. 34
  • Full Context (V) declines as unrelated histories grow, whereas Semantic RAG (V) and MMA remain comparatively stable across memory scales.Figure 9 reports MCQ EM and LLM-as-a-Judge trends across two controlled four-domain combinations.Source: Figure 9 · p. 34, §D.6 · p. 34
  • Full Context (V) may decline because larger histories require filtering more irrelevant visual-textual evidence and may truncate or dilute early relevant evidence.The truncation or dilution risk increases as long histories approach the context window.Source: §D.6 · p. 34
  • M2A shows partial resilience in open-ended evaluation, but its lower absolute scores suggest that abstraction must preserve fine-grained visual evidence.Agentic operations such as memory writing, conflict checking, and iterative querying may help manage noisy histories.Source: §D.6 · p. 34
  • Scaling multimodal memory requires evidence routing that filters unrelated history while preserving visual details and state information needed for the current question.The ablation indicates that simply increasing the context window is not sufficient.Source: §D.6 · p. 34

D.7. Limitations and Broader Impacts

MemEye is a diagnostic benchmark, not a complete simulation of multimodal-agent deployments or a universal ranking of memory systems. More reliable visual memory may reduce stale reasoning but can increase privacy risks, motivating safeguards.

D.7. Limitations and Broader Impacts

  • MemEye’s scenarios, model panel, captioning pipeline, and human validation sample may not cover every real-world setting, and its oracle check is only a small sanity check.The human oracle check in Appendix B.3 is not a full human ceiling estimate.Source: §D.7 · p. 34
  • Comparisons across memory architectures can reflect method-specific encoders and implementations, so results indicate design trade-offs rather than a universal ranking.Source: §D.7 · p. 34
  • More reliable visual memory may reduce stale or incorrect state reasoning, but systems storing user images or evolving states raise privacy risks and need consent, minimization, deletion, and access controls.Source: §D.7 · pp. 34–35

E. Case Studies · E.1. Textual Memory Case Study

The case studies show that caption-based memory can discard decisive visual details, while retrieval of related images alone may not resolve evolving states. The textual-memory cases add that compact state abstractions can help track updates, but still need preserved visual evidence.

E. Case Studies

  • Cases 1–5 expose visual-evidence loss: captions omit identity cues, fine-grained attributes, or the precise visual state needed to answer.Cross-session identity requires cues such as body shape, color patterns, and character pairings; other cases depend on comparing cave ceilings, recalling exact eye color, tracking an A→B→A recommendation change, or counting cards at a specific state.Source: E. Case Studies · pp. 40–42
  • Cases 6–8 show that resolving evolving states requires temporal reasoning: stale-majority evidence, object migration, and mixed local events can mislead retrieval-based answers.The correct answer requires later evidence to override an older tag, temporal ordering to identify an object's current location, or sequence-level integration to determine an overall narrative direction.Source: Figure 12 · p. 42, E. Case Studies · pp. 42–43
  • Cases 9–11 isolate retrieval failures: missing later evidence or the trigger state leaves models with stale or incomplete visual evidence.The backpack's later nearby appearance is missed; the card-game answer requires retrieving the preceding hand-size change as well as the answer state; and methods retrieve the bag's old shelf location but miss its later desk location.Source: E. Case Studies · p. 43, Figure 13 · p. 44, Figure 14 · p. 45

E.1. Textual Memory Case Study

  • The textual-memory cases suggest that compact state extraction can outperform visually similar image retrieval in some Y3 scenarios, but does not replace visual evidence.Figure 15 compares A-Mem, M2A, and MMA retrieval for two evolving-state cases; the reported advantage comes from a structured textual evidence chain tracking state changes.Source: §E.1 · p. 45, Figure 15 · p. 46

Related papers

Loading 2605.15128v1…