PaperlayerView exact revision on arXiv

Source-linked AI summary

MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory

Minghao Guo, Qingyue Jiao, Zeru Shi, Yihao Quan, Boxuan Zhang, Danrui Li, Liwei Che, Wujiang Xu, Shilong Liu, Zirui Liu, Mubbasir Kapadia, Vladimir Pavlovic, Jiang Liu, Mengdi Wang, Yiyu Shi, Dimitris N. Metaxas, Ruixiang Tang

arXiv:2605.15128v1cs.CVcs.CLcs.IR

TL;DR

Existing evaluations rarely test whether multimodal agents preserve fine-grained visual evidence or reason over visual states that change over time. MemEye introduces a two-dimensional benchmark and finds that current systems lose detailed evidence, retrieve stale states, and struggle to synthesize the currently valid state.

  • Problem

    Existing evaluations provide limited evidence about preserving fine-grained visual evidence and reasoning over visual states that change across sessions.

  • Method

    MemEye evaluates multimodal memory along visual-evidence granularity and memory-reasoning depth, using 371 questions across eight life-scenario tasks.

  • Results

    Current systems remain far from reliable long-term visual memory, losing fine-grained evidence, retrieving stale evidence, and failing to synthesize currently valid states.

  • Takeaways & Limitations

    Effective multimodal memory must preserve visual details, track temporal validity, and select the right evidence over long histories.

  • Takeaways & Limitations

    MemEye is diagnostic rather than exhaustive, and architecture comparisons can be affected by method-specific encoders and implementations.

Abstract

from arXiv · show

Long-term agent memory is increasingly multimodal, yet existing evaluations rarely test whether agents preserve the visual evidence needed for later reasoning. In prior work, many visually grounded questions can be answered using only captions or textual traces, allowing answers to be inferred without preserving the fine-grained visual evidence. Meanwhile, harder cases that require reasoning over changing visual states are largely absent. Therefore, we introduce MemEye, a framework that evaluates memory capabilities from two dimensions: one measures the granularity of decisive visual evidence (from scene-level to pixel-level evidence), and the other measures how retrieved evidence must be used (from single evidence to evolutionary synthesis). Under this framework, we construct a new benchmark across 8 life-scenario tasks, with ablation-driven validation gates for assessing answerability, shortcut resistance, visual necessity, and reasoning structure. By evaluating 13 memory methods across 4 VLM backbones, we show that current architectures still struggle to preserve fine-grained visual details and reason about state changes over time. Our findings show that long-term multimodal memory depends on evidence routing, temporal tracking, and detail extraction.

1. Introduction

MemEye addresses the lack of evaluations for preserving irreplaceable visual evidence and reasoning over changing visual states in long-term multimodal memory. It evaluates memory along visual evidence granularity and memory reasoning depth, revealing persistent trade-offs between detail preservation and state tracking.

  • Motivation: Existing benchmarks often emphasize short-context image understanding or textual long-term memory, while many visual questions remain answerable without retained visual evidence.Captions, dialogue, or answer options can provide shortcuts that bypass fine-grained visual memory.
  • Motivation: Visual compression into captions can lose region layouts, object identities, and fine-grained textures, while agents also struggle to synthesize evidence across sessions as observations override earlier states.These challenges require both detailed visual preservation and reasoning beyond simple retrieval.
  • MemEye framework: MemEye evaluates multimodal memory along visual evidence granularity, from scene-level to pixel-level, and memory reasoning depth, from atomic retrieval to evolutionary synthesis.The two dimensions measure whether systems preserve decisive visual details and reason over retrieved evidence.
  • Findings: 13 memory methods across four vision-language model backbones show that current systems remain far from reliable long-term visual memory.The evaluation identifies persistent limitations in preserving visual details and reasoning about which state remains valid over time.
  • Findings: Text-based memory helps organize state transitions and updates but loses fine-grained visual details, whereas native image memory preserves visual evidence more directly but struggles with temporal state validity.The benchmark includes 371 mirrored questions across real-world scenarios to test preservation and reasoning over irreplaceable visual evidence.

2. Related Work

Prior memory systems largely store textual abstractions, while newer multimodal methods retain visual experience but lack evaluations that isolate visual-memory failures. Existing benchmarks also rarely test whether decisive visual evidence survives compression, must be preserved at fine granularity, or changes over time.

  • Memory Systems: Earlier computer-use and interactive-agent memory systems primarily operate over textual memories or text abstractions of prior experience.These methods use explicit memory writing, updating, and maintenance procedures to store and reuse past information.
  • Multimodal Memory: Recent multimodal methods retain or retrieve visual experience, but existing evaluations often report end-task performance without isolating which memory operation fails.The cited systems make architectural trade-offs among coverage, retrieval selectivity, abstraction, and revision.
  • MemEye Taxonomy: MemEye organizes evaluation along visual-evidence granularity and the reasoning operation required over memory.Its taxonomy uses one axis for decisive visual-evidence granularity and another for the required reasoning operation.
  • Benchmarks: Long-horizon benchmarks mostly test recovery, summarization, or use of linguistic facts, while multimodal benchmarks introduce images within dialogue.Examples include LoCoMo, LongMemEval, TwinVoice, MemoryAgentBench, MMDU, ATM-Bench, MMRC, and Mem-Gallery.
  • Evaluation Gap: Prior benchmarks rarely test whether captions can bypass decisive image content, whether fine-grained evidence must be preserved at instance or pixel granularity, or whether visual evidence changes over time.The missing object of study is the coupled failure mode between visual evidence compression and state-evolving memory use.

3. MemEye Framework and Benchmark

MemEye organizes memory evaluation along visual-evidence granularity and retrieval reasoning depth, assigning each question an (X, Y) coordinate. Its benchmark combines diverse multimodal life-scenario questions with filtering mechanisms intended to retain items requiring visual information and memory capabilities.

  • Framework dimensions: MemEye’s X-axis measures visual-evidence granularity from scene-level X1 to increasingly fine-grained evidence at X4.X1 captures scene type, activity, and global semantic gist, while higher levels require more localized visual evidence.
  • Framework dimensions: The Y-axis measures retrieval reasoning depth from atomic single-fact retrieval Y1 through relational association Y2 to Y3.This dimension progresses from basic memory access toward associating, revising, and synthesizing evidence across sessions and modalities.
  • Framework dimensions: Each question receives an (X, Y) coordinate indicating its required visual-evidence level and reasoning depth over memory.The benchmark assigns each item the most demanding X and Y labels needed to answer it.
  • Benchmark construction: The benchmark contains 371 questions across 221 sessions, 848 dialogue rounds, and 438 images, with mirrored multiple-choice and open-ended forms.It spans eight tasks across four life-scenario domains and includes photographs, screenshots, comic panels, user-interface renderings, archival media, public media, and generated content.
  • Benchmark construction: Three filtering mechanisms remove questions answerable without visual evidence, isolate foundation-model answerability, and retain items suitable for evaluating memory capabilities.Answer rotations mitigate VLM bias; dialogue-only checks test answer leakage, while visual-bypassability checks distinguish foundation-model limitations from memory limitations.

4. Experiments and Analysis

Experiments evaluate 13 memory methods across four VLM backbones and show that current systems remain far from saturating MemEye. Their failures reflect two interacting bottlenecks: preserving fine-grained visual evidence and tracking which visual state remains valid over time.

  • Experimental Setup: 13 memory methods are evaluated across 4 model backbones, comprising seven text-based and six multimodal approaches.Text-based methods replace images with dense GPT-5.2 captions, whereas multimodal methods operate on original visual inputs.
  • Metrics and Diagnostics: Cohen’s κ = 0.94 indicates strong agreement between automated and human accept/reject judgments on a stratified sample of 72 predictions.Exact-match accuracy is used for multiple-choice questions, while LLM-as-a-Judge is the primary open-ended metric.
  • Main Results: SRAG(V) achieves the best aggregate open-ended performance with LLM-Judge = 0.4937 and the best multiple-choice performance with EM = 0.6177.Table 2 reports cell-level EM for multiple-choice questions and LLM-Judge for free-response questions.
  • Visual Evidence Granularity: At (X3, Y1), SRAG(V) reaches LLM-as-a-Judge = 0.6554 versus A-Mem at 0.4459, while at (X4, Y1), MMA and SRAG(V) both reach 0.6389 versus 0.3889 for the best text-based method.Native visual memory becomes more important at high X, whereas caption-based memory remains competitive at low X.
  • Evolving Visual State Tracking: Y3 failures arise because systems may retrieve the right topic while missing the decisive latest clue or complete update chain, making state selection the bottleneck.Native-image benefits weaken in Y3 because the answer depends on which visual state remains valid after later updates; semantic similarity does not guarantee temporal validity.

5. Implications for Memory Architecture Design

MemEye’s findings imply that multimodal memory should combine image-based and text-based representations rather than rely on a single retrieval module. It must preserve fine-grained visual evidence, track changing states, and select temporally valid evidence while avoiding stale or unrelated history.

  • Architecture requirements: Memory architecture should not be designed as a single retrieval module because visual-evidence granularity and temporally valid state selection impose complementary requirements.The X-axis concerns preserving decisive visual evidence at the right granularity, while the Y-axis concerns selecting valid evidence as memory evolves.
  • Storing visual information: Text-based memory captures coarse visual information but often loses fine-grained details when images are converted into text.Image-based memory is needed for questions whose answers depend on visual information omitted by captions.
  • Selecting valid visual states: Preserving image evidence alone is insufficient when visual states change across sessions, because the system must determine which state remains valid.Text-based or structured memory helps record updates, conflicts, and overrides, while image-based memory supports checking those states.
  • Combined memory design: A robust multimodal memory should retain both image-based and text-based memory, combining visual detail preservation with cross-session state updates.The proposed direction builds on both representations rather than treating either as sufficient alone.
  • Evidence selection: Evidence-selection mechanisms should filter unrelated history, use temporal signals, and avoid stale evidence when retrieving memory.These mechanisms are needed on top of image and text memory to select evidence that remains valid.

6. Conclusion · A. Benchmark Construction and Dataset Details

MemEye evaluates whether multimodal memory preserves visual evidence and reasons over it across time, using a two-axis taxonomy and a validated benchmark. Results across memory architectures and VLM backbones expose persistent failures in detail preservation, temporal retrieval, and state synthesis.

  • 6. Conclusion: MemEye separates visual evidence granularity from memory reasoning depth in its two-axis evaluation taxonomy.The framework targets memory systems that preserve and reason over visual evidence across time.
  • 6. Conclusion: The benchmark contains 371 questions spanning eight life-scenario tasks.It was constructed under the MemEye framework for long-term multimodal agent memory evaluation.
  • 6. Conclusion: Clue-centered construction and validation gates test answerability, shortcut resistance, visual grounding, and reasoning-structure alignment.These gates are used to validate the benchmark’s intended evaluation properties.
  • 6. Conclusion: 13 memory architectures evaluated across 4 VLM backbones remain far from saturation.The evaluation indicates substantial room for improvement in current multimodal memory systems.
  • 6. Conclusion: The MemEye matrix identifies three recurring failures: losing fine-grained visual evidence, retrieving stale evidence, and failing to synthesize the currently valid state.These failure modes span visual detail retention, temporal evidence validity, and state reasoning.
  • 6. Conclusion: Future systems should combine image evidence with text or structured state records and select temporally valid evidence over long histories.The proposed direction addresses the observed failures in evidence preservation, retrieval, and state synthesis.

A.1. Task Statistics … B. Benchmark Validation

MemEye constructs and validates a multimodal memory benchmark by organizing questions around decisive visual evidence and required memory operations. Its pipeline combines provenance-aware task data, taxonomy audits, shortcut and oracle gates, human adjudication, and reproducibility checks.

  • A.1. Task Statistics: The benchmark spans eight life-scenario tasks with archival/public or generated/rendered visual sources and mirrored multiple-choice and open-ended questions.Examples include advertisement images, public-domain comics, interior-design photographs, rendered game-state screenshots, and synthetic face images.
  • A.2. Taxonomy Cell Distribution: Table 5 reports the number of questions in each (X_i, Y_j) taxonomy cell, documenting the benchmark’s distribution across evidence and memory-operation levels.The distribution is presented as a two-axis taxonomy matrix.
  • A.3. Detailed Taxonomy Definitions: MemEye assigns each question an (X, Y) coordinate using the finest decisive visual-evidence level and deepest required memory operation.The taxonomy is attached to evidence requirements and memory operations rather than task names.
  • A.4. Item-Level Filtering and Taxonomy Audit Details: Stage 2 filters candidates with four-way answer rotations and gpt-5.4-mini and gpt-5.2, flagging robust shortcut success, oracle-unsolved items, or mismatched Y-level clue structures.Robust shortcut success requires both models to answer all rotations correctly, while oracle unsolved means rotation-averaged EM ≤0.25 for both models.
  • 5 Clue Sufficiency: The validation gates test option-only and text-only shortcut resistance, oracle visual answerability, and whether clue structures satisfy Y1, Y2, or Y3 reasoning requirements.Y1 uses one evidence unit, Y2 associates multiple nonredundant units, and Y3 resolves temporally ordered state updates, overrides, or conflicts.
  • A.5. Annotation Agreement: 100 stratified questions were independently labeled by GPT-5.4 and Gemini-2.5-Pro using the full taxonomy definitions, without access to original images.The study checks label reproducibility across all (X, Y) cells and does not replace human adjudication.
  • 7. Text-Leak Rejection: The construction pipeline generates target-coordinate questions, creates mirrored MCQ/open-ended items with clue annotations and answer rotations, then removes shortcut-answerable or underspecified candidates.It also verifies that clue structures match annotated Y levels and tests whether captions can replace visual evidence.
  • B. Benchmark Validation: All 371 final benchmark questions underwent human review of images, dialogue context, and clue structure, while LLM agreement served only as a taxonomy reproducibility check.Disagreements were cross-checked against human-adjudicated gold labels.

B.1. Caption-Proof Validation Details … C. Evaluation Protocol and Implementation

The validation suite tests whether MemEye requires native visual evidence and whether its axes capture increasing reasoning difficulty and visual specificity. Oracle-evidence and human checks support the benchmark’s intended distinctions while showing that state synthesis remains challenging.

  • B.1. Caption-Proof Validation Details: Caption-Proof validation compares matched textual and multimodal memories, replacing images with dense captions in the textual system and reporting visual gain ∆= scoreV −scoreT.Positive visual gain indicates that native images preserve critical evidence unavailable from captions alone.
  • B.1. Caption-Proof Validation Details: Caption-Proof gaps diagnose where captions lose evidence, especially instance identity, exact attributes, small text, and temporally valid visual states.Scene- and region-level evidence are often included in captions, whereas finer-grained evidence is more likely to be lost.
  • B.2. Oracle-Evidence Validation Details: Oracle-evidence validation provides gold-clue rounds and original images while removing retrieval difficulty, testing whether the annotated axes reflect intended memory requirements.The protocol controls search over the full history while preserving the evidence needed to answer.
  • B.2. Oracle-Evidence Validation Details: 0.673 →0.601 →0.558 LLM-Judge scores from Y1 to Y3 show that reasoning remains harder with relevant evidence provided, supporting atomic retrieval, relational association, and evolutionary synthesis.LLM-Judge is the primary metric; EM and BLEU-1 are auxiliary metrics.
  • B.2. Oracle-Evidence Validation Details: +0.122 at X1 to +0.298 at X4 shows that native visual evidence becomes more important as decisive evidence shifts from scene-level to instance- and pixel-level details.The X-axis comparison uses text-only and multimodal gold-evidence settings.
  • B.3. Human Validation Under Oracle Evidence: Three annotators independently answer a stratified MCQ subsample using only gold clue rounds and original images, with Y3 rounds presented chronologically for latest-valid-state judgments.Annotators do not see the full conversation history.
  • B.3. Human Validation Under Oracle Evidence: Near-perfect human accuracy on Y1 decreases on Y3, but the lower Y3 score reflects possible state-resolution errors across updates, conflicts, or overrides rather than visual unanswerability.The study is a small-scale human oracle sanity check, not a full human ceiling estimate.

C.1. Benchmark Runner and Method Settings … D.2. Caption Robustness Ablation

The benchmark standardizes multimodal memory evaluation across four VLM backbones and thirteen methods, with explicit controls for caption-only versus native-visual evidence. Validation and ablation analyses establish judge reliability, uncertainty estimation, complete result reporting, and the remaining high-detail advantage of visual streams.

  • C.1. Benchmark Runner and Method Settings: The benchmark evaluates 4 VLM backbones and 13 memory methods under a common comparison framework.Backbones include Qwen3-VL-8B-Instruct, gpt-4.1-nano, gpt-5.4-mini, and gemini-2.5-flash-lite.
  • C.1. Benchmark Runner and Method Settings: Textual methods receive dialogue plus dense captions, whereas native multimodal methods receive the original visual inputs.This separates caption-based memory from native-image memory during evaluation.
  • C.2. LLM-as-a-Judge Validation: Automated judging agrees with human labels on 97.2% of items, with Cohen’s κ = 0.94.Agreement excludes one borderline case and covers 69 of 71 items.
  • C.3. Prompt Examples: The prompts require answers to remain grounded in dialogue and images, while caption-only settings explicitly prohibit assuming access to raw pixels.The taxonomy spans X1 scene-level through X4 fine visual evidence and Y1 single-evidence through Y3 state-change reasoning.
  • C.4. Bootstrap Confidence Intervals: Uncertainty is estimated with paired, question-level percentile 95% confidence intervals from 10,000 bootstrap resamples.Multiple-choice rotations remain within-question variants rather than independent samples.
  • C.5. Memory Method Implementations: All methods use the same runner, while implementations differ in memory construction, retrieval, truncation, reflection, and multimodal evidence handling.Examples include FC full-history baselines, SRAG top-K retrieval, SimpleMem raw-image pointers, and M2A evidence expansion.
  • D.1. Complete Result Matrices: Complete evaluation matrices report EM, BLEU-1, and LLM-as-a-Judge results for gpt-5.4-mini and three additional VLM backbones.The additional matrices cover gpt-4.1-nano, Qwen3-VL-8B-Instruct, and gemini-2.5-flash-lite.
  • D.2. Caption Robustness Ablation: At high X, task-aware captions reduce SRAG(T)’s visual-stream gap from +0.194 to +0.041, while at X4 the gap decreases from +0.215 to +0.094.At low X, task-aware captions raise SRAG(T) from 0.425 to 0.595 and close the gap, with ∆= −0.005.

D.3. Effective Visual Information Analysis

This analysis tests whether answer-relevant visual information can be replaced by text, using matched no-visual, caption-only, and native-image settings with gold clue contexts. MemEye shows a larger caption-to-multimodal gain than prior benchmarks, indicating greater visual irreplaceability.

  • Analysis Setup: The analysis isolates visual information by using gold clue rounds, removing memory search as the primary bottleneck.For MemEye, annotated clue rounds are used; prior benchmarks receive comparable gold-clue contexts from answer-relevant evidence.
  • Analysis Setup: The same answering model is evaluated with no visual information, dense captions, or original images while keeping question text fixed.The three settings are No Visual Info., Caption Only, and Multimodal, with gold clue contexts held constant.
  • Evaluation Protocol: The evaluation uses open-ended answers scored with the same LLM-as-a-Judge protocol as the main experiments.Captions are generated with gpt-5.2 using the dense captioning prompt from the caption-based memory experiments.
  • Findings: MemEye has a larger gain from caption-only to multimodal input than prior benchmarks.This comparison measures visual irreplaceability: a larger multimodal advantage means native images preserve information that captions do not.

D.4. Retrieval Diagnostics · D.5. Evolving Visual State Probe

The retrieval diagnostics show that native visual retrieval improves evidence access but still misses complete, current update chains, while recency re-ranking corrects some stale rankings without recovering absent latest clues. The evolving visual-state probe confirms that answering requires temporally authoritative visual evidence and reasoning across changing image-grounded states.

  • D.4. Retrieval Diagnostics: SRAG(V) improves Any-Clue Recall@10 over SRAG(T), but Y3 Full-Clue Recall@10 is only 0.367 and Latest-Clue Recall@10 is only 0.533.These results indicate that many failures involve incomplete evidence chains or missing current-state clues rather than total retrieval failure.
  • D.4. Retrieval Diagnostics: Semantic relevance can surface the right topic while selecting an outdated visual state, with MMA Stale-Dominance at 0.750 versus 0.767 for SRAG(V).MMA provenance is expanded from semantic-memory evidence rather than direct raw-round retrieval, so it requires careful interpretation.
  • D.4. Retrieval Diagnostics: The recency probe reduces Stale-Dominance and Rank-Inversion, but Latest-Miss does not improve because re-ranking cannot recover missing latest evidence.The candidate pool remains fixed, so the probe diagnoses ranking errors rather than adding a new memory capability.
  • D.4. Retrieval Diagnostics: The α=0.7 setting yields the larger Y3 answer-quality point estimate (+0.067 Judge), whereas α=0.5 yields the larger Stale-Dominance reduction (−0.183 versus −0.083).The trade-off shows that recency is useful diagnostically but does not replace retrieval of missing evidence or reasoning over update chains.
  • D.5. Evolving Visual State Probe: The evolving visual-state subset requires sequences of visual clues in which later images update, conflict with, or override earlier images.Correct answers require recovering relevant visual evidence and determining which visual state remains valid.
  • D.5. Evolving Visual State Probe: The all-clue oracle obtains the highest LLM-as-a-Judge score, while Latest-only is slightly lower and Stale-only is lower than both.Latest-only can lack earlier states needed to compare changes, whereas stale-only can support plausible but outdated answers.
  • D.5. Evolving Visual State Probe: MemEye exposes temporal-authority failures over native visual evidence, including missing the latest clue, over-weighting stale evidence, and retrieving fragments without tracking the full temporal chain.These failure patterns are difficult to expose when updates are primarily textual.

D.6. Cross-topic Dialogue Scaling Ablation

The cross-topic scaling ablation tests memory robustness as answer-relevant evidence is embedded in increasingly diverse histories. Results show that context expansion can cause interference, whereas retrieval-based and structured methods are comparatively more stable.

  • Experimental setup: The ablation evaluates three memory scales across two controlled four-domain combinations using the same underlying questions.The combinations are Brand Memory, Social Chat, Cartoon Entertainment, and Card Playlog; or Health Care, Home Renovation, CrossScene Memory, and Outdoor Navigation.
  • Scaling trends: Full Context (V) declines as unrelated histories are added, especially for the Health Care, Home Renovation, CrossScene Memory, and Outdoor Navigation combination.The decline may reflect increased filtering of irrelevant visual-textual evidence and histories approaching the context window, risking loss of early answer-relevant evidence.
  • Scaling trends: Semantic RAG (V) remains comparatively robust across memory scales, suggesting targeted retrieval reduces irrelevant context before answer generation.This pattern contrasts with the interference observed for Full Context (V).
  • Scaling trends: MMA exhibits flatter trends, indicating that structured multimodal memory can reduce interference as histories become more diverse.MMA is grouped with retrieval-based and structured methods as more stable than Full Context (V).
  • Scaling trends: M2A shows partial resilience in open-ended evaluation, suggesting agentic operations can help manage noisy histories.The passage identifies memory writing, conflict checking, and iterative querying as potentially helpful operations.
  • Implication: Scaling multimodal memory requires evidence-routing mechanisms that filter unrelated history while preserving visual details and state information needed for the current question.The ablation indicates that simply increasing the context window is insufficient.

D.7. Limitations and Broader Impacts

MemEye is a diagnostic benchmark whose scenarios, models, captioning pipeline, and human validation may not represent every deployment, so comparisons indicate design trade-offs rather than universal rankings. Stronger visual memory may improve visual-state reasoning but also heighten privacy risks, requiring safeguards such as consent and access control.

  • Limitations: MemEye’s scenarios, model panel, captioning pipeline, and human validation sample may not cover every real-world multimodal-agent setting.The benchmark is intended as a diagnostic evaluation rather than a complete simulation of all deployments.
  • Limitations: Method-specific encoders and implementations can affect architecture comparisons, so results should be interpreted as design-trade-off evidence rather than a universal ranking.The limitation applies to comparisons across memory architectures.
  • Broader Impacts: More reliable visual memory may reduce stale or incorrect visual-state reasoning, but storing user images, environments, or evolving states may increase privacy risks.The broader-impact concern arises when agents retain personal visual information over time.
  • Broader Impacts: Future multimodal memory systems should pair evidence preservation with consent, data minimization, deletion, and access-control mechanisms.These safeguards address privacy risks associated with retaining personal visual evidence.

E. Case Studies · E.1. Textual Memory Case Study

The case studies show that MemEye exposes failures in preserving fine-grained visual evidence and resolving evolving visual states, including identity, attributes, temporal triggers, migrations, and trajectories. The textual-memory cases further show that compact evidence chains can outperform visually similar retrieval when answers depend on state evolution.

  • E. Case Studies: Cases 1–5 show that captions lose decisive visual evidence for identity tracking, micro-attribute comparison, exact color recall, belief revision, and game-state counting.These cases require native visual input because captions omit identity cues, contrastive scene details, exact eye color, the latest message state, or structured card counts.
  • E. Case Studies: The case-study interpretation separates visual-evidence loss in Cases 1–5 from evolving-state failures in Cases 6–11.Preserving dialogue or semantically related images is insufficient when decisive details or temporally valid evidence are missing.
  • E. Case Studies: Cases 6–8 show that stale-majority retrieval, object migration, and narrative trajectories require temporal ordering or sequence integration rather than semantic similarity alone.The correct answers depend on later evidence overriding earlier evidence, distinguishing current from historical locations, or judging the overall direction across local ups and downs.
  • E. Case Studies: Cases 9–11 expose retrieval failures in which systems miss later evidence, retrieve the wrong trigger state, or let visually salient stale evidence determine the answer.The failures involve a nearby object reappearance, a card-state transition, and a plastic bag moving from an earlier shelf to a later desk location.
  • E.1. Textual Memory Case Study: In the brass-compass case, A-Mem preserves a textual chain from fossil-case presence and absence to appearance on the restoration table, yielding the correct current location.M2A and MMA retain stale or visually similar evidence without explicitly representing the move.
  • E.1. Textual Memory Case Study: Across the textual-memory cases, A-Mem’s advantage comes from compact state extraction that preserves evolving evidence chains more reliably than retrieving visually similar raw images.This advantage concerns structured state tracking rather than richer visual recall.
  • E.1. Textual Memory Case Study: In the paint case, A-Mem’s chain identifies sage green as tested but rejected after the design pivots to terracotta and the final room uses terracotta.M2A loses the exact label, while MMA separates the visual entries without extracting the tested-but-not-final relation.
Loading 2605.15128v1…