Source-linked AI summary
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu, Yepeng Liu, Lin Long, Yichen Guo, Nuo Chen, Zhaotian Weng, Elena Kochkina, Simerjot Kaur, Charese Smiley, Xiaomo Liu, James Zou, Sheng Liu, Yuheng Bu, Songyou Peng, Xin Eric Wang
TL;DR
Long-horizon multimodal agents need memory that tracks evolving worlds and supports decisions, yet existing benchmarks largely test static recall and underdiagnose lifecycle failures. The paper reframes memory as an Action-World Interaction Loop and introduces WorldMemArena to evaluate writing, maintenance, retrieval, and use across evolving-state and agentic tasks. Results show that storage quality alone does not predict performance, visual evidence remains difficult to reuse, and harness-based memory is flexible but costly and less reliable.
Problem
Existing evaluations focus on static dialogue recall, omit lifecycle-level diagnosis, and provide limited pressure for multimodal evidence use.
Method
The paper formulates memory as an Action-World Interaction Loop and introduces WorldMemArena with staged lifecycle annotations for multi-session multimodal tasks.
Results
Storage quality alone does not guarantee better performance; visual evidence remains difficult to reuse, and harness memory is flexible but costly and less reliable.
Takeaways & Limitations
Effective agent memory should be developed as an adaptive capability grounded in interaction rather than optimized solely as a static module.
Takeaways & Limitations
Current systems rarely revise or remove obsolete entries, showing that state maintenance remains constrained by append-only behavior.
Abstract
from arXiv · showhide
Multimodal large language models are increasingly deployed as long-horizon agents, where memory must do more than recall: it must track an evolving world, revise what has gone stale, and surface the right evidence at decision time. Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions, leaving us unable to localize failures to writing, maintenance, retrieval, or use. The rise of agent harnesses that author their own memory sharpens this gap, since we have no principled way to compare hand-designed pipelines with self-managing alternatives. To close these gaps, we formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle, and instantiate it in WorldMemArena: 400 multi-session multimodal tasks spanning Lifelong Evolution (evolving personal and task states) and Agentic Execution (memory from real observations, actions, and feedback), annotated with gold memory points, updates, distractors, and evidence chains for stage-level diagnosis. This enables the first head-to-head comparison of long-context, manually designed (RAG and external memory systems), and harness-based memory agents. Results show that: (1) better memory writing and storage do not guarantee better performance; (2) multimodal memory still struggles to fully use visual evidence; (3) systems are unstable across domains and degrade on realistic agentic trajectories; and (4) harness memory is more flexible but remains costly and less reliable.
1. Introduction
Long-horizon multimodal agents need memory that tracks evolving states, revises stale information, and retrieves evidence for decisions, but existing evaluations mostly test static textual recall. WorldMemArena addresses this gap with lifecycle-level diagnosis and comparisons across memory paradigms.
- Long-horizon agents need memory to track task state, learn from actions, and support decisions through real-world interaction.
- Existing benchmarks often test remembered dialogue rather than how memory guides future actions.
- Final-answer accuracy alone cannot identify whether failures arise in writing, updating, retrieval, or evidence use.
- Many benchmarks reduce images to captions and provide limited interaction pressure on multimodal evidence use.
- WorldMemArena evaluates memory as an Action–World Interaction Loop with observable writing, maintenance, retrieval, and use stages.
- The benchmark spans Lifelong Evolution and Agentic Execution, with gold memory points, updates, distractors, and evidence chains for diagnosis.
- Across long-context, manually designed, and harness-based systems, better storage does not guarantee performance, visual evidence remains difficult, and harness memory is flexible but costly and unreliable.
3. Problem Formulation
The paper models each task as a partially observed, multi-session agent–world process and evaluates memory through four architecture-agnostic lifecycle phases. This formulation separates memory preservation and reuse from final answer accuracy.
- 3.1. Memory as an Action-World Interaction Loop: Each interaction step maps latent world state to observations, selects actions using memory, and produces updated state and feedback.
- 3.1. Memory as an Action-World Interaction Loop: A trajectory records observations, actions, and feedback, then is segmented into sessions while the world state persists across session boundaries.
- 3.2. Memory Lifecycle as a Diagnostic Framework: The architecture-agnostic lifecycle evaluates writing, maintenance, retrieval, and use as observable phases.
- 3.2. Memory Lifecycle as a Diagnostic Framework: Writing retains a selective memory delta from the previous state and current session rather than storing the full trajectory.
- 3.2. Memory Lifecycle as a Diagnostic Framework: Maintenance integrates new information through revision and consolidation because long-horizon interaction is not purely additive.
- 3.2. Memory Lifecycle as a Diagnostic Framework: Retrieval returns decision-relevant evidence for a query, while use evaluates whether that evidence is faithfully translated into an answer or action.
- 3.2. Memory Lifecycle as a Diagnostic Framework: WorldMemArena extends benchmark comparison beyond static dialogue and long-context settings to interactive, multimodal, lifecycle-based evaluation.
- 3.2. Memory Lifecycle as a Diagnostic Framework: The comparison includes benchmarks with differing support for multimodality, interaction, and the four lifecycle dimensions.
4. WorldMemArena: Agent Memory in Action-World Interaction
WorldMemArena is a multi-session benchmark combining evolving-state tasks with realistic agent trajectories, annotated to diagnose memory across its lifecycle. Its scale and evaluation protocol support both multimodal grounding and stage-specific analysis.
- Overview: WorldMemArena contains 461 multi-session tasks across Lifelong Evolution and Agentic Execution.
- Agentic Execution: Agentic Execution tests reuse of observations, actions, and feedback from realistic trajectories in later decisions.
- Lifelong Evolution: Lifelong Evolution tests coherent state tracking for evolving personal information and long-horizon projects.
- Why both Regimes are Needed: The two regimes jointly test persistent state tracking and action-grounded experience reuse.
- Data Construction: The construction pipeline segments sessions, extracts gold memory points and updates, removes redundancy, and creates staged QA checkpoints.
- Domain and Annotations: The benchmark covers GUI and embodied settings, with domain variation and lifecycle annotations for key, update, and interference memory points.
- Dataset Scale and Coverage: It contains 24,258 QA pairs and 15,595 images or screenshots across 461 samples averaging 18.4 sessions and approximately 9.1K tokens.
- Evaluation: Evaluation measures writing coverage and quality, update correctness, retrieval coverage and ranking, and final use across twelve capability axes.
5. Experiments
The experiments compare long-context, manually designed, and harness-based memory agents across memory quality and downstream QA, using lifecycle-oriented evaluation metrics. The results emphasize that storing or retrieving memories is not sufficient unless systems can use them effectively for answering.
- 5. Experiments: Three paradigms are evaluated: long-context agents, manually designed memory systems, and harness-based memory agents.Long-context agents rely on concatenated interaction history; manually designed systems use external memory or RAG modules; harnesses manage memory during interaction.
- 5. Experiments: Table 2 measures memory construction, correctness, updating, interference rejection, retrieval coverage, and QA quality in percentage means.The QA metrics include correctness, hallucination, omission, and retrieval coverage; evaluation uses GPT-5.4-mini.
- 5.1. Main Results: Multimodal systems show limited downstream gains from visual inputs, while text-based systems achieve more stable final answer quality.The comparison suggests that visual evidence is not yet reliably encoded and reused as long-term memory.
- 5.1. Main Results: High memory storage and recall quality does not guarantee strong QA because systems may fail to retrieve and use the right evidence at answer time.The experiments distinguish memory quality from evidence retrieval and final answer correctness.
- 5.1. Main Results: Table 3 reports QA correctness, hallucination, and omission for base models and harness agents.Harness-based evaluation is primarily end-to-end because its internal memory process is difficult to decompose.
6. Analysis
The analysis localizes memory failures across retrieval, updating, domain transfer, and multimodal reasoning. It shows that performance depends on how memory is maintained and used, not simply on how much information is stored or retrieved.
- [RQ1] Where do memory failures occur in the lifecycle?: Stored memories may remain unusable when systems fail to retrieve the evidence needed for the current decision.Figure 4(a) tracks average QA accuracy across sessions, while the accompanying analysis separates storage coverage from evidence use.
- [RQ1] Where do memory failures occur in the lifecycle?: Most systems append new information rather than revising, deleting, or merging obsolete memories as evidence changes.Figure 4(b-c) summarizes update operations and fact salience over dialogue sessions.
- [RQ1] Where do memory failures occur in the lifecycle?: Early omissions and incorrect outputs can compound over long trajectories by reducing later evidence availability and contaminating future updates.The analysis links lifecycle failures to progressively weaker evidence for later decisions.
- [RQ2] Are memory system designs constrained by domain-specific data?: Most systems perform better in Lifelong Evolution than Agentic Execution, where usable memory must be extracted from actions and environment feedback.Performance also varies across task types, with long-horizon embodied tasks such as visual navigation posing greater challenges.
- [RQ3] How does multimodal affect the memory lifecycle?: Systems remain relatively stable on simple visual fact recall but degrade on complex tasks requiring long interaction histories and cross-modal reasoning.The core difficulty is maintaining visual states over time and integrating visual evidence with historical context.
- [RQ3] How does multimodal affect the memory lifecycle?: Increasing retrieval scope does not always improve answer quality because longer contexts can add redundant, outdated, or irrelevant evidence.This problem is especially pronounced in multimodal tasks with substantial visual redundancy.
[RQ5] Can agents turn memory into action?
Agents often store memory successfully without reliably converting it into reusable knowledge for reasoning, learning, or action. Harness-based memory is more adaptable in complex settings, but long-horizon memory remains fragile.
- Systems perform worse on reasoning and test-time learning tasks, indicating a gap between storing information and using it for future decisions.Retrieved memories are dominated by explicit textual facts, while tool feedback, failed actions, and visual details are often omitted.
- Harness-based memory is more flexible than fixed pipelines in complex agentic settings because it can record, retrieve, and revise memory during interaction.Human-designed systems are comparable on simpler long-horizon tasks, but fixed pipelines become limiting as feedback and environmental conditions change.
- Higher storage quality does not necessarily produce better performance, and harness-based agents without explicit memory modules can outperform some manually designed pipelines.
- Effective memory requires consistent state maintenance, usable visual representations, and evaluation of learning from experience rather than retrospective question answering.The paper highlights stale-memory accumulation, loss of spatial and procedural visual details, and the need to assess improved behavior across sessions.
- WorldMemArena enables stage-level diagnosis of memory writing, maintenance, retrieval, and use across long-context, manually designed, and harness-based agents.Its annotations include gold memory points, updates, and distractors.
A. Experimental Setting
The evaluation standardizes model execution and decomposes memory quality into coverage, correctness, update handling, and interference rejection. These metrics distinguish selective, maintained memory from indiscriminate storage.
- All baselines share the same backbone and decoding configuration, while retrieval is capped at top-K = 10 items per query.Answer-stage and judge models use temperature 0.0; the effective answer context is 128,000 tokens.
- Memory Recall measures importance-weighted coverage of gold memory points supported by the system’s add/update delta.Sessions without gold memory points are excluded, and semantic judging avoids penalizing harmless paraphrases or summarization.
- Correctness, hallucination, and irrelevance labels expose the storage quality that recall alone cannot capture.The labels distinguish faithful items from concrete factual contradictions and fundamentally wrong or irrelevant content.
- Update Handling classifies whether changed facts are updated, retained alongside old facts, or left outdated.Keeping both old and new facts receives half-credit because downstream QA may still surface the wrong answer.
- Interference Rejection measures whether casual remarks, jokes, and corrections are rejected rather than committed to memory.High Recall paired with low IntRej identifies an indiscriminate writer that hoards everything.
B.3. QA metrics
The QA metrics evaluate whether stored memory supports answers when evidence is no longer local. They separate correct answers, hallucinations, omissions, and partial surface-form overlap.
- QA-C, QA-H, and QA-O classify answers as Correct, Hallucination, or Omission based on comparison with the gold answer and evidence.The labels distinguish confident-but-wrong answers from refusals or “I don’t know” responses.
- Answer F1 adds a fine-grained signal for partial token overlap between generated and gold short factual answers.Normalization lowercases, removes selected stopwords, strips punctuation, and applies Porter stemming.
- BLEU-1 provides a precision-leaning unigram metric alongside F1, making their gap informative about answer padding.BLEU-1 uses the same normalization without Porter stemming.
B.4. Retrieval metrics
The retrieval metrics distinguish whether gold evidence appears anywhere, reaches the top-K, and is ranked early, using semantic matching rather than exact string identity.
- The soft match predicate accepts matching identifiers, source sessions, or normalized content with substring or at least 0.75 token overlap.These rules absorb superficial identifier mismatches across heterogeneous baselines without awarding credit solely for verbatim equality.
- Retrieval Coverage uses an LLM judge to count how many gold evidence points are supported anywhere in the full top-K list.It is rank-agnostic because an answer can succeed when the evidence appears anywhere and the answerer reads the list.
- Recall@K deterministically measures whether gold evidence appears within the top-K results using a soft match predicate.Recall@1 is the strictest setting because it rewards placing the correct evidence first.
- NDCG@K discounts later-ranked evidence to distinguish ordering quality within the top-K list.Retrieved items are assigned binary relevance greedily so one item cannot receive credit for two gold points, and DCG applies a logarithmic rank discount.
- NDCG@K is the ratio of discounted gain to ideal discounted gain, with QAs lacking gold evidence contributing zero.The ideal ranking places all reachable golds as early as possible, where K* = min(|𝒱q|, K).
B.5. Per-question-type accuracy
Per-question-type accuracy decomposes aggregate QA performance across eleven mutually exclusive semantic types and four skill axes, while preventing frequent types from dominating the average.
- Each gold QA receives one of eleven mutually exclusive semantic types grouped along four skill axes.This decomposition exposes strengths and weaknesses that aggregate accuracy can hide.
- For each type, QA-C is computed only over questions of that type and instances containing at least one such question.This produces a type-specific accuracy value rather than mixing unrelated question categories.
- The Avg. column is the unweighted mean of the eleven per-type values per instance, preventing types with more questions from dominating the headline number.
- The trajectories combine four upstream agent benchmarks with an in-house long-horizon dialogue collection released alongside WorldMemArena.
C.2. Quality validation
WorldMemArena validates generated multi-session data with automated checks while covering evolving personal and task states, realistic agent trajectories, and multimodal traces.
- Quality validation: Each generated session passes validators for memory-point coverage, image-caption coverage, interference detectability, and update-chain consistency.Samples failing any validator are regenerated up to three times.
- Lifelong Evolution: Lifelong Evolution models hidden world states that change across ordered sessions, covering personal memory and long-horizon project progress.
- Lifelong Evolution: The professional-vertical domain spans six fields and includes multimodal checkpoint references tied to documents, interfaces, or scene captures.It contains 18 samples in total.
- Lifelong Evolution: The holistic life-course domain separates career and life-goal main arcs from daily-life, family, and health side arcs across 20 trajectories.
- Agentic Execution: Agentic Execution records observations, reasoning, actions, environment feedback, and optional screenshots from navigation, embodied manipulation, and desktop GUI settings.Its trajectory is a time-ordered log of acting on a world and reading consequences back.
- Action-World Interaction Loop: All three domains require symbolic state evolution integrated with visual grounding rather than dialogue-only recall.
E. More Experiment
Experiments reveal large differences in retrieval quality, latency, and backbone competence, while long-horizon and agentic settings expose compounding failures, text bias, and limited reuse of experience.
- Efficiency: Total memory-method cost spans almost two orders of magnitude, from M2A at 10.0 s to SimpleMem at 786.3 s.Read-heavy methods spend more time rescanning dialogue at query time, whereas write-heavy methods allocate more cost to writing and storage.
- Retrieval: Dense RAG with Qwen3-VL-Embedding-8B achieves the strongest ranking quality at larger K, while UniversalRAG trails on recall-oriented metrics and NDCG.
- Retrieval: Agent-memory architectures show a large performance spread: SimpleMem and MIRIX are weaker, while Omni-SimpleMem and M2A recover toward mid-tier recall and NDCG.AUGUSTUS tracks A-Mem on recall without superior NDCG@K, reinforcing that high recall alone is insufficient.
- Backbone competence: Deepseek V4 has the highest average and leads Fact Recall, Memory Boundary, Memory Conflict, and most multimodal axes, including Visual Fact Recall, Visual Search, and Visual Update.
- Failure analysis: Long-horizon lifecycle failures compound: omissions reduce later evidence, retrieval failures weaken grounding, and incorrect answers can pollute subsequent updates.The degradation is particularly severe in reconstructed agentic worlds involving actions, tool outcomes, visual states, and causal consequences.
- Failure analysis: Retrieval faces a recall–precision trade-off and remains text-centric, so adding more items may introduce outdated or conflicting memories while visual evidence depends on textualization.
- Failure analysis: Systems often recall explicit facts but struggle to convert prior experiences into action-guiding knowledge for reasoning, test-time learning, and future decisions.