PaperlayerView exact revision on arXiv

Source-linked AI summary

EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks

Lizhou Liang, Xinyu Zhong, Miao Pan, Xiaohe Zhou, Xuanyu Liu, Qinfeng Li, Peng Li, Jintao Chen, Xuhong Zhang, Wenqi Zhang

arXiv:2609.28236v1cs.CV

TL;DR

Long-horizon embodied interaction requires agents to retain changing environmental information, but existing benchmarks do not directly test memory use during embodied interaction. EMem-Bench evaluates that capability through 2,554 episodes, and EMem improves memory-guided behavior while current models still struggle to maintain and reuse world state.

  • Problem

    Existing benchmarks do not directly assess whether agents can maintain and use memory during long-horizon embodied interaction.

    Source: §1 · p. 3

  • Method

    EMem-Bench evaluates memory across 2,554 episodes, while EMem organizes multimodal experience into spatial, event, and scene memories for agents to write, retrieve, and use.

    Source: §1 · p. 3

  • Results

    Current MLLMs still struggle to maintain and reuse world state over long-term interaction, while EMem improves memory-guided behavior.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The benchmark evaluates four embodied-memory capabilities through executable task families, and EMem provides a system for organizing and using interaction memories.

    Source: §6 · p. 10

    The benchmark and cross-benchmark evaluations are conducted primarily in simulation, so results do not establish long-term memory performance on physical robots.

    Source: E Limitations · p. 30

Abstract

from arXiv · show

Long-horizon embodied interaction requires agents to retain and continually update information about the environment as they observe, act, and encounter change. Yet current agents struggle to maintain such memory reliably. Our analysis traces this limitation to four key deficiencies: weak fine-grained visual memory, unreliable dynamic world-state tracking, failing to record world state revealed by interaction outcomes, and limited generalization from prior experience. However, existing benchmarks do not directly assess these memory capabilities during long-horizon embodied interaction. To address this gap, we introduce EmbodiedMemory-Bench (EMem-Bench), comprising 2,554 interactive episodes across four task families. EMem-Bench requires agents to build and update memory from interaction history, then use it to complete a later task by acting in the environment. We further present Embodied-Memorizer (EMem), an external memory system that organizes embodied experience into spatial, event, and scene memories. We also train EMem-8B, an 8B policy that manages and uses these memories. We evaluate a diverse range of open-source and proprietary MLLMs and representative multimodal memory systems. Results show that current models remain weak and uneven across the four challenges. Under matched backbones, EMem achieves the best overall performance among the evaluated memory systems and improves both open-source and proprietary models, while EMem-8B further improves over its backbone. Project page: https://zju-omniai.github.io/EmbodiedMemoryBench/

1 Introduction

Long-horizon embodied interaction requires memory that links observations, actions, and feedback while tracking a changing environment. EMem-Bench tests whether agents can build memory from interaction histories and use it to act; evaluations expose persistent memory failures and report gains from EMem.

1 Introduction

  • Embodied memory must preserve causal links between observations, actions, and feedback because actions change the environment and their outcomes reveal otherwise hidden states.Interaction histories also unfold over long horizons in environments subject to disturbance and change.Source: §1 · p. 1, Figure 1 · p. 2
  • The four identified memory failures are forgetting fine-grained visual cues, missing environmental changes, not recording interaction-revealed state, and failing to generalize prior experience.In reviewed trajectories, these categories represented 35%, 18%, 31%, and 7% of failures, respectively, about 91% combined.Source: §1 · pp. 2–3
  • The authors attribute 91% of 100 reviewed failed trajectories to four recurring memory error categories.The failures were sampled from Gemini-3-Flash and Qwen3-VL-32B trajectories on EB-ALF and EB-Hab.Source: §1 · p. 2, Figure 2 · p. 2
  • Unlike benchmarks that test single-turn question answering or exploration without supplied context, EMem-Bench tests whether agents consolidate offline trajectories into memory and use it for embodied action.The benchmark varies interaction history and target task across episodes.Source: §1 · p. 3
  • EMem improves average SR by 16.4 points on Mistral-Small-3.1-24B and 23.6 points on GPT-5.4-mini; EMem-8B improves over Qwen3-VL-8B by 20.6 points.EMem organizes experience into scene, spatial, and event memories; EMem-8B is trained to write, retrieve, and use them.Source: §1 · p. 3
  • EMem-Bench contains 2,554 episodes that require agents to extract task-relevant information from multimodal interaction histories and then act in the environment.Its four families test fine-grained visual memory, dynamic tracking, state revealed by interaction outcomes, and experience generalization.Source: §1 · p. 3

2 Related Work

Prior embodied benchmarks introduce memory within individual tasks, while general memory benchmarks evaluate histories without requiring remembered information to guide later embodied interaction. EMem-Bench evaluates embodied memory jointly across interactive episodes.

2 Related Work

  • Embodied benchmarks such as FindingDory, LMEE-Bench, SpaMEM, and WorldLines introduce memory within individual tasks; EMem-Bench evaluates it jointly across interactive episodes.The cited benchmarks address visual-history navigation, spatial-state revision, and state question answering or planning over multi-day traces.Source: §2 · p. 3
  • LoCoMo, LongMemEval, MemoryAgentBench, Evo-Memory, and WorldMemArena assess memory over histories or experience, but do not require it to guide subsequent embodied interaction.Their stated foci include retrieval, test-time learning, experience reuse, and memory writing, maintenance, retrieval, and use.Source: §2 · pp. 3–4
  • The benchmark comparison tracks explicit evaluation of visual memory, dynamic tracking, interaction-derived state, and experience generalization.The comparison table marks explicit evaluation with Y.Source: Table 1 · p. 4

3 EMem-Bench

EMem-Bench tests whether an agent can recover information from multimodal interaction history and use it during later embodied action. It organizes this challenge into four memory-focused task families and validates episodes built from task-specific cues and distractor trajectories.

3.1 Task Definition

  • Each episode pairs a multimodal interaction history with a target task and feasible actions; success requires reaching an environment state that satisfies the target.The evaluation reports success rate by family and an equally weighted mean across the four families.Source: §3.1 · p. 4, §3.4 · p. 6

3.2 Task Families

  • The four families test fine-grained visual recall, updating changed object states, retaining interaction-revealed states, and transferring regularities to new objects.The history supplies the relevant information, while the later task determines how it must be used.Source: §3.2 · p. 5

3.3 Interaction-History Construction

  • Episodes are built from multi-room scenes, task-specific memory cues, unrelated distractor trajectories, and automated checks followed by manual review.Review removed 143 candidates; all 2,554 retained episodes passed automated execution checks and manual screening.Source: §3.3 · pp. 5–6, Figure 4 · p. 5

3.4 Benchmark Statistics and Evaluation

  • The benchmark contains 2,554 episodes across four families, and average success rate weights the families equally; ERR tracks recurrence of memory-dependent errors.The families contain 1,036 Passive Observation, 1,052 Dynamic Tracking, 263 Interaction Failure, and 203 Experience Generalization episodes.Source: §3.4 · pp. 6–7

4 Our Method: Embodied-Memorizer

EMem is an external system that organizes and updates world state during embodied interaction. Its three memory types support visual context, current object states, and interaction experience, which the model can write, retrieve, and use in an ongoing action loop.

4 Our Method: Embodied-Memorizer

  • EMem continually organizes and updates world state through scene, spatial, and event memories.Scene memory stores visual context, spatial memory maintains object states, and event memory records interactions and reusable experience.Source: §4 · p. 7, Figure 6 · p. 7

4.1 Three Complementary Memories

  • Spatial memory tracks current object locations and states; event memory preserves actions, feedback, and corrections; scene memory binds entities to their rooms.Event memory can summarize correction patterns for retrieval with related new objects, while scene memory distinguishes same-named objects across rooms.Source: §4.1 · p. 7

4.2 Operating the Memory Loop

  • During interaction, EMem stores observations, changes, outcomes, and corrections, then retrieves relevant memories to guide actions and updates memory from new feedback.Supervised fine-tuning teaches EMem-8B what to write, which memories to query, and how to act on retrieved information.Source: §4.2 · p. 7

5 Experiments

Experiments compare models and memory systems, then test the effects of added visual history, memory components, transfer, and failure types. EMem leads the evaluated memory systems, while current models show uneven performance and recurring state-use and reasoning errors.

5.1 Experimental Setup

  • The evaluation compares open-source and proprietary MLLMs and fixes GPT-5.4-mini as the backbone for memory-system comparisons.The baselines include general-purpose and embodied models; the memory comparison includes MIRIX, MemVerse, TeleMem, MMA, and EMem.Source: §5.1 · p. 8

5.2 Main Results

  • Spatial and embodied specialization does not ensure strong memory use: several embodied models score below Qwen3-VL-8B-Instruct’s 25.4% average SR.RynnBrain-8B scores 3.6%, RoboBrain2.5-8B-NV 20.0%, Cambrian-S-7B 0.8%, MiMo-Embodied-7B 38.8%, and Robotics-ER 1.5 52.4%.Source: §5.2 · p. 9
  • Current models remain weak and uneven; EMem reaches 58.9% average SR, 21.1 points above TeleMem, and EMem-8B adds 9.8 points over frozen-backbone EMem.Across model baselines, Gemini-3-Flash reaches 64.2% average SR; the table reports full benchmark results.Source: Table 2 · p. 8, §5.2 · p. 9

5.3 Does More Visual History Help?

  • Adding historical images lowers average SR from 36.6% to 26.2%, while increasing input tokens by 4.9× and model-step latency by 9.5×.Performance declines in every history-length quartile.Source: §5.3 · p. 9, Table 3 · p. 9

5.4 Ablation Study

  • Removing any memory reduces average SR; spatial memory matters most for Dynamic Tracking, event memory for Interaction Failure and Experience Generalization, and scene memory for Passive Observation.Source: §5.4 · pp. 9–10, Table 4 · p. 9

5.5 Cross-Benchmark Generalization

  • EMem-8B improves EB-ALF average SR by 5.0 points and BLINK macro accuracy by 1.3 points, gains 0.5 points on MMMU-Pro, and decreases 0.4 points on HallusionBench.These cross-benchmark results indicate strongest transfer to interactive embodied execution while general multimodal performance remains comparable.Source: §5.5 · p. 10, Table 5 · p. 10

5.6 Failure Analysis

  • Memory-use errors dominate three task families, contextual reasoning errors dominate Interaction Failure, and action errors remain uncommon across all four.In Dynamic Tracking, the main difficulty is replacing outdated state with newer evidence; in Interaction Failure, avoiding a known failed action does not necessarily produce a revised decision.Source: §5.6 · p. 10, Figure 7 · p. 10

6 Conclusion

EMem-Bench evaluates four embodied-memory challenges, while EMem organizes experience into spatial, event, and scene memories. The paper situates these contributions among prior memory benchmarks and describes how EMem represents and updates experience.

6 Conclusion

  • EMem-Bench evaluates four executable memory task families, and EMem organizes multimodal experience into spatial, event, and scene memories; experiments find that current MLLMs struggle while EMem improves memory-guided behavior.Source: §6 · p. 10

A Extended Related Work

  • EMem-Bench groups complementary memory demands in one interactive evaluation, unlike prior benchmarks focused on navigation, manipulation, question answering, state reconstruction, or long-range evidence integration.Source: A Extended Related Work · p. 15
  • Prior embodied memory architectures vary in what they store and how memory enters control; EMem focuses on scene, spatial, and event records for its four benchmark challenges.Source: A Extended Related Work · p. 15

B Embodied-Memorizer Details

  • EMem is modular and episode-local: its policy selects writes, retrievals, and actions, while the backend performs deterministic record updates and searches.Source: B Embodied-Memorizer Details · p. 16

B.1 Memory Representation

  • Spatial Memory stores scene-scoped entities and relations, replacing conflicting location records while preserving compatible properties; Event Memory retains both successful and failed outcomes.Source: §B.1 · p. 16
  • Event Memory can aggregate recurring corrections into reusable rules after at least two related corrections, while Scene Memory stores selected visual snapshots with captions, entities, and RGB references.Source: §B.1 · p. 16

B.2 Writing Memory from Interaction History

  • During chronological ingestion, the writer receives the current observation and preceding action but not the future target instruction; the same write path updates memory after task-time actions.Source: §B.2 · p. 17

B.3 Task-Conditioned Retrieval and Execution

EMem retrieves task-relevant records from the appropriate memory stores, then plans and executes actions using those records alongside the current observation and legal actions. The loop refreshes memory and restarts after failures or material state changes.

B.3 Task-Conditioned Retrieval and Execution

  • EMem selects spatial, event, or scene queries based on the instruction, then searches each store using structured constraints, lexical matches, and semantic similarity as a fallback.Source: §B.3 · p. 17
  • Retrieved records join the instruction, current RGB observation, and legal actions in the action context; invalid actions or sufficiently large memory-backed state changes interrupt the sequence and start a new turn.Source: §B.3 · pp. 17–18
  • The procedure uses fixed runtime settings across benchmark families, including an eight-action primitive budget, deterministic decoding, and a retrieval budget of at most 12 records per call.Source: §B.3 · pp. 17–18

B.4 Prompt Templates

  • Prompt templates populate runtime fields with observations, legal actions, tool schemas, or executed-call results; the controller validates JSON and restricts available tools by phase.Source: §B.4 · p. 18

B.5 Training Procedure

The appendix details EMem-8B’s training procedure and the benchmark’s evaluation, difficulty, coverage, and release-audit protocols. It also reports trajectory statistics and explains how error recurrence is operationalized.

B.5 Training Procedure

  • Training uses teacher-generated ProcTHOR trajectories and supervises context writes, memory queries, and action generation; Qwen3-VL-8B is fine-tuned with LoRA for one epoch and 1,816 optimization steps.The corpus contains 8,852 turns: 3,930 context-write turns and 2,461 each for memory and action phases.Source: §B.5 · pp. 18–19, Table 7 · p. 19

C Benchmark Details

  • EMem-Bench has 2,554 episodes across 1,118 globally unique scenes, 125 visible object types, 83 target object types, and 33 receptacle types; family counts may overlap.The benchmark assigns episodes by the simulator of their target-task scene, with 352 AI2-THOR episodes and 2,202 ProcTHOR episodes.Source: C Benchmark Details · p. 21, Table 9 · p. 23, §C.4 · p. 24

C.1 Full-Context Evaluation Protocol

  • Full-context evaluation exposes ordered interaction records and current observations, retains the complete textual history for every episode, and refreshes legal actions and observations after each valid transition.The 80-step history limit applies separately to each session; the longest canonical-release session has 78 steps, and at most eight target-task steps are allowed.Source: §C.1 · p. 21

C.2 Operational Definition of Error Recurrence Rate

  • Error Recurrence Rate flags family-specific avoidable errors—wrong remembered locations, superseded states, repeated failed targets, or violated owner habits—and clips each episode’s score to an indicator.ERR complements terminal-state success rate: an agent can trigger ERR, recover, and still complete the task.Source: §C.2 · p. 22

C.3 Model Output Trajectory Length

  • 3.07 actions for Gemini-3-Flash to 6.77 for InternVL3-8B; Interaction Failure averages 6.66 actions partly because its execution structure requires six actions, so length is descriptive, not a performance ranking.The family means are 4.56 actions for Passive Observation, 4.92 for Dynamic Tracking, 6.66 for Interaction Failure, and 5.29 for Experience Generalization.Source: §C.3 · p. 23, Table 8 · p. 23

C.4 Composition and Interaction-History Difficulty

  • Experience Generalization has a median 111-step history and four to five scene transitions; other families have median histories of 78–83 steps, two transitions, and cue-to-target gaps of 52–56 steps.Passive Observation averages 11.0 visible objects at the decisive cue; the passage connects these statistics to recovering evidence across long, changing, visually cluttered histories.Source: Table 10 · p. 23, §C.4 · p. 24

C.5 Object Coverage and Distribution Concentration

  • Target-object concentration is low in Passive Observation, Dynamic Tracking, and Experience Generalization: the most frequent target accounts for 11.0%, 9.6%, and 5.9% of family episodes, respectively.Interaction Failure is narrower, with 18 target types and five receptacle types; receptacle distributions are more concentrated because tasks admit only receptacles supporting the executable terminal action.Source: §C.5 · p. 24, Table 11 · p. 24

C.6 Release Audit and Executability

  • Release checks pair executability and terminal-state validation with six-dimensional expert review; the process rejected 143 candidates before retaining 2,554 episodes.Review criteria include evidence sufficiency, instruction clarity, leakage, trajectory plausibility, cue–probe alignment, and terminal-judgment correctness.Source: §C.6 · pp. 24–25, Human Review Protocol and Audit… · pp. 25–26

C.7 Task-Family Sensitivity

Model success-rate rankings are positively but nonuniformly correlated across task families, consistent with models having uneven memory-task profiles.

C.7 Task-Family Sensitivity

  • SR-ranking correlations range from 0.52 to 0.94; Dynamic Tracking correlates only 0.52 with both Interaction Failure and Experience Generalization.Passive Observation and Experience Generalization are the most aligned task-family rankings.Source: §C.7 · p. 26

C.8 Case Studies

The case studies contrast successful and failed uses of remembered visual cues, updated world states, interaction feedback, and recurring household conventions.

C.8 Case Studies

  • In Passive Observation, GPT-5.4-mini retrieves a remembered ButterKnife location, but in a parallel case repeatedly searches the bed instead of the CellPhone’s recoverable dresser location.The comparison isolates the later policy’s use of historical evidence.Source: §C.8 · p. 26
  • In Dynamic Tracking, the model retrieves a RemoteControl from its updated armchair location, while another case repeatedly returns to the superseded coffee-table location of a CreditCard.The contrast distinguishes updating world state from retaining an earlier observation.Source: §C.8 · p. 26
  • After feedback identifies a locked cabinet, the successful case avoids that instance, while the failed case retries it and cannot place the SoapBottle.Successful transfer associates earlier feedback with the identity of the later interaction target.Source: §C.8 · p. 26
  • The model transfers a desk-placement convention from corrections involving a CellPhone and Laptop to a new CD, but ignores a side-table convention and repeatedly places a CreditCard on the dining table.The contrasting cases test induction of a reusable rule from multiple corrected experiences.Source: §C.8 · pp. 26–27

D Additional Experimental Results

Two controlled experiments test how episode evidence and historical images relate to task success, using fixed episode identifiers and paired evaluation procedures.

D Additional Experimental Results

  • The experiments test whether success changes with episode evidence and whether adding historical images to the full-context setting affects task success.Both use the same fixed episode identifiers released with the evaluation ledger.Source: D Additional Experimental Results · p. 27
  • Average SR is computed as the arithmetic mean of the four task-family success rates.This matches the main-paper aggregation method.Source: D Additional Experimental Results · p. 27
  • Paired differences reuse the same resampled episode identifiers, with episode-paired bootstrap 95% confidence intervals computed from 10,000 resamples.Source: D Additional Experimental Results · p. 27

D.1 Causal Dependence on Interaction History

History interventions show that EMem-Bench depends on preceding evidence, while adding maximum historical RGB input lowers performance rather than helping.

D.1 Causal Dependence on Interaction History

  • The fixed 800-episode evaluation samples 200 episodes per task family and compares Correct history, Current only, Cue removed, and Minimal oracle conditions.Cue removed replaces decisive evidence with a length- and type-matched neutral record; Minimal oracle supplies only the shortest proposition sufficient to identify the terminal relation.Source: §D.1 · p. 27
  • 11.1-point and 21.4-point paired macro-SR differences separate correct-history from cue-removed conditions for Qwen3-VL-8B and GPT-5.4-mini, respectively.Positive cue effects across all distance quartiles indicate that the requirement extends to temporally remote evidence.Source: §D.1 · p. 27
  • 58.2 and 69.1 macro SR under Minimal oracle exceed Correct history by 34.4 and 32.5 points for Qwen3-VL-8B and GPT-5.4-mini, respectively.The authors interpret this gap as separating memory retrieval from downstream execution.Source: §D.1 · p. 27

D.2 Historical Visual Inputs in Full Context

  • Adding maximum historical RGB input changed GPT-5.4-mini macro SR by -10.4 points on the 800-episode subset.Mean input rose from 41,770 to 206,489 tokens per episode, and mean model-step latency rose from 4.1 to 39.1 seconds.Source: Table 13 · p. 30, §D.2 · p. 30
  • Maximum-history-image preserves full sanitized text history and the current RGB observation, then adds as many historical RGB observations as context permits.When visual history exceeds the limit, frames are sampled uniformly across sessions and timelines, then restored to temporal order.Source: §D.2 · p. 30

E Limitations

EMem-Bench and cross-benchmark evaluations are conducted primarily in simulation, so the reported results do not establish physical-robot memory performance.

E Limitations

  • Primarily simulation-based evaluations do not establish long-term memory performance on physical robots, where sensor noise, actuation error, open-world changes, and ambiguous human feedback arise.The authors identify standardized real-robot test suites as future work.Source: E Limitations · p. 30
Loading 2609.28236v1…