PaperlayerView exact revision on arXiv

Source-linked AI summary

What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents

Junxiang Wang, Yuchen Liu, Yi Wu, Guofei Chen, Nir Pechuk, Tao Chen, Zackory Erickson, Yuzhe Qin, Tianyu Li

arXiv:2610.12561v1cs.RO

TL;DR

Embodied agents may need to retain visual and spatial details for later tasks, but it is unclear which memory forms best support recall and action at what inference cost. MeMoREval compares modalities through a controlled experience–request benchmark. Frames with scene graphs achieve the highest aggregate success across tested models, while frames outperform generated descriptions for the primary model at lower write-time cost.

  • Problem

    For embodied agents, the evidence worth retaining can include visual and spatial details, but the memory modality affects what remains available for later tasks and its inference costs.

    Source: §1 · p. 1

  • Method

    MeMoREval compares memory modalities through a controlled protocol that separates experience from later information and action requests and measures task success and inference costs.

    Source: §1 · p. 2, §6 · p. 11

  • Results

    Frames with scene graphs achieve the highest aggregate task success for every tested model; for the primary model, frames outperform generated descriptions at lower write-time inference cost.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    Memory-modality benefits vary across models and task categories, and the benchmark supports investigation of accumulated experience, selective frame retention, and writer inputs.

    Source: §1 · p. 2, §6 · p. 11

    Experience is fixed per task across trials, so performance differences reflect memory configuration rather than variation in what was observed.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests. For embodied agents that control robots, visual and spatial observations introduce a broader choice of memory modalities, affecting both the information available for subsequent tasks and the model inference costs of constructing and using memory. We introduce MeMoREval, a benchmark for studying this problem through a controlled experience-request protocol. Agents first experience a sequence of events without knowing the subsequent tasks, and subsequently they use memory to respond to information and action requests without receiving new observations. The benchmark comprises 53 tasks across four simulated indoor environments and measures task success alongside the inference costs of memory construction and use. Our reference evaluations reveal that storing frames outperforms storing generated descriptions at lower write-time inference cost for the primary model, while storing frames with scene graphs achieves the highest aggregate task success for every tested model. Extensions to accumulated experience, selective frame retention, and visual input to memory writers demonstrate how the benchmark accommodates further memory-design investigations. Two physical-robot demonstrations establish the feasibility of applying its protocol beyond simulation.

1 Introduction

Embodied agents must preserve visual and spatial evidence for later tasks, but the useful memory form and its inference costs are uncertain. MeMoREval compares memory modalities under a controlled protocol and finds that frames with scene graphs achieve the highest aggregate success across tested models.

Motivation

  • Text descriptions can omit details needed by an unknown future task, while frames retain visual evidence and spatial records expose object locations and relations.Forgotten information may require physical searches to recover and can become unobservable after objects move or are removed.Source: §1 · pp. 1–2

Benchmark and protocol

  • MeMoREval compares textual records, generated image descriptions, camera frames, scene graphs, and combinations using common observations and a fixed agent implementation.It measures task success and memory construction and use costs.Source: §1 · p. 2

Reference evaluations

  • Frames with scene graphs achieve the highest aggregate task success for every tested model.For the primary model, frames also outperform generated descriptions in success at lower write-time inference cost; benefits vary by model and task category.Source: §1 · p. 2

Benchmark and protocol

  • The protocol separates an experience sequence from later information and action requests, which agents answer using memory without new observations.Experience phases can also be stacked to test longer histories.Source: §1 · p. 2

Contributions

  • The benchmark contains 53 tasks across four simulated environments, and two physical-robot demonstrations instantiate its protocol beyond simulation.Extensions examine accumulated experience, selective frame retention, and visual input to memory writers.Source: §1 · p. 2

2 Related Work

Prior memory research spans language-agent systems and embodied benchmarks, but comparisons of memory modality can also vary other agent components. This work holds the agent and observations fixed to study modality across backends.

Memory for LLM agents

  • Language-agent memory systems organize experience as archives, extracted facts, linked notes, reusable procedures, or files, while several benchmarks test long-conversation recall.MemLens and MemEye evaluate preservation of visual evidence across extended multimodal interactions.Source: §2 · p. 3

Memory for embodied agents

  • Embodied memory work includes scene graphs, combined image-and-graph storage, changing-state records, retrieval methods, and selective observation retention.Retrieval and indexing are complementary to deciding what to store.Source: §2 · p. 3

Embodied benchmarks with memory demands

  • Embodied evaluations often vary complete systems, and OpenEQA uses different model types for caption- and scene-graph-based versus frame-based agents.MeMoREval keeps the agent, recorded observations, and writer input common within each backend; requests begin without new observations, and costs are measured separately.Source: §2 · pp. 3–4

3 Preliminaries: LLM-Based Embodied Agents and Memory

The paper models embodied interaction as an LLM selecting robot skills from context and observations, with a memory module retaining experience for later use.

LLM-based embodied agents

  • The LLM selects physical skills such as navigation or grasping, while separate robot controllers execute those skills.Observations can include images, detected objects and locations, user messages, and tool feedback.Source: §3 · p. 4

Memory module

  • The memory module holds persistent state and records observations through a writer or by directly archiving artifacts such as camera frames.The agent reads stored information through supplied context and retrieval tools to answer questions or plan actions.Source: §3 · p. 4

Memory properties

  • Memory modality is the form of retained evidence, whereas backend describes how that evidence is organized and retrieved.A backend may organize textual entries that reference frames, generated descriptions, or scene-graph snapshots.Source: §3 · p. 4

4 MeMoREval: Evaluation Protocol and Benchmark

MeMoREval separates experience from later requests that rely on stored evidence, then instantiates the protocol in simulated indoor tasks with visual, spatial, and textual observations.

4 MeMoREval: Evaluation Protocol and Benchmark

  • MeMoREval evaluates embodied-agent memory modalities through task success and inference costs for memory construction and use.Its protocol and benchmark are presented through task composition, metrics, environments, and tasks.Source: §4 · p. 4

4.1 Evaluation Protocol

  • Tasks separate scripted experience, when observations and action outcomes are recorded before requests are known, from requests answered without new observations.Experience is fixed per task across trials, so performance differences reflect memory configuration rather than different observations.Source: §4.1 · pp. 4–5
  • Information requests test recovery of facts, while action requests require complete plans executed open-loop without observations or revision.The request context is fresh and contains only the request and the robot’s current room.Source: §4.1 · p. 5
  • Stacked tasks concatenate multiple experience phases before their requests, testing whether information remains available as history accumulates.Component tasks retain their original grading criteria, and plans start from the same restored physical state.Source: §4.1 · p. 5
  • Task success requires correct responses to all requests, while inference costs are measured separately during memory construction and request-time use.Write-time cost scales with actions and read-time cost with requests; read-time cost per successful request also includes failed attempts.Source: §4.1 · p. 5

4.2 Benchmark Environments and Tasks

  • The benchmark uses four indoor simulated environments and a mobile manipulator, with 53 tasks spanning seven categories of remembered information.Categories include object location, spatial configuration, appearance binding, set aggregation, user preference, occluded state, and skill outcome.Source: §4.2 · pp. 5–6
  • Textual observations omit color and texture, frames provide visual evidence, and scene graphs store object positions plus support and containment relations.The spatial snapshots do not contain appearance attributes or explicitly encode ordinal or viewpoint-dependent relations.Source: §4.2 · p. 6
  • The benchmark lets memory retain textual records, generated image descriptions, camera frames, spatial snapshots, or combinations of these forms.Transforming observations, including generating descriptions, is part of memory construction.Source: §4.2 · p. 6

5 Experiments

Across the tested models, frames with scene graphs achieve the highest aggregate task success; for Gemini 3.7 Flash, frames outperform generated descriptions at lower write-time cost. The experiments also examine success-cost trade-offs, deployment-oriented variants, and physical-robot demonstrations.

5.1 Experimental Setup

  • The protocol compares six memory conditions across five backends, with Gemini 3.7 Flash evaluated on 53 tasks over three seeds.It separately reports write-time and read-time inference costs; the additional models are evaluated on the single tasks with one seed.Source: §5.1 · p. 7

5.2.1 RQ1: How does memory modality affect task success?

  • Frames with scene graphs achieve the highest aggregate task success for every tested model.Frame-only storage beats desc@1024 for Gemini 3.7 Flash and GPT 5.6 Sol, but not GLM 5.3 Flash.Source: Figure 5 · p. 8, §5.2.1 · p. 9

5.2 Results

  • Without memory, the agent solves 3 of 53 tasks; with full-information memory, it solves all 53.The controls bound performance and show that each task is solvable when its relevant facts are explicitly available as text.Source: §5.2 · p. 8

5.2.1 RQ1: How does memory modality affect task success?

  • Description-based success rises from 56% to 60% as the budget grows from 64 to 1024 tokens, versus 39% without descriptions and 70% with frames.Frames improve mean success in six of seven categories; adding scene graphs raises overall success to 75%, with category-specific gains.Source: §5.2.1 · pp. 8–9

5.2.2 RQ2: How do memory modalities trade off success against cost?

  • Frames require approximately 15% fewer weighted tokens per successful information request and 7% fewer per successful action request than the longest descriptions.Frames and longest descriptions have approximately the same read-time cost per attempted request, while frames achieve higher success; scene graphs further improve success but increase cost per successful request.Source: §5.2.2 · p. 10

5.2.3 RQ3: What do deployment-oriented variants show about memory design?

  • Deployment variants show that memory performance depends on history length, frame retention, and whether writers receive images.Transcript performance degrades when stacking histories exceed its shared context budget and truncate earlier observations. Selective writers retain 8–55% of frames with lower mean success than automatic archival; visual input does not consistently improve success, though it changes write-time cost.Source: §5.2.3 · pp. 10–11

5.3 Real-World Instantiation

  • In two physical-robot demonstrations, the robot used wiki memory with frames and scene graphs to complete both deliveries without human assistance.The instantiation used camera observations and a manually annotated spatial map.Source: §5.3 · p. 11

6 Conclusion

MeMoREval evaluates embodied-agent memory through a controlled experience–request protocol, measuring task success and inference costs. Reference results compare memory modalities and extend the protocol to deployment-oriented variants and physical-robot demonstrations.

6 Conclusion

  • For the primary model, camera frames outperform generated descriptions at lower write-time cost; adding scene graphs improves success further but raises read-time cost.The reported extensions also find that writer image access does not consistently improve success, though it can reduce write-time cost for some backends.Source: §6 · p. 11
  • Accumulated experience affects backends differently; selecting frames from textual observations reduces archival volume but lowers task success, while writer image access does not consistently improve success.Giving images to the writer can reduce write-time cost for some backends.Source: §6 · p. 11
  • Two physical-robot demonstrations apply the protocol using camera observations and an annotated spatial map, supporting evaluations beyond simulation.The benchmark provides common experiences, interfaces, and grading as an experimental basis for memory-system development.Source: §6 · p. 11

A Additional Results

The additional-results tables catalog evaluation values across task categories, base models, memory backends and conditions, and writer image access.

A Additional Results

  • Table 3 reports success and inference costs for Gemini 3.7 Flash across 53 tasks, five backends, and three seeds.It separates information and action success and pools writer tokens across four backends.Source: Table 3 · p. 14
  • Table 4 reports task success by category and plotted differences with 95% intervals from a paired bootstrap over tasks.The evaluation pools five backends and three seeds.Source: Table 4 · p. 14
  • Table 5 reports task success and read tokens by base model, with each model serving as writer, describer, and agent on the same recorded tours.Weighted token costs use model-specific input and output prices.Source: Table 5 · p. 14
  • Tables 6 and 7 cover backend-condition success and the cost of writer image access, respectively.Table 6 also reports the share of offered frames retained when the writer selects frames.Source: Table 6 · p. 15, Table 7 · p. 15

B Task inventory

The task inventory organizes 53 tasks across four indoor scenes, spanning information and action requests in seven task categories.

B Task inventory

  • Task categories cover object location, spatial configuration, appearance binding, set aggregation, user preference, occluded state, and skill outcome.The inventory abbreviates these categories as loc, spat, app, set, pref, occl, and skill.Source: Table 8 · p. 16
  • Most tasks have one information and one action request; info-only and skill-outcome tasks have only one request type, while some preference tasks and qa_inventory have two information requests.Tasks marked with † are excluded from their scene’s stacked composite.Source: Table 8 · p. 16
  • The inventory contains 12 office tasks, 14 studio tasks, 14 home tasks, and 13 clinic tasks.The table labels the inventory as 53 tasks across four scenes.Source: B Task inventory · p. 16, Table 8 · p. 16

Related papers

Loading 2610.12561v1…