PaperlayerView exact revision on arXiv

Source-linked AI summary

ECoMEM: Explicit Concept Memory for Memory-Dependent Robot Control

Yize Liu, Ke Wang, Mac Schwager, Yiqing Xu, Jiajun Wu

arXiv:2610.00801v1cs.RO

TL;DR

When a robot’s current observation omits facts needed for action, history-based VLA policies leave unclear which facts should persist or be revised. ECoMEM uses an evidence-based Writer and reusable concept library to build explicit memory, while a learned Reader conditions the VLA on it. It leads on 15 of 16 RoboMME tasks and achieves 86.1% success on new real-robot tasks, compared with 8.6% for a no-memory baseline.

  • Problem

    History-based VLA policies do not specify which past facts to retain, when to revise beliefs, or whether observations refer to the same event, although those facts may no longer be visible when action is needed.

    Source: §1 · p. 2

  • Method

    ECoMEM uses a shared library of grounded concepts and an evidence-based Writer to maintain records, while a learned Reader maps them into tokens that condition the VLA.

    Source: §1 · p. 2

  • Results

    ECoMEM leads evaluated policies on 15 of 16 RoboMME tasks and succeeds in 68/79 real-robot trials (86.1%), versus 5/58 (8.6%) for no-memory π0.5.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across the evaluated tasks, the concept library serves as a reusable and extensible memory interface, transferring directly to one new task and requiring one new concept for another.

    Source: §1 · p. 3, §6 · p. 12

    On InsertPeg, ECoMEM reaches 40.67% versus 7.56% for the strongest baseline but remains below human performance; precise execution must still be learned by the policy.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

A robot may lose sight of an object it must later retrieve, need to recall what a person demonstrated earlier, or track which steps of a task it has already completed. Current vision-language-action (VLA) policies often fail once the information needed for action disappears from the current observation, making memory critical for long-horizon robot behavior. Existing approaches typically provide longer histories or learn implicit memory from observation-action trajectories. But action supervision tells a policy how to act, not what to remember: it does not specify which past facts should persist or how they should change as new evidence arrives. We therefore separate maintaining an evidence-grounded account of the past from learning how to act on it. This insight motivates Explicit Concept Memory (ECoMEM), which represents task-relevant history with a reusable library of grounded concepts. An evidence-based Writer selects and updates these records, while a learned Reader turns them into memory tokens that directly condition the VLA. Across 16 RoboMME tasks, ECoMEM leads the evaluated robot policies on 15 tasks. On two new real-robot tasks, the same memory library either transfers directly or requires only one new concept, achieving 86.1% success versus 8.6% for a no-memory VLA. These results show that explicit concepts provide a reusable and extensible memory interface for robot control. Project website: https://ecomem.github.io/

1 Introduction

Long-horizon robots need compact records of task-relevant facts that may no longer be visible. ECoMEM separates evidence-based memory construction from learning how to use memory for action, and reports strong benchmark and real-robot results.

1 Introduction

  • Existing history-based policies leave implicit which facts to retain, when to revise beliefs, or whether observations refer to the same event; action prediction alone does not establish reliable memory.Explicit memories also face construction challenges in entity tracking, state estimation, and temporal interpretation.Source: §1 · p. 2
  • ECoMEM represents task-relevant history as explicit facts about entities, states, completed events, progress, and temporal order, updated only when visual and robot-state evidence supports them.Instructions specify what should be achieved, while memory records what has happened.Source: §1 · p. 2
  • ECoMEM uses a shared library of grounded concepts, with an evidence-based Writer maintaining records and a learned Reader mapping them into tokens that condition the VLA.Recognition and writing remain fixed during policy training, separating what evidence determines should be remembered from how action supervision teaches the policy to use memory.Source: §1 · pp. 2–3
  • ECoMEM achieves the highest average success among evaluated policies and leads on 15 of 16 RoboMME tasks; on real robots it succeeds in 68/79 trials (86.1%) versus 5/58 (8.6%) for no-memory π0.5.CUPSWITCH reuses the library directly, while SCOOPPOUR requires only one new event concept.Source: §1 · p. 3

2 Related Work

Prior work stores history implicitly, retains selected experience explicitly, or uses symbolic concepts for reasoning and planning. ECoMEM instead maintains evidence-linked facts and feeds them directly to a low-level VLA policy.

2.1 Implicit memory in robot policies

2.2 Explicit memory in robot policies

2.3 Neuro-symbolic methods in robotics

  • Neuro-symbolic robotics grounds reusable symbolic concepts in learned representations; ECoMEM uses that principle to interface perceptual history with a learned control policy.Prior examples include structured world models (Mao et al., 2019, 2022) and symbolic planning abstractions (Liang et al., 2025; Silver et al., 2023).Source: §2.3 · p. 4

3 Preliminaries

A task is memory-dependent when identical instructions and current observations can call for different actions because the histories differ. ECoMEM compresses history into concept records and conditions the VLA on their token representations.

3 Preliminaries

  • A task is memory-dependent when two histories with the same instruction and current observation require different expert actions.For example, the scene may look identical after the first and third button presses, although the correct action differs.Source: §3 · p. 4
  • ECoMEM compresses the history into a set of concept records, then separates what to remember from how the policy uses it.The Writer selects concepts, recognizes them from history, and updates records; the Reader maps records to memory tokens that condition the VLA.Source: §3 · pp. 4–5
  • The shared concept library spans tasks: Figure 3 maps task rows to concepts grouped in four semantic-family columns, with filled cells indicating which concepts each task uses.The figure shows reuse across the 16 RoboMME tasks and two real-robot tasks.Source: Figure 3 · p. 5

4 ECoMEM

ECoMEM pairs a shared library of task-relevant concepts with a Writer that builds evidence-grounded records and a Reader that turns them into tokens for VLA control. Its update rules preserve supported facts, deduplicate events, and track progress against goals.

4 ECoMEM

  • ECoMEM separates memory construction from memory use: the Writer selects, recognizes, and updates concepts, while the Reader encodes structured memory as tokens that condition the VLA.Those tokens join image and language inputs to the policy.Source: §4 · p. 5, §4.3 · p. 7

4.1 Concept Library

  • A shared library reuses fact types across tasks, organized into four families: entity and spatial grounding, state and relations, events and progress, and temporal and procedural structure.Each family resolves a distinct ambiguity, from what and where to the order of events and steps.Source: §4.1 · pp. 5–6
  • Concepts specify arguments, recognizers, and explicit update logic, and can compose; new concepts reuse existing perception models, lower-level concepts, and record formats.The paper’s scooping example adds a check that the scooper carries beans to existing grounding, containment, and event-counting concepts.Source: §4.1 · p. 6

4.2 Constructing Concept Memory

  • The Writer activates instruction-required concepts and dependencies, grounds them to instruction-derived entities and properties, and recognizes them using visual and robot-state evidence.Recognition composes spatial and relational checks, event checks, and temporal operations into grounded, time-stamped results.Source: §4.2 · pp. 6–7
  • The Writer stabilizes memory by confirming states with accumulated evidence, recording each event once, counting distinct events, and requiring prerequisites before ordered progress advances.A missed detection does not negate a state, so an occluded object retains its last location; demonstrations can set bindings and order but do not count as the robot’s progress.Source: §4.2 · p. 7

4.3 Integrating Concept Memory with VLA

  • The Reader embeds each record’s discrete fields with field-specific embeddings, then uses a Transformer to produce one token per record and relate facts without pooling them away.Records form a set without position embeddings; temporal information enters through age and goal dependencies.Source: §4.3 · p. 7
  • The Reader’s tokens join image and language tokens in the policy prefix, and the Reader is trained jointly with the language backbone and action expert while the vision encoder and Writer remain frozen.Training memory uses only observations available up to each decision, as at execution.Source: §4.3 · p. 7

5 Experiments

The experiments compare ECoMEM with memory baselines, isolate concept-family and Writer-construction choices, and test transfer to two real-robot tasks. Results show strong RoboMME performance and transfer through library reuse or a one-concept extension.

5.1 Experiment Setup

  • The evaluation covers 16 RoboMME tasks and two new real-robot tasks, with baselines spanning no memory, history conditioning, and visual memory.RoboMME tasks cover event counting, object persistence through occlusion, past-interaction reference, and demonstrated behavior; each task has 50 validation episodes with three policy sampling seeds.Source: §5.1 · p. 8

5.2 Concept Memory Outperforms Implicit and Visual Memory

  • ECoMEM achieves 82.42% average success, 37.91 points above FrameSamp+Modul, and leads the evaluated policies on 15 of 16 RoboMME tasks.Its average is within 8.08 points of human performance.Source: §5.2 · p. 9
  • ECoMEM's advantage is largest when target, location, or event order must persist beyond visibility; on InsertPeg it reaches 40.67% versus 7.56% for the strongest baseline, but remains below human performance.PickHighlight is the only task where ECoMEM does not lead; MemER is 8 points higher, consistent with a transient visual cue preserved in a retrieved keyframe.Source: §5.2 · p. 9

5.4 Reliable Memory Requires Structured Construction

  • With the policy fixed, full-library recognition adds task-unselected records, reduces success on VideoUnmask and VideoRepick (64% to 8% on VideoRepick), and increases median initialization latency 12.0×–18.1× on matched episodes using the same GPUs.Source: Figure 5 · p. 9, §5.4 · p. 11

5.3 Concept Families Provide Complementary Memory

  • Removing spatial grounding, event and progress information, or temporal order drops success from 84% to 18% on VideoUnmask, 62% to 14% on PickXtimes, and 60% to 12% on PatternLock (Figure 4).The interventions remove indirect cues and are applied during both training and evaluation on matched data.Source: Figure 4 · p. 9, §5.3 · p. 10

5.4 Reliable Memory Requires Structured Construction

  • Generic role-level queries drop target-location agreement from 98% to 20% and VideoUnmask success from 86% to 24%; ordered-location agreement and PatternLock success fall to 0% and 2% (Figure 6).Concept selection and the policy are fixed, and the policy still receives the full instruction.Source: Figure 6 · p. 9, §5.4 · p. 11
  • Across eight concepts on 95 held-out videos, direct VLM extraction reaches only 5.9–11.8% for ordered sequences, while Concept Memory achieves 88.2–100% on temporal concepts.Direct extraction is stronger on local state and simple event counts; similar Structured and QA results indicate output format is not the main bottleneck.Source: Table 2 · p. 10, §5.4 · p. 10

5.5 The Library Transfers to New Tasks on a Real Robot

  • On real-robot tasks, ECoMEM succeeds in 68/79 trials (86.1%) versus 5/58 (8.6%) for no-memory; CupSwitch reuses the library, while ScoopPour adds SCOOP so only successful bean-carrying scoops advance the counter.The same counting machinery tracks completed round trips in CupSwitch and successful scoops in ScoopPour; empty scoops leave progress unchanged and trigger a retry.Source: Table 3 · p. 11, Figure 7 · p. 11, §5.5 · p. 12

6 Conclusion

ECoMEM separates evidence-based memory construction from learned action generation, using a shared concept library and Writer–Reader pipeline to provide reusable memory for VLA control.

6 Conclusion

  • ECoMEM separates evidence-based memory construction from learned action generation.This explicit memory interface is presented as reusable and extensible for VLA control.Source: §6 · p. 12
  • Across 16 RoboMME tasks, ECoMEM substantially outperforms no-memory, history-based, and visual-memory alternatives.The conclusion also reports transfer to new real-robot tasks through direct library reuse or adding one task-specific concept.Source: §6 · p. 12
  • A shared, compositional concept library represents task-relevant world state, events, progress, and temporal structure.The Writer maintains these records; the Reader learns how to use them for control.Source: §6 · p. 12

A Policy Training

Policy training jointly optimizes the Reader and unfrozen policy weights while keeping recognition and memory writing fixed; training uses RoboMME data and an action-chunk objective.

A.1 Shared policy configuration.

  • The policy uses pretrained π0.5 with a 2B language backbone and 300M action expert; the image backbone is frozen, while the language backbone, action expert, and Reader are jointly optimized.Recognition and memory writing remain fixed, and the policy predicts 20 actions but executes the first 16 before requesting another chunk.Source: §A.1 · p. 17

A.2 Main benchmark training.

  • ECoMEM is initialized from pretrained π0.5 and trained across all 16 RoboMME tasks, sampling uniformly from retained observation–action frames.Results use a single EMA checkpoint; training takes 4× H200 GPUs for 24 hours.Source: §A.2 · p. 17

A.3 Controlled comparisons.

  • Each controlled comparison is independently trained from the same pretrained π0.5 policy, with interventions matched across training and evaluation.Pairs use identical observation/action indices and normalization, each with its own matched full-memory control; budgets and memory-construction settings differ from the main benchmark.Source: §A.3 · p. 17

A.4 Flow-matching objective.

  • The flow-matching objective predicts conditional velocity from the noisy expert action chunk, instruction, current observation, and causal memory.The optimized parameters are the Reader weights ϕ and unfrozen policy weights θ, with squared error averaged over action-chunk entries.Source: §A.4 · p. 17

B Main Benchmark Evaluation

This material describes benchmark baselines and the methods that construct, update, and present ECoMEM’s concept memory; it does not include benchmark outcome comparisons.

B.1 Current-observation and past-action baselines.

  • Standard π0.5 uses current images, robot state, and instruction, while its past-action variant adds issued actions but not their visual outcomes.The symbolic SimpleSG and GroundSG baselines append VLM-predicted subgoals to the task instruction; GroundSG also supplies image coordinates.Source: §B.1 · p. 18, §B.2 · p. 18

B.3 Perceptual memory: FrameSamp and TokenDrop.

  • FrameSamp retains sampled whole observations, whereas TokenDrop selects changing image patches under a fixed token budget; RMT carries token-valued memory state and TTT carries parameter-valued state across chunks.Context, Modul, and Expert compare joint input processing, action-feature modulation, and a dedicated memory-processing pathway.Source: §B.3 · p. 18, §B.4 · p. 18, §B.5 · p. 18

B.6 MemER.

  • MemER selects historical keyframes for high-level language subtasks, while SAM2Act+ retrieves stored visual features and spatial action information for subsequent actions.RoboMME adapts MemER with fine-tuned Qwen3-VL-4B and GroundSG; SAM2Act+ uses planner-executed predicted keyframe actions and demonstration-prefilled memory.Source: §B.6 · p. 19, §B.7 · p. 19

C Concept Library

  • The concept library groups remembered concepts into semantic families and represents entity-bound states such as attachment, holding, visibility, containment, and motion.It distinguishes missing evidence from confirmed negative states, and physical states such as attachment, holding, and lifting require separate support.Source: C Concept Library · pp. 19–20, Table 4 · p. 19
  • Progress is represented by distinct completed-unit records and goal records that mark satisfaction, executable frontier, or blocked status.Ordered bindings and temporal records preserve event timing, evidence provenance, and demonstrated entity, control, or waypoint sequences.Source: C Concept Library · pp. 20–21

D.1 Language-grounded perception.

  • Language-grounded entity descriptions are detected and assigned episode-local identities, while temporal sampling and tracking maintain entity associations and last-supported locations through occlusion.Figure 8 illustrates concept recognition and update through a pick-and-place interaction.Source: D Observation & Perception · p. 21, Figure 8 · p. 21, §D.1 · p. 22

E.1 Concept selection.

  • Task-conditioned concept selection chooses recognizers, whose outputs combine visual and robot-state evidence to identify grounded relations, events, and ordered bindings.For pick-and-place, gripper transitions delimit an interaction interval, and object, action, and target checks support recognition of the completed instance.Source: §E.1 · p. 22, §E.2 · p. 22, §E.3 · pp. 22–23

F.1 Evidence merging.

  • Evidence Merge retains provenance while combining active recognizers’ outputs, and the Writer integrates these with previous records to refresh event history and progress.The Reader converts the updated records into memory tokens for the VLA policy.Source: F Concept Merge & Update · p. 23, §F.1 · p. 23

F.5 Reader architecture.

The Reader converts structured memory records into tokens for VLA conditioning, with training samples aligned to evidence available by each decision frame.

F.5 Reader architecture.

  • The Reader embeds eleven fields into 256-dimensional vectors, combines them per record, and uses two Transformer blocks before projecting one 2,048-dimensional token per record.These memory tokens join image and language tokens in the VLA prefix, and the Reader is trained jointly with the policy.Source: §F.5 · p. 24

F.6 Causal training alignment.

  • Each training sample pairs the current observation and memory available through that frame with the subsequent expert action chunk.Demonstrated references remain separate from execution progress, and events enter memory once their supporting evidence is available; shared identifier renaming preserves record connections.Source: §F.6 · pp. 24–25

G Ablations of pipeline

The appendix specifies interventions on memory-record content and task-conditioned selection, measures initialization latency, and evaluates language-conditioned grounding. It also describes the RoboMME data and supervision used for concept learning.

G.1 Content interventions.

  • No Spatial zeros coordinate fields, No Temporal removes age and ordering cues, and No Event/Progress removes completion records and progress bits while retaining current-state evidence.The interventions preserve other record content and masks as specified for each condition.Source: §G.1 · p. 25

G.2 Initialization latency.

  • Median per-episode initialization was 2.23 s versus 40.47 s for VideoUnmask and 4.38 s versus 52.66 s for VideoRepick.Timing covers recognition through evidence merging, memory construction, and interface validation; VLA loading and action execution are excluded.Source: §G.2 · p. 25, Table 6 · p. 26

G.3 Language-conditioned grounding.

  • 98/100 versus 20/100 VideoUnmask matches and 86/98 versus 0/98 PatternLock matches favor specific entity queries; closed-loop success is 43/50 versus 12/50 and 36/50 versus 1/50, respectively.PatternLock excludes episodes 29 and 83 because the required reference row was absent at the retrieval frame.Source: §G.3 · p. 26, Table 7 · p. 26

H.1 Dataset and supervision.

  • 504 recorded RoboMME videos from six tasks are split into 365 training, 44 validation, and 95 test videos, with scene-seed groups kept within one split.Reference labels come from simulator replays using physical events, rendered highlights, and button states; they are algorithmically generated and audited rather than exhaustively hand-annotated.Source: §H.1 · p. 26

H.2 Training.

The paper compares structured-record and textual-answer concept training, then describes the real-robot tasks, memory generation, policy training, and deployment setup. The robot policies use shared task data and an identical execution interface across memory and no-memory conditions.

H.2 Training.

  • Structured training generates JSON records, while QA training produces concept-specific textual answers; both use matched visual inputs, semantic supervision, and sample order.Separate adapters are trained for historical and current-state concepts, updating only language-model attention LoRA parameters.Source: §H.2 · p. 27

I.1 Tasks and data.

  • Each task uses 59 demonstrations; ScoopPour has 80,203 frames and CupSwitch 81,200, with evaluation covering ScoopPour 1–3× and CupSwitch 2–3×.The one-cycle ScoopPour target is absent from fine-tuning data.Source: §I.1 · p. 27

I.2 Concept adaptation and memory generation.

  • ScoopPour adds event.scoop while reusing grounding, ordering, and counting; CupSwitch composes existing entity, pick-and-place, sequence, and counting concepts into round trips.A scoop cycle completes when a loaded-to-empty transition above the plate is confirmed across consecutive observations.Source: §I.2 · pp. 27–28

I.3 Policy training.

  • Concept Memory and the no-memory π0.5 baseline use the same retained data, inputs, normalization, and pretrained backbone, with the baseline omitting the memory encoder and tokens.Both policies are fully fine-tuned; dense-frame sampling supplies 20-step action targets.Source: §I.3 · p. 28

I.4 Deployment and evaluation.

  • The remote server predicts 20-step action chunks, the robot executes the first 16 steps before replanning, and streamed-observation memory updates run at 5 Hz for ScoopPour and 10 Hz for CupSwitch.Memory resets at episode boundaries, and both methods share the execution interface and command-increment limits.Source: §I.4 · p. 28

I.5 Results.

Concept Memory has higher pooled real-robot success than the no-memory baseline across the reported ScoopPour and CupSwitch conditions. The paper reports outcome-annotation rules and both pooled and condition-level scoring scope.

I.5 Results.

  • 68/79 successes (86.08%) for Concept Memory versus 5/58 (8.62%) for the baseline is a gain of 77.46 percentage points.The reported conditions are ScoopPour 1–3× and CupSwitch 2–3×.Source: §I.5 · pp. 28–29, Table 8 · p. 29
  • The mean weights the five conditions equally, whereas the pooled rate weights individual trials equally.The reported conditions comprise ScoopPour 1–3× and CupSwitch 2–3×.Source: §I.5 · p. 29
  • Concept Memory trials marked fail or timeout count as failures; baseline annotations count as failures for textual failure labels, including timeouts after reaching the target count.Baseline numeric suffixes matching the requested count, or absent suffixes, count as successes.Source: §I.5 · p. 29, Table 9 · p. 29

Related papers

Loading 2610.00801v1…