PaperlayerView exact revision on arXiv

Source-linked AI summary

MemER: Scaling Up Memory for Robot Control via Experience Retrieval

Ajay Sridhar, Jennifer Pan, Satvik Sharma, Chelsea Finn

arXiv:2510.20328v1cs.ROcs.AIcs.LG

TL;DR

Robot policies need long-term visual memory, but long histories are costly and brittle, while indiscriminate subsampling can retain irrelevant information. MemER uses hierarchical experience retrieval to select keyframes and generate subtasks for a low-level controller. Across three real-world tasks requiring minutes of memory, it improves performance while retaining low-latency inference and compatibility with VLA backbones.

  • Problem

    Robot policies generally lack long-term visual memory, while conditioning on long observation histories is expensive and can be brittle under distribution shift.

  • Method

    MemER uses a high-level VLM to select and consolidate task-relevant keyframes and emit subtasks for a low-level robot policy to execute.

  • Results

    Across three real-world long-horizon manipulation domains, MemER significantly improves performance on tasks requiring minutes of recall while retaining low-latency inference.

  • Takeaways & Limitations

    Experience retrieval provides a compact episodic-memory strategy compatible with existing VLA backbones for long-horizon robot control.

  • Takeaways & Limitations

    The method currently accumulates keyframes without deleting them, which may limit tasks requiring hours of memory; throughput also depends on VLM scheduling and backbone speed.

Abstract

from arXiv · show

Humans routinely rely on memory to perform tasks, yet most robot policies lack this capability; our goal is to endow robot policies with the same ability. Naively conditioning on long observation histories is computationally expensive and brittle under covariate shift, while indiscriminate subsampling of history leads to irrelevant or redundant information. We propose a hierarchical policy framework, where the high-level policy is trained to select and track previous relevant keyframes from its experience. The high-level policy uses selected keyframes and the most recent frames when generating text instructions for a low-level policy to execute. This design is compatible with existing vision-language-action (VLA) models and enables the system to efficiently reason over long-horizon dependencies. In our experiments, we finetune Qwen2.5-VL-7B-Instruct and $π_{0.5}$ as the high-level and low-level policies respectively, using demonstrations supplemented with minimal language annotations. Our approach, MemER, outperforms prior methods on three real-world long-horizon robotic manipulation tasks that require minutes of memory. Videos and code can be found at https://jen-pan.github.io/memer/.

1 INTRODUCTION

MemER addresses the lack of long-term memory in robot policies by retrieving task-relevant past information through a hierarchical policy. It uses a high-level policy to select keyframes and issue subtasks to a low-level controller, enabling minutes-long reasoning across three real-world tasks.

  • Motivation: Robot policies often lack long-term visual memory needed for complex, partially observable, multi-step tasks.The paper motivates memory by examples such as recalling previously viewed objects during sandwich preparation.
  • Motivation: Long observation histories are computationally expensive, slow deployment, and can encourage shortcut reliance that harms generalization.Prior approaches also remain difficult to scale naively to very long histories.
  • Approach: MemER filters and stores task-relevant information from historical context to prevent memory growth from becoming unmanageable.This addresses the need to preserve long-range dependencies without retaining indiscriminate history.
  • Approach: Its hierarchical design uses a video-understanding high-level policy to select keyframes and predict subtasks, while a low-level policy executes them.The approach adapts open-source VLMs using 50 teleoperated demonstrations with subtask annotations.
  • Scope: MemER uses task-relevant past information across three complex long-horizon tasks requiring up to a few minutes of memory.The overview describes one policy trained with only a small number of expert demonstrations.

2 RELATED WORK

Prior work extends robot-policy context only modestly, while MemER targets memory built from hundreds of frames. Related video keyframe-selection methods offer a relevant precedent but may impose per-frame costs incompatible with robotic latency.

  • Robot Policy Memory: Prior robot-memory studies extend imitation-policy context from a few frames to at most two dozen.Fang et al. stores at most N = 10 of the most recent context frames.
  • Robot Policy Memory: MemER investigates tasks requiring memory construction from hundreds of frames, beyond the horizons emphasized by prior work.The comparison frames MemER as targeting substantially longer temporal dependencies.
  • Foundation Models in Robotics: The paper uses open-weight models to reason about robot affordances in its long-horizon manipulation tasks.This choice appears in the discussion of foundation models for robotics.
  • Video Keyframe Selection: Video-understanding research has used keyframe selection to incorporate longer contexts for VLMs and video question answering.These methods provide conceptual precedent outside robotics.
  • Video Keyframe Selection: Many video keyframe methods estimate frame importance with separate multimodal-LLM calls, creating high per-frame costs for robotic use.The paper notes that such costs conflict with the latency requirements of robotic tasks.

3 MEMER

MemER decomposes long-horizon robot control into a high-level memory policy and a low-level control policy. The high-level policy selects task-relevant keyframes from recent context, filters them into persistent memory, and generates subtasks for execution.

  • Architecture: MemER hierarchically decomposes robot control into a high-level policy that generates instructions and a low-level policy that executes them.The high-level policy reasons about memory, while the low-level policy handles robot-specific, high-frequency control.
  • High-level policy: The high-level policy conditions on recent frames, the task instruction, and previously selected keyframes to predict the current subtask and candidate keyframes.Candidate keyframes are nominated from the recent context and selected keyframes persist information from earlier observations.
  • Visual memory: 1D single-linkage clustering consolidates candidate keyframes into disjoint memory entries, using a merge distance of d = 5 frames.One representative frame is added to memory for each cluster.
  • Visual memory: The keyframe filter pools nominated frame indices across the trajectory, preserves duplicate nominations, clusters nearby indices, and selects each cluster’s median frame.Clusters group indices at most d apart; clusters sufficiently far in the past need not be recalculated.
  • Low-level control: The low-level policy predicts direct robot actions from the current images, joint states, and language subtask, while the high-level policy supplies subtasks and memory selections.The policies are trained using language-subtask annotations and trajectories containing images, proprioceptive inputs, task instructions, and subtasks.
  • Training: The implementation finetunes Qwen2.5-VL-7B-Instruct as the high-level policy and π0.5 as the low-level policy, using a single policy for all three tasks.The low-level policy uses 50 long-horizon demonstrations plus 10–15 intervention examples per task; model merging improves or maintains performance across tasks.

4 EXPERIMENTS

The experiments evaluate memory-based hierarchical robot policies across three long-horizon manipulation tasks, comparing MemER with history-based, modality, model, and proprietary-VLM alternatives. MemER outperforms naive history baselines, exceeds 90% on all tasks, and shows robust visual-memory performance.

  • Evaluation Tasks: The evaluation covers three real-robot tasks requiring object-location memory, action tracking, or repeated-step counting, with task-specific completion metrics.The setup uses 20 trials per task and method, with complex instructions, intermediate subtasks, and keyframe predictions evaluated across three domains.
  • Main Results: MemER clearly outperforms no-history, 8-frame, and 32-frame baselines across the three long-horizon memory tasks and performs on par with the human high-level policy.The baselines use the same low-level policy while varying the high-level policy’s input context.
  • Main Results: > 90% performance on all tasks is reported for MemER, while the 32-frame policy remains 34% worse on average and incurs 1 second of inference cost.The dominant remaining failures occur in the low-level policy’s execution of subtasks.
  • API-Based VLMs: Zero-shot GPT-5 and Gemini Robotics–ER 1.5 perform poorly relative to the finetuned Qwen2.5-VL high-level policy, chiefly predicting too many non-informative keyframes.The comparison uses offline trajectory and boundary accuracy on held-out low-level-policy trajectories.
  • Modality Results: Visual-only memory performs better than text-only and text-plus-image alternatives, while adding text to MemER provides no benefit and underperforms across tasks.The reported explanation is that the high-level policy over-attends to text tokens and misses visual details.

5 DISCUSSION AND FUTURE WORK

MemER scales robotic memory through hierarchical experience retrieval, retaining task-relevant keyframes while supporting low-latency control. The authors identify limits in memory growth, throughput, sensory coverage, and robot embodiment as directions for future work.

  • Discussion: MemER uses a high-level policy to retain keyframes and emit language subtasks for a low-level controller.An online consolidation strategy converts timestep-level candidates into compact episodic memory.
  • Future Work: The current system continuously accumulates keyframes without deleting them, limiting scalability to tasks requiring hours of memory.Future work would let the high-level policy add and delete keyframes.
  • Future Work: Throughput depends on the VLM backbone and scheduling, which can limit very high-frequency control and reactivity to rapid environmental changes.The stated example schedules are πh at 1 Hz and πl at 2 Hz.
  • Future Work: MemER currently stores only visual observations, uses one robot embodiment, and does not yet address mobile manipulation or multi-room tasks.The authors propose tactile and audio sensing, broader embodiments, and spatial-memory settings as extensions.

A MODEL INITIALIZATION AND HYPERPARAMETERS

The high-level and low-level policies are fine-tuned separately, with their respective model configurations documented in Tables 4 and 5.

  • Model Initialization and Hyperparameters: The high-level policy is fine-tuned using Qwen2.5-VL-7B-Instruct.Its fine-tuning hyperparameters are documented in Table 4.
  • Model Initialization and Hyperparameters: Table 4 documents the hyperparameters used for high-level policy fine-tuning.
  • Model Initialization and Hyperparameters: The low-level policy is fine-tuned using the π0.5 model.Its fine-tuning hyperparameters are documented in Table 5.
  • Model Initialization and Hyperparameters: Table 5 documents the hyperparameters used for low-level policy fine-tuning.

B DATA COLLECTION AND LABELING THE SUBTASKS

Robot trajectories are collected through Oculus teleoperation, with subtasks prepared before collection and completion marked by keyboard input.

  • Data Collection and Labeling the Subtasks: Robot trajectory data are collected using the Oculus teleoperation procedure from Khazatsky et al. (2025).
  • Data Collection and Labeling the Subtasks: A list of subtasks for the whole task trajectory is generated before data collection begins.This preparation simplifies the subsequent subtask-labeling process.
  • Data Collection and Labeling the Subtasks: During collection, operators follow the current subtask and indicate its completion with a simple keyboard input.

C CROSS-TASK OBJECT GENERALIZATION

The cross-task evaluation swaps objects between Object Search and Dust & Replace to test unseen object-task combinations. Multi-task MemER generalizes more effectively than single-task MemER, while original-task performance is roughly similar.

  • Cross-Task Object Generalization: Single-task and multi-task MemER perform roughly similarly on their original Object Search and Dust & Replace tasks.
  • Cross-Task Object Generalization: Cross-task evaluation swaps all objects between Object Search and Dust & Replace, creating object-task combinations unseen during training.Figure 9 shows the objects used for Object Search, Counting, and Dust & Replace.
  • Cross-Task Object Generalization: The multi-task model generalizes more effectively than the single-task version to new object-task combinations.The figure caption states that the multi-task policy generalizes despite never seeing those combinations during training.
  • Cross-Task Object Generalization: The reported results associate scaling the number of memory-based training tasks with improved generalization.

D KEYFRAME SELECTION ALGORITHM

The keyframe-selection algorithm converts candidate frame sets into temporal indices, groups nearby indices, and selects representative keyframes from each group.

  • The algorithm takes candidate keyframe sets, a merge distance d, and outputs the selected-keyframe list K_t.It extracts temporal indices from the candidate frames before selection.
  • If no candidate indices exist, the algorithm returns an empty set.
  • Sorted temporal indices are clustered when consecutive indices differ by at most the merge distance d.
  • The procedure then selects the median index from each temporal cluster as a representative keyframe.

E ANNOTATION RULES FOR KEYFRAMES

Ground-truth keyframes are defined with simple semantic rules that select final frames for milestone subtasks while omitting intermediate or reset actions.

  • Each subtask interval receives a semantic annotation rule that determines its ground-truth keyframes.
  • Object-search annotations select the last frame of looking inside a location bin and no frames from object-transfer intervals.
  • Ingredient-scooping annotations select the last frame of placing a scoop in a bowl, while omitting pickup, reset, and drop-scooper intervals.
  • Shelf-cleaning annotations select the last frame for removing each object and dusting each shelf, while omitting duster reset and placement intervals.
  • Shelf-restoration annotations select the last frame of placing each object back on the bottom and top shelves.

F PROMPTS FOR TRAINING MEMER

Training prompts combine selected memory frames with the robot’s most recent context frames and require a high-level policy to return the current subtask and relevant frame positions.

  • The training prompt presents a video of the robot’s most recent actions and asks for the current subtask plus keyframe positions.
  • The object-search example predicts taking fried chicken from the right bin and placing it in the white bin, with keyframe position 7.
  • The system prompt describes selected frames as important actions from the robot’s full history and requests a JSON response.
  • The prompt schema requires current_subtask to identify the action at the current timestep and keyframe_positions to identify where actions change.

G PROMPTS FOR GPT-5 / GEMINI ROBOTICS–ER 1.5 EVALUATION

Evaluation prompts specify task-specific action spaces, object or ingredient vocabularies, and memory cues that guide selecting recent frames for long-horizon state tracking.

  • The object-search evaluation prompt orders bin exploration from left to center to right and defines the permitted search and transfer actions.
  • Object search uses a fixed vocabulary of locations and target objects, including fried chicken, and asks the model to remember which objects are in each bin.
  • For ingredient tasks, the prompt requires resetting the scooper between scoops and dropping it after all scoops are obtained.
  • Ingredient evaluation emphasizes remembering completed scoops and returns frame indices that preserve evidence of scoop counts.
  • Shelf-cleaning evaluation uses global and wrist cameras, defines removal, dusting, reset, and replacement actions, and asks the model to remember object locations and dusted shelves.
Loading 2510.20328v1…