MemER: Scaling Up Memory for Robot Control via Experience Retrieval

Motivation

• Robot policies often lack long-term visual memory needed for complex, partially observable, multi-step tasks.The paper motivates memory by examples such as recalling previously viewed objects during sandwich preparation.Source: §1 · p. 1
• Long observation histories are computationally expensive, slow deployment, and can encourage shortcut reliance that harms generalization.Prior approaches also remain difficult to scale naively to very long histories.Source: §1 · p. 1

Approach

• MemER filters and stores task-relevant information from historical context to prevent memory growth from becoming unmanageable.This addresses the need to preserve long-range dependencies without retaining indiscriminate history.Source: §1 · p. 2
• Its hierarchical design uses a video-understanding high-level policy to select keyframes and predict subtasks, while a low-level policy executes them.The approach adapts open-source VLMs using 50 teleoperated demonstrations with subtask annotations.Source: §1 · p. 2

Scope

• MemER uses task-relevant past information across three complex long-horizon tasks requiring up to a few minutes of memory.The overview describes one policy trained with only a small number of expert demonstrations.Source: §1 · p. 2, Figure 1 · p. 2

Robot Policy Memory

• Prior robot-memory studies extend imitation-policy context from a few frames to at most two dozen.Fang et al. stores at most N = 10 of the most recent context frames.Source: §2 · p. 2
• MemER investigates tasks requiring memory construction from hundreds of frames, beyond the horizons emphasized by prior work.The comparison frames MemER as targeting substantially longer temporal dependencies.Source: §2 · p. 2

Foundation Models in Robotics

• The paper uses open-weight models to reason about robot affordances in its long-horizon manipulation tasks.This choice appears in the discussion of foundation models for robotics.Source: §2 · p. 3

Video Keyframe Selection

• Video-understanding research has used keyframe selection to incorporate longer contexts for VLMs and video question answering.These methods provide conceptual precedent outside robotics.Source: §2 · p. 3
• Many video keyframe methods estimate frame importance with separate multimodal-LLM calls, creating high per-frame costs for robotic use.The paper notes that such costs conflict with the latency requirements of robotic tasks.Source: §2 · p. 3

Architecture

• MemER hierarchically decomposes robot control into a high-level policy that generates instructions and a low-level policy that executes them.The high-level policy reasons about memory, while the low-level policy handles robot-specific, high-frequency control.Source: §3.1 · p. 3, §3.2 · p. 4

High-level policy

• The high-level policy conditions on recent frames, the task instruction, and previously selected keyframes to predict the current subtask and candidate keyframes.Candidate keyframes are nominated from the recent context and selected keyframes persist information from earlier observations.Source: §3.2 · p. 4

Visual memory

• 1D single-linkage clustering consolidates candidate keyframes into disjoint memory entries, using a merge distance of d = 5 frames.One representative frame is added to memory for each cluster.Source: Figure 3 · p. 5
• The keyframe filter pools nominated frame indices across the trajectory, preserves duplicate nominations, clusters nearby indices, and selects each cluster’s median frame.Clusters group indices at most d apart; clusters sufficiently far in the past need not be recalculated.Source: §3.2 · pp. 4–5

Low-level control

• The low-level policy predicts direct robot actions from the current images, joint states, and language subtask, while the high-level policy supplies subtasks and memory selections.The policies are trained using language-subtask annotations and trajectories containing images, proprioceptive inputs, task instructions, and subtasks.Source: §3.1 · p. 3, §3.3 · p. 5

Training

• The implementation finetunes Qwen2.5-VL-7B-Instruct as the high-level policy and π0.5 as the low-level policy, using a single policy for all three tasks.The low-level policy uses 50 long-horizon demonstrations plus 10–15 intervention examples per task; model merging improves or maintains performance across tasks.Source: §3.3 · pp. 5–6

Evaluation Tasks

• The evaluation covers three real-robot tasks requiring object-location memory, action tracking, or repeated-step counting, with task-specific completion metrics.The setup uses 20 trials per task and method, with complex instructions, intermediate subtasks, and keyframe predictions evaluated across three domains.Source: §4 · p. 6, Figure 4 · p. 7

Main Results

• MemER clearly outperforms no-history, 8-frame, and 32-frame baselines across the three long-horizon memory tasks and performs on par with the human high-level policy.The baselines use the same low-level policy while varying the high-level policy’s input context.Source: Figure 5 · p. 7, §4.1 · p. 8
• > 90% performance on all tasks is reported for MemER, while the 32-frame policy remains 34% worse on average and incurs 1 second of inference cost.The dominant remaining failures occur in the low-level policy’s execution of subtasks.Source: §4.1 · p. 9

API-Based VLMs

• Zero-shot GPT-5 and Gemini Robotics–ER 1.5 perform poorly relative to the finetuned Qwen2.5-VL high-level policy, chiefly predicting too many non-informative keyframes.The comparison uses offline trajectory and boundary accuracy on held-out low-level-policy trajectories.Source: §4.1 · p. 9

Modality Results

• Visual-only memory performs better than text-only and text-plus-image alternatives, while adding text to MemER provides no benefit and underperforms across tasks.The reported explanation is that the high-level policy over-attends to text tokens and misses visual details.Source: Figure 6 · p. 8, §4.1 · p. 10

Discussion

• MemER uses a high-level policy to retain keyframes and emit language subtasks for a low-level controller.An online consolidation strategy converts timestep-level candidates into compact episodic memory.Source: §5 · p. 10

Future Work

• The current system continuously accumulates keyframes without deleting them, limiting scalability to tasks requiring hours of memory.Future work would let the high-level policy add and delete keyframes.Source: §5 · p. 11
• Throughput depends on the VLM backbone and scheduling, which can limit very high-frequency control and reactivity to rapid environmental changes.The stated example schedules are πh at 1 Hz and πl at 2 Hz.Source: §5 · p. 11
• MemER currently stores only visual observations, uses one robot embodiment, and does not yet address mobile manipulation or multi-room tasks.The authors propose tactile and audio sensing, broader embodiments, and spatial-memory settings as extensions.Source: §5 · p. 11

Model Initialization and Hyperparameters

• The high-level policy is fine-tuned using Qwen2.5-VL-7B-Instruct.Its fine-tuning hyperparameters are documented in Table 4.Source: A MODEL INITIALIZATION AND… · p. 15
• Table 4 documents the hyperparameters used for high-level policy fine-tuning.Source: Table 4 · p. 15
• The low-level policy is fine-tuned using the π0.5 model.Its fine-tuning hyperparameters are documented in Table 5.Source: A MODEL INITIALIZATION AND… · p. 15
• Table 5 documents the hyperparameters used for low-level policy fine-tuning.Source: Table 5 · p. 15

Data Collection and Labeling the Subtasks

• Robot trajectory data are collected using the Oculus teleoperation procedure from Khazatsky et al. (2025).Source: B DATA COLLECTION AND LABELING THE… · p. 15
• A list of subtasks for the whole task trajectory is generated before data collection begins.This preparation simplifies the subsequent subtask-labeling process.Source: B DATA COLLECTION AND LABELING THE… · p. 15
• During collection, operators follow the current subtask and indicate its completion with a simple keyboard input.Source: B DATA COLLECTION AND LABELING THE… · p. 15

Cross-Task Object Generalization

• Single-task and multi-task MemER perform roughly similarly on their original Object Search and Dust & Replace tasks.Source: C CROSS-TASK OBJECT GENERALIZATION · p. 15, Figure 8 · p. 16
• Cross-task evaluation swaps all objects between Object Search and Dust & Replace, creating object-task combinations unseen during training.Figure 9 shows the objects used for Object Search, Counting, and Dust & Replace.Source: C CROSS-TASK OBJECT GENERALIZATION · p. 16, Figure 9 · p. 16
• The multi-task model generalizes more effectively than the single-task version to new object-task combinations.The figure caption states that the multi-task policy generalizes despite never seeing those combinations during training.Source: C CROSS-TASK OBJECT GENERALIZATION · p. 16
• The reported results associate scaling the number of memory-based training tasks with improved generalization.Source: C CROSS-TASK OBJECT GENERALIZATION · p. 16

• The algorithm takes candidate keyframe sets, a merge distance d, and outputs the selected-keyframe list K_t.It extracts temporal indices from the candidate frames before selection.Source: D KEYFRAME SELECTION ALGORITHM · p. 17
• If no candidate indices exist, the algorithm returns an empty set.Source: D KEYFRAME SELECTION ALGORITHM · p. 17
• Sorted temporal indices are clustered when consecutive indices differ by at most the merge distance d.Source: D KEYFRAME SELECTION ALGORITHM · p. 17
• The procedure then selects the median index from each temporal cluster as a representative keyframe.Source: D KEYFRAME SELECTION ALGORITHM · p. 17

• Each subtask interval receives a semantic annotation rule that determines its ground-truth keyframes.Source: E ANNOTATION RULES FOR KEYFRAMES · p. 17
• Object-search annotations select the last frame of looking inside a location bin and no frames from object-transfer intervals.Source: E ANNOTATION RULES FOR KEYFRAMES · p. 17
• Ingredient-scooping annotations select the last frame of placing a scoop in a bowl, while omitting pickup, reset, and drop-scooper intervals.Source: E ANNOTATION RULES FOR KEYFRAMES · p. 17
• Shelf-cleaning annotations select the last frame for removing each object and dusting each shelf, while omitting duster reset and placement intervals.Source: E ANNOTATION RULES FOR KEYFRAMES · p. 17
• Shelf-restoration annotations select the last frame of placing each object back on the bottom and top shelves.Source: E ANNOTATION RULES FOR KEYFRAMES · p. 18

• The training prompt presents a video of the robot’s most recent actions and asks for the current subtask plus keyframe positions.Source: F PROMPTS FOR TRAINING MEMER · pp. 18–19
• The object-search example predicts taking fried chicken from the right bin and placing it in the white bin, with keyframe position 7.Source: F PROMPTS FOR TRAINING MEMER · p. 19
• The system prompt describes selected frames as important actions from the robot’s full history and requests a JSON response.Source: F PROMPTS FOR TRAINING MEMER · p. 19
• The prompt schema requires current_subtask to identify the action at the current timestep and keyframe_positions to identify where actions change.Source: F PROMPTS FOR TRAINING MEMER · pp. 18–20

• The object-search evaluation prompt orders bin exploration from left to center to right and defines the permitted search and transfer actions.Source: G PROMPTS FOR GPT-5 / GEMINI… · p. 21
• Object search uses a fixed vocabulary of locations and target objects, including fried chicken, and asks the model to remember which objects are in each bin.Source: G PROMPTS FOR GPT-5 / GEMINI… · pp. 21–22
• For ingredient tasks, the prompt requires resetting the scooper between scoops and dropping it after all scoops are obtained.Source: G PROMPTS FOR GPT-5 / GEMINI… · p. 22
• Ingredient evaluation emphasizes remembering completed scoops and returns frame indices that preserve evidence of scoop counts.Source: G PROMPTS FOR GPT-5 / GEMINI… · p. 22
• Shelf-cleaning evaluation uses global and wrist cameras, defines removal, dusting, reset, and replacement actions, and asks the model to remember object locations and dusted shelves.Source: G PROMPTS FOR GPT-5 / GEMINI… · pp. 22–23