PaperlayerView exact revision on arXiv

Source-linked AI summary

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder, Suraj Nair, Brian Ichter, Allen Z. Ren, Haohuan Wang, Jiaming Tang, Kyle Stachowicz, Karan Dhabalia, Michael Equi, Quan Vuong, Jost Tobias Springenberg, Sergey Levine, Chelsea Finn, Danny Driess

arXiv:2603.03596v2cs.ROcs.LG

TL;DR

Real-world multi-stage robotic tasks require both semantic long-term memory and detailed short-term memory, while dense observation histories are difficult to scale. MEM combines compressed video-based short-horizon memory with language-based long-horizon memory, enabling π0.6 policies to perform tasks lasting up to fifteen minutes and adapt manipulation strategies in context.

Abstract

from arXiv · show

Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.

I. INTRODUCTION

MEM addresses the need for robotic memory at multiple abstraction levels by combining dense visual short-horizon memory with compressed language-based long-horizon memory. Integrated into π0.6, it supports complex tasks lasting up to fifteen minutes and state-of-the-art manipulation performance.

I. INTRODUCTION

  • Long-horizon robotic tasks require memories ranging from recent visual events for occlusion handling to semantic events such as completed recipe stages.These memory types differ in the information they must preserve and their useful time scales.Source: §I · pp. 1–2
  • MEM combines a video encoder for compact dense image-based memory with a language mechanism that summarizes semantic events over long time periods.The mixed-modal design uses visual memory for short-term detail and language memory for high-level compression.Source: §I · pp. 1–2
  • The language-based component lets policies retain high-level memories, while short-term memory supports in-context mistake correction and resilience to partial observability and self-occlusion.The paper describes adapting manipulation strategies after errors as one capability enabled by short-term memory.Source: §I · p. 2
  • MEM integrated into π0.6 achieves state-of-the-art performance across diverse complex manipulation tasks.π0.6 is a generalist VLA trained on robot, vision-language, and video data.Source: §I · p. 2
  • MEM enables tasks such as cleaning a whole kitchen or preparing a grilled cheese sandwich that require memories lasting up to fifteen minutes.These examples involve extended, multi-stage manipulation rather than only short-horizon memory.Source: §I · p. 2
  • Prior memory approaches use recurrent modules, dense observation histories, latent representations, or heuristic compression, but no single modality fits all robot-memory needs.The cited approaches trade off long-horizon scalability against precise spatial information needed for manipulation correction.Source: §II · p. 2

III. MULTI-SCALE EMBODIED MEMORY FOR VLAS

MEM factorizes action prediction across time scales: a low-level policy handles recent dense observations, while a high-level policy uses language memory to generate subtasks and preserve semantic history.

III. MULTI-SCALE EMBODIED MEMORY FOR VLAS

  • MEM targets policies that predict action chunks from task goals, dense observation sequences, and language context rather than only a single current observation.The observation sequence can include images and proprioceptive state.Source: A. Multi-Scale Embodied Memory (MEM) · p. 3
  • Conditioning directly on observations spanning minutes is infeasible, motivating a factorization into low-level and high-level policies.The factorization separates fine-grained action modeling from longer-term semantic context.Source: A. Multi-Scale Embodied Memory (MEM) · p. 3, §III · p. 3
  • The low-level policy predicts actions from the goal, a short observation window K ≪ T, and a subtask instruction generated by the high-level policy.The short window preserves recent context without passing the full observation history to the action policy.Source: A. Multi-Scale Embodied Memory (MEM) · p. 3
  • The high-level policy conditions on the goal and language summary m_t to generate both the next subtask instruction and updated language memory m_t+1.This lets semantic history cover many minutes while reducing the number of dense observations required by the low-level policy.Source: A. Multi-Scale Embodied Memory (MEM) · p. 3

B. Language Memory for Long-Term Memory

MEM combines language memory for semantic long-term events with an efficient video encoder for dense short-term observations. The video encoder interleaves spatial and causal-temporal processing, compressing past observations before they reach the VLA backbone.

B. Language Memory for Long-Term Memory

  • Language memory m_t summarizes semantic events from task execution and is updated by the high-level policy using prior memory, current observations, and the task.The policy explicitly decides when and how to update the summary.Source: B. Language Memory for Long-Term Memory · p. 3
  • Language-memory training data is generated by asking a pretrained language model to summarize annotated subtasks together with execution success or failure.The pipeline produces transitions between successive language-memory states.Source: B. Language Memory for Long-Term Memory · p. 3
  • The language memory compresses details when appropriate, retaining sufficient semantic information such as where multiple bowls were placed instead of every bowl’s color.Compression keeps the memory succinct and supports long-term storage.Source: B. Language Memory for Long-Term Memory · p. 3

C. Video Encoder for Dense Short-Term Visual Memory

  • Dense past observations remain necessary for fine-grained details, dynamics, and resolving self-occlusion, but encoding them one by one quickly violates acceptable inference latency.Image encoding is described as a major contributor to VLA computation and inference speed.Source: C. Video Encoder for Dense Short-Term… · pp. 3–4
  • The video encoder extends ViTs by interleaving bidirectional spatial attention within observations with causal-temporal attention across observations.This design processes spatial and temporal context without treating the entire video as one fully jointly attended sequence.Source: C. Video Encoder for Dense Short-Term… · p. 4, Figure 4 · p. 4
  • Its attention complexity decreases from O(n^2K^2) to O(Kn^2 + nK^2), and past-timestep representations are dropped before the subsequent VLA backbone.Dropping past tokens reduces the number of tokens passed downstream.Source: C. Video Encoder for Dense Short-Term… · p. 5
  • The encoder adds no new learnable parameters relative to standard single-image ViTs and can be initialized from pretrained ViT weights.It modifies attention patterns and adds fixed sinusoidal temporal position encoding.Source: C. Video Encoder for Dense Short-Term… · p. 5
  • The video encoder scales observation-based memory to tens of seconds without prohibitive training or inference overhead.It also permits initialization from pretrained vision-language model weights.Source: C. Video Encoder for Dense Short-Term… · p. 5

D. Integrating MEM into the π0.6 VLA

The π0.6-MEM VLA adapts π0.6 to use MEM’s video encoder and trains on heterogeneous robot, vision-language, and video-language data. Its observation memory includes past camera frames and continuously embedded proprioceptive states.

D. Integrating MEM into the π0.6 VLA

  • π0.6-MEM adapts the π0.6 VLA architecture to support MEM’s video encoder and initializes from a pretrained Gemma3-4B vision-language model.The model retains π0.6’s discrete FAST action prediction and flow-matching action expert.Source: D. Integrating MEM into the π0.6 VLA · p. 5
  • Past observation memory includes camera frames and proprioceptive states such as joint angles.Proprioceptive states are projected into the backbone embedding space rather than represented as separate text tokens.Source: D. Integrating MEM into the π0.6 VLA · p. 5
  • Pretraining uses teleoperated demonstrations, policy rollouts, human corrections, vision-language tasks, and video-language tasks such as video captioning.The training mixture follows the broader data strategy used for π0.6.Source: D. Integrating MEM into the π0.6 VLA · p. 5
  • During pretraining, the model uses six observations spaced one second apart; post-training expands observation memory to up to 18 frames and 54 seconds.The longer horizon is accommodated flexibly after pretraining.Source: D. Integrating MEM into the π0.6 VLA · p. 5

IV. EXPERIMENTAL EVALUATION

The evaluation tests whether MEM supports long-term memory across challenging manipulation tasks lasting up to fifteen minutes. Results indicate that combining short-horizon video memory with long-horizon language memory substantially improves performance, while uncompressed language history suffers from distribution shift.

B. In-Context Adaptation of Manipulation Strategies

The study examines whether short-horizon memory enables VLAs to adapt manipulation strategies after failures. MEM uses prior context to incorporate corrections and change behavior during tasks where π0.6 struggles with repeated mis-grasps or uncertain door-opening directions.

C. Analysis Experiments

MEM is evaluated against alternative memory designs across core memory capabilities and dexterous manipulation tasks. It is the only approach reported to perform strongly across all tested memory capabilities while matching non-memory π0.6 performance on challenging dexterous tasks.

  • The evaluation suite tests partial observability, counting, visual memory, and dexterous manipulation across single-arm, dual-arm, and mobile robots.Source: C. Analysis Experiments · p. 7
  • Existing memory approaches improve simpler tasks, but Pool-Memory struggles with longer-term visual memory and Proprio-Memory struggles when environmental state must be remembered.Examples include remembering multiple mug positions, remaining grocery items, or which drawer contains an object.Source: C. Analysis Experiments · p. 8
  • MEM is the only model reported to achieve strong performance across all tested memory capabilities.It handles partial observability and applies memory in dexterous tasks such as unpacking groceries.Source: C. Analysis Experiments · p. 8
  • Pre-training observation-based memory on diverse robot and non-robot data improves memory use even when the memory horizon expands from 5 seconds to up to 1 minute.Introducing memory only during post-training performs noticeably worse.Source: C. Analysis Experiments · p. 8, Figure 9 · p. 8
  • MEM matches the state-of-the-art performance of the non-memory π0.6 VLA on challenging dexterous manipulation tasks.The authors attribute this partly to diverse pre-training data, which can prevent spurious correlations associated with smaller, more uniform robot datasets.Source: C. Analysis Experiments · pp. 8–9, Figure 10 · p. 9

V. CONCLUSION

The conclusion presents MEM as a mixed-modal architecture for long-horizon VLA memory. It supports memory over tens of minutes, in-context manipulation adaptation, and state-of-the-art performance across diverse manipulation tasks.

  • MEM combines short-horizon video memory with long-horizon language memory for robot policies.The conclusion describes this mixed-modal design as the basis for managing memory across multiple time scales.Source: §V · p. 9
  • MEM enables VLAs to perform long-horizon tasks requiring tens of minutes of memory while obeying real-world latency constraints.Source: §V · p. 9
  • MEM enables in-context adaptation of manipulation strategies and achieves state-of-the-art performance across a wide range of manipulation tasks when combined with π0.6.Source: §V · p. 9
  • Future work is proposed to extend memory beyond a single episode to weeks, months, or years of deployment.Source: §V · p. 9

APPENDIX

The appendix illustrates the long-horizon kitchen-task suite used to evaluate MEM. Tasks require retrieving, placing, cleaning, and organizing many items across randomized kitchen locations.

Recipe setup

Cleaning the kitchen

2) In-context adaptation (Section IV-B):

The in-context adaptation evaluation targets manipulation failures caused by unusual geometry or ambiguous object mechanisms. It tests whether correction data can support strategy changes rather than repeated failed attempts.

In-context adaptation

Chopstick Pick Up

Open Refrigerator

3) Analysis Experiments (Section IV-C):

The analysis experiments cover diverse manipulation tasks requiring object tracking, step tracking, precise actions, and memory across partially observed or extended procedures.

Evaluation

  • Task success is evaluated through binary completion criteria or incremental item-level scores tied to correct placement and execution.The three-way swap, table bussing, and grilled cheese tasks provide point-based scoring, while several household tasks use episode success.Source: 3) Analysis Experiments (Section IV-C) · p. 14

Memory-intensive tasks

  • The task suite spans object retrieval, grocery unpacking, coffee scooping, sandwich preparation, and window cleaning, each requiring memory across sequential actions.Examples include remembering a hidden object's drawer, remaining grocery items, completed cleaning steps, or cooking intervals.Source: 3) Analysis Experiments (Section IV-C) · p. 14

Multi-object manipulation

  • Several tasks require tracking multiple objects or destinations, including sorting 12 tabletop objects, swapping three mugs, and moving counter items into storage.These tasks score correct placement across multiple objects or subtasks.Source: 3) Analysis Experiments (Section IV-C) · p. 14

Household manipulation

  • The suite also includes folding clothes, making a bed, cleaning a kitchen counter, and assembling a cardboard box.Success depends on completing the specified physical subtasks or achieving the required final arrangement.Source: 3) Analysis Experiments (Section IV-C) · pp. 14–15

C. Video encoder with Space-Time separable attention

The video encoder modifies a ViT to model temporal context alongside spatial attention while retaining standard transformer projections and computation.

Temporal encoding

Attention projections

  • The encoder reuses the ViT's standard query, key, and value projections, with attention-head indexing and layer normalization applied as in the underlying model.The implementation uses RMSNorm, depending on the starting ViT.Source: C. Video encoder with Space-Time… · p. 15

Attention mechanism

  • Attention is defined as a softmax over query-key interactions, with selectable spatial and temporal index sets determining where attention operates.The formulation supports space-only attention, time-only attention, and joint space-time attention patterns.Source: C. Video encoder with Space-Time… · p. 15

Transformer computation

  • After attention weights combine with values, the outputs pass through an MLP following the standard transformer-layer computation.The video-specific changes therefore occur in the attention pattern and temporal encoding before standard layer processing resumes.Source: C. Video encoder with Space-Time… · p. 15

Related papers

Loading 2603.03596v2…