PaperlayerView exact revision on arXiv

Source-linked AI summary

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder, Suraj Nair, Brian Ichter, Allen Z. Ren, Haohuan Wang, Jiaming Tang, Kyle Stachowicz, Karan Dhabalia, Michael Equi, Quan Vuong, Jost Tobias Springenberg, Sergey Levine, Chelsea Finn, Danny Driess

arXiv:2603.03596v2cs.ROcs.LG

TL;DR

Real-world multi-stage robotic tasks require both semantic long-term memory and detailed short-term memory, while dense observation histories are difficult to scale. MEM combines compressed video-based short-horizon memory with language-based long-horizon memory, enabling π0.6 policies to perform tasks lasting up to fifteen minutes and adapt manipulation strategies in context.

  • Problem

    Complex multi-stage robotic tasks require memory at multiple abstraction levels, but a single dense observation history is difficult to scale and different memory types preserve different information.

  • Method

    MEM combines a video encoder for dense short-horizon observation memory with a language mechanism that summarizes semantic events for long-horizon memory.

  • Results

    MEM enables policies to solve tasks lasting up to fifteen minutes and is the only tested model reported to perform strongly across all tested memory capabilities.

  • Takeaways & Limitations

    With π0.6, MEM achieves state-of-the-art performance across diverse manipulation tasks while supporting long-horizon memory and in-context adaptation.

Abstract

from arXiv · show

Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.

I. INTRODUCTION

MEM addresses the need for robotic memory at multiple abstraction levels by combining dense visual short-horizon memory with compressed language-based long-horizon memory. Integrated into π0.6, it supports complex tasks lasting up to fifteen minutes and state-of-the-art manipulation performance.

  • I. INTRODUCTION: Long-horizon robotic tasks require memories ranging from recent visual events for occlusion handling to semantic events such as completed recipe stages.These memory types differ in the information they must preserve and their useful time scales.
  • I. INTRODUCTION: MEM combines a video encoder for compact dense image-based memory with a language mechanism that summarizes semantic events over long time periods.The mixed-modal design uses visual memory for short-term detail and language memory for high-level compression.
  • I. INTRODUCTION: The language-based component lets policies retain high-level memories, while short-term memory supports in-context mistake correction and resilience to partial observability and self-occlusion.The paper describes adapting manipulation strategies after errors as one capability enabled by short-term memory.
  • I. INTRODUCTION: MEM integrated into π0.6 achieves state-of-the-art performance across diverse complex manipulation tasks.π0.6 is a generalist VLA trained on robot, vision-language, and video data.
  • I. INTRODUCTION: MEM enables tasks such as cleaning a whole kitchen or preparing a grilled cheese sandwich that require memories lasting up to fifteen minutes.These examples involve extended, multi-stage manipulation rather than only short-horizon memory.
  • I. INTRODUCTION: Prior memory approaches use recurrent modules, dense observation histories, latent representations, or heuristic compression, but no single modality fits all robot-memory needs.The cited approaches trade off long-horizon scalability against precise spatial information needed for manipulation correction.

III. MULTI-SCALE EMBODIED MEMORY FOR VLAS

MEM factorizes action prediction across time scales: a low-level policy handles recent dense observations, while a high-level policy uses language memory to generate subtasks and preserve semantic history.

  • III. MULTI-SCALE EMBODIED MEMORY FOR VLAS: MEM targets policies that predict action chunks from task goals, dense observation sequences, and language context rather than only a single current observation.The observation sequence can include images and proprioceptive state.
  • III. MULTI-SCALE EMBODIED MEMORY FOR VLAS: Conditioning directly on observations spanning minutes is infeasible, motivating a factorization into low-level and high-level policies.The factorization separates fine-grained action modeling from longer-term semantic context.
  • III. MULTI-SCALE EMBODIED MEMORY FOR VLAS: The low-level policy predicts actions from the goal, a short observation window K ≪ T, and a subtask instruction generated by the high-level policy.The short window preserves recent context without passing the full observation history to the action policy.
  • III. MULTI-SCALE EMBODIED MEMORY FOR VLAS: The high-level policy conditions on the goal and language summary m_t to generate both the next subtask instruction and updated language memory m_t+1.This lets semantic history cover many minutes while reducing the number of dense observations required by the low-level policy.

B. Language Memory for Long-Term Memory

MEM combines language memory for semantic long-term events with an efficient video encoder for dense short-term observations. The video encoder interleaves spatial and causal-temporal processing, compressing past observations before they reach the VLA backbone.

  • B. Language Memory for Long-Term Memory: Language memory m_t summarizes semantic events from task execution and is updated by the high-level policy using prior memory, current observations, and the task.The policy explicitly decides when and how to update the summary.
  • B. Language Memory for Long-Term Memory: Language-memory training data is generated by asking a pretrained language model to summarize annotated subtasks together with execution success or failure.The pipeline produces transitions between successive language-memory states.
  • B. Language Memory for Long-Term Memory: The language memory compresses details when appropriate, retaining sufficient semantic information such as where multiple bowls were placed instead of every bowl’s color.Compression keeps the memory succinct and supports long-term storage.
  • C. Video Encoder for Dense Short-Term Visual Memory: Dense past observations remain necessary for fine-grained details, dynamics, and resolving self-occlusion, but encoding them one by one quickly violates acceptable inference latency.Image encoding is described as a major contributor to VLA computation and inference speed.
  • C. Video Encoder for Dense Short-Term Visual Memory: The video encoder extends ViTs by interleaving bidirectional spatial attention within observations with causal-temporal attention across observations.This design processes spatial and temporal context without treating the entire video as one fully jointly attended sequence.
  • C. Video Encoder for Dense Short-Term Visual Memory: Its attention complexity decreases from O(n^2K^2) to O(Kn^2 + nK^2), and past-timestep representations are dropped before the subsequent VLA backbone.Dropping past tokens reduces the number of tokens passed downstream.
  • C. Video Encoder for Dense Short-Term Visual Memory: The encoder adds no new learnable parameters relative to standard single-image ViTs and can be initialized from pretrained ViT weights.It modifies attention patterns and adds fixed sinusoidal temporal position encoding.
  • C. Video Encoder for Dense Short-Term Visual Memory: The video encoder scales observation-based memory to tens of seconds without prohibitive training or inference overhead.It also permits initialization from pretrained vision-language model weights.

D. Integrating MEM into the π0.6 VLA

The π0.6-MEM VLA adapts π0.6 to use MEM’s video encoder and trains on heterogeneous robot, vision-language, and video-language data. Its observation memory includes past camera frames and continuously embedded proprioceptive states.

  • D. Integrating MEM into the π0.6 VLA: π0.6-MEM adapts the π0.6 VLA architecture to support MEM’s video encoder and initializes from a pretrained Gemma3-4B vision-language model.The model retains π0.6’s discrete FAST action prediction and flow-matching action expert.
  • D. Integrating MEM into the π0.6 VLA: Past observation memory includes camera frames and proprioceptive states such as joint angles.Proprioceptive states are projected into the backbone embedding space rather than represented as separate text tokens.
  • D. Integrating MEM into the π0.6 VLA: Pretraining uses teleoperated demonstrations, policy rollouts, human corrections, vision-language tasks, and video-language tasks such as video captioning.The training mixture follows the broader data strategy used for π0.6.
  • D. Integrating MEM into the π0.6 VLA: During pretraining, the model uses six observations spaced one second apart; post-training expands observation memory to up to 18 frames and 54 seconds.The longer horizon is accommodated flexibly after pretraining.

IV. EXPERIMENTAL EVALUATION

The evaluation tests whether MEM supports long-term memory across challenging manipulation tasks lasting up to fifteen minutes. Results indicate that combining short-horizon video memory with long-horizon language memory substantially improves performance, while uncompressed language history suffers from distribution shift.

  • The evaluation asks whether MEM supports long-term memory for up to fifteen-minute tasks and improves on prior VLA memory approaches.
  • MEM policies are tested on long-horizon tasks including recipe setup, kitchen cleanup, and grilled cheese preparation.
  • Without memory, even the state-of-the-art generalist policy π0.6 struggles with these challenging long-horizon tasks.
  • MEM provides context across short and long time intervals and significantly increases policy success rate on the evaluated tasks.
  • Both video memory and language memory are essential: video tracks recent task context, while language retains distant semantic events.
  • Naive concatenation of previous language instructions performs worse because repeated failed subtasks create a train-inference distribution shift.MEM instead delays language-memory updates until a subtask succeeds, discarding failed attempts and improving overall performance.

B. In-Context Adaptation of Manipulation Strategies

The study examines whether short-horizon memory enables VLAs to adapt manipulation strategies after failures. MEM uses prior context to incorporate corrections and change behavior during tasks where π0.6 struggles with repeated mis-grasps or uncertain door-opening directions.

  • Short-horizon memory lets policies use previous failed attempts to modify manipulation strategies instead of repeating the same mistake.
  • Without memory, policies cannot recall attempted strategies and therefore remain stuck with a suboptimal approach after mistakes.
  • The evaluation targets chopstick pickup with an out-of-distribution table height and fridge opening with an unknown door direction.
  • Targeted human feedback supplies corrected strategies after failures, while fridge exploration rollouts contain failed attempts followed by corrective demonstrations.
  • MEM-VLA is much more effective at leveraging corrections and adapts its manipulation strategy on the fly.

C. Analysis Experiments

MEM is evaluated against alternative memory designs across core memory capabilities and dexterous manipulation tasks. It is the only approach reported to perform strongly across all tested memory capabilities while matching non-memory π0.6 performance on challenging dexterous tasks.

  • The evaluation suite tests partial observability, counting, visual memory, and dexterous manipulation across single-arm, dual-arm, and mobile robots.
  • Existing memory approaches improve simpler tasks, but Pool-Memory struggles with longer-term visual memory and Proprio-Memory struggles when environmental state must be remembered.Examples include remembering multiple mug positions, remaining grocery items, or which drawer contains an object.
  • MEM is the only model reported to achieve strong performance across all tested memory capabilities.It handles partial observability and applies memory in dexterous tasks such as unpacking groceries.
  • Pre-training observation-based memory on diverse robot and non-robot data improves memory use even when the memory horizon expands from 5 seconds to up to 1 minute.Introducing memory only during post-training performs noticeably worse.
  • MEM matches the state-of-the-art performance of the non-memory π0.6 VLA on challenging dexterous manipulation tasks.The authors attribute this partly to diverse pre-training data, which can prevent spurious correlations associated with smaller, more uniform robot datasets.

V. CONCLUSION

The conclusion presents MEM as a mixed-modal architecture for long-horizon VLA memory. It supports memory over tens of minutes, in-context manipulation adaptation, and state-of-the-art performance across diverse manipulation tasks.

  • MEM combines short-horizon video memory with long-horizon language memory for robot policies.The conclusion describes this mixed-modal design as the basis for managing memory across multiple time scales.
  • MEM enables VLAs to perform long-horizon tasks requiring tens of minutes of memory while obeying real-world latency constraints.
  • MEM enables in-context adaptation of manipulation strategies and achieves state-of-the-art performance across a wide range of manipulation tasks when combined with π0.6.
  • Future work is proposed to extend memory beyond a single episode to weeks, months, or years of deployment.

APPENDIX

The appendix illustrates the long-horizon kitchen-task suite used to evaluate MEM. Tasks require retrieving, placing, cleaning, and organizing many items across randomized kitchen locations.

  • Recipe setup: Recipe setup requires retrieving all ingredients and cookware from randomized kitchen locations and placing them at specified destinations.The prompt can specify items in the fridge, cabinets, drawers, or on the stove, along with their target locations.
  • Recipe setup: Example recipe prompts require moving a pot and lid, retrieving potatoes, butter, milk, and a masher, and placing them at specified countertop or sink locations.
  • Recipe setup: Other recipe prompts require retrieving rice, spam, soy sauce, a pan, and a spatula from cabinets, the fridge, and a drawer.
  • Recipe setup: A further prompt requires retrieving a baking tray, pizza dough, pepperoni, cheese, and a dough roller and placing them on the countertop.
  • Cleaning the kitchen: Kitchen cleaning requires wiping and drying the countertop, storing food, organizing dishes, and washing dishes from the sink.Episodes average eight subtasks, with one point awarded per completed subtask.
  • Cleaning the kitchen: An example cleaning prompt combines countertop wiping and drying, towel disposal, mustard storage, dish placement, and washing two plates.

2) In-context adaptation (Section IV-B):

The in-context adaptation evaluation targets manipulation failures caused by unusual geometry or ambiguous object mechanisms. It tests whether correction data can support strategy changes rather than repeated failed attempts.

  • In-context adaptation: The chopstick task uses an out-of-distribution low table height that causes frequent mis-grasps, while the refrigerator task has an unclear door-opening direction.
  • Chopstick Pick Up: Chopstick success requires both picking up the chopstick and placing it in the bin, for a total success score of 2.
  • Open Refrigerator: Refrigerator episodes count as successful when the door opens within 4 grasps, testing intentional strategy switching rather than repeated random sampling.

3) Analysis Experiments (Section IV-C):

The analysis experiments cover diverse manipulation tasks requiring object tracking, step tracking, precise actions, and memory across partially observed or extended procedures.

  • Evaluation: Task success is evaluated through binary completion criteria or incremental item-level scores tied to correct placement and execution.The three-way swap, table bussing, and grilled cheese tasks provide point-based scoring, while several household tasks use episode success.
  • Memory-intensive tasks: The task suite spans object retrieval, grocery unpacking, coffee scooping, sandwich preparation, and window cleaning, each requiring memory across sequential actions.Examples include remembering a hidden object's drawer, remaining grocery items, completed cleaning steps, or cooking intervals.
  • Multi-object manipulation: Several tasks require tracking multiple objects or destinations, including sorting 12 tabletop objects, swapping three mugs, and moving counter items into storage.These tasks score correct placement across multiple objects or subtasks.
  • Household manipulation: The suite also includes folding clothes, making a bed, cleaning a kitchen counter, and assembling a cardboard box.Success depends on completing the specified physical subtasks or achieving the required final arrangement.

C. Video encoder with Space-Time separable attention

The video encoder modifies a ViT to model temporal context alongside spatial attention while retaining standard transformer projections and computation.

  • Temporal encoding: Input embeddings for spatial patch p and timestep t are augmented with a sinusoidal temporal position embedding, with e(0) = 0.The timestep range is t ∈ [-K, 0].
  • Attention projections: The encoder reuses the ViT's standard query, key, and value projections, with attention-head indexing and layer normalization applied as in the underlying model.The implementation uses RMSNorm, depending on the starting ViT.
  • Attention mechanism: Attention is defined as a softmax over query-key interactions, with selectable spatial and temporal index sets determining where attention operates.The formulation supports space-only attention, time-only attention, and joint space-time attention patterns.
  • Transformer computation: After attention weights combine with values, the outputs pass through an MLP following the standard transformer-layer computation.The video-specific changes therefore occur in the attention pattern and temporal encoding before standard layer processing resumes.
Loading 2603.03596v2…