PaperlayerView exact revision on arXiv

Source-linked AI summary

UniMem: Unifying Multimodal Memory and Control for Vision-Language-Action Models

Lars Osterberg, Maggie Wang, Mac Schwager

arXiv:2608.22869v1cs.ROcs.CV

TL;DR

VLAs struggle with non-Markovian tasks because they lack architectural memory, while existing memory approaches can create bottlenecks and fragmented training pipelines. UniMem unifies multimodal memory and control within one VLA backbone, using event-driven updates and visual keyframes, and achieves higher task success with low-latency inference across simulation and hardware.

  • Problem

    VLAs lack architectural memory for non-Markovian tasks requiring sequential and spatial context, while existing memory methods can introduce bottlenecks and fragmented training pipelines.

  • Method

    UniMem unifies multimodal textual and visual memory with control under one VLA backbone, using an event classifier, keyframes, and jointly trained memory updates.

  • Results

    UniMem achieves superior task success across nine simulation and hardware tasks, reaching 93.4% simulation and 80.0% hardware average success rates.

  • Takeaways & Limitations

    The unified system provides a simpler training pipeline and low-latency rollouts while supporting long-horizon spatial and sequential memory.

  • Takeaways & Limitations

    UniMem has not yet been evaluated on tasks extending tens of minutes or hours.

Abstract

from arXiv · show

While Vision-Language-Action (VLA) models have leveraged internet-scale pretraining and task-focused finetuning to achieve strong performance on long-horizon tasks, they often struggle with non-Markovian tasks that require memory. Existing approaches to memory typically involve additional Vision-Language-Models (VLMs) for long-term memory management, introducing a memory bottleneck and a fractured training pipeline. Conditioning on multiple historical frames can provide the VLA with access to more descriptive features of past scenes, but can degrade performance if frames are chosen at arbitrary, fixed intervals. To address these limitations, we present UniMem, a framework that unifies high-level, multimodal memory and low-level control under one backbone. UniMem employs an event classifier for memory updates, a keyframe encoder for dense spatial memory, and a keyframe caching technique to minimize overhead during policy rollouts. We evaluate UniMem across five simulation and four hardware tasks targeting sequential and spatial memory, demonstrating that our unified, single-model system outperforms fixed-interval image sampling baselines (93.4% vs. 68.2%) in simulation and hierarchical baselines (80.0% vs. 43.5%) in hardware, while offering faster inference and a simple training pipeline for easy adoption. Project website: https://losterberg3.github.io/unimem-vla/

I. INTRODUCTION

UNIMEM addresses VLA failures on non-Markovian tasks by unifying event-driven multimodal memory and control under one backbone. It improves success across simulation and hardware while preserving low-latency inference.

  • Motivation: VLA models struggle with non-Markovian tasks because similar observations can require different actions based on preceding events.This perceptual aliasing requires both sequential memory for task progress and spatial memory for scene context.
  • Motivation: Existing memory systems use an additional VLM for textual milestone summaries, isolating the VLA from rich historical context and spatiotemporal video information.The factorized design creates artificial silos and discards information already encoded in past frames.
  • Approach: UNIMEM unifies historical textual events and visual context with memory and control under a shared backbone.Its memory loop is positioned adjacent to the control loop to reduce inference latency.
  • Approach: An event classifier dynamically detects task-critical milestones and constructs task-relevant memory from informative language and vision keyframes.The architecture retains selected history rather than indiscriminately conditioning on all observations.
  • Results: 93.4% simulation success and 80.0% hardware success exceed the best baselines’ 72.6% and 43.5% across nine benchmark tasks.UNIMEM also preserves approximately 90 ms single-frame inference speed and achieves a 6× speedup over hierarchical memory baselines.

II. RELATED WORK

Prior robotic-memory methods use recurrent, sequence-based, hierarchical, or latent-memory designs, but they face limits in horizon, computation, or direct multimodal conditioning. UNIMEM instead unifies event-driven visual and textual memory retrieval with action generation in one backbone.

  • A. Recurrent & Sequence-Based Robotic Memory: Recurrent, state-space, and proprioceptive approaches maintain temporal or implicit memory, but proprioceptive memory applies only when past scene states are unnecessary.These approaches target partial observability but do not uniformly preserve visual scene history.
  • A. Recurrent & Sequence-Based Robotic Memory: Transformer methods ingest frame sequences, but fixed-interval sampling can miss task-critical moments and dense histories create computational overhead and context degradation.The limitation worsens as task horizons grow because observation streams are not selectively retained.
  • B. Hierarchical & Latent Memory: Reactive VLAs condition on the immediate camera frame and instruction, making them effective for short-horizon, fully observable decisions but limited when historical context is required.Their action prediction lacks access to preceding observations.
  • B. Hierarchical & Latent Memory: Hybrid and latent-memory systems construct environment graphs or retain past scene states, while MemER saves relevant keyframes but decouples memory from the low-level actor.These designs provide partial solutions across navigation and tabletop manipulation.
  • C. UNIMEM: UNIMEM uniquely conditions the VLA directly on event-driven visual and textual memory under a single architecture while maintaining low-latency control.This unifies memory retrieval and action generation rather than separating them across hierarchical modules.

A. Problem Formulation and Method Overview

UNIMEM formulates non-Markovian manipulation as a POMDP and resolves perceptual aliasing with online textual-event and visual-keyframe memories. An event classifier in the shared VLA backbone updates both memories and conditions subsequent actions.

  • A. Problem Formulation and Method Overview: The task is modeled as a partially observable, long-horizon manipulation problem in which reactive policies condition on the current observation and instruction.The policy is represented as π(a_t:t+H|o_t, g).
  • A. Problem Formulation and Method Overview: Perceptual aliasing occurs when near-identical observations require different actions because the agent is at different points in task history.Historical context is therefore necessary to resolve otherwise ambiguous states.
  • A. Problem Formulation and Method Overview: UNIMEM maintains textual memory of discrete task events and visual keyframe memory of important spatial information, updating both online through an integrated event classifier.Both histories jointly condition later action prediction.
  • B. Event-Driven Multimodal Memory: The implementation adds an MLP event-classifier head to the final latent representation of a π0.5 VLA with a PaliGemma backbone and Gemma action expert.The classifier predicts among valid milestones and a null event.
  • B. Event-Driven Multimodal Memory: Jointly training the event classifier with the VLA couples memory learning to the shared representation used for action generation.The event loss updates the backbone and prioritizes features supporting both event prediction and control.
  • B. Event-Driven Multimodal Memory: A non-null predicted event is appended as natural language to textual history, modifying the subsequent language instruction and anchoring task progress.Textual history starts empty at each rollout.
  • B. Event-Driven Multimodal Memory: The same event detection stores the corresponding wrist and external multi-view observation as a visual memory update.Keyframes are collected at discrete event timestamps and the current timestep.
  • B. Event-Driven Multimodal Memory: Keyframe history starts with the initial scene, is capped at four frames including the current frame, and drops the oldest frame when the limit is exceeded.Depending on event sparsity, this can provide over a minute of visual memory.

C. Efficient Keyframe Encoding and Caching

UNIMEM combines automated event labeling, event-window keyframe construction, and end-to-end auxiliary supervision to train memory-aware policies efficiently. Keyframe representations are cached so historical visual context can be reused without repeated encoding.

  • Keyframe encoding and caching: UNIMEM caches historical keyframe representations before temporal attention, reusing them while the current image supplies the query.This avoids repeatedly re-encoding raw historical pixels during control steps.
  • Data labeling: UNIMEM labels demonstrations with event classes and textual memory, then samples one keyframe from each past event window to construct visual history.Memory updates occur only after an event window has passed, preventing target leakage during training.
  • End-to-end training: The training objective jointly optimizes the standard action-chunking loss and an auxiliary class-weighted event-classification loss.The event classifier is trained against automatically generated event labels while preserving π0.5’s flow-matching objective.
  • End-to-end training: Null events receive weight 0.02 because they dominate the dataset, while remaining supervised so the classifier learns when not to update memory.The weighting reflects the append-only memory design, where false positives persist in textual and visual memory.
  • End-to-end training: Event-classification gradients flow through the shared language-model trunk rather than a detached feature, coupling memory learning with control.The classifier uses the shared backbone representation, allowing auxiliary supervision to update features used for action generation.

IV. EXPERIMENTS AND ANALYSES

UNIMEM is evaluated on nine memory tasks spanning simulation and real-world hardware, using different robot platforms, camera streams, and retained event-keyframe budgets. The evaluation tests memory-aware control under both environments.

  • Evaluation setup: UNIMEM is evaluated on nine memory tasks divided between simulation and hardware.The simulation platform is a 7DoF Franka Panda in robosuite, while hardware uses a UFactory xArm6.
  • Evaluation setup: The robots use both wrist-mounted and external image streams in simulation and hardware experiments.Hardware inference uses real-time chunking and queries UNIMEM at approximately 10 Hz.
  • Evaluation setup: UNIMEM retains up to three event keyframes for simulation and four for hardware.The retained-memory budgets are |K| = 3 for simulation and |K| = 4 for hardware.

B. Baselines and Ablations

The experiments compare UNIMEM with fixed-interval visual sampling in simulation and a hierarchical memory baseline on hardware, while also isolating textual and keyframe memory contributions. The task suite includes simulated robosuite and real-world xArm6 manipulation tasks.

  • Simulation baseline: The simulation baseline augments π0.5 with a video encoder using frames sampled at 6-second intervals.The baseline uses the keyframe encoder architecture with |K| = 4 to test arbitrary frame sampling against event-driven memory.
  • Hardware baseline: The hardware comparison uses MemER with π0.5 as the primary hierarchical-memory baseline.MemER represents a decoupled memory system for comparison with UNIMEM’s unified architecture.
  • Ablations: Hardware ablations remove memory, retain text memory only, or retain keyframe memory only to isolate each memory input.All ablations receive auxiliary classifier supervision.
  • Task suite: The task overview covers manipulation tasks in robosuite and real-world tasks on the xArm6 setup.These correspond to the simulation and hardware evaluation domains.

C. Simulation Experiments and Ablations

UNIMEM is evaluated on five simulation tasks spanning sequential and spatial memory. It substantially outperforms fixed-interval visual-history baselines, while ablations show complementary roles for textual and visual memory.

  • Overall performance: 93.4% simulation average success exceeds the fixed-window video baseline (π0.5+V.E., 68.2%) across non-Markovian environments.Basic tasks such as UpDown and OccludedTap also verify that history conditioning does not impede low-level control.
  • Sequential memory: Text-only conditioning reaches 93% success on UpDown3Times, whereas keyframe-only conditioning achieves 23%.The result illustrates the value of textual memory for compressing progress into discrete events during counting tasks.
  • Spatial memory: UNIMEM reaches 79% success on UpDownSpatial, compared with 30% for text-only, 49% for video encoding, and 52% for keyframes.Combining textual and visual memory addresses both geometric-awareness and task-progression failures.
  • Spatial memory: In OccludedTap, keyframe ablation performs 12% better than text ablation, while PlateRecall text-only reaches 20%.These results indicate that visual history is particularly useful for spatial disambiguation, whereas text alone can approach chance-level selection.
  • Ablation analysis: Keyframes and the video baseline both match the full model at 96% when discrete spatial ambiguities are easy to resolve.This result shows that visual memory can suffice when progress is readily inferred.

D. Real-world Experiments and Analyses

Real-world experiments test whether UNIMEM overcomes hierarchical memory bottlenecks, benefits from joint textual and visual conditioning, and maintains low inference latency. Across progressively longer and more spatially demanding tasks, the unified model improves success while caching limits overhead.

  • Hierarchical memory: UNIMEM reaches 93.0% success in BeanScoop, preventing high-level command errors that leave hierarchical control unrecoverable.Continuous memory access and near single-frame inference speeds let the robot adjust its trajectory before physical drift becomes uncorrectable.
  • Hierarchical memory: 80% of TableClean trials achieve the correct wipe within 10 cm, compared with 13% for MemER.UNIMEM combines keyframes and textual memory to access the bottle’s exact prior location.
  • Hierarchical memory: UNIMEM achieves 60% success in TapScoopPour, compared with 7% for MemER, by providing visual history directly to the low-level policy.The task requires disambiguating among eight cups after a human selects the target.
  • Memory modalities: Joint textual and visual conditioning reaches 87% in HammerMeasure, versus 13% without memory and 53% with either single modality.The ablations characterize perceptual aliasing in a task where progression must be distinguished from similar observations.
  • Memory modalities: UNIMEM achieves 93% in BeanScoop, while text-only and keyframe-only ablations reach 27% and 20%, respectively.Textual memory supports long-horizon counting, while keyframes help commit individual pour events; their combination provides both short- and long-term memory.
  • Inference efficiency: Caching a 16-keyframe context across four camera streams adds approximately 25 milliseconds beyond the single-frame, two-camera base policy.The mechanism caches precomputed keyframe representations to eliminate redundant vision-backbone computation.

V. CONCLUSION

UNIMEM unifies long-horizon spatial and sequential memory within a single VLA architecture and reports strong performance across simulation and hardware. The authors identify evaluation horizon, memory editing, event labeling, and multiple temporal contexts as directions for further work.

  • UNIMEM unifies long-horizon spatial and sequential memory within a single VLA architecture using an event classifier to update multimodal history.The framework is designed to avoid the memory bottleneck and high latency of dual-system architectures.
  • Across nine tasks in simulation and hardware, UNIMEM demonstrates superior task success, simpler training pipelines, and low-latency real-world rollouts.
  • UNIMEM has not yet been evaluated on extended tasks spanning tens of minutes or hours.The reported robust state tracking covers multi-minute execution horizons.
  • Future work includes memory editing mechanisms such as pruning and consolidation to maintain high performance.
  • The current keyframe extraction pipeline relies on automated offline labeling, while future work could let the policy determine which events warrant long-term retention.
  • UNIMEM currently maintains only one temporal context extending several minutes, leaving short- and long-term memory distinctions for future work.

APPENDIX I DATA CURATION

The data-curation appendix describes a VLM-generated labeling process that converts robot demonstrations into discrete events and textual memory strings. It specifies the input channels, event vocabulary, detection rules, and temporal windows used for annotation.

  • Data collection: Human teleoperated demonstrations are collected at 10 Hz in simulation and 20 Hz on hardware, with labeling generated by a VLM script except for a human tap marker.The human tap in TapScoopPour is flagged during demonstration collection.
  • Inputs and outputs: The labeling script takes one LeRobot parquet per episode with end-effector pose, gripper, and human-event channels.The human event channel marks the single frame where the operator pressed the tap key during teleoperation.
  • Inputs and outputs: Each frame receives a discrete event id from a vocabulary set and a textual memory string.
  • Event vocabulary: The vocabulary contains five once-per-episode events: human tap, grabbed spoon, scooped beans, poured beans, and placed spoon.Frames belonging to no event receive the remaining unlabeled category described by the curation procedure.
  • Event rules: Human tap is detected at the first frame with human event == 1.0, while grabbed spoon is detected at the first post-tap fully closed-gripper plateau.
  • Event rules: Scooped beans is identified using a 40-frame window constrained by roll, yaw, height, and pitch statistics, with the event assigned to the window’s final frame.The supplied criteria include roll standard deviation, mean roll, yaw standard deviation, mean z, and sustained pitch increase.
  • Event rules: The poured-beans event uses the frame of minimum roll, and placed spoon uses the first gripper opening after scooping.
  • Temporal memory: Event labels cover a window from five frames before through twenty frames after detection, while textual memory reveals an event only after its labeled window ends.Before the first event is visible, the memory string is rendered as “History: none.”

APPENDIX II TRAINING DETAILS

Training details report LoRA fine-tuning of several VLA checkpoints, including MemER variants, π0.5, π0.5 +V.E., ablations, and full UNIMEM. The appendix also notes improved performance on BeanScoop and TapScoopPour when a probability component is upweighted.

  • VLA checkpoints are fine-tuned with LoRA, including MemER low-level π0.5, π0.5 +V.E., all ablations, and full UNIMEM.
  • BeanScoop and TapScoopPour show improved performance when the probability component is upweighted.
Loading 2608.22869v1…