PaperlayerView exact revision on arXiv

Source-linked AI summary

Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation

Yun Zhang, Zhiyu Huang, Tianhui Cai, Chen Tang, Wei Wang, Jiaqi Ma

arXiv:2610.22323v1cs.RO

TL;DR

Long-horizon manipulation needs memory for information that may disappear from current observations, while existing ways of retaining history can expose long contexts or lose task-relevant details. Mem4Act learns fixed-size latent memory through temporal aggregation, complementary semantic and perceptual supervision, and memory-first training. It reports improved results across RoboMME, RMBench, and real-world manipulation tasks.

  • Problem

    Long-horizon manipulation requires retaining past information that may disappear from current observations, while existing history representations can burden policy context or omit later-relevant details.

    Source: §1 · p. 1

  • Method

    Mem4Act combines multi-granularity temporal attention, semantic QA and key-frame feature reconstruction, and memory-first two-stage training for fixed-size latent memory.

    Source: §1 · pp. 2–3

  • Results

    Across 29 manipulation tasks, Mem4Act achieves 49.88% on RoboMME, 61.0% on RMBench, and 81.50% on real-world tasks, outperforming strong baselines.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Mem4Act’s fixed-size memory enables the policy to preserve and exploit task-relevant historical information across diverse memory requirements.

    Source: §5 · p. 10

Abstract

from arXiv · show

Long-horizon robotic manipulation fundamentally requires memory: agents must retain task-relevant information from past observations and interactions and use it to guide future actions. Existing approaches typically preserve raw historical observations, compressed visual tokens, or language summaries, but these representations either expose the action policy to increasingly long histories or risk discarding task-relevant temporal, semantic, and perceptual information. We introduce Mem4Act, a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control. Mem4Act addresses three coupled challenges: how to efficiently cover long histories, what information memory should retain, and how to ensure the policy actually relies on memory. To this end, it organizes history via multi-granularity temporal aggregation, preserves complementary semantic and perceptual information through question-answering supervision and key-frame DINO feature reconstruction, and employs memory-first two-stage training to encourage greater use of memory by the action policy. The resulting memory captures both high-level task state and fine-grained perceptual evidence while providing a bounded interface to the action policy. Across RoboMME, RMBench, and four real-world manipulation tasks, Mem4Act achieves state-of-the-art performance on memory-dependent control, with success rates of 49.88%, 61.0%, and 81.5%, respectively. Extensive ablations further demonstrate the importance of temporal aggregation, complementary memory supervision, and memory-first training.

1 INTRODUCTION

Long-horizon manipulation requires retaining information that may disappear from current observations. Mem4Act learns fixed-size latent memory to address history coverage, retained content, and policy use, and reports gains across simulation and real-world tasks.

1 INTRODUCTION

  • Long-horizon tasks need past information because decision-critical details can disappear from the current observation before they are needed.For example, block colors observed early can become hidden after covering.Source: §1 · p. 1
  • Directly exposing full history increases policy context and computational costs, while selected frames, compressed tokens, or language summaries may omit later-relevant information.Mem4Act defines memory as a compact, fixed-capacity latent state updated throughout the episode, distinguishing it from historical context explicitly exposed to the policy.Source: §1 · p. 1
  • Mem4Act uses multi-granularity temporal attention for coverage, semantic QA and key-frame feature reconstruction for content, and memory-first training for utilization.The three components respectively organize memory across temporal regions, supervise task-relevant facts and perceptual information, and establish memory representations before action learning.Source: §1 · pp. 2–3
  • 49.88%, 61.0%, and 81.5% success rates on RoboMME, RMBench, and real-world tasks, respectively, outperform the strongest evaluated baselines.The evaluations cover 29 tasks, while the action policy receives a fixed-size interface of 512 latent memory tokens.Source: §1 · p. 2

2 RELATED WORK

Prior VLA policies often rely on current observations or short contexts, while memory methods vary in how they retain and expose history. Mem4Act is presented against this landscape as a fixed-size memory pipeline for long-horizon control.

2 RELATED WORK

  • Most VLA policies rely on current observations or short fixed contexts, limiting action when relevant information is occluded or temporally unavailable.The related-work discussion also describes VLA applications in manipulation, navigation, and autonomous driving.Source: §2 · p. 3
  • RoboMME spans diverse memory forms, whereas RMBench varies whether a task requires recalling one past observation or multiple observations.The paper evaluates on these benchmarks as complementary tests of memory capability.Source: §2 · p. 3
  • Memory-augmented VLA approaches retain recent windows, selected or compressed records, or bounded banks and recurrent states, each with stated coverage or detail trade-offs.Mem4Act instead allocates fixed-size memory slots across temporal regions before integrating information over wider windows.Source: §2 · p. 3
  • Figure 2 depicts a pipeline from encoded historical observations through temporal aggregation and memory supervision to memory-conditioned action generation and staged training.The figure also shows state, phase, and time embeddings in the historical observation encoding.Source: Figure 2 · p. 4

3 METHOD

Mem4Act turns historical observations into a fixed-size latent state intended to cover long histories, retain task-relevant information, and condition action generation. Its method combines temporally structured memory, complementary supervision, memory modulation, and two-stage training.

3.1 PROBLEM FORMULATION

  • The problem formulation represents history with S latent tokens of dimension d and conditions action prediction on current observation, instruction, and memory.The stated objective is effective historical coverage, retention of task-relevant information, and reliable policy use of memory.Source: §3.1 · p. 4

3.2 STRUCTURED MEMORY FORMATION

  • Historical frames are encoded with visual, state, relative-time, and execution-phase information, then arranged into a history sequence for memory formation.Memory tokens attend within local temporal windows before progressively integrating broader windows; final global self-attention produces fixed-size memory.Source: §3.2 · pp. 4–5

3.3 MEMORY-SPECIFIC SUPERVISION

  • Memory supervision combines questions derived from subgoal annotations with reconstruction of DINO features from selected past keyframes.The QA prompts and answers supervise memory through answer-token likelihood; keyframe feature reconstruction encourages retention of fine-grained visual content.Source: §3.3 · p. 5

3.4 MEMORY-CONDITIONED ACTION POLICY

  • At each action-expert layer, action hidden states attend to memory, whose projected features modulate the feed-forward branch while preserving current-observation and language processing.The policy is trained with a flow-matching objective to predict action chunks.Source: §3.4 · p. 6

3.5 MEMORY-FIRST TWO-STAGE TRAINING

  • Stage 1 trains memory components with auxiliary supervision while the pretrained VLA backbone is frozen; Stage 2 jointly trains policy and memory components.Stage 2 retains memory objectives and uses a 0.1× learning rate for the memory encoder to limit disruption to its pretrained representation.Source: §3.5 · p. 6

4 EXPERIMENTS

Mem4Act is evaluated on RoboMME, RMBench, and real-world manipulation tasks, with one shared policy across RMBench tasks. It achieves the strongest overall results reported across these settings, while ablations highlight the roles of temporal organization, complementary supervision, and memory adaptation.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers 16 single-arm RoboMME tasks, nine dual-arm RMBench tasks, and four real-world dual-arm tasks.Real-world policies are trained on 70 demonstrations per task and evaluated over 50 trials; Mem4Act uses 10k Stage-1 and 30k Stage-2 steps.Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2 MAIN RESULTS

  • 49.88% success across 16 RoboMME tasks gives Mem4Act the best overall score, 5.37 percentage points above FrameSamp+Modul.It scores highest on Counting, Reference, and Imitation, while MemER leads on Permanence at 53.17%.Source: §4.2 · p. 7, Table 1 · p. 7
  • 61.0% overall success across nine RMBench tasks exceeds NativeMEM by 12.3 percentage points with one policy trained across all tasks.Mem4Act reaches 41.3% versus NativeMEM's 14.8% on longer M(n) tasks, compared with 76.8% versus 75.8% on M(1) tasks.Source: §4.2 · p. 7, Table 2 · p. 8
  • 81.50% average real-world success across tasks exceeds GPT-6 Astra-xhigh at 42.50% and FrameSamp+Modul at 42.00%.Real-world episodes have a median length of 982 frames, versus 430 in RoboMME; under the same 512-token budget, FrameSamp+Modul can use up to 32 images.Source: §4.2 · pp. 7–8, Table 3 · p. 9

4.3 ABLATION STUDIES

  • At 512 tokens, 32 →16 →8 temporal windows achieve 49.88% success, versus 36.88% for repeated 32 →32 →32 aggregation.With the 32 →16 →8 hierarchy, increasing capacity from 128 to 256 and 512 tokens raises success from 40.13% to 44.25% and 49.88%, while 1024 tokens yields 42.75%.Source: §4.3 · pp. 8–9, Table 4 · p. 9
  • QA supervision performs better on semantic tasks (58.80% vs. 55.60%), while DINO feature training performs better on perceptual tasks (50.67% vs. 45.33%).Combining both gives the best joint-task performance (37.60%) and overall success (49.88%); under one-stage training, adding QA and DINO raises success from 31.50% to 33.38%.Source: §4.3 · p. 9, Table 5 · p. 9
  • Updating pretrained memory for control improves performance: 0.1× learning-rate adaptation reaches 49.88%, versus 33.38% for one-stage training and 43.25% for full-rate adaptation.Freezing memory yields 18.38%; both adapted two-stage variants show larger drops under mismatched history than one-stage training, with the 0.1× variant showing the largest drop.Source: §4.3 · pp. 9–10, Table 5 · p. 9

5 CONCLUSIONS

Mem4Act learns compact structured memory for long-horizon manipulation, combining temporal aggregation, complementary supervision, and memory-first training; it outperforms strong baselines across 29 manipulation tasks.

5 CONCLUSIONS

  • 49.88% on RoboMME, 61.0% on RMBench, and 81.50% on real-world tasks, with Mem4Act outperforming strong baselines across diverse memory requirements.The evaluations span 29 manipulation tasks.Source: §5 · p. 10
  • Mem4Act addresses historical coverage, memory content, and utilization through temporal aggregation, semantic QA and perceptual feature reconstruction, and memory-first two-stage training.The resulting memory is fixed-size and intended to preserve and exploit task-relevant historical information.Source: §5 · p. 10
  • Ablations show that hierarchical temporal structure improves historical coverage, semantic and perceptual supervision provide complementary benefits, and memory-first training strengthens reliance on rollout-specific information.The paper concludes that explicitly learning and utilizing structured memory matters for long-horizon robotic control.Source: §5 · p. 10

Mem4Act Supplementary Material

The supplement details Mem4Act’s visual and memory inputs, temporal hierarchy, auxiliary supervision, keyframe reconstruction, memory-conditioned policy, and two-stage optimization.

Base VLA and visual inputs

  • The policy uses π0.5 with a PaliGemma VLM and Gemma-2B language backbone, a Gemma-300M action expert, and a shared SigLIP So400m/14 vision encoder.Policy and memory images are resized to 224×224; the supplement specifies current-view camera inputs and the action chunks and parameterization used for RoboMME and RMBench.Source: A MODEL DETAILS · pp. 14, 16

History construction

  • The memory encoder samples third-person history at fixed intervals, includes initial and current frames, and uses 16 projected visual tokens per frame with state, time, and source embeddings.The history uses 164 sampled frames for RoboMME and 128 for RMBench and real-world evaluation; the stated capacities avoid truncation during training and evaluation.Source: A MODEL DETAILS · p. 14

Temporal windowing

  • The three-layer temporal hierarchy partitions history into 32, 16, and 8 windows, then globally exchanges information across memory slots.When fewer frames than windows are available, adjacent groups may read the same frame so no window is empty.Source: A MODEL DETAILS · p. 14

Auxiliary supervision construction

  • Training targets are generated automatically from instructions and subgoal annotations; QA questions respect episode phase, and reconstruction uses only observed initial or subgoal-boundary keyframes.Subgoal annotations and auxiliary targets are training-only, and the procedure requires no manual QA labels or additional keyframe labeling beyond subgoal annotations. Task-specific rules track progress in BinFill and PatternLock; QA movement directions use the robot’s egocentric frame.Source: A MODEL DETAILS · pp. 14–15, Figure S1 · p. 15

Keyframe reconstruction

  • Keyframe reconstruction masks unobserved frames and unused query slots, with frozen DINOv3 patch features as targets and learned queries carrying relative-time encodings.The decoder uses two Perceiver layers to attend to 512 memory slots and reconstruct dense features with rank-16 coefficients and basis features, plus a shared spatial bias.Source: A MODEL DETAILS · p. 15

Memory-conditioned action generation

  • At every action-expert layer, action states query 512 memory slots, whose retrieved features modulate the feed-forward branch through scale and shift.The policy combines that history with its standard language-instruction and current multi-view observation conditioning.Source: A MODEL DETAILS · pp. 15–16

Training and parameter freezing

  • Stage 1 trains the memory encoder, projection, and DINO decoder while freezing the language model, visual encoder, and action expert; Stage 2 jointly trains the language model, action expert, memory components, and decoder.In Stage 2 the memory encoder uses a 0.1× learning-rate multiplier; the shared SigLIP encoder remains frozen, and QA and DINO losses each have weight 0.05 while action loss has unit weight.Source: A MODEL DETAILS · p. 16

B BENCHMARK DETAILS

The supplement describes RoboMME and RMBench as complementary long-horizon manipulation benchmarks and specifies their task groupings, observations, and evaluation protocols.

B.1 ROBOMME

  • RoboMME covers 16 non-Markovian single-arm tasks spanning event tracking, occlusion or hidden movement, object-reference maintenance, and imitation of demonstrations.The robot receives front and wrist RGB views and proprioception; imitation and Video tasks also provide a demonstration before execution.Source: §B.1 · p. 17
  • RoboMME uses 100 demonstrations per task, joint training across all 16 tasks, and evaluation on 50 held-out environment seeds per task.Its maximum horizon is 1,300 simulator steps, and success follows each task’s completion criterion.Source: §B.1 · p. 17

B.2 ROBOMME TASK GROUPS FOR AUXILIARY-SUPERVISION ANALYSIS

  • RoboMME’s auxiliary-supervision analysis groups tasks by whether success requires semantic history, perceptual history, or both; models remain jointly trained on all 16 tasks.The task categories are listed in Table S3.Source: §B.2 · p. 17, Table S3 · p. 18

B.3 RMBENCH

  • RMBench covers nine long-horizon dual-arm tasks, distinguishing M(1) tasks needing one observation or a small fixed state set from M(n) tasks needing information accumulated across interactions or outcomes.It provides head and wrist-camera observations and proprioception; the protocol uses 50 synthesized expert demonstrations per task and 100 rollouts per task, requiring completion of the full sequence for success.Source: §B.3 · p. 17

B.4 REAL-WORLD EVALUATION

The supplement describes a four-task real-world dual-arm evaluation, its data and success protocol, baseline comparisons, and GPT-6’s direct-control setup.

B.4 REAL-WORLD EVALUATION

  • The real-world benchmark adapts four memory-dependent RMBench tasks on a YAM dual-arm platform, using 70 teleoperated demonstrations per task and 280 trajectories total.Memory processes head-view history while the policy receives current head and wrist views; about 10% of training data includes unlabeled failure-recovery behavior.Source: §B.4 · p. 18
  • Each real-world task uses one fixed instruction, randomized initial configurations, and 50 evaluation trials; success requires completing the full instruction in order.An operator assigns binary outcomes, and overall success is the mean across four tasks.Source: §B.4 · pp. 18–19, Table S4 · p. 18

C.1 COMPARISON PROTOCOL

  • Mem4Act and π0.5-based MME-VLA RoboMME comparisons use 80,000 optimization steps, while Mem4Act trains one joint policy per benchmark and follows official evaluation protocols.Other methods and reproduced RMBench baselines follow their original task-specific training protocols; the comparison uses published results except for specified RMBench reproductions.Source: §C.1 · p. 19

C.2 ROBOMME BASELINES

  • The baselines span short-window or current-observation policies, visual-history memory, VLM-mediated keyframe memory, recurrent memory, and hierarchical or compressed visual memory.Examples include SAM2Act+, TokenDrop and FrameSamp, MemER, MME-VLA TTT+Expert, Mem-0, HAMLET, and NativeMEM.Source: §C.2 · p. 19, §C.3 · p. 20

C.4 REAL-WORLD BASELINES

  • The real-world comparison trains each learned method jointly across four tasks on 280 demonstrations, with shared current observations and action interfaces; GPT-6 Astra is evaluated zero-shot without access to that training data.GPT-6 receives the current three views and issues one Cartesian end-effector pose target per decision, limited relative to the measured pose and executed for at most 15 control ticks.Source: §C.4 · p. 20

C.5 MEMORY-TRAINING ABLATION DETAILS

The ablations compare memory-supervision losses and memory-update schedules under a shared 80,000-step RoboMME setup. Two-stage variants first train memory with auxiliary losses, then combine them with action training.

C.5 MEMORY-TRAINING ABLATION DETAILS

  • All Table 5 variants use 80,000 steps and share architecture, data, history sampling, action horizon, and evaluation protocol.The effective global batch size is 64.Source: §C.5 · p. 21
  • In the supervision ablation, two-stage variants spend 20,000 steps on selected QA and/or DINO losses, then 60,000 steps combining those losses with action training.Auxiliary-loss weights are 1.0 in Stage 1 and 0.05 in Stage 2; the action loss weight is 1.0 in Stage 2.Source: §C.5 · p. 21
  • The optimization ablation holds the two-stage loss schedule fixed while comparing a frozen memory module, a 0.1× Stage 2 update, and a full-rate update.The one-stage comparator uses action, QA, and DINO losses throughout, with the memory learning-rate multiplier fixed at 1.0.Source: §C.5 · p. 21
  • Two-stage schedules restart the learning-rate schedule at Stage 2, and the pretrained VLA backbone remains frozen during Stage 1.Stage 2 initializes from the pretrained backbone and Stage 1 memory encoder, projection, and DINO decoder parameters.Source: §C.5 · p. 21

D.1 COMPARING SIMULATION AND REAL-WORLD PERFORMANCE GAPS

Mem4Act’s advantage over evaluated baselines is larger in the real-world comparisons than in simulation, but the comparisons span different tasks and separately trained policies. The discussion notes design differences that limit direct interpretation of those margins.

D.1 COMPARING SIMULATION AND REAL-WORLD PERFORMANCE GAPS

  • Mem4Act’s margin over π0.5 is 31.95 points on RoboMME and 53.50 points in the real world.The passage cautions that these are comparisons across different tasks and separately trained policies, not a controlled measure of transfer.Source: §D.1 · p. 21
  • Mem4Act’s margin over Mem-0 is 19.0 points on RMBench and 59.00 points in the real world.Mem-0’s executor is trained separately for each task, while the real-world Mem4Act implementation learns all four tasks jointly.Source: §D.1 · p. 21
  • Against MME-VLA-FrameSamp+Modul, Mem4Act’s margin is 5.37 points on RoboMME and 39.50 points in the real world.The comparison may reflect both broader temporal coverage and more useful memory content, but does not isolate their contributions.Source: §D.1 · p. 21
  • MME-VLA-FrameSamp+Modul retains at most 32 uniformly sampled images, while real-world episodes have a median length of 982 frames.As episodes grow, the fixed image set becomes more widely spaced and may miss intermediate events.Source: §D.1 · p. 21

D.2 REAL-WORLD INFERENCE EFFICIENCY

Mem4Act provides a fixed-size history interface, with history processing and action prediction contributing separately to policy-cycle latency. Its real-world comparisons show a trade-off between cycle latency and success, alongside similar-cost and higher-cost baselines.

D.2 REAL-WORLD INFERENCE EFFICIENCY

  • The efficiency evaluation uses batch size one on an NVIDIA RTX A6000, and FLOPs and latency include visual encoding, history processing, and action-chunk generation.Peak memory excludes framework-reserved memory.Source: §D.2 · p. 22
  • A policy cycle comprises history processing and action prediction; Mem4Act’s 512-slot interface keeps action-policy input length fixed as history grows.History processing includes visual encoding and memory updates; action prediction produces the next action chunk.Source: §D.2 · p. 22
  • 81.50% real-world success for Mem4Act contrasts with 28.00% for π0.5, while policy-cycle latency rises from 131 to 335 ms.Successful rollouts average 40 seconds for Mem4Act and 47 seconds for π0.5; the added per-cycle latency does not increase task-completion time.Source: §D.2 · p. 22
  • Mem4Act reaches a 39.50-percentage-point higher success rate than MME-VLA (FrameSamp+Modul) at nearly identical latency, 335 versus 340 ms.Peak memory is 7.2 versus 7.0 GiB, respectively.Source: §D.2 · p. 22
  • Mem-0 uses 4,923 versus 489 G FLOPs per cycle while achieving 22.50% success; its reported cost excludes its optional high-level planner.The passage attributes the cost to repeatedly applying Qwen3-VL-2B to newly observed history frames before compressing features into memory.Source: §D.2 · p. 22
  • GPT-6 Astra with xhigh reasoning takes approximately 19 seconds per policy cycle and 17 minutes per successful rollout, versus 335 ms and 40 seconds for Mem4Act.Source: §D.2 · p. 22

D.3 REAL-WORLD FAILURE ANALYSIS

The failure analysis assigns each unsuccessful rollout its earliest outcome-determining failure, making failure categories competing events. Mem4Act has substantially fewer failures than the baselines, while their failure profiles differ.

D.3 REAL-WORLD FAILURE ANALYSIS

  • The analysis manually inspects unsuccessful real-world rollouts and assigns each exactly one primary observable failure mode.The failure table reports counts of failed rollouts by category.Source: §D.3 · p. 22, Table S6 · p. 23
  • Failure categories are competing events because assigning the earliest outcome-determining failure prevents later failures from being observed.The total failure count is the most directly comparable quantity across methods; individual categories describe how each method fails.Source: §D.3 · p. 23
  • Mem4Act has 37 failures, corresponding to 81.50% success: 34 incorrect behaviors and three non-recoverable states, with no stalls.The authors characterize unsuccessful Mem4Act trials as usually maintaining stable execution, with remaining failures concentrated in challenging task-level decisions.Source: §D.3 · p. 23
  • GPT-6 Astra-xhigh records 10 incorrect-behavior failures, but 73 non-recoverable states, 20 timeouts, and 12 stalls occur earlier in other rollouts.Its timeout count reflects the 80-decision or 30-minute evaluation limit; earlier failures prevent subsequent task-level errors from being observed.Source: §D.3 · p. 23
  • π0.5 has 70 incorrect behaviors and 62 stalls, MME-VLA (FrameSamp+Modul) has 89 incorrect behaviors, and Mem-0 has 155 failures.Mem-0’s failures include 57 non-recoverable states and 49 stalls.Source: §D.3 · p. 23

Related papers

Loading 2610.22323v1…