Source-linked AI summary
Memory before Goal: Role of Pre-History Memory for Humanoid Loco-Manipulation
Chenyuan Chen, Ran Ding, Lechen Zhang, Zewen He, Yoshihiko Nakamura
TL;DR
In dynamic humanoid loco-manipulation, cues can disappear or leave the camera view before decisions, while VLA policies see only the present. WholeBodyMem selects object-event frames and object tokens for an adaptive memory, and HuMeGo evaluates memory-dependent humanoid tasks. It improves results over memoryless and uniform-memory baselines, including a reported simulation success rate of 72.6% versus 29.1% for Ψ0.
Problem
Whole-body VLA policies typically act on current observations, leaving them without earlier cues that vanish, are hidden, or leave view before a decision.
Source: §I · p. 1Method
WholeBodyMem uses SAM3-tracked object events to select key frames within an adaptive budget, which the action expert reads together with object tokens.
Source: §I · pp. 1–2Results
72.6% success versus 29.1% for Ψ0 and 44.4% for Uniform memory on memory-dependent tasks; WholeBodyMem also reaches 86.9% versus 70.2% for Ψ0 on SIMPLE.
Source: C. Results on Memory-Dependent Tasks… · p. 7, B. Results on Standard… · p. 7Takeaways & Limitations
Keeping event-relevant frames supports decisions about vanished cues in simulation and on a real G1 without losing reactive skills.
Source: §VI · p. 8SAM3 can miss cues when their objects are never detected, online tracking adds inference cost, simulation leaves a sim-to-real gap, and simulated demonstrations are scripted.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Humanoid loco-manipulation in dynamic scenes is non-Markovian: cues vanish as objects change or are hidden, or leave the view as the robot turns and walks, yet whole-body vision-language-action (VLA) policies see only the present. We present WholeBodyMem, an object-guided perceptual memory that keeps the pre-history of object events seen before the robot acts on its goal. SAM3 tracks the task objects, and their events causally select key frames within an adaptive budget, which the action expert reads by cross-attention together with object tokens encoding appearance, geometry and time. We also contribute HuMeGo, a humanoid memory benchmark with six simulated and six real-world Unitree G1 tasks. Keeping the right frames rather than more frames, WholeBodyMem surpasses memoryless and uniform-memory baselines, raising its base policy's success from 29.1% to 72.6% in simulation and from 36.7% to 88.3% on the real robot, while also improving the standard SIMPLE tasks (70.2% to 86.9%).
I. INTRODUCTION
Whole-body VLA policies must act on brief object events that may disappear or leave view before a decision. WholeBodyMem selects event-relevant frames and object tokens, and HuMeGo evaluates memory-dependent humanoid tasks in simulation and on a real G1.
I. INTRODUCTION
- Whole-body loco-manipulation is non-Markovian when earlier object events determine decisions after cues vanish, are hidden, or leave the moving camera’s view.The introduction calls the record of object events seen before acting on a goal “pre-history memory.”Source: §I · p. 1
- WholeBodyMem’s object events select key frames within an adaptive budget, which the action expert reads with object tokens.The method uses SAM3 to track task objects and detect changes, then reads memory through cross-attention and FiLM.Source: §I · pp. 1–2
- HuMeGo includes six simulated tasks and six real-world G1 tasks, with scene-ending cues and decisions scored apart from manipulation in simulation.The real-world tasks also require the robot to turn away from the cue.Source: §I · p. 2
- WholeBodyMem is reported to outperform memoryless and uniform-memory baselines, help on SIMPLE, and carry over to a real G1.The motivating real-task example shows the seats looking identical when the robot returns, despite the earlier cola placement cue.Source: Figure 1 · p. 1, §I · p. 2
II. RELATED WORK
Prior whole-body humanoid VLA work conditions actions on current observations and short proprioceptive histories, while robot memory work has focused mainly on tabletop and navigation settings. WholeBodyMem selects past views by object events to address cues displaced by humanoid motion, and the paper introduces a humanoid benchmark for decisions after cues leave view.
A. Whole-Body Humanoid Loco-Manipulation VLAs
- Whole-body VLA policies share the limitation that events leaving camera view can no longer affect actions; the paper contrasts WholeBodyMem’s past views with POT-VLA’s record of current object locations [27].The related-work discussion also notes prior memory work for wheeled or head-camera robots, but not whole-body loco-manipulation.Source: A. Whole-Body Humanoid… · p. 2
B. Memory and Datasets for Robot Policies
- Robot-policy memory has mainly been studied for tabletop arms and mobile or navigation agents, using long histories, compression, or fixed-budget frame selection [8], [9],,,, [13],,,,.Long histories can dilute short cues, while compressed representations can lose decision-relevant detail.Source: B. Memory and Datasets for Robot… · p. 2
- Humanoid memory must handle cues moved out of view by walking or turning and update causally during whole-body control; WholeBodyMem selects frames using object events and reads them with action-query cross-attention and FiLM.The related-work discussion distinguishes SAM3 as a tracker from SAM2Act+’s use of SAM2’s memory mechanism.Source: B. Memory and Datasets for Robot… · p. 3
- Existing memory benchmarks are mostly tabletop, while humanoid benchmarks evaluate skills with the goal in view; the proposed benchmark tests decisions after cues leave view, –[21].The paper also contrasts these benchmark settings with memory benchmarks and humanoid imitation benchmarks on the same platform and control frequency.Source: B. Memory and Datasets for Robot… · p. 3
III. MEMORY-DEPENDENT LOCO-MANIPULATION BENCHMARK
HuMeGo defines memory-dependent humanoid tasks around cues that no longer determine the correct action from the decision-time observation. It specifies six simulated tasks and describes how demonstrations and train/test scenes are generated.
A. Task Definition
- A task is memory-dependent when the decision-time observation cannot determine the correct action, and the instruction and pre-decision motion are answer-independent while answers are balanced.The cue becomes unobservable through scene changes or robot motion; memory-decision accuracy is scored separately from manipulation success.Source: A. Task Definition · p. 3
- The six simulated tasks cover object, spatial, and temporal memory, while the real-world tasks additionally introduce motion-induced cue loss.Simulated cues include flashes, hidden or swapped objects, and brief blinks; the real-world suite includes the robot turning away from cues.Source: A. Task Definition · p. 3
B. Data Generation
- A scripted whole-body expert generates demonstrations in MuJoCo, with task-specific cue timing and randomized scene parameters.Episodes are retained when the cue is visible during its window and, for carrying tasks, the object remains held after lifting.Source: B. Data Generation · p. 3
- Each memory task has 100 training demonstrations and 90 test scenes from distinct seeds, balanced across answers and three randomization levels.Test scenes remain within the training distribution for layout, rollout, and table materials.Source: B. Data Generation · p. 4
C. Data Analysis
HuMeGo pairs long humanoid episodes with decisions that require remembering cues after they disappear. This tests a gap for whole-body VLA policies that condition on current observations and cannot respond to events no longer in view.
C. Data Analysis
- 1,155 control steps (23 s) is the average HuMeGo episode length, versus 818, 611, and 322 steps in HumanoidMimicGen, HumanoidArena, and SIMPLE, respectively.HuMeGo decisions require memory of a short-lived cue occupying 0.8–14.7% of an episode and remembered for 13.5–22.5 s after disappearance, using task medians.Source: C. Data Analysis · p. 4
- 40.7% of HuMeGo steps occur after the decision point, when the vanished cue is the sole evidence for the correct action.No step of the humanoid benchmarks depends on a vanished cue, while the memory benchmarks use arms rather than humanoids.Source: C. Data Analysis · p. 4
A. Preliminaries: Whole-Body VLA Policies
- Whole-body VLA policies condition on the current image, proprioceptive state, and instruction; without memory, an event that has left the view cannot affect their actions.WholeBodyMem adds past-view memory tokens and object tokens to the policy inputs.Source: A. Preliminaries: Whole-Body VLA… · p. 4
B. Object-Guided Memory Selection
WholeBodyMem selects task-object event frames within a causal, adaptive memory budget rather than relying only on even sampling. Tracking and event scoring guide which past views are retained while preserving native visual detail.
B. Object-Guided Memory Selection
- The memory budget allows up to seven frames, each retaining 77 native vision tokens, plus at most 14 object tokens.The text specifies 539 memory tokens and says cached frames can be reselected without cost.Source: B. Object-Guided Memory Selection · p. 4
- Native tokens preserve small visual regions that pooled summaries could lose, making frame choice consequential when only six past frames fit.The paper notes that a cue may occupy less than 1% of the image.Source: B. Object-Guided Memory Selection · p. 5
- Frozen SAM3 tracks prompted task objects and returns a mask, box, and confidence for each object and frame; background motion is removed before measuring object motion.Task prompts are fixed before evaluation, and simulation tracking runs at 50 Hz.Source: B. Object-Guided Memory Selection · p. 5
- Objects become event-frame candidates when they appear, disappear, reappear, move, or change shape; disappearance and reappearance rank first, and detections below 0.35 are ignored.Movement is measured after camera-motion removal, and shape change uses mask overlap below 65%.Source: B. Object-Guided Memory Selection · p. 5
- The selector always keeps first, middle, and current frames, then adds up to four high-scoring event frames while avoiding redundant frames within five frames of a selected view.It retains 3 to 7 frames, averaging 5.4; unused budget slots remain empty.Source: B. Object-Guided Memory Selection · p. 5
- Only frames up to the current time are eligible, and the SIMPLE setting fills all seven slots with evenly spaced frames.The evenly spaced seven-frame memory is the uniform control.Source: B. Object-Guided Memory Selection · p. 5
C. Memory Cross-Attention in the Action Expert
The action expert reads selected past views and object tokens through cross-attention, then uses the readout to modulate action-token processing. Object and pad tokens provide additional task-specific information to the expert.
C. Memory Cross-Attention in the Action Expert
- Each selected native vision token receives a sinusoidal code for its frame index and grid cell before memory tokens are computed and shared across blocks and flow-matching steps.Empty slots are masked.Source: C. Memory Cross-Attention in the… · p. 5
- In each of six action-expert blocks, action tokens query the memory, so future action steps retrieve the frames they need while pretrained joint attention stays unchanged.The memory readout modulates normalized action tokens through scale and shift ahead of the adaptive layer norm and feed-forward network.Source: C. Memory Cross-Attention in the… · p. 5
- The readout produces scale and shift terms for action tokens, with near-zero initialization so the policy initially matches the pretrained one.The modulation is applied as normalized action tokens multiplied by one plus scale, then offset by the shift.Source: C. Memory Cross-Attention in the… · p. 5
- Object tokens combine mask-weighted native features with time, centroid, box, area, visibility, and slot information, and a second cross-attention adds their readout.The representation allows up to two tracked objects per memory frame, yielding up to 14 object tokens.Source: C. Memory Cross-Attention in the… · pp. 5–6
- Fig. 4 presents four teleoperated steps per real-world task, with blue arrows for cue and answer and pink arrows for robot motion.Dashed blue arrows mark where the cue was after it disappeared; S1–S3 identify cafe seats from front to back.Source: Figure 4 · p. 6
- A learned query attends over memory, object, and null tokens to predict an answer, and the resulting weights enter each block’s adaptive layer norm.The answer logits are trained with cross-entropy from the decision step onward.Source: C. Memory Cross-Attention in the… · p. 6
- Visible pad tokens combine pooled native features, box geometry, and pad index; attention weights favor the selected pad when added to action-token readout.This branch activates only when the largest weight and the weight on visible pads reach 0.6.Source: C. Memory Cross-Attention in the… · p. 6
D. Training Details
Each task policy is fine-tuned from a pretrained base policy with flow-matching and decision losses; the memory modules are trained from scratch.
Training objective
- Fine-tuning combines the SIMPLE and Ψ0 flow-matching loss with a decision-head cross-entropy loss weighted by λ=0.5.The action expert predicts actions corrupted by noise with σ sampled uniformly from 0 to 1.Source: D. Training Details · p. 6
Simulation training
- The Qwen3-VL-2B backbone stays frozen while the memory encoder, memory attention, and WF are trained from scratch.Training uses AdamW, a 10^-4 learning rate with cosine decay, 1k warm-up steps, bf16, dropout 0.1, and a global batch of 128 for 40k steps.Source: D. Training Details · p. 6
Real-robot training
- Real-robot tasks use 15 teleoperated demonstrations each, with memory modules trained from scratch and the 5k-step checkpoint deployed.The action expert's transformer blocks start from the post-trained action expert.Source: D. Training Details · p. 6
V. EXPERIMENTS
The experiments test whether memory preserves reactive skills, supports decisions after cues vanish, and transfers to a real G1; ablations examine which memory components contribute.
Q1: Reactive skills
- Q1 tests whether adding memory costs or improves performance on the 14 standard SIMPLE tasks that need no cue memory.The experiments also ask whether WholeBodyMem matches or exceeds existing baselines on these tasks.Source: §V · p. 6
Q2: Vanished cues
- Q2 tests whether memory-dependent policies can act on vanished cues that memoryless policies miss.The question concerns six memory tasks.Source: §V · p. 6
Q3: Memory components
- Q3 examines whether object guidance rather than capacity explains gains and what key frames, object tokens, the decision head, adaptive memory, and memory attention each contribute..Source: §V · p. 6
Q4: Real-robot transfer
- Q4 tests whether memory carries over to a real Unitree G1 when the robot turns away from the cue.Source: §V · p. 6
A. Setup
The setup evaluates policies on standard and memory-dependent tasks in simulation and on the real G1, using trained memoryless and uniform-memory baselines.
Simulation
- Simulation uses SIMPLE with a 36-D whole-body action; AMO drives the lower body in memory tasks and two motion-planning tasks, while SIMPLE's controller serves the other 12 tasks.Policies are queried once per action chunk by the unmodified SIMPLE evaluation client.Source: A. Setup · p. 6
Real robot
- The real-robot setup uses a 29-DoF Unitree G1 with Dex3 hands and a head-mounted RealSense D435; SAM3 tracks one frame per executed action chunk.Of 30 predicted actions, the first 24 are executed before the next chunk; SONIC runs at 50 Hz and tracking is about 1.25 Hz.Source: A. Setup · p. 6
Baselines and ablations
- Baselines include seven memoryless policies and a RoboMME-style uniform-memory control using evenly spaced frames at a 539-token budget.The control omits object tokens and the decision head; Table IV reports six-task averages of successes per level for ablations removing adaptive memory, object tokens with the decision head, the decision head, or memory attention.Source: A. Setup · pp. 6–7, Table IV · p. 7
Evaluation
- The 14 SIMPLE tasks use 10 episodes per randomization level, while each of six memory tasks uses 30 balanced-answer episodes per level.Success requires the final condition; memory tasks also report decision accuracy, or placements/touches on the cued pad when no decision head is present.Source: A. Setup · p. 7
B. Results on Standard Loco-Manipulation Tasks (Q1)
WholeBodyMem improves performance on standard tasks and memory-dependent tasks, with object-event frame selection and its other memory components contributing to the gains.
B. Results on Standard Loco-Manipulation Tasks (Q1)
- 86.9% success on the 14 SIMPLE tasks, versus 70.2% for Ψ0 and 76.9% for Uniform memory; WholeBodyMem matches or exceeds the best baseline on every task.T6 yields 21 versus 13 successes and T8 yields 20 versus 11, together accounting for 17 of the 42 extra successes over Uniform memory.Source: B. Results on Standard… · p. 7
C. Results on Memory-Dependent Tasks (Q2)
- 72.6% success on six memory tasks, versus 29.1% for Ψ0 and 44.4% for Uniform memory; WholeBodyMem identifies the cue in 93.7% of decisions.Uniform memory is clearly above chance only on FlashPad and HideMug, and WholeBodyMem's success is stable across randomization levels.Source: C. Results on Memory-Dependent Tasks… · p. 7
D. Object-Guided vs. Uniform Memory (Q3)
- At Uniform memory's seven-frame, 539-token budget, object-event frame selection without object tokens raises success from 44.4% to 60.4%, while adaptive memory uses 22.9% fewer tokens without losing performance.The results support keeping the right frames rather than more frames.Source: D. Object-Guided vs. Uniform Memory (Q3) · p. 7
- Of the 28.2-point gain over Uniform memory, key frames contribute 16.0 points, object tokens 5.3, the decision head 4.7, and adaptive memory 2.2.Memory supplied as context to the frozen VLM instead of through memory attention falls below memoryless Ψ0.Source: D. Object-Guided vs. Uniform Memory (Q3) · p. 7
E. Real-World Experiments (Q4)
Across six real-world tasks, WholeBodyMem outperformed Ψ0, including decisions that combine what, where, and when. The conclusion notes limits from missed object detections, inference cost, rendering differences, and scripted simulated demonstrations.
E. Real-World Experiments (Q4)
- Tasks tested memory for cues that leave view during robot turns, an untraceable can touch, and progress after remaining cans leave view.There were six tasks; cue tasks used three marked areas or seats on table B before the robot turned to table A and back.Source: E. Real-World Experiments (Q4) · p. 7
- Each task used 15 whole-body teleoperation demonstrations, 90 total, with every answer in cue tasks demonstrated five times.Each frame paired a head-camera image and robot state with a motion token and hand targets as the action label.Source: E. Real-World Experiments (Q4) · p. 7
- 88.3% (53/60) of real-world trials succeeded with WholeBodyMem, versus 36.7% (22/60) with Ψ0 across six tasks.The advantage held even when decisions combined what, where, and when.Source: E. Real-World Experiments (Q4) · p. 8
VI. LIMITATIONS AND CONCLUSION
- The conclusion identifies missed cues when SAM3 never detects an object, online tracking inference cost, a rendering-related sim-to-real gap, and scripted simulated demonstrations.It frames remembering before acting on the goal as a first step toward humanoids acting on what happened as well as what is visible.Source: §VI · p. 8