Source-linked AI summary
Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation
Zirui Zheng, Jiaqian Yu, Xiongfeng Peng, jun shi, Mingyi Li, Chao Zhang, Weiming Li, Dong Wang, Huchuan Lu, Xu Jia
TL;DR
Occlusions and rapid wrist-camera motion make current observations insufficient for persistent manipulation rollouts. Mem-World uses W-VMem to retrieve geometry-relevant history, improving policy-evaluation correlation and long-horizon synthetic-data training results.
Problem
End-effector occlusions and rapid wrist-camera motion leave incomplete current views, making persistent prediction difficult in long-horizon manipulation.
Source: §1 · p. 1Method
W-VMem anchors observations to temporally evolving surfels and scores their visibility, task relevance, and recency to retrieve history for future prediction.
Source: §1 · p. 2, §3.2 · pp. 4–5Results
Pearson correlation with real-world performance improves by 14.5%, and synthetic-data training raises average success on long-horizon tasks from 58% to 72%.
Source: §1 · p. 2, §4.3 · p. 8Takeaways & Limitations
Mem-World rollouts support more reliable policy evaluation and policy improvement through generated training data.
Source: §1 · p. 2, §4.3 · p. 8Mem-World does not explicitly enforce physical constraints, so it may predict a successful grasp without antipodal gripper contacts.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Action-conditioned world models have emerged as a promising paradigm for robot learning, offering a scalable alternative to costly real-world experimentation by generating action-consistent video rollouts. However, persistent world modeling remains challenging in manipulation: frequent end-effector occlusions and rapid wrist-camera motion make the current observation insufficient for predicting future views, causing models to forget or hallucinate scene details seen in earlier frames. Existing memory retrieval strategies often fail to identify informative history in dynamic manipulation scenarios. To address this limitation, we propose Mem-World, a memory-augmented multi-view action-conditioned world model. At its core, we present W-VMem, a 4D wrist-view-centered surfel-indexed memory that anchors historical observations to temporally evolving surface elements. By explicitly modeling when and where scene elements are observed, W-VMem enables geometry-aware retrieval of relevant history frames conditioned on future actions. During generation, relevant history frames are selected via surfel-based rendering and scoring, providing informative and non-redundant context for prediction. Extensive experiments show that Mem-World generates persistent rollouts in complex manipulation scenarios, enables more reliable policy evaluation than Ctrl-World, improving the Pearson correlation with real-world performance by 14.5\%, and supports effective policy improvement through synthetic data generation, increasing success rates from 58\% to 72\% on long-horizon tasks.
1 Introduction
Manipulation world models can lose or hallucinate scene details when wrist-camera motion and occlusions leave the current view incomplete. Mem-World uses geometry-aware history retrieval to support persistent rollouts, with reported gains in policy evaluation and training.
1 Introduction
- End-effector occlusions and rapid wrist-camera motion leave partial scene evidence, leading models to forget or hallucinate earlier-visible objects, especially in long rollouts.Source: §1 · p. 1
- Joint-pose similarity is only an indirect relevance proxy and misses visibility constraints, while static-scene retrieval assumptions do not fit dynamic manipulation and robot-arm occlusions.Source: §1 · p. 2
- Pearson correlation with real-world performance improves by 14.5%, while long-horizon synthetic-data training raises average success from 58% to 72%.Mem-World also outperforms prior world models on frame-level and object-level consistency metrics.Source: §1 · p. 2
- Mem-World is proposed as a memory-augmented multi-view action-conditioned model for persistent manipulation rollouts and more reliable policy evaluation and training.Source: §1 · p. 2
- W-VMem anchors historical wrist observations to temporally evolving surface elements, enabling geometry-aware retrieval under occlusions, rapid camera motion, and dynamic hand-object interactions.Source: §1 · p. 2
2 Related Work
Related work uses action-conditioned video models for robotics and pursues long-horizon consistency through explicit reconstruction or implicit memory. Mem-World distinguishes itself with temporally aware, manipulation-centric surfel retrieval.
2 Related Work
- Action-conditioned world models condition video generation on low-level actions for policy evaluation, while IRASim [5], Ctrl-World [1], and Interactive World Simulator [6] represent related approaches.Related applications of video prediction include zero-shot planning [19] [20], direct control [21] [22] [23] [24], and data generation [25] [26] [27].Source: §2 · pp. 2–3
- Long-horizon consistency methods use explicit 3D reconstruction [29] [30] [31] or implicit context retrieval [16] [15] [17] [32]; VMem [17] retrieves views by rendering static surfels from future camera poses.Explicit reconstruction can provide geometric guidance but depends on reconstruction quality and adds computational overhead.Source: §2 · p. 3
- Unlike static camera-pose overlap alone, Mem-World makes retrieval temporally aware and manipulation-centric through dynamic surface-level associations.Source: §2 · p. 3
3 Method
The method extends surfel-indexed view memory into a wrist-centered, time-aware memory for action-conditioned manipulation. It retrieves history by rendering and scoring surfels from estimated future wrist viewpoints, then updates memory during rollout.
3 Method
- Mem-World alternates between retrieving history from W-VMem, predicting future multi-view observations conditioned on that history and actions, and updating memory with predicted observations.Source: §3 · p. 3, §3.2 · p. 5
3.1 Preliminary
- VMem [17] stores surfels with observed-view indices and retrieves views covering rendered surfels, but assumes available camera poses and does not distinguish changing scene elements.Source: §3.1 · p. 4
3.2 W-VMem for Persistent Manipulation World Modeling
- Future wrist-camera poses are derived from robot actions using forward kinematics and the fixed wrist-camera-to-end-effector transform, then converted to the point-cloud frame using camera extrinsics.Source: §3.2 · p. 4
- W-VMem gives surfels observed-timestep and manipulated-object attributes; it initializes from three camera views but uses subsequent wrist views for updates to preserve wrist-observation timing.Source: §3.2 · pp. 4–5
- For retrieval, W-VMem renders surfels separately by timestep from an estimated future wrist viewpoint, scores geometric visibility, task relevance, and recency, then applies NMS to reduce redundant views.NMS promotes broader scene coverage when the wrist repeatedly visits the same region.Source: §3.2 · p. 5
4 Experiments
The experiments assess long-horizon rollout consistency, policy evaluation, and policy training with synthetic data. They evaluate manipulation scenarios involving occlusions, wrist-camera motion, and increasingly complex tasks.
4.1 World Model Consistency Evaluation
- Evaluation used 34 DROID replay trajectories selected for gripper occlusion or large back-and-forth wrist-camera motion, with 20-step, 20-second rollouts.The evaluation compared predicted and ground-truth frames using PSNR, SSIM, and LPIPS, and assessed object-level consistency with DINOv2 feature similarity.Source: §4.1 · p. 6
4.3 World Model for Policy Training
- Real-world average success on three long-horizon tasks rose from 58% to 72% after post-training on Mem-World-generated successful trajectories.The study generated 200 trajectories per task and retained 20–30 successful trajectories using human success annotations.Source: Figure 5 · p. 7, §4.3 · p. 8
4.2 World Model for Policy Evaluation
- Mem-World policy success rates correlated more strongly with real-world rates (r = 0.97, p < 0.001) than Ctrl-World rates did (r = 0.85, p < 0.01).Mem-World's absolute success rates were generally lower than real-world rates, but the gap was relatively uniform across tasks.Source: Figure 4 · p. 7, §4.2 · p. 8
4.1 World Model Consistency Evaluation
- W-VMem retrieved relevant historical frames, helping rollouts recover occluded or previously observed scene content during back-and-forth camera motion.The retrieval ablation compares W-VMem with stride memory and short-term memory; the latter strategies provide less informative context and can distort or lose objects.Source: §4.1 · p. 7, Figure 3 · p. 7
5 Conclusion
Mem-World uses W-VMem to retrieve relevant past observations, producing more persistent manipulation rollouts than prior works and widely adopted memory mechanisms. Its rollouts correlate with real-world policy performance and can improve policy training, while initial wrist-view quality and the lack of explicit physical constraints remain limitations.
5 Conclusion
- Mem-World produces more persistent rollouts than prior works and widely adopted memory mechanisms, with rollouts that correlate with real-world policy performance and can improve policy training.The paper presents the rollouts as a data engine for policy training.Source: §5 · p. 8
- Rollout consistency can be limited by an occluded or narrow-field-of-view initial wrist frame, and the model does not explicitly enforce physical constraints.The paper notes that a spherical-object grasp may be predicted as successful without antipodal gripper contacts.Source: §5 · p. 8
- Future work may incorporate contact or force information to improve the physical fidelity of imagined rollouts.Source: §5 · p. 9