Source-linked AI summary
Memory as Plans: World-Action Modeling with Memory-Grounded Planning
Sizhe Zhao, Haozhe Xie, Weiyu Zhao, Chenchu Zhang, Huan Wang, Chenyang Wang, Qinglin Liu, Shengping Zhang
TL;DR
Memory-dependent robotic manipulation requires information beyond the current observation, while existing memory mechanisms either lose visual detail or incur growing execution costs. MaP-WAM converts multimodal episodic history into memory-grounded language and visual plans, then uses progress-aware execution for adaptive segment transitions. It achieves 83.3% success on RMBench and 78.0% on real-robot tasks while maintaining approximately constant executor latency as history grows.
Problem
Memory-dependent manipulation requires long-horizon information beyond the current observation, while language summaries can omit visual evidence and growing visual histories increase execution costs.
Method
MaP-WAM converts completed segment records with sparse visual context into language-and-visual plans, then uses WAP to execute plans with progress-aware adaptive transitions.
Results
83.3% success on RMBench and 78.0% on real-robot tasks, with approximately constant executor inference latency as task history grows.
Takeaways & Limitations
Separating planning-time memory use from fixed-context execution preserves long-horizon visual grounding while avoiding history-dependent executor latency.
Abstract
from arXiv · showhide
Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world manipulation tasks are inherently non-Markovian, requiring long-horizon memory beyond the current observation. Existing memory mechanisms often rely on language summaries, growing visual windows, or their combinations, and may therefore lose fine-grained visual evidence or face a trade-off between history coverage and execution efficiency. We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history. MaP-WAM represents memory as completed segment records containing language instructions and sparse visual context, and converts this episodic memory into compact plans comprising the next segment-level language plan and corresponding visual guidance. A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predicting action chunks and corresponding execution progress at inference time, calibrating predicted progress through plan-observation alignment for adaptive segment transitions and closed-loop context updates. MaP-WAM keeps the executor context length fixed, while structured attention further enables key-value caching in both planning and execution. MaP-WAM achieves state-of-the-art performance on RMBench with an 83.3% success rate and attains 78.0% success on real-robot tasks, while maintaining approximately constant executor inference latency as task history grows.
1. Introduction
MaP-WAM addresses memory-dependent manipulation by using long-horizon multimodal context for segment-level planning rather than repeatedly feeding full history to execution. It combines memory-grounded visual-language planning with progress-aware execution, achieving strong benchmark and real-robot success while keeping executor latency approximately constant.
- Memory-dependent manipulation requires long-horizon memory because information needed for future decisions may no longer be visible in the current observation.
- Growing visual histories preserve perceptual evidence but increase computational and GPU-memory costs as context expands.
- MaP-WAM uses episodic multimodal context as planning-time evidence, decoupling memory-grounded planning from fixed-context plan-conditioned execution.
- The framework represents completed segments with language instructions and sparse visual context, then produces language plans and corresponding visual guidance.
- WAP jointly models visual dynamics, action chunks, and execution progress to support variable-duration execution and adaptive segment transitions.
- 83.3% success on RMBench and 78.0% on real-robot tasks, with approximately constant executor inference latency as task history grows.
2. Related Work
Related work spans reactive VLA and world-action policies, goal- or plan-conditioned control, and memory mechanisms based on language or visual context. These approaches trade compactness and interpretability against retention of fine-grained history and execution efficiency.
- Reactive VLA policies improve instruction following and low-latency control but struggle with memory-dependent tasks.
- World-action models enrich action generation by predicting future states, observations, or action-conditioned scene evolution beyond single-frame control.
- Goal- and plan-conditioned policies generate intermediate subgoals, trajectories, or videos before producing actions.
- Language-memory methods provide compact, interpretable summaries but may discard fine-grained visual and spatial evidence.
- Growing visual windows retain richer temporal evidence, while fixed windows truncate distant history and growing windows increase latency and GPU-memory costs.
3. Our Approach
MaP-WAM decomposes memory-dependent control into episodic memory-grounded planning and fixed-context execution. Its visual plans, progress-aware WAP executor, and alignment-based transitions support variable-duration closed-loop control while avoiding history-dependent executor inputs.
- 3.1. Overview: MaP-WAM decouples long-horizon visual-context processing from short-horizon action generation using structured segment-level episodic context.
- 3.1. Overview: The executor predicts short-horizon actions from the current observation, robot state, instruction, and memory-grounded plan, with inputs independent of history length.
- 3.2. Memory-Grounded Planning: Memory-grounded planning predicts a next segment language plan and uses a causal world model to generate corresponding visual guidance from completed-segment context.
- 3.2. Memory-Grounded Planning: Block-causal attention makes completed visual evidence a static, cacheable prefix while preventing future leakage across segments.
- 3.3. World-Action-Progress Modeling: WAP jointly models future visual states, action chunks, and progress from the plan, current observation, robot state, and progress condition.
- 3.3. World-Action-Progress Modeling: Plan-observation alignment calibrates recursively predicted progress against temporally indexed visual-plan frames, while thresholded progress triggers context updates and replanning.
4. Experiments
MaP-WAM is evaluated on simulation and real-robot memory-dependent manipulation tasks, with ablations examining visual planning, progress modeling, and inference efficiency.
- Simulation Experiments: 83.3% overall success on RMBench is the highest among representative baselines, while MaP-WAM reaches 96% on both Swap T and Press Button.Observe and Pick Up improves from 9% to 19% over the strongest baseline.
- Real-World Experiments: 88% and 68% success rates are achieved on Find Button and Press Buttons over 50 independent trials per task.Baselines achieve non-zero success on Find Button through random selection but fail on Press Buttons.
- Memory-Grounded Visual Planning: Memory-grounded visual planning preserves object identities and spatial positions that current-observation-conditioned planning often gets wrong.The ablation compares variants with no visual plan and no visual memory under matched retained backbones.
- Progress Modeling: Progress calibration mitigates autoregressive drift through plan-observation alignment on the substantially longer Blocks Ranking Try task.The subgoal swap requires three consecutive pick-and-place operations and averages 423.10 steps versus 116.32 for the other evaluated tasks.
- Inference Efficiency: MaP-WAM keeps executor inference efficient as history grows by using fixed plan context, whereas full-context execution latency increases rapidly.KV caching reduces repeated prefix computation, but the full-context executor still incurs roughly 4× the zero-history latency.
5. Conclusion
MaP-WAM addresses memory-dependent robotic manipulation by converting structured episodic context into memory-grounded plans instead of repeatedly conditioning execution on dense visual histories.
- 5. Conclusion: MaP-WAM couples task semantics with anticipated visual evolution through memory-grounded planning and uses progress-aware execution to connect planning, execution, and context updates.The framework preserves planning-time access to long-term, fine-grained visual context.
A. Conditional Flow Matching
Conditional flow matching trains a velocity-field model to transform noisy interpolations toward target variables under conditioning information.
- A. Conditional Flow Matching: The flow-matching objective trains a velocity-field model on noisy samples formed by interpolating a target variable with Gaussian noise at a flow time.The objective is conditioned on c and compares the predicted velocity with the target noise-to-sample direction.
B. Construction of Multimodal Episodic Context
MaP-WAM stores completed segments as structured multimodal records with language instructions and sparse visual evidence. These records provide compact planning-time context for predicting the next segment plan and its visual evolution.
- Segment Records: Each completed segment is represented as a record combining its language instruction with sparse visual evidence.The episodic context also includes the global task instruction and prior completed-segment records.
- Sparse Visual Context: Eight temporally uniform key observations are retained per segment by selecting the last frame from each of eight equal bins.The initial observation is retained separately as G0.
- Planning Context: Before planning segment k, the structured context supplies completed segment instructions and visual evidence for predicting the next language plan and desired visual evolution.Language planning uses G0 and the final frame of each completed segment, while the CWM receives the full sparse visual context.
C. Attention Mask and KV Caching Details
Structured causal attention separates reusable planning and execution prefixes from target predictions. This organization enables KV caching and refreshes execution context only when progress indicates a segment transition.
- CWM Planning: Completed visual evidence forms a causal planning prefix whose hidden states can be cached because they do not depend on the target segment plan.The block mask restricts each block to earlier or same-index blocks, while completed evidence attends only to the global instruction.
- WAP Execution: During WAP execution, the generated visual plan is a static prefix, while current-state and future visual, action, and progress tokens use structured branch-specific attention.Progress tokens additionally attend to target tokens, and the future visual branch can be omitted during deployment after auxiliary training.
- Segment Transitions: The execution cache is refreshed only after a progress-gated transition generates a new segment plan.Algorithm 1 initializes progress and repeatedly predicts actions and progress until the completion threshold is reached.
D. Plan-Observation Alignment Details
MaP-WAM aligns current observations with generated visual plans to calibrate execution progress. WAP then uses progress predictions and completion scores to support variable-duration execution and segment transitions.
- Reference Sequence: At each segment start, the reference sequence contains the segment’s initial observation and eight generated plan frames with progress increasing from 0 to 1.The resulting reference sequence has nine frames in total.
- Progress Calibration: Before each action chunk, current progress is calibrated by comparing the observation with the two reference frames nearest to its estimated progress using RGB mean absolute pixel difference.This alignment updates the recurrent progress estimate before action inference.
- Chunked Execution: WAP jointly predicts action chunks and progress sequences, retaining prefixes for execution, progress updates, and completion-score computation.The retained prefix determines when the current plan is completed and the system can transition to a new segment.
F. Implementation Details
The implementation uses flow-matching configurations for CWM and WAP and trains task-specific models on multi-GPU hardware. The language planner receives ordered visual evidence alongside task and completed-subtask instructions.
- Hardware and Training: MaP-WAM is trained on eight NVIDIA A800 GPUs with 80 GB of memory each, while WAP training epochs vary by task according to available training frames.Additional training configurations are reported in Table 3.
- Language-Planner Prompt: The language planner receives the initial observation, final sampled frames from completed segments, the global task instruction, and corresponding segment instructions.Its prompt asks it to analyze the next subtask after showing the results of completed subtasks.
H. Latency Measurement Protocol
Latency measurements use a controlled single-GPU protocol with standardized precision, batch size, denoising steps, warm-up, and repeated runs.
- Latency is measured on one NVIDIA A800-SXM4 GPU using BF16 precision, batch size one, and 10 flow-matching denoising steps.Measurements cover latent-space inference only and exclude loading, encoding, transfer, and VAE operations.