PaperlayerView exact revision on arXiv

Source-linked AI summary

Rolling-WAM: World Action Models with Rolling Imagination

Yinghua Zhou, Junjie Ye, Yiqi Zhao, Hao Dong, Celina Shiyu Wang, Ruohai Ge, Tingyi Yang, Basile Van Hoorick, Gaurav Sukhatme, Vitor Guizilini, Yue Wang

arXiv:2609.30247v1cs.ROcs.AIcs.CV

TL;DR

Joint video-action denoising at every replanning cycle delays action updates and limits closed-loop responsiveness. Rolling-WAM distributes denoising across successive cycles by retaining and refining staggered video-action chunks. It achieves competitive manipulation performance and a 4.5× steady-state inference speedup over standard WAMs.

  • Problem

    Repeatedly denoising the full prediction horizon at each replanning cycle adds latency that delays action updates and limits closed-loop responsiveness.

    Source: §I · pp. 1–2

  • Method

    Rolling-WAM maintains a sliding window of video-action chunks at staggered noise levels, denoising the imminent chunk for execution while refining retained future chunks across cycles.

    Source: §I · p. 2

  • Results

    Rolling-WAM achieves competitive manipulation performance on LIBERO, RoboTwin, and real-world humanoid tasks, with a 4.5× inference speedup over standard WAMs.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    Rolling-WAM reduces the denoising steps needed for the next executable action chunk while preserving evolving visual-action predictions across chunk boundaries.

    Source: §V · p. 7

    Retained predictions may lag behind rapid scene changes and misguide action generation, particularly with long windows.

    Source: §V · p. 8

Abstract

from arXiv · show

World Action Models (WAMs) couple action generation with future visual prediction for robotic manipulation. However, completing the joint video-action denoising process at each replanning cycle incurs substantial latency, delaying action updates and limiting closed-loop responsiveness. We present Rolling-WAM, a formulation that distributes joint denoising across successive replanning cycles. Our method maintains a sliding window of video-action chunks at staggered noise levels. At each step, a rolling noise schedule fully denoises the imminent action chunk for execution, while partially refining farther-future chunks. As the window advances with new camera observations, the retained future chunks continue their denoising process. This distributes the computational cost over time while carrying an evolving visual-action context across chunk boundaries. Evaluations on LIBERO, RoboTwin, and a real-world Unitree G1 humanoid show that Rolling-WAM achieves competitive manipulation performance. By removing the need to denoise the entire prediction horizon from scratch, it delivers a 4.5x steady-state replanning speedup over standard joint WAMs.

I. INTRODUCTION

World Action Models jointly predict actions and future observations, but repeated joint denoising makes closed-loop replanning slow. Rolling-WAM carries partially denoised future chunks across replans, achieving competitive benchmark performance with a reported 4.5× speedup over standard WAMs.

A. World Action Models for Robotic Manipulation

  • WAMs jointly predict robot actions and future observations, providing visual context that helps the policy anticipate scene evolution,, [3].Source: §I · p. 1

I. INTRODUCTION

  • Repeatedly denoising the full video-action horizon delays action updates and limits closed-loop responsiveness.Some methods omit future-video generation at test time, but then lose explicit visual predictions that could inform action generation.Source: §I · p. 1
  • Rolling-WAM distributes joint denoising across replanning cycles, fully denoising the imminent action chunk while retaining partially denoised future chunks.As new observations arrive, retained chunks continue to be refined rather than discarded and regenerated from scratch.Source: §I · p. 2
  • 98.1% average success on LIBERO and 93.3% on RoboTwin, with competitive or better-than-state-of-the-art performance and a 4.5× inference speedup over standard WAMs.The evaluations also include three real-world Unitree G1 humanoid manipulation tasks.Source: §I · p. 2

B. Efficient Inference for World Action Models

Prior efficiency approaches reduce, cache, or reuse future visual prediction, while action-only rolling methods do not couple it to action generation. Rolling-WAM instead refines joint video-action predictions across replanning cycles to address the latency of closed-loop WAM inference.

B. Efficient Inference for World Action Models

  • Rolling-WAM distributes joint video-action denoising across replanning cycles rather than removing or caching future predictions.Other approaches omit future-video generation, cache visual context, reduce model or token costs, distill sampling, reuse context, or overlap inference with execution.Source: B. Efficient Inference for World… · p. 2

C. Rolling Diffusion for Sequence Generation

  • Action-only rolling methods maintain partially denoised action buffers, whereas Rolling-WAM couples that mechanism to visual futures and evaluates broader multitask settings.Rolling Diffusion provides a sliding-window precedent with progressively higher noise toward the future,,.Source: C. Rolling Diffusion for Sequence… · p. 2

III. METHOD

  • The method section develops the formulation, rolling denoising, sampling and execution, then model architecture and training objective.Source: §III · p. 2

A. Problem Formulation

  • Closed-loop WAM inference requires multiple denoising steps per replanning cycle, increasing latency and limiting responsiveness to new observations.Rolling-WAM maintains a sliding window of joint predictions, completing near-term predictions while refining farther-future chunks at staggered noise levels under new observations.Source: A. Problem Formulation · p. 3

B. Rolling Video-Action Denoising

Rolling video-action denoising partitions the prediction horizon into aligned chunks and assigns them staggered noise levels. Each cycle completes the nearest chunk while retaining and refining future predictions as the window advances.

B. Rolling Video-Action Denoising

  • The prediction window contains W aligned video-action chunks, each pairing a video segment with K actions over the same physical interval.The total horizon is H = WK, and execution advances one chunk at a time.Source: B. Rolling Video-Action Denoising · p. 3
  • During rolling denoising, the first chunk becomes clean while later chunks remain at progressively higher noise levels.The noise phase τ decreases over a replanning cycle, moving chunks toward the noise level assigned to their preceding window position.Source: B. Rolling Video-Action Denoising · p. 3
  • After execution, retained chunks are already at the starting noise level for their new positions, and a Gaussian-noise chunk restores the window.Source: B. Rolling Video-Action Denoising · p. 3
  • At episode initialization, all chunks start at Gaussian noise; nearer chunks begin denoising earlier than more distant chunks.Initialization establishes the rolling window because partially denoised predictions are not yet available.Source: B. Rolling Video-Action Denoising · pp. 3–4

C. Sampling and Execution

Sampling jointly updates video and action predictions across the window, then executes the clean nearest action chunk and shifts the window. Later replanning cycles require fewer denoising steps to produce the next executable chunk.

C. Sampling and Execution

  • N denoising steps are distributed across chunks, so subsequent replanning cycles require only N/W steps to produce the next executable action chunk.This allocation assumes N is a multiple of W.Source: C. Sampling and Execution · p. 4
  • Each denoising step jointly updates video and action predictions throughout the window using the predicted flow velocity.The update operates on each chunk's current denoising state.Source: C. Sampling and Execution · p. 4
  • After the first chunk's K actions execute, the window retains its remaining predictions, appends a Gaussian-noise chunk, and conditions the next cycle on the latest camera observation and state.Each chunk receives all N denoising steps before execution, but that computation is distributed across cycles; larger windows can lower steady-state latency with fixed N and sufficient GPU parallelism.Source: C. Sampling and Execution · p. 4

D. Model Architecture and Training

Rolling-WAM couples pretrained video and action Transformers so action generation can use an evolving visual prediction. Its training objective supports both initializing the prediction window and refining it during rolling execution.

D. Model Architecture and Training

  • A pretrained video Diffusion Transformer and lightweight action Transformer are coupled through masked joint attention, with language and proprioception conditioning both experts.The video expert processes observation and future-video VAE latents; the action expert processes projected noisy actions.Source: D. Model Architecture and Training · p. 4
  • Chunk-wise noise modulation lets predictions at different refinement stages be processed together, while each action chunk attends to visual features across the full window.Action-to-action attention is restricted to the same chunk, and video tokens do not attend to action tokens.Source: D. Model Architecture and Training · p. 4
  • Flow matching trains the joint denoiser on rolling and initialization modes, using independent Gaussian samples for video and actions and noise-weighted losses for each modality.The activity mask excludes initialization chunks held at pure noise; training on both modes equips the model to initialize and continue refining the window.Source: D. Model Architecture and Training · p. 4

IV. EXPERIMENTS

The experiments compare task success and replanning latency across simulation benchmarks, real-world humanoid tasks, and a broad set of vision-language-action and world-action models. LIBERO evaluation uses four suites, with success rate as the primary performance metric.

IV. EXPERIMENTS

  • The evaluation compares task success with vision-language-action and world-action models on LIBERO, RoboTwin 2.0, and Unitree G1 tasks, and measures replanning latency.Ablations examine window size, training noise schedules, and cross-chunk action attention.Source: §IV · p. 4, A. Experimental Setup · p. 4

A. Experimental Setup

1) LIBERO:

  • LIBERO evaluation covers Spatial, Object, Goal, and Long suites, with 10 tasks and 500 demonstrations per suite and 50 rollouts per task.One policy is trained across the four suites for 10 epochs; observations include external and wrist RGB views.Source: 1) LIBERO · p. 4, Table I · p. 5

2) RoboTwin 2.0:

The paper evaluates one Unitree G1 policy across three real-world manipulation tasks and describes its training and deployment configuration. It also reports RoboTwin 2.0 success rates in Table II.

3) Real-world humanoid manipulation:

  • On the Unitree G1, the policy is evaluated on Doll Placement, Plate Stacking, and Bead Pouring, with 20 trials per task.It uses 50 demonstrations per task, trains for 7,500 steps, and executes actions at 10 Hz.Source: 3) Real-world humanoid manipulation · p. 5
  • The real-world policy receives egocentric RGB and robot state, and predicts actions combining a SONIC motion latent with commands for both hands.The input is a 320 × 224 image and 43-dimensional state; actions comprise a 64-dimensional motion latent and 7-dimensional commands for each hand.Source: 3) Real-world humanoid manipulation · p. 5

B. Implementation Details

  • The video expert is initialized from Wan2.2-TI2V-5B, while the action expert is initialized from interpolated video-expert weights; training uses benchmark demonstrations without additional embodied pretraining.The VAE and text encoder remain frozen during joint training.Source: B. Implementation Details · p. 5
  • The default configuration uses a five-chunk window, 10 denoising steps per chunk, and two denoising steps per steady-state replanning cycle.Initialization and rolling modes are sampled with probabilities 0.2 and 0.8, respectively.Source: B. Implementation Details · p. 5
  • Table II reports RoboTwin 2.0 success rates and states that Rolling-WAM achieves the highest success rate in both settings.Source: Table II · p. 5
  • Figure 3 compares imagined video with ground-truth observations across successive moments during Put Object Cabinet execution.Source: Figure 3 · p. 5

C. Performance on Simulation Benchmarks

Rolling-WAM records strong success rates on both simulation benchmarks, while reducing steady-state replanning latency relative to joint and fast WAM baselines. Its latency also remains lower in a window-size comparison, and imagined rollouts qualitatively track observations in one RoboTwin example.

C. Performance on Simulation Benchmarks

  • 98.1% on LIBERO is within 0.4 percentage points of LingBot-VA and Joint-WAM (both 98.5%), while 93.3% on RoboTwin 2.0 leads LingBot-VA (92.2%), Fast-WAM (91.8%), and Joint-WAM (90.6%).RoboTwin clean and randomized success rates are 93.5% and 93.0%, respectively.Source: C. Performance on Simulation Benchmarks · p. 5

D. Inference Efficiency

  • 215 ms versus 978 ms for Joint-WAM gives Rolling-WAM a 4.5× speedup, while 548 ms for Fast-WAM corresponds to a 2.5× speedup.The default comparison uses N = 10 and W = 5; each update takes two denoising steps while retaining an 80-action prediction window.Source: Figure 4 · p. 6, D. Inference Efficiency · p. 6

C. Performance on Simulation Benchmarks

D. Inference Efficiency

  • At W = 5, Rolling-WAM takes 322 ms, compared with 832 ms for Fast-WAM and 1529 ms for Joint-WAM in the window-size comparison.Rolling-WAM uses H = 16W while the baselines use H = 16; latency flattens for W = 6–8 as rolling denoising remains at two steps per cycle.Source: Figure 5 · p. 6, D. Inference Efficiency · p. 6

E. Real-World Humanoid Evaluation

Rolling-WAM achieved the highest average success rate among five policies in real-world humanoid manipulation, with especially strong Bead Pouring performance. The evaluation covered three tasks on the Unitree G1, and execution was qualitatively more continuous across action-chunk boundaries than with Joint-WAM.

E. Real-World Humanoid Evaluation

F. Ablation Studies

The ablations examine window size, training noise mixtures, and cross-chunk action attention. The conclusion reports competitive LIBERO performance, stronger results than compared methods on RoboTwin 2.0 and real-world humanoid tasks, and substantial inference speedups over standard joint WAMs.

F. Ablation Studies

  • 78.2% success at W = 5 exceeded 77.3% at W = 3 and 69.5% at W = 8; larger windows did not consistently improve performance.The authors conjecture that distant visual predictions may offer limited guidance for imminent actions; W = 5 is the default.Source: F. Ablation Studies · p. 7
  • Random noise with p = 0.5 raised selected RoboTwin success to 78.5% but lowered LIBERO to 97.3%; no alternative improved both benchmarks.The full rolling schedule scored 78.2% on selected RoboTwin tasks and 98.1% on LIBERO, and was retained for non-initialization samples.Source: F. Ablation Studies · p. 7
  • Cross-chunk action attention scored 76.3% on selected RoboTwin tasks and 97.9% on LIBERO, below within-chunk attention at 78.2% and 98.1%.The results favor sharing information through the visual window while restricting direct action attention to each chunk.Source: F. Ablation Studies · p. 7

V. CONCLUSION

  • Rolling-WAM achieved competitive LIBERO performance and outperformed compared methods on RoboTwin 2.0 and real-world humanoid manipulation, with substantial inference speedups over standard joint WAMs.Source: §V · p. 7

APPENDIX ROBOTWIN DETAILED RESULTS

Table V presents per-task RoboTwin 2.0 success rates for Rolling-WAM and baselines across clean and randomized evaluations.

APPENDIX ROBOTWIN DETAILED RESULTS

Loading 2609.30247v1…