PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-WAM: Scaling the Context of World-Action Models

Wei Huang, Bohan Zhang, Chenzhi Liu, Isabella Liu, Shuai Yang, Weian Mao, Luozhou Wang, Yicheng Xiao, Weifeng Lin, Qixin Hu, Bryan Chu, Sifei Liu, Linxi Fan, Xiaojuan Qi, Song Han, Yukang Chen

arXiv:2610.10528v1cs.ROcs.AIcs.CV

TL;DR

Real-time robot control needs visual history to infer motion and task progress, but processing that history can delay action. Long-WAM learns causal video prediction before action adaptation and preserves that structure; it improves performance across benchmarks and physical tasks, with longer context benefiting AR-pretrained foundations more than bidirectional initialization.

  • Problem

    Real-time robot control needs visual history to infer motion and task progress, but processing more history can delay the response it is meant to improve.

    Source: §1 · p. 2

  • Method

    Long-WAM first learns AR video prediction from robot and egocentric videos, then adapts to actions while preserving causal history-to-future structure.

    Source: §1 · p. 2

  • Results

    Long-WAM raises GR-1 success from 63.3% to 78.7% and leads compared methods on LIBERO-Long, RoboTwin 2.0, and DOMINO.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    Longer context pays off more with AR-pretrained foundations, while robot-domain AR pretraining raises peak success on GR-1 and LIBERO-Long.

    Source: §1 · p. 2

Abstract

from arXiv · show

Real-time robot control demands enough visual history to infer motion and task progress, but processing that history can delay action. We present Long-WAM, a model-system framework for scaling the context of causal world-action models under real-time control constraints. Our central finding is that access to history is not the same as using it: longer histories pay off far more when the video foundation is pretrained autoregressively (AR). We first learn causal prediction from robot and egocentric videos without action labels, then preserve this history-to-future structure during world-action adaptation. On RoboCasa GR-1, increasing context from 0.0 to 19.2 seconds raises success from 63.3% to 78.7%, whereas a bidirectionally pretrained initialization shows no net gain; robot-domain AR pretraining further raises peak success on GR-1 and LIBERO-Long. Long-WAM also achieves the best results among compared methods on LIBERO-Long, RoboTwin 2.0, and DOMINO. Streaming observation encoding, asynchronous execution, and hardware-specific acceleration enable deployment on RTX 5090, DGX Spark, and Jetson AGX Thor without dropping future prediction; on RTX 5090, each action chunk, including future-video latent prediction, takes 107.4 ms. Real-time deployment on Unitree G1 and YAM supports dynamic and long-horizon manipulation, including 95% success on dynamic cup stacking, where Pi0.5 and Fast-WAM succeed in none of 20 trials. As a memory-informed executor, Long-WAM also complements higher-level planning in composite tasks.

1. Introduction

Long-WAM treats visual history as useful when causal predictive pretraining teaches the controller to exploit it, and co-designs execution to retain future prediction under real-time constraints. It improves benchmark and physical-robot performance and can complement higher-level planning.

1. Introduction

  • Long-WAM first learns AR video prediction from robot and egocentric videos, then preserves its history-to-future structure during action adaptation.The approach distinguishes learning to predict from history from merely providing a controller access to history.Source: §1 · p. 2
  • On GR-1, longer history benefits AR-pretrained foundations but not bidirectional initialization; the robot-domain AR model’s lead grows from 3.3 to 17.1 points.LongLive2.0-Robot is pretrained on roughly 10,000 window-equivalent hours of robot and egocentric video, and robot-domain pretraining raises peak success on GR-1 and LIBERO-Long.Source: §1 · p. 2
  • Streaming observation encoding, asynchronous execution, and hardware-specific acceleration retain future prediction while enabling deployment on RTX 5090, DGX Spark, and Jetson AGX Thor.On RTX 5090, each action chunk takes 107.4 ms, including observation VAE computation.Source: §1 · p. 2
  • Long-WAM raises GR-1 success from 63.3% to 78.7%, leads on RoboTwin 2.0 and DOMINO, and stacks moving cups at 95%.Its executor also raises RoboCasa365 success from 31.4% to 54.4% when paired with GPT-6 Astra, versus 25.2% for the planner alone.Source: §1 · p. 2
  • Pairing Long-WAM’s unchanged checkpoint with GPT-6 Astra raises RoboCasa365 success from 31.4% to 54.4%, versus 25.2% for the planner alone.Planner augmentation yields a larger gain for Long-WAM (+23.0 points) than for π0.5 (+13.6 points).Source: §1 · p. 2

2. Related Work

Prior work explores video-generator-based world-action models, memory interfaces for retaining history, and methods for reducing inference cost. Long-WAM’s approach combines causal video prediction with action adaptation and targets efficient asynchronous control.

World-action modeling from video generators

  • World-action models connect visual dynamics to executable actions, with prior approaches varying in video pretraining, causal action coupling, and alignment of generated video with action consequences.Long-WAM instead uses action-free robot-video AR pretraining followed by action adaptation under the same causal ordering.Source: §2 · p. 2

Memory and context scaling

  • Causal caches, compressed histories, event retrieval, and test-time adaptation provide complementary memory interfaces; Long-WAM studies the robot’s own interaction history.Its focus is how predictive pretraining shapes the benefit of longer context, alongside closed-loop success and online computation.Source: §2 · pp. 2–3

Efficient and deployable generative control

  • Figure 2 distinguishes current-observation, history-only, co-denoising, and predict-then-act strategies; IDM conditions actions on observed history and predicted future latents.It reuses visual KV across action-denoising steps.Source: Figure 2 · p. 3
  • Prior systems reduce generative-control cost through caching, few-step generation, pipelining, action-only decoding, and hierarchical update rates.Long-WAM retains future visual conditioning and addresses sequential cost with streaming observation encoding and device-specific acceleration.Source: §2 · p. 3

3. Scaling the Context of WAMs from AR Video Generation

Long-WAM scales causal visual context by first training AR video prediction and then adapting it to action generation without losing its history-to-future structure. The section describes robot-video pretraining, causal video–action coupling, and context-scaling experiments.

3. Scaling the Context of WAMs from AR Video Generation

  • Long-WAM learns robot motion and interaction dynamics through long-sequence AR video pretraining, then transfers the predictive foundation to action generation while preserving causal temporal structure.Actions condition on observed history and anticipated futures, making prediction an intermediate representation for control.Source: §3 · p. 3

3.1. LongLive2.0-Robot: Robot-Domain AR Video Pretraining

  • LongLive2.0-Robot continues from LongLive-2.0’s 16-second AR checkpoint using approximately 10,000 window-equivalent hours of robot and egocentric video, without requiring a shared action space.Teacher-forcing training uses robot-video sequences up to 30 seconds, block-causal attention, and ground-truth prefixes; the model predicts coherent robot motion and interactions from an image and language prompt.Source: §3.1 · pp. 3–4

3.2. Causal-to-Causal World-Action Adaptation

  • The asymmetric video–action interface preserves causal visual dependencies: video queries read visual blocks only, while action queries use observed history, partially denoised futures, and the noisy action chunk.In IDM, the video expert predicts future latents and supplies a visual cache; training uses separate video and action flow-matching passes, while inference reuses that cache through action denoising.Source: §3.2 · p. 4

3.3. Context Scaling of World-Action Models

  • Context experiments vary the duration of the causal prefix while keeping the forecast and action horizon fixed, comparing bidirectional, AR, and robot-domain AR initializations.The sweep tests 0–38.4 seconds on RoboCasa GR-1 and also studies LIBERO-Long; longer prefixes increase prefill, attention, and cache costs.Source: §3.3 · p. 4
  • With the same trigger stride and overlap, streaming VAE encoding meets the handoff deadline, whereas full-window encoding delays handoffs and accumulates robot idle time.The figure expresses time in units of Δt.Source: Figure 3 · p. 5

4. Long-WAM Infrastructure: Real-Time Edge Deployment

Long-WAM co-designs asynchronous scheduling, streaming observation encoding, and edge acceleration to meet handoff deadlines while retaining predictive conditioning. Its infrastructure combines shared inference optimizations with platform-specific tuning.

4.1. Asynchronous Execution

  • On RoboTwin 2.0, Long-WAM nearly preserves synchronous success under asynchrony, while Fast-WAM and LingBot-VA lose 15.4 and 45.5 percentage points.The execution scheme discards the elapsed prefix and runs the new chunk’s aligned suffix at handoff, waiting if inference arrives late.Source: §4.1 · p. 5
  • Streaming causal VAE encoding avoids repeated waits and accumulated idle time, allowing later inference triggers and shorter overlap while retaining the latest observation.Reducing overlap from 12 to 8 control steps lowers overlap RMSE about fourfold and jerk nearly threefold, but tightens the handoff deadline.Source: §4.1 · p. 5

4.2. Efficient Edge Deployment

  • Video-expert linear layers use W4A4 NVFP4, while action compute and KV storage remain BF16; CUDA Graph replay and compilation reduce launch overhead.Within-call KV reuse reduces redundant computation; streaming-VAE state updates remain outside CUDA Graph capture.Source: §4.2 · p. 6
  • Shared input quantization reuses Q/K/V activations and scales across three GEMMs, while online softmax combines attention over separate video and action KV buffers.The combination avoids concatenating the KV buffers; additional memory-access and VAE-layout changes are subject to shape and backend constraints.Source: §4.2 · p. 6
  • Shared optimizations and device-specific tuning yield 3.2–4.1× total speedups over BF16 eager execution.The tuning adapts Quant/GEMM settings and backends to hardware constraints rather than using device-exclusive algorithms.Source: §4.2 · p. 6, Figure 4 · p. 6

5. Experiments

Long-WAM is evaluated across simulation benchmarks and ablations of context length, video initialization, and video–action denoising. The results show strong benchmark performance and distinguish the benefit of longer history from the role of autoregressive pretraining.

5.1. Implementation Details

  • Robot experiments use a Unitree G1 humanoid and a YAM bimanual manipulator; pretraining uses approximately 30,720 GPU-hours on 64 H100 GPUs.Downstream WAMs train on 16 GB200 GPUs.Source: §5.1 · p. 7

5.2. Results on Simulation Benchmarks

  • The evaluation covers LIBERO, RoboTwin 2.0, DOMINO, and RoboCasa; GR-1 hosts the longer-context study, and RoboCasa365 tests compositional tasks with GPT-6 Astra as planner.For the first three simulation benchmarks, the main results use up to 2.4 seconds of context.Source: §5.2 · pp. 7–8

5.3. Ablation Studies

  • On GR-1, robot-domain AR’s lead over bidirectional initialization widens from 3.3 points without history to 17.1 points at 19.2 seconds.The bidirectional variant returns to 61.6% at 19.2 seconds, while both AR variants gain over the 2.4–19.2-second interval.Source: Figure 6 · p. 8, §5.3 · pp. 8–9
  • On RoboCasa GR-1, scaling context from 2.4 to 19.2 seconds raises success from 66.3% to 78.7% with robot-domain AR and from 65.7% to 76.7% with LongLive-2.0.At 38.4 seconds, success falls to 75.2% and 74.2%; 80.4% of sampled history frames are padding, and the paper hypothesizes limited history coverage explains the decline.Source: Table 4 · p. 8, §5.3 · p. 8

5.2. Results on Simulation Benchmarks

  • Long-WAM leads LIBERO at 99.5% average, RoboTwin 2.0 at 94.4% average, and DOMINO at 34.9% SR and 45.1 MS.On DOMINO, it exceeds the strongest baseline on each metric: Fast-WAM by 15.0 SR points and PUMA by 10.1 MS points.Source: Table 3 · p. 8, §5.2 · p. 8

5.3. Ablation Studies

  • IDM reaches 99.5% on LIBERO-Long, versus 94.5% without future-video denoising and 97.8% with co-denoising.IDM predicts future visual latents before denoising actions conditioned on history and prediction, without pixel-level synthesis.Source: §5.3 · p. 9

6. Atomic Execution and Compositional Planning

Long-WAM combines physical execution with high-level planning, and its planner pairing extends performance to unseen skill compositions. Real-world and efficiency evaluations show dynamic and long-horizon task performance alongside low-latency execution.

6. Atomic Execution and Compositional Planning

  • The RoboCasa365 hierarchy keeps the Human300-trained policy checkpoint and 2.4 seconds of visual context unchanged while GPT-6 Astra decomposes goals into atomic instructions.The planner also selects execution-prefix lengths and can issue bounded end-effector corrections; Long-WAM supplies action chunks.Source: §6 · p. 9
  • Long-WAM alone achieves 67.9% Atomic-Seen success, versus 39.6% for π0.5 and 31.5% for GPT-6 Astra alone.Querying the policy every 15 steps yields 84.4% without a planner, close to the hierarchical system’s 85.6%.Source: §6 · p. 9
  • Planner pairing raises Composite-Seen success from 15.8% to 38.8%, Composite-Unseen from 6.1% to 35.0%, and Overall from 31.4% to 54.4%.On unseen compositions, Long-WAM + GPT-6 Astra reaches 35.0%, versus 20.8% for GPT-6 Astra alone and 19.0% for the π0.5 hierarchy.Source: §6 · pp. 9–10

6.1. Real-world Deployment

  • Long-WAM succeeds in 19 of 20 moving-cup stacking trials, maintains 90–100% conveyor grasping success, and reaches 81.7% average success on YAM tasks lasting over 40 seconds.The figure captions identify the dynamic Unitree G1 comparison and YAM long-horizon trials.Source: §6.1 · pp. 10–11, Figure 7 · p. 10, Figure 8 · p. 10

6.2. Efficiency

  • 107.4 ms per action chunk on RTX 5090 includes future-video prediction; Long-WAM is 2.3× faster than Fast-WAM at 244.1 ms.Device-specific tuning yields 107.4, 328.2, and 378.7 ms on RTX 5090, Spark, and Thor, respectively.Source: §6.2 · p. 11, Table 6 · p. 11, Table 8 · p. 11
  • At S=12 (R=24), asynchronous execution reduces Long-WAM success by 0.2 points for IDM and 0.4 for CoD, while Fast-WAM and LingBot-VA lose 15.4 and 45.5 points.With a shorter overlap at S=16, asynchronous success reaches 95.0%, on par with synchronous execution.Source: §6.2 · p. 11, Table 7 · p. 11

A. Discussion and Conclusion

Long-WAM treats recent physical history as useful for control, while assigning hour-scale goals and task decomposition to a higher-level planner. The discussion identifies task and compute as factors in choosing context length and outlines limits of the current evaluation.

A. Discussion and Conclusion

  • The low-level controller is intended to retain motion and recent interaction state, while a higher-level planner handles hour-scale goals, reasoning, and task decomposition.The planner can maintain semantic continuity as Long-WAM executes local objectives using physical context.Source: A. Discussion and Conclusion · p. 17
  • The appropriate context window may depend on the task and available compute rather than follow one universal duration.The discussion frames adaptive context allocation as matching retained history to the interaction and computation to response requirements.Source: A. Discussion and Conclusion · p. 17
  • Each context length was trained as a separate model; test-time window adjustment and real-robot context-length studies remain future steps.At 38.4 seconds, performance decline coincides with 80.4% padding in current datasets, while longer, denser robot recordings are proposed as a route to extending the useful window.Source: A. Discussion and Conclusion · p. 17
  • The observed trends motivate evaluating generative robot policies jointly by what they remember, how well they act, and how quickly they incorporate new observations.The discussion does not prescribe a power law or fixed memory optimum.Source: A. Discussion and Conclusion · p. 17

B.1. Pretraining Data

The paper describes a large video-only pretraining corpus, then details training objectives and inference interfaces for video and world-action modeling. Qualitative examples show task-conditioned robot-video predictions.

B.1. Pretraining Data

  • The pretraining corpus contains 2,294,889 model-ready text-and-image-to-video samples from five dataset families; its reported splits contain 2,251,021 training, 21,931 validation, and 21,937 test samples.The approximately 10,000-hour scale is a window-equivalent estimate using 16 seconds per training sample, not unique raw footage.Source: §B.1 · p. 17, Table 9 · p. 17
  • Video-prediction pretraining uses no action supervision, so its source datasets need not share action coordinates or actuator dimensions.This action-space independence applies to pretraining; transfer to an unseen embodiment requires separate evaluation.Source: §B.1 · p. 17

B.2. Training Objectives and Inference Interface

  • Video pretraining uses paired clean and noisy streams so each target block accesses preceding context and its own noisy tokens, but not later blocks or its clean target.The velocity target corrects latent-input corruption toward the original target; the conditioning image remains clean and outside the loss.Source: §B.2 · p. 18
  • World-action adaptation uses separate video and action passes, masking padded future latents and action steps while preventing action-loss gradients from flowing into the video cache.The action pass uses a detached cache built from the observed prefix and ground-truth future latents forward-noised to σ⋆=0.9.Source: §B.2 · p. 18
  • At inference, partial video rollout starts from Gaussian noise and stops at σ⋆ without ground-truth future frames; future latents remain in latent space and the visual KV cache is reused during action-chunk denoising.Changing robot state, evicting context, or changing temporal positions can invalidate previously computed visual keys and values.Source: §B.2 · p. 18

C. LongLive2.0-Robot Video Predictions

  • Selected LongLive2.0-Robot predictions follow task prompts through coherent arm movements and object transitions, including lid placement, shoe transfer, and drawer opening.The examples support a task-conditioned visual prior for later world-action adaptation.Source: C. LongLive2.0-Robot Video Predictions · p. 18

D. Baseline Details

The baseline details list policy and world-action comparisons, additional benchmark references, and video-model initializations. A figure presents selected task-conditioned prediction examples with prompts and frame indices.

D. Baseline Details

  • Policy baselines include Diffusion Policy, OpenVLA variants, GR00T, π0 and π0.5, UniVLA, X-VLA, InternVLA-M1, and StarVLA.The paper identifies UniVLA as Wang et al.’s Unified Vision-Language-Action Model and PUMA as the method introduced with DOMINO.Source: D. Baseline Details · p. 18
  • World-action baselines include Motus, LingBot-VA and LingBot-VA 2.0, Fast-WAM, AHA-WAM, DreamZero, Cosmos Policy, and ABot-M0.5.RoboCasa365 comparisons also include additional methods and external leaderboard scores, with Qwen-RobotManip scores drawn from its technical report.Source: D. Baseline Details · pp. 18–19
  • Within each Figure 9 row, frames progress from left to right, beginning with the prompt to cover a frying pan with a glass lid.The figure caption states that all frames retain the original field of view.Source: D. Baseline Details · p. 19, Figure 9 · p. 19
  • Video initialization comparisons use bidirectional Wan2.2, LongLive-2.0, and LongLive2.0-Robot; GPT-6 Astra serves as the high-level planner in Appendix 6.Figure 9 shows selected text-and-image-to-video examples with task prompts above each row and zero-based frame indices.Source: D. Baseline Details · p. 19, Figure 9 · p. 19

E. Discussion of Asynchronous Overlap Length

Shorter overlaps reduce RMSE and jerk, but task success does not improve monotonically with smoothness; the highest observed success occurs at an intermediate stride.

E. Discussion of Asynchronous Overlap Length

  • With R=24, increasing trigger stride shortens overlap; RMSE compares aligned predictions within that overlap, while jerk is a per-step proxy measured across chunk handoffs.Jerk uses measured non-gripper joint states and is averaged equally across episodes with valid measurements, including successful and failed episodes.Source: E. Discussion of Asynchronous Overlap… · pp. 19–20, Table 10 · p. 20
  • 95.0% success occurs at S=16; shorter overlaps reduce RMSE and jerk, but success falls to 94.3% at the next tested stride.The full sequence is 94.2%, 95.0%, then 94.3% as the trigger stride increases.Source: Table 10 · p. 20, E. Discussion of Asynchronous Overlap… · p. 20
  • Smoother chunk transitions can come with slower feedback: at fixed control interval, nominal decision frequency is 1/(SΔt), while actions still execute at 1/Δt.The authors interpret the observed success pattern as a trade-off between action continuity and decision frequency.Source: E. Discussion of Asynchronous Overlap… · p. 20

F. IDM versus Co-Denoising: Capability and Inference Latency

Video-first IDM provides an explicit future prediction before action generation and has higher reported average success than co-denoising, while taking longer in optimized inference measurements.

F. IDM versus Co-Denoising: Capability and Inference Latency

  • IDM predicts future visual latents independently of action tokens before denoising actions, whereas CoD jointly updates future video and actions with bidirectional interaction.IDM achieves modestly higher reported average success on LIBERO and RoboTwin 2.0.Source: F. IDM versus Co-Denoising: Capability… · p. 20
  • The latency comparison uses each variant’s trained schedule and cache implementation, so it does not isolate inference order as the sole source of the difference.Timing runs are separate from task-success evaluations.Source: F. IDM versus Co-Denoising: Capability… · p. 20
  • IDM takes 107.4 ms with V4/A4 and 81.8 ms with V2/A2, versus CoD at 90.9 ms and 65.7 ms, respectively.These are optimized RTX 5090 inference latencies including observation VAE computation; IDM reaches 9.3 and 12.2 Hz at the two budgets.Source: F. IDM versus Co-Denoising: Capability… · p. 20, Table 11 · p. 21
  • Observed-video KV reuse reduces V4/A4 latency on RTX 5090 from 112.6 to 107.4 ms, but gives no reduction with V2/A2 on that device.With V2/A2, latency is 81.8 ms with reuse versus 81.3 ms without it; the effect differs on Spark.Source: F. IDM versus Co-Denoising: Capability… · p. 21

G. Context-Dependent Inference Latency

Long-WAM’s infrastructure supports longer-context inference, while selected real-robot visualizations show dynamic interception and long-horizon task execution across Unitree G1 and YAM.

G. Context-Dependent Inference Latency

  • 341.0 ms per action chunk at 19.2 seconds of history is about 3.2× the 107.4 ms latency at 2.4 seconds on RTX 5090.The context window increases eightfold; streaming VAE encoding moves prefix work ahead of the inference trigger.Source: G. Context-Dependent Inference Latency · p. 21

H. Additional Real-World Deployment Visualizations

  • The additional deployment montages are selected rollouts, while aggregate success rates are reported in the main text.The examples retain overview and wrist-camera views, with red annotations for failures and green annotations for successes.Source: H. Additional Real-World Deployment… · p. 21

H.1. Dynamic Composite Manipulation

  • Long-WAM’s illustrated Unitree G1 rollout intercepts a moving green cup and nests it in a blue cup; baseline examples miss during interception.Figure 10 depicts Pi0.5 grasping the blue cup but missing the moving green cup.Source: §H.1 · p. 21, Figure 10 · p. 22, Figure 11 · p. 23

H.2. Moving-Object Grasping at Different Speeds

  • Across illustrated conveyor-grasping rollouts, Long-WAM completes grasps at all four speeds, while baseline examples show missed interceptions at higher speeds.The speeds are 3.0, 4.5, 6.0, and 7.5 cm/s.Source: §H.2 · p. 22, Figure 13 · p. 25, Figure 14 · p. 26

H.3. Long-Horizon Manipulation

  • Long-WAM’s YAM sequences show three long-horizon tasks—brick sorting, placing dumplings in a pan, and stacking bowls—lasting over 40 seconds on average.The sequences illustrate progress through repeated pickup and placement.Source: §H.3 · p. 22, Figure 16 · p. 28
Loading 2610.10528v1…