PaperlayerView exact revision on arXiv

Source-linked AI summary

RealtimeWAM: One-Step Asynchronous World Action Models

Chengtao Lv, Jinyang Du, Shuyi Feng, Yang Yong, Shiqiao Gu, Shunzi Yang, Ruihao Gong, Shen Ren, Tianwei Zhang, Wenya Wang

arXiv:2610.06617v1cs.CVcs.LGcs.RO

TL;DR

MoT-based WAMs still face latency from multi-step action denoising and sequential expert execution, while local consistency alone may not ensure accurate final actions. RealtimeWAM combines teacher-anchored one-step action distillation with block-wise asynchronous expert execution. Across diverse benchmarks, it reports approximately 14∼25× acceleration with an average < 1% accuracy drop, but targets MoT-based rather than shared-backbone WAMs.

  • Problem

    MoT-based WAMs remain slow because multi-step action denoising dominates latency and sequential expert execution adds waiting; local consistency distillation may also leave global action-target deviations.

    Source: §1 · p. 2, §4.1 · p. 5

  • Method

    RealtimeWAM combines teacher-endpoint supervision for one-step action generation with block-wise overlap of Video and Action Expert execution.

    Source: §1 · p. 2

  • Results

    Across LIBERO, LIBERO-Plus, and RoboTwin 2.0, RealtimeWAM achieves approximately 14∼25× inference acceleration with an average < 1% accuracy drop versus original multi-step models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported inference latency is shorter than corresponding execution time, enabling real-time action generation across diverse robot configurations, task horizons, and distribution conditions.

    Source: §1 · p. 2

    RealtimeWAM targets MoT-based WAMs and is not directly applicable to shared-backbone architectures such as DreamZero.

    Source: A LIMITATIONS · p. 14

Abstract

from arXiv · show

World Action Models (WAMs) incorporate visual representations from video generation backbones to guide action prediction. Recent efficient WAMs adopt Mixture-of-Transformers (MoT) architectures and compute video representations once for reuse by the action expert. However, intra-expert iteration (\ie, multi-step action denoising) and inter-expert waiting (\ie, sequential execution of the video and action experts) still limit inference efficiency. To this end, we present RealtimeWAM, an extremely efficient WAM variant with one-step action generation and asynchronous inference, addressing these two bottlenecks. To reduce intra-expert iteration, we propose Teacher-Anchored Consistency Distillation (TACD) to address a local-global error gap: low local consistency error alone does not guarantee accurate final actions. TACD supplements local consistency with explicit supervision from the frozen teacher's multi-step rollout endpoint, enabling accurate one-step action generation. Additionally, we propose Cross-Expert Wavefront Pipelining (CEWP) to eliminate unnecessary expert-level waiting. It overlaps the two experts through block-wise sharing of the video KV cache, synchronizing only immediately before the corresponding action attention consumes it. Extensive experiments across diverse benchmarks (\eg, LIBERO, LIBERO-Plus and RoboTwin) and model variants (\eg, Fast-WAM and Faster-WAM) demonstrate the superiority of RealtimeWAM. Notably, RealtimeWAM maintains near-lossless performance (\ie, $<1\%$ drop) across these benchmarks while delivering significant end-to-end speedup (\eg, $\sim25\times$ on H100). Our code and checkpoints are available via this \href{https://github.com/ModelTC/LightX2V/tree/main/examples/realtimewam}{link}.

1 INTRODUCTION

RealtimeWAM targets the latency of MoT-based WAMs by pairing one-step action generation with asynchronous execution of the Video and Action Experts. It combines teacher-anchored action distillation with block-wise expert pipelining and reports near-lossless results across models and benchmarks.

1 INTRODUCTION

  • WAMs add future visual prediction to action learning, while efficient MoT-based designs compute video representations once for reuse by a separate action expert.This contrasts with shared-backbone WAMs that jointly denoise video and action tokens within one backbone.Source: §1 · p. 1
  • Existing MoT-based WAMs still incur over 300 ms per inference call; multi-step action denoising accounts for nearly 90% of latency, while sequential expert execution adds waiting.The latency analysis identifies intra-expert iteration and inter-expert waiting as distinct bottlenecks.Source: §1 · p. 2
  • Teacher-Anchored Consistency Distillation (TACD) supplements local consistency supervision with the frozen teacher’s multi-step endpoint to constrain global action error for one-step generation.The teacher endpoint serves as a numerical proxy for the exact clean endpoint.Source: §1 · p. 2
  • Cross-Expert Wavefront Pipelining (CEWP) overlaps Video and Action Expert execution through block-wise video KV-cache sharing, synchronizing before the corresponding action attention consumes each block.Each action block needs only its corresponding video block’s KV cache, rather than the Video Expert’s complete output.Source: §1 · p. 2
  • Across LIBERO, LIBERO-Plus, and RoboTwin 2.0, RealtimeWAM delivers approximately 14∼25× inference acceleration with an average < 1% accuracy drop versus original multi-step models.The reported results include Fast-WAM and Faster-WAM across diverse downstream benchmarks.Source: §1 · p. 2

2 RELATED WORK

Related work spans WAM architectures, few-step distillation, and asynchronous robot-policy inference. RealtimeWAM focuses on one-step action distillation for MoT-based WAMs and block-level asynchrony.

2.1 WORLD ACTION MODELS

  • Shared-backbone WAMs model video and actions together, while MoT-based WAMs use separate video and action experts; prior efficient variants can still degrade on challenging benchmarks or offer limited speedups.Fast-WAM reuses video representations computed once by the Video Expert.Source: §2.1 · p. 3

2.2 FEW-STEP DISTILLATION

  • Flash-WAM [Akbari et al., 2026] addresses modality-dependent noise regimes but retains future-video denoising and shows noticeable one-step degradation on RoboTwin and real-world tasks.These limitations motivate one-step action distillation for MoT-based WAMs.Source: §2.2 · p. 3

2.3 ASYNCHRONOUS INFERENCE FOR ROBOT POLICIES

  • Prior asynchronous robot-policy methods operate at execution, module, or sampling level, whereas RealtimeWAM introduces finer-grained block-level asynchrony.The three prior categories respectively overlap action chunks, decouple modules, or assign distinct sampling schedules.Source: §2.3 · p. 3

3 PRELIMINARIES

Efficient MoT-based WAMs compute video representations once and use them to condition iterative action generation. Consistency distillation trains a student to map noisy action chunks toward clean endpoints, enabling few-step generation.

3.1 EFFICIENT MIXTURE-OF-TRANSFORMERS (MOT)-BASED WAMS

  • MoT-based WAMs use separate Video and Action Experts whose corresponding blocks interact through shared attention; efficient variants run the Video Expert once to extract block-wise keys and values.The Action Expert uses the video representation to parameterize the action distribution, which is generated through n denoising steps.Source: §3.1 · p. 4

3.2 CONSISTENCY DISTILLATION FOR ACTION GENERATION

  • In consistency distillation, training data action chunks are mixed with Gaussian noise at a sampled time, and the student and EMA target learn an endpoint mapping.Noisy action chunks on the same teacher trajectory map toward a common clean endpoint, enabling few-step generation.Source: §3.2 · p. 4
  • For neighboring time steps, one numerical integration step of the frozen teacher is approximated by treating its velocity as locally constant.This yields the first-order approximation used for the teacher step.Source: §3.2 · p. 4

4 REALTIMEWAM

RealtimeWAM targets both multi-step action denoising and sequential expert waiting in MoT-based WAMs, using TACD for one-step action generation and CEWP for asynchronous execution.

4.1 TEACHER-ANCHORED CONSISTENCY DISTILLATION

  • Consistency distillation can reduce local error without considerably reducing deviation from the global target action, leaving a local-global error gap.The paper notes that deviations from clean action targets can induce inaccuracies in robotic control.Source: §4.1 · p. 5
  • TACD supplements local consistency with supervision from the teacher’s interval-averaged velocity, directly constraining combined endpoint error.For 0 < t ≤ 1, the paper gives an upper bound on expected squared endpoint error that also depends on teacher integration error.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 CROSS-EXPERT WAVEFRONT PIPELINING

  • Action attention is the operation in an action block that directly consumes its corresponding video KV cache, so it need not wait for the complete cache.The dependency analysis identifies block-level cache availability as sufficient for the corresponding action attention.Source: §4.2 · p. 7
  • CEWP overlaps the Video and Action Experts with block-wise KV-cache release, synchronizing only before action attention consumes each entry.The video stream signals cache readiness after projection while the action stream continues its projection independently.Source: §4.2 · p. 7

5 EXPERIMENTS

Across LIBERO, RoboTwin 2.0, and LIBERO-Plus, RealtimeWAM preserves success with one-step generation, while ablations show gains from endpoint anchoring and latency reductions from combined optimizations.

5.1 IMPLEMENTATION DETAILS

  • Experiments use frozen Video Experts and rank-128 Action-Expert LoRA; TACD uses a 10-step teacher rollout, λ=0.2, and 30,000 training iterations.Latency evaluation includes VAE encoding but excludes text encoding, which is performed once per episode.Source: §5.1 · p. 7

5.2 EVALUATION BENCHMARKS

  • Evaluation covers four LIBERO suites, 50 RoboTwin tasks under clean and randomized conditions, seven LIBERO-Plus perturbations, and real-world dual-arm manipulation.Task success rate is the evaluation metric.Source: §5.2 · pp. 7–8

5.3 MAIN RESULTS

  • RealtimeWAM† scores 92.64% on RoboTwin, 0.29% below 10-step Faster-WAM; RealtimeWAM* scores 97.0% on LIBERO, matching 10-step Fast-WAM; RealtimeWAM† reaches 73.0% on LIBERO-Plus, 0.6% below Faster-WAM.Each result uses one-step action generation.Source: §5.3 · p. 8

5.4 ABLATION STUDY

  • Fine-tuning the Video Expert lowers success and adds training overhead, while frozen-video LCD+LTA raises success from 89.85% to 90.84%.LCD and LTA are the consistency and teacher-anchored losses, respectively.Source: Table 3 · p. 9, §5.4 · p. 9
  • On RoboTwin, K=5, 10, and 20 yield 90.39%, 90.84%, and 90.77%; K=10 balances policy performance with teacher-rollout cost.The authors use K=10 by default.Source: Table 4 · p. 9, §5.4 · p. 9
  • With all optimizations, latency is 12.2 ms on Fast-WAM (24.55× speedup) and 16.1 ms on Faster-WAM (13.56×).The latency measurements are on a single NVIDIA H100 GPU.Source: Figure 5 · p. 9, §5.4 · p. 10

6 CONCLUSION

RealtimeWAM combines teacher-anchored one-step action distillation and block-wise asynchronous expert execution to accelerate MoT-based WAMs while retaining near-lossless policy performance.

6 CONCLUSION

  • Across Fast-WAM and Faster-WAM, RealtimeWAM reports near-lossless policy performance with 12.2 ms example latency and approximately 14–25× H100 speedups.The framework targets intra-expert iteration and inter-expert waiting.Source: §6 · p. 10

A LIMITATIONS

RealtimeWAM targets MoT-based WAMs, but it is not directly applicable to shared-backbone architectures. CEWP supports both MoT variants that omit and retain future-video prediction at inference.

A LIMITATIONS

  • RealtimeWAM is not directly applicable to shared-backbone WAMs, though it supports MoT models with or without future-video prediction at inference.The limitation is tied to shared-backbone step distillation more closely resembling video-generation distillation, making action-specific characteristics harder to isolate.Source: A LIMITATIONS · p. 14
  • Teacher-anchored loss adds multi-step frozen-teacher rollouts, increasing training time about 50% while reducing peak GPU memory 62.5% versus joint fine-tuning.Training time rises from 5h 21min to 8h 3min; peak memory falls from 65.98 to 24.73 GiB.Source: A LIMITATIONS · p. 14, Table A.1 · p. 14

B CROSS-EXPERT WAVEFRONT PIPELINING: PSEUDOCODE

  • CEWP schedules paired Video and Action Expert passes on separate streams, publishing each video KV cache after fusion and waiting only before corresponding action attention consumes it.Video-conditioned blocks are all blocks for Fast-WAM and {1, 5, ..., N} for Faster-WAM; intervening action blocks need no cross-expert synchronization.Source: B CROSS-EXPERT WAVEFRONT PIPELINING:… · p. 15

C.1 LOCAL CONSISTENCY AND ENDPOINT ERROR

The appendix distinguishes local consistency error from global target error, then relates teacher-anchored velocity supervision to endpoint supervision and bounds total error. It also argues that CEWP preserves sequential inference outputs under identical computation.

C.1 LOCAL CONSISTENCY AND ENDPOINT ERROR

  • Small expected local consistency error alone does not ensure small total endpoint error without control of global target error.If expected local error is zero, expected total squared error equals expected global squared error.Source: §C.1 · p. 16

C.2 EQUIVALENCE TO WEIGHTED ENDPOINT SUPERVISION

  • The teacher-anchored velocity loss is exactly a weighted endpoint loss, with 1/t^2 inside the expectation, for any teacher solver.The equivalence follows from the definition of the teacher's interval-averaged velocity.Source: §C.2 · p. 16

C.3 TOTAL ENDPOINT ERROR BOUND

  • With uniform L and M, the teacher numerical-integration contribution is O(K^-2); increasing rollout steps K tightens its error bound.For L = 0, the bound is Mt^2/(2K). The combined local-plus-global error bound does not require each component to decrease separately.Source: §C.3 · p. 17

D INFERENCE EQUIVALENCE OF ASYNCHRONOUS EXECUTION

  • With identical initial states and unchanged operators, CEWP produces identical expert states and action outputs to sequential execution in exact arithmetic.CEWP is another topological ordering of the original computation graph and requires no architectural modification or additional training; it is independent of TACD.Source: D INFERENCE EQUIVALENCE OF… · p. 17
  • For sparse conditioning, the equivalence still applies after required KV fusion, and retaining future-video prediction preserves the video output.Non-conditioning action blocks require no video KV.Source: D INFERENCE EQUIVALENCE OF… · p. 17

E INFERENCE LATENCY ANALYSIS

Latency analysis models expert execution and the overlap achieved by CEWP, while describing result aggregation and implementation optimizations. Ideal overlap is only a reference because dependencies and shared GPU resources constrain it.

E INFERENCE LATENCY ANALYSIS

  • TACD reduces action passes to one, while CEWP overlaps the remaining Video and Action Expert passes; block dependencies and shared GPU resources limit attainable overlap.The model excludes common costs such as VAE encoding and includes additional synchronization, scheduling, and resource-contention costs in Δ.Source: E INFERENCE LATENCY ANALYSIS · pp. 17–18

F.1 RESULT SOURCES AND AGGREGATION

  • For prior-work results, Tabs. 1 and 2 retain aggregates reported in Light-WAM, Flash-WAM, and Faster-WAM; the appendix also specifies evaluation steps and Overall aggregation.Overall averages four suite-level success rates on LIBERO, Clean and Random success rates on RoboTwin 2.0, and trial-weighted perturbation-category rates on LIBERO-Plus.Source: §F.1 · p. 18

F.2 EFFICIENT KERNEL IMPLEMENTATION

  • Five operator-level optimizations fuse kernels and projections, adjust attention masks, and preserve specified precision behavior; CUDA Graph separately reduces CPU submission overhead.The listed operations include modulation with residual gating, Q/K RMSNorm, RoPE conversions and rotation, and self-attention Q/K/V and cross-attention K/V projections.Source: §F.2 · p. 18

F.3 DETAILED LIBERO RESULTS

Detailed LIBERO results show one-step variants matching or exceeding 10-step overall success rates, with suite-level variation. Additional studies compare teacher rollout intervals and report latency gains on two more GPUs.

F.3 DETAILED LIBERO RESULTS

  • 97.0% overall success for RealtimeWAM* matches 10-step Fast-WAM, while RealtimeWAM† reaches 99.0%, comparable to 10-step Faster-WAM at 98.9%.RealtimeWAM† improves Goal success over Faster-WAM by 1.2 percentage points (99.6% vs. 98.4%), with modest reductions on Spatial and LIBERO-10.Source: §F.3 · p. 18, Table F.1 · p. 19

F.4 ADDITIONAL ABLATION STUDY

  • 90.84% overall success makes the t → 0 teacher rollout interval outperform both 1 → 0 and t → r, so t → 0 is the default.Here, 1 denotes pure noise, 0 the clean endpoint, and t and r < t intermediate noise levels.Source: §F.4 · p. 19

F.5 LATENCY ON ADDITIONAL GPUS

  • With all optimizations, Fast-WAM speeds up 14.1× on RTX 5090 and 17.7× on RTX 4090D; Faster-WAM speeds up 6.9× and 8.1×, respectively.The authors note their kernels are optimized primarily for NVIDIA H-series GPUs, which may explain smaller gains on these RTX GPUs.Source: §F.5 · p. 19

F.6 REAL-WORLD EVALUATION OF LONG-HORIZON GARMENT FOLDING

RealtimeWAM* is evaluated on long-horizon, real-world T-shirt folding with two PiPER arms and inference on an RTX 4090. Consecutive actions proceed without perceptible pauses, supporting smooth, low-latency execution through the folding sequence.

F.6 REAL-WORLD EVALUATION OF LONG-HORIZON GARMENT FOLDING

  • The bimanual evaluation used two AgileX PiPER arms and required successive grasping and folding decisions as garment geometry and available grasp locations changed.Because early misalignment can affect later folds, the task probes long-horizon deformable-object manipulation.Source: §F.6 · pp. 19–20
  • The rollout progressed from an unfolded T-shirt through successive folds to a compact final configuration.Figure F.1 shows four keyframes, and the consecutive actions appear smooth and low-latency.Source: Figure F.1 · p. 20
  • Consecutive actions proceeded without perceptible pauses during real-world T-shirt folding, sustaining continuous control through the long-horizon sequence.The evaluation used RealtimeWAM* distilled from Fast-WAM, with TACD for one-step action generation and CEWP for block-wise overlap between the Video and Action Experts; policy inference ran on an RTX 4090.Source: §F.6 · p. 20, Figure F.1 · p. 20

Related papers

Loading 2610.06617v1…