PaperlayerView exact revision on arXiv

Source-linked AI summary

RealtimeWAM: How Fast Can I Run My World Action Model?

Huanan Liu, Ye Li, Kangye Ji, Xiaoyu Chen, Hanyun Cui, Yutian Shen, Yuan Meng, Chenglei Wu, Jingyan Jiang, Bo Li, Zhi Wang

arXiv:2610.10079v1cs.RO

TL;DR

High WAM inference latency limits responsive robot control, and existing acceleration approaches leave scheduling and changing action demands insufficiently addressed. RealtimeWAM coordinates parallel execution with adaptive computation; it reports large speedups with near-native simulation success and higher success across five real-world tasks.

  • Problem

    High WAM inference latency limits responsive robot control, while existing computation reduction and caching do not address scheduling constraints and changing action precision demands.

    Source: §1 · p. 2

  • Method

    RealtimeWAM coordinates layerwise parallel execution with observation-feature reuse and motion-adaptive Transformer residual refinement, without additional training.

    Source: §1 · p. 2

  • Results

    FastWAM and OpenWAM achieve average speedups of 8.90× and 10.67×, with success rates decreasing by 0.02 and 0.53 percentage points, respectively, versus native inference.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across five real-world tasks, average success rates improve over native inference by 17.2 percentage points on FastWAM and 37.2 percentage points on OpenWAM.

    Source: §5 · p. 9

    The described independence of observation tokens assumes first-frame-causal attention and fixed clean-frame conditioning.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

World Action Models (WAMs) combine visual dynamics modeling with action generation, but their high inference latency limits responsive robot control. Recent efforts accelerate inference by removing explicit future-video generation at test time, as in FastWAM, an approach that requires a specially tailored architectural design. More general caching strategies exploit feature redundancy, but redundancy alone does not capture the changing computational demands of closed-loop control. To address these challenges, we present RealtimeWAM, a general, training-free framework that coordinates parallel execution with adaptive computation for low-latency inference across diverse WAM architectures. We exploit layerwise dependencies to overlap observation processing with prediction. However, concurrent branches still compete for GPU resources, limiting the benefit of parallel execution. We therefore adapt computation throughout the pipeline through selective reuse, caching observation features in visually stable regions and reusing Transformer residuals while reserving additional refinement for small predicted adjustments. We evaluate RealtimeWAM on FastWAM and OpenWAM across RoboTwin, LIBERO, and LIBERO-Plus. On an RTX 4090, measured mean inference latencies are 24.09 and 63.09 ms, corresponding to average speedups of 8.90$\times$ and 10.67$\times$. Average success rates are 82.75% and 87.41%, respectively, within 0.02 and 0.53 percentage points of native inference. Across five real-world tasks, RealtimeWAM improves average success rates over native inference by 17.2 and 37.2 percentage points on FastWAM and OpenWAM, respectively.

1 INTRODUCTION

RealtimeWAM targets WAM latency by coordinating prediction with observation processing and adapting computation to resource contention and changing motion precision needs. Its evaluation reports large speedups with nearly unchanged simulation success.

1 INTRODUCTION

  • High WAM inference cost can delay robot responses as scenes change, while existing simplification and caching leave scheduling, action-specific precision, and changing computational demands unresolved.A moving object may shift from its observed position before the predicted action executes.Source: §1 · p. 2
  • Layerwise release of observation K/V lets prediction overlap Prefill, but concurrent branches still compete for GPU resources.The schedule overlaps action prediction and, when present, future-video generation with ongoing observation processing.Source: §1 · p. 2, Figure 2 · p. 2
  • Token reuse refreshes observation features based on their changes, using latency plateaus to refresh additional tokens with larger first-layer feature changes.This reduces workload competing with prediction while keeping observation features current.Source: §1 · p. 2
  • Motion-adaptive refinement uses coarse predicted displacement to allocate fewer evaluations to large movements and more to small adjustments.Figure 3 illustrates that the same offset can fit a large movement’s tolerance but exceed a small adjustment’s tolerance.Source: §1 · p. 2, Figure 3 · p. 3
  • FastWAM and OpenWAM achieve average speedups of 8.90× and 10.67×, with success-rate decreases of 0.02 and 0.53 percentage points, respectively.These results combine execution optimizations with adaptive computation.Source: §1 · p. 3

2 RELATED WORK

The related work covers WAM inference, feature reuse, and adaptive computation. RealtimeWAM is positioned as overlapping observation processing with prediction and adapting reuse and refinement to current inputs.

2 RELATED WORK

  • FastWAM omits future-video generation at inference, whereas OpenWAM jointly predicts video and actions.The section also distinguishes Flash-WAM’s distillation of joint prediction to fewer steps (Akbari et al., 2026).Source: §2 · p. 3
  • Prior real-time systems overlap policy inference with action execution, while RealtimeWAM overlaps observation processing and prediction within one request.Source: §2 · p. 3
  • Prior diffusion caching reuses intermediate features, while RealtimeWAM reduces Prefill contention and uses latency plateaus to refresh additional tokens.Related approaches apply reuse at visual-token, block, and diffusion-head levels.Source: §2 · p. 3
  • Figure 4 depicts selected Prefill tokens receiving fresh FFN outputs while other positions reuse cached outputs, with 4F refreshes at Euler updates 0, 3, 6, and 9.Source: Figure 4 · p. 4
  • Adaptive-computation methods include token and channel pruning and motion-guided caching; RealtimeWAM allocates Prefill and refinement budgets from observation changes and predicted motion without training a selector.Source: §2 · p. 4

3 METHOD

RealtimeWAM combines dependency-aware parallel execution with selective reuse and motion-adaptive refinement to reduce latency across a WAM request. The method allocates fresh computation both across observation tokens and across later predictive evaluations.

3 METHOD

  • RealtimeWAM overlaps observation processing with prediction, then uses token reuse and motion-adaptive refinement to allocate fresh computation across each WAM request.The framework coordinates execution order and fresh computation within existing WAM architectures.Source: §3 · p. 4

3.1 ANATOMY OF WAM INFERENCE

  • Request latency is decomposed into full predictive evaluations, light per-step work, and final action postprocessing; Rt controls first-stage workload while Nt controls later refinement.Measured request latency also includes online selection, data movement, and synchronization.Source: §3.1 · pp. 4–5

3.2 DEPENDENCY-AWARE PARALLEL EXECUTION

  • Observation K/V are released after projection so prediction can proceed alongside remaining Prefill work, while fused operations and CUDA Graph replay reduce transfer and launch overhead.Input preparation also overlaps VAE encoding with context preparation; joint attention waits for required K/V.Source: §3.2 · p. 5

3.3 HARDWARE-AWARE TOKEN REUSE

  • RGB changes identify required token refreshes, and feature changes rank additional positions to fill the smallest profiled execution capacity covering required and reserved updates.The refreshed set is reused in subsequent layers, while unselected positions retain cached FFN outputs; action and future-video tokens remain fully computed.Source: §3.3 · pp. 5–6, Figure 5 · p. 5

3.4 MOTION-ADAPTIVE REFINEMENT

  • Motion-adaptive refinement caches Transformer residuals and uses predicted maximum displacement to allocate fewer evaluations to large movements and additional computation to small adjustments.Both schedules share two initial refreshes; the 4F schedule adds refreshes at Euler updates 6 and 9, and the 2F schedule does not.Source: §3.4 · p. 6

4 EXPERIMENTS

RealtimeWAM reduces inference latency on FastWAM and OpenWAM with small average simulation success-rate decreases, while improving average success on five real-world tasks. The experiments compare execution and adaptive-computation components across simulation and real-world settings.

4.1 EXPERIMENTAL SETUP

  • The setup evaluates FastWAM and OpenWAM against native inference and four training-free caching baselines, reporting success rate and full-request latency on an RTX 4090.Measurements use BF16 and batch size one; compilation, graph capture, warmup, profiling, and robot execution are excluded.Source: §4.1 · p. 6

4.2 OVERALL COMPARISON

  • FastWAM latency falls from 214.33 to 24.09 ms (8.90×) with success changing from 82.77% to 82.75%; OpenWAM latency falls from 672.83 to 63.09 ms (10.67×) with success changing from 87.94% to 87.41%.Table 1 reports success rates and mean latency across manipulation benchmarks, with settings equally weighted in the average.Source: §4.2 · p. 7, Table 1 · p. 7
  • TeaCache is the fastest competing baseline at 49.28/245.99 ms on FastWAM/OpenWAM, but has lower average success than RealtimeWAM on both.BAC retains higher OpenWAM success than RealtimeWAM, 87.77% versus 87.41%, at 305.08 ms.Source: §4.2 · p. 7

4.3 SOURCES OF LATENCY REDUCTION

  • With graph replay and compilation, parallel execution reduces first-forward latency from 36.21 to 17.77 ms on FastWAM and from 60.62 to 37.12 ms on OpenWAM.Token reuse further lowers these values to 17.69 and 31.62 ms; the separate 10F request latencies fall from 214.33 to 29.47 ms and from 672.83 to 217.24 ms.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 ABLATION STUDIES

  • Ablations assess token-selection and motion-gating components, while diagnostics compare token-count latency profiles and fixed versus adaptive evaluation budgets.The rollout figure tracks evaluation budgets and token allocation across 50 requests in an OpenWAM block-sorting rollout.Source: Table 3 · p. 8, §4.4 · p. 8, Figure 6 · p. 8

4.5 REAL-WORLD EVALUATION

  • Average real-world success rises from 35.6% to 52.8% on FastWAM and from 33.6% to 70.8% on OpenWAM, while mean inference latency decreases on both.Across five tasks, the success-rate gains are 17.2 and 37.2 percentage points, respectively.Source: §4.5 · p. 9, Table 4 · p. 9, Figure 7 · p. 9

5 CONCLUSION

RealtimeWAM combines parallel execution, hardware-aware token reuse, and motion-adaptive refinement in a training-free framework. It reports substantial inference speedups with near-native simulation success rates and higher average success across five real-world tasks.

5 CONCLUSION

  • RealtimeWAM combines parallel execution, hardware-aware token reuse, and motion-adaptive refinement without additional training.On FastWAM and OpenWAM, average speedups are 8.90× and 10.67×, with success rates within 0.02 and 0.53 percentage points of native inference; five-task real-world success improves by 17.2 and 37.2 percentage points.Source: §5 · p. 9

AI USE STATEMENT

  • The authors report manuscript assistance from generative AI for writing, editing, and LaTeX formatting, with authors retaining responsibility for accuracy and integrity.They state that authors reviewed and revised the assisted content.Source: AI USE STATEMENT · p. 10

A.1 PLATFORMS AND TASKS

  • The five real-world tasks use a Franka Research 3 robot and RealSense D435i camera across conveyor, inclined-tabletop, and level-tabletop setups.The conveyor runs at 1 m/min for shuttlecock stacking, carrot placement, and tennis-ball placement; the rolling-can task uses a tabletop edge raised 5 cm.Source: §A.1 · p. 14, Figure 8 · p. 14

A.2 EVALUATION PROTOCOL

The real-world evaluation compares native inference with RealtimeWAM under a shared synchronous policy. In conveyor experiments, success was substantially lower when measured inference latency exceeded approximately 90 ms.

A.2 EVALUATION PROTOCOL

  • Each task uses 50 trials per backbone and inference configuration, with the same synchronous inference and action-execution policy for both methods.Averages weight the five tasks equally.Source: §A.2 · p. 14
  • Each inference request predicts a chunk of 32 future actions, executed by the robot at a 20 Hz control frequency.Inference runs locally, with negligible network transmission latency.Source: §A.2 · p. 14
  • In conveyor experiments, task success was substantially lower when measured inference latency exceeded approximately 90 ms.The paper marks this empirically observed threshold with a dashed line in Figure 1.Source: §A.2 · p. 14

B.1 BENCHMARKS AND COMPARISON SETTINGS

The benchmark comparison covers FastWAM and OpenWAM across RoboTwin 2.0, LIBERO, and LIBERO-Plus. Methods share checkpoints, seeds, and initial scenes within each backbone and setting.

B.1 BENCHMARKS AND COMPARISON SETTINGS

  • Evaluation spans RoboTwin 2.0 Clean and Random, plus Spatial, Object, Goal, and Long categories in both LIBERO and LIBERO-Plus.LIBERO-Plus includes perturbed task variants involving layout, cameras, robot initial state, language, lighting, background, and sensor noise.Source: §B.1 · pp. 14–15
  • Success uses each benchmark's task checks: a rollout succeeds when the goal is satisfied within the evaluation horizon, and otherwise fails.RoboTwin uses 100 rollouts per task and setting, LIBERO uses 50, and LIBERO-Plus uses one per perturbed variant.Source: §B.1 · p. 15
  • For each backbone and setting, methods use the same released model weights, evaluation seeds, and initial scenes; caching baselines and RealtimeWAM use compilation and CUDA Graph replay.Caching baselines retain their original execution schedules, while RealtimeWAM adds parallel execution.Source: §B.1 · p. 15

B.2 METRICS AND AGGREGATION

This section defines how success and latency are aggregated across benchmark settings and specifies the component variants used in ablations.

B.2 METRICS AND AGGREGATION

  • Success rates average rollout outcomes per task; RoboTwin averages task rates within settings, while LIBERO and LIBERO-Plus average within categories before equally weighting four categories.Latency is the mean request time measured during complete closed-loop rollouts.Source: §B.2 · p. 15

B.3 COMPONENT ABLATION SETTINGS

  • Token ablations compare full refresh, random selection at the selected capacity, and refresh without capacity filling.The random variant uniformly samples the same number of positions; without capacity filling, it refreshes the required count rather than the selected execution capacity.Source: §B.3 · pp. 15–16
  • Motion-gate ablations compare fixed two- or four-evaluation budgets with a random budget sampled independently of predicted motion.Both fixed variants use ten Euler updates, and their token refresh counts follow the observation.Source: §B.3 · p. 16

B.4 CATEGORY-LEVEL BENCHMARK RESULTS

Table 6 breaks the benchmark success-rate summaries into their underlying categories and clarifies how the reported averages are formed.

B.4 CATEGORY-LEVEL BENCHMARK RESULTS

  • Table 6 reports category-level success rates underlying Table 1; RoboTwin averages Clean and Random, while LIBERO and LIBERO-Plus equally weight Spatial, Object, Goal, and Long.Each backbone uses the same methods and success-rate evaluations as Table 1.Source: §B.4 · p. 16, Table 6 · p. 16

C.1 EXECUTION SETTINGS AND FIXED BUDGETS

These sections specify execution controls and report diagnostic results for scheduling, caching, and token selection. They examine runtime, numerical agreement, feature correspondence, and selection quality across the stated settings.

C.1 EXECUTION SETTINGS AND FIXED BUDGETS

  • At ten evaluations, Parallel reduces Compile + Graph latency from 450.90 to 223.27 ms on OpenWAM and from 99.68 to 33.52 ms on FastWAM.The dense-execution controls use BF16 on an RTX 4090; timing boundaries include input preparation through CPU action return but exclude sensor acquisition and robot execution.Source: §C.1 · p. 17

C.2 NUMERICAL AGREEMENT

  • Parallel and Native predictions are checked at matched budgets; direct execution and graph replay agree bitwise on the tested FastWAM cases, as do OpenWAM’s official and modified Native controls.The checks vary FastWAM state, instruction, images, and noise.Source: §C.2 · p. 17, Table 8 · p. 17

C.3 COMPILED CACHING BASELINES

  • Table 9 compares original execution with compilation and CUDA Graph replay, preserving each method’s execution schedule and using rollout-mean latency under the Table 1 protocol.Native remains the original implementation for main-table speedup calculations.Source: Table 9 · p. 18, §C.3 · p. 18

C.4 PREPARATION SCHEDULING

  • Preparation overlap saves paired medians of 0.778 ms for VAE/condition overlap and 0.274 ms for early RGB routing, with bitwise-identical actions and selected positions.The fixed-budget historical OpenWAM profiling compares 100-request summaries at 80 tokens.Source: §C.4 · p. 18, Table 10 · p. 18

D.1 CACHE STATE AND FEATURE CORRESPONDENCE

  • Mean RGB/FFN-MAE correlations are 0.827/0.817 at layer 0 and 0.716/0.701 across layers 1–29; small RGB changes can still accompany larger feature changes.The analysis uses 14 FastWAM pairs and 44 OpenWAM pairs; Figures 10 and 11 show typical and weaker spatial correspondence.Source: §D.1 · p. 19, Figure 10 · p. 20, Figure 11 · p. 20
  • Across 44 OpenWAM pairs, layer-0 selection overlaps later layers by 85.4%, versus 46.7% by chance, supporting a shared observation-layer selection.The selected 56 positions have the smallest layer-0 changes.Source: §D.1 · p. 19

D.2 SELECTION AT MATCHED REFRESH COUNTS

  • The matched-count comparisons fix cache history, inputs, noise, and two Transformer evaluations; the displayed spatial masks use an RGB-only diagnostic rule.The full selector is evaluated through the Random tokens variant in Table 3.Source: §D.2 · p. 21
  • At matched refresh counts, RGB-guided selection lowers action MAE by 74.2% on real video and 74.5% in simulation versus random masks.The real-video comparison reuses 37 of 80 visible positions; the simulation comparison refreshes 64 of 120 positions. Figure 12 presents the selection diagnostics.Source: §D.2 · p. 21, Figure 12 · p. 21

D.3 CAPACITY FILLING AND RECURSIVE REUSE

This section evaluates refresh-count capacity filling and recursive reuse for OpenWAM, reporting latency–error trade-offs and the effect of limiting cache age.

D.3 CAPACITY FILLING AND RECURSIVE REUSE

  • The refresh-count sweep uses three observation pairs, timing five rounds per pair across all 120 counts; action MAE is measured against full refresh.Timing includes online selection, packing, cache writes, transfers, and waits; the full-refresh reference has mean MAE 0.00113 against the original dense engine.Source: §D.3 · p. 22
  • Within the middle plateau, filling from 32 to 80 tokens reduces MAE from 0.00503 to 0.00389 (22.7%), with request latencies of 50.71 and 50.36 ms.Reducing refresh from 120 to 80 changes First latency from 36.41 to 30.95 ms and Request latency from 55.94 to 50.36 ms.Source: §D.3 · p. 22
  • At 64 refreshed positions, an age cap raises mean MAE from 0.002571 to 0.004006; increasing refresh to 96 with the cap gives 0.001596.The comparison covers 44 transitions from four OpenWAM adjust_bottle trajectories and measures error against dense FFN computation.Source: §D.3 · p. 22, Table 11 · p. 22

E.1 SAMPLER AND MOTION SCORE

This appendix specifies the cached-residual sampler and motion score, compares 2F and 4F predictions with 10F references, and records visualization settings.

E.1 SAMPLER AND MOTION SCORE

  • After update 3, coarse predicted motion can trigger refreshes at updates 6 and 9; 2F and 4F use two and four full Transformer evaluations across ten Euler updates.The RoboTwin motion probe uses maximum XYZ displacement across 32 predicted positions and a 0.08 m threshold, excluding rotation and gripper coordinates.Source: §E.1 · p. 23

E.2 PAIRED PREDICTION COMPARISONS

  • 30 of 93 requests triggered 4F; for small-motion requests, 2F/4F relative deviation was 0.164/0.040, and refinement reduced deviation on 90/93 observations.The motion gate uses a coarse 2F displacement threshold of 0.08 m.Source: §E.2 · pp. 23–24, Table 12 · p. 23, Figure 13 · p. 24
  • Four evaluations reduced mean executed-prefix action MAE against the 10F reference from 0.007304 to 0.001224, and P95 from 0.010497 to 0.001713.The fixed-budget comparisons share the observation, initial noise, integration schedule, and action normalization.Source: §E.2 · p. 24, Table 13 · p. 24

E.3 VISUALIZATION SETTINGS

  • Figure 3 displays requests 1–25 of a 26-request episode, while Figure 6 covers all 50 requests and ends in task failure.At the 8 cm threshold in Figure 6, 30 requests use 2F and 20 use 4F.Source: §E.3 · p. 24
Loading 2610.10079v1…