PaperlayerView exact revision on arXiv

Source-linked AI summary

Faster-WAM: Efficient Inference-Time Future Conditioning for Robust World Action Models

Weiheng Zhao, Haoyi Jiang, Xin Shi, Liu Liu, Fan Huang, Zhizhong Su, Wei Sui, Xinggang Wang

arXiv:2608.04404v1cs.CV

TL;DR

WAMs use future visual modeling, but it remains unresolved whether those representations are needed during inference for robustness under distribution shift. Faster-WAM reuses future representations through sparse video–action interaction, achieving higher OOD LIBERO-Plus success than Fast-WAM and faster inference than Joint-WAM.

  • Problem

    Whether future representations are essential at inference, rather than only useful as a training signal, remains unresolved; removing them may leave policies without temporal information needed for unseen environments.

    Source: Introduction · p. 1

  • Method

    Faster-WAM computes future representations once and reuses them during action denoising through selective SparseMoT interactions and multi-depth Interval KV-Fusion.

    Source: Introduction · p. 2

  • Results

    73.57% success on OOD LIBERO-Plus versus 49.14% for Fast-WAM, with a 2.21× speedup over Joint-WAM.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Faster-WAM achieves state-of-the-art in-distribution performance and robust OOD generalization in simulated and real-world manipulation while reducing redundant video–action interaction.

    Source: Conclusion · p. 7

    Without inference-time future conditioning, a policy may lose temporal information required to handle unseen environments.

    Source: Introduction · p. 1

Abstract

from arXiv · show

World Action Models (WAMs) improve robot manipulation by learning how the environment evolves beyond the current observation. However, existing approaches face a fundamental dilemma: Joint-WAMs preserve future-aware representations during inference but incur prohibitive computation costs, while efficient alternatives remove future modeling at inference time and may lose the robustness benefits of temporal reasoning. In this work, we revisit the role of future representations in WAMs and show that inference-time future conditioning is critical for generalization under distribution shifts. This observation motivates Faster-WAM, an efficient future-conditioning WAM that preserves future representations while avoiding expensive video-action interaction. Faster-WAM introduces a sparse future-conditioning framework that computes future representations once and selectively reuses them throughout action denoising. Specifically, we propose SparseMoT to replace ubiquitous layer-wise fusion with selective video-action interaction at a compact subset of network stages, and Interval KV-Fusion to aggregate multi-depth future representations without increasing attention complexity. Experiments demonstrate that Faster-WAM achieves a substantially better performance-efficiency trade-off than existing WAMs. On the out-of-distribution LIBERO-Plus benchmark, Faster-WAM improves success rate from 49.14% to 73.57% compared with Fast-WAM, while running 2.21$\times$ faster than Joint-WAM. It further achieves state-of-the-art performance on LIBERO and RoboTwin 2.0, while demonstrating strong robustness in real-world manipulation.

Introduction

The paper asks whether future representations are needed during inference for robust WAMs, rather than serving only as a training signal. Faster-WAM preserves them through selective, efficient interaction and improves OOD performance while reducing latency.

Introduction

  • The paper frames the unresolved question as whether future predictions are only a training signal or whether future representations provide essential information during inference.WAMs extend action prediction with future visual modeling, unlike approaches relying primarily on current observations and language instructions.Source: Introduction · p. 1
  • Joint-WAMs use evolving future representations during inference but incur substantial computation from repeated video-branch execution and dense video–action interaction.Fast-WAM improves efficiency by restricting future modeling to training, but this raises concern that policies may lose temporal information needed for unseen environments.Source: Introduction · p. 1
  • Removing future representations at inference substantially harms robustness under distribution shift; Fast-WAM degrades relative to Joint-WAM.This motivates preserving future representations at inference while making their interaction with action prediction selective and efficient.Source: Introduction · p. 2
  • Faster-WAM computes future representations once, reuses them during action denoising, and applies SparseMoT at selected stages with action-only refinement between interactions.Interval KV-Fusion aggregates representations from multiple video depths without increasing attention complexity.Source: Introduction · p. 2
  • 73.57% success on OOD LIBERO-Plus versus 49.14% for Fast-WAM, with a 2.21× speedup over Joint-WAM.Faster-WAM also achieves state-of-the-art performance on LIBERO and RoboTwin 2.0 and demonstrates strong robustness in real-world manipulation.Source: Introduction · p. 2

Related Work

Prior work spans action policies without explicit future modeling, WAMs that use future visual dynamics, and efficient variants that omit future representations at inference. Faster-WAM instead focuses on selectively reusing future context during action denoising.

Vision-Language-Action Models

World Action Models

  • WAMs incorporate future visual dynamics; Joint-WAMs couple future-video and action generation but require iterative video computation and dense interaction at inference.Fast-WAM and Light-WAM retain future modeling as a training objective and remove future representations at inference (Yuan et al. 2026; Li et al. 2026c).Source: Related Work · p. 2
  • Whether discarding future representations at inference compromises robustness under distribution shift remains underexplored.Concurrent Efficient-WAM compresses future generation, whereas Faster-WAM focuses on selectively exposing multi-depth future representations to the action pathway during denoising.Source: Related Work · p. 2

Method

Faster-WAM constructs a future-aware visual interface once, then reuses it during action generation through selective video–action interaction. SparseMoT limits where actions read video context, while Interval KV-Fusion summarizes information across video depths without lengthening action attention.

Problem Formulation

  • The policy uses the current image, language instruction, and proprioceptive state to predict an action chunk, with the video branch deriving an internal visual representation from the observation.Joint-WAMs update future-video and action states together, while Fast-WAM removes future slots at inference; Faster-WAM instead builds a fixed future-aware interface in one video pass.Source: Problem Formulation · pp. 2–3

Faster-WAM

  • Faster-WAM pairs aligned video and action experts in a Mixture-of-Transformers, using a single video pass to produce a layer-wise K/V hierarchy and compact action-facing summaries.Language and proprioception condition both experts.Source: Faster-WAM · p. 3, Figure 2 · p. 3
  • At inference, future slots initialized from Gaussian noise are processed once at the noisy endpoint to create a fixed video K/V hierarchy for action conditioning.Future-slot flow estimation can encode plausible scene-dynamics cues without reconstructing a rollout.Source: Faster-WAM · p. 3
  • SparseMoT restricts video–action interaction to selected stages; the other action stages refine the action state while carrying previously injected future information.All L action stages remain active, but only selected stages read video K/V.Source: Faster-WAM · pp. 3–4
  • Interval KV-Fusion aggregates each selected stage’s preceding interval of video K/V pairs into one pair, preserving sequence length and avoiding increased attention complexity.The resulting M fused pairs form the reusable action-facing interface.Source: Faster-WAM · p. 4
  • Training jointly learns video and action flow fields with separate flow-matching losses; independent flow-time sampling exposes the action pathway to varied video and action noise levels.The video loss supervises future dynamics, while the action loss encourages control-relevant information in the fused hierarchy.Source: Faster-WAM · p. 4
  • One video-side pass followed by N sparse, cache-conditioned action evaluations replaces N dense joint video–action evaluations at inference.The cached interface is reused at every action-flow step without updating or decoding future-video latents.Source: Faster-WAM · p. 4

Experiment

Faster-WAM is evaluated on manipulation benchmarks, real-world tasks, and inference latency, with ablations examining future conditioning and video–action interaction. Results cover standard and distribution-shift settings.

Implementation Details

  • Faster-WAM uses a 30-layer action Transformer to predict 32 actions per planning step from nine frames, with video–action interaction every four stages (M = 8).Video and action prediction share a flow-matching objective.Source: Implementation Details · p. 4

Main Results

  • Without embodied pretraining, Faster-WAM averages 99.0% on LIBERO and 92.6% on RoboTwin 2.0; in the standard real-world setting, its success rate is 95.8%, versus 90.8% for Joint-WAM and 88.3% for Fast-WAM.Source: Main Results · pp. 5–6
  • 73.6% vs 49.1% for Fast-WAM on LIBERO-Plus; Faster-WAM also reaches 71.1% average success across real-world unseen conditions.On LIBERO-Plus it outperforms Joint-WAM’s 66.3% on six shifts and is comparable on Layout; it is best under every real-world unseen condition.Source: Table 3 · p. 5, Main Results · p. 6, Figure 4 · p. 6

Experiment Setup

  • Evaluation spans LIBERO, RoboTwin 2.0, LIBERO-Plus, and dual-arm real-world tasks; LIBERO-Plus tests seven distribution shifts without additional training.Real-world OOD evaluation varies lighting, backgrounds, and distractor objects for Pick Strawberries.Source: Experiment Setup · p. 5

Main Results

  • Faster-WAM’s inference latency is 252.95 ms, a 2.21× speedup over Joint-WAM’s 559.84 ms and faster than Fast-WAM’s 320.97 ms.SparseMoT reduces repeatedly incurred action-denoising latency from 276.56 ms to 192.11 ms, offsetting the one-time visual-context increase from 27.67 ms to 43.04 ms.Source: Table 4 · p. 6, Main Results · p. 6

Ablation Study

  • Restoring future temporal slots raises average success from 51.00% to 73.57%, with improvements across all seven distribution shifts.The current-only counterpart retains the same interaction design and training protocol, isolating the contribution of future-aware representations at inference.Source: Ablation Study · p. 7, Table 5 · p. 7
  • Removing Interval KV-Fusion lowers average success from 73.57% to 69.99%, while stride 4 reaches 73.57% versus 69.78% for dense stride 1.Replacing SparseMoT with dense MoT yields 69.78%, iterative joint denoising yields 66.27%, and strides 7 and 14 yield 71.05% and 70.05%, respectively.Source: Ablation Study · p. 7, Table 5 · p. 7

Conclusion

Faster-WAM treats future representations as essential inference-time context for robust action prediction under distribution shift. Its sparse conditioning framework preserves this context while reducing redundant video–action interaction.

Conclusion

  • Future representations are essential inference-time context for robust action prediction under distribution shift, not merely an auxiliary training signal.Source: Conclusion · p. 7
  • Faster-WAM uses SparseMoT for interaction at selected stages and Interval KV-Fusion to aggregate multi-depth future information into compact action-facing context.Source: Conclusion · p. 7
  • Faster-WAM achieves state-of-the-art performance on LIBERO and RoboTwin 2.0, with robust OOD generalization in simulated and real-world manipulation and reduced inference latency.Source: Conclusion · p. 7
  • Learning-based selection of video–action interaction stages and graph compilation or custom CUDA kernels are proposed as future directions.Source: Conclusion · p. 7

Training Details of Faster-WAM

Faster-WAM jointly learns future visual prediction and action generation, using a shared optimization recipe and dataset-specific data construction for LIBERO, RoboTwin 2.0, and real-world experiments.

Training Details of Faster-WAM

  • Faster-WAM jointly optimizes the video and action experts with Interval KV-Fusion and the proprioceptive encoder, while supplying task instructions through precomputed language embeddings.Source: Training Details of Faster-WAM · p. 10
  • Joint-WAM and Fast-WAM follow the same training recipe as Faster-WAM to ensure controlled comparisons.Source: Training Details of Faster-WAM · p. 10
  • All datasets use BF16, AdamW, a 1 × 10^-4 learning rate, 1 × 10^-2 weight decay, cosine decay after 5% warm-up, and gradient clipping at 1.0.Video and action flow times are sampled independently; the two timestep-weighted losses receive equal weight, and runs use seed 42.Source: Training Details of Faster-WAM · p. 10
  • Each sample contains 33 consecutive observation-state pairs and 32 action steps, with one current and eight future multi-view observations paired to the action chunk.LIBERO uses 224 × 448 inputs and 10 epochs; RoboTwin and real-world data use 384 × 320 inputs and 5 epochs, with global batch sizes 1,024 and 512, respectively.Source: Training Details of Faster-WAM · p. 10

Effect of Video Denoising Steps

A single video pass at the noisy endpoint supplies reusable future-aware context for action denoising. Further video denoising does not improve LIBERO-Plus success in the reported ablation.

Effect of Video Denoising Steps

  • The default context comes from one video-expert pass at the noisy endpoint τv = 1, where future slots remain initialized from Gaussian noise.Source: Effect of Video Denoising Steps · p. 10
  • The ablation uses the same trained checkpoint and varies video denoising steps from one to ten while reusing the resulting context during action denoising.Source: Effect of Video Denoising Steps · p. 10
  • The default one-pass setting achieves the highest average success rate of 73.57%; using two video steps does not improve performance, yielding 73.24%, while performance generally declines with further denoising and reaches 68.33% at ten steps.Source: Effect of Video Denoising Steps · p. 10
  • The authors interpret endpoint flow-direction estimation as encoding control-relevant future cues before future latents are explicitly resolved.Source: Effect of Video Denoising Steps · p. 10

Details of the Latency Comparison

Latency measurements isolate inference components under fixed hardware and input settings. The component ablation attributes the primary efficiency gain to SparseMoT, while Interval KV-Fusion adds negligible measured overhead.

Details of the Latency Comparison

Details of the Real-World Dataset

The appendix specifies four real-world dual-arm tasks and describes a balanced, synchronized multimodal dataset, including the distribution shifts used for the strawberry OOD evaluation.

Detailed Results on RoboTwin 2.0

The appendix supplements RoboTwin 2.0 average success-rate reporting with per-task results for Faster-WAM and compared methods under clean and randomized evaluation.

Related papers

Loading 2608.04404v1…