PaperlayerView exact revision on arXiv

Source-linked AI summary

TurboWAM: Learning What Matters for Fast World Action Models

Guile Wu, Shuangzhi Li, Xingxin Chen, Chengjie Huang, Sam Hosseini, Tongtong Cao, Yingxue Zhang, Dongfeng Bai

arXiv:2610.33941v1cs.CV

TL;DR

World action models face high inference latency because future-visual generation requires substantial computation, while prior efficiency methods often do not adapt layer use to action relevance. TurboWAM combines action-conditioned layer aggregation and layer-wise action supervision with inference-time layer reduction and caching; experiments report lower latency with competitive task success rates against state-of-the-art methods.

  • Problem

    World action models incur high inference latency from computationally costly future-visual generation, and existing methods often do not adapt layer use to its changing relevance for action prediction.

    Source: §1 · p. 2

  • Method

    TurboWAM adaptively fuses world-model layer representations based on the action denoising state, modulates action-supervision gradients across layers, and reduces inference work through layer limits and caching.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    TurboWAM achieves lower inference latency with competitive task success rates against state-of-the-art methods; on LIBERO, its highest average success rate is 99.0% and its lowest inference latency is 49.7 ms among compared methods.

    Source: §4.3 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    Experiments on robot manipulation benchmarks support TurboWAM as a low-latency approach that maintains competitive task success rates.

    Source: §1 · p. 2, §5 · p. 8

    TurboWAM may face challenges in tasks requiring complex spatial reasoning and intricate object interactions.

    Source: §5 · p. 9

Abstract

from arXiv · show

World action models have demonstrated promising performance in robot policy learning, yet their high inference latency remains a critical bottleneck for real-world deployment. Recent approaches have sought to improve inference efficiency by reducing test-time future-frame imagination or accelerating action prediction. However, these methods often leverage representations from all video world model layers without explicitly considering their varying relevance to action prediction, leading to redundant computation and suboptimal representation utilization. In this work, we propose TurboWAM, a fast world action model that efficiently aggregates structured world representations conditioned on action predictions. Specifically, TurboWAM introduces a structured world representation aggregation module that adaptively fuses representations from different world model layers according to their relevance to the evolving action denoising state. Furthermore, TurboWAM employs a structured action supervision mechanism that adaptively modulates gradients from action supervision across different world model layers during backward propagation, facilitating world representation learning. During inference, by reducing the number of world model layers executed and caching relevant representations across action denoising steps, TurboWAM substantially reduces computational overhead. Extensive experiments on multiple robot manipulation benchmarks demonstrate that TurboWAM achieves low inference latency while maintaining competitive task success rates compared with state-of-the-art methods.

1. Introduction

World action models combine future-visual generation with action prediction, but their inference cost limits real-time robotic control. TurboWAM addresses redundant layer use with action-conditioned representation aggregation and layer-wise action supervision, while reducing inference computation.

1. Introduction

  • World action models combine future visual generation and action prediction, but iterative denoising in large video models creates latency that limits real-time robotic control.WAMs [2] [12] [14] [22] [23] [26] aim to let robots anticipate how the physical world evolves during interaction.Source: §1 · pp. 1–2
  • Existing efficiency methods [13] [15] [18] [23] [27] often use all world-model layers without accounting for changing relevance, risking redundant computation and suboptimal representations.They also typically lack explicit mechanisms to regulate how action supervision shapes different layers, which may leave representations insufficiently aligned with action prediction.Source: §1 · p. 2
  • TurboWAM adaptively aggregates representations from different world-model layers according to their relevance to action prediction at each denoising step.It also limits the layers executed at inference and caches reusable representations; training retains video co-training while omitting future visual prediction at inference.Source: §1 · p. 2
  • In this paper, “structured” refers to layer-wise representation aggregation and action supervision across world-model layers, not geometric structure.The contributions also include experiments on multiple robot manipulation benchmarks showing lower inference latency with competitive task success rates versus state-of-the-art methods.Source: §1 · p. 2
  • TurboWAM adds structured action supervision that modulates action-supervision gradients across world-model layers during backward propagation.The mechanism operates only during backward propagation, adding no forward-inference computation.Source: §1 · p. 2

2. Related Work

World action models use visual generative models to incorporate physical dynamics into action prediction, while recent work has targeted their inference cost. TurboWAM positions itself as a layer-aware alternative that adapts representation fusion to the evolving action denoising state.

2. Related Work

  • WAMs [12] [22] [24] [25] [29] jointly model future visuals and actions, using visual generative models to capture environment dynamics for robot control.VLA models [3] [10] [11], by contrast, typically predict actions directly from visual observations and language instructions.Source: §2 · p. 2
  • Prior efficiency efforts include removing test-time future imagination [23], aggregating layer features with learned fusion [18], compact intermediate states [8], and compression or sparse interactions [13] [15] [27].These approaches reduce inference overhead through different strategies.Source: §2 · p. 3
  • TurboWAM dynamically adjusts layer contributions using the current action denoising state and modulates action-supervision gradients across layers.Its framework overview presents these as complementary designs for lower inference latency with competitive task success rates.Source: §2 · p. 3, Figure 2 · p. 3

3. Methodology

TurboWAM decouples future imagination from inference while retaining video co-training, then uses action-conditioned layer aggregation and structured action gradients to support efficient action prediction. At inference, it executes selected world-model layers once and reuses their cached representations across denoising steps.

3.1. Approach Overview

  • At inference, TurboWAM executes only the first m world-model layers once per control decision, caches their key/value representations, and reuses them across action-denoising steps without future imagination.Video generation and action prediction remain jointly optimized during training.Source: §3.1 · pp. 3–4

3.2. Structured World Representation

  • The layer budget m selects the first m layers for aggregation, allowing the remaining n − m layers to be skipped at inference while all n layers still run for video co-training.The selected layers’ cached keys and values are aggregated across denoising steps.Source: §3.2 · p. 5
  • Fusion weights change with the current noisy action latents: their average produces a layer-relevance score that informs aggregation at each denoising step.The action-conditioned weights are applied to both keys and values; cached representations stay fixed while weights are updated.Source: §3.2 · p. 5

3.3. Structured Action Supervision

  • Structured action supervision modulates action-loss gradients to balance supervision across selected layers while leaving forward aggregation and action prediction unchanged.The mechanism applies only during training; video-loss gradients and other parameters follow ordinary backpropagation.Source: §3.3 · p. 6
  • The backward gradients combine a layer-shared component with layer-specific residuals, with their weights interpolating between the learned fusion distribution and a uniform distribution.The interpolation is controlled by ρ, with 0 ≤ ρ ≤ 1 and μ_com > 0.Source: §3.3 · p. 6

4. Experiments

TurboWAM combines competitive task success with low inference latency on LIBERO and RoboTwin2.0, and its two structured mechanisms outperform the tested alternatives. Evaluation covers benchmark and real-world robot settings, with zero-shot robustness assessed on LIBERO-Plus.

4.1. Experimental Setup

  • Evaluation spans LIBERO, LIBERO-Plus, RoboTwin2.0, and real-world robot manipulation tasks.LIBERO results cover 2,000 trials across 40 tasks; LIBERO-Plus tests zero-shot generalization on 10,030 instances, and RoboTwin2.0 includes clean and randomized conditions.Source: §4.1 · p. 6

4.2. Implementation Details

  • Implementation uses Wan2.2-5B as the pretrained video world model, with action horizon H=32, layer budget m=24, and routing dimension r=16.Latency is measured with horizon 32 and cached instruction embeddings.Source: §4.2 · p. 7

4.3. Comparison with State-of-the-Art Methods

  • TurboWAM records 99.0% average success and 49.7 ms inference latency on LIBERO, and 90.5% average success and 47.1 ms on RoboTwin2.0.These are the highest average success rate and lowest latency among compared methods on LIBERO, and the lowest latency on RoboTwin2.0.Source: §4.3 · p. 7, Table 1 · p. 7
  • On LIBERO-Plus, TurboWAM achieves competitive performance across seven perturbation categories versus state-of-the-art methods, using models trained on original LIBERO suites and evaluated without additional adaptation.Figure 4 qualitatively compares task execution, with each row showing progression from left to right.Source: §4.3 · p. 7, Figure 4 · p. 8

4.4. Ablation Study

  • Structured representation aggregation consistently beats removing its learned static logit, averaging all layers, or substituting Faster-WAM's DoT strategy.Directly averaging world representations yields substantially lower task success rates.Source: §4.4 · p. 8, Table 3 · p. 8
  • Structured action supervision outperforms ordinary supervision, gradient-norm-matched isotropic scaling, and gradient insulation, without adding inference overhead.The mechanism operates only during backward propagation.Source: §4.4 · p. 8, Table 4 · p. 8

5. Conclusion

TurboWAM reduces inference latency while maintaining competitive task success rates through structured representation aggregation and action supervision. The authors note challenges with complex spatial reasoning and intricate object interactions.

5. Conclusion

  • TurboWAM adaptively aggregates world representations conditioned on the evolving action denoising state and modulates action-supervision gradients across world model layers.Experiments report significantly lower inference latency while maintaining competitive task success rates against state-of-the-art methods.Source: §5 · p. 8
  • TurboWAM may face challenges in scenarios requiring complex spatial reasoning and intricate object interactions.The authors identify improving success rates while preserving low inference latency as further work.Source: §5 · p. 9

Related papers

Loading 2610.33941v1…