PaperlayerView exact revision on arXiv

Source-linked AI summary

Faster-WAM: Do World Action Models Need Deep Action Modules?

Liheng Ma, Rui Heng Yang, Zhanguang Zhang, Mateo Clemente, Ziwen Hu, Tongtong Cao, Yingxue Zhang

arXiv:2608.02365v1cs.AIcs.LGcs.RO

TL;DR

Existing WAMs often tie deep action modules to video backbones, creating inference overhead. The paper introduces DoT and Faster-WAM, which use a lightweight action head docked to a video representation hub; Faster-WAM preserves strong control performance and generalization while achieving 66.5 ms latency, a 3.2× speedup over Fast-WAM.

  • Problem

    Existing WAM architectures generally couple action-module depth to video-backbone depth, producing substantial computational overhead and high inference latency.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    DoT treats a pretrained video world model as a representation hub and connects a lightweight task-specific head through KV-Fusion and video–action RoPE alignment.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Faster-WAM achieves competitive control performance and generalization while generating an action chunk in 66.5 ms, a 3.2× speedup over Fast-WAM.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    A single-layer action head can maintain strong control performance and generalization while reducing inference latency to match VLA models.

    Source: §5 · p. 9

    DoT cannot exploit the training-time shared-attention computation used by MoTs, although its lightweight head keeps the resulting overhead negligible.

    Source: §3.1.2 · p. 6

Abstract

from arXiv · show

World Action Models (WAMs) couple robot action prediction with video world models. Existing WAMs with shared-backbone and Mixture-of-Transformers designs generally tie the depth of the action module to that of the video backbone, resulting in substantial computational overhead and high inference latency. To address this limitation, we introduce Dock of Transformer (DoT), a video-centric design principle that treats a pretrained video Transformer as a representation hub and connects lightweight output-heads through docking interfaces. This enables flexible output-head design while providing direct access to representations from all layers of the backbone. We then introduce \textbf{Faster-WAM}, an instantiation of DoT for WAMs, which docks a single-layer action head onto a 30-layer video backbone. The docking interface fuses keys and values from all video layers and applies RoPE realignment. Without additional embodied pretraining, Faster-WAM achieves competitive performance on LIBERO and RoboTwin 2.0 while demonstrating strong out-of-distribution generalization on LIBERO-Plus. Faster-WAM also achieves the lowest end-to-end latency in our controlled comparison, requiring only 66.5 ms per inference --- a \(3.2\times\) speedup over Fast-WAM. Overall, these results demonstrate that the video-centric DoT architecture supports flexible task-specific head design while delivering low inference latency, strong action-prediction performance, and robust generalization.

1 Introduction

WAMs use pretrained video world models for action prediction, but existing architectures often incur high latency by coupling action computation to deep video backbones. DoT addresses this with a representation-hub design, instantiated by Faster-WAM, which uses a lightweight action head while retaining strong benchmark performance and substantially lower latency.

1 Introduction

  • WAMs exploit video-world-model pretraining to model physical dynamics, but this benefit commonly comes with substantial inference cost.Fast-WAM requires 211.7 ms per action chunk, compared with 71.4 ms for π0.5, motivating lower-latency WAMs without sacrificing control performance.Source: §1 · p. 1
  • Shared-backbone WAMs process video and action tokens through a heavy trunk, while MoT designs use separate action DiTs whose depth often mirrors the video backbone.MoT decouples the streams during inference but typically depends on ordered video–action layer correspondence for shared-attention co-training.Source: §1 · pp. 1–2
  • DoT treats the pretrained video world model as a representation hub whose distributed representations are accessed by lightweight task-specific heads through docking interfaces.This video-centric design makes output-head construction independent of duplicating the full backbone computation.Source: §1 · p. 2
  • Faster-WAM docks a lightweight action head onto the full video backbone using KV-Fusion and video–action RoPE alignment.KV-Fusion aggregates keys and values from all video DiT layers, while RoPE alignment reconciles the positional coordinates of video keys and action queries, enabling a single-layer action DiT.Source: §1 · p. 2
  • 66.5 ms per action chunk gives Faster-WAM a 3.2× speedup over Fast-WAM while achieving 98.50% on LIBERO, 89.17% on RoboTwin 2.0, and 75.0% on LIBERO-Plus.The LIBERO-Plus result exceeds Fast-WAM by 23.5 percentage points, and the comparison reports lower latency than representative VLA models.Source: §1 · p. 2

2 Related Work

Related WAM designs differ in how video and action streams interact, but most retain predetermined layer coupling that limits action-head compression. DoT instead provides flexible access to representations across the video backbone through a docking interface.

2 Related Work

  • MoT uses one-to-one video–action layer correspondence, H-Bridge restricts interaction to fixed intermediate layers, and DoT uses flexible docking interfaces to access the video backbone as a representation hub.The architectural comparison contrasts predetermined cross-stream connections with DoT’s task-specific access pattern.Source: Figure 1 · p. 3
  • Fast-WAM lowers latency by skipping test-time future-video denoising, while Light-WAM and Efficient-WAM narrow the VLA latency gap using compact backbones or simplified decoding.The related-work passage contrasts these efficiency strategies with Faster-WAM’s retention of a full-scale pretrained video backbone.Source: §2 · p. 3
  • Most MoT WAMs use action experts whose depth follows the video DiT, making it difficult to reduce action-expert depth while retaining distributed backbone representations.Asymmetric designs relax the coupling but retain predetermined and ordered connections.Source: §2 · p. 3
  • Prior feature-routing methods select fixed backbone depths, whereas DoT flexibly accesses and combines representations across the video Transformer.Faster-WAM instantiates this principle with KV-Fusion, allowing a single-layer action DiT to aggregate keys and values from all video layers.Source: §2 · p. 3, Figure 2 · p. 4

3 Method

DoT treats the video backbone as a representation hub that lightweight task heads can access through flexible docking interfaces rather than fixed layerwise coupling. Faster-WAM instantiates this design with KV-Fusion, RoPE realignment, and a single-layer action head, reducing latency while maintaining competitive performance and generalization.

3.1 Dock of Transformer (DoT)

  • DoT decouples video and task-head depths and layer correspondence while preserving hub–head interaction through explicit docking mechanisms.Unlike MoT, the design does not require the task head and video backbone to have equal depths or a predetermined layer map.Source: §3.1 · p. 4

3.1.1 KV-Fusion Docking Mechanism

  • KV-Fusion remaps video keys and values into the action-head feature space, then performs head-specific cross-layer aggregation with shared coefficients for keys and values.The fused cache is formed from representations collected across all video backbone layers.Source: §3.1.1 · pp. 4–5

3.1.2 Video-Action RoPE Alignment

  • RoPE alignment removes the video keys’ 3D positional basis before fusion, then normalizes and applies the action head’s 1D RoPE for mixed-context attention.Directly reusing video keys would create a mismatch because video keys use 3D RoPE while action queries use 1D RoPE.Source: §3.1.2 · pp. 5–6

3.2 Instantiation of DoT: Faster-WAM

  • Faster-WAM uses a 30-layer video DiT as a representation hub and docks a single-layer action DiT that aggregates representations from all video layers.The action head removes text cross-attention because visual and language information are incorporated through the video backbone and docking interface.Source: §3.2 · p. 6
  • The design substantially reduces inference latency while maintaining competitive prediction performance and strong generalization.Although DoT cannot exploit MoTs’ training-time shared-attention computation, its lightweight head keeps the resulting computational overhead negligible.Source: §3.2 · p. 6, §3.1.2 · p. 6

4 Experiments

Faster-WAM combines a single-layer action head with a full video backbone through cross-layer fusion and positional alignment. Across standard control, latency, generalization, and ablation evaluations, it maintains strong performance while substantially reducing inference cost.

4.1 Implementation Details

  • The implementation uses a 30-layer video DiT hub, a single Transformer-layer action head, cross-layer KV-Fusion, and video–action positional alignment.The action head omits text cross-attention, while language information is incorporated through the video hub and docking interface.Source: §4.1 · p. 6

4.2.1 Results on LIBERO and Inference Efficiency

  • 98.5% average LIBERO success matches or exceeds Fast-WAM across every suite despite using a single-layer action head without embodied pretraining.Faster-WAM gains 2.6 percentage points on LIBERO-Long and matches or outperforms several embodied-pretrained baselines.Source: §4.2.1 · p. 7, Table 1 · p. 7
  • 66.5 ms produces a complete 32-step action chunk, the lowest latency among identically evaluated models and a 3.2× speedup over Fast-WAM.Measured latencies include 68.2 ms for π0, 71.4 ms for π0.5, 105.3 ms for X-VLA, and 211.7 ms for Fast-WAM.Source: §4.2.1 · p. 7, Table 1 · p. 7

4.2.2 Results on RoboTwin 2.0

  • 89.17% average RoboTwin 2.0 success remains close to Fast-WAM while the single-layer action expert performs similarly in clean and randomized scenes.Success rates are 89.70% in clean scenes and 88.64% in randomized scenes; Faster-WAM trails Fast-WAM by 2.47 percentage points on average.Source: §4.2.2 · p. 7

4.2.3 Generalization on LIBERO-Plus

  • 75.0% LIBERO-Plus success exceeds Fast-WAM by 23.5 percentage points and improves across all seven perturbation categories.The largest gains occur for camera perturbations, from 16.4% to 67.9%, and sensor noise, from 37.7% to 82.7%.Source: §4.2.3 · p. 8, Table 2 · p. 8

4.3 Ablation Studies

  • Sequential ablations raise LIBERO-Plus success from 49.5% to 75.0% as final-layer access, KV-Fusion, RoPE alignment, and removal of text cross-attention are added.KV-Fusion increases success to 66.8%, RoPE alignment to 71.3%, and removing text cross-attention to 75.0%.Source: §4.3 · pp. 8–9, Figure 3 · p. 8

5 Conclusion

The paper introduces DoT, which treats a pretrained video world model as a representation hub for lightweight task-specific heads. Faster-WAM instantiates this design with cross-layer fusion and positional alignment, showing that a single-layer action head can preserve strong control and generalization while reducing latency to VLA-like levels.

  • DoT treats a pretrained video world model as a representation hub connected to lightweight task-specific heads through docking interfaces.Source: §5 · p. 9
  • Faster-WAM docks a single-layer action head using cross-layer KV-Fusion and video–action RoPE alignment, decoupling action-head depth from video-backbone depth.Source: §5 · p. 9
  • A single-layer action head maintains strong control performance and generalization while reducing inference latency to match that of VLA models.Source: §5 · p. 9

A.1 Misalignment of 3D-RoPE and 1D-RoPE

The section shows that combining 1D action RoPE with 3D video RoPE breaks the usual relative-position property because the streams use different coordinate systems. DoT restores a common action-side basis by unrotating, fusing, and rerotating video keys.

Misalignment of 3D-RoPE and 1D-RoPE

  • The mismatch arises because corresponding rotary operators, despite pairing adjacent channels, map each 3D axis pair to a corresponding 1D pair, yielding phases from three different coordinate systems.Source: §A.1 · p. 15
  • 3D video RoPE and 1D action RoPE produce cross-modal attention logits that depend jointly on absolute coordinates rather than one relative displacement.Video channels encode temporal, vertical, and horizontal coordinates separately, whereas action RoPE applies one scalar position across rotary pairs.Source: §A.1 · pp. 14–15
  • DoT first recovers canonical unrotated video keys, performs channel and layer fusion there, then reapplies the action head’s 1D RoPE.This unrotate–fuse–rerotate sequence preserves the video representation while expressing it in the positional basis required by mixed-context attention.Source: §A.1 · p. 15
  • After realignment, the fused video key shares the action query’s rotary basis, restoring a well-defined 1D relative displacement for the mixed-context logit.Within a common rotary basis, the positional contribution to attention depends only on relative displacement.Source: §A.1 · pp. 14–15

Related papers

Loading 2608.02365v1…