PaperlayerView exact revision on arXiv

Source-linked AI summary

Light-WAM: Efficient World Action Models with State-Fusion Action Decoding

Ziang Li, Dongzhou Cheng, Yibin Wang, Shiyue Wang, Xiaoyang Xu, Lingxuan Weng, Juan Wang, Jiaqi Wang

arXiv:2606.08242v1cs.CV

TL;DR

Efficient deployment of World Action Models is limited by the cost and latency of large generative architectures. Light-WAM retains future-video supervision through compact latent representations and single-pass action decoding, achieving strong benchmark performance with substantially lower resource use.

  • Problem

    Current World Action Models couple future-video prediction and action generation in large generative architectures, limiting efficient training and low-latency closed-loop deployment.

    Source: §1 · p. 1

  • Method

    Light-WAM combines a compact video backbone, downsampled latent-space future-video supervision, and a StateFusionActionExpert for single-pass action-chunk prediction.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Results

    Light-WAM maintains strong LIBERO performance and achieves 76.4% average success across 50 RoboTwin 2.0 tasks while reducing trainable parameters to 0.44B and inference latency to 72.03ms.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Light-WAM provides a favorable performance-efficiency trade-off for robot manipulation across LIBERO, RoboTwin 2.0, and real-world dual-arm tasks.

    Source: §4.5 · p. 7, §5 · p. 8

    Larger WAMs and embodied-pretrained policies continue to achieve higher success in more challenging multi-task settings, indicating that model capacity and large-scale embodied data remain important.

    Source: §5 · p. 8

Abstract

from arXiv · show

World Action Models (WAMs) extend robot policy learning by incorporating future prediction as an additional training objective, encouraging the policy to encode task-relevant temporal structure in its representations. Current WAMs often rely on large-scale generative architectures that incur high training costs and inference latency, making them difficult to deploy as efficient closed-loop policies. We propose Light-WAM, a lightweight World Action Model for efficient robot manipulation. Specifically, it is built with a compact video backbone and performs future-video supervision in a downsampled latent space, reducing the cost of video co-training while retaining its benefits for representation learning. For action prediction, Light-WAM introduces the StateFusionActionExpert, which reads adapted states from multiple backbone layers, fuses them through learned-query pooling, and directly predicts action chunks in a single forward pass. This design provides an efficient interface between video backbone representations and robot actions, avoiding the need for heavy generative action experts. Experiments demonstrate that Light-WAM maintains strong performance on LIBERO and achieves usable multi-task performance on RoboTwin 2.0, while using only 0.44B trainable parameters. It also achieves 72.03ms inference latency with 4.1GiB peak GPU memory and improved training throughput.

1 Introduction

Light-WAM is a lightweight World Action Model that retains training-time future-video representation learning while reducing video-supervision and action-decoding costs. It combines downsampled latent-space supervision with StateFusionActionExpert decoding and achieves strong manipulation performance with substantially lower resource use.

Architecture

  • Light-WAM uses a frozen Wan2.1-T2V-1.3B video backbone adapted with lightweight modules for efficient robot manipulation.The compact backbone and lightweight adaptation support the model’s reduced training and inference costs.Source: §1 · pp. 1–2

Experimental Results

  • 97.2% average success on LIBERO is achieved without embodied pretraining, while RoboTwin 2.0 reaches 76.4% average success across 50 tasks.The LIBERO result is competitive with larger WAM baselines, and the RoboTwin result demonstrates usable multi-task performance.Source: §1 · p. 2

Efficiency Results

  • 0.44B trainable parameters, 4.25× training-throughput improvement, 72.03ms inference latency, and 4.1GiB peak GPU memory are reported versus Fast-WAM.Compared with Fast-WAM, trainable parameters decrease from 6.02B to 0.44B, alongside improved throughput and lower inference resource requirements.Source: §1 · p. 2

Efficient World Action Modeling

  • Light-WAM retains future-video co-training benefits while applying supervision in a downsampled latent space to reduce WAM training cost.The approach is motivated by evidence that test-time future-video generation is unnecessary for strong policy performance, with video prediction primarily benefiting training-time representation learning.Source: §1 · pp. 1–2

StateFusionActionExpert

  • StateFusionActionExpert fuses multi-level adapted backbone states through learned-query pooling and predicts action chunks in one forward pass.It bridges video representations and robot actions without test-time future-video generation or a generative action expert.Source: §1 · p. 2

2 Related Work

Related work spans instruction-following Vision Language Action models and World Action Models that augment robot action learning with future-video prediction. Light-WAM pursues efficient WAM policies, drawing on efficiency-focused VLA research and Fast-WAM while improving the overall pipeline rather than performing inference-time video rollout.

Vision Language Action Models

  • VLA models condition robot actions on visual observations and language instructions, supporting instruction-following manipulation and scalable multi-task learning.Recent work, including SmolVLA, also explores compact architectures for more efficient training and deployment.Source: §2 · p. 2

World Action Models

  • WAMs couple robot action learning with future-video prediction to encourage representations of object motion, interaction dynamics, and task progress.Motus, LingBot-VA, and Fast-WAM exemplify this direction.Source: §2 · p. 2

Light-WAM Positioning

  • Light-WAM shares VLA research’s efficiency-oriented goal but applies it to WAM policies, using future-video prediction to shape visual representations for robot control.Its focus is efficient WAM policy design rather than VLA-only deployment.Source: §2 · p. 2
  • Like Fast-WAM, Light-WAM treats video prediction as training-time supervision without executing it during inference, while targeting efficiency across the overall WAM pipeline.This distinguishes the work from approaches centered on inference-time video rollout.Source: §2 · p. 2

3 Methodology

Light-WAM combines a compact, minimally adapted video backbone with downsampled latent future-video supervision and a direct StateFusionActionExpert action interface. The action branch fuses sparse multi-level backbone states through learned-query pooling and predicts action chunks directly from the current observation at inference.

Video backbone adaptation

  • Light-WAM freezes the pretrained Wan2.1-T2V-1.3B backbone, applies low-rank updates, and inserts sparse bottleneck adapters for lightweight robot-domain adaptation.The adapters are placed at selected transformer depths, while low-rank updates modify attention and feed-forward projections.Source: §3.2 · pp. 3–4

Multi-level state interface

  • The action branch reads a small set of adapted states from different backbone levels, exposing multi-granular visual information without decoding from all activations.These selected states form the interface between the video backbone and the action expert while keeping action decoding efficient.Source: §3.2 · p. 4

Latent-space video supervision

  • Future-video supervision operates on spatially downsampled latent videos, while action prediction retains the original-resolution current-observation latent.This separates lower-cost temporal supervision from the higher-resolution observation needed for manipulation.Source: §3.3 · p. 4

StateFusionActionExpert

  • The StateFusionActionExpert compresses each dense backbone state with learned-query pooling, then fuses the resulting level-wise representations into a fixed-width action state.The number of learned queries controls the information bottleneck: too few can discard manipulation-relevant details, whereas too many weaken compression and burden decoding.Source: §3.4 · p. 4

Action decoding and inference

  • At inference, Light-WAM predicts the complete action sequence directly from the current observation without future-video rollout.Training combines future-video supervision with action regression, but deployment uses the direct action pathway.Source: §3.4 · p. 5

4 Experiments

Light-WAM achieves competitive LIBERO performance and usable 50-task RoboTwin 2.0 performance without embodied pretraining. It also substantially reduces training and inference costs while retaining effective latent-space video supervision and single-pass action decoding.

LIBERO

  • 97.2% average success ranks Light-WAM first without embodied pretraining and third overall on LIBERO.Its suite success rates are 98.2% Spatial, 99.6% Object, 97.8% Goal, and 93.0% Long.Source: §4.2 · p. 5

RoboTwin 2.0

  • 76.4% average success on RoboTwin 2.0 demonstrates usable performance across 50 bimanual manipulation tasks without embodied pretraining.Light-WAM outperforms π0 and X-VLA and is competitive with Motus without embodied pretraining, though it trails Fast-WAM and the strongest embodied-pretrained WAMs.Source: §4.3 · pp. 5–6

Efficiency

  • 13.7× fewer trainable parameters and 3.4× fewer total training-time parameters than Fast-WAM reduce Light-WAM’s training burden.Trainable parameters decrease from 6.02B to 0.44B, total training-time parameters from 6.73B to 1.99B, peak memory from 70.7GiB to 43.1GiB, and throughput rises from 0.49 to 2.08 steps/s.Source: §4.4 · p. 6
  • 72.03ms overall inference latency with 4.1GiB peak GPU memory makes Light-WAM substantially faster and lighter than prior WAM methods.The action branch takes 2.1ms, while the reported latency includes VAE encoding and policy forward with cached language context.Source: §4.4 · p. 7, Table 4 · p. 7

Ablations

  • 99.0% LIBERO-Spatial success with original-resolution video latents improves over 98.2%, but substantially increases training cost.Adding adapter layers from 3 to 5 changes success from 98.2% to 98.0%, whereas reducing learned queries from 16 to 8 decreases success to 95.4%.Source: §4.5 · p. 7

Qualitative analysis

  • Downsampled latent predictions remain smoother than reference frames while capturing main motion and scene changes, indicating useful temporal information.The qualitative analysis compares predicted and reference future frames at t = {+8, +16, +24, +32}.Source: §4.6 · p. 8

5 Conclusion and Limitations

Light-WAM combines compact video modeling, latent-space video supervision, and StateFusionActionExpert decoding to improve World Action Model efficiency. Experiments indicate a favorable performance-efficiency trade-off, while challenging multi-task settings remain a limitation.

Conclusion

  • Light-WAM is presented as a lightweight World Action Model for efficient robot manipulation.The paper evaluates it on LIBERO, RoboTwin 2.0, and real-world dual-arm tasks.Source: §5 · p. 8
  • Compact video backbones, downsampled latent-space video supervision, and StateFusionActionExpert decoding improve WAM training and inference efficiency.These components are combined in the proposed Light-WAM design.Source: §5 · p. 8

Limitations

  • A favorable performance-efficiency trade-off is demonstrated on LIBERO, RoboTwin 2.0, and real-world dual-arm tasks.The passage identifies larger WAMs as relevant in more challenging multi-task settings, indicating a remaining limitation.Source: §5 · p. 8

A Algorithmic Details

Light-WAM freezes the Wan2.1-T2V-1.3B video backbone while adapting it with LoRA and sparse residual bottleneck adapters. Adapted states from layers {8, 16, 24} are pooled and fused by the StateFusionActionExpert for action prediction, while the final backbone output supports future-video co-training.

Backbone adaptation

  • Light-WAM freezes Wan2.1-T2V-1.3B and applies LoRA across self-attention, cross-attention, and feed-forward projections, with sparse WAM adapters at layers {8, 16, 24}.Each WAM adapter is a residual bottleneck module with a 256-dimensional bottleneck.Source: A Algorithmic Details · p. 12
  • The selected adapted states feed the StateFusionActionExpert, while the final backbone output feeds the video prediction head for future-video co-training.The default adapter scale is γ = 1.0.Source: A Algorithmic Details · p. 12

State-fusion

  • The StateFusionActionExpert uses layer-specific learnable queries to pool adapted video tokens from each selected backbone layer.This query-based pooling provides the interface from backbone states to action chunks.Source: A Algorithmic Details · p. 12
  • 16 queries and 8 attention heads are used per layer, with pooled states projected to 4608 dimensions and concatenated into a 6144-dimensional fused state.The fused state is further processed by a residual MLP block before temporal decoding and action prediction.Source: A Algorithmic Details · p. 12

B Training and Implementation Details

Light-WAM training uses AdamW with cached video latents, frozen pretrained components, and task-specific batch sizes on four H100 GPUs. Training jointly optimizes future-video and action objectives, with checkpoints selected at suite-dependent training steps.

Training setup

  • 1 × 10−4 learning rate, 1 × 10−2 weight decay, cosine scheduling, and 1,000 warmup steps define the AdamW training setup.All models are trained on 4 NVIDIA H100 GPUs; global batch sizes are 64 for LIBERO and 128 for RoboTwin 2.0.Source: B Training and Implementation Details · p. 12
  • Cached Wan2.1 VAE latents remove online VAE encoding during training, while evaluation uses online encoding and pretrained video-backbone weights remain frozen.The setup also freezes the VAE, while trainable adaptation modules are optimized around the frozen pretrained components.Source: B Training and Implementation Details · p. 12

Checkpoint selection

  • 60K, 60K, 12.5K, and 80K steps select LIBERO checkpoints for Spatial, Goal, Object, and Long, respectively, while RoboTwin 2.0 uses 460K steps.Checkpoint selection is performed separately for each LIBERO suite.Source: B Training and Implementation Details · p. 12

Parameter breakdown

Training algorithm

  • Training jointly optimizes future-video and action losses, using downsampled noisy video latents for video supervision and multi-level adapted states for action prediction.The RoboTwin 2.0 procedure constructs a three-camera canvas, subsamples frames with stride 4, and uses L = Lvideo + Laction.Source: B Training and Implementation Details · p. 13

C Full RoboTwin 2.0 Results

Table 7 presents the complete per-task results for RoboTwin 2.0.

D Real-World Rollouts

Additional real-world rollouts visualize Light-WAM on three dual-arm tasks and compare its future-video predictions with ground-truth future frames.

  • Figure 5 presents additional real-world rollout frames for Light-WAM.Source: Figure 5 · p. 15
  • The rollouts cover three dual-arm tasks.Source: Figure 5 · p. 15
  • Future-video predictions are shown alongside ground-truth future frames.Source: Figure 5 · p. 15

Related papers

Loading 2606.08242v1…