PaperlayerView exact revision on arXiv

Source-linked AI summary

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies

Jialei Chen, Kai Wang, Kang Chen, Shuaihang Chen, Feng Gao, Wenhao Tang, Zhiyuan Li, Weilin Liu, Zhuyu Yao, Boxun Li, Yuanbo Xu, Chao Yu

arXiv:2606.15768v1cs.ROcs.AI

TL;DR

VLAs often lack explicit foresight into how actions change scenes, while pixel-space WAMs make future prediction costly and slow. LaWAM predicts compact latent visual subgoals for action generation, achieving competitive performance across simulated and real-world manipulation tasks with substantially lower latency.

  • Problem

    VLAs often predict actions from current context without explicitly modeling how candidate actions change the scene, while WAMs incur costly pixel-level future synthesis and latency.

  • Method

    LaWAM repurposes a latent-action model’s forward decoder to predict action-relevant future observation features as compact latent visual subgoals for policy action generation.

  • Results

    Across simulated and real-world manipulation tasks, LaWAM achieves state-of-the-art or competitive success rates, runs in 187 ms per action-chunk prediction, and reaches up to 24× lower latency than pixel-space WAMs.

  • Takeaways & Limitations

    LaWAM provides action-relevant future context through a compact latent interface while avoiding the latency and parameter cost of pixel-space WAMs.

  • Takeaways & Limitations

    LaWAM is most effective with stable camera views and has limited reliability for moving-camera settings and fine-grained deformable-object dynamics.

Abstract

from arXiv · show

Vision-Language-Action models (VLAs) leverage large-scale vision-language pretraining for semantic robot control, but often lack explicit foresight into how robot actions change the scene. World-Action Models (WAMs) address this limitation by conditioning policies on predicted futures, yet existing approaches typically rely on computationally expensive video generation with substantial pixel-level redundancy. We present LaWAM, a Latent World Action Model that exposes predictive dynamics to robot policies through compact latent visual subgoals instead of reconstructed future video. At the core of LaWAM is a latent-action-conditioned Latent World Model (LaWM). We obtain LaWM by training a latent action model in the latent space of a pretrained vision foundation model and repurposing its forward decoder to predict future observation features for scene evolution. LaWAM then conditions action generation on these predicted latent visual subgoals to enable dynamics-aware robot control. LaWAM achieves state-of-the-art or competitive success rates (SRs) across LIBERO (98.6% SR), RoboTwin (91.22% SR), and real-world manipulation tasks while retaining low-latency inference. LaWAM runs in 187 ms per action-chunk prediction and achieves up to 24x lower wall-clock latency than pixel-space WAMs.

1 Introduction

LaWAM addresses the lack of explicit scene-evolution modeling in VLAs and the inefficiency of pixel-space WAMs by predicting compact latent visual subgoals for dynamics-aware action generation. Its latent world model supports competitive benchmark and real-world performance with substantially lower inference latency.

  • VLAs often generate actions from current visual-language context without explicitly modeling how candidate actions change the scene.
  • LaWAM predicts compact latent visual subgoals representing the scene change required for the next action chunk, avoiding future-image or video synthesis.Operating entirely in latent space enables efficient modeling of action-relevant dynamics without expensive or iterative pixel generation.
  • LaWM repurposes a latent-action model’s decoder to predict future observation features from the current latent state and inferred latent action.The latent-action model uses an inverse-dynamics encoder to infer latent actions from visual transitions.
  • LaWAM first trains LaWM as a latent-action-conditioned world model, then uses latent-action distillation for subgoal-conditioned action generation.
  • 187 ms per action-chunk prediction and up to 24× lower wall-clock latency distinguish LaWAM’s low-latency inference from pixel-space WAMs.LaWAM uses a compact 230M-parameter LaWM, about 95% fewer world-modeling parameters than the 5B WAN backbone, while achieving state-of-the-art or competitive success across simulated and physical tasks.

2 Related Work

Prior work gives robot policies strong semantic grounding or predictive future context, but these capabilities remain distinct: VLAs lack explicit scene-evolution reasoning, while pixel-space WAMs incur costly generation and appearance-level redundancy. Latent action models instead learn compact transition variables from unlabeled videos, with prior work emphasizing their representation properties and transfer.

  • Vision-Language-Action Models: VLAs transfer large-scale vision-language pretraining into robot control, grounding policies in instructions, objects, and task compositions.This semantic prior specifies what the robot should achieve but does not by itself explain how the scene evolves during embodied interaction.
  • World-Action Models: World-Action Models condition action generation on predicted future context, modeling how scenes may evolve under embodied interaction.Pixel-space WAMs provide physical foresight through generated images or videos, but require costly iterative generation and contain substantial appearance-level redundancy.
  • Latent Action Models: Latent action models learn compact transition variables from unlabeled videos using latent inverse dynamics and forward prediction.Prior methods use these variables for cross-embodiment action representations, policy learning, and future prediction, while chiefly studying latent-space properties and downstream transfer.

3 Method

LaWAM replaces pixel-space future generation with compact latent visual subgoals predicted by a latent-action-conditioned world model. It trains the model in two stages: first learning latent dynamics and then using policy-predicted latent actions to guide action-chunk generation.

  • Latent world model: Pixel-space WAMs generate dense future images or videos, whereas LaWAM represents the future in a frozen visual feature space for compact scene-change prediction.LaWAM defines current and future features as u = fψ(o) and uT = fψ(oT ).
  • Latent world model: LaWM infers a latent action z from observed feature transitions and decodes the current feature plus z into the horizon feature, retaining this decoder as the world model.The posterior qϕ acts as a latent inverse-dynamics model, while the decoder predicts future observation features.
  • Stage-one pretraining: The first stage supervises horizon-feature prediction, adds end-effector-state prediction, and regularizes the latent-action space with KL divergence.The auxiliary predictor encourages z to encode embodied motion, and the auxiliary head is discarded after pretraining.
  • Stage-two policy: At deployment, the policy prior predicts the latent action from the observation and instruction, then LaWM decodes it into a latent chunk-level visual future without generating pixels.The future-dependent inverse-dynamics encoder is unavailable at test time, so the policy prior supplies the latent action.
  • Stage-two policy: An Alternate-DiT action expert combines VLM semantic context with latent dynamics context from (u, ˆuT ) when denoising action chunks.Stage-two training combines latent-action distillation, subgoal supervision, and action flow matching while applying Knowledge Insulation to preserve pretrained dynamics.

4 Experiments

LaWAM is evaluated across simulated benchmarks, efficiency, real-world transfer, latent dynamics, and component contributions. It delivers strong performance on LIBERO, RoboTwin, and real-world manipulation while using compact latent subgoals for low-latency dynamics-aware control.

  • Simulated benchmark performance: LaWAM achieves the best average success rate among compared baselines on the four LIBERO suites.The comparison includes VLA, latent-action, and WAM baselines; results use 50 trials per task.
  • Inference efficiency: 230M LaWM parameters replace a 5B WAN backbone, reducing world-modeling parameters by about 95%.The full LaWAM model is 2.3B and reaches 187 ms latency, up to 24× faster than pixel-space WAMs.
  • RoboTwin benchmark performance: LaWAM achieves the best clean-scene average on RoboTwin 2.0 and remains close to the strongest pixel-space WAMs in randomized scenes.RoboTwin 2.0 covers 50 coordinated bimanual manipulation tasks with 100 trials per task.
  • Real-world transfer: LaWAM ranks first across Pick-and-Place, Drawer Opening, and Towel Folding in real-world evaluation.The trials include in-distribution and out-of-distribution spatial configurations, and the advantage is especially clear for dynamic towel folding.
  • Latent-dynamics behavior: Applying the same latent action to unseen environments and embodiments produces coherent latent-space changes.The latent action captures an embodiment-agnostic visual transition while LaWM grounds it in the current latent visual state.

5 Limitations

LaWAM is currently most effective for manipulation with relatively stable camera views, while camera-dominated transitions can disrupt latent action learning. Its data coverage also limits performance on fine-grained deformable-object dynamics.

  • LaWAM is currently most effective in manipulation settings with relatively stable camera views.
  • Camera motion dominated by abrupt shake or large viewpoint changes can prevent LaWM from learning a coherent latent action space.
  • This limitation affects humanoid or mobile robots whose observations are strongly shaped by self-motion.
  • Limited data coverage remains a challenge for fine-grained deformable-object dynamics, such as subtle cloth deformation.

6 Conclusion

LaWAM introduces predictive dynamics into robot policy inference through latent visual subgoals rather than reconstructed pixel-space futures. Its LaWM repurposes a latent action model’s forward decoder to expand predicted latent actions into embodiment-grounded subgoals for action-chunk generation.

  • 6 Conclusion: LaWAM brings predictive dynamics into robot policy inference without reconstructing pixel-space futures.The approach uses latent visual subgoals to provide action-relevant future context.
  • 6 Conclusion: LaWAM repurposes the forward decoder of a latent action model as its Latent World Model.This decoder predicts embodiment-grounded latent visual subgoals from policy-predicted latent actions.
  • 6 Conclusion: The resulting latent visual subgoals support action-chunk generation across simulated and real-world manipulation tasks.They provide action-relevant future context while avoiding pixel-space reconstruction costs.

A Detailed RoboTwin Results … C.3 Physical-Time Alignment for Mixed-Frequency Data

LaWAM exposes dynamics through latent DINO-space subgoals and implements its policy with transformer-based world-model and action-expert architectures. For mixed-frequency data, it aligns action chunks and temporal encodings by physical elapsed time rather than token index.

  • B Qualitative Visualization Protocol: LaWM subgoals are visualized directly in latent DINO feature space instead of reconstructed future pixels.
  • B Qualitative Visualization Protocol: Cosine-similarity heatmaps locate where a selected robot-arm feature is expected to move in the predicted latent subgoal.The method compares an initial-observation robot-arm patch with every patch in the predicted subgoal feature map.
  • C.1 Latent World Model Architecture: LaWM uses distilled DINOv3 ViT-B/16 features with 24-layer transformer inverse-dynamics encoder and decoder modules.The encoder jointly processes flattened visual patches from current and horizon observations to infer a latent action posterior.
  • C.2 Latent World Action Model Architecture: LaWAM follows Qwen-GR00T with the first 16 Qwen3-VL layers, four Alternate-DiT action-expert blocks, and hidden dimension 1024.Its input sequence includes observations, task instructions, latent-action query tokens, optional auxiliary views, and action-query tokens.
  • C.3 Physical-Time Alignment for Mixed-Frequency Data: LaWAM keeps each dataset or embodiment branch at its native control frequency while defining action chunks over a fixed physical interval τ.This makes the horizon represent consistent real elapsed time despite differing numbers of discrete action tokens.
  • C.3 Physical-Time Alignment for Mixed-Frequency Data: Under the same τ, a 5 Hz branch uses fewer action tokens than a 20 Hz branch while both supervise the future visual state after the same physical duration.The approach preserves native temporal resolution and keeps latent subgoals comparable across datasets.
  • C.3 Physical-Time Alignment for Mixed-Frequency Data: Each action token receives a physical-time encoding because token index i is ambiguous across branches with different control frequencies hb.For token i from branch b, the timestamp is computed as tb,i = i/hb; valid native timestamps are retained after padding and masking.

C.4 Mixed-Frequency Training Experiment

The experiment isolates whether physical-time encoding resolves inconsistencies introduced when mixed-frequency demonstrations map identical discrete action indices to different elapsed times. On controlled LIBERO data, encoding largely restores performance lost by joint training without it, approaching the native 20 Hz reference.

  • Experiment design: The study isolates physical-time encoding in mixed-source pretraining, where identical discrete action indices can represent different physical times across control frequencies.A full-mixture analysis is difficult because dataset scale, embodiment, task distribution, camera setup, and language coverage vary simultaneously.
  • Experiment design: Researchers downsample native 20 Hz LIBERO trajectories to 10 Hz and 5 Hz, then co-train LaWAM on the combined 5/10/20 Hz data.This preserves task distribution, embodiment, visual domain, and language instructions while changing the mapping between discrete action indices and elapsed physical time.
  • Results: Without physical-time encoding, joint mixed-frequency training substantially degrades success relative to the native 20 Hz upper-bound reference.Figure 7 reports the comparison under controlled LIBERO training.
  • Results: Adding physical-time encoding largely recovers performance toward the native 20 Hz reference, supporting its role as the temporal coordinate for consistent mixed-frequency training.The result indicates that encoding helps align data whose discrete action indices correspond to different elapsed physical times.

C.5 Training Details … D.2 RoboTwin Protocol

LaWAM’s experiments use specified latent-world and policy-training procedures, standardized evaluation settings, and benchmark protocols for LIBERO and RoboTwin. These protocols combine controlled training schedules with success-rate evaluation across tasks and scene conditions.

  • C.5 Training Details: 100k steps on 16 H100 GPUs train LaWM with AdamW, learning rate 3 × 10−4, weight decay 10−2, and global batch size 1024.The KL regularization weight is β = 10−5, with fixed physical-time horizons of 1.2 s for teleoperation videos and 0.4 s for egocentric human videos.
  • C.5 Training Details: Joint training across 5/10/20 Hz LIBERO versions shows that physical-time encoding resolves control-frequency confusion.The versions are downsampled from the same LIBERO data.
  • C.5 Training Details: λdistill = λwm = 0.1 in all experiments during stage-two policy integration with trajectories carrying explicit language instructions.Egocentric human videos inform LaWM’s dynamics prior but are excluded from policy integration because they usually lack task descriptions.
  • D Experimental Details: 10−4 is the post-training learning rate for each benchmark, using cosine decay.Benchmark-specific schedules and evaluation protocols are provided in the following subsections.
  • D Experimental Details: 1,000 repeated action-chunk predictions on an A100 GPU determine average wall-clock inference latency.Policies use 10 denoising steps unless otherwise specified, and WAM parameter counts exclude the video-diffusion VAE and text encoder.
  • D.1 LIBERO Protocol: 25k steps with global batch size 256 train LaWAM across four LIBERO suites, with success rates reported over 2,000 trials across 40 tasks.The suites are LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long, following the standard protocol and removing failed demonstrations.
  • D.2 RoboTwin Protocol: 100k steps with global batch size 1024 train LaWAM on over 27,500 RoboTwin demonstrations spanning over 50 tasks.Training requires about 20 hours on 64 H100 GPUs, and average success rates are reported over 100 trials per task under clean and randomized settings.
  • D.2 RoboTwin Protocol: Predicted latent subgoals guide action generation without iterative pixel-space video prediction in qualitative LIBERO executions.The latent world rollouts provide compact future dynamics during action execution.

D.3 Real-World Protocol

The real-world evaluation covers single-arm pick-and-place and drawer opening on a Franka Panda, plus long-horizon towel folding on a Quanta X1, with LaWAM strongest among compared methods on towel-folding subgoal prediction.

  • Robot platforms: The protocol uses Franka Panda setups for pick-and-place and drawer opening, and a Quanta X1 bimanual robot for towel folding.An external RGB camera observes the Franka workspace; the Quanta X1 performs the long-horizon cloth task.
  • Data and evaluation: 150 demonstrations train each Franka task, while towel folding uses 280 demonstrations and 30 trials evaluate each task.Trials span seen conditions and unseen test conditions.
  • Pick-and-place: Pick-and-place tests grasping and bowl placement across irregular toys, smooth plastic, size mismatch, randomized initial states, and modest spatial extrapolation.The setup varies object location, bowl location, and object pose, with some positions beyond the training region.
  • Drawer opening: Drawer opening requires grasping a closed-drawer handle and pulling parallel to the drawer motion because misaligned pulls create substantial sliding resistance.The drawer front edge and orientation receive randomized placement and rotational perturbations.
  • Towel folding: Towel folding comprises shaking and flattening, two long-edge folds, and one short-edge fold, stressing bimanual coordination, cloth contact, and temporal consistency.Even an experienced teleoperator takes about 70 s for the full sequence.
  • Results and failures: LaWAM achieves the strongest towel-folding performance among compared methods for predicting robot-arm subgoals, while Fast-WAM and LingBot-VA exhibit distinct failure modes.Fast-WAM failures involve spatial perception near targets, whereas LingBot-VA failures are tied to inference latency during towel manipulation.

D.4 Additional LaWM Rollout Analysis

LaWM rollout analyses show that its predicted latent futures track true future states while diverging from initial observations, and that identical latent actions produce coherent, context-specific changes across environments and embodiments. Real-world visualizations further show action experts following LaWM subgoals during manipulation tasks.

  • Aggregate rollout behavior: 500 LIBERO trajectories show predicted horizon features staying close to ground-truth future features while moving away from initial features.This supports modeling nontrivial latent dynamics rather than simply preserving the current observation.
  • Cross-environment and cross-embodiment rollouts: Applying the same latent action in different visual contexts yields coherent but context-specific latent changes across environments and embodiments.The visualizations support grounding latent actions in the current latent visual state, embodiment, and scene, while the latent action specifies the abstract transition.
  • Real-world manipulation rollouts: Post-chunk observations show the executed robot arm overlapping LaWAM subgoal heatmaps during real-world pick-and-place and drawer-opening tasks.The visualizations cover approaching, grasping, transporting, placing, grasping a handle, and pulling a drawer open on the Franka Emika Panda platform.
  • Real-world manipulation rollouts: The towel-folding rollout tests whether the action expert follows LaWAM subgoals during long-horizon bimanual deformable-object manipulation.The sequence includes shaking out and flattening the towel, two long-edge folds, and one short-edge fold on the Quanta X1 platform.
Loading 2606.15768v1…