PaperlayerView exact revision on arXiv

Source-linked AI summary

villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models

Xiaoyu Chen, Hangxing Wei, Pushi Zhang, Chuheng Zhang, Kaixin Wang, Yanjiang Guo, Rushuai Yang, Yucen Wang, Xinquan Xiao, Li Zhao, Jianyu Chen, Jiang Bian

arXiv:2507.23682v3cs.ROcs.AIcs.LG

TL;DR

Latent-action pretraining for vision-language-action models is promising, but improving how latent actions are learned and integrated remains an open challenge. villa-X grounds latent actions in physical dynamics and integrates them into VLA pretraining; it reports strong performance across simulation and real-world tasks, with zero-shot generalization to unseen embodiments and open-vocabulary icons.

  • Problem

    Despite the promise of latent-action pretraining, how to improve latent actions and integrate them more effectively into VLA pretraining remains an open challenge.

    Source: §1 · p. 1

  • Method

    villa-X grounds latent-action learning in physical dynamics and jointly trains latent-action and robot-action experts through diffusion, conditioning robot-action prediction on latent actions.

    Source: §1 · p. 2

  • Results

    villa-X reports state-of-the-art performance across SIMPLER tasks and two real-world robot setups, including a 77.7% average success rate on the Google robot, and zero-shot generalization to unseen embodiments and open-vocabulary icons.

    Source: §1 · p. 2, §4.4 · p. 8

  • Takeaways & Limitations

    The authors present villa-X as a foundation for future research on generalizable robot manipulation policies.

    Source: §1 · p. 2

    The visual-change consistency objective ignores physical dynamics, leaving latents insufficiently grounded when robot states are available.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent works have begun to explore the incorporation of latent actions, abstract representations of motion between two frames, into VLA pre-training. In this paper, we introduce villa-X, a novel Vision-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. We demonstrate that villa-X can generate latent action plans in a zero-shot fashion, even for unseen embodiments and open-vocabulary symbolic understanding. This capability enables villa-X to achieve superior performance across diverse simulation tasks in SIMPLER and on two real-world robotic setups involving both gripper and dexterous hand manipulation. These results establish villa-X as a principled and scalable paradigm for learning generalizable robot manipulation policies. We believe it provides a strong foundation for future research.

1 Introduction

villa-X advances both latent-action learning and its integration into VLA pre-training. It grounds latent actions in robot dynamics and uses them to condition robot-action generation, enabling broad zero-shot generalization and strong task performance.

1 Introduction

  • Latent Action Models encode motion between successive frames as compact pseudo-action labels, enabling VLA pre-training on abundant action-free robot and human video data.The paper identifies two central challenges: improving how latent actions are learned and how they are integrated into VLA pre-training.Source: §1 · p. 1
  • A proprioceptive Forward Dynamics Model grounds latent actions in physical dynamics by predicting future robot states and actions, helping capture control-critical changes such as subtle rotations or gripper movements.The model uses embodiment context to distinguish heterogeneous data.Source: §1 · pp. 1–2
  • villa-X achieves state-of-the-art performance across SIMPLER tasks and two real-world setups, while scaled pre-training enables zero-shot generalization to unseen embodiments and open-vocabulary symbolic icons.The real-world setups include gripper and dexterous-hand manipulation; the latent-action expert transfers knowledge across simulated environments and real-world tasks.Source: §1 · p. 2
  • A joint diffusion policy combines latent-action and robot-action experts, conditioning robot-action prediction on latent actions to transfer information more effectively.The two components are ACT-latent and ACT-robot.Source: §1 · p. 2

2 Related Work

Prior VLA work uses pretrained vision-language models to generate robot actions, but cross-embodiment generalization remains challenging. villa-X advances latent-action modeling and its integration into VLA pretraining.

Vision-Language-Action Models

  • VLAs use pretrained VLMs to predict robot actions directly or through action experts, while cross-embodiment generalization remains challenging despite large-scale training.Unlabeled trajectories with pseudo-labels such as latent actions, language sub-goals, or visual sub-goals are used to address this challenge.Source: §2 · p. 2

Modeling Latent Actions for VLA Pretraining

  • Latent-action research began with game-focused LAPO [59] and Genie [6], followed by Dynamo [14], which used inverse and forward dynamics models to shape state representations.Source: §2 · p. 3
  • villa-X jointly models latent and robot actions through a joint diffusion process, conditioning robot-action generation on latent actions for structured information transfer.This is presented as a contrast with prior approaches that do not jointly model the two action types.Source: §2 · p. 3
  • It integrates latent actions more tightly than LAPA [68] and GR00T [50], incorporates immediate visual context unlike Moto-GPT [10], and avoids Go-1's [1] teacher-forcing inconsistencies.The paper attributes these differences to more robust reasoning at test time.Source: §2 · p. 3
  • Robotic approaches learn latent actions from videos [68] [9], co-fine-tune latent and robot-action labels [10], treat latents as embodiments [50] or discrete conditioning tokens [1], and learn task-centric actions [7].Other work explores continuous latent actions [38], analyzes learned latents [69], or uses supervision for distractor robustness in MuJoCo.Source: §2 · p. 3

3 Method

villa-X combines a physically grounded Latent Action Model (LAM) with an ACTor Module (ACT) that jointly models latent and robot actions. Training proceeds through LAM pretraining, joint latent–robot ACT pretraining, and embodiment-specific finetuning.

3 Method

  • Training uses three stages: LAM pretraining on diverse datasets, ACT pretraining with joint latent–robot modeling, and embodiment-specific finetuning.Source: §3 · p. 3

3.1 Latent Action Model (LAM)

  • A context vector lets proprio-FDM separate embodiment-specific dynamics across datasets while preserving latent-action consistency.Dataset IDs use learnable embeddings, while control frequencies are encoded as sinusoidal features passed through an MLP.Source: §3.1 · p. 4
  • LAM grounds latent actions in physical state transitions, extending visual-change modeling with proprioceptive forecasts; the framework could also use structural cues such as keypoints or hand pose.The proprioceptive prediction term is omitted for human video without proprioceptive labels, and the continuous vector from the VQ codebook center serves as the latent action.Source: §3.1 · p. 4

3.2 Actor Module (ACT)

  • ACT jointly models future latent and robot actions through a factorized policy and conditional flow matching, treating latents as a mid-level bridge rather than only a pretraining initialization as in LAPA [68].The low-level policy conditions on visual-language features, predicted latents, proprioceptive state, and embodiment context; wrist-camera input is optional.Source: §3.2 · pp. 4–5
  • ACT uses VLM, ACT-latent, and ACT-robot experts, with stochastic attention masking to keep robot actions from overly relying on latent tokens.In half of training cases all robot-to-latent attention is masked; otherwise, half of the latent tokens are randomly masked.Source: §3.2 · pp. 4–5

4 Experiments

Experiments show that proprioception-informed latent actions improve prediction and policy pre-training, while villa-X generates plans for unseen embodiments and symbols and outperforms baselines in simulation and real-world evaluations.

4.1 Does our improved LAM learn higher-quality latent actions?

  • The proprio FDM variant yields more low-error action predictions than the variant without it, and its policies outperform that variant and the no-latent baseline.The probing experiment freezes the latent-action models and evaluates an MLP’s maximum L1 error across action dimensions on LIBERO, which was not used to train the latent-action models.Source: Figure 3 · p. 5, §4.1 · p. 6, Table 1 · p. 6

4.2 Can the actor module effectively leverage the pre-trained latent actions?

  • On the same SIMPLER dataset mixture, villa-X significantly outperforms LAPA-style and GO-1-style approaches, supporting its design for incorporating latent actions into VLA pre-training.The LAPA-style variant uses two-stage training with a switched prediction head, while the GO-1-style variant adds a separate autoregressive latent planner.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Zero-shot Generalization for Latent Actor Module

  • Rendered plans follow instructions on an unseen Realman arm and correctly interact with novel symbol concepts, demonstrating zero-shot embodiment transfer and open-vocabulary understanding.ACT-latent generates latent actions from a starting image and language instruction; a separately trained world model renders them as video for evaluation.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4 Evaluating villa-X in Simulation

  • 77.7% on Google and 62.5% on WidowX: villa-X achieves the highest average success rates in SIMPLER, outperforming VLA, video-learning, and latent-action baselines.The gap from the “villa-X w/o latent” ablation indicates that the latent-action expert is essential; the comparison also demonstrates benefits from incorporating human videos into policy learning.Source: §4.4 · p. 8, Table 2 · p. 8

4.5 Evaluating villa-X on Real-world Robots

  • villa-X outperforms all baselines in both task and generalization evaluations on Realman, and outperforms existing baselines on Xarm with the dexterous Xhand.Realman generalization changes block and table-cover colors; Xarm tests seen tasks with object changes or distractors and unseen tasks with unseen objects or backgrounds.Source: §4.5 · pp. 8–9, Table 4 · p. 9, Table 3 · p. 9

5 Conclusion, Limitations, and Future Works · A Additional Implementation Details for LAM · A.1 Architecture Overview

villa-X improves latent-action learning and use in VLA pre-training, with reported generalization to an unseen embodiment and strong performance in simulation and real-world tasks. The appendix outlines the LAM architecture and notes that future work could use a vision-language-informed critic to filter planned trajectories.

5 Conclusion, Limitations, and Future Works

  • villa-X learns higher-quality latent actions, uses them more effectively in its policy, generalizes zero-shot to an unseen embodiment, and performs strongly in simulation and real-world tasks.The conclusion describes the latent expert’s future planning through visual and proprioceptive state planning as effective, while noting that it is not fully explored.Source: §5 · p. 9
  • The latent expert is not fully explored; future work could use a foundation vision-language-informed critic to reject sampled trajectories that do not follow the instruction.The proposed critic would allow multiple latent-expert samples and filter planned trajectories based on language alignment.Source: §5 · p. 9

A Additional Implementation Details for LAM

A.1 Architecture Overview

  • LAM comprises four modules: a Spatial–Temporal Transformer inverse dynamics model, a vector-quantization module, an image-reconstruction forward dynamics model, and a proprioceptive forward dynamics model.Source: §A.1 · p. 16
  • The inverse dynamics model processes default 8 × 224 × 224 video clips using patch size 14 and 12 ST-blocks with hidden dimension 768 and 32 attention heads.Source: §A.1 · p. 16
  • The vector-quantization module maps continuous inverse-dynamics outputs to discrete tokens using a codebook of size 32, with continuous codebook centers passed downstream.Source: §A.1 · p. 16
  • The image-reconstruction forward model predicts a future frame from the current frame and latent action, while the proprioceptive model predicts future robot states and low-level actions from current state, latent action, and embodiment context.The image model is a 12-layer ViT-base; the proprioceptive model is a two-layer MLP with dual output heads.Source: §A.1 · p. 16
  • Rather than predicting one latent action per frame pair, the model processes 8 frames into 7 latent tokens and learns visually and physically grounded representations by reconstructing future frames, states, and actions.Source: §A.1 · p. 16

A.2 Training Details · A.3 Inference Behavior and Diagnostics

The LAM is trained on human egocentric data and robot trajectories with annotation-dependent objectives, while inference uses the IDM and retains FDM branches for reconstruction-based diagnostics.

A.2 Training Details

  • LAM training combines human egocentric data such as Ego4D [21] with robot trajectories such as OpenX [12].Source: §A.2 · p. 16
  • For samples without low-level robot annotations, such as human videos, training excludes the proprio-FDM branch and uses only the visual FDM objective.Source: §A.2 · p. 16
  • Training uses a batch size of 512, a learning rate of 1.5 × 10−4, and a 2000-step linear warmup.Source: §A.2 · p. 16
  • Training lasts approximately 4 days on 128 NVIDIA A100 GPUs, and visual FDM and proprio-FDM receive equal overall-loss weights.Source: §A.2 · p. 16
  • Each latent token is discretized through the VQ module but represented by its continuous codebook center in subsequent network components.Source: §A.2 · p. 16

A.3 Inference Behavior and Diagnostics

  • At inference, only the IDM is required to extract latent tokens from consecutive frames; FDM and proprio-FDM are retained for reconstruction-based diagnostics of future content, robot states, and actions.This analysis helps assess and debug the physical grounding of the latent representation.Source: §A.3 · p. 16

B Additional Implementation Details for Actor Module · C Dataset

The supplied passages describe the actor module’s architecture, embodiment-specific policy head, wrist-camera integration, regularization, and training setup; they provide no dataset details.

B Additional Implementation Details for Actor Module

  • PaliGemma [3] supplies the 3B-parameter vision–language encoder, while separate 18-layer latent-action and robot-action Transformers use hidden dimension 1,024 and 8 attention heads.The latent-action sequence has length N = 6 and the robot-action sequence has length M = 4.Source: B Additional Implementation Details… · p. 16
  • A variant of HPT [63] gives each embodiment separate state- and action-projection layers while sharing all other policy-head parameters.Each expert contains approximately 300 M parameters and is trained from scratch.Source: B Additional Implementation Details… · p. 17
  • A pretrained ResNet-18 [23] encodes wrist-camera features, which a shared cross-attention head maps into 16 tokens; wrist views are randomly masked 50% of training time.Source: B Additional Implementation Details… · p. 17
  • To limit overreliance on latent actions, training applies 50% attention-weight dropout on that stream and randomly masks 50% of its remaining tokens.The stated aim is to encourage a robust policy balancing predicted latent actions with the input image and instruction.Source: B Additional Implementation Details… · p. 17
  • Training samples τ from different beta distributions for latent and robot actions, biasing latent-action timesteps toward noisier regimes.All components are trained jointly with a 5e −5 learning rate, 200-step linear warmup, and gradient clipping at maximum norm 1.0.Source: B Additional Implementation Details… · p. 17
  • Pretraining takes 4 days on 64 NVIDIA A100 GPUs.Source: B Additional Implementation Details… · p. 17

C.1 Data Mixture … D.2 Transfer Video Demonstrations into Robot Actions through LAM and Proprio FDM

The visualizations support villa-X’s cross-embodiment latent-action representations: similar latent actions correspond to similar behaviors, and decoded actions reproduce video demonstrations in simulation. The accompanying sections describe the pretraining data mixture and its preprocessing.

C.1 Data Mixture

  • The pretraining mixture contains 1.6M robot trajectories (223.5M frames) and 3.6M human-video clips.Robot data draws primarily from OpenX and AgiBot; human videos combine nine datasets. Table 5 provides the full mixture breakdown.Source: §C.1 · p. 17, Table 5 · p. 18
  • LAM pretraining uses only the primary third-person camera view, while policy pretraining optionally includes available wrist-mounted views with 50% dropout.The data mixture combines robot data and action-free human videos.Source: §C.1 · p. 17

C.2 Data Preprocessing

  • Data preprocessing uses EgoHOD’s curated Ego4D subset and filters videos for visual quality; brightness, contrast, saturation, and hue are randomly adjusted.Robot proprioceptive states and actions are represented using Euler angles.Source: §C.2 · p. 17

D.1 Image Pairs with Similar Latent Actions

  • Similar latent actions correspond to similar behaviors and low-level actions across humans and different robot embodiments, supporting cross-embodiment manipulation priors.Figure 6 visualizes image pairs sharing the same latent action.Source: §D.1 · p. 17

D.2 Transfer Video Demonstrations into Robot Actions through LAM and Proprio FDM

  • The transfer experiment extracts latent actions from task-demonstration videos, maps them to robot actions with proprioceptive FDM, and executes them in SIMPLER.The figures show source demonstrations above and corresponding simulated results below.Source: §D.2 · p. 17
  • Simulated motions closely reproduce video demonstrations after latent actions are decoded into robot actions, supporting alignment between villa-X’s latent actions and robot actions.The demonstrations are transferred through the proprioceptive FDM in SIMPLER; Figure 7 shows robot-to-robot transfer and Figure 8 human-to-robot transfer.Source: §D.2 · p. 18

D.3 More Ablations on LAM · E Latent Action Expert Visualization

The embodiment context improves latent-action modeling and generalization, while the latent action expert generates instruction-following behaviors from image-based rollouts using only latent actions.

D.3 More Ablations on LAM

  • The zero-shot test uses a Realman arm unseen during training and probes inferred latent actions with a newly trained MLP.The MLP predicts robot actions from latent actions inferred by IDM.Source: §D.3 · p. 18, Table 7 · p. 19
  • Across both experiments, embodiment context improves performance and aids generalization to novel embodiments, with experiments addressing reconstruction loss on a validation set and zero-shot generalization to an unseen embodiment.Source: §D.3 · pp. 18–19, Table 6 · p. 18, Table 7 · p. 19
  • Figure 6 visualizes image pairs with similar latent actions.Source: Figure 6 · p. 19
  • Visual FDM aligns latent actions with visual changes, while proprio-FDM grounds them in physical dynamics and embodiment context disambiguates heterogeneous actions and controllers.The authors hypothesize that this context supports a more consistent, grounded latent-action space across embodiments.Source: §D.3 · p. 19

E Latent Action Expert Visualization

  • The latent expert is evaluated by passing its predictions through an image-reconstruction FDM that predicts future observations, creating an iterative simulated rollout.Source: E Latent Action Expert Visualization · p. 19
  • From RT-1 and Bridge initial images, the latent expert follows diverse language instructions by identifying target objects and predicting actions that move toward them.Generated videos use only latent actions and are shown in Figures 9 and 10.Source: E Latent Action Expert Visualization · p. 19

F More Ablations on policy model · G Simulation Evaluation Details

The ablations report that both the attention mask and embodiment context improve performance on Google Robot and WidowX Robot. The experiments use the same setting as Table 1 in the main paper.

F More Ablations on policy model

  • Both the attention mask and embodiment context improve performance on Google Robot and WidowX Robot.The ablation experiments follow the setting of Table 1 in the main paper.Source: F More Ablations on policy model · p. 19

G.1 SIMPLER Benchmark

The evaluation covers all eight SIMPLER visual-matching tasks across Google Robot and WidowX. Success rates are averaged across randomized environment trials, with published baseline metrics used for comparison where available.

G.1 SIMPLER Benchmark

  • The benchmark spans Google Robot tasks for picking, moving, drawer operation, and apple placement, plus WidowX tasks for placing three objects and stacking cubes.Google Robot includes multiple can configurations and drawer subtasks; WidowX uses 24 configurations per task.Source: §G.1 · p. 20
  • Each WidowX configuration is tested 10 times for 240 rollouts per task; Google Robot trials total 300 for Pick Coke Can, 240 for Move Near, 216 for Open/Close Drawer, and 108 for Place Apple in Closed Drawer.Reported results are average success rates across trials.Source: §G.1 · p. 20
  • Table 8 presents policy-model ablation results as success rates across four Google Robot and four WidowX task columns.The caption identifies the task grouping but provides no cell values.Source: Table 8 · p. 21
  • Published performance metrics are used for RT-1-X, Octo-base, OpenVLA, RoboVLMs, MoTo, and LAPA, while GR00T is fine-tuned on RT-1/Bridge using the authors’ guidelines.The GR00T evaluation starts from the official pretrained checkpoint.Source: §G.1 · p. 21

H LIBERO Benchmark … I.2 XHand dexterous hand

On LIBERO, villa-X outperforms existing methods across all four task suites, and its latent-action version outperforms the version without latent actions. The remaining sections describe the Realman and XHand real-world evaluation setups.

H LIBERO Benchmark

  • LIBERO [39] evaluates transfer across four suites, each containing 10 tasks with 50 human demonstrations per task for fine-tuning.Spatial varies layouts, Goal varies tasks, Object varies object types, and Long varies objects, layouts, and backgrounds.Source: H LIBERO Benchmark · p. 21
  • The LIBERO comparison includes villa-X, villa-X without latent actions, and seven existing models, evaluated after two-stage pretraining and suite-specific fine-tuning.The baselines are Diffusion Policy [11], Octo, OpenVLA [29], π0 [4], π0 FAST [54], TraceVLA [71], and SpatialVLA [55]; evaluation uses 10 tasks and 20 trials per task in each suite.Source: H LIBERO Benchmark · p. 21
  • Villa-X outperforms existing methods on all four LIBERO task suites, and adding latent actions improves performance across every suite and on average.The supplied passages do not provide the table’s cell values.Source: H LIBERO Benchmark · p. 21, Table 9 · p. 21

I.1 Realman robot arm

  • The Realman setup mounts an Inspire Robot gripper on a Realman RM75 arm, uses primary and wrist cameras, and fine-tunes for 60k gradient steps.Fine-tuning reinitializes the linear state encoder, action encoder, and action decoder, and tunes all parameters except the vision encoder.Source: §I.1 · p. 22
  • Realman data comprise 375 trajectories across five tasks, recorded at 10Hz and post-processed to average 120 steps per trajectory.The tasks are putting a green block into a bowl, taking it out, pushing it to a marked position, stacking a wooden block, and unstacking it.Source: §I.1 · p. 22
  • Realman evaluation covers seven groups with 10 trials each, including five collected tasks and generalization to changed block colors and table covers.The color group repeats the five tasks with blue or red blocks; the table-cover group uses brown or blue covers.Source: §I.1 · p. 22

I.2 XHand dexterous hand

  • The XHand setup mounts a 12-DoF hand on a 7-DoF XArm and uses main and wrist cameras, with state and action encoder-decoder modules reinitialized for fine-tuning.The module changes accommodate the hand’s higher dimensionality.Source: §I.2 · p. 22
  • XHand fine-tuning uses roughly 4,000 trajectories across 13 categories and over 50 objects, with five tasks evaluated in seen and unseen conditions.The tasks are pick-and-place, cube stacking, upright cup placement, water pouring, and ball flicking. Unseen conditions include novel target objects or backgrounds and more than 20 novel objects; evaluation uses 50 runs for pick-and-place, 20 for cube stacking, and 10 for each other task.Source: §I.2 · p. 22
Loading 2507.23682v3…