ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models

Motivation

• Existing action representations primarily model trajectories, while effective manipulation representations should also account for the visual dynamics actions induce.The gap is how visual dynamics should shape an executable action latent space.Source: §1 · pp. 1–2

Approach

• ViDAL trains an Action VAE to preserve executable actions while predicting future visual latents from the current visual latent and action latent.The frozen Action VAE supplies action-latent targets for downstream policies and decodes predicted latents into actions.Source: §1 · p. 2

Contributions

• The plug-in Action VAE supports regression and flow-matching policies without modifying the VLM backbone.The paper evaluates those policy types on LIBERO, RoboTwin 2.0, Franka, and ARX.Source: §1 · p. 2

Results

• 11.2% Clean and 9.9% Random gains for π0.5 on RoboTwin 2.0 accompany 20.0% and 23.4% absolute gains on Franka and ARX, respectively.The reported evaluation also finds consistent improvements across two simulation benchmarks and two real-world platforms.Source: §1 · p. 2

Vision-Language-Action Models

• VLA action experts include flow-matching policies for continuous control and one-step regression policies for faster inference; ViDAL changes the action target without modifying these architectures.The predicted dynamics-grounded latent is decoded back into executable actions.Source: §2 · pp. 2–3

Action Representations for VLA Models

• Raw action chunks, discrete tokens, and continuous latents are generally shaped by behavior cloning, quantization, or reconstruction rather than scene evolution.ViDAL directly supervises action latents with visual dynamics.Source: §2 · p. 3

Visual Dynamics for VLA Policies

• Other visual-dynamics approaches condition or supervise VLA policies, entangling dynamics learning with policy training and tying it to an architecture.ViDAL confines dynamics learning to the action latent and freezes the resulting Action VAE before policy training.Source: §2 · p. 3

3.1 Overview

• The Action VAE encodes each action chunk into a continuous latent, and a frozen video model supplies visual latents for joint action-reconstruction and dynamics supervision.After representation learning, the video model is no longer required by the downstream policy.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Learning the Action Latent Space

• Action reconstruction alone may omit scene-relevant information, so ViDAL adds future-visual-latent prediction conditioned on the initial visual latent.The objective combines time-domain reconstruction, DCT-domain reconstruction, KL regularization, and visual-dynamics alignment.Source: §3.2 · pp. 3–4
• The Action VAE maps an action chunk through a Gaussian posterior to a continuous latent and decodes that latent back into the action chunk.Its encoder applies 4× temporal downsampling, with latent length T=H/4.Source: §3.2 · p. 3
• A frozen video encoder extracts initial and future visual latents, while a lightweight predictor estimates the future latent from the initial visual latent and action latent.Conditioning on the initial visual latent accounts for the fact that the same action chunk can correspond to different scene transitions in different contexts.Source: §3.2 · p. 3

3.3 Integrating ViDAL into VLA Policies

• For downstream VLA training, the frozen Action VAE encoder supplies a deterministic, normalized latent target, leaving the VLM backbone unchanged.The target differs from the sampled latent used during Action VAE training.Source: §3.3 · p. 4
• Regression experts predict the latent target, while flow-matching experts shift their modeled distribution from raw actions to dynamics-grounded latents; the decoder returns predicted latents to executable actions.The Action VAE acts as a drop-in interface between an existing VLA action expert and the robot, without a video model in the loop.Source: §3.3 · pp. 4–5

4.2 Main Results

• RoboTwin 2.0 gains were +11.2%/+9.9% on π0.5 and +4.2%/+1.8% on StarVLA-OFT (Clean/Random); LIBERO gains were +0.3% on OpenVLA-OFT and +1.2% on π0.5.The RoboTwin 2.0 results average over 50 tasks; on LIBERO, π0.5+ViDAL attained the highest average in the comparison.Source: §4.2 · p. 5, Table 2 · p. 6, Table 1 · p. 6

4.3 Ablation Studies

• Removing either the visual-dynamics loss or DCT reconstruction degrades both RoboTwin 2.0 splits, with the steepest drop when visual-dynamics supervision is disabled.The visual-dynamics term is identified as the structural source of the gain, while frequency-domain reconstruction adds a smaller refinement.Source: Table 3 · p. 6, §4.3 · p. 6
• Latent dimension, chunk length, dynamics weight, and KL weight all trade off around the default rather than improving monotonically.The default balances action reconstruction, dynamics anchoring, and prior regularization; overly strong KL regularization collapses the latent, while weak regularization leaves it under-regularized for the flow expert.Source: §4.3 · pp. 6–7, Figure 3 · p. 7

4.4 Further Analysis

• A frozen Wan VAE target with dense L1 dynamics supervision performs best among the tested choices, and the frozen ViDAL latent alone is the most effective policy target.Replacing L1 with InfoNCE or combining them degrades performance; raw-only prediction reproduces the π0.5 baseline, while joint prediction sits between it and latent-only prediction.Source: Figure 4 · p. 7, §4.4 · p. 7
• ViDAL improves π0.5 at every tested task and demonstration scale, with the gain widening as either data axis increases.The evaluation subset varies with task count, so the task-scale trend is read from the comparable improvement values; demonstration scaling uses a fixed 50-task evaluation.Source: §4.4 · p. 7, Figure 5 · p. 8

4.5 Real-Robot Experiments

• On real robots, ViDAL raises average success by +20.0% on Franka and +23.4% on ARX, improving every task across both platforms.Each platform's Action VAE is trained on its own demonstrations and reused as a frozen plug-in representation; each task is evaluated over 10 rollouts.Source: Figure 6 · p. 8, §4.5 · p. 8

4.6 Qualitative Visualization and Analysis

• ViDAL latents are less geometrically collapsed than raw actions and also support recognizable future-video rollouts.Raw actions need 14 principal components to reach 95% cumulative variance versus 40 for ViDAL; decoded predictions track object location, gripper pose, and contact events, while the predictor and decoder are analysis-only.Source: Figure 7 · p. 9, §4.6 · p. 9, Figure 8 · p. 9

• ViDAL jointly trains an Action VAE on action reconstruction and video-latent prediction to ground action latents in future visual dynamics.Source: §5 · p. 9
• The frozen Action VAE serves as a plug-in action representation for regression and flow-matching policies without modifying the VLM backbone.Source: §5 · p. 9
• ViDAL improves performance across LIBERO, RoboTwin 2.0, single-arm Franka, and dual-arm ARX, with its advantage widening as tasks and demonstrations scale up.Source: §5 · p. 9
• Extending ViDAL to diverse action spaces and embodiments requires retraining only the lightweight Action VAE; large-scale egocentric human videos are proposed as a next supervision source.Source: §5 · p. 9

Action VAE architecture

• The Action VAE encodes normalized action chunks into temporally ordered continuous latent tokens and decodes them back into executable action chunks.Its encoder uses temporal convolutions with residual blocks and strided downsampling; downstream supervision uses the posterior mean as a deterministic target.Source: A Method details · p. 13

Frozen video-latent target

• A frozen video VAE encodes the aligned observation clip, whose future visual tokens serve as the dynamics target for the action chunk.The clip contains one more frame than the action horizon.Source: A Method details · p. 13

Future-latent predictor

• The predictor infers future visual latents from current visual context and action latents, encouraging scene-conditioned action effects to be encoded in the latent.It is used only during representation learning, not in the deployed VLA policy.Source: A Method details · p. 13

Training objective

• The four-term objective combines action reconstruction, frequency-domain reconstruction, KL regularization, and visual-dynamics prediction.The reconstruction and frequency terms use L1 losses; the dynamics loss compares predicted tokens with fixed video-latent targets.Source: A Method details · p. 13

Downstream VLA interface

• After representation learning, the frozen Action VAE encodes action chunks into deterministic latent targets and decodes policy predictions into executable chunks.The video VAE and future-latent predictor are discarded, so downstream training changes the action target space while leaving the VLM backbone and action expert otherwise unchanged.Source: A Method details · p. 13

Action VAE training

• A separate Action VAE is trained for each benchmark or robot platform, with LIBERO configurations used for single-arm settings and RoboTwin 2.0 configurations for dual-arm settings.The ARX setting follows the RoboTwin 2.0 configuration; all settings use a frozen Wan video VAE and λdyna=1, λfreq=1, and λKL=10^-4.Source: B Implementation details · pp. 13–14, Table 4 · p. 14

Downstream VLA integration

• ViDAL replaces raw action targets with normalized Action VAE latents; for π0.5, the action-expert learning rate is set to 10× the VLM learning rate.At inference, only the Action VAE decoder is additionally deployed; the video VAE and future-latent predictor are not used, and decoder latency is negligible relative to the VLA forward pass.Source: B Implementation details · p. 14

Training data and evaluation

• RoboTwin 2.0 training uses 2,500 Clean episodes across 50 tasks, with 100 evaluation rollouts per task in both Clean and Random settings.Franka uses four tasks with 50 demonstrations each, ARX uses three tasks with 100 demonstrations each, and each real-robot task is evaluated with 10 rollouts.Source: B Implementation details · p. 14

C Full RoboTwin 2.0 results

• The full per-task RoboTwin 2.0 results are reported in Table 5, corresponding to Table 2.Source: C Full RoboTwin 2.0 results · p. 14
• All methods are evaluated under both Clean and Random settings, with 100 rollouts per task in each setting.Source: C Full RoboTwin 2.0 results · p. 14
• The representative rows in the main paper are a subset of the full table, and the averages match Table 2 up to rounding.Source: C Full RoboTwin 2.0 results · p. 14

D Future prediction examples

• Figures 9 and 10 pair ground-truth future frames with predictions decoded from future visual latents conditioned on ViDAL action latents across simulation and real-robot examples.The simulation examples use LIBERO and RoboTwin 2.0; the real-robot examples use Franka and ARX.Source: D Future prediction examples · p. 14, Figure 9 · p. 16, Figure 10 · p. 17
• The future-video visualizations are qualitative analysis only; the video VAE, future-latent predictor, and video decoder are not part of the deployed policy.Source: D Future prediction examples · p. 14
• Table 5 reports success rates across all 50 RoboTwin 2.0 tasks, with gray columns marking the corresponding +ViDAL variants.Source: Table 5 · p. 15