Unified Video Action Model

• Video generation provides scene information for action prediction, while actions reveal interaction-driven visual changes for modeling dynamics.Source: §I · p. 1
• Action modeling requires high temporal speed, whereas video generation requires high spatial resolution, creating a processing-speed trade-off.Source: §I · p. 1
• UVA jointly learns visual and action data in a unified latent representation that captures shared dynamics with reduced computational overhead.Source: §I · pp. 1–2
• Two lightweight diffusion heads separately decode videos and actions, allowing inference to bypass video generation for fast policy prediction.Source: §I · p. 2
• 20% higher success rate on PushT Multitask and 5% higher performance on Libero10 than the best baselines demonstrate strong multi-task results.Source: §I · p. 2
• UVA matches state-of-the-art policy-learning and planning approaches, achieves similar speed to Diffusion Policy, and supports policy, dynamics, video, and combined planning functions.Source: §I · p. 2

• Prior robotics video-policy methods generate high-resolution videos before predicting actions, while action-only methods omit video supervision and its scene-dynamics benefits.Source: §III · p. 3
• UVA differs from prior masked-training work by applying one model directly to video generation, forward dynamics, inverse dynamics, and policy learning without downstream fine-tuning.Source: §II · p. 3
• Given h historical observations and action chunks, UVA predicts future actions and observations over future horizon h′, with experiments setting h = h′.Source: §III · p. 3
• Masked training is presented as the mechanism that lets one unified framework learn from varied video-action input combinations.Source: §III · p. 3
• The architecture encodes historical observations, actions, and masked future observations with a Transformer before decoding actions and videos.Source: §III · p. 3, Figure 2 · p. 3
• Each image becomes N visual tokens, while each corresponding action chunk contains L higher-frequency actions and is repeated to match the visual-token sequence.Source: A. Encode History · p. 3

Masked Autoencoder for Observation Prediction

• Future observation frames are encoded into N-token latent sequences, randomly masked during training, and reconstructed by the model.Source: B. Masked Autoencoder for Observation… · p. 4
• Language instructions are encoded as CLIP tokens and fused through Transformer cross-attention for tasks such as Libero10.Source: B. Masked Autoencoder for Observation… · p. 4
• At inference, complete videos are generated from empty sequences by autoregressively predicting tokens at shared spatial positions across frames.Source: B. Masked Autoencoder for Observation… · p. 4

Decoupled Video and Action Diffusions

• Joint training lets video generation enrich latent scene information for action prediction, while policy inference skips video generation and video generation can bypass action decoding.Source: C. Decoupled Video and Action Diffusions · p. 4
• Two lightweight diffusion decoders restrict denoising to the decoders rather than the entire model, reducing inference time.Source: C. Decoupled Video and Action Diffusions · p. 4
• The video decoder predicts patches from individual latent tokens, whereas the action decoder aggregates latent tokens into an action latent.Source: C. Decoupled Video and Action Diffusions · p. 4
• The total objective sums action and video diffusion losses across the time horizon, with decoders refining noise into actions or videos.Source: C. Decoupled Video and Action Diffusions · p. 4

D. Masked Training with Flexible Objectives

• UVA trains on five tasks by varying input-output combinations, replacing unused components with learned mask tokens and applying selective action or video losses.Source: D. Masked Training with Flexible… · pp. 4–5
• The masked strategy supports incomplete datasets, including video-only data without corresponding action labels.Source: D. Masked Training with Flexible… · p. 5
• UVA can function as a robot policy, video model, forward dynamics model, inverse dynamics model, or combined policy and planner.Source: D. Masked Training with Flexible… · p. 5, §IV · p. 5

Evaluation settings

• UVA is evaluated in single-task and multi-task settings across simulated and real-world environments.Simulation includes PushT and Toolhang; multi-task evaluation includes image-defined PushT-M and language-defined Libero10 goals.Source: §V · p. 5, A. Simulation Benchmarks · p. 5, Figure 3 · p. 5

Single-task evaluation

• Simulation evaluation reports success rates for PushT and Toolhang, with separate policies trained for different single tasks.The best-performing checkpoint is averaged across 50 rollouts for each task.Source: A. Simulation Benchmarks · p. 5

Multi-task evaluation

• Multi-task evaluation trains one policy across multiple goals and reports average performance over PushT-M and Libero10 environments.PushT-M varies target “T” positions, while Libero10 contains 10 tasks evaluated across 50 environments per task.Source: A. Simulation Benchmarks · p. 5, Figure 3 · p. 5

Real-world evaluation

• On real-world UMI tasks, UVA performs worse than DP-UMI in single-task evaluation but better in multi-task evaluation.Both methods use 16 denoising steps, and speed is measured for a 16-action trajectory.Source: Table II · p. 6

Baseline Methods

• The evaluation compares UVA with Diffusion Policy, OpenVLA, π0, π0-FAST, UniPi, and the UVA-action ablation using matched training or evaluation conditions.The comparison spans action accuracy, inference speed, visual-disturbance robustness, history-length robustness, and the effect of joint video-action modeling.Source: C. Baselines · p. 6

Action Prediction Accuracy

• 20% on PushT-M and 5% on Libero10: UVA surpasses the best baseline in multi-task action prediction.In simulation single-task settings, UVA matches DP-C and outperforms UniPi and OpenVLA.Source: D. Policy Learning Results · p. 7
• 15% higher success rate on Cup and 40% higher success rate on Mouse: UVA exceeds DP-UMI in real-world multi-task evaluation.For real-world single-task evaluation, UVA performs comparably with DP-UMI on the described training dataset.Source: D. Policy Learning Results · p. 7

Inference Speed

• UVA accelerates inference by applying diffusion iterations only to a lightweight action head and skipping video generation during policy inference.With 100 denoising steps, UVA is faster than DP-C and DP-T in simulation; in real-world tasks, it is slightly slower than DP-UMI because the Transformer attention module accounts for half of inference time.Source: D. Policy Learning Results · pp. 7–8

Visual Generalization

• 64% success rate with changed goal color: UVA exceeds UniPi at 40% and OpenVLA at 32% under this visual disturbance.The simulation evaluation also varies background color and adds object distractors outside the training distribution.Source: D. Policy Learning Results · p. 8

History-Length Robustness

• UVA maintains robust performance with longer history inputs, whereas DP-C often degrades as history length increases on PushT-M.The reported comparison attributes UVA’s robustness to jointly predicting video and action.Source: D. Policy Learning Results · p. 8, Figure 6 · p. 8

• UVA can generate videos by bypassing its action diffusion head during inference.Source: §VI · p. 8
• UVA outperforms UniPi on video generation in both simulated Libero10 and real-world Cup Arrangement environments.Source: Table IV · p. 8
• Masked autoencoder training enables autoregressive video generation, with visual tokens generated progressively across predefined steps.Source: §VI · p. 8
• UVA produces videos that closely match ground truth, while UniPi sometimes generates blurry, mismatched, or missing objects.Source: §VI · p. 8
• 8 generation steps improve video detail and performance compared with a single autoregressive step.Source: §VI · p. 8

• UVA performs forward dynamics prediction by modeling future observations as a function of current observations and actions.The model is used to guide a pretrained policy in block-pushing tasks.Source: §VII · p. 8
• The block-pushing evaluation uses four settings with varying initial object and robot positions, and each setting is tested 10 times.Source: §VII · p. 9
• UVA predicts future observations for sampled trajectories and selects actions whose predicted frames place blocks nearer their targets.Source: §VII · p. 9
• 60% success rate with UVA-guided DP-C exceeds the 38% achieved by DP-C alone.Source: Table V · p. 10, §VII · p. 10
• The ground-truth simulator reaches 75% success, exceeding the UVA-guided policy while remaining an upper-bound comparison.Source: §VII · p. 10, Table V · p. 10

• UVA infers actions from consecutive observations in an inverse dynamics setting using UMI data and unseen test data.Source: §VIII · p. 10
• UVA predicts 16 actions simultaneously, producing more temporally coherent predictions than UniPi’s consecutive-image approach.Source: §VIII · p. 10
• The evaluation compares UVA and UniPi inverse dynamics against a motion-capture ground truth, with SLAM included as a stronger accuracy reference.Source: Table VI · p. 10, §VIII · p. 10
• SLAM achieves the best inverse-dynamics accuracy but is more complex to implement than UVA.Source: Table VI · p. 10
• UVA achieves position errors under 1 cm and rotation errors around 1 degree, while SLAM records 0.41 cm and 0.30 degrees.Source: §VIII · p. 10

• The supplementary materials cover autoregressive video generation, simulation and real-world benchmarks, and inference-speed details.Source: §X · p. 14
• UVA extends masked discrete-token image-generation methods to video and predicts continuous latent representations for diffusion-based video and action decoding.Source: A. Autogregressive Video Generation · p. 14
• A one-step autoregressive setting generates the entire video in a single pass, while larger predefined step counts generate it sequentially.Source: A. Autogregressive Video Generation · p. 14, Figure 9 · p. 14

Single-Task Evaluation

• Single-task benchmarks include PushT and Toolhang, with average success rates reported over 50 rollouts or environments.Source: B. Simulation Benchmarks · p. 14

Multi-Task Evaluation

• Multi-task evaluation trains one policy across multiple goals defined by images or text, including varying-target PushT-M and long-horizon Libero10.Source: B. Simulation Benchmarks · p. 14
• Libero10 contains 10 language-directed tasks evaluated over 500 total test episodes, with average reward reported across tasks.Source: B. Simulation Benchmarks · p. 14

Training Data

• The real-world evaluation uses two public UMI datasets and a single ARX X5 robotic arm across cup arrangement, towel folding, and mouse arrangement tasks.The datasets contain demonstrations for three manipulation tasks evaluated on the same robotic setup.Source: C. Real-world Benchmarks · pp. 14–15

Single-Task Evaluation

• Single-task evaluation trains on the Cup dataset and compares UVA directly with DP-UMI using the same data across 20 test rollouts.The rollouts vary object positions and orientations, with cup rotation required in 85% of cases.Source: C. Real-world Benchmarks · p. 15

Multi-Task Evaluation

• Multi-task evaluation combines 1,500 episodes sampled from three datasets and tests one model independently on all three tasks across 20 scenarios.The original public datasets contain 6,764 episodes; training uses 500 episodes from each dataset.Source: C. Real-world Benchmarks · p. 15

Inference Speed Measurement and Decomposition

• UVA predicts a 16-action trajectory in 95ms with 16 diffusion steps, or 173ms with 100 steps, while flash attention reduces either total by 10ms.The reported real-world measurements use an NVIDIA RTX 3080 GPU; 100 steps produce smoother actions and higher success at lower speed.Source: D. Inference Speed Measurement and… · p. 15
• Video generation takes 100ms with 16 diffusion steps and 625ms with 100 steps, but policy inference skips it to significantly accelerate action prediction.The action head takes 15ms with 16 diffusion steps and 93ms with 100 steps.Source: D. Inference Speed Measurement and… · p. 15

Action-Free Video Pretraining

• UVA uses 3,175 action-free Human Video videos for video-generation pretraining before masked co-training on Human Video and LIBERO-10 data.This procedure is designed to leverage video-only data alongside video-action data.Source: E. Utilize Action-Free Video · p. 15

Additional Human Data

• Table VIII reports LIBERO-10 results with additional human data across 10 tasks using either 30 or 500 total tests per task.Each task is evaluated three times with different seeds in the 30-test setting and 50 times in the 500-test setting.Source: Table VIII · p. 15

Masking Strategies

• Application-dependent masking follows each task’s input-output structure, whereas application-independent masking randomly masks inputs regardless of task semantics.The two strategies are evaluated across multiple mask ratios.Source: F. Impact of masking strategy · p. 15, Table IX · p. 16

Evaluation

• The evaluation measures policy learning by success rate, video generation and forward dynamics by FVD, and inverse dynamics by L2 error.These metrics cover the distinct prediction tasks supported by masked training.Source: F. Impact of masking strategy · p. 15

Masking Results

• Under application-independent masking, 25% masking works best for video generation and forward dynamics, while 50% is optimal for policy and inverse dynamics.In the application-dependent setting, higher mask ratios degrade overall performance.Source: F. Impact of masking strategy · pp. 15–16