PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Video Action Model

Shuang Li, Yihuai Gao, Dorsa Sadigh, Shuran Song

arXiv:2503.00200v3cs.ROcs.CV

TL;DR

Unified video-action modeling is promising for robotics but remains difficult because action prediction requires speed while video generation requires high spatial resolution. UVA addresses this mismatch with a unified latent representation and decoupled decoding, and matches or surpasses tailored approaches across policy-learning settings while supporting multiple robotics functions.

  • Problem

    Combining video generation and action prediction remains challenging because action modeling requires high temporal speed, whereas video generation requires high spatial resolution and slower processing.

  • Method

    UVA jointly models videos and actions through a unified latent representation, then separately decodes them so action inference can skip video generation.

  • Results

    UVA matches state-of-the-art tailored approaches, surpasses the best baseline by 20% on PushT-M and 5% on Libero10, and outperforms larger π0 and π0-FAST models despite using fewer resources.

  • Takeaways & Limitations

    UVA serves as a versatile framework for policy learning, video modeling, forward and inverse dynamics, and combined policy-planning functions.

  • Takeaways & Limitations

    UVA does not currently use large amounts of actionless video data and consequently sometimes achieves only comparable real-world performance to DP-UMI.

Abstract

from arXiv · show

A unified video and action model holds significant promise for robotics, where videos provide rich scene information for action prediction, and actions provide dynamics information for video prediction. However, effectively combining video generation and action prediction remains challenging, and current video generation-based methods struggle to match the performance of direct policy learning in action accuracy and inference speed. To bridge this gap, we introduce the Unified Video Action model (UVA), which jointly optimizes video and action predictions to achieve both high accuracy and efficient action inference. The key lies in learning a joint video-action latent representation and decoupling video-action decoding. The joint latent representation bridges the visual and action domains, effectively modeling the relationship between video and action sequences. Meanwhile, the decoupled decoding, powered by two lightweight diffusion heads, enables high-speed action inference by bypassing video generation during inference. Such a unified framework further enables versatile functionality through masked input training. By selectively masking actions or videos, a single model can tackle diverse tasks beyond policy learning, such as forward and inverse dynamics modeling and video generation. Via an extensive set of experiments, we demonstrate that UVA can serve as a general-purpose solution for a wide range of robotics tasks, such as policy learning, forward/inverse dynamics and video observation prediction, without compromising performance compared to methods tailored for specific applications. Results are best viewed on https://unified-video-action-model.github.io/.

I. INTRODUCTION

UVA addresses the conflict between video generation’s spatial demands and action prediction’s speed requirements by jointly modeling videos and actions while decoupling their decoding. Its unified latent representation and masked training support accurate, efficient, and versatile robotics functions.

  • Video generation provides scene information for action prediction, while actions reveal interaction-driven visual changes for modeling dynamics.
  • Action modeling requires high temporal speed, whereas video generation requires high spatial resolution, creating a processing-speed trade-off.
  • UVA jointly learns visual and action data in a unified latent representation that captures shared dynamics with reduced computational overhead.
  • Two lightweight diffusion heads separately decode videos and actions, allowing inference to bypass video generation for fast policy prediction.
  • 20% higher success rate on PushT Multitask and 5% higher performance on Libero10 than the best baselines demonstrate strong multi-task results.
  • UVA matches state-of-the-art policy-learning and planning approaches, achieves similar speed to Diffusion Policy, and supports policy, dynamics, video, and combined planning functions.

II. RELATED WORK

UVA builds on video generation, masked training, and multimodal sequence modeling to jointly predict future observations and actions. The formulation uses historical observations and higher-frequency action chunks as inputs for unified prediction.

  • Prior robotics video-policy methods generate high-resolution videos before predicting actions, while action-only methods omit video supervision and its scene-dynamics benefits.
  • UVA differs from prior masked-training work by applying one model directly to video generation, forward dynamics, inverse dynamics, and policy learning without downstream fine-tuning.
  • Given h historical observations and action chunks, UVA predicts future actions and observations over future horizon h′, with experiments setting h = h′.
  • Masked training is presented as the mechanism that lets one unified framework learn from varied video-action input combinations.
  • The architecture encodes historical observations, actions, and masked future observations with a Transformer before decoding actions and videos.
  • Each image becomes N visual tokens, while each corresponding action chunk contains L higher-frequency actions and is repeated to match the visual-token sequence.

B. Masked Autoencoder for Observation Prediction

UVA predicts observations through masked visual-token reconstruction and uses decoupled diffusion heads to generate videos or actions from a shared latent representation. Joint training enriches the latent with scene information while allowing modality-specific inference.

  • Masked Autoencoder for Observation Prediction: Future observation frames are encoded into N-token latent sequences, randomly masked during training, and reconstructed by the model.
  • Masked Autoencoder for Observation Prediction: Language instructions are encoded as CLIP tokens and fused through Transformer cross-attention for tasks such as Libero10.
  • Masked Autoencoder for Observation Prediction: At inference, complete videos are generated from empty sequences by autoregressively predicting tokens at shared spatial positions across frames.
  • Decoupled Video and Action Diffusions: Joint training lets video generation enrich latent scene information for action prediction, while policy inference skips video generation and video generation can bypass action decoding.
  • Decoupled Video and Action Diffusions: Two lightweight diffusion decoders restrict denoising to the decoders rather than the entire model, reducing inference time.
  • Decoupled Video and Action Diffusions: The video decoder predicts patches from individual latent tokens, whereas the action decoder aggregates latent tokens into an action latent.
  • Decoupled Video and Action Diffusions: The total objective sums action and video diffusion losses across the time horizon, with decoders refining noise into actions or videos.

D. Masked Training with Flexible Objectives

Masked training varies which video and action components are available and supervised, allowing UVA to learn from complete or incomplete data. The same framework therefore supports multiple robotics functions without separate task-specific models.

  • D. Masked Training with Flexible Objectives: UVA trains on five tasks by varying input-output combinations, replacing unused components with learned mask tokens and applying selective action or video losses.
  • D. Masked Training with Flexible Objectives: The masked strategy supports incomplete datasets, including video-only data without corresponding action labels.
  • D. Masked Training with Flexible Objectives: UVA can function as a robot policy, video model, forward dynamics model, inverse dynamics model, or combined policy and planner.

V. UVA AS POLICY

UVA is evaluated as a policy across single-task and multi-task simulation and real-world settings, including image- and language-defined goals. It generally performs strongly, especially in multi-task scenarios, while real-world UMI results show a single-task versus multi-task trade-off.

  • Evaluation settings: UVA is evaluated in single-task and multi-task settings across simulated and real-world environments.Simulation includes PushT and Toolhang; multi-task evaluation includes image-defined PushT-M and language-defined Libero10 goals.
  • Single-task evaluation: Simulation evaluation reports success rates for PushT and Toolhang, with separate policies trained for different single tasks.The best-performing checkpoint is averaged across 50 rollouts for each task.
  • Multi-task evaluation: Multi-task evaluation trains one policy across multiple goals and reports average performance over PushT-M and Libero10 environments.PushT-M varies target “T” positions, while Libero10 contains 10 tasks evaluated across 50 environments per task.
  • Real-world evaluation: On real-world UMI tasks, UVA performs worse than DP-UMI in single-task evaluation but better in multi-task evaluation.Both methods use 16 denoising steps, and speed is measured for a 16-action trajectory.

C. Baselines

UVA is evaluated against policy, video-based, and vision-language-action baselines across accuracy, speed, robustness, and joint video-action modeling. Across these comparisons, it matches or exceeds tailored approaches, with especially strong multi-task and visual-generalization results.

  • Baseline Methods: The evaluation compares UVA with Diffusion Policy, OpenVLA, π0, π0-FAST, UniPi, and the UVA-action ablation using matched training or evaluation conditions.The comparison spans action accuracy, inference speed, visual-disturbance robustness, history-length robustness, and the effect of joint video-action modeling.
  • Action Prediction Accuracy: 20% on PushT-M and 5% on Libero10: UVA surpasses the best baseline in multi-task action prediction.In simulation single-task settings, UVA matches DP-C and outperforms UniPi and OpenVLA.
  • Action Prediction Accuracy: 15% higher success rate on Cup and 40% higher success rate on Mouse: UVA exceeds DP-UMI in real-world multi-task evaluation.For real-world single-task evaluation, UVA performs comparably with DP-UMI on the described training dataset.
  • Inference Speed: UVA accelerates inference by applying diffusion iterations only to a lightweight action head and skipping video generation during policy inference.With 100 denoising steps, UVA is faster than DP-C and DP-T in simulation; in real-world tasks, it is slightly slower than DP-UMI because the Transformer attention module accounts for half of inference time.
  • Visual Generalization: 64% success rate with changed goal color: UVA exceeds UniPi at 40% and OpenVLA at 32% under this visual disturbance.The simulation evaluation also varies background color and adds object distractors outside the training distribution.
  • History-Length Robustness: UVA maintains robust performance with longer history inputs, whereas DP-C often degrades as history length increases on PushT-M.The reported comparison attributes UVA’s robustness to jointly predicting video and action.

VI. UVA AS A VIDEO GENERATOR

UVA functions as a video generator by decoding video from its unified latent representation, with masked autoregressive training supporting flexible generation. It outperforms UniPi across simulated and real-world video-generation settings, and additional generation steps improve detail and performance.

  • UVA can generate videos by bypassing its action diffusion head during inference.
  • UVA outperforms UniPi on video generation in both simulated Libero10 and real-world Cup Arrangement environments.
  • Masked autoencoder training enables autoregressive video generation, with visual tokens generated progressively across predefined steps.
  • UVA produces videos that closely match ground truth, while UniPi sometimes generates blurry, mismatched, or missing objects.
  • 8 generation steps improve video detail and performance compared with a single autoregressive step.

VII. UVA AS A FORWARD DYNAMICS MODEL

UVA predicts future observations from current observations and actions, then uses those predictions to guide trajectory selection for pretrained policies. On block pushing, this improves success over DP-C alone, though a ground-truth simulator remains stronger.

  • UVA performs forward dynamics prediction by modeling future observations as a function of current observations and actions.The model is used to guide a pretrained policy in block-pushing tasks.
  • The block-pushing evaluation uses four settings with varying initial object and robot positions, and each setting is tested 10 times.
  • UVA predicts future observations for sampled trajectories and selects actions whose predicted frames place blocks nearer their targets.
  • 60% success rate with UVA-guided DP-C exceeds the 38% achieved by DP-C alone.
  • The ground-truth simulator reaches 75% success, exceeding the UVA-guided policy while remaining an upper-bound comparison.

VIII. UVA AS A INVERSE DYNAMIC MODEL

UVA also operates as an inverse dynamics model, inferring actions from consecutive observations and evaluating them against unseen motion-capture actions. It produces temporally coherent predictions with errors near, though slightly above, those of SLAM.

  • UVA infers actions from consecutive observations in an inverse dynamics setting using UMI data and unseen test data.
  • UVA predicts 16 actions simultaneously, producing more temporally coherent predictions than UniPi’s consecutive-image approach.
  • The evaluation compares UVA and UniPi inverse dynamics against a motion-capture ground truth, with SLAM included as a stronger accuracy reference.
  • SLAM achieves the best inverse-dynamics accuracy but is more complex to implement than UVA.
  • UVA achieves position errors under 1 cm and rotation errors around 1 degree, while SLAM records 0.41 cm and 0.30 degrees.

X. SUPPLEMENTARY MATERIALS

The supplementary materials describe UVA’s autoregressive video-generation process, implementation basis, benchmark settings, and evaluation tasks. These include single-task and multi-task simulation benchmarks with defined environments, goals, and reporting protocols.

  • The supplementary materials cover autoregressive video generation, simulation and real-world benchmarks, and inference-speed details.
  • UVA extends masked discrete-token image-generation methods to video and predicts continuous latent representations for diffusion-based video and action decoding.
  • A one-step autoregressive setting generates the entire video in a single pass, while larger predefined step counts generate it sequentially.
  • Single-Task Evaluation: Single-task benchmarks include PushT and Toolhang, with average success rates reported over 50 rollouts or environments.
  • Multi-Task Evaluation: Multi-task evaluation trains one policy across multiple goals defined by images or text, including varying-target PushT-M and long-horizon Libero10.
  • Multi-Task Evaluation: Libero10 contains 10 language-directed tasks evaluated over 500 total test episodes, with average reward reported across tasks.

C. Real-world Benchmarks

UVA is evaluated on real-world robotic benchmarks spanning single-task and multi-task policy learning. Its decoupled inference omits video generation, enabling fast action prediction with configurable diffusion steps.

  • Training Data: The real-world evaluation uses two public UMI datasets and a single ARX X5 robotic arm across cup arrangement, towel folding, and mouse arrangement tasks.The datasets contain demonstrations for three manipulation tasks evaluated on the same robotic setup.
  • Single-Task Evaluation: Single-task evaluation trains on the Cup dataset and compares UVA directly with DP-UMI using the same data across 20 test rollouts.The rollouts vary object positions and orientations, with cup rotation required in 85% of cases.
  • Multi-Task Evaluation: Multi-task evaluation combines 1,500 episodes sampled from three datasets and tests one model independently on all three tasks across 20 scenarios.The original public datasets contain 6,764 episodes; training uses 500 episodes from each dataset.
  • Inference Speed Measurement and Decomposition: UVA predicts a 16-action trajectory in 95ms with 16 diffusion steps, or 173ms with 100 steps, while flash attention reduces either total by 10ms.The reported real-world measurements use an NVIDIA RTX 3080 GPU; 100 steps produce smoother actions and higher success at lower speed.
  • Inference Speed Measurement and Decomposition: Video generation takes 100ms with 16 diffusion steps and 625ms with 100 steps, but policy inference skips it to significantly accelerate action prediction.The action head takes 15ms with 16 diffusion steps and 93ms with 100 steps.

E. Utilize Action-Free Video

UVA can incorporate action-free human videos through video pretraining and masked co-training. Masking performance depends on the strategy and ratio, with different optimal ratios across tasks.

  • Action-Free Video Pretraining: UVA uses 3,175 action-free Human Video videos for video-generation pretraining before masked co-training on Human Video and LIBERO-10 data.This procedure is designed to leverage video-only data alongside video-action data.
  • Additional Human Data: Table VIII reports LIBERO-10 results with additional human data across 10 tasks using either 30 or 500 total tests per task.Each task is evaluated three times with different seeds in the 30-test setting and 50 times in the 500-test setting.
  • Masking Strategies: Application-dependent masking follows each task’s input-output structure, whereas application-independent masking randomly masks inputs regardless of task semantics.The two strategies are evaluated across multiple mask ratios.
  • Evaluation: The evaluation measures policy learning by success rate, video generation and forward dynamics by FVD, and inverse dynamics by L2 error.These metrics cover the distinct prediction tasks supported by masked training.
  • Masking Results: Under application-independent masking, 25% masking works best for video generation and forward dynamics, while 50% is optimal for policy and inverse dynamics.In the application-dependent setting, higher mask ratios degrade overall performance.
Loading 2503.00200v3…