PaperlayerView exact revision on arXiv

Source-linked AI summary

CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators

Kechen Liu, Ola Shorinwa

arXiv:2608.27406v1cs.ROcs.AIcs.CV

TL;DR

Action-conditioned video models are typically limited to one robot embodiment, restricting use of heterogeneous videos for learning generalizable physics. CLAP reconciles cross-embodiment action spaces and uses curriculum learning to scale across unlabeled human and robot videos, approaching or surpassing single-embodiment baselines while enabling few-shot adaptation.

  • Problem

    Single-embodiment action-conditioned video models cannot readily leverage heterogeneous human and robot videos, whose morphologies and action spaces differ and whose human videos often lack action labels.

  • Method

    CLAP harmonizes action spaces with end-effector, language, and latent actions, then uses curriculum learning to pretrain on unlabeled videos before grounding physical priors in end-effector actions.

  • Results

    CLAP approaches or surpasses state-of-the-art single-embodiment baselines and supports sample-efficient adaptation to target embodiments for zero-shot real-world tasks.

  • Takeaways & Limitations

    CLAP provides a cross-embodiment route to training action-conditioned video world models and adapting them into single-embodiment models.

  • Takeaways & Limitations

    CLAP remains prone to hallucinations, while broader bimanual and humanoid data and scaling improvements incur significant training and inference costs.

Abstract

from arXiv · show

State-of-the-art action-conditioned video models are typically restricted to a single robot embodiment, preventing them from leveraging the vast corpus of heterogeneous video data that contains rich signals for learning generalizable physics. To bridge this gap, we introduce CLAP, a framework for cross-embodiment action-conditioned video generation capable of being trained on diverse, internet-scale videos across human and robotic agents. CLAP is grounded in the insight that universal physical laws govern spatiotemporal dynamics regardless of the actor. However, cross-embodiment learning is non-trivial because action representations vary sharply across robot platforms and are typically absent in human videos. CLAP addresses this fundamental challenge through the following core contributions. First, CLAP reconciles disparate action spaces using end-effector poses, language instructions, and latent actions. Second, to resolve their individual limitations, CLAP introduces a curriculum-based cross-embodiment learning recipe that first learns foundational physical priors across unlabeled video data using latent actions and subsequently grounds them in end-effector action spaces for zero-shot deployment to real-world tasks. Crucially, CLAP approaches or surpasses state-of-the-art single-embodiment video models in challenging environments like DROID. These performance advantages compound via few-shot adaptation to establish a novel paradigm for training single-embodiment video world models. Ultimately, CLAP delivers the most comprehensive suite of action-conditioned video world models to date - spanning diverse action-conditioning spaces (end-effector, language, and latent) and robot morphologies (including cross-embodiment, DROID, Bridge, bimanual YAM robots, and G1 humanoids). We open-source all code and models. Project Website at https://omni-clap.github.io .

1 Introduction

CLAP addresses the limitations of single-embodiment action-conditioned video models by learning physical priors across heterogeneous human and robot videos. It harmonizes disparate action spaces and uses curriculum learning to support deployment and adaptation across robot embodiments.

  • Single-embodiment video models restrict access to heterogeneous internet-scale videos that could provide signals for learning generalizable physical priors.
  • CLAP exploits shared physical laws by training cross-embodiment action-conditioned video models on human and robot data.
  • CLAP harmonizes action spaces with end-effector poses, language instructions, and learned latent actions.
  • Its curriculum first learns physical priors from unlabeled videos with latent actions, then refines them using end-effector actions for zero-shot real-world deployment.
  • CLAP approaches or surpasses state-of-the-art single-embodiment baselines in challenging environments such as DROID and compounds this advantage through sample-efficient adaptation.
  • The resulting suite spans end-effector, language, and latent conditioning across cross-embodiment, DROID, Bridge, bimanual YAM, and G1 humanoid settings.

2 Related Work

Prior work uses video generation for trajectory synthesis and policy learning, while cross-embodiment methods unify heterogeneous robot spaces through shared representations or latent actions. CLAP builds on these directions to address action and observation heterogeneity in unlabeled video data.

  • Video generation models support trajectory synthesis with pseudo-actions and serve as policy backbones through tracking, inverse dynamics, or unified video-action prediction.
  • Cross-embodiment robot learning addresses heterogeneous degrees of freedom, kinematic structures, and observations using padded spaces, shared end-effector or language representations, action heads, projectors, and latent actions.

3 CLAP: Training Cross-Embodiment Video World Models

CLAP trains cross-embodiment video world models by reconciling heterogeneous action spaces and scaling latent-action learning to unlabeled videos. A curriculum then grounds learned physical priors in end-effector actions and supports few-shot adaptation to target embodiments.

  • Harmonizing Heterogeneous Action Spaces: CLAP reconciles heterogeneous robot and human action spaces with end-effector poses, language, and latent action representations.
  • Harmonizing Heterogeneous Action Spaces: Unified action spaces are needed because robots such as WidowX and Franka have different degrees of freedom while video models require fixed-dimensional conditioning inputs.
  • Harmonizing Heterogeneous Action Spaces: CLAP-EE uses absolute end-effector actions normalized into finite bounds [−1, 1], with robot-specific factors accounting for morphological differences.
  • Harmonizing Heterogeneous Action Spaces: Language actions represent control primitives as natural text and provide coarse-to-fine controllability with minimal architectural adaptation.
  • Harmonizing Heterogeneous Action Spaces: Latent actions enable training from unlabeled videos by describing transformations between video frames, but require downstream alignment to a target embodiment.
  • Curriculum-based Latent-to-EE Actions: CLAP-CURR first learns physical priors from unlabeled videos using latent actions and subsequently refines them with end-effector actions for zero-shot deployment.
  • Model Architecture: The video model uses video diffusion with continuous-VAE tokenization and a U-Net for salient spatiotemporal modeling.
  • History Conditioning and Adaptation: History conditioning supplies temporal velocity information by using past observations and robot poses, while broad training learns priors that can be refined through target-embodiment finetuning.

4 Experiments

CLAP evaluates whether cross-embodiment video models can match single-embodiment systems, support different action spaces, generalize to real-world manipulation, and adapt efficiently to new robots. Across these experiments, CLAP matches or exceeds strong baselines, while action-space choice and adaptation determine fidelity and controllability.

  • Action-space harmonization: Latent actions can match or surpass end-effector actions on OXE-Mix, while enabling training on unlabeled human videos; CLAP-CURR combines latent-action learning with end-effector grounding.This combination addresses the mismatch between action-free training data and geometric robot actions at deployment.
  • Cross-embodiment performance: CLAP matches single-embodiment baseline performance on challenging DROID despite using the same model size and fewer domain-specific samples.In Bridge, CLAP achieves high-accuracy predictions but does not precisely match the Bridge-Base baseline.
  • Action-space harmonization: Absolute actions outperform relative actions for end-effector-conditioned prediction, including an approximately 14.6% LPIPS gap on DROID.Relative actions instead achieve higher fidelity for language-conditioned video models, partly because compact bounds use limited tokenization resolution more effectively.
  • Zero-shot manipulation: CLAP improves real-world manipulation by selecting among action proposals from π0.5 and MolmoAct-2, including sequential receding-horizon planning when policies fail on different objects.The planner evaluates predicted future trajectories and can improve robustness even when both policies struggle on a shared task.
  • Adaptation to target embodiments: Few-shot finetuning transfers cross-embodiment physical priors into single-embodiment models that surpass SOTA baselines across nearly all perceptual metrics.The transfer improves both DROID and Bridge performance and supports high-fidelity prediction for bimanual YAM and G1 humanoid embodiments.

5 Conclusion, Limitations, and Future Work

CLAP learns physical priors from heterogeneous human and robot videos, bridges diverse action spaces, and supports zero-shot and few-shot adaptation across robot embodiments. The authors also identify hallucinations, data-diversity limits, and scaling costs as areas for future work.

  • Conclusion: CLAP learns generalizable physical priors from heterogeneous human and robot videos.
  • Conclusion: CLAP combines end-effector, language, and latent action representations to bridge disparate embodiment action spaces.
  • Conclusion: CLAP’s cross-embodiment models approach or surpass state-of-the-art single-embodiment baselines and support zero-shot real-world task generalization.
  • Conclusion: Few-shot adaptation of cross-embodiment models provides a paradigm for training high-fidelity single-embodiment video world models.
  • Limitations and Future Work: CLAP remains prone to hallucinations, while extending training to bimanual and humanoid data could increase diversity and improve future prediction and adaptation.
  • Limitations and Future Work: Model and data scaling offer improvement paths but incur significant training and inference costs, motivating efficient architectures.

Safety, Data Privacy, and Consent

The work is computational and theoretical, uses publicly available benchmarks and open-source datasets, and does not involve human participants, live user studies, or personally identifiable information. Video world models condition future-frame prediction on language, observations, and action sequences, with history horizons providing temporal context.

  • Safety, Data Privacy, and Consent: The paper does not involve human participants, live user studies, or personally identifiable information, so IRB approval was not required.
  • Safety, Data Privacy, and Consent: Experiments use publicly available benchmarks and open-source datasets in accordance with their respective licenses.
  • Preliminaries: Video world models predict future outcomes from an initial camera observation conditioned on per-frame actions.
  • Preliminaries: CLAP’s video world model takes a language instruction and past and current observations as inputs, then predicts future frames.
  • Preliminaries: The history and prediction horizons are denoted by H and P, respectively.

B Method

CLAP represents embodiment-specific controls through end-effector, language, and latent actions. End-effector states encode pose and gripper information, while latent actions are inferred directly from video and remain embodiment- and sensor-agnostic.

  • End-effector Actions: End-effector actions store each demonstration as video-aligned trajectories of seven-dimensional end-effector states.
  • End-effector Actions: Each end-effector state contains gripper position, roll–pitch–yaw orientation, and a raw gripper signal.
  • End-effector Actions: CLAP-EE encodes each end-effector state into a 1024-dimensional vector used as a video-model conditioning signal.
  • Language Actions: CLAP-LANG maps per-frame end-effector actions to one natural-language string per timestep.
  • Language Actions: Language actions express gripper position numerically on a zero-to-one scale and may use absolute or relative coordinates.
  • Latent Actions: Latent actions encode transitions between consecutive observed frames as compact 32-dimensional vectors, providing action signals without embodiment or sensor dependence.

B.2 Model Architecture

CLAP uses a frozen-VAE latent video backbone with history and future windows, cross-attention conditioning, and multiple action encoders. Its latent-action model learns compact representations from consecutive frames using a β-VAE objective and a weighted multi-dataset training mixture.

  • Latent Action Model: The latent-action model trains on two-frame RGB clips at 240 × 320 resolution, split into 16 × 16 patches forming a 15 × 20 token grid.
  • Latent Action Model: The model uses a standard β-VAE objective combining pixel reconstruction with KL prior matching, using β = 10^-6.
  • Training: Training uses AdamW with learning rate 5 × 10^-6, weight decay 10^-2, gradient clipping 0.3, fp16, eight GPUs, and global batch size 64.
  • Video Model: The video backbone is an SVD spatio-temporal UNet operating in frozen-VAE latent space with six history frames and five future frames.
  • Conditioning: A shared interface maps per-frame control signals into the UNet’s 1024-dimensional cross-attention token space, using MLPs for end-effector and latent actions and CLIP for language.
  • Conditioning: Action context is applied at the frame level, and dropping it with probability 5% enables classifier-free guidance.
  • Inference: Inference uses an EDM sampler with 50 steps, frame-wise classifier-free guidance, and autoregressive chunked rollout.

C.1 Evaluation Setup

CLAP trains cross-embodiment models on diverse datasets with standardized visual inputs and action conditioning. Real-world deployment uses a lightweight adapter to map end-effector actions into a frozen model’s latent-action space.

  • Training Data and Inputs: CLAP samples diverse cross-embodiment datasets using ratios proportional to dataset complexity and quality.
  • Training Data and Inputs: Multi-view images are resized and vertically stacked into fixed-size 576 × 320 frames, while single-view inputs are repeated across views.The text notes that repeating single-view inputs was preliminarily found unnecessary.
  • Training Data and Inputs: The latent-action model is trained on Open X-Embodiment datasets and additionally incorporates EgoDex.
  • Action Conditioning: A lightweight adapter maps end-effector actions into a 32-dimensional latent-action space while the video world model remains frozen.The adapter’s output head is zero-initialized during training.

C.2 Inference and Timing Results

CLAP is designed for inference on consumer hardware, requiring under 12 GB of VRAM. Inference speed varies by GPU architecture and is affected by cuDNN kernel-autotuning at initialization.

  • Memory and Runtime: Under 12 GB of VRAM is required at inference, with approximately 9.7 GB consumed per call.The measurements were verified on A100 and H200 GPU nodes, and the model can fit on an RTX 3060.
  • Memory and Runtime: 1.49s average subsequent inference time is measured on the H200 under the stated 11-frame, 25-step configuration.The reported variation is 1.49s ± 0.00s across 20 trials.
  • Memory and Runtime: 3.24s average subsequent inference time is measured on the A100-PCIe-40GB, compared with 2.88s on the A100-SXM4-80GB.Both measurements are reported across 20 trials.
  • Memory and Runtime: The first inference call takes about 3 seconds with an initialized cuDNN cache and roughly 15 seconds otherwise.Subsequent calls are faster and depend on GPU architecture.

C.3 Additional Results on Latent Action Models

CLAP’s latent-action models are evaluated as inverse-dynamics models through frame reconstruction and autoregressive rollouts across in-domain and held-out embodiments. The cross-embodiment IDM generalizes better outside training domains than single-embodiment baselines.

  • Evaluation Protocol: CLAP and single-embodiment latent-action models are trained on cross-embodiment, Bridge, and DROID data, then evaluated against DreamDojo-IDM.Evaluation uses latent actions extracted from unseen trajectories to reconstruct ground-truth trajectories.
  • Evaluation Protocol: Frame-to-frame prediction accuracy and open-loop rollout accuracy are evaluated, including trajectories from unseen embodiments such as the UR5 arm.Rollouts use a two-frame skip and begin from a single ground-truth frame for 30 steps in the figure protocol.
  • Generalization Results: CLAP’s IDM achieves the highest or second-highest perceptual scores across datasets, including held-out data and OXE-Mix.This pattern holds for both frame-to-frame reconstruction and open-loop rollout results.
  • Generalization Results: CLAP’s IDM outperforms off-domain single-embodiment IDMs while matching or closely trailing them on their target datasets.Single-embodiment IDMs perform best on native distributions but degrade sharply out of domain.
  • Generalization Results: CLAP’s IDM achieves superior 30-step autoregressive generalization beyond the training-dataset domains compared with other methods.

C.4 Additional Results on Cross-Embodiment Video Modeling

Additional experiments evaluate CLAP’s planning, adaptation, and reinforcement-learning capabilities across single-arm, bimanual, and novel-embodiment settings. The reported results include improved carrot-task success through video-model RL and broad cross-embodiment comparisons, alongside explicit scope and fairness questions.

  • Policy Finetuning: CLAP supports policy finetuning with reinforcement learning using Diffusion Steering via Reinforcement Learning and dense per-frame progress rewards.A convolutional neural network estimates task progress for the RL reward signal.
  • Policy Finetuning: 80% to 88% success improves for the carrot task after RL-based finetuning in CLAP’s video model.The towel-task success rate is not degraded, based on 25 carrot-task and 20 towel-task trials.
  • Cross-Embodiment Modeling: Tables 4–6 report cross-embodiment video-world-model performance on DROID, Bridge, and OXE-Mix.
  • Action Conditioning: Tables 7–9 compare relative and absolute action-space conditioning across DROID, Bridge, and OXE-Mix.
  • Planning and Adaptation: Planning and few-shot adaptation are evaluated in single-arm DROID, bimanual YAM, pretrained embodiments, and novel embodiments.The corresponding evaluations are summarized in Tables 10–14.
  • Policy Finetuning: Table 15 reports success rates after RL-based policy finetuning with CLAP.

Detailed Q&A

The Q&A clarifies CLAP’s novelty, practical trade-offs, evaluation choices, and deployment boundaries. It supports cross-embodiment adaptation while acknowledging latency, representation-specific limitations, hallucinations, and uneven performance across platforms.

  • Novelty: CLAP’s novelty is a cross-embodiment paradigm for action-conditioned video modeling, rather than a new backbone or dataset.The framework is positioned as distinct from derivative single-embodiment models.
  • Implementation constraints: CLAP does not run in real time: inference averages 3.24s on A100-PCIe-40GB, 2.88s on A100-SXM4-80GB, and 1.49s on H200.These timings use 11 frames and 25 denoising steps, averaged over 20 trials.
  • Limitations: CLAP is not yet a replacement for well-tuned physics-based simulators and does not universally outperform domain-specific single-embodiment models.Its accuracy also depends on training scale, distribution coverage, and proximity between target and training platforms.
  • Action representations: No single action representation is universally optimal: language reduces domain gaps but lacks precision, end-effector poses are precise but need labels, and latent actions use unlabeled data but create deployment gaps.The representations encode different trade-offs rather than forming a uniformly ranked set.
  • Action representations: Learned latent actions enable cross-embodiment training on unlabeled internet-scale videos by avoiding manual action annotations.This expands usable training data, although latent actions introduce a deployment domain gap.
Loading 2608.27406v1…