PaperlayerView exact revision on arXiv

Source-linked AI summary

OSCAR: Omni-Embodiment Action-Conditioned World Model for Robotics

Zhuoyuan Wu, Jun Gao

arXiv:2606.04463v2cs.RO

TL;DR

Robot policy evaluation with video world models requires diverse scenarios, precise action following, and generalization across embodiments. OSCAR combines a curated robotics and egocentric-human dataset with unified 2D skeleton conditioning. It improves several video-generation qualities over existing baselines, and its virtual policy evaluations correlate significantly with real-world evaluations.

  • Problem

    Action-conditioned world models for robot policy evaluation must address limited scenario diversity, imprecise action following, and poor generalization across embodiments.

    Source: §1 · p. 1

  • Method

    OSCAR curates and filters robotics and egocentric-human data and uses 2D kinematic skeletons as a shared conditioning representation across robot arms and human hands.

    Source: §1 · p. 2, §3.2 · p. 4, §4 · p. 5

  • Results

    OSCAR significantly improves action following, appearance quality, and motion consistency over existing baselines, and its virtual policy evaluation significantly correlates with real-world evaluation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    OSCAR’s correlation with real-world evaluation supports using virtual generated worlds to reduce policy-evaluation costs and speed development iterations.

    Source: §1 · p. 2

    Data scale is limited by the availability and quality of camera calibration and kinematic annotations, while scaling beyond the 2B-parameter backbone requires more compute.

    Source: §6 · p. 8

Abstract

from arXiv · show

We present OSCAR, a precise action-conditioned video world model that generalizes across different robot embodiments and enables robot policy evaluation. Existing video world models face three main challenges for real-world robot evaluation: limited scenario diversity in current robot training datasets, imprecise action following, and poor generalization across embodiments for broad adoption. We tackle these challenges from two perspectives. At its core is a large-scale standardized data pipeline that curates, filters, and deduplicates broad robotics and egocentric human datasets, yielding a clean joint-training dataset that spans diverse tasks, scenarios, actions, and robot embodiments. To condition the video model, we adopt 2D kinematic skeleton rendering as a unified conditioning representation that generalizes across different robot arms or even human hands. We finetune the Cosmos-Predict2.5-2B model on a single GH200 GPU. Our model achieves significant improvement on action following, appearance quality, and motion consistency, compared to existing baselines, which either have a much larger model size or require more GPUs. We further deploy OSCAR to evaluate robot policies from RoboArena. Extensive experiments demonstrate the significant correlation between our virtual policy evaluation in OSCAR and real-world evaluation, paving the way for the future where robot policies can be purely evaluated in virtual generated worlds.

1 Introduction

Action-conditioned world models could support policy evaluation and robot action-planning pretraining, but faithful evaluation requires precise action following, broad scenario coverage, and generalization across embodiments. OSCAR combines a diverse standardized dataset with skeleton conditioning, and its virtual evaluations correlate significantly with real-world evaluations.

1 Introduction

  • Action-conditioned world models predict future states from robot action sequences, supporting policy evaluation, reinforcement learning, and robot action-planning pretraining.These uses motivate the need for action-conditioned forecasting.Source: §1 · p. 1
  • A generalizable world model must precisely indicate when and where actions occur, cover diverse tasks and scenarios, and generalize across robot embodiments.Precise frame- and pixel-level action signals are needed to provide meaningful signals for downstream policy evaluation.Source: §1 · pp. 1–2
  • Latent-action approaches can handle multiple embodiments but often follow actions imprecisely, while explicit conditioning trades off accuracy, robustness, and whole-arm motion.VAP uses skeleton rendering but is constrained by limited, unfiltered data across two robot embodiments and human clips.Source: §1 · p. 2
  • OSCAR curates diverse robotics and egocentric human data and uses 2D skeletons as a shared conditioning representation for robots and people.Skeletons depend on the kinematic chain rather than embodiment-specific appearance, which may mitigate texture overfitting.Source: §1 · p. 2
  • OSCAR’s virtual policy evaluation shows significant correlation with real-world evaluation, while improving action following, appearance quality, and motion consistency over existing baselines.The cited comparison includes baselines with much larger models or greater GPU requirements.Source: §1 · p. 2

2 Related Work

Video world models for manipulation condition video generation using text, latent actions, or explicit spatially aligned inputs. Policy-evaluation approaches range from open-loop scoring to closed-loop policy execution, with OSCAR evaluated against RoboArena rankings.

Video World Models for Robot Manipulation

  • Manipulation world models use text guidance, latent-action embeddings, or explicit spatially aligned renderings to condition video generation.Explicit renderings include pointmaps, occupancy, gripper states, and 2D skeleton projections; OSCAR adopts the skeleton-based interface.Source: §2 · pp. 2–3
  • OSCAR’s overview encodes the first frame and rendered skeleton with a VAE, injects the skeleton latent with the noisy video latent, then denoises and decodes the video.The diagram presents conditioning and video generation as separate stages.Source: Figure 2 · p. 3

Video World Models for Policy Evaluation

  • Policy-evaluation methods include open-loop scoring from recorded actions and closed-loop policy runs scored by VLMs or humans; OSCAR follows the RoboArena protocol using Pearson correlation and MMRV against BT/Elo rankings.The related work also identifies EWMBench as a complementary metric suite.Source: §2 · p. 3

3 Method

OSCAR builds on a video diffusion model that conditions future-frame generation on an initial image, and uses 2D skeleton renderings as explicit action guidance. The rendering represents robot and human motion in a shared form while leaving fine visual detail to the video prior.

3.1 Preliminaries

  • Cosmos-Predict2.5 encodes a video into spatiotemporal latents and trains its DiT to predict a velocity field between noise and the target latent.With image conditioning, the first frame’s latent is kept clean at each denoising step while future frames are denoised.Source: §3.1 · p. 3

3.2 Skeleton Rendering as a Unified Conditioning

  • Skeleton renderings provide explicit action guidance while remaining largely invariant to arm textures and materials; the first RGB frame anchors the robot’s appearance and scene.The design balances action precision and cross-embodiment generalization without adding fine-grained surface textures to the action condition.Source: §3.2 · p. 4
  • Rasterisation draws pixel-space line segments between projected kinematic-tree endpoints and circles at projected vertices, leaving all other pixels black.The resulting 2D visualization of the kinematic chain is described as the cheapest geometry information indicating the action.Source: §3.2 · p. 4
  • The skeleton rendering is supplied to the DiT as a frame-aligned second RGB video stream, encoded into a latent and summed with the target-video tokens.Both streams use the WAN 2.1 VAE and separate patch embedders before their token tensors are summed for denoising.Source: §3.2 · p. 4
  • Robot arms and human hands use the same 2D line-drawing representation by swapping in the hand’s kinematic model and pose parameters.The skeleton constrains coarse motion while the pretrained video prior supplies plausible fine-grained visual detail, allowing human egocentric demonstrations to broaden training diversity.Source: §3.2 · pp. 4–5

4 Data Pipeline

OSCAR’s data pipeline filters a broad mix of robotics and egocentric human videos to build a diverse training corpus. It removes low-quality and redundant clips, then captions retained episodes.

4 Data Pipeline

  • The four-stage pipeline filtered 180,657 episodes from 2,165,359 source videos to construct a diverse dataset for action-conditioned world models.The dataset statistics are reported in Table 1.Source: §4 · p. 5

4.1 Data Curation

  • Robotics and egocentric human sources were selected to broaden scene coverage across robot embodiments, tasks, and environments.The collection details span robot and human video sources.Source: §4.1 · p. 5

4.2 Data Filtering

  • Clips were filtered for sufficient length, static cameras, meaningful manipulator action, and a visible robot skeleton.The filters were applied to human and robot data.Source: §4.2 · p. 5

4.3 Semantic Deduplication

  • Semantic deduplication first clusters visually similar clips, then checks trajectory similarity so shared scenes with substantially different motion are retained.Pairs above 0.95 SigLIP cosine similarity proceed to trajectory verification; duplicates must fall below an adaptive RMS-distance threshold.Source: §4.3 · pp. 5–6

4.4 Data Captioning

  • Retained episodes are captioned with Qwen3-VL-30B-A3B-Instruct, sampled at 15 fps, except DROID episodes, sampled at 1–2 fps for efficiency.DROID episodes are typically five times longer than episodes from other sources.Source: §4.4 · p. 6

5 Experiments

OSCAR is evaluated against seven baselines on a four-embodiment benchmark, with ablations of its conditioning and training-data choices. It ranks best or second-best on most metrics and its virtual policy evaluations correlate significantly with real-world deployment.

5.1 Experimental Settings

  • The benchmark contains 200 manipulation clips across six datasets and four embodiments, comparing OSCAR with seven baselines using quality, coherence, fidelity, and speed metrics.All models are timed on the same GH200 GPU and evaluated on their first 49 frames.Source: §5.1 · p. 6

5.2 Comparison with Baselines

  • OSCAR ranks best or second-best on most metrics and outperforms 14B-parameter Kinema4D despite using a 2B model.Figure 4 qualitatively compares two embodiments; OSCAR is reported to have better visual quality and precise action following than five baselines.Source: §5.2 · p. 6, Figure 4 · p. 7
  • Text-only control is weakest, latent-action guidance is limited by training embodiments, and explicit spatial guidance performs best; skeleton conditioning offers the best accuracy/generalisation trade-off.Dense point-maps overfit in-distribution.Source: §5.2 · p. 7

5.3 Ablation Studies

  • Ablations find latent actions imprecise, while mesh and skeleton conditioning are statistically indistinguishable across seven metrics; adding human data improves performance, and warm-starting accelerates convergence.Skeleton rendering is selected because it supports incorporating human data without robot-specific URDF assets.Source: §5.3 · pp. 7–8

5.4 Policy Evaluation

  • On 65 sessions covering seven RoboArena policies, OSCAR’s virtual policy evaluation significantly correlates with real-world deployment, and skeleton rendering gives the strongest correlation among the tested conditions.The evaluation also compares pairwise policy rankings against RoboArena results.Source: §5.4 · p. 8, Table 4 · p. 8

6 Conclusion

OSCAR combines a standardized joint-training dataset with 2D kinematic skeleton conditioning to support action following and cross-embodiment generalization. It outperforms larger or more resource-intensive baselines on several video qualities, and its policy evaluations correlate strongly with real-world evaluations.

6 Conclusion

  • OSCAR’s standardized pipeline builds a joint-training corpus spanning diverse tasks, scenes, and actions across four robot embodiments and the human hand.The pipeline curates, filters, and deduplicates robotics and egocentric human video.Source: §6 · p. 8
  • A 2D kinematic skeleton provides one conditioning representation across robot arms and human hands, with embodiment changes handled by updating the kinematic specification.Texture-free rendering is used to avoid binding motion to specific robot appearance.Source: §6 · p. 8
  • Finetuned on one GH200 GPU, the 2B Cosmos-Predict2.5 model outperforms baselines using more parameters or GPUs on action following, appearance quality, and motion consistency.The model is deployed on RoboArena, where virtual policy evaluation correlates strongly with real-world evaluation.Source: §6 · p. 8
  • Training-data scale is constrained by camera-calibration and kinematic-annotation availability and quality; larger backbones may improve fidelity and generalization but require more compute.Calibration errors degrade skeleton–RGB alignment, limiting which raw videos can be reliably converted into training data.Source: §6 · p. 8

A.1.1 Dataset details

The appendix details the sources and dataset-specific characteristics behind OSCAR’s collection. Robot sources span distinct manipulation settings, while the human sources are recorded from head-mounted cameras in everyday environments.

Robot sources

  • Robot sources cover tabletop, mobile, humanoid, and synthetic settings, with varied tasks, skills, and scenes across several embodiments.Examples include 147 RH20T tasks, 86 DROID tasks across 564 scenes, and 217 AgiBot World tasks.Source: §A.1.1 · p. 12

Human sources

  • EgoDex records 194 everyday tabletop tasks, while EPIC-Kitchens captures unscripted cooking in 45 home kitchens across four cities using head-mounted cameras.The head-mounted setup allows recording across homes and kitchens without a fixed rig.Source: §A.1.1 · p. 12

RH20T

  • RH20T is restricted to cfg5 Franka and cfg7 KUKA because cfg1 lacks proprioceptive state and cfg3 has camera-extrinsic miscalibration that causes large skeleton–image offsets.Released hours are estimated from cfg5 and cfg7 episode counts because per-configuration hours are not published.Source: §A.1.1 · p. 12

A.1.2 Captioning prompt

The captioning prompt directs a video captioner to produce a factual, detailed English description of the video and its motion. It specifies content and style requirements and provides rewritten-prompt examples.

A.1.2 Captioning prompt

  • The prompt requires an English caption that accurately describes the video’s subject, actions, context, and environment.It calls for factual, precise, natural-sounding language that preserves the input video’s intent.Source: §A.1.2 · p. 12
  • Captions should emphasize movement and camera angles, use simple direct verbs, and include visible details such as clothing and backgrounds.Source: §A.1.2 · p. 12
  • The prompt targets captions of around 80–100 words and includes four rewritten-prompt examples spanning photographic, anime, game-art, and poster imagery.Source: §A.1.2 · pp. 12–13

A.2 Evaluation benchmark selection

The appendix details benchmark selection, training, conditioning, and evaluation procedures, alongside additional quantitative and qualitative results. It also describes human-scene generation and the VLM-based comparison with real RoboArena evaluations.

A.2 Evaluation benchmark selection

  • The 200-clip evaluation benchmark removes near-duplicate scenes using caption and motion embeddings, requires the end effector to remain visible, and balances per-embodiment quotas.The quotas are proportional to source-corpus size, balancing long-tail KUKA and Franka data against AgiBot.Source: §A.2 · p. 13

A.3 Training implementation details

  • Training uses frequency-tempered sampling and event-biased 81-frame windows, with 15k robot-only iterations before training on the robot-plus-human mixture.The model is finetuned from Cosmos-Predict2.5-2B [6] with AdamW, learning rate 3×10−5, batch size 16, and inference guidance scale 6.Source: §A.3 · p. 13

A.4 Latent-action conditioning pathway

  • Across embodiments, the method uses a fixed 14-D action vector: it concatenates AgiBot G1’s two 7-D arms and zero-pads the remaining dimensions for single-arm robots, avoiding embodiment-specific prediction heads.Each clip has 80 transitions; its (80, 14) action tensor is flattened to 1120 dimensions and projected into conditioning tokens. The tokens are added to the timestep embedding and adaptive LayerNorm modulation signal at every frame.Source: §A.4 · p. 14

A.6 Per-embodiment quantitative results

A.7 Per-embodiment qualitative results

  • Additional qualitative comparisons cover four remaining robot embodiments, two more AgiBot G1 and DROID examples, and conditioning and data-composition ablations across six embodiments.The ablation visuals compare ground truth, skeleton, mesh, latent-action, robot-only, and human-data configurations.Source: §A.7 · p. 15, §A.8 · p. 15, Figure 7 · p. 20

A.9 Human-data qualitative samples

  • OSCAR also generates human scenes conditioned on MANO skeletons, with EgoDex and EPIC-Kitchens in training and Ego4D subsets reserved as out-of-distribution probes.The samples pair ground truth and predictions at three time steps.Source: §A.9 · p. 15, Figure 9 · p. 22

A.10.1 Platform and camera calibration

  • RoboArena sessions use estimated camera calibration, and evaluation compares seven-policy rankings and success rates using rank violation, Spearman ρ, Pearson r, and success-rate error.The appendix also shows paired OSCAR and real-robot rollouts at six time steps, with disagreements between human labels and VLM verdicts highlighted; Table 12 lists external-asset licenses.Source: §A.10.1 · p. 16, §A.10.2 · p. 16, §A.10.4 · p. 17

A.10.3 VLM success scoring

  • The VLM matched human binary labels on 78 of 100 balanced real-robot clips, with specificity 0.90, recall 0.66, and Pearson r = 0.58 (p < 10−7).The authors report that it under-reports success rather than inflating it; qualitative disagreements include false negatives on partial attempts.Source: §A.10.3 · pp. 16–17, §A.10.4 · p. 17

Related papers

Loading 2606.04463v2…