PaperlayerView exact revision on arXiv

Source-linked AI summary

Vidar: Embodied Video Diffusion Model for Generalist Manipulation

Yao Feng, Hengkai Tan, Xinyi Mao, Chendong Xiang, Guodong Liu, Shuhe Huang, Hang Su, Jun Zhu

arXiv:2507.12898v4cs.LGcs.AIcs.CVcs.RO

TL;DR

Vidar targets the high data requirements and embodiment sensitivity of general-purpose manipulation. It combines an embodied video diffusion prior with a masked inverse dynamics adapter, using unified cross-robot pre-training and lightweight target adaptation. With only 20 minutes of demonstrations on an unseen robot, it outperforms baselines and generalizes to unseen tasks and backgrounds.

  • Problem

    General-purpose manipulation requires large, hardware-specific demonstrations, while control can degrade under embodiment, background, and viewpoint changes.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    Vidar decouples video generation from action prediction, using a video model for temporally coherent rollouts and a lightweight inverse dynamics model for robot-specific controls.

    Source: §1 · p. 2, §2.1 · p. 3, §2.2 · p. 4

  • Results

    With only 20 minutes of demonstrations on an unseen robot, Vidar achieves state-of-the-art multi-task performance and outperforms baselines by 58% over VPP [14] and 40% over UniPi [11].

    Source: §1 · p. 2, §3.2 · p. 6, §5 · p. 10

  • Takeaways & Limitations

    Vidar supports a scalable one-prior-many-embodiments recipe in which large video priors are aligned to new robots with minimal on-robot data.

    Source: §1 · p. 2, §5 · p. 10

    Deployment of generalist robotic systems raises privacy, safety, and accountability concerns in sensitive human-robot interaction settings.

    Source: §6 · p. 10

Abstract

from arXiv · show

Scaling general-purpose manipulation to new robot embodiments remains challenging: each platform typically needs large, homogeneous demonstrations, and end-to-end pixel-to-action pipelines may degenerate under background and viewpoint shifts. Based on previous advances in video-based robot control, we present Vidar, consisting of an embodied video diffusion model as the generalizable prior and a masked inverse dynamics model (MIDM) as the adapter. We leverage a video diffusion model pre-trained at Internet scale, and further continuously pre-train it for the embodied domain using 750K multi-view trajectories collected from three real-world robot platforms. For this embodied pre-training, we introduce a unified observation space that jointly encodes robot, camera, task, and scene contexts. The MIDM module learns action-relevant pixel masks without dense labels, grounding the prior into the target embodiment's action space while suppressing distractors. With only 20 minutes of human demonstrations on an unseen robot (1% of typical data), Vidar outperforms state-of-the-art baselines and generalizes to unseen tasks, backgrounds, and camera layouts. Our results suggest a scalable recipe for "one prior, many embodiments": strong, inexpensive video priors together with minimal on-robot alignment.

1 Introduction

Vidar addresses data scarcity and embodiment heterogeneity in general-purpose bimanual manipulation by combining a transferable video prior with lightweight action decoding. Its video-based pipeline uses cross-embodiment pre-training and masked attention to support precise control from limited demonstrations.

Problem

  • General-purpose and bimanual manipulation remains difficult because combinatorial action spaces, contact dynamics, long-horizon reasoning, and platform-specific details increase data requirements.Human demonstration corpora typically contain only tens to hundreds of hours [3], far below Internet-scale video collections [6].Source: §1 · p. 1

Motivation

  • Video provides abundant temporal interaction cues and supports embodiment-agnostic knowledge that can later specialize to new morphologies with fewer platform-specific samples.The formulation shifts representation learning toward abundant Internet and robotic video rather than costly demonstrations.Source: §1 · pp. 1–2

Video prior

  • 750K multi-view bimanual clips from three robot platforms are aligned into a unified observation space for embodied video pre-training.The unified space is intended to mitigate viewpoint and morphology gaps while supporting feasible, physically credible control.Source: §1 · p. 2

Action adaptation

  • Vidar uses a video diffusion model as a transferable prior and a masked inverse dynamics model to decode robot-specific actions.The MIDM learns action-relevant regions without pixel-level supervision, filtering distractors during action prediction.Source: §1 · p. 2

Empirical results

  • 58% over VPP [14] and 40% over UniPi [11] with only 20 minutes of demonstrations on an unseen robot, while generalizing to reflective environments.Vidar also achieves state-of-the-art performance on the RoboTwin [13] multi-task benchmark.Source: §1 · p. 2

2 Method

Vidar factorizes manipulation through a transferable video prior and a lightweight inverse-dynamics adapter, addressing embodiment shifts, limited demonstrations, and visual distractors. Its video model uses unified multi-view conditioning and test-time selection, while MIDM grounds generated videos into robust robot actions.

2.1 Problem Formulation

  • Vidar factorizes the manipulation policy into video generation and inverse dynamics, shifting representation learning to abundant video while limiting target-platform training to a lightweight adapter.The generator produces temporally coherent, physically plausible rollouts, and the inverse dynamics model maps short video windows to robot-specific controls.Source: §2.1 · p. 3

2.2 Video Generation Model

  • The video generator uses rectified flow to model pixel velocities from noisy videos toward target videos, trained by matching the predicted velocity to the constant flow between endpoints.This provides the generative mechanism for producing temporally coherent video rollouts.Source: §2.2 · p. 4
  • A unified observation space aggregates multiple camera views with robot, camera, and task instructions to support video prediction across heterogeneous embodiments.The space excludes actions, allowing the video model to learn world evolution rather than embodiment-specific control outputs.Source: §2.2 · p. 4
  • The embodied video model is pretrained on approximately 750K multi-view episodes from three robot platforms and fine-tuned with clipped human demonstrations from the target platform.The pipeline is designed to use limited demonstrations while adapting video prediction to the target robot.Source: Figure 1 · p. 3, §2.2 · p. 4
  • Test-time scaling samples multiple candidate trajectories and ranks them with a pretrained evaluator, selecting the highest-scoring rollout to reduce stochastic generation failures.The strategy addresses variance in quality, physical plausibility, and task relevance without retraining the model.Source: §2.2 · p. 4

2.3 Masked Inverse Dynamics Model

  • MIDM learns spatial masks and predicts actions from masked frames, using sparse action-relevant regions to suppress background noise and distractors without pixel-level segmentation labels.Its loss combines action-regression error with L1 sparsity regularization, and it can predict embodiment-specific or embodiment-agnostic actions.Source: §2.3 · pp. 4–5

3 Experiments

Vidar is evaluated across simulation and real-world manipulation settings using heterogeneous robotic data, limited target-platform demonstrations, and comparisons against video- and robot-policy baselines. It achieves strong benchmark performance and generalization, while ablations support the contributions of embodied pre-training, MIDM, and test-time scaling.

3.1.1 Datasets

  • Vidar uses 20 minutes of real-world human demonstrations covering 81 tasks across 232 episodes on target domains unseen during pre-training.The experiments include seen tasks and backgrounds, unseen tasks, and unseen backgrounds, including reflective surfaces.Source: §3.1.1 · p. 5, §3.2 · pp. 6–7

3.1.2 Training and Inference

  • The experiments use Wan2.2 for simulation and Vidu 2.0 for real-world tests, with continued pre-training followed by target-domain fine-tuning and MIDM training.Vidar uses open-loop control; real-world test-time scaling generates three videos in parallel and reranks them with GPT-4o.Source: §3.1.2 · pp. 5–6

3.2 Experimental Results

  • Vidar achieves state-of-the-art average success rates across RoboTwin scenarios compared with Pi0.5 in the multi-task setting.The benchmark covers 50 tasks and 100 episodes, with results summarized in Table 1.Source: §3.2 · p. 6, Table 1 · p. 7
  • Vidar outperforms the compared methods across seen-task, unseen-task, and unseen-background scenarios, with Table 2 reporting the corresponding success rates.The authors attribute baseline weaknesses to limited heterogeneous robotic data for UniPi and noisy single-pass predicted features for VPP.Source: §3.2 · p. 7, Table 2 · p. 7
  • Vidar surpasses Pi0.5 by 35% on seven seen tasks and 54% on seven unseen tasks when evaluated with Wan2.2 and HunyuanVideo.These additional experiments assess the generality of the approach across open-source video-generation models.Source: §3.2 · p. 7
  • Embodied pre-training in the unified observation space improves VBench video consistency and frame quality in the unseen target domain.Table 3 evaluates the original and pre-trained video models under the target-domain setting.Source: §3.2 · p. 7, Table 3 · p. 7
  • MIDM generalizes better than the ResNet baseline on testing success rates and l1 errors while learning action-relevant masks without additional supervision.Figure 3 shows masks focusing on robotic arms despite reflective unseen backgrounds.Source: §3.2 · p. 8, Table 4 · p. 8, Figure 3 · p. 9

3.3 Ablation Study

  • The ablation study finds that both MIDM and test-time scaling improve success rates across the three evaluated scenarios.The “w/o MIDM” condition uses the ResNet baseline.Source: §3.3 · p. 9, Table 5 · p. 9

4 Related Work

Prior work explores video and action generation for robotic control, but existing approaches remain constrained by data requirements, coupled training, single-arm or single-view settings, and limited heterogeneous embodied pre-training.

Vision-Language-Action Models (VLAs)

  • Vision-language-action models require large, task-specific datasets and couple actions to individual models, limiting transfer across heterogeneous embodiments.Examples include RT-1 with 130K episodes and RT-2 with 1B image-text pairs.Source: §4 · p. 9

Coupled Video Generation and Action Synthesis

  • Joint video-generation and action-synthesis methods improve physical consistency but require end-to-end training, reducing flexibility and adaptability.VPP, UVA, and VidMan connect spatiotemporal representations with action prediction in different ways.Source: §4 · p. 9

Video Generation Models for Embodied AI

  • Video-generation models for embodied AI encode physical dynamics and reduce reliance on robotics data, but many remain limited to single-arm, single-camera, or tool-dependent settings.Most existing methods also do not use heterogeneous embodied videos for pre-training.Source: §4 · pp. 9–10

5 Conclusion

Vidar addresses data scarcity and embodiment heterogeneity in bimanual manipulation by combining unified-space diffusion pre-training with masked inverse dynamics. The framework uses multi-view observations and language instructions to support action prediction and generalization with minimal demonstrations.

5 Conclusion

  • Vidar combines large-scale diffusion-based video pre-training with a masked inverse dynamics model for generalizable bimanual manipulation.The framework targets data scarcity and embodiment heterogeneity.Source: §5 · p. 10
  • Vidar supports accurate action prediction from multi-view visual observations and language instructions with minimal demonstrations in new environments.The conclusion reports strong generalization to unseen tasks and backgrounds.Source: §5 · p. 10

6 Ethics Statement

Vidar may accelerate deployment of capable bimanual robots, while generalist robotic systems raise privacy, safety, and accountability concerns in sensitive human-robot interaction settings.

6 Ethics Statement

  • Deployment of generalist robotic systems raises privacy, safety, and accountability concerns, particularly in sensitive close human-robot interaction domains.These concerns accompany the potential deployment of capable bimanual robots in real-world environments.Source: §6 · p. 10

A Dataset Details

The datasets are standardized across robot and camera configurations to support the unified observation space, with additional filtering, annotation augmentation, and view adjustments. The target datasets also include camera layouts substantially different from pre-training data.

A Dataset Details

  • Dataset preparation records robot and camera setups as part of the unified observation space and samples datasets proportionally to their sizes.RoboTwin camera positions capture both arms, while Agibot-World episodes are segmented using frame-level annotations.Source: A Dataset Details · p. 14
  • The datasets filter out episodes with fewer than three views or shorter than four seconds and augment Vidar task annotations with GPT-4o.These operations standardize the pre-training data and task descriptions.Source: A Dataset Details · p. 14
  • RoboTwin and Vidar use robot arms and camera viewpoints that differ from the pre-training datasets, including a far, high central camera in Vidar.This creates a substantial viewpoint shift between pre-training and target data.Source: A Dataset Details · p. 14

B Training and Inference Details

Vidar combines large-scale video diffusion training with lightweight inverse-dynamics adaptation, using cloud-based generation and local action inference. Training and inference include multi-view processing, MIDM masking, and optional GPT-4o test-time selection.

Training

  • 64 NVIDIA Ampere-series 80GB GPUs trained Vidu 2.0 for 23,000 iterations in about 64 hours, while MIDM training used 8 NVIDIA Hopper-series 80GB GPUs for 60,000 iterations in about 5 hours.Both procedures used full-parameter fine-tuning for the video model; Pi0.5 was trained for 55,000 iterations as a baseline.Source: B Training and Inference Details · pp. 14–15
  • MIDM hyperparameters are documented in Table 7, while Pi0.5 hyperparameters and the baseline training setup are documented in Table 8 and the accompanying training description.The supplied passages identify these tables as configuration references but do not provide their parameter values.Source: Table 7 · p. 15, Table 8 · p. 15, B Training and Inference Details · p. 15

Inference

  • During inference, video diffusion runs in the cloud while lightweight MIDM runs locally, and GPT-4o selects among uniformly sampled generated frames for test-time scaling.Each pairwise GPT-4o comparison costs about $0.003 and contributes about 25% of total latency when K = 3.Source: B Training and Inference Details · p. 15, Figure 4 · p. 15

Diagnostics

  • The implementation includes visual diagnostics of MIDM masks across λ values and segmentation results obtained by splitting concatenated frames into three views before applying RoboEngine [21].These visuals document mask behavior and the multi-view segmentation procedure rather than reporting numerical outcomes.Source: Figure 5 · p. 16, Figure 6 · p. 16

C Experimental Results

The experiments evaluate Vidar in challenging multi-task simulation and real-world settings, including hyperparameter sensitivity and segmentation quality. The reported results support strong performance with limited data, while RoboEngine shows shortcomings in this setting.

Simulation Experiments

  • Vidar and Pi0.5 are evaluated in multi-task RoboTwin 2.0 experiments under clean and adjusted-view low-data regimes, with success rates averaged over 100 episodes.The testing setup uses K = 1, a maximum of 180 steps, and 50 or 20 episodes per task depending on the data regime.Source: C Experimental Results · p. 16

Hyperparameter Analysis

  • λ = 3 × 10−3 yields the best real-world performance for MIDM, while success rates remain robust across a broad range of λ values.The result is reported in Table 12 and illustrated in Figure 5.Source: C Experimental Results · p. 16

Segmentation Analysis

  • RoboEngine [21] often detects only one arm per frame, misses grippers in wrist-camera views, and lacks temporal consistency across frames.These segmentation failures are illustrated in Figure 6.Source: C Experimental Results · p. 16

D Additional Real-World Experiments

Additional experiments test Vidar with alternative video backbones and provide expanded benchmark, hyperparameter, and visualization results. Across the reported Wan 2.2 evaluation, Vidar consistently outperforms Pi0.5 on average.

D Additional Real-World Experiments

  • Vidar consistently outperforms Pi0.5 on average across 14 Wan 2.2 evaluation tasks comprising 7 seen and 7 unseen tasks.The comparison uses an expanded fine-tuning set of 2,307 episodes, with both models trained for 55,000 iterations.Source: D Additional Real-World Experiments · p. 16, Table 14 · p. 20
  • Vidar is also evaluated with the HunyuanVideo backbone on six tasks, with results reported in Table 15 and Figure 7.The authors state that implementations for HunyuanVideo and MIDM are planned for open release.Source: D Additional Real-World Experiments · p. 16, Table 15 · p. 20
  • The additional real-world experiments include detailed success rates for different methods and configurations, with L, R, and B denoting left-arm, right-arm, and bimanual use.These results are organized in Table 11, while Tables 9 and 10 report RoboTwin 2.0 success rates under clean and randomized scenarios.Source: Table 9 · p. 17, Table 10 · p. 18, Table 11 · p. 19
  • Table 12 analyzes MIDM’s λ, Table 13 lists Wan 2.2 and HunyuanVideo hyperparameters, and Tables 14–15 report success rates for the reproduced backbones.The supplied passages identify the scope of these tables but do not provide their individual cell values.Source: Table 12 · p. 19, Table 13 · p. 19, Table 14 · p. 20

E Additional Visualizations

  • Additional visualizations include more demonstrations with two failed cases in Figure 8.The figure extends the qualitative evidence beyond the primary reported evaluations.Source: E Additional Visualizations · p. 20

F Hardware Details

The target platform differs substantially from the platforms represented during pre-training, motivating its use as a test of scarce-data adaptation. Supplementary figures and tables document the hardware, generated predictions, executions, and language-model evaluation prompt.

F Hardware Details

  • Vidar assumes the intermediate video contains all information needed for action prediction, but this fails for some platforms when arm joints fall outside the camera’s field of view.The target platform changes the center-camera position so both arms are fully captured, unlike the pre-training configuration.Source: F Hardware Details · p. 20
  • The target platform differs from every platform used during pre-training, making it a testbed for adaptation with scarce data.The hardware details are provided in Figure 9 and Table 16.Source: F Hardware Details · p. 20, Figure 9 · p. 22, Table 16 · p. 22

G Large Language Model Usage

  • Figures 7 and 8 show Vidar’s predicted videos alongside corresponding executions for challenging tasks, with Figure 8 including both successful and failed cases.Figure 9 presents the robotic platform.Source: Figure 7 · p. 21, Figure 8 · p. 22, Figure 9 · p. 22

Related papers

Loading 2507.12898v4…