PaperlayerView exact revision on arXiv

Source-linked AI summary

VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation

Youpeng Wen, Junfan Lin, Yi Zhu, Jianhua Han, Hang Xu, Shen Zhao, Xiaodan Liang

arXiv:2411.09153v1cs.CVcs.RO

TL;DR

Robot action prediction must make effective use of limited robotic data, including relationships between visual observations and actions. VidMan first learns environment dynamics through video diffusion, then uses a layer-wise adapter to infer actions. It reports an 11.7% relative improvement over state-of-the-art baselines on CALVIN and over 9% precision gains on small-scale OXE; the authors identify 2D-only vision as a limitation for tasks requiring accurate 3D spatial understanding.

  • Problem

    High-quality robot data are costly to acquire, and fitting data without accounting for visual observation–action relationships may use them suboptimally.

    Source: §1 · p. 1

  • Method

    VidMan pre-trains on OXE to predict future visual trajectories, then uses a layer-wise self-attention adapter with shared parameters to infer dynamics-modulated actions.

    Source: §1 · p. 2

  • Results

    VidMan achieves an 11.7% relative improvement over state-of-the-art baselines on CALVIN and over 9% precision gains on the OXE small-scale dataset.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that learning data facets in distinct stages gives VidMan an inverse-dynamics inductive bias that improves generalization, especially with limited data.

    Source: §1 · p. 2

    The model uses only 2D vision and lacks 3D perception, affecting tasks that demand accurate 3D spatial understanding.

    Source: §A.1 · p. 14

Abstract

from arXiv · show

Recent advancements utilizing large-scale video data for learning video generation models demonstrate significant potential in understanding complex physical dynamics. It suggests the feasibility of leveraging diverse robot trajectory data to develop a unified, dynamics-aware model to enhance robot manipulation. However, given the relatively small amount of available robot data, directly fitting data without considering the relationship between visual observations and actions could lead to suboptimal data utilization. To this end, we propose VidMan (Video Diffusion for Robot Manipulation), a novel framework that employs a two-stage training mechanism inspired by dual-process theory from neuroscience to enhance stability and improve data utilization efficiency. Specifically, in the first stage, VidMan is pre-trained on the Open X-Embodiment dataset (OXE) for predicting future visual trajectories in a video denoising diffusion manner, enabling the model to develop a long horizontal awareness of the environment's dynamics. In the second stage, a flexible yet effective layer-wise self-attention adapter is introduced to transform VidMan into an efficient inverse dynamics model that predicts action modulated by the implicit dynamics knowledge via parameter sharing. Our VidMan framework outperforms state-of-the-art baseline model GR-1 on the CALVIN benchmark, achieving a 11.7% relative improvement, and demonstrates over 9% precision gains on the OXE small-scale dataset. These results provide compelling evidence that world models can significantly enhance the precision of robot action prediction. Codes and models will be public.

1 Introduction

VidMan uses a two-stage video-diffusion training strategy to use robot trajectory data for dynamics-aware action prediction. It reports gains on CALVIN and small-scale OXE.

1 Introduction

  • High-quality robotic data are difficult and costly to acquire, motivating ways to use diverse robot trajectories without neglecting visual observation–action relationships.Video generators trained on large-scale online video show potential for understanding real-world physical dynamics.Source: §1 · pp. 1–2
  • VidMan’s two-stage training draws on dual-process theory to improve stability and data utilization.The framework learns different facets of data at distinct stages, acquiring an inverse-dynamics inductive bias that the authors say particularly improves generalization with limited data.Source: §1 · p. 2
  • The first stage pre-trains on OXE to predict future trajectories from past observations and language, while the second uses a layer-wise adapter and shared parameters to infer actions without generating future visuals.The second stage is described as an implicit inverse dynamics model for real-world control.Source: §1 · p. 2
  • VidMan achieves an 11.7% relative improvement over state-of-the-art baselines on CALVIN and over 9% precision gains on the OXE small-scale dataset.The OXE gains are particularly evident when data from the target robot is small.Source: §1 · p. 2

2 Related Work

Related work covers language-guided manipulation and robot pre-training. VidMan is positioned alongside GPT-style video-and-action prediction methods, while using video’s future information to predict action sequences.

2 Related Work

  • Language-guided manipulation methods include LLM-based planners with low-level policies, hierarchical 2D subgoal policies, and 3D policies trained with depth maps and camera extrinsics.Other approaches use object detection with constrained optimization or learn language-conditioned policies from unstructured play data.Source: §2 · p. 3
  • Robot pre-training includes masked-image and contrastive representation learning, as well as language-conditioned policies and GPT-style action-token prediction.GR-1 and RoboFlamingo use GPT-style frameworks to predict video and action; VidMan is described as using video’s future information to predict long action sequences.Source: §2 · p. 3

3 Preliminaries

The preliminaries distinguish forward dynamics from inverse dynamics and describe how image observations can be used to infer state sequences. The state estimate is framed using posterior probability and its likelihood and prior terms.

3 Preliminaries

  • Forward dynamics model the next-state distribution given the current state and action, whereas inverse dynamics model the action distribution given a state transition.The text notes inverse dynamics’ relevance when passive observation data lack explicit actions.Source: §3 · p. 3
  • When direct state access is infeasible, the state sequence can be inferred from image or proprioceptive observations by maximizing its posterior probability given those observations..Source: §3 · p. 3
  • The state-sequence posterior is expressed using the observation likelihood and the prior probability of the sequence.The text identifies P(O | S) as the likelihood and P(S) as the prior.Source: §3 · p. 3

4 Method

VidMan separates learning visual dynamics from predicting actions: it first forecasts future trajectories, then adapts the video model into an inverse dynamics model that uses implicit dynamics knowledge for action prediction.

4 Method

  • VidMan uses a two-stage training strategy that first learns visual dynamics and then adapts the video model for action prediction.The stages share model parameters while separating dynamics learning from embodiment-specific action prediction.Source: §4 · pp. 3–4

4.1 Dynamics-aware Visionary Stage

  • The Dynamics-aware Visionary Stage trains a video diffusion model to predict future frames from historical frames and language instructions.This stage uses Open-Sora to acquire dynamic priors and predict future state transitions.Source: §4.1 · p. 4, §4 · p. 4

4.2 Dynamics-modulated Action Stage

  • The Dynamics-modulated Action Stage uses learnable action tokens and layer-wise adapters on VDT features to form action embeddings for a diffusion-based action head.The adapter reuses VDT parameters without its observation-generation function, and the action head decodes embeddings into low-level control signals.Source: §4.2 · p. 5

5 Experiment

The experiments evaluate VidMan’s two-stage training on CALVIN and OXE, comparing it with policy baselines and testing design choices through ablations. VidMan improves over GR-1 on CALVIN, while the ablations examine the effects of co-training, pretraining data, the adapter, and frame intervals.

5.1.1 2-stage Training Setting

  • The first stage uses Open-Sora initialization and OXE video sequences, while the second stage trains with OXE and CALVIN data and adds wrist-camera observations.The first-stage sequences contain two historical and two future frames sampled at intervals of three; the second-stage action horizons differ between OXE and CALVIN.Source: §5.1.1 · p. 6

5.1.2 Benchmark and Baselines

  • CALVIN evaluation tests zero-shot generalization from environments A, B, and C to D using five-instruction chains, and all models must predict robot pose trajectories.The comparison includes hierarchical 2D, large-scale 2D transformer, and 3D policies; OXE baselines include RT-1-X and Octo, alongside VidMan-GPT.Source: §5.1.2 · pp. 6–7

5.2 Comparison Results

  • VidMan achieves an 11.7% relative improvement over GR-1 in average length on CALVIN and remains competitive with 3D-based policies.The comparison uses CALVIN’s language-labeled data for VidMan training; the table reports zero-shot long-horizon evaluation.Source: Table 1 · p. 7, §5.2 · p. 7
  • VidMan outperforms Octo-base by 5.6% on Bridge, 2.6% on Taco Play, 9.9% on Cable Routing, and 9.0% on Autolab UR5.Figure 3 reports offline performance for models trained on OXE and validated across four datasets.Source: Figure 3 · p. 8

5.3 Extra Ablation Studies

  • Adding frame-generation loss to action supervision in stage two severely degrades average score compared with action-only training.The authors frame action-only training as learning an inverse dynamics model rather than the joint distribution of actions and future frames.Source: §5.3 · p. 8
  • OXE pretraining increases CALVIN average task length by 0.53 versus no pretraining, whereas Ego4D pretraining decreases it by 0.13.The reported comparison uses robot-specific OXE pretraining and general-video Ego4D pretraining.Source: §5.3 · p. 9
  • Removing the layer-wise adapter significantly reduces performance, while freezing Open-Sora blocks converges faster and training them yields better accuracy.The efficiency comparison contrasts frozen and unfrozen Open-Sora blocks.Source: §5.3 · p. 9, Figure 4 · p. 9
  • Increasing the frame interval from 1 to 3 improves Bridge and CALVIN performance, but gains saturate when the interval increases to 4.The authors speculate that frames too far in the future may provide less useful guidance for immediate local action prediction.Source: §5.3 · p. 9, Table 3 · p. 9

6 Conclusion

VidMan combines video-diffusion pretraining on Open X-Embodiment with a dynamics-modulated action stage for robot imitation learning. The paper reports high precision and computational efficiency, and proposes expanding the model’s perceptual inputs in future work.

6 Conclusion

  • VidMan combines a Dynamics-aware Visionary Stage with a Dynamics-modulated Action Stage to integrate environment-dynamics knowledge into action prediction.The first stage pre-trains on Open X-Embodiment; the second integrates that knowledge into action prediction.Source: §6 · p. 9
  • The two-stage framework is reported to achieve high precision and computational efficiency, with improved performance on CALVIN and OXE.Source: §6 · p. 9
  • Future work will expand VidMan to perceive more dimensions of information.Source: §6 · p. 9

A Appendix

The appendix outlines material on negative impacts and limitations, network and training details, OXE, additional experiments, and prediction visualizations.

A Appendix

  • The appendix includes a section on negative impacts and limitations.Source: A Appendix · p. 14
  • It covers network and training details and provides an OXE description and usage guide.Source: A Appendix · p. 14
  • Additional experimental results are included.Source: A Appendix · p. 14
  • The appendix includes visualizations of video prediction and action prediction.Source: A Appendix · p. 14

A.1 Negative Impacts and Limitations

The appendix states that the benchmarks are public and transparent, and identifies limits in VidMan’s spatial, language, and fine-grained visual perception.

A.1 Negative Impacts and Limitations

  • The authors state that public, transparent benchmarks mean the method has no ethical risk from dataset usage or privacy violations.Source: §A.1 · p. 14
  • VidMan uses only 2D vision and lacks 3D perception, affecting performance on tasks requiring accurate 3D spatial understanding.Source: §A.1 · p. 14
  • The model’s comprehension of complex human instructions is limited, and it lacks the sophistication of a CLIP language encoder.Source: §A.1 · p. 14
  • VidMan processes the entire image as one input for action prediction, which the authors describe as a limit on fine-grained perception.Source: §A.1 · p. 14

A.2 Network and Training Details

The appendix specifies VidMan’s pretrained encoders and diffusion model, training configurations, and the layer-wise adapter used to refine visual-dynamics information into the action query.

A.2 Network and Training Details

  • The video autoencoder is Stabilityai’s VideoAutoencoderKL, the text encoder is CLIP’s text encoder, and the diffusion model is STDiT-XL/2.Source: §A.2 · p. 14
  • Stage-1 OXE video prediction used 16 V100 GPUs and batch size 384 for 42 hours; stage-2 OXE training used 16 GPUs and batch size 1440 for 32 hours.For stage 2, gradient accumulation updated parameters every five steps.Source: §A.2 · p. 14
  • CALVIN stage-2 training used eight V100 GPUs with total batch size 400, and training used AdamW with inverse square root learning-rate decay and weight decay 0.1.Source: §A.2 · p. 14
  • The layer-wise adapter combines self-attention and a feed-forward network with gating to refine video-transformer information into the action query.Source: §A.2 · p. 15, Figure 5 · p. 15

A.3 More Details of Open X-Embodiment Datasets

VidMan’s OXE training data was curated for diversity and task relevance, then balanced and standardized across datasets.

A.3 More Details of Open X-Embodiment Datasets

  • The curated OXE data spans robot forms, environments, tasks, sensor configurations, and labeling styles, including datasets with and without language instructions.Source: §A.3 · p. 15
  • The authors selected datasets for diversity and task relevance, excluding those without image streams or delta end-effector control, and omitting repetitive, low-resolution, or overly specialized data.The filtering approach was inspired by Octo [7].Source: §A.3 · p. 15
  • More diverse datasets received higher sampling weights, while some large datasets were reweighted to maintain a balanced mix.Source: §A.3 · p. 15
  • Camera-channel gaps were zero-padded, and gripper commands were standardized so +1 meant open and 0 meant closed.Source: §A.3 · p. 15

A.4 Additional Experimental Results

The additional experiments examine stage-two inputs, frame-history and future length, OXE offline performance, and RLBench evaluation; visualizations show video and action predictions. The reported results favor VidMan over Octo-base on four OXE datasets and show that shortening history hurts more than shortening the future horizon.

A.4 Additional Experimental Results

  • Using pure noise in stage two allowed some evaluation performance; the authors suggest it may align with denoising and distill the pretrained model’s future perception into action prediction.The input variants were zero embeddings, pure noise, and unnoised images; experiments used offline Bridge validation with the VDT block frozen.Source: §A.4 · p. 15, Table 5 · p. 16
  • Shortening historical frames significantly decreased performance on Bridge and CALVIN, while shortening future frames also reduced performance but less substantially.The experiments compared history/future settings n=1, m=1; n=2, m=1; and n=2, m=2, with a frame interval of 3.Source: §A.4 · p. 16, Table 6 · p. 16
  • RLBench evaluation used 18 tasks, 100 demonstrations per agent, and 25 evaluation episodes per task, comparing PerAct, RVT, and Act3D.Tasks included multiple variations and tests with novel object poses, sampled goals, and sampled scenes; the reported measure was average success rate with five-seed variance.Source: §A.4 · pp. 16–17, Table 8 · p. 17
  • VidMan outperformed Octo-base by 5.6% on Bridge, 2.6% on Taco Play, 9.9% on Cable Routing, and 9.0% on Autolab UR5.These are offline validation results for models trained on OXE.Source: Table 7 · p. 16

A.5 Visualization

  • VidMan qualitatively predicts future frames, though some details such as occluded objects are missing.Figure 6 distinguishes ground-truth images in yellow boxes from predictions in blue boxes, with the language instruction below.Source: §A.5 · p. 17, Figure 6 · p. 18
  • On OXE offline evaluation, VidMan’s predicted actions fit ground-truth actions across varied scenarios.Figure 7 pairs subsampled episode frames with true and predicted 7D pose results over time.Source: §A.5 · p. 17, Figure 7 · p. 19

Related papers

Loading 2411.09153v1…