PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations

Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, Jianyu Chen

arXiv:2412.14803v2cs.CVcs.RO

TL;DR

Generalist robot policies need visual representations that capture dynamics, not only static appearance. VPP fine-tunes video diffusion models for manipulation and conditions an implicit inverse dynamics model on their predictive representations, achieving improvements across simulated and real-world tasks. The paper reports a 41.5% improvement on Calvin ABC→D over the previous SOTA and a 31.6% real-world success-rate improvement over the strongest baseline.

  • Problem

    Previous vision encoders often capture static information from one or two images while underusing dynamic information in sequential video data.

  • Method

    VPP fine-tunes a general video diffusion model on human and robot manipulation data, then learns an implicit inverse dynamics model conditioned on predictive visual representations.

  • Results

    41.5% improvement on the Calvin ABC→D benchmark over the previous SOTA and 31.6% higher success rate on real-world dexterous manipulation than the strongest baseline.

  • Takeaways & Limitations

    Predictive representations inside video diffusion models yield consistent improvements across simulated and real-world robotic tasks.

Abstract

from arXiv · show

Visual representations play a crucial role in developing generalist robotic policies. Previous vision encoders, typically pre-trained with single-image reconstruction or two-image contrastive learning, tend to capture static information, often neglecting the dynamic aspects vital for embodied tasks. Recently, video diffusion models (VDMs) demonstrate the ability to predict future frames and showcase a strong understanding of physical world. We hypothesize that VDMs inherently produce visual representations that encompass both current static information and predicted future dynamics, thereby providing valuable guidance for robot action learning. Based on this hypothesis, we propose the Video Prediction Policy (VPP), which learns implicit inverse dynamics model conditioned on predicted future representations inside VDMs. To predict more precise future, we fine-tune pre-trained video foundation model on robot datasets along with internet human manipulation data. In experiments, VPP achieves a 18.6\% relative improvement on the Calvin ABC-D generalization benchmark compared to the previous state-of-the-art, and demonstrates a 31.6\% increase in success rates for complex real-world dexterous manipulation tasks. Project page at https://video-prediction-policy.github.io

1. Introduction

Existing vision encoders often miss dynamic information because they process only one or two images, whereas video diffusion models provide representations of current and predicted future frames. VPP uses these predictive representations to learn robot actions and reports gains across simulated and real-world settings.

  • Motivation: Single-image and two-image visual pretraining may not fully exploit the dynamic information in sequential video data.These methods typically operate on only one or two sampled images.
  • Predictive representations: Video diffusion representations explicitly encode one current step and multiple predicted future steps, unlike previous vision encoders.The representation has dimensions (T, H, W), corresponding to one current frame and T −1 predicted future frames.
  • Approach: VPP implicitly learns inverse dynamics by tracking robot movements within predictive representations and aligning robot actions with visual space using few demonstrations.Accurate future prediction is intended to transfer video-model generalization to robotic policies.
  • Approach: VPP fine-tunes a general video diffusion model on internet human and robot manipulation data, then learns an inverse dynamics model conditioned on its predictive representations.Directly using internal representations avoids multiple denoising steps and supports high-frequency closed-loop operation.
  • Results: 41.5% improvement in the Calvin ABC→D benchmark was achieved over the previous SOTA method, while real-world dexterous manipulation success rates improved by 31.6% over the strongest baseline.VPP also outperformed baselines across two simulated and two real-world settings.

2. Related Works

Robotics has used self-supervised representations and future prediction, but video diffusion representations remain comparatively unexplored. This work positions their predictive properties as useful for sequential embodied control.

  • Visual representation learning: Self-supervised contrastive, distillation-based, and reconstructive methods produce vision representations used for embodied AI tasks.Prior work reports both high-level semantic and low-level visual information in these representations.
  • Future prediction for control: Future-prediction methods such as SuSIE and UniPi condition control on generated future frames or learn inverse dynamics between two generated frames.These approaches use future visual predictions to support policy learning.
  • Diffusion representations: Image diffusion representations have shown utility for discrimination, semantic segmentation, and embodied tasks.Their usefulness has been demonstrated across several visual and embodied applications.
  • Video diffusion representations: Video diffusion representations have not been extensively explored, and the paper identifies a predictive property that may suit sequential embodied control.The claimed distinction is their ability to represent future evolution within video-model features.

3. Preliminaries

The preliminaries describe diffusion models as iterative noising and denoising processes for video generation, with text guidance controlling generation from an initial frame and language prompt. They also introduce diffusion policy for multimodal action-sequence learning.

  • Video diffusion models: The forward diffusion process progressively adds Gaussian noise to a real video sequence.The noisy state and noise amplitude are represented by x_t and α_t.
  • Video diffusion models: The reverse process starts from a highly noisy sample and estimates progressively less noisy video states.The reverse transition is expressed as a variational approximation of q(x_t−1|x_t).
  • Video diffusion models: In text-guided video generation, a learned noise estimator controls generation using the initial frame and language prompt.The estimator approximates the score function during denoising.
  • Diffusion policy: Diffusion policy denoises action sequences conditioned on observations and instructions.It can represent complex multimodal action distributions and stabilize training.

4. Video Prediction Policy

VPP fine-tunes a video foundation model for manipulation, then uses its predictive representations in a policy that generates robot actions. The design uses a single forward step, feature aggregation, and a diffusion action head to retain high control frequency while modeling future dynamics.

  • 4. Video Prediction Policy: VPP uses two-stage learning: manipulation-focused TVP fine-tuning followed by inverse-dynamics learning conditioned on TVP predictive representations.The first stage uses internet human, robot, and downstream manipulation data; the second trains the action policy.
  • 4.1. Text-guided Video Prediction Model: The general video model is specialized because pretrained video generators are not fully controllable for robot manipulation.Fine-tuning aims to improve prediction accuracy in the manipulation domain.
  • 4.2. Action Learning Conditioned on Predictive Visual Representation: VPP uses the video diffusion model as a vision encoder through one forward step rather than denoising an entire video sequence.This avoids the time cost and potential open-loop issues of full-sequence denoising while providing a rough future-state trajectory.
  • 4.2. Action Learning Conditioned on Predictive Visual Representation: Features from multiple up-sampling layers are interpolated to a common spatial resolution and stacked along the channel dimension into a predictive visual representation.The representation can be predicted independently for multiple camera views.
  • 4.2. Action Learning Conditioned on Predictive Visual Representation: Video Former compresses high-dimensional spatial, temporal, and multi-view features into a fixed number of learnable tokens using spatial-temporal attention and feed-forward layers.The resulting tokens condition a diffusion policy action head through cross-attention.
  • 4.3. Experiments: VPP improves average task completion length on the zero-shot Calvin ABC→D benchmark and reports multi-task success-rate evaluation on Metaworld.Table 1 evaluates five chained tasks sequentially; Table 2 reports Metaworld multi-task success rate.

5. Experiments

The experiments evaluate VPP across simulated and real-world manipulation settings, testing predictive representations, training components, architecture, and generalization. VPP outperforms baselines, while ablations show that predictive features, video pre-training, the Video Former, and feature aggregation contribute to performance.

  • Quantitative Results: VPP improves the previous CALVIN ABC→D state of the art from 3.35 to 4.33 average task completion length.With only 10% of annotated Calvin ABC data, VPP reaches 3.25, exceeding related methods trained with full data.
  • Quantitative Results: VPP outperforms the strongest GR-1 baseline by 10.8% in average success rate on the 50-task MetaWorld benchmark.
  • Ablation Study: Predictive visual representations outperform replacement representations, whose substitution causes a clear performance decline.
  • Ablation Study: Removing internet manipulation data decreases performance from 4.33 to 3.97, while also removing video pre-training causes a substantial further drop.
  • Ablation Study: Removing the Video Former lowers the VPP score from 4.33 to 3.86 and nearly triples inference time.
  • Ablation Study: Replacing aggregated multi-layer features with final-layer features reduces Calvin average task completion length from 4.33 to 3.60.
  • Real World Experiments: VPP outperforms GR-1, Susie, and diffusion policy across seen, unseen, and tool-use real-world tasks.The evaluation includes 200+ Panda rollouts and 500+ dexterous-hand rollouts.
  • Real World Experiments: On unseen tasks, predicted future states are reasonable and execution trajectories closely align with them, supporting generalization through predictive representations and implicit inverse dynamics.

6. Conclusion

VPP is presented as a generalist robot policy that conditions an implicit inverse dynamics model on predictive representations inside video diffusion models. The authors report consistent improvements across simulated and real-world tasks and emphasize video generation models’ potential for embodied tasks.

  • VPP learns an implicit inverse dynamics model conditioned on predictive representations inside video diffusion models.
  • VPP yields consistent improvements across simulated and real-world tasks.
  • The authors highlight the potential of video generation models for building physical intelligence and supporting embodied tasks.

A. Real-world experiments

The real-world evaluation covers Panda-arm and dexterous-hand manipulation, with demonstrations and task-level success rates reported for seen, unseen, and tool-use settings.

  • Panda Arm: Panda demonstrations comprise 2k trajectories across 30+ tasks in six categories, including picking, placing, pressing, routing, opening, and closing.
  • Evaluation: Success rates are reported separately for seen and unseen tasks across the Panda-arm and dexterous-hand evaluations.
  • Dexterous Hand: Dexterous-hand demonstrations comprise 2.5k trajectories across 100+ tasks in ten manipulation categories.
  • Evaluation: Tool-use tasks are evaluated separately in the dexterous-hand setting, with each tool-use task tested 10 times.
  • Training Data: The training data use different sampling ratios to balance datasets with varying quality and scale.

B.2. Quantitative result on Prediction Quality

The manipulation-focused TVP models successfully predict future frames across diverse validation tasks and outperform an earlier TVP model on bridge-dataset prediction quality. The authors attribute this advantage to using the pre-trained SVD video foundation model.

  • TVP models successfully predict future frames across diverse manipulation tasks on validation datasets.
  • The models are quantitatively evaluated with the FVD metric on bridge datasets using settings from Seer.
  • The proposed model outperforms the previous TVP model on bridge-dataset prediction quality.The authors attribute the improvement to leveraging the pre-trained SVD video foundation model.

B.3. More Visualization of Complete Prediction Results

The fine-tuned manipulation TVP generates complete future predictions for human and robotic manipulation validation data, while predictive representations capture physical evolution despite imprecise visual details.

  • Complete Prediction Results: The manipulation TVP predicts trajectories from human and robotic manipulation validation datasets using 30 denoising steps and classifier-free guidance of 7.5.Human predictions are shown in Figure 8 and robotic predictions in Figure 10.
  • Predictive Representations: Intermediate predictive representations reflect object and robot-arm movements even when one-step forward videos have imprecise textures and details.These representations provide information about physical evolution rather than precise visual appearance.
  • Implementation Details: The VPP architecture and hyperparameter settings are provided for four evaluation environments in Table 13.The TVP transformer block follows Stable Video Diffusion settings, while other Diffusion Transformer hyperparameters follow prior work.

C.2. More ablation

Additional CALVIN ABC→D ablations examine temporal attention, denoising steps, viewpoints, feature layers, and diffusion time-steps. The reported findings favor temporal attention and one-step representations, while a single view achieves strong Task 5 performance.

  • Ablation Setup: The ablation studies are summarized in Table 12 alongside the CALVIN ABC→D experiments.The passage identifies the evaluation setting but does not provide the individual table values.
  • Temporal Attention: Removing the Temporal-attn module reduces the Video Former’s temporal comprehension capabilities relative to the full VPP configuration.The comparison is reported in Table 12 under the CALVIN ABC→D setting.
  • Denoising Steps: A 2-step denoising process does not yield superior performance and nearly doubles inference time while halving control frequency.The main experiments therefore use a one-step direct encoder.
  • Single-View Ablation: Task 5 reaches a success rate of 3.58 using only a single static observation viewpoint in the Calvin ABC→D task.This result surpasses the reported success rate of 3D Diffuser Actor, which uses two viewpoints and depth images.
  • Feature and Time-Step Ablations: Additional ablations evaluate different feature layers and diffusion time-steps using average task completion length.The feature-layer results are listed in Table 10 and the diffusion-time-step results in Table 11.

C.3. Baseline Implementations

The baseline implementations use official repositories for several policies and reproduce SuSIE with an InstructPix2Pix future-frame predictor plus image-goal Diffusion Policy. The section also includes VPP hyperparameters and visual prediction comparisons.

  • Baseline Implementations: RT-1, GR-1, and Diffusion Policy are implemented using their official repositories.These methods serve as baseline algorithms in the reported comparisons.
  • SuSIE Comparisons: For SuSIE comparisons, InstructPix2Pix predicts future frames and an image-goal Diffusion Policy generates the state sequence.This setup is used in both Metaworld and real-world manipulation scenarios.
  • Hyperparameters: Table 13 reports the hyperparameters used in the Video Prediction Policy.The supplied passage identifies the table but does not list its individual settings.
  • Prediction Visualizations: Figure 8 visualizes video prediction results on Internet human manipulation validation datasets using green ground-truth frames and red predicted futures.The visualizations use 30 denoising steps.
  • Predictive Representations: Figure 9 visualizes predictive representations alongside ground-truth and predicted future states.Green frames represent ground truth, red frames predicted futures, and blue frames predictive representations.
  • Prediction Visualizations: Figure 10 visualizes robotic-dataset predictions, separating internet robotic samples from self-collected samples.Green frames denote ground truth and red frames denote predicted futures, with 30 denoising steps.
Loading 2412.14803v2…