PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Prediction Models as Rewards for Reinforcement Learning

Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, Pieter Abbeel

arXiv:2305.14343v2cs.LGcs.AIcs.CV

TL;DR

Reward design limits reinforcement learning, motivating methods that extract behavioral preferences from scalable unlabeled videos. VIPER trains an autoregressive video model on expert videos and uses its likelihoods as rewards, achieving expert-level control across 28 tasks and generalizing across embodiments. The method provides a scalable approach to reward specification from videos, though it depends on in-domain expert data and can prefer low-uncertainty states in stochastic settings.

  • Problem

    Manually designed rewards are laborious and can produce undesirable outcomes, motivating scalable reward learning from unlabeled videos.

    Source: §1 · p. 1

  • Method

    VIPER trains a video prediction model on expert videos and uses trajectory log-likelihoods as rewards for reinforcement learning.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    VIPER achieves expert-level control without task rewards across 15 DMC, 6 RLBench, and 7 Atari tasks, and generalizes to new environments for cross-embodiment tabletop manipulation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Video prediction likelihoods provide a scalable reward specification approach from unlabeled videos across multiple reinforcement-learning domains.

    Source: §1 · p. 2, §5 · p. 9

    VIPER relies on in-domain expert data, and rewards trained on stochastic data may favor low-uncertainty states and produce sub-optimal behavior.

    Source: §5 · p. 9

Abstract

from arXiv · show

Specifying reward signals that allow agents to learn complex behaviors is a long-standing challenge in reinforcement learning. A promising approach is to extract preferences for behaviors from unlabeled videos, which are widely available on the internet. We present Video Prediction Rewards (VIPER), an algorithm that leverages pretrained video prediction models as action-free reward signals for reinforcement learning. Specifically, we first train an autoregressive transformer on expert videos and then use the video prediction likelihoods as reward signals for a reinforcement learning agent. VIPER enables expert-level control without programmatic task rewards across a wide range of DMC, Atari, and RLBench tasks. Moreover, generalization of the video prediction model allows us to derive rewards for an out-of-distribution environment where no expert data is available, enabling cross-embodiment generalization for tabletop manipulation. We see our work as starting point for scalable reward specification from unlabeled videos that will benefit from the rapid advances in generative modeling. Source code and datasets are available on the project website: https://escontrela.me/viper

1 Introduction

VIPER addresses reward-design bottlenecks by using likelihoods from a video prediction model trained on expert videos as reinforcement-learning rewards. It achieves expert-level control without task rewards across 28 tasks and supports cross-environment generalization.

1 Introduction

  • Unlike observation-level video rewards, VIPER’s trajectory likelihoods capture temporally meaningful behavior.Prior methods reward similarity to the current observation and can suffer mode collapse under adversarial training.Source: §1 · pp. 1–2
  • VIPER uses a video prediction model trained on expert videos, then rewards agents by the log-likelihood of their trajectories.This directly encourages trajectories matching the model’s distribution and captures temporal consistency.Source: §1 · p. 2
  • Evaluating video-model likelihoods is faster than rolling out the video model, enabling faster training and more environment interactions.The paper also discusses implementation details intended to improve VIPER’s robustness.Source: §1 · p. 2
  • VIPER achieves expert-level control without task rewards across 15 DMC, 6 RLBench, and 7 Atari tasks.The evaluation uses raw pixels without ground-truth rewards or expert actions.Source: §1 · p. 2, Figure 2 · p. 2
  • VIPER generalizes to environments without training data, enabling cross-embodiment generalization for tabletop manipulation.The result extends reward specification beyond environments represented in the expert videos.Source: §1 · p. 2

2 Related Work

Prior learning-from-observation methods either infer actions for videos or learn rewards from expert behavior. VIPER instead directly models expert video distributions to provide stable, temporally informed rewards without action annotations.

Related work

  • Learning from observations seeks policies from expert videos without ground-truth rewards, expert actions, or interaction with the expert.The field includes behavioral cloning with predicted actions and reinforcement learning with video-derived rewards.Source: §2 · p. 2

Labelling videos with predicted actions

  • Inverse-dynamics approaches label video transitions with predicted actions, but performance depends on action-label quality and training-data quantity and diversity.The inferred action distribution can be learned from environment interactions or curated action-labelled data.Source: §2 · p. 2

Reinforcement learning with videos

  • Video-derived reward methods estimate task progress or expert-agent trajectory divergence, while adversarial discriminators can suffer mode collapse and require stabilization techniques.Spurious associations between task-irrelevant features and labels contribute to discriminator failure.Source: §2 · p. 3

Using video models as policies

  • UniPi generates future video trajectories and then infers low-level controls, whereas VIPER uses video prediction likelihoods to guide online policy learning.VIPER avoids slow video generations for planning.Source: §2 · p. 3

3 Video Prediction Rewards

VIPER uses pretrained video prediction likelihoods as action-free rewards, training agents to match expert trajectory distributions without task rewards or action annotations. The method combines autoregressive video modeling, likelihood-based rewards, entropy-driven exploration, and curated expert videos across multiple environments.

3 Video Prediction Rewards

  • VIPER trains agents with video-model likelihoods as rewards, requiring only videos of desired behaviors and supporting any reinforcement-learning algorithm.The reward is implemented as part of the environment and does not require ground-truth rewards or action annotations.Source: §3 · p. 3

3.1 Video Modeling

  • The video model uses an autoregressive transformer over VQ-GAN frame codes, approximating frame likelihoods with conditional latent-code likelihoods.This approximation is used because exact pixel-level conditional likelihoods require marginalizing over all code combinations.Source: §3.1 · pp. 3–4

3.2 Reward Formulation

  • VIPER rewards each observed transition by its conditional video likelihood, encouraging trajectories that are probable under the expert video distribution.Limited-context models compute sequence likelihoods using sliding windows over subsequences of length k.Source: §3.1 · p. 3, §3.2 · p. 4
  • Because maximizing likelihood alone can favor a single mode or cause early convergence, VIPER adds an entropy-based exploration term to match a diverse trajectory distribution.The exploration coefficient β controls the amount of exploration, and the implementation uses Plan2Explore or RND.Source: §3.2 · pp. 4–5

3.3 Data Curation

  • The evaluation curates expert videos from task-oracle agents or motion planners and studies generalization across DMC, Atari, and RLBench settings.The authors use pretrained models across varied environments and identify fine-tuning larger text-conditioned video models as a future route to broader generalization.Source: §3 · p. 3, §3.1 · p. 3, §3.3 · p. 5

4 Experiments

VIPER is evaluated across DMC, Atari, and RLBench using video-model likelihoods as rewards, with experiments covering training details, task performance, cross-embodiment generalization, and design ablations.

4 Experiments

  • Across 28 tasks, the evaluation uses 15 DMC, 7 Atari, and 6 RLBench tasks with pixel observations and no task rewards for the evaluated agents.The study compares VIPER with single-task and multi-task AMP using DrQ and DreamerV3.Source: §4 · p. 5, §4.2 · p. 6

4.2 Video Model Likelihoods as Rewards for Reinforcement Learning

  • VIPER achieves near expert-level DMC performance from pixels using video-prediction rewards alone and outperforms both AMP variants.It slightly underperforms a state-based Task Oracle with dense task rewards.Source: §4.2 · p. 6
  • VIPER approaches sparse-reward Atari Task Oracle performance and outperforms AMP, while scoreboard masking improves downstream Pong training.Without masking, out-of-distribution scores can produce noisier rewards and more erratic behavior.Source: §4.2 · pp. 6–7
  • VIPER outperforms the Task Oracle on RLBench, where dense rewards extracted from expert videos help solve tasks with sparse rewards after long action sequences.Training and reward likelihoods use observations subsampled by a factor of 4 to avoid high likelihoods for stationary trajectories.Source: §4.2 · p. 7

4.3 Generalization of Video Prediction Rewards

  • A task-conditioned autoregressive video model generalizes to an unseen Franka Panda saucepan task and enables VIPER to learn it without demonstrations for that arm-task combination.The model assigns higher likelihood to reference videos than random trajectories in the out-of-distribution setting.Source: §4.3 · p. 8

4.4 Ablation Studies

  • Equal exploration weighting and longer context improve downstream performance, while MaskGIT underperforms VideoGPT and BYOL is limited by deterministic multi-step prediction.MaskGIT also requires approximately 8× slower reward computation and 2.5× slower overall RL training.Source: §4.4 · pp. 8–9, Figure 9 · p. 8

5 Conclusion

VIPER uses pretrained video models to derive rewards from expert videos, enabling broad task learning without task rewards while exposing important data and modeling limitations.

5 Conclusion

  • VIPER learns rewards from video models and succeeds across 3 benchmarks and 28 tasks, including DMC, Atari, and RLBench.The method uses conditional frame probabilities and entropy maximization to match diverse trajectories.Source: §5 · p. 9
  • VIPER is trained on in-domain expert data, which is not readily available in real-world settings.Source: §5 · p. 9
  • Stochastic data can make video-prediction rewards favor low-uncertainty states and produce sub-optimal behavior.This issue may arise from stochastic environments or noisy demonstrations.Source: §5 · p. 9
  • VQCode size and context length strongly affect reward quality, with small codes omitting important components and short contexts causing myopic behavior.Source: §5 · p. 9
  • Larger video models, fine-tuning, human preferences, text-to-video models, and video diffusion models are proposed as future directions.Source: §5 · p. 9

A.1 Bootstrap Your Own Latent Details

The BYOL-based implementation predicts future latent observations without action inputs and converts prediction uncertainty into a reward for staying near demonstrated trajectories.

A.1 Bootstrap Your Own Latent Details

  • The BYOL-Explore architecture uses online predictions trained against targets from an exponential-moving-average observation encoder.Observations are encoded, processed through recurrent networks, and passed to a predictor head.Source: §A.1 · p. 13
  • Action inputs are omitted from the recurrent model so the reward mechanism can be trained solely from videos.This differs from the original BYOL-Explore architecture, which conditions on actions.Source: §A.1 · p. 13
  • Because the recurrent architecture is deterministic and action-free, it may perform poorly in multimodal or stochastic environments.A probabilistic recurrent state-space model is proposed as a more principled future alternative.Source: §A.1 · p. 13
  • The loss uses the valid open-loop horizon K(t) = min(K, T −1 −t) and a stop-gradient operator.Source: §A.1 · p. 13
  • The transition uncertainty is computed by summing the corresponding prediction losses, and rewards are defined as their negative.This keeps the agent close to the trajectory distribution found in the original dataset.Source: §A.1 · p. 13

A.2 Video Model Hyperparameters

The appendix provides hyperparameters and training details for the VQ-GAN, VideoGPT, and MaskGit video models used in the experiments.

A.2 Video Model Hyperparameters

  • Table 1 lists hyperparameters and training details for all VQ-GAN models.Source: Table 1 · p. 14
  • The appendix covers both VQ-GAN and autoregressive or masked video-model configurations.Source: Table 1 · p. 14, Table 2 · p. 14
  • Table 2 lists hyperparameters and training details for all VideoGPT and MaskGit models.Source: Table 2 · p. 14

A.3 Reinforcement Learning Hyperparameters

The appendix reports training hyperparameters and implementation details for the DreamerV3 and DrQ reinforcement-learning baselines.

A.3 Reinforcement Learning Hyperparameters

A.4 Environment Training Curves

The section presents results across RLBench, Atari, and DeepMind Control Suite benchmark tasks, with results aggregated across three seeds.

  • Results span 6 Reinforcement Learning Benchmark tasks, with means and standard deviations computed across 3 seeds.Source: Figure 10 · p. 17
  • Results span 7 Atari tasks using Human-Normalized Mean scores computed across 3 seeds.Source: Figure 11 · p. 17
  • Results span 15 DeepMind Control Suite tasks, with means and standard deviations computed across 3 seeds; AMP runs stopped early because of poor performance.Source: Figure 12 · p. 18

A.5 Training Environments

The paper trains autoregressive video models across robotic, DeepMind Control, and Atari environments, then visualizes their uncertainty through conditional VQCode likelihoods. These visualizations associate higher uncertainty with randomized initial configurations and trajectories exhibiting poor behavior.

A.5 Training Environments

  • A single autoregressive video model is trained on 30 Franka Panda tasks and 23 ReThink Robotics Sawyer tasks using 16 × 16 VQCodes, context length 4, and frame skip 4.Source: Figure 13 · p. 19
  • A single task-conditioned autoregressive video model is trained on 15 DeepMind Control tasks using 8 × 8 VQCodes and context length 16.Source: Figure 14 · p. 19
  • A single autoregressive video model is trained on 7 Atari tasks using 16 × 16 VQCodes and context length 16.Source: Figure 15 · p. 20

A.6 Visualizing Video Prediction Uncertainty

  • Conditional likelihoods over VQCodes are upsampled to visualize which local image regions the video model predicts with greater uncertainty.Source: §A.6 · p. 21
  • Brighter visualized values indicate higher likelihoods, while random trajectories receive lower likelihoods around the ball because expert-only training cannot accurately predict sub-optimal play.Source: Figure 16 · p. 21
  • For a random cartpole-balance trajectory, the visualization reports brighter values as higher likelihoods without specifying a further comparison.Source: Figure 17 · p. 21
  • Randomized saucepan positions create initial uncertainty, while poor arm trajectories create high uncertainty because such behavior is absent from training data.Source: Figure 18 · p. 22

Related papers

Loading 2305.14343v2…