VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training

Yecheng Jason Ma; Shagun Sodhani; Dinesh Jayaraman; Osbert Bastani; Vikash Kumar; Amy Zhang

Source-linked AI summary

VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training

Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, Amy Zhang

arXiv:2210.00030v2Sep 30, 2022cs.ROcs.AIcs.CVcs.LG

TL;DR

Robot manipulation needs scalable visual representations and general reward specification despite scarce task-specific data. VIP learns an action-free value-based representation from unlabeled human videos, and its frozen embeddings provide effective rewards across unseen robot tasks, including few-shot offline RL with 20 trajectories.

Problem
Reward specification remains an open challenge for pre-training robot control because real-world tasks lack privileged state information and well-shaped rewards.
Method
VIP treats human-video representation learning as offline goal-conditioned reinforcement learning and derives a self-supervised dual value objective without action labels.
Results
VIP’s frozen representation significantly outperforms prior pre-trained representations across simulated and real-robot reward-based control settings.
Takeaways & Limitations
VIP provides dense visual reward for unseen goal-image tasks and enables simple few-shot offline RL on real-robot tasks with as few as 20 trajectories.
Takeaways & Limitations
Downstream policies use frozen out-of-domain representations and act on embedded observations for single-goal tasks.

Abstract

from arXiv · show

Reward and representation learning are two long-standing challenges for learning an expanding set of robot manipulation skills from sensory observations. Given the inherent cost and scarcity of in-domain, task-specific robot data, learning from large, diverse, offline human videos has emerged as a promising path towards acquiring a generally useful visual representation for control; however, how these human videos can be used for general-purpose reward learning remains an open question. We introduce $\textbf{V}$alue-$\textbf{I}$mplicit $\textbf{P}$re-training (VIP), a self-supervised pre-trained visual representation capable of generating dense and smooth reward functions for unseen robotic tasks. VIP casts representation learning from human videos as an offline goal-conditioned reinforcement learning problem and derives a self-supervised dual goal-conditioned value-function objective that does not depend on actions, enabling pre-training on unlabeled human videos. Theoretically, VIP can be understood as a novel implicit time contrastive objective that generates a temporally smooth embedding, enabling the value function to be implicitly defined via the embedding distance, which can then be used to construct the reward for any goal-image specified downstream task. Trained on large-scale Ego4D human videos and without any fine-tuning on in-domain, task-specific data, VIP's frozen representation can provide dense visual reward for an extensive set of simulated and $\textbf{real-robot}$ tasks, enabling diverse reward-based visual control methods and significantly outperforming all prior pre-trained representations. Notably, VIP can enable simple, $\textbf{few-shot}$ offline RL on a suite of real-world robot tasks with as few as 20 trajectories.

1 INTRODUCTION

VIP addresses reward specification for unseen real-world manipulation tasks by learning from unlabeled human videos. Its frozen representation provides visual reward and representation without task-specific fine-tuning, outperforming prior representations across simulated and real-robot settings.

Motivation: Real-world robot learning lacks privileged state information and well-shaped rewards, making reward specification a key pre-training challenge.Prior approaches either assume dense simulated rewards or use demonstrations, requiring substantial engineering for each new task.
Motivation: Goal images offer a simple, general task specification, but existing pre-trained representations do not reliably turn embedding distance into effective reward.This motivates learning representations specifically for reward-based visual control rather than transferring representations optimized for other objectives.
Approach: VIP treats representation learning from diverse human videos as offline goal-conditioned reinforcement learning and derives an action-free dual value-function objective.Fenchel duality makes self-supervised value learning possible even though human videos lack robot action labels.
Approach: VIP’s implicit time-contrastive interpretation attracts initial and goal frames while recursively repelling intermediate frames, producing long-range temporal structure and local smoothness.These properties make embedding distances suitable for dense visual reward construction.
Results: ≈30% of tasks are solved with a standard trajectory optimizer and no task-specific fine-tuning, increasing to ≈45% with more optimization budget.Other representations can worsen with additional optimization because their reward landscapes are vulnerable to reward hacking.
Results: VIP enables few-shot offline reinforcement learning on real-robot manipulation tasks using as few as 20 trajectories.The perceptual reward is learned entirely from out-of-domain human videos.

2 RELATED WORK

Prior work has used out-of-domain visual pre-training for control, but VIP differs by combining human-video pre-training with fully self-supervised value learning. Related approaches span supervised and unsupervised image objectives, masked autoencoding, and time-contrastive learning.

Scope: The related literature also includes perceptual reward learning from human videos and goal-conditioned reinforcement learning as representation learning.These topics are reviewed in the paper, with two areas covered in the appendix because of space constraints.
Out-of-Domain Representation Pre-Training for Control: Out-of-domain visual pre-training has been applied to simulated dexterous manipulation, navigation, and online reinforcement learning.Reported backbones include ImageNet-trained ResNet, unsupervised MoCo ResNet, and video-trained masked autoencoders.
Out-of-Domain Representation Pre-Training for Control: R3M is the closest related approach because it also uses Ego4D and temporal information, whereas VIP is fully self-supervised and does not require video textual descriptions.The comparison distinguishes VIP’s training supervision from R3M’s text-alignment requirement.

3 PROBLEM SETTING AND BACKGROUND

The paper evaluates frozen visual encoders trained on out-of-domain images or videos by defining goal-image tasks over their embedding spaces. Rewards measure progress toward the goal using changes in embedding-based distance.

Out-of-Domain Pre-Training Visual Representation: The pre-training dataset contains out-of-domain video or image observations and excludes robot-task or domain-specific data.A learning algorithm maps raw RGB images to a visual embedding space of dimension K.
Representation Evaluation: Each evaluation task is a Markov decision process whose state space is the representation space and whose task is specified by one or more goal images.The encoder remains frozen during downstream policy learning.
Representation Evaluation: The reward is the difference in goal-embedding distance between consecutive observations.It combines a raw embedding-distance term with a reward-shaping term that encourages progress toward the goal while preserving the optimal policy.
Representation Evaluation: The goal similarity is defined as negative Euclidean distance between the observation and goal embeddings.Policies act on embedded observations and do not explicitly condition on a goal because each task has one goal.

4 VALUE-IMPLICIT PRE-TRAINING

VIP derives an action-free value-learning objective from offline human videos and interprets it as implicit temporal contrastive learning. The resulting embedding emphasizes long-range task progress and local temporal smoothness for downstream reward specification.

4.1 FOUNDATION: SELF-SUPERVISED VALUE LEARNING FROM HUMAN VIDEOS: VIP begins by formulating representation learning from human videos as offline goal-conditioned reinforcement learning over human behavior.The derivation addresses the absence of action labels by moving from the primal policy problem to a dual value-function problem.
4.1 FOUNDATION: SELF-SUPERVISED VALUE LEARNING FROM HUMAN VIDEOS: The dual objective contains no actions and can be estimated entirely from consecutive observations in the offline dataset.This makes self-supervised value learning possible on passive human videos.
4.1 FOUNDATION: SELF-SUPERVISED VALUE LEARNING FROM HUMAN VIDEOS: Using a constant negative reward except at the goal, the learned value captures the discounted number of steps required to reach the goal.The objective therefore encourages visual features that predict discounted temporal distance between video frames.
4.2 ANALYSIS: IMPLICIT TIME CONTRASTIVE LEARNING: VIP’s implicit time-contrastive view treats distant initial frames as positives and intermediate frames as negatives.This differs from standard time-contrastive learning, where positives are temporally closer to the anchor than negatives.
4.2 ANALYSIS: IMPLICIT TIME CONTRASTIVE LEARNING: Attraction between initial and goal frames captures long-range temporal dependencies, while recursive temporal-difference structure makes embedding distances locally smooth.Together, these effects support effective visual reward functions.
4.3 ALGORITHM: VALUE-IMPLICIT PRE-TRAINING (VIP): VIP approximates the value function with negative L2 distance and trains by sampling sub-trajectories whose initial and final frames define the relevant endpoints.The implementation uses offline human videos and a simple visual architecture, with the core loop summarized in Algorithm 1.

5 EXPERIMENTS

The experiments evaluate VIP as a visual reward and representation across trajectory optimization, online RL, real-robot offline RL, and qualitative embedding analyses. Across these settings, VIP generally outperforms prior representations and supports few-shot task learning.

Evaluation setup: The evaluation covers trajectory optimization, online RL, real-robot offline RL, and qualitative analyses using VIP and several pretrained baselines.Experiments use FrankaKitchen tasks, Ego4D and real-robot videos, and comparisons including R3M, ResNet, and other pretrained representations.
Trajectory optimization and online RL: VIP substantially outperforms all baselines in FrankaKitchen trajectory optimization and is the only representation making non-trivial progress in the Hard setting.With stronger MPPI optimizers, VIP reaches an average success rate of 44%, while baselines often worsen as compute increases.
Trajectory optimization and online RL: VIP rewards correlate with human-engineered dense rewards on at least one camera view for 8 of 12 tasks, with correlation coefficients as high as R2 = 0.95.This supports VIP’s ability to capture task-salient information without prior robot-domain or task-specific knowledge.
Real-world few-shot offline RL: VIP-RWR improves upon VIP-BC on all real-robot tasks, while in-domain VIP methods fail completely, indicating that scarce task-specific data can overfit the representation.The harder PickPlaceMelon and FoldTowel tasks particularly require useful reward information in this low-data regime.
Qualitative analysis: VIP produces smoother embedding-distance curves and reward distributions than competing representations, including approximately 20% more first positive-reward-bin counts than R3M.VIP also exhibits fewer negative rewards and approximately 100% fewer extreme rewards than R3M.

6 CONCLUSION

VIP is a self-supervised value-based pre-training objective that provides visual rewards and representations for unseen robotics tasks. Trained on diverse human videos, it captures temporal structure that supports zero-shot reward specification and gains across policy-learning settings.

VIP provides both visual reward and representation for downstream unseen robotics tasks.
Implicit time contrastive learning captures long-range temporal dependencies and local temporal smoothness, supporting zero-shot reward specification.
VIP is trained entirely on diverse, in-the-wild human videos and demonstrates gains over prior pre-trained visual representations across policy-learning settings.
VIP enables sample-efficient real-world robot learning using visual rewards derived from the pre-trained representation.

REPRODUCIBILITY STATEMENT

The supplied material documents VIP’s implementation resources, goal-conditioned RL formulation, contrastive-learning background, and relationship to prior approaches for learning from human videos.

VIP code, model-training instructions, inference instructions, configured hyperparameters, and PyTorch pseudocode are publicly available.
Goal-conditioned RL is formulated over RGB-image observations and goals, with policies maximizing discounted cumulative return.
Occupancy distributions describe goal-conditioned visitation frequencies and satisfy a Bellman flow constraint for stationary policies.
InfoNCE uses positives and negatives in an unsupervised contrastive objective, while TCN attracts temporally close frames and separates frames farther apart in time.
Prior human-video methods commonly learn human–robot invariant representations but require task-specific demonstrations, whereas VIP extends dual offline GCRL to action-free, out-of-domain human-video pre-training.

C.1 PROOF OF PROPOSITION 4.1

The proof develops VIP’s dual optimization under deterministic dynamics and establishes the representation’s implicit repulsion and monotonic distance behavior along optimal paths.

The dual objective is expressed using the initial-state value term and a log-expectation over consecutive dataset observations.
Fixing the representation reduces the inner optimization to a representation-induced state and goal space, after which a prior dual-RL proposition is applied.
Deterministic dynamics allow the transition expectation to be replaced by the observed offline-dataset sample.
The derivation assumes no embedding collisions when equating embedded-state sampling with original observation-pair sampling.
Under the stated assumptions, VIP’s implicit repulsion holds on optimal paths, producing monotonically decreasing embedding distances to the goal.

D.1 DATASET PROCESSING AND SAMPLING

VIP uses pre-processed Ego4D clips and samples video sub-trajectories for training, with a compact PyTorch implementation and documented architecture and hyperparameters.

The pre-processed Ego4D dataset contains approximately 72,000 clips and 4.3 million frames, with clips spanning 10–150 frames.
Training samples a set of videos and then a sub-trajectory from each video, treating each sub-trajectory as a video segment.
Three additional negative observation pairs are sampled from separate video sequences for the log-sum-exp term.
The reported training configuration uses Adam, learning rate 0.0001, mini-batch size 32, and discount factor γ 0.98.
The PyTorch training loop computes goal-relative embedding values for sampled observations, optimizes VIP loss, and updates the vision model.

E.1 FRANKAKITCHEN TASK DESCRIPTIONS

The experiments use a 12-task FrankaKitchen simulation suite and a real-robot setup with RGB observations, randomized object placements, and goal images derived from demonstrations.

FrankaKitchen suite: The FrankaKitchen suite contains 12 tasks from environment version v0.1 version 2.
FrankaKitchen suite: Hard-setting initial frames use the environment default state, while Easy-setting initial frames use the 20th demonstration frame and goals use the final frame.
Trajectory optimization: The trajectory-optimization experiments use MPPI with a 12-step planning horizon and 32 proposed action sequences per step.
Real-robot setup: The real-robot environment uses an RGB-only RealSense camera mounted at the table edge, without depth observations.
Real-robot data: Real-robot data include 20 expert demonstrations for three tasks and 10 expert plus 20 sub-optimal trajectories for CloseDrawer.
Goal specification: Each task’s goal embedding is the average representation of goal images taken from the final frames of its demonstrations.

F.4 QUALITATIVE ANALYSIS

Qualitative analyses show VIP-RWR reaches task-critical states and can recover through re-attempts, while comparisons examine whether these behaviors depend on the pre-training method or dataset.

Robust key action execution: On PickPlaceMelon, failed VIP-RWR trajectories still reach the critical state of gripping the watermelon, unlike baseline failures.
Task re-attempt: VIP-RWR often performs recovery actions after initially failing CloseDrawer or FoldTowel, a behavior not observed for R3M-RWR.
Alternative pre-training methods: The additional Ego4D comparison tests VIP against alternative representations trained on the same pre-training dataset.

G.2 VALUE-BASED PRE-TRAINING ABLATION: LEAST-SQUARE TEMPORAL-DIFFERENCE

The ablation compares VIP with LSTD and examines whether value-based objectives, temporal smoothness, and reward correlation explain VIP’s reward-specification performance.

LSTD comparison: LSTD uses the same embedding-distance parameterization and training setup as VIP but differs in its value-training objective.
LSTD comparison: LSTD outperforms prior baselines in Easy settings but is less effective in Hard settings because it does not capture VIP’s long-range temporal dependency.
Objective properties: VIP’s superior reward specification is attributed to combining long-range temporal dependencies with local temporal smoothness.
Reward correlation: VIP rewards show greater correlation with ground-truth rewards on successful trajectories, with high correlation on at least one camera view for 8 of 12 tasks.
Embedding smoothness: On real-robot data, VIP produces smoother embedding-distance curves than other representations, while randomly sampled Ego4D snippets may not represent complete tasks.
Reward distributions: VIP reward histograms consistently contain more small positive rewards and fewer extreme rewards in either direction than baselines.

H LIMITATIONS AND FUTURE WORK

VIP’s current scope is bounded by its goal-image formulation, symmetric value parameterization, frozen evaluation setup, and focus on robot manipulation, motivating broader goals, asymmetric distances, fine-tuning, and other embodied tasks.

Goal specification: VIP currently provides rewards only for tasks specified via a goal image, which excludes tasks requiring intermediate instructions and steps.The authors also tested visual goals from the same domain, although robots need not appear in the goal image.
Value-function structure: VIP’s symmetric embedding-distance value function assumes environmental reversibility, which may not hold in practice.The authors suggest quasimetrics as a future parameterization for asymmetric structures.
Adaptation: VIP was evaluated only as a frozen visual reward and representation module, so task-specific fine-tuning may improve absolute performance.The authors identify learning how best to fine-tune VIP as future work.
Future applications: Although this work focuses on robot manipulation, VIP’s training objective may also support pre-training for other goal-directed tasks such as visual navigation.The authors propose exploring these additional embodied AI applications.