PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos

Annie S. Chen, Suraj Nair, Chelsea Finn

arXiv:2103.16817v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Generalist robots need reward functions that recognize task success from onboard observations despite limited broad robotic data. DVD learns task-conditioned rewards by classifying whether human and robot videos perform the same task, leveraging diverse human videos and modest robot demonstrations. The resulting rewards generalize to unseen environments and tasks, support visual model predictive control, and enable a real WidowX200 robot to perform an unseen task from one human demonstration.

  • Problem

    Robots need reward functions that generalize across environments and tasks, but broad, high-quality robotic interaction datasets remain difficult to collect.

  • Method

    DVD trains a classifier to predict whether two human or robot videos complete the same task, then uses a human demonstration and robot behavior to score task success.

  • Results

    Diverse human-video training improves generalization to unseen environments and tasks, supports visual model predictive control, and enables an unseen-task demonstration on a real WidowX200 robot.

  • Takeaways & Limitations

    Human videos can provide broad experience for learning robotic reward functions that generalize beyond the robot data’s environments and tasks.

  • Takeaways & Limitations

    DVD still assumes some robot demonstrations and task labels, does not directly learn a generalizable policy or visual dynamics model, and has only been tested on coarse tasks.

Abstract

from arXiv · show

We are motivated by the goal of generalist robots that can complete a wide range of tasks across many environments. Critical to this is the robot's ability to acquire some metric of task success or reward, which is necessary for reinforcement learning, planning, or knowing when to ask for help. For a general-purpose robot operating in the real world, this reward function must also be able to generalize broadly across environments, tasks, and objects, while depending only on on-board sensor observations (e.g. RGB images). While deep learning on large and diverse datasets has shown promise as a path towards such generalization in computer vision and natural language, collecting high quality datasets of robotic interaction at scale remains an open challenge. In contrast, "in-the-wild" videos of humans (e.g. YouTube) contain an extensive collection of people doing interesting tasks across a diverse range of settings. In this work, we propose a simple approach, Domain-agnostic Video Discriminator (DVD), that learns multitask reward functions by training a discriminator to classify whether two videos are performing the same task, and can generalize by virtue of learning from a small amount of robot data with a broad dataset of human videos. We find that by leveraging diverse human datasets, this reward function (a) can generalize zero shot to unseen environments, (b) generalize zero shot to unseen tasks, and (c) can be combined with visual model predictive control to solve robotic manipulation tasks on a real WidowX200 robot in an unseen environment from a single human demo.

I. INTRODUCTION

General-purpose robots need reward functions that determine task success from onboard observations and generalize across environments and tasks. DVD addresses limited robotic interaction data by learning functional similarity from diverse human videos and modest robot demonstrations.

  • Generalist robots remain difficult to realize because their reward functions must assess task proficiency from onboard observations across environments and tasks.
  • Large-scale robotic interaction data is difficult to collect, whereas in-the-wild human videos provide plentiful, diverse visual experience.Human videos still introduce domain, action-space, viewpoint, and quality differences relative to robots.
  • DVD classifies whether two videos depict the same task, using human activity labels and robot demonstrations to learn functional similarity across visual domains.At deployment, a human demonstration is compared with robot behavior to produce a task-success score.
  • Training on diverse human videos improves generalization to unseen environments and tasks by 15-20% absolute in downstream task success over robot-only data.The method also combines with visual model predictive control and enables an unseen task in an unseen environment from one human demonstration on a WidowX200 robot.
  • The work extends reward learning beyond single-task, single-environment settings toward multi-task visual manipulation conditioned on one human task video.

B. Robotic Learning from Human Videos

Prior approaches translate human demonstrations or collect broad robotic datasets, but DVD instead learns human-robot functional similarity from diverse, weakly aligned videos. Its training setup uses labeled human and robot demonstrations to support reward generalization without a close one-to-one data mapping.

  • Some prior methods track objects or hands in human videos and hand-design mappings from human sequences to robot actions or motion primitives.
  • End-to-end approaches commonly translate human demonstrations or goals into the robot perspective using paired or unpaired pixel-based data.
  • DVD explicitly targets large, diverse real-world video collections containing varied individuals, viewpoints, backgrounds, objects, and tasks.
  • Unlike prior work focused on single tasks in single environments, this work learns multi-task rewards conditioned on a single human video.
  • The training discriminator predicts whether two videos correspond to the same task, with videos drawn from either human or robot demonstration sets.
  • DVD differs from broad robotic-dataset approaches by sidestepping challenges associated with collecting diverse robotic interaction data.

A. Problem Statement

The problem setting learns task-conditioned, non-Markovian rewards from visual demonstrations for robots and humans that may differ in actions and dynamics. Training combines abundant human videos with limited robot data, without requiring visual similarity or action access.

  • The robot operates in a fixed-horizon MDP whose state space is RGB images, with task-specific rewards, actions, and stochastic dynamics.
  • Human and robot MDPs share the state space, task reward, and horizon but may have different action spaces and transition dynamics.
  • Rewards are inferred from task videos because the underlying task reward functions are unobserved and may depend on recent history rather than only the current state.
  • The learned model estimates each task’s reward from a sequence of H states and a task-demonstrating video, supporting planners or policies with memory.
  • Training uses human and robot demonstration datasets, with no constraints on human-video viewpoints, backgrounds, quality, or task balance.
  • The reward should generalize to new tasks from a new demonstration while relying only on visual observations despite potentially large domain shifts.

B. Domain-Agnostic Video Discriminators

DVD learns functional similarity by classifying whether pairs of human or robot videos depict the same task. Its training procedure is designed to bridge substantial visual differences between human and robot domains.

  • DVD trains a classifier on two videos to predict whether they complete the same task, using task labels to obtain pairwise supervision.
  • The method must identify functional behavior across highly diverse human videos and visually different robot environments to bridge the domain gap.
  • Training samples an anchor video, positive and negative examples, and updates the discriminator using the pairwise classification objective.
  • During planning, the trained reward function and video prediction model are used with a human task demonstration to evaluate predicted action trajectories.

C. DVD Implementation

DVD represents videos with a fixed pretrained encoder and learns a task-similarity classifier over their latent representations. Training balances pair types across the larger human dataset and smaller robot dataset.

  • DVD computes similarity by applying a parameterized similarity network to the pretrained encoder representations of two videos.
  • The encoder is pretrained on Something-Something-V2 and fixed, while the fully connected similarity classifier is randomly initialized and trained for binary prediction.
  • Although human videos substantially outnumber robot videos, batches are sampled to be roughly balanced between human and robot demonstrations.

D. Using DVD for Task Execution

DVD is used as a visual reward inside model-predictive control: predicted trajectories are scored against a human demonstration, and the highest-scoring action sequence is executed. The experiments evaluate environment, task, single-demonstration, and real-robot generalization.

  • VMPC conditions DVD on a human demonstration and uses predicted video similarity as the reward for optimizing actions with a learned visual dynamics model.
  • The planner samples action sequences, rolls out imagined visual trajectories, scores them with DVD against the demonstration, and executes the highest-scoring sequence.
  • The experiments test whether diverse human data improves generalization to unseen environments and tasks, single-human-demonstration performance, and reward inference on a real robot.

A. Simulated Experimental Set-Up

The simulated study tests DVD across progressively shifted tabletop environments and three target manipulation tasks specified by unseen human videos. It examines whether diverse human data improves reward-function generalization beyond robot demonstrations alone.

  • Environments: The simulated tabletop setup includes a drawer, faucet, coffee cup, and coffee machine across four environment variants with increasing visual and arrangement shifts.The variants include the training environment, changed colors, changed colors and viewpoint, and changed colors, viewpoint, and object arrangement.
  • Tasks: The three target tasks are closing an open drawer, turning the faucet right, and pushing the cup toward the coffee machine.Each task is specified by an unseen in-the-wild human video.
  • Training Data: Training uses robot videos for the target tasks together with Something-Something-V2 human videos spanning many objects, environments, and up to 15 human tasks.The simulated robot demonstrations comprise 120 videos for three tasks, with the human-data amount varied by experiment.
  • Environment Generalization: 20% on average: training with human videos significantly improves environment generalization over robot-only training, even when robot-only receives a test-time robot demonstration.Success rates are computed over three randomized sets of 100 trials.
  • Environment Generalization: DVD remains generally robust to the number of human tasks, and nine completely unrelated tasks still outperform using no human videos.During planning, DVD assigns high similarity to trajectories completing the demonstrated task and low similarity to irrelevant behavior.

C. Experiment 2: Task Generalization

The task-generalization experiment withholds target-task data during training and evaluates human-video-conditioned rewards in the original environment, alongside prior methods and a real-robot extension. Human videos improve unseen-task performance, and DVD also transfers to an unseen WidowX200 environment.

  • Experiment 2: Task Generalization: Task generalization is evaluated without human or robot training data from the target tasks, using three different robot tasks and evaluation in the training environment.The robot training tasks are opening the drawer, moving something right to left, and not moving objects.
  • Experiment 2: Task Generalization: Roughly 10% on average: human videos significantly improve task generalization over robot-only training, even when the robot-only model receives a robot demonstration.Six unrelated human tasks create a gap of more than 20% over robot-only training.
  • Experiment 3: Prior Work Comparison: DVD outperforms both other baselines by over 30% in task generalization.Concept2Robot cannot naturally test an unseen task specified by a human video because it is not demo-conditioned and is trained on all 174 human-video tasks.
  • Experiment 4: Real Robot Efficacy: On the real WidowX200, DVD is tested in a toy-kitchen environment with one seen task and one unseen task after training in a file-cabinet environment.The training tasks are closing something and pushing left to right; testing uses closing something and pushing right to left.
  • Experiment 4: Real Robot Efficacy: Over 65-70% success: DVD trained with six human tasks succeeds on the real robot, whereas robot-only training succeeds at most 40%.The results are reported over 20 trials, and unrelated human tasks generally improve performance over human videos limited to training tasks.

F. Ablation on Amount of Robot Data for Training

The robot-data ablation tests whether DVD retains environment-generalization performance with fewer robot demonstrations. Performance declines only slightly at 20 demonstrations per task, indicating that diverse human data supports low-robot-data training.

  • Ablation on Amount of Robot Data for Training: Using as few as 20 robot demonstrations per task decreases DVD performance by only a small margin in environment generalization.The prior simulation experiments used 120 robot demonstrations per task.
  • Ablation on Amount of Robot Data for Training: The authors report that leveraging diverse human data enables good performance with very little robot data.This conclusion concerns the tested environment-generalization setting and does not establish a generalizable policy or dynamics model.
  • Ablation on Amount of Robot Data for Training: DVD performs comparably with fewer robot demonstrations, while 120 demonstrations per task provide only a slight benefit over 20 or 40.The comparison is summarized in the ablation figure.

APPENDIX

The appendix documents DVD’s data, optimization, video-prediction, and comparison procedures. It specifies how robot and human videos are sampled, preprocessed, and used with SV2P and behavioral-cloning baselines.

  • Dataset Details: Simulation DVD uses 120 robot demonstrations for three tasks, split between the original and rearranged training environments.These videos are collected with model predictive control using random shooting, a ground-truth video-prediction model, and task-specific shaped rewards.
  • Dataset Details: The validation set combines all available human validation videos for selected tasks with 48 robot demonstrations, plus eight demonstrations per task for WidowX200 experiments.The robot validation videos are split between the original and rearranged environments.
  • Hyperparameters: DVD is trained with SGD using learning rate 0.01, momentum 0.9, weight decay 0.00001, batch size 24, and 120 epochs.Each batch contains triplets with two videos sharing a task label and one video with a different label.
  • Hyperparameters: Input clips are randomly rotated, resized to height 120, and cropped to 120 × 120 × 3 during training; planning clips are center-cropped to the same spatial size.Planning demonstrations are rescaled and spliced to 30-40 frames.
  • SV2P Training: SV2P supplies action-conditioned video prediction models for visual model predictive control when evaluating DVD on potentially unseen tasks.SV2P samples a latent variable and generates image predictions conditioned on actions.
  • SV2P Training: Each simulation environment uses 10,000 random 60-frame episodes to train SV2P, which predicts the next 15 frames from five input frames.The robot model is trained on 58,500 autonomously collected interaction frames in the original training environment.
  • Additional DVD Details: DVD training batches sample triplets so two videos share a task label while the third has a different task label, with robot and human sources sampled probabilistically.The detailed sampling procedure selects same-task partners and repeatedly samples a different-task video.
  • Comparisons: The behavioral-cloning baseline concatenates conditioning-video and image-state encodings before predicting robot actions through an MLP trained with mean squared error.It uses DVD’s pretrained video encoder and a pretrained ResNet18 for image states.

B. Experimental Details

The experiments evaluate DVD in simulation and on a real WidowX200 robot, using trajectory planning and task-specific success criteria across environments and tasks.

  • Simulation Experiments: Simulation uses RGB observations with five-dimensional gripper actions in Meta-World-based Mujoco environments.Actions comprise linear and angular gripper velocity plus discrete gripper open/close control.
  • Simulation Experiments: Simulation trials plan 3 trajectories of length 20, sample 100 action sequences per trajectory, and execute one of the top 5 DVD-ranked trajectories.Each method is evaluated over 100 trials with 3 different seeds for every task.
  • Simulation Experiments: The simulation evaluation covers closing a drawer, turning a faucet right, and moving a cup toward the coffee machine.Success is defined using task-specific final-state distance thresholds.
  • Real Robot Experiments: Real-robot trials use one length-10 trajectory, two CEM iterations, 100 sampled action sequences, and execution of one of the top 5 DVD-ranked trajectories.The WidowX200 experiments evaluate manipulation tasks including closing an unseen toy kitchen door.

C. Additional Experimental Results

Additional experiments show that human videos improve DVD’s environment generalization and multi-task reward quality, while adding data reduces discriminator accuracy but preserves robust validation performance. On a real WidowX200, DVD ranks task-completing trajectories highly.

  • Environment Generalization: Human videos enable more effective generalization across new environments than relying only on robot videos.This conclusion holds across the three simulation tasks and varying amounts of human data.
  • Environment Generalization: DVD outperforms the next-best method by over 40% on the drawer task and 30% on the faucet task across environments.Comparisons include Concept2Robot, demo-conditioned behavioral cloning, and a random policy.
  • Environment Generalization: DVD’s multi-task reward signals are more effective across environments than Concept2Robot and behavioral cloning, whose behavior is task- or environment-specific.Concept2Robot performs particularly well on the cup task, while behavioral cloning shows erratic task imitation.
  • Training Trade-off: Robot-only DVD reaches 99% validation accuracy, whereas DVD trained with robot data and 12 human tasks still reaches 89%.The lower accuracy from adding human videos is associated in these experiments with greater generalization to unseen environments and tasks.
  • Real Robot Results: On the WidowX200, DVD highly ranks trajectories that complete the same task as the human demonstration.The real-robot examples concern closing an unseen toy kitchen door.
Loading 2103.16817v1…