PaperlayerView exact revision on arXiv

Source-linked AI summary

AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos

Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, Sergey Levine

arXiv:1912.04443v3cs.ROcs.CVcs.LG

TL;DR

Long-horizon robotic reinforcement learning needs less burdensome task specification and training scaffolding. AVID learns multi-stage tasks from human videos by translating them into robot images and applying stage-wise model-based reinforcement learning with learned resets. It learns tasks such as coffee making from 20 minutes of human demonstrations and about 180 minutes of robot interaction, while remaining bounded by the cost of training separate translation models for different tasks.

  • Problem

    Robot reinforcement learning for long-horizon tasks requires laborious reward definition, robot-specific demonstrations, and substantial human effort for task scaffolding.

  • Method

    AVID uses CycleGAN to translate human demonstrations into robot images, extracts stage instructions, and trains a stage-wise model-based reinforcement learning system with learned resets and limited human feedback.

  • Results

    AVID learns coffee making and cup retrieval from human demonstrations, using 20 minutes of human demonstration time and about 180 minutes of robot interaction for coffee making.

  • Takeaways & Limitations

    AVID enables image-based robots to learn multi-stage tasks from human videos while reducing reliance on robot-specific demonstrations and manual resets.

  • Takeaways & Limitations

    The framework may require separate CycleGAN training for each task, although the authors identify amortizing this cost across tasks as future work.

Abstract

from arXiv · show

Robotic reinforcement learning (RL) holds the promise of enabling robots to learn complex behaviors through experience. However, realizing this promise for long-horizon tasks in the real world requires mechanisms to reduce human burden in terms of defining the task and scaffolding the learning process. In this paper, we study how these challenges can be alleviated with an automated robotic learning framework, in which multi-stage tasks are defined simply by providing videos of a human demonstrator and then learned autonomously by the robot from raw image observations. A central challenge in imitating human videos is the difference in appearance between the human and robot, which typically requires manual correspondence. We instead take an automated approach and perform pixel-level image translation via CycleGAN to convert the human demonstration into a video of a robot, which can then be used to construct a reward function for a model-based RL algorithm. The robot then learns the task one stage at a time, automatically learning how to reset each stage to retry it multiple times without human-provided resets. This makes the learning process largely automatic, from intuitive task specification via a video to automated training with minimal human intervention. We demonstrate that our approach is capable of learning complex tasks, such as operating a coffee machine, directly from raw image observations, requiring only 20 minutes to provide human demonstrations and about 180 minutes of robot interaction.

I. INTRODUCTION

AVID addresses the burden of specifying and training multi-stage robotic tasks by learning from human videos rather than robot demonstrations. It translates human demonstrations into robot images, extracts stage instructions, and combines them with stage-wise reinforcement learning and automated resetting.

  • Multi-stage tasks are difficult for reinforcement learning because defining a reward for every stage requires substantial human effort.
  • Robot imitation learning commonly requires laborious kinesthetic teaching or specialized teleoperation hardware and expertise.
  • AVID asks whether robots can learn multi-stage tasks by observing human demonstrations, analogous to how humans learn through observation.
  • Human videos reduce demonstration burden because they require less setup, hardware, and expertise than demonstrations collected directly on the robot.
  • AVID uses CycleGAN to translate human videos into robot images, extracts stage instruction images, and uses them for model-based reinforcement learning with learned resets.
  • AVID evaluates coffee-machine operation and cup retrieval, learning coffee making from 30 human demonstrations in 20 minutes and 180 minutes of robot interaction.

II. RELATED WORK

Prior work addresses robotic reward design, data efficiency, multi-stage learning, and human demonstrations through varied assumptions about rewards, embodiment, and instrumentation. AVID instead combines human videos, pixel-level translation, stage-wise learning, and learned resets.

  • Real-world reinforcement learning is hindered by reward definitions that may require motion capture, object tracking, accelerometers, or thermal cameras.
  • Representation learning and model-based reinforcement learning are established approaches for improving data efficiency in image-based robotic control.
  • Existing multi-stage methods often assume stage-specific rewards or demonstrations matching the robot’s observations and actions.
  • Learned resets can reduce manual resetting, but AVID uses human key presses to indicate stage success or failure because it lacks automatic stage-completion tests.
  • Most robotic imitation methods use demonstrations in the robot’s own observation and action space, whereas AVID learns from human videos.
  • Single-view TCN also handles embodiment changes from human demonstrations, but its typical setup uses multiple human views that are unavailable in AVID’s setting.

III. PRELIMINARIES

AVID treats human-to-robot demonstration conversion as unsupervised image-to-image translation between unpaired domains. CycleGAN combines adversarial translation with cycle consistency, enabling frame-by-frame conversion without paired examples.

  • AVID frames human-to-robot conversion as mapping unpaired source human images to target robot images.
  • The translation assumes an underlying bijective mapping between human and robot image domains, supported by morphological and perspective similarities.
  • CycleGAN learns mappings in both directions using adversarial objectives for the human-to-robot and robot-to-human domains.
  • The cycle-consistency loss penalizes discrepancies after translating an image to the other domain and back.
  • The overall CycleGAN objective combines the two adversarial losses with a cycle-consistency term weighted by λ.
  • Although CycleGAN does not use temporal information, prior work supports translating demonstration videos frame by frame.

B. Structured Representation Learning

AVID represents robot images and actions with a temporally structured latent-variable model. Learned encoders, dynamics, and decoders support instruction-image encoding and model-based planning in latent space.

  • The model represents robot images and actions with a latent-variable model whose generative components are shown separately from the variational family and encoder.
  • AVID treats the underlying state as unobserved, evolving from previous states and actions, while robot images are observations of that state.
  • The generative model includes an initial state distribution, learned neural-network dynamics, and a convolutional decoder for observations.
  • A variational distribution approximates the posterior over latent trajectories conditioned on observations and actions.
  • The encoder and decoder are convolutional networks, and the model parameters are jointly learned by maximizing the variational lower bound.
  • The learned encoder and dynamics model are used to encode instruction images and perform stage-wise model-based planning in latent space.

IV. AUTOMATED VISUAL INSTRUCTION-FOLLOWING WITH DEMONSTRATIONS

AVID specifies robot tasks from human demonstration videos without teleoperation, kinesthetic teaching, or instrumented rewards. It translates demonstrations into the robot domain, extracts stage-completion images, and uses them to construct stage-specific rewards.

  • Problem setting: The task is specified by human demonstration videos rather than teleoperation or kinesthetic teaching, with no instrumented reward signals assumed.The goal is to reduce required human effort, expertise, and instrumentation.
  • A. Translation for Goal Concept Acquisition: CycleGAN is trained on diverse human and randomly collected robot images to translate human demonstrations into the robot’s visual domain.Robot data covers varied settings, including scenes where the cup is moved or placed in the gripper.
  • A. Translation for Goal Concept Acquisition: The learned mapping automatically converts human demonstration videos into robot demonstration videos without manually specified human–robot image correspondences.The method uses the translated videos as the basis for subsequent instruction extraction.
  • A. Translation for Goal Concept Acquisition: AVID extracts key instruction images corresponding to natural stage completions instead of learning directly from every frame of the translated demonstration.Using select frames avoids the poor performance observed for full-video imitation and keeps stage goals explicit.
  • A. Translation for Goal Concept Acquisition: Users manually specify the time steps corresponding to stage completions, a process described as easy because only a modest number of videos is used.The paper identifies automatic stage discovery as future work for more complex tasks.
  • A. Translation for Goal Concept Acquisition: For each stage, AVID trains a success classifier using translated instruction images as positives and other robot images as negatives.The classifiers operate on learned latent states, and their log probabilities provide stage-specific reward signals.

B. Model-Based RL with Instruction Images

AVID combines latent-space model-predictive control with classifier rewards and forward-reset behavior to learn multi-stage tasks while limiting manual resets. Human feedback confirms proposed stage completions and supplies corrective data after failures.

  • B. Model-Based RL with Instruction Images: Latent-space MPC-CEM searches over action sequences using learned dynamics and classifier-based rewards, then executes the first action of the optimized sequence.CEM refines the search by evaluating model-generated trajectories in latent space.
  • B. Model-Based RL with Instruction Images: During stage s, the planner uses classifier Cs and aims to exceed threshold α; otherwise, it automatically switches to Cs−1 to reset toward the preceding stage.This forward-reset loop reduces the need for human-provided episode resets.
  • B. Model-Based RL with Instruction Images: When the classifier threshold is met, human feedback marks success or failure; success advances to Cs+1, while failure triggers exploration and retraining.Stage-wise learning avoids compounding errors from learning the entire task simultaneously.
  • B. Model-Based RL with Instruction Images: Algorithm 1 initializes stage-wise training with a pretrained model, classifiers, robot data, a maximum attempt count K, and threshold α.Each stage repeatedly plans, records trajectories, and checks whether the stage has been completed.
  • B. Model-Based RL with Instruction Images: If a stage is completed, the procedure advances; otherwise, it retrains the model and classifiers using newly collected data before continuing.The algorithm explicitly incorporates both successful goal images and additional exploration data.
  • B. Model-Based RL with Instruction Images: Failure-triggered exploration samples states near suspected successes, while labeling the failed current image negative improves classifier accuracy.These updates combat false positives and artifacts in translated images.
  • B. Model-Based RL with Instruction Images: After the final stage, AVID runs the procedure in reverse to return the robot to its initial state and avoid manual resets between training attempts.The complete forward-and-reverse process repeats until the robot reliably completes the task.

V. EXPERIMENTS

The experiments test whether AVID learns temporally extended visual tasks from human demonstrations, whether instruction images and latent planning help, and what is lost without robot demonstrations. They compare AVID with targeted ablations and teleoperated-demonstration methods while measuring supervision burden.

  • V. EXPERIMENTS: The experiments ask whether AVID can solve temporally extended vision-based tasks directly from human demonstrations.This evaluates the central capability of the proposed system.
  • V. EXPERIMENTS: They test the benefits of instruction images and latent-space planning through ablations that use full translated videos or pixel-space planning.These comparisons isolate two design choices in AVID.
  • V. EXPERIMENTS: The experiments examine the cost of lacking robot demonstrations by comparing imitation learning with access to teleoperated robot demonstrations.The teleoperated demonstrations are treated as oracle information for this comparison.
  • V. EXPERIMENTS: The evaluation also analyzes the human supervision burden of AVID alongside task performance.The paper reports this analysis with the experimental results and a supplementary video.

A. Comparisons

The comparison suite contrasts stage-aware and non-stage-aware learning, full-video versus instruction-image training, pixel-space versus latent planning, and human versus teleoperated demonstrations. It also documents the data and computational requirements associated with these choices.

  • A. Comparisons: Behavioral cloning and BCO use teleoperated robot demonstrations and do not support AVID’s stage-wise training procedure.Behavioral cloning additionally learns directly from observation-action pairs, whereas BCO infers actions with an inverse model.
  • A. Comparisons: The full-video ablation trains BCO on translated human demonstrations rather than the selected instruction images used by AVID.It tests whether learning from entire translated demonstrations is effective.
  • A. Comparisons: The pixel-space ablation retains translated instruction images, stage-wise training, learned resets, and human feedback but replaces latent-space planning with DVF.This isolates the planning representation while preserving several AVID components.
  • A. Comparisons: Table I compares stage awareness, demonstration supervision, image counts, pretraining data, and online training data across methods.Only AVID and the pixel-space ablation are stage-aware; BCO and behavioral cloning require teleoperated demonstrations.
  • A. Comparisons: The instruction sequences show human-demonstration frames translated into robot-domain images for coffee making and cup retrieval.The sequences expose stage structure while also showing artifacts such as displaced grippers and deformed cups.
  • A. Comparisons: Time-contrastive networks learn temporally consistent embeddings from human demonstrations and are evaluated with the same RL subroutine as AVID.Unlike AVID’s classifier rewards, TCN uses its learned embedding for cross-domain generalization.

B. Experimental Setup

The experiments use two temporally extended visual tasks on a Sawyer robot, with methods trained from human, random-trajectory, and direct robot-demonstration data. AVID is evaluated against other methods using stage-wise success rates over ten trials.

  • Tasks and robot: AVID is evaluated on coffee-machine operation and cup retrieval using 64-by-64-by-3 RGB observations and a 7 DoF Sawyer manipulator.The robot uses end-effector velocity control.
  • Data collection: Training data combines human action videos, random robot trajectories, and domain-specific data for training CycleGAN and method-specific models.Human videos are reused to generate translated robot demonstrations, while random robot trajectories support model training.
  • Evaluation: Table II reports success through each stage over 10 trials, comparing human-demonstration methods with methods trained from direct robot demonstrations.AVID succeeds 8 times out of 10 on coffee making and 7 times out of 10 on cup retrieval.
  • Coffee making: Coffee making contains three stages: picking up the cup, placing it in the machine, and pressing the machine button.The task uses 30 human demonstrations and 900 images, collected in about 20 minutes.
  • Cup retrieval: Cup retrieval contains five stages, including grasping and opening the drawer, moving the arm aside, picking up the cup, and placing it on the drawer.The intermediate arm-movement stage prevents the planner from bumping into the drawer door.

C. Experimental Results

AVID achieves the strongest performance among human-demonstration methods on both tasks, while stage-wise training and resetting reduce later-stage failures. Training requires limited human feedback, whereas full-video and other comparison methods struggle on longer-horizon behavior.

  • Overall performance: AVID achieves the best performance among human-demonstration methods on both tasks, learning earlier stages reliably and accumulating fewer later-stage errors.It uses automated resetting and retrying during both reinforcement learning and final evaluation.
  • Ablations: The full-video ablation picks up the cup 70% of the time for coffee making but fails later stages and fails entirely on cup retrieval.The result is attributed to translation artifacts and the absence of stage-wise training.
  • Baselines: BCO performs better with teleoperated robot demonstrations than with translated demonstrations but remains significantly worse than AVID from translated demonstrations.This comparison indicates that full demonstrations can accumulate errors on long-horizon image-based tasks.
  • Baselines and human burden: Behavioral cloning outperforms AVID on coffee making but performs worse on cup retrieval, where the greater number of stages favors explicit stage-wise training.AVID training took about one hour per task and used fewer than 150 human key presses in both cases.

VI. DISCUSSION AND FUTURE WORK

AVID combines CycleGAN translation, stage-wise latent-space model-based reinforcement learning, learned resets, and human feedback to learn multi-stage tasks from human videos. The authors identify reuse of translation models across tasks as a path toward reducing future data-collection and training costs.

  • Discussion: AVID learns visual multi-stage robotic tasks directly from human demonstration videos using CycleGAN-generated robot instruction images.The method learns coffee making and cup retrieval from raw image observations.
  • Discussion: The system uses latent-space model-based planning, learned resets, and human feedback to train temporally extended tasks robustly and efficiently.These components support stage-wise learning from translated visual instructions.
  • Discussion: AVID outperforms prior methods and human-demonstration ablations, and exceeds behavioral cloning from teleoperated demonstrations on one task.The reported demonstrations cover coffee making and cup retrieval.
  • Future work: The authors propose amortizing CycleGAN training and data collection across multiple tasks rather than training a separate translation model for each task.They suggest that a cup-retrieval translator could support learning a related cup-placement task without additional upfront cost.
Loading 1912.04443v3…