PaperlayerView exact revision on arXiv

Source-linked AI summary

Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller

Pratyusha Sharma, Deepak Pathak, Abhinav Gupta

arXiv:1911.09676v1cs.LGcs.CVcs.ROstat.ML

TL;DR

Learning to imitate a third-person human demonstration with a robot remains challenging when the agent must adapt goals to its own context without access to environment state. This paper decouples sub-goal generation from control, and reports better generalization to novel objects in unseen configurations, including on a real Baxter robot.

  • Problem

    Imitating a third-person human demonstration requires translating demonstrated goals to the robot’s context and mapping actions, using raw camera images without environment state-space information.

    Source: §1 · pp. 1–2

  • Method

    A high-level module generates first-person visual sub-goals conditioned on the human video and robot observation, while a low-level controller predicts actions to reach them.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    The decoupled approach outperforms baselines on all four reported metrics and generalizes better to novel objects; the complete system is deployed on Baxter.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The reported results demonstrate improved generalization to novel objects in unseen configurations using raw-pixel input on a real robot.

    Source: §8 · p. 9

    The modules operate per time step without temporal information, resulting in shaky predicted trajectories.

    Source: §8 · p. 9

Abstract

from arXiv · show

We study a generalized setup for learning from demonstration to build an agent that can manipulate novel objects in unseen scenarios by looking at only a single video of human demonstration from a third-person perspective. To accomplish this goal, our agent should not only learn to understand the intent of the demonstrated third-person video in its context but also perform the intended task in its environment configuration. Our central insight is to enforce this structure explicitly during learning by decoupling what to achieve (intended task) from how to perform it (controller). We propose a hierarchical setup where a high-level module learns to generate a series of first-person sub-goals conditioned on the third-person video demonstration, and a low-level controller predicts the actions to achieve those sub-goals. Our agent acts from raw image observations without any access to the full state information. We show results on a real robotic platform using Baxter for the manipulation tasks of pouring and placing objects in a box. Project video and code are at https://pathak22.github.io/hierarchical-imitation/

1 Introduction

Third-person imitation requires inferring a demonstrator’s intent, translating goals to a robot’s context, and controlling the robot; this combination is difficult to learn jointly. The paper proposes a decoupled hierarchy for learning from third-person videos and evaluates it on Baxter.

1 Introduction

  • Third-person imitation requires interpreting intent, adapting goals to novel objects, mapping actions to a robot, and performing low-level control.The introduction contrasts this with learning from robot demonstrations or videos captured from the robot’s or a similar viewpoint [19] [30].Source: §1 · p. 1
  • The target setting uses a human demonstration video and raw camera images, without access to environment state-space information.The robot must translate demonstrated goals to its own context and map the observed actions to its trajectory.Source: §1 · pp. 1–2
  • The approach separates task-specific first-person sub-goal generation from a task-independent controller, sharing the controller across tasks for sample efficiency and robustness.Both modules are trained with paired but temporally unaligned human videos and robot joint-angle trajectories.Source: §1 · p. 2
  • Experiments use Baxter for pouring and placing objects in a box, evaluating modules individually, testing generalization, and deploying the complete system.The tests include novel objects and demonstrations.Source: §1 · p. 2

2 Problem Setup: Third Person Visual Imitation

The problem is to make a robot follow a novel human demonstration video by predicting robot joint-angle configurations from its initial state. Training uses paired human and robot demonstrations, while inference uses the human video.

2 Problem Setup: Third Person Visual Imitation

  • At inference, the agent follows a novel human video from its initial state by predicting a sequence of robot joint-angle configurations.The action space is a vector of robot joint angles.Source: §2 · pp. 2–3
  • The hierarchy predicts a robot visual goal from human frames and the current robot state, then trains the controller on state-action-next-state triplets.At inference, the modules run in sequence, with a new goal generated after the robot reaches the current one.Source: Figure 2 · p. 3
  • Training uses human-video frames, robot demonstration images, and robot joint-angle states from paired demonstrations that are not exactly time-aligned.The sequences are subsampled to help them become roughly aligned; at test time, only the human image sequence is available.Source: §2 · p. 3

3 Hierarchical Controllers for Imitation

The approach separates goal inference from control: a high-level module generates robot-view sub-goals, and a low-level inverse controller acts to reach them. The independently trained modules alternate at inference.

3 Hierarchical Controllers for Imitation

  • Decoupling goal inference from control makes the approach modular and more sample-efficient than end-to-end learning, while allowing intermediate sub-goals to be visualized.The goal generator translates the human demonstration into robot-context pixel-level goals, which the inverse controller follows.Source: §3 · p. 3

3.1 High-Level Module: Goal Generator

  • The goal generator translates changes in human demonstration images into changes to the current robot image, simplifying goal generation to re-rendering rather than full image generation.It uses a conditional GAN with a U-Net-style architecture to generate realistic sub-goals.Source: §3.1 · p. 4
  • The goal generator takes the current robot visual state and two human-video frames as input, predicting the robot visual state at a later step.Wider human-frame context can be used when the human and robot demonstrations are not aligned; L1 loss encourages the correct frame and adversarial loss encourages realistic samples.Source: §3.1 · p. 4

3.2 Low-Level Module: Inverse Controller

  • The inverse controller maps the current robot visual state and predicted goal state to an action, and is not specific to a particular task.Decoupling allows controller sharing across tasks and offers the possibility of training with additional self-supervised data, left for future work.Source: §3.2 · pp. 4–5

3.3 Inference: Third-person Imitation

  • At inference, the goal generator and inverse controller run alternately, producing a sub-goal and actions to reach it until the human video ends.The next sub-goal is generated from the robot’s current observation and the human demonstration sequence.Source: §3.3 · p. 5

4 Implementation Details and Baselines

The experiments use MIME human demonstrations paired with Baxter trajectories and compare module variants as well as joint-system baselines.

4 Implementation Details and Baselines

  • MIME supplies 8,260 paired human-kinesthetic robot demonstrations spanning 20 tasks, with task-specific and multi-task splits for training, validation, and testing.Trajectories are subsampled to 200 time steps, and the inverse model regresses 14-dimensional robot joint-angle actions.Source: §4 · p. 5
  • The joint system is compared with an end-to-end model that predicts actions directly and DAML, which adapts task parameters with one test-time gradient update.Module ablations include CycleGAN and L1- and L2-loss prediction models.Source: §4 · p. 5

5 Results: Generalization of Individual Hierarchical Modules

The module evaluations test generalization to new positions, objects, and tasks while assuming ground-truth access to the other hierarchical module. The inverse model performs better than baselines on held-out and novel tasks, while the goal generator generalizes to novel objects and locations but is not task-agnostic.

5.1 Generalization to new positions of the same object

  • With the human demonstration fixed, the goal generator produces sub-goals responsive to changes in object position.Figure 4 also shows sub-goal predictions at ten-step intervals, approximately two seconds apart, that are consistent across the trajectory.Source: §5.1 · p. 6, Figure 4 · p. 6

5.2 Generalization to new objects

  • Across four metrics, the goal generator outperforms the baselines on novel objects in different configurations.The evaluation uses the pouring test set.Source: Table 1 · p. 7, §5.2 · p. 7
  • On novel objects and locations, the inverse model significantly outperforms all other baselines.Position generalization is tested using ground-truth test images, with results reported jointly with novel-object evaluation.Source: §5.1 · p. 6, Table 2 · p. 7, §5.2 · p. 7
  • Feature-matching baselines trail the model: temporal-feature nearest neighbors give rMSE 22.20 (stderr 2.14), while SIFT gives rMSE 45.32 (stderr 6.12).The authors attribute SIFT’s poor correspondence matching to the large human–robot domain gap.Source: §5.2 · p. 7

5.3 Generalization to new tasks

  • The goal generator is not task-agnostic; training one that is task-agnostic is left for future work.The authors suggest that task-agnostic training may be possible because the goal generator and inverse model do not depend on temporal information.Source: §5.3 · p. 7
  • On held-out data from 15 training tasks and 5 novel tasks, the inverse model has much lower error than baseline methods.The evaluation uses an inverse model trained on the first 15 MIME tasks.Source: Table 3 · p. 7, §5.3 · p. 7
  • DAML’s cited results concern single planar-manipulation tasks, and scaling it across task categories may require changes.The authors note that DAML was not designed specifically for third-person task transfer.Source: §5.3 · pp. 7–8

6 Results: Generalization and Evaluation of Joint Hierarchical Model

The joint hierarchical system is evaluated on pouring and box placement, including tests with novel objects in random locations. The reported comparison favors the alternating goal-generator and inverse-controller approach, while completion remains imperfect.

6 Results: Generalization and Evaluation of Joint Hierarchical Model

  • The joint evaluation is the final test of running the decoupled models together on the robot.The passage points readers to the project website for robot demonstration videos.Source: §6 · p. 8
  • Pouring success requires reaching and pouring within 5 cm of the target cup.These criteria are described as similar to those used by Yu et al. [30].Source: §6 · p. 8
  • For box placement, success requires reaching and dropping the object within 5 cm of the box, and the models are trained on pouring alone.The box task tests generalization from pouring to placing.Source: §6 · p. 8
  • The alternating goal generator and inverse controller outperform other approaches on long-horizon tests with novel objects in random locations, including imperfect predictions and actions.The passage reports this joint comparison via Table 4 but gives no numerical values.Source: §6 · p. 8

7 Related Work

The related work covers domain adaptation, learning from demonstrations, reward inference, and visual foresight. The paper also reports a joint evaluation in which its approach outperforms baselines on pouring and box placement, though completion is imperfect.

7 Related Work

  • The proposed approach outperforms baselines on pouring and box placement by a significant margin, but remains far from perfect task completion.The joint evaluation is reported in Table 4.Source: Table 4 · p. 8

Domain Adaptation

  • Prior domain-adaptation methods map between domains or learn invariant representations, while third-person imitation methods may require multi-view training data; modular controllers address these transfer issues.The passage cautions that task-invariant features alone may not bridge differences in physical configurations.Source: §7 · p. 8

Learning from Demonstrations (LfD)

  • Most learning-from-demonstration work learns a task from a handful of demonstrations, whereas this paper targets imitation of a task with new objects from one human video.The cited LfD demonstrations may come from kinesthetic teaching, teleoperation, or motion capture [31].Source: §7 · p. 8

Explicitly Inferring Rewards

  • Reward-inference approaches use reinforcement learning after constructing task rewards, but the passage says they require substantial real-world data and retraining for each new task.The paper contrasts this with supervised maximum-likelihood training as more scalable to real robots.Source: §7 · p. 9

Visual Foresight

  • Visual-foresight methods rely on image-space dots and action-conditioned predictions, while this setting specifies goals directly from human videos without hand-specified goals.The passage gives pouring as a harder task to specify with image dots alone.Source: §7 · p. 9

8 Discussions

The paper presents decoupled hierarchical controllers for third-person imitation from raw pixels, reporting improved generalization to novel objects in unseen configurations. It identifies missing temporal information as a source of shaky predicted trajectories and proposes several future directions.

8 Discussions

  • The approach infers a task from one third-person human demonstration and executes it on a real robot from first-person perspective using raw pixels.The paper reports improved generalization over end-to-end and other baselines on novel objects in unseen configurations.Source: §8 · p. 9
  • The high- and low-level modules operate per time step without temporal information, resulting in shaky predicted trajectories.Training across many tasks could potentially alleviate this limitation; the passage also suggests using additional self-supervised data to improve the low-level controller.Source: §8 · p. 9
  • Explicit decoupling may allow additional self-supervised data to train the low-level controller for robustness to failures and different joint configurations.The authors leave this direction for future work.Source: §8 · p. 9

A.1 Implementation Details

The implementation uses an image-based goal generator followed by a low-level inverse controller. The generator predicts robot sub-goals from paired human and robot observations, while the controller maps robot states and goals to joint actions.

A.1 Implementation Details

  • The goal generator uses a pix2pix-inspired U-Net with skip connections, takes 128×128 images, and is trained with random resizing and cropping.It is optimized with Adam at learning rate 0.0002, with β1 = 0.5 and β2 = 0.999.Source: §A.1 · p. 9
  • The generator receives human demonstration images at t and t + k plus the robot image at t, and predicts the robot goal image at t + k.After subsampling trajectories to 200 time steps, the authors find k = 10 best balances precise smaller steps with goals sufficiently different from the current state.Source: §A.1 · p. 9
  • The inverse controller uses ResNet-18 convolution blocks and fully connected layers to predict a 14-dimensional tuple of Baxter joint angles from the current robot image and next goal image.The input images are randomly cropped to 85% to improve robustness to arm and camera vibrations; training uses Adam with learning rate 0.001.Source: §A.1 · p. 10

A.2 Generalization of Inverse Model: Simulation Experiments

In Sawyer simulation, the learned inverse-model controller achieved a mean RMSE of 6.09 (stderr 2.8) on new objects at different test locations. The authors interpret this as evidence of controller robustness.

A.2 Generalization of Inverse Model: Simulation Experiments

  • Mean RMSE was 6.09 (stderr 2.8) for new objects at different test locations.The authors state that this suggests controller robustness.Source: §A.2 · p. 10
  • Sawyer training trajectories came from a reaching policy trained with different objects placed in front of the robot.Demonstrations were collected using policies trained with proximal policy optimization (PPO).Source: §A.2 · p. 10
  • The PPO policies were trained on diverse objects to collect 500 demonstrations.Source: §A.2 · p. 10

Related papers

Loading 1911.09676v1…