Source-linked AI summary
Zero-Shot Visual Imitation
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A. Efros, Trevor Darrell
TL;DR
The paper addresses the burden of learning imitation from expert action supervision and the challenge of inferring actions from visual goals. It learns goal-conditioned skills from unsupervised exploration using forward consistency, then evaluates visual zero-shot imitation in manipulation and navigation. The method improves knot-tying accuracy from 36% to 60%, generalizes navigation to unseen offices, and benefits from curiosity-driven exploration in VizDoom.
Problem
Action-supervised imitation requires tedious expert-controlled demonstrations for each new task, while visual-only imitation leaves the agent to infer how to act.
Method
The agent explores without expert supervision, relabels visited states as goals, and learns a goal-conditioned skill policy with forward consistency and a goal recognizer.
Results
36% to 60% knot-tying accuracy, generalization to unseen office environments, and more accurate demonstration following with curiosity-driven than random exploration in VizDoom.
Takeaways & Limitations
Visual observations alone can communicate task goals while the agent learns action strategies from self-supervised exploration, without expert actions.
Takeaways & Limitations
The approach depends on exploration quality and currently assumes first-person demonstrations with similar visual observation statistics between expert and agent settings.
Abstract
from arXiv · showhide
The current dominant paradigm for imitation learning relies on strong supervision of expert actions to learn both 'what' and 'how' to imitate. We pursue an alternative paradigm wherein an agent first explores the world without any expert supervision and then distills its experience into a goal-conditioned skill policy with a novel forward consistency loss. In our framework, the role of the expert is only to communicate the goals (i.e., what to imitate) during inference. The learned policy is then employed to mimic the expert (i.e., how to imitate) after seeing just a sequence of images demonstrating the desired task. Our method is 'zero-shot' in the sense that the agent never has access to expert actions during training or for the task demonstration at inference. We evaluate our zero-shot imitator in two real-world settings: complex rope manipulation with a Baxter robot and navigation in previously unseen office environments with a TurtleBot. Through further experiments in VizDoom simulation, we provide evidence that better mechanisms for exploration lead to learning a more capable policy which in turn improves end task performance. Videos, models, and more details are available at https://pathak22.github.io/zeroshot-imitation/
1 INTRODUCTION
The paper replaces action-supervised imitation with self-supervised exploration and visual goal communication, so agents learn how to imitate without expert actions. A goal-conditioned skill policy uses forward consistency to handle multiple valid action trajectories and is evaluated across manipulation and navigation tasks.
- Motivation: Conventional learning from demonstration requires experts to control robots and provide multiple observation-action demonstrations for each new task.This supervision is described as tedious for human experts and task-specific.
- Motivation: Visual demonstrations communicate desired states rather than actions, allowing experts to demonstrate multiple tasks more quickly and easily.The agent must infer the actions needed to achieve the demonstrated goals.
- Motivation: An agent without prior knowledge struggles to imitate from visual demonstrations, while manually defined state variables can be inaccurate and difficult for downstream planners to use robustly.The paper motivates learning useful prior knowledge through self-supervised exploration instead.
- Method: Self-supervised exploration data is relabeled to train a goal-conditioned skill policy that predicts action sequences from current and goal observations.The policy learns goal-directed skills without extrinsic reward or expert supervision.
- Method: Forward consistency addresses multimodal trajectories by favoring actions whose predicted outcomes match the ground-truth next state, rather than requiring exact action matching.A jointly optimized goal recognizer handles the varying number of steps needed to reach different goals.
- Results: The method improves knot-tying accuracy from 36% to 60%, generalizes navigation to unseen offices, and performs better with curiosity-driven than random exploration in VizDoom.Experiments use a Baxter for rope manipulation and a TurtleBot for office navigation.
2 LEARNING TO IMITATE WITHOUT EXPERT SUPERVISION
The method learns a goal-conditioned skill policy from unsupervised exploration, then uses visual goal demonstrations to infer actions without expert action supervision. Forward consistency, feature-space dynamics, multi-step prediction, and goal recognition support imitation across intermediate visual goals.
- 2.1 Learning the Goal-Conditioned Skill Policy (GSP): Exploration data trains a goal-conditioned skill policy that maps current and goal observations to a variable-length action sequence.The policy can use intermediate sub-goals, not only final task goals.
- 2.1 Learning the Goal-Conditioned Skill Policy (GSP): During imitation, the agent sequentially targets demonstration images, repeatedly acting until a goal recognizer indicates proximity before advancing to the next image.This allows variable numbers of actions between consecutive visual goals and limits compounding error across the demonstration.
- 2.2 Forward Consistency Loss: Forward consistency addresses multi-modal trajectories by penalizing differences between predicted and ground-truth next observations rather than requiring identical actions.The loss permits distinct actions when they produce the same next state, while a learned forward model supplies differentiable state predictions.
- 2.2 Forward Consistency Loss: The learned forward model is needed because analytic dynamics are generally unavailable for image-based states, and its parameters are trained alongside the policy objective.The forward model first explains observed transitions; the consistency term then aligns outcomes from predicted and ground-truth actions.
- 2.2 Forward Consistency Loss: The method extends to feature-space dynamics and variable-length multi-step skills, using recurrent memory and applying forward consistency at each trajectory step.The multi-step objective jointly optimizes action prediction and step-wise forward consistency over the trajectory.
- 2.3 Goal Recognizer; 2.4 Ablations and Baselines: A goal recognizer determines when intermediate visual goals are reached, while experiments compare complete GSP against ablations removing action history, forward consistency, or forward-model feature regularization.The comparison includes classical navigation methods and an inverse-model baseline.
3 EXPERIMENTS
Experiments evaluate the zero-shot imitator on rope manipulation and indoor navigation, including unseen goals, sparse visual demonstrations, and novel environments. The forward-consistent GSP outperforms baselines and generalizes beyond training layouts.
- Evaluation settings: The evaluation covers rope manipulation with Baxter, office navigation with TurtleBot, and visual imitation in simulated 3D navigation.The experiments test generalization to unseen environments and new goals while handling visual distractors.
- Rope manipulation: 60% knot-tying success was achieved by the proposed method, compared with 36% for the baseline.The method also significantly outperformed the baseline on constructing the rope into an ‘S’ shape.
- Goal finding: The full GSP reached single-image goals successfully at a much higher rate, although successful runs took longer on average.This trade-off is reported in the Table 1 comparison of methods across different initial and goal image pairs.
- Goal finding: The TurtleBot learned to rotate in place until visual overlap with a single-image goal appeared, then moved toward the target in an unseen office environment.The initial and goal observations were deliberately 20–30 steps apart without overlap, and success required reaching the goal within 200 steps without collision.
- Visual imitation: Sparse landmark demonstrations enabled successful navigation when SIFT matching failed and ORBSLAM2 could not generate a map.The demonstrations included maze and perturbed-loop tasks, with each orientation initialized so the initial demonstration frame was not visible.
- VizDoom and generalization: Curiosity-collected exploration data improved imitation across methods, while forward consistency improved generalization to new maps with novel textures.Feature-space forward consistency further enhanced generalization even when pixel- and feature-space models performed similarly on training environments.
4 RELATED WORK
Related work spans action-supervised imitation learning, visual demonstration methods, forward and inverse dynamics, and goal-conditioned policies. This paper differs by requiring less supervision and emphasizing generalization across tasks during inference.
- Positioning: The paper addresses a less-supervised imitation problem that requires generalization across tasks during inference.Its setting is distinguished from conventional imitation learning by communicating task goals through observations rather than expert actions.
- Imitation learning: Behavioral cloning directly maps states to actions, whereas inverse reinforcement learning recovers a reward function under which demonstrations are optimal or nearly optimal.Inverse reinforcement learning is commonly formulated with state-action information.
- Visual demonstration: Learning from demonstration commonly assumes full expert states and actions, while visual-demonstration methods relax supervision to visual observations alone.Some related methods still require expert supervision during training or demonstrations for learning.
- Visual demonstration: Third-person imitation and imitation-from-observation translate expert observations into agent observations before policy optimization, but require demonstrations during learning.Nair et al. use image sequences for rope manipulation, while other approaches retain training-time expert supervision.
- Forward/inverse dynamics: Prior dynamics methods learn forward models or jointly learn forward and inverse dynamics without optimizing consistency between them.The paper positions its forward consistency loss as an empirical alternative to these approaches.
- Goal conditioning: Goal-conditioned policies parameterize value or policy functions with goals so agents can learn and perform multiple tasks.This work draws goals from experience but uses supervised learning and dynamics modeling rather than reinforcement learning for policy optimization.
5 DISCUSSION
The method imitates visual demonstrations without expert actions, but its capabilities depend on exploration data and first-person, distribution-matched observations. The framework could also accept formal language as a goal-communication channel.
- 5 DISCUSSION: The learned goal-conditioned skill policy is limited by the coverage and structure of self-supervised exploration data.Random navigation exploration rarely connects rooms, so cross-room goals require intermediate sub-goals; more structured exploration could support broader search behaviors.
- 5 DISCUSSION: The approach requires first-person-view demonstrations and assumes similar visual-observation statistics during expert demonstration and agent execution.Changes such as daylight versus evening lighting may reduce performance, motivating robustness to nuisance changes or domain adaptation.
- 5 DISCUSSION: After training the goal-conditioned skill policy, formal language could potentially replace image-based goal communication by mapping instructions into the policy’s feature space.The paper states that its framework makes no image-specific assumptions.
A SUPPLEMENTARY MATERIAL
The supplementary material reports evaluations across multiple environments and tasks, then gives additional experimental setup and hyperparameter details.
- A SUPPLEMENTARY MATERIAL: The proposed approach was evaluated across multiple environments and tasks.
- A SUPPLEMENTARY MATERIAL: The supplementary section provides additional details about the experimental task setup.
- A SUPPLEMENTARY MATERIAL: The supplementary section provides additional hyperparameter details.
A.1 ROPE MANIPUATION
The rope-manipulation experiments reuse self-supervised Baxter interaction data and use an image-based skill-policy architecture that predicts grasp and displacement actions.
- A.1 ROPE MANIPULATION: The Baxter rope-manipulation setup follows Nair et al. (2017) and reuses approximately 60K self-supervised interaction pairs.
- A.1 ROPE MANIPULATION: The base architecture uses pre-trained AlexNet features as input to a skill-policy network.
- A.1 ROPE MANIPULATION: The skill-policy network predicts grasp location, displacement direction, and displacement magnitude.
- A.1 ROPE MANIPULATION: The forward-regularizer baseline jointly regularizes AlexNet features and the skill-policy network with forward-model loss weight 0.1.
A.2 NAVIGATION IN INDOOR OFFICE ENVIRONMENTS
The indoor-navigation experiments use a TurtleBot2 with discrete motion actions and collect training data through automated self-supervised action sampling. The recurrent forward-consistent policy is trained from these exploration data with a pre-trained visual encoder.
- A.2 NAVIGATION IN INDOOR OFFICE ENVIRONMENTS: The TurtleBot2 uses an Orbbec Astra RGB camera and a four-action space: forward, left, right, and stand still.Forward motion is approximately 10 cm, while turns are approximately 14–18 degrees, with variation from velocity control.
- A.2 NAVIGATION IN INDOOR OFFICE ENVIRONMENTS: Automated self-supervised collection samples an action and its repeat count without human supervision.The no-op action has probability 0.05, other actions are equally likely, and no-op repeats are uniformly sampled from {1, 2}.
- A.2 NAVIGATION IN INDOOR OFFICE ENVIRONMENTS: The exploration data train a recurrent forward-consistent goal-conditioned skill policy using an ImageNet-pre-trained ResNet-50 encoder.The forward-consistency model is pretrained, then fine-tuned end-to-end with the policy; its loss weight is 0.1 and optimization uses Adam.
A.3 3D NAVIGATION IN VIZDOOM
The VizDoom navigation evaluation uses self-supervised exploration data and measures how agents follow visual demonstrations across environments. Performance is summarized using landmark progress, completion, efficiency, and robust aggregation across randomized runs.
- Self-Supervised Data Collection: Curiosity and random exploration each collect 1.5 million frames with action repeat 4 for training and validation.A separate map with different textures is reserved for generalization experiments.
- TurtleBot Evaluation: TurtleBot performance is evaluated in maze and loop conditions using the fraction of demonstration landmarks reached and actions taken to reach the farthest landmark.Table 4 reports both landmark progress and action counts.
- VizDoom Evaluation: VizDoom navigation compares the proposed GSP with baseline models using demonstration completion and efficiency over 50 seeds and five human paths per environment type.Table 5 reports means and standard errors for these measures.
- Evaluation Protocol: Each scenario uses five human demonstrations, with every tenth frame provided to the agent as the visual-imitation input.Each human path is evaluated over 50 seeds, yielding 250 runs per environment type.
- Metric Choice: Median accuracy is the main reported statistic because randomized initial orientations create high variance from outliers.Mean completion results are additionally reported in Table 5.
- Implementation Details: Models use 42x42 grayscale images, batch size 64, Adam with learning rate 1e-4, and uniformly sampled landmark slices spanning 5 to 15 action steps.All models share a goal recognizer trained on curiosity data.