Source-linked AI summary
One-Shot Visual Imitation Learning via Meta-Learning
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, Sergey Levine
TL;DR
Robots need to acquire many complex skills without collecting extensive task-specific supervision. The paper combines gradient-based meta-learning with imitation learning so vision-based policies adapt to new tasks from one demonstration, and demonstrates this on simulated and real robots.
Problem
Generalist robots require complex skills from raw sensory inputs, but learning each skill independently demands substantial task-specific supervision.
Method
The method combines imitation learning with gradient-based meta-learning to train policies that adapt to new tasks from a single visual demonstration.
Results
The method learns new task variants end-to-end from one visual demonstration on simulated domains and a real robotic system.
Takeaways & Limitations
Meta-imitation learning improves robotic learning efficiency while retaining end-to-end training for complex visual inputs.
Takeaways & Limitations
The evaluation uses tasks with limited diversity beyond object diversity, and domain shift between human video and the robot’s view is left for future work.
Abstract
from arXiv · showhide
In order for a robot to be a generalist that can perform a wide range of jobs, it must be able to acquire a wide variety of skills quickly and efficiently in complex unstructured environments. High-capacity models such as deep neural networks can enable a robot to represent complex skills, but learning each skill from scratch then becomes infeasible. In this work, we present a meta-imitation learning method that enables a robot to learn how to learn more efficiently, allowing it to acquire new skills from just a single demonstration. Unlike prior methods for one-shot imitation, our method can scale to raw pixel inputs and requires data from significantly fewer prior tasks for effective learning of new skills. Our experiments on both simulated and real robot platforms demonstrate the ability to learn new tasks, end-to-end, from a single visual demonstration.
1 Introduction
The paper targets generalist robots that can acquire complex skills efficiently by reusing experience across tasks. It introduces meta-imitation learning for adapting vision-based policies to new skills from one demonstration.
- 1 Introduction: Meta-imitation learning reuses prior task experience so robots can learn new skills from a single demonstration.The method learns a parameterized policy that adapts through gradient updates across tasks.
- 1 Introduction: The approach demonstrates one-shot imitation learning from raw pixels across simulated reaching, pushing, and real-robot visual placing tasks.The evaluations include new task variants and settings with limited demonstration information.
- 1 Introduction: The method can use a raw video demonstration without access to the demonstrator’s applied controls.Its parameter-efficient meta-learning requires relatively few demonstrations during meta-learning and supports real robotic systems.
2 Related Work
Prior imitation methods can be data-efficient when environment state is known, but raw-sensory imitation remains difficult because each task requires many demonstrations. This work shares demonstrations across tasks to reduce that burden for new tasks.
- 2 Related Work: Raw-pixel imitation removes the need for predefined vision systems and supports vision-based manipulation when environmental state is unknown.The paper frames this setting as applicable to unknown, dynamic environments.
- 2 Related Work: A central limitation of existing real-world imitation methods is the need for many demonstrations for each task.The related-work discussion distinguishes this issue from compounding errors, which the paper does not address.
- 2 Related Work: The proposed strategy shares demonstrations from many other tasks so a new task can be learned from one demonstration.This extends the broader idea of sharing information across tasks beyond task-to-task mappings, gating, and shared features.
3 Meta-Imitation Learning Problem Formulation
The problem formulation treats tasks as samples and trains a policy to adapt to a held-out task from one expert demonstration. MAML supplies the gradient-based mechanism: optimize initial parameters for effective post-update behavior across tasks.
- 3 Meta-Imitation Learning Problem Formulation: The visual meta-imitation problem requires a vision-based policy to adapt to a new task from a single demonstration.The section introduces the problem before extending prior meta-learning into meta-imitation learning.
- 3.1 Problem Statement: Meta-learning reuses demonstrations from other tasks, treating entire tasks as datapoints while keeping per-task data relatively small.This targets generalist robots that receive limited supervision for each new task.
- 3.1 Problem Statement: Each imitation task contains an expert trajectory of observations and actions plus a loss used to compare predicted and demonstrated actions.The loss may be mean squared error for continuous actions or cross-entropy for discrete actions.
- 3.1 Problem Statement: The learned parameters are returned as an initialization that can be quickly adapted to new tasks through imitation.Meta-testing measures performance on held-out tasks after one-demonstration adaptation.
- 3.1 Problem Statement: MAML samples tasks from p(T), adapts parameters using one demonstration, evaluates the adapted policy on another demonstration, and updates the initial parameters.The algorithm uses step sizes α and β for adaptation and meta-update steps.
- 3.2 Background: Model-Agnostic Meta-Learning: The adaptation rule computes θ′ from θ by taking a gradient step on the task loss.The formulation considers one update for simplicity, while multiple updates are a straightforward extension.
- 3.2 Background: Model-Agnostic Meta-Learning: MAML optimizes θ while computing the objective with updated parameters θ′, so gradient descent on new tasks produces effective behavior.Meta-optimization across sampled tasks uses stochastic gradient descent.
4 Meta-Imitation Learning with MAML
The method extends MAML for imitation learning by adapting a policy to new tasks from one demonstration, and can learn from visual demonstrations without expert actions at test time.
- Meta-Imitation Learning with MAML: MAML is extended so a policy adapts to new imitation tasks through gradient updates using demonstrations.Meta-training uses two demonstrations per task: one for adaptation and one for the meta-objective.
- Meta-Imitation Learning with MAML: One-shot adaptation uses a single demonstration, while multiple demonstrations can resolve ambiguity.At meta-test time, the adapted policy can address new goals or previously unseen objects.
- Two-Head Architecture: Meta-Learning a Loss for Fast Adaptation: A two-head architecture separates pre-update and post-update policy parameters while meta-learning both for effective post-adaptation performance.The post-update head is not updated using the demonstration, but both heads are meta-learned.
- Two-Head Architecture: Meta-Learning a Loss for Fast Adaptation: The two-head design acts as a meta-learned loss, providing more flexibility in adapting policy parameters and enabling one-shot adaptation without expert actions.The paper reports increased performance in some experiments from the added adaptation flexibility.
- Learning to Imitate without Expert Actions: At test time, the method can use a visual demonstration without the demonstrator’s controls, although expert actions remain available during meta-training.The paper leaves domain shift between human video and the robot’s view to future work.
5 Model Architectures for Meta-Imitation Learning
The policy architecture adds a bias transformation to give gradient-based adaptation more control, while Figure 3 illustrates reaching, pushing, and placing task variants across scenes.
- Policy Architecture: The vision policy uses a CNN, and the policy observation includes both camera images and robot configuration.The architecture uses strided convolutions followed by ReLU nonlinearities.
- Bias Transformation: A bias transformation concatenates learned parameters to hidden activations, reparameterizing the next layer’s bias as W2z + b.This separates the transformed bias from direct dependence on the input.
- Bias Transformation: The bias transformation gives gradient updates more direct control over the updated bias by reducing coupling with other network parameters.The construction uses y = W1x + W2z + b and treats z as a meta-learned parameter.
- Bias Transformation: The authors found the bias transformation made gradient-based meta-learning significantly more stable and effective in experiments.The architecture is illustrated in Figure 2.
- Example Tasks: Figure 3 contrasts demonstration scenes with learned-policy roll-outs for reaching, pushing, and placing the same task-relevant object or target in new settings.The examples cover color-conditioned reaching, object pushing toward a goal, and placing an object on a target.
6 Experiments
The experiments evaluate one-shot imitation across simulated reaching and pushing domains and real-world placing, comparing MIL with contextual, LSTM, and random policies under varied demonstration conditions. MIL handles raw visual inputs, adapts to held-out objects, and remains effective when demonstrations omit actions or robot state.
- Experimental domains: The evaluation spans simulated planar reaching, simulated 7-DoF pushing, and real-world visual placing with held-out objects.The experiments compare methods across vision and non-vision settings and real RGB-camera inputs.
- Simulated reaching: MIL handles raw pixel inputs more effectively than contextual and LSTM policies across modest meta-learning dataset sizes.In non-vision reaching, LSTM policies improve, especially with attention, but still perform worse than MIL; bias transformation improves consistency across dataset sizes.
- Simulated pushing: 85.8% one-shot learning success was achieved by MIL on held-out simulated pushing objects using the largest dataset size.MIL also achieves, on average, 6.5% higher success than the LSTM-based approach across dataset sizes.
- Simulated pushing: MIL is less affected than LSTM when test demonstrations omit expert actions or robot state, while all approaches improve when five demonstrations are available.The additional evaluations use the largest dataset size and average predictions or gradients across five demonstrations.
- Real-world placing: MIL achieves 90% success placing held items onto previously unseen target objects from a single visual demonstration, compared with 25% for LSTM and contextual policies.LSTM and contextual policies failed to localize the correct target object; MIL also handled video-only demonstrations with lower success.
7 Discussion and Future Work
The real-robot evaluation used held-out objects and showed that one-shot visual imitation can work with video-only demonstrations. The discussion identifies limited task diversity as an evaluation boundary while pointing to broader demonstrations as a future direction.
- Table 2 evaluates one-shot placement on a real PR2 robot using 29 held-out test objects after meta-training on approximately 100 objects.
- The method learns tasks from visual inputs in a single demonstration and substantially outperforms a prior recurrent-neural-network one-shot imitation method.
- The experimental evaluation uses tasks with limited diversity beyond object diversity, while more diverse meta-training demonstrations are expected to expand the method’s capabilities.
A Additional Experimental Details
This appendix section provides additional experimental details covering data collection, evaluation, and training hyperparameters.
- The appendix includes additional details about data collection.
- The appendix includes additional details about evaluation.
- The appendix includes additional details about training hyperparameters.
A.1 Simulated Reaching
The simulated reaching experiments compare vision and no-vision policies using robot state inputs, with vision policies additionally receiving RGB images. Evaluation uses held-out target-color tasks and noisy expert demonstrations.
- Experimental Setup: Vision policies receive an 80 × 64 RGB image in addition to arm joint angles and end-effector position.
- Experimental Setup: Non-vision policies receive object positions but must infer the target-object index from the demonstration, while policy outputs are joint torques.
- Experimental Setup: Expert demonstrations are generated with iLQG using goal knowledge, and meta-testing evaluates 150 held-out color tasks across 1500 total trials.
A.2 Simulated Pushing
The simulated pushing experiment evaluates image-based policies on object-placement success and compares one- versus two-head architectures. The two-head design is reported to improve performance in this more challenging domain.
- Experimental Setup: Pushing policies receive a 125 × 125 RGB image and robot proprioceptive inputs, with success requiring target alignment for at least 10 timesteps.
- Experimental Setup: The pushing success rates are computed over 74 tasks with 6 trials per task, totaling 444 trials.
- Experimental Setup: The expert policies are trained with TRPO using target and distractor poses rather than vision inputs.
- Architecture Ablation: Using two heads leads to significantly better performance than one head in the simulated pushing domain.
A.3 Real-World Placing
The real-world placing evaluation uses standardized video preprocessing, task-specific demonstrations, and held-out trials to assess whether objects reach target containers.
- Demonstrations use 320 × 240 RGB camera videos downsampled by 2, cropped to 100 × 90, and fixed to a 30-step horizon.
- Training data contain 1293 demonstrations, including 96 validation examples, collected from varied objects, containers, and object positions.
- Evaluation covers 18 tasks with 4 trials each, totaling 72 trials, using one held-out placing target and one holding object per task.
- Success is defined by whether the held object lands in or on the target container after a manually coded open-gripper action.
- The policy combines convolutional processing, spatial softmax, fully connected layers, fixed VGG-19 initialization, and an auxiliary target-position regression loss.