PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-Embedded Control Networks for Few-Shot Imitation Learning

Stephen James, Michael Bloesch, Andrew J. Davison

arXiv:1810.03237v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Robotic learning often struggles to reuse knowledge across tasks, while existing meta-learning approaches can scale poorly and forget learned tasks. TecNets learn compact task embeddings that condition control from one or more demonstrations, surpass MIL with visual information, and transfer simulation-trained few-shot ability to real-world one-shot learning. The method is expected to perform poorly when new tasks differ drastically from its training domain.

  • Problem

    Robot learning approaches often focus on single tasks and provide limited knowledge reuse, while related meta-learning methods face scalability and forgetting limitations.

  • Method

    TecNets jointly learn a compact task embedding and a control network that uses demonstrations and current observations to predict actions for new task variations.

  • Results

    TecNets surpass MIL in two experimental domains using only visual information, and achieve 72.97% one-shot success in real-world placing after simulation training.

  • Takeaways & Limitations

    Few-shot learning can be trained in simulation, transferred to the real world, and used there to learn new tasks from a single demonstration.

  • Takeaways & Limitations

    TecNets are expected to perform poorly when new tasks are drastically different from the training domain.

Abstract

from arXiv · show

Much like humans, robots should have the ability to leverage knowledge from previously learned tasks in order to learn new tasks quickly in new and unfamiliar environments. Despite this, most robot learning approaches have focused on learning a single task, from scratch, with a limited notion of generalisation, and no way of leveraging the knowledge to learn other tasks more efficiently. One possible solution is meta-learning, but many of the related approaches are limited in their ability to scale to a large number of tasks and to learn further tasks without forgetting previously learned ones. With this in mind, we introduce Task-Embedded Control Networks, which employ ideas from metric learning in order to create a task embedding that can be used by a robot to learn new tasks from one or more demonstrations. In the area of visually-guided manipulation, we present simulation results in which we surpass the performance of a state-of-the-art method when using only visual information from each demonstration. Additionally, we demonstrate that our approach can also be used in conjunction with domain randomisation to train our few-shot learning ability in simulation and then deploy in the real world without any additional training. Once deployed, the robot can learn new tasks from a single real-world demonstration.

1 Introduction

Robotic few-shot learning aims to reuse prior task knowledge while avoiding the scalability and forgetting problems of gradient-based meta-learning. TecNets address this with task embeddings and demonstrate visual few-shot learning, simulation-to-real transfer, and higher success rates than MIL.

  • 1 Introduction: TecNets address limitations of MAML-based meta-imitation learning, whose adapted weights can forget prior tasks and raise scalability concerns when storing task-specific copies.
  • 1 Introduction: The task-embedding space places visually similar tasks near one another and different tasks far apart, supporting few-shot learning and potentially zero-shot similarity inference.
  • 1 Introduction: TecNets learn a compact task embedding that conditions a control network on visual demonstrations and current observations to produce actions for new task variations.The embedding and control networks are jointly trained, and the approach accepts single or multiple demonstrations.
  • 1 Introduction: The approach trains few-shot learning in simulation and transfers it to the real world without additional training, where a robot learns new tasks from a single demonstration.
  • 1 Introduction: TecNets achieve higher success rates than MIL when using only visual information from demonstrations across visually guided manipulation tasks.

2 Related Work

This work combines behavioural cloning and meta-learning to address data-intensive, independently learned robotic tasks. It builds on metric-learning approaches that represent tasks or skills in embedding spaces, while contrasting with MIL's task-specific weight-retention burden.

  • 2 Related Work: The paper focuses on behavioural cloning from pixels, learning a direct mapping from observations to motor-control actions given demonstrations.
  • 2 Related Work: Behavioural cloning commonly requires substantial data and often learns tasks independently, limiting reuse of knowledge across tasks.
  • 2 Related Work: Meta-learning evaluates adaptation to new tasks from few examples, while MIL combines MAML with behaviour cloning for end-to-end learning from visual demonstrations.
  • 2 Related Work: Metric-learning methods such as Matching Networks and Prototypical Networks use learned embedding spaces to compare examples and support classification from limited data.
  • 2 Related Work: Prior skill-embedding work accelerated training but did not target few-shot learning, and related multimodal embedding methods required component pre-training.
  • 2 Related Work: Compared with MIL's need for task-specific weight copies to retain learned information, TecNets store a compact sentence for each task.

3 Task-Embedded Control Networks

Task-Embedded Control Networks learn a task sentence from demonstrations and use it with current observations to produce actions for unseen task variations. The method combines metric-based task embedding with jointly trained control, using support examples to describe tasks and query examples to evaluate performance.

  • Task-Embedded Control Networks: TecNets learn a universal policy π(o, s) that uses a learned task sentence s to emulate an expert policy for task T.The task is represented by a set of example trajectories, and the sentence modulates the policy's behavior.
  • Task Embedding: A task sentence is formed by embedding support examples and taking their normalized mean, while query examples provide task-specific example vectors for training.The support and query sets are disjoint, and other tasks in the batch supply negative comparisons.
  • Task Embedding: The embedding loss trains each example vector to have higher dot-product similarity with its own task sentence than with sentences from other tasks.The hinge rank loss enforces a margin between the positive and negative task comparisons.
  • Control: The task-embedding and control networks are trained jointly, allowing the control loss to shape an embedding that is meaningful for selecting actions.The final objective combines embedding and control losses, with an auxiliary support-set control objective used to ease learning.
  • Control: At test time, demonstrations are converted into a sentence, which is combined with observations to generate actions iteratively until the task is complete.The sentence is tiled and concatenated channel-wise with the control network's visual input, while the environment supplies observations during execution.
  • Experimental Domains: The experiments evaluate one-shot task description and modified-task control across simulated reaching, simulated pushing, and real-world placing domains.Each domain uses support examples to describe the task and query examples to test performance on a modified version.

4 Experiments

The experiments evaluate TecNets across simulated reaching and pushing and real-world placing, including ablations, embedding-size sensitivity, and sim-to-real transfer. TecNets outperform MIL in the simulated domains and achieve one-shot real-world placing after simulation training.

  • 4 Experiments: The evaluation covers simulated reaching and pushing before testing placing on a real robot, with fair comparison enabled by a control architecture similar to MIL.The experiments address embedding quality, ablations, few-shot performance, and sim-to-real transfer.
  • 4.1 Simulated Reaching: TecNets outperformed MIL by a large margin in simulated reaching, while removing the embedding loss reduced success by over 70%.The support loss also substantially assisted task learning in this domain.
  • 4.1 Simulated Reaching: Increasing embedding dimensionality can hurt performance by encouraging trivial embeddings, so capacity must be balanced against overfitting risk.The authors report that useful performance persists across a wide range of embedding sizes, despite the additional hyperparameter.
  • 4.2 Simulated Pushing: TecNets surpassed MIL in both 1-shot and 5-shot simulated pushing when using visual demonstrations alone.The pushing task involved 7-DoF torque control and varied object and environment properties.

5 Conclusion

TecNets provide few-shot imitation learning through task embeddings and can continue learning tasks without forgetting earlier ones. Their transfer is limited when new tasks differ drastically from the training domain.

  • 5 Conclusion: TecNets surpass MIL in two visually guided few-shot imitation domains while retaining few-shot learning during continual task acquisition.The method uses only visual information and can learn new tasks from single or multiple demonstrations.
  • 5 Conclusion: TecNets support continual learning of new tasks without forgetting old tasks or losing their few-shot ability.After simulation training and real-world deployment, the robot can continue learning from single or multiple demonstrations.
  • 5 Conclusion: A TecNet is expected to perform poorly when the new task is drastically different from its training domain, such as pushing after training on placing.The authors suggest that broader task training sets may enable broader generalization, but identify this as future work.

A Experimental Details

The experiments use convolutional task-embedding and control networks with layer normalization and ELU activations, optimized with Adam.

  • A Experimental Details: Both networks use CNNs with layer normalization and ELU activations after each layer, except for linear final layers.The task-embedding and control networks are jointly specified with these activation and normalization choices.
  • A Experimental Details: Adam optimization uses a learning rate of 5 × 10−4.The passage also states that lambda values were set, but does not provide their complete specification.
  • A Experimental Details: The appendix provides additional experiment details, network architecture, and hyperparameters.

A.1 Simulated Reaching

The simulated reaching setup combines visual observations with robot proprioception in CNN-based embedding and control networks that output joint torques.

  • A.1 Simulated Reaching: The reaching networks process 80 × 64 RGB images together with arm joint angles and end-effector position.Proprioceptive features are concatenated with CNN features in the control network before the fully connected layers.
  • A.1 Simulated Reaching: The reaching CNN uses three strided convolution layers with 40 3 × 3 filters, followed by four fully connected layers of 200 neurons.
  • A.1 Simulated Reaching: The reaching task-embedding network outputs a vector of length 20, while the control network outputs torques for the arm’s two joints.

A.2 Simulated Pushing

The simulated pushing setup uses visual and proprioceptive inputs with a deeper CNN architecture for task embedding and control.

  • A.2 Simulated Pushing: The pushing networks process 125 × 125 RGB images together with joint angles, joint velocities, and end-effector pose.Proprioceptive data are concatenated with control-network CNN features before the fully connected layers.
  • A.2 Simulated Pushing: The pushing CNN uses four strided convolution layers with 16 5 × 5 filters, followed by three fully connected layers of 200 neurons.
  • A.2 Simulated Pushing: The passage specifies that the pushing task-embedding network produces a vector, but its full output length is not stated.

A.3 Real-world Placing

The real-world placing control network used visual and proprioceptive inputs, with the task sentence integrated into the control architecture. A U-Net variant performed comparably to input-layer concatenation.

  • A.3 Real-world Placing: The control network received 125 × 125 RGB images and joint angles, while the embedding network produced a 20-dimensional task vector.Proprioceptive features were concatenated with CNN features before the fully connected layers.
  • A.3 Real-world Placing: A U-Net control architecture with the sentence concatenated at the bottleneck performed just as well as channel-wise sentence concatenation at the input layer.

B Sim-to-Real Embedding Visualisation

The sim-to-real visualisation examined TecNet embeddings for 164 real-world placing tasks after training entirely in simulation. It shows how individual demonstrations cluster by task and how visually similar tasks remain nearby.

  • B Sim-to-Real Embedding Visualisation: The visualisation used 5 demonstrations from each of 164 real-world tasks, despite the TecNet being trained entirely in simulation without real-world data.The demonstrations were collected through human teleoperation using an HTC Vive controller.
  • B Sim-to-Real Embedding Visualisation: Figure 7’s t-SNE plot places demonstrations from the same task next to one another while positioning other visually similar tasks nearby.It embeds individual demonstration sentences without averaging.
  • B Sim-to-Real Embedding Visualisation: Figure 8 averages the 5 demonstrations for each task into one task sentence, producing a more legible view of relationships between tasks.
Loading 1810.03237v1…