PaperlayerView exact revision on arXiv

Source-linked AI summary

An Algorithmic Perspective on Imitation Learning

Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, Jan Peters

arXiv:1811.06711v1cs.ROcs.LG

TL;DR

Imitation learning addresses the challenge of programming autonomous behavior by learning from expert demonstrations. This paper surveys its assumptions, methods, relationships to related fields, and implementation tools, while identifying open challenges and research directions.

  • Problem

    Imitation learning seeks to efficiently acquire autonomous behavior from expert demonstrations when manually programming the behavior is difficult or expensive.

  • Method

    The paper provides a structured survey of imitation-learning assumptions, algorithms, relationships to related fields, and practical implementation tools.

  • Results

    The survey organizes existing imitation-learning methods and identifies future research directions, including approaches for transferring expert skills to robots.

  • Takeaways & Limitations

    The paper offers machine-learning and robotics practitioners a broader framework for understanding and applying imitation-learning methods.

  • Takeaways & Limitations

    Minimizing the learner-to-expert trajectory KL divergence is difficult in practice because the expert policy likelihood cannot be evaluated.

Abstract

from arXiv · show

As robots and other intelligent agents move from simple environments and problems to more complex, unstructured settings, manually programming their behavior has become increasingly challenging and expensive. Often, it is easier for a teacher to demonstrate a desired behavior rather than attempt to manually engineer it. This process of learning from demonstrations, and the study of algorithms to do so, is called imitation learning. This work provides an introduction to imitation learning. It covers the underlying assumptions, approaches, and how they relate; the rich set of algorithms developed to tackle the problem; and advice on effective tools and implementation. We intend this paper to serve two audiences. First, we want to familiarize machine learning experts with the challenges of imitation learning, particularly those arising in robotics, and the interesting theoretical and practical distinctions between it and more familiar frameworks like statistical supervised learning theory and reinforcement learning. Second, we want to give roboticists and experts in applied artificial intelligence a broader appreciation for the frameworks and tools available for imitation learning.

5 Challenges in Imitation Learning for Robotics

This section contrasts behavioral cloning with inverse reinforcement learning and surveys open questions in imitation learning, including demonstrated data, design choices, algorithms, and performance evaluation.

  • 5.1 Behavioral Cloning vs Inverse Reinforcement Learning: The section first examines behavioral cloning versus inverse reinforcement learning.
  • 5.2.1 Problems Related to Demonstrated Data: It then considers problems related to demonstrated data.
  • 5.2 Open Questions in Imitation Learning: It addresses open questions related to design choices and problems related to algorithms.
  • 5.2.4 Performance Evaluation: It concludes with performance evaluation.

Introduction

Imitation learning transfers desired behavior from expert demonstrations to autonomous systems, often enabling intuitive programming when manual engineering is difficult. This survey organizes imitation learning from an algorithmic perspective, covering its methods, design choices, and relationships to other fields.

  • Purpose and scope: Imitation learning learns desired behavior by recording expert demonstrations and training a policy to reproduce the demonstrated behavior.Experts are often human operators, while learners are robotic systems.
  • Motivation: Human demonstrations provide an efficient and intuitive way to teach robots tasks, including applications where robots work alongside people.Such applications include manufacturing, elder care, service work, collaboration with human operators, and reducing caregivers’ physical workload.
  • Historical development: Imitation learning has progressed from higher-level task teaching toward trajectory-level planning and learning robot motions from demonstrations.Earlier work focused on tasks such as “pick,” “move,” and “place,” while later work increasingly addressed trajectories.
  • Design choices: Designing an imitation learning system requires decisions about who demonstrates, how demonstrations are recorded, what behavior is extracted, how policies are represented, and how policies are learned.These choices involve issues such as multiple experts, recording methods, feature selection, motion segmentation, policy representation, and algorithm selection.
  • Survey contribution: The survey provides an algorithmic overview of existing methods and examines the design choices needed to formulate imitation problems and develop efficient algorithms.It aims to support both readers beginning imitation learning practice and readers seeking deeper theoretical understanding.

Design of Imitation Learning Algorithms

Designing an imitation learning algorithm requires choosing how the problem is formalized, including reward access, behavior description, system dynamics, similarity, features, and policy representation. Information-theoretic analysis further connects maximum-likelihood, maximum-entropy, and projection-based views of imitation learning.

  • Reward function: Reward access separates imitation learning from reinforcement learning, especially when rewards are difficult to define for tasks such as driving or knot-tying.When rewards are available, expert demonstrations can reduce the need for expensive global exploration and may speed learning, with theoretically potentially exponential improvement.
  • Representation and modeling: Other design choices concern whether to learn system dynamics, how to measure policy similarity, which features to use, and how to represent the policy and task abstraction level.Dynamics may be needed for under-actuated robot motion planning but are unnecessary when a sufficiently capable controller is available; similarity is often defined over jointly generated trajectories.
  • Design interactions: These design choices are interdependent and can be made in flexible order, such as choosing policy similarity jointly with policy representation.Features and representations must be informative enough to express the desired behavior while limiting learning complexity.
  • Behavior description: The desired behavior may be represented directly by behavioral cloning or indirectly through inverse reinforcement learning, depending on whether direct mappings or long-horizon planning are more parsimonious.Direct state or context-to-action/trajectory mappings can be learned from demonstrated state-action trajectories.
  • Information-theoretic perspective: Maximum likelihood in an exponential family is equivalent to finding the maximum-entropy distribution under average feature constraints, and M-projection gives the same solution.With bounded feature-matching accuracy, the maximum-entropy formulation also yields a regularized likelihood equivalent to a maximum-a-posterior estimate over dual parameters.

Behavioral Cloning … 3.2. Design Choices for Behavioral Cloning

Behavioral cloning learns a direct policy mapping from contexts or states to trajectories or control inputs using expert demonstrations, framing imitation as supervised regression. Its design centers on jointly choosing a surrogate loss and an expressive yet efficiently trainable regression method.

  • Behavioral Cloning: Behavioral cloning directly maps states or contexts to trajectories or actions without recovering a reward function, efficiently reproducing behavior when that representation is parsimonious.The chapter reviews both model-free methods and model-based methods that leverage system dynamics.
  • 3.1 Problem Statement: In robotic imitation learning, upper-level controllers plan desired trajectories while lower-level controllers generate control inputs to follow them.The main target is to learn these hierarchical controllers.
  • 3.1 Problem Statement: Expert demonstrations provide trajectory-context or control-state datasets from which policies learn direct mappings from context to trajectory or state to control input.This learning problem is formulated as supervised learning and can be solved as regression.
  • 3.1 Problem Statement: Behavioral cloning records demonstrations, selects a policy representation πθ and objective L, then optimizes L with respect to θ.The objective represents similarity between demonstrated behaviors and the learner’s policy.
  • 3.2 Design Choices for Behavioral Cloning: BC design requires a surrogate loss that quantifies demonstrated-versus-produced behavior differences and strongly influences efficient policy training.The regression method should be sufficiently expressive for the desired behavior but simple enough for efficient training.
  • 3.2.1.1 Quadratic Loss Function: Quadratic loss, also called ℓ2-loss or least-squares regression, is equivalent to maximizing expected log likelihood under a Gaussian distribution assumption.Weighted quadratic loss can use a known weight, including the covariance-based Mahalanobis distance.
  • 3.2.1.1 Quadratic Loss Function: DMP and ProMP learn trajectory representations by minimizing quadratic loss functions.This connects the quadratic-loss formulation to established behavioral-cloning trajectory methods.

3.2. Design Choices for Behavioral Cloning · 3.3. Model-Free and Model-Based Behavioral Cloning Methods

This section presents behavioral-cloning design choices for losses and regression models, emphasizing robustness, probabilistic objectives, classification margins, distributional differences, and model complexity. It also notes how these choices support model-free behavioral-cloning methods.

  • 3.2. Design Choices for Behavioral Cloning: ℓ1-loss minimization is more robust to outliers than ℓ2-loss minimization because it yields the median rather than the mean of training samples.A few large outliers can significantly influence the mean while leaving the median largely unaffected.
  • 3.2.1.3 Log Loss Function: The log loss is also called cross-entropy loss because the two loss functions are equivalent.The passage identifies log loss as equivalent to cross entropy.
  • 3.2.1.3 Log Loss Function: In binary imitation-learning classification, minimizing log loss is equivalent to maximizing the expected log likelihood under a Bernoulli distribution.Classification can learn a discrete control policy from expert demonstrations.
  • 3.2.1.4 Hinge Loss Function: Hinge loss assigns zero cost to correct classifications and a linear cost to wrong classifications, with a hinge at x1x2 = 1.Although discontinuous at the hinge, the convex loss can be optimized efficiently in practice.
  • 3.2.1.5 Kullback-Leibler Divergence: KL divergence quantifies differences between probability distributions and is useful when learning stochastic policies.Its application is motivated by comparing policy distributions rather than only point predictions.
  • 3.3. Model-Free and Model-Based Behavioral Cloning Methods: KL divergence is not symmetric, so DKL (p(x)||q(x)) = DKL (q(x)||p(x)) does not generally hold.Behavioral-cloning methods such as use KL divergence as the loss function.
  • 3.2.2 Choice of Regression Methods for Behavioral Cloning: Behavioral cloning requires selecting a regression method with appropriate model complexity for the task.The literature includes regression methods listed in Table 3.1, and model complexity must be chosen appropriately.
  • 3.2.2 Choice of Regression Methods for Behavioral Cloning: Linear-regression models are easy to train but may be insufficiently informative, whereas neural networks can represent highly nonlinear mappings but require substantial training resources.The passage contrasts simple and complex regression models without specifying a quantitative resource requirement.

3.3 Model-Free and Model-Based Behavioral Cloning Methods

Behavioral cloning methods are categorized as model-free or model-based. Model-free methods are simpler because they avoid learning system dynamics, whereas model-based methods can plan feasible trajectories but typically require difficult, time-consuming iterative learning.

  • Behavioral cloning methods are categorized into model-free and model-based approaches.
  • Model-free methods learn policies reproducing expert behavior without estimating system dynamics or recovering the reward function.They often avoid iterative learning and are relatively simple to implement.
  • Model-free trajectory learning does not ensure that the resulting trajectory is feasible in a given system.
  • Model-based methods use system-dynamics information to plan feasible trajectories close to expert behavior, including for underactuated robots.Learning a forward model is non-trivial, and these methods often require time-consuming iterative learning.

3.4 Model-Free Behavioral Cloning Methods in Action-State space

Model-free behavioral cloning learns policies directly from demonstrations, but naive supervised learning can fail when learners encounter states absent from the demonstrations. Neural-network methods have achieved strong results in several applications, while DAGGER addresses distribution mismatch by collecting demonstrations on learner-induced states.

  • Distribution mismatch: Naive supervised imitation learning can fail when the learner encounters states whose distribution differs from that of the demonstrations.Robust driving requires examples of recovery after mistakes, not only accurate driving.
  • Model-free behavioral cloning: Model-free methods directly learn a policy from data without learning a dynamics model or iterating between trajectory and behavior generation.They are difficult to apply to underactuated systems because, without a model, predicting desired behavior is hard.
  • Neural-network methods: Neural-network imitation learning has produced impressive results in Go, handwriting, natural-language generation, and image captioning.Supervised neural networks can also serve as building blocks for learning policies or cost functions in inverse reinforcement learning.
  • Neural-network methods: In Go, demonstrations cannot cover all possible states, so supervised training must generalize from limited data.The board has more than 10^170 possible configurations, yet imitation learning learned a competitive policy and self-play improved it.
  • Distribution mismatch: DAGGER collects expert demonstrations on states encountered by the learner, reducing state-distribution mismatch and the training data needed for satisfactory performance.The approach often achieves much better performance even asymptotically.

3.5 Model-Free Behavioral Cloning for Learning Trajectories

Model-free behavioral cloning learns task trajectories directly from demonstrations, typically by mapping contexts to trajectories without explicitly estimating system dynamics. The section emphasizes choosing parsimonious trajectory representations and enforcing physical feasibility, while reviewing discrete, dynamical-system, and probabilistic representations.

  • Trajectory learning: Model-free behavioral cloning can learn trajectories without explicitly estimating system dynamics when low-level controllers achieve desired states.This setting is particularly applicable to nearly fully actuated robotic manipulators with available low-level controllers.
  • Trajectory learning: A supervised policy maps contexts s ∈ S directly to trajectories τ ∈ T, with trajectory representation determining the parameterized space for learning.The representation should be as parsimonious as the application permits.
  • Trajectory learning: Planned trajectories must be physically feasible, so policies may need explicit constraints such as smooth convergence to the goal state.Naive regression may not adequately satisfy these requirements.
  • Discrete-state representations: Hidden Markov models represent trajectories as probabilistic transitions among discrete latent states and support motion recognition, clustering, and segmentation.Their main drawback is discreteness, motivating combinations with continuous Gaussian models and hidden semi-Markov models.
  • Dynamical-system representations: Dynamic Movement Primitives require separate primitives for each dimension in multidimensional motions, while trajectory adaptation can perform Hilbert-norm minimization.The cited norm is also used in trajectory optimization algorithms such as CHOMP.
  • Dynamical-system and probabilistic representations: Constrained Gaussian-mixture dynamical systems can be globally asymptotically stable, while probabilistic ProMPs learn correlated movements and tolerate noisy trajectory observations.Interaction ProMPs model correlated movements as correlated weight vectors.

3.6 Model-Free Behavioral Cloning for Task-Level Planning

Model-free behavioral cloning for task-level planning decomposes complex tasks into primitive motions, then learns how to sequence those primitives. Key approaches address trajectory segmentation and clustering, skill-structure or transition modeling, incremental correction, and language annotations.

  • 3.6 Model-Free Behavioral Cloning for Task-Level Planning: Demonstrated trajectories must be segmented into primitive motions because they may contain sequences of different motion types rather than a single motion.Manual segmentation and subsequent clustering are time-consuming, motivating automated methods for both operations.
  • 3.6 Model-Free Behavioral Cloning for Task-Level Planning: After segmentation, task-level learners model skill structure and transitions between primitive motions to plan complex behaviors as motion sequences.Approaches include skill trees, movement-primitive graphs, and probabilistic transition models.
  • 3.6 Model-Free Behavioral Cloning for Task-Level Planning: Skill trees merge segmented skill chains by identifying similar skills, while SVM-based graphs classify the next movement primitive from the current one.HMM-based methods instead model probabilistic transitions, with STARHMM conditioning latent-variable distributions on observed state and task phase.
  • 3.6 Model-Free Behavioral Cloning for Task-Level Planning: Incremental imitation learning uses unstructured demonstrations and corrective interactions to segment primitives, construct an FSA, and replan task execution when refinement is needed.Niekum et al. use BP-AR-HMM segmentation and allow human experts to provide corrections during execution.
  • 3.6 Model-Free Behavioral Cloning for Task-Level Planning: Annotated motions can support bidirectional motion-language modeling, mapping motion sequences to sentences and sentences to motion symbols and sequences.The framework represents primitives with HMMs and learns p(λ|y) and p(y|λ) using a motion language model alongside a natural language model.

3.7 Model-Based Behavioral Cloning Methods

Model-based behavioral cloning iteratively learns and uses a forward dynamics model to adjust demonstrated trajectories and plan feasible learner behavior. Methods have progressed from regression-based models such as locally weighted and Gaussian mixture regression to Gaussian Processes, while iterative control learning can also avoid a forward model.

  • Forward dynamics and correspondence: Model-based BC requires iterative learning with access to a forward dynamics model.The learned model supports trajectory adjustment and planning for the learner’s system.
  • Forward dynamics and correspondence: The correspondence problem arises when expert and learner embodiments differ, or when demonstrated velocities are infeasible under the learner’s dynamics.Planned trajectories must therefore be adjusted to satisfy the learner’s kinematic and dynamic constraints.
  • Regression methods: Forward dynamics learning is framed as regression, progressing from locally weighted regression to Gaussian mixture regression and more recently Gaussian Processes.Gaussian Processes are useful because they can incorporate input uncertainty and approximately propagate uncertainty.
  • Gaussian-process methods: Gaussian-process model-based imitation learning iteratively updates a probabilistic forward model and minimizes KL divergence between expert and learned trajectory distributions.Analytically computed gradients enable gradient descent, while the method matches the first and second trajectory-distribution moments.
  • Iterative control learning: Iterative control learning can reproduce a desired trajectory without a forward dynamics model by repeatedly executing a controller and updating the target trajectory.The approach may use an LQR controller, but it is not limited to a specific controller.

3.8. Robot Applications with Model-Free BC Methods

Under a Gaussian trajectory-distribution assumption, matching feature expectations can be interpreted as an M-projection onto the manifold of maximum-entropy distributions.

  • 3.8. Robot Applications with Model-Free BC Methods: The trajectory distribution is assumed to be Gaussian.This assumption underlies the subsequent geometric interpretation.
  • 3.8. Robot Applications with Model-Free BC Methods: Gaussian distributions are identified as maximum-entropy distributions.Park and Bera [2009] are cited for this characterization.
  • 3.8. Robot Applications with Model-Free BC Methods: Matching feature expectations under the Gaussian assumption corresponds to an M-projection onto the maximum-entropy distribution manifold.The interpretation follows the formulation in (3.78) and the discussion in §2.7.1.

3.8 Robot Applications with Model-Free BC Methods

Model-free behavioral cloning has been successfully applied to diverse robotic tasks, including helicopter flight, ball-hitting, rhythmic ball-paddling, and human-robot handovers. These applications use demonstrations with methods such as DMPs and ProMPs to reproduce task-relevant motions.

  • Robot Applications with Model-Free Behavioral Cloning Methods: Model-free behavioral cloning has been successfully used in autonomous RC helicopter flight, ball-hitting tasks, and robotic surgery.The section presents these examples to demonstrate the capability of model-free BC methods in robotic applications.
  • Point-to-Point Motion: A tennis swing was demonstrated with a motion-capture suit measuring 35 human-body DoFs at 100Hz and reproduced on a 30-DoF humanoid robot using DMPs.The recorded joint-angle trajectories were used to learn and reproduce the point-to-point tennis motion.
  • Rhythmic Motion: Kober and Peters learned periodic ball-paddling motions from kinesthetic demonstrations with a seven-DoF Barrett WAM arm using rhythmic DMPs.The task required repeatedly bouncing the ball, and ten basis functions per motor primitive represented the task.
  • Human-Robot Collaboration: ProMPs modeled coupled human-robot motion for a KUKA LWR handover in which the robot grasped and gave a plate or screw to a human operator.The motion was demonstrated using kinesthetic teaching and addressed coordination between the two agents.

3.8. Robot Applications with Model-Free BC Methods

Model-free behavioral-cloning methods support adaptive human-robot collaboration and complex robotic manipulation by learning movement distributions from demonstrations. Applications include interaction ProMPs for conditional motion planning and surgical knot-tying trajectories updated in real time across changing contexts.

  • Collaborative motion learning: Interaction ProMPs learned correlations between human and robot movements, enabling robot motion to be planned conditionally on the observed human movement.Maeda et al. applied interaction ProMPs to several human-robot collaborative tasks.
  • Movement primitive learning: Probabilistic trajectory segmentation enabled learning sequences of movement primitives represented by ProMPs for table tennis, writing, and chair assembly.
  • Surgical manipulation: Behavioral cloning addressed surgical knot-tying, where the looping trajectory’s topology matters more than its exact start and goal positions.The task was demonstrated under varied contexts using a bimanual teleoperated master-slave surgical system.
  • Surgical manipulation: The surgical system performed knot-tying and soft-tissue-cutting tasks, with trajectories updated in real time as execution contexts changed.Demonstrations were collected 9-20 times under various contexts.

3.9. Robot Applications with Model-Based BC Methods

Model-based behavioral cloning was applied to learning autonomous helicopter maneuvers from expert demonstrations, including acrobatic flights under highly nonlinear dynamics. An iterative LQR controller executed the learned trajectories.

  • 3.9. Robot Applications with Model-Based BC Methods: Autonomous helicopter acrobatic maneuvers were learned from expert demonstrations in a system with highly nonlinear dynamics.The learned trajectory was executed using an iterative LQR controller.

3.9 Robot Applications with Model-Based Behavioral Cloning Methods

This section presents model-based behavioral cloning applications for nonlinear robotic control, cross-embodiment learning, and planning in action-state space. Examples include transferring expert helicopter aerobatics and learning ball hitting with an underactuated manipulator.

  • Applications: Model-based behavioral cloning methods are applied to nonlinear robotic systems, learning from different embodiments, and planning in action-state space.The section introduces these application areas before discussing specific examples.
  • Acrobatic helicopter flight: Abbeel et al. [2010] learned acrobatic RC helicopter flight from expert demonstrations despite the system’s nonlinear dynamics.The method normalized demonstrated trajectories’ temporal alignment using an Expectation Maximization-like approach.
  • Acrobatic helicopter flight: The helicopter controller reproduced in-place flips, rolls, loops, hurricanes, auto-rotation landings, chaos, and tic-toc maneuvers previously limited to exceptional experts.Expert skills were transferred by combining model-based behavioral cloning with iterative controller learning.
  • Underactuated manipulation: Englert et al. [2013] learned ball hitting with an underactuated robot by learning trajectories and a controller from kinesthetic demonstrations.Underactuation makes learning challenging because feasible trajectories are limited; the work used a forward model with PILCO to train a policy reproducing demonstrated trajectories.

3.9. Robot Applications with Model-Based BC Methods

Model-based behavioral cloning can learn robot-specific controllers that are robust to the correspondence problem by exploiting learned forward dynamics, while DAGGER addresses learner-induced state distributions through online expert queries. These methods have been applied to video-game control and autonomous UAV flight, including in real forest environments.

  • DAGGER Applications: DAGGER complements initial demonstrations by querying an expert online for states induced by the learner’s policy.This targets states that may be absent from the original expert demonstrations.
  • DAGGER Applications: DAGGER was also applied to autonomous UAV control, where the UAV flew autonomously in real forest environments.A small early error can lead the learner to an unseen state that deviates substantially from expert demonstrations, making online learning essential.
  • Model-Based Behavioral Cloning: Model-based behavioral cloning learns robot-specific controllers that are robust to the correspondence problem compared with model-free behavioral cloning.It can address underactuated robots with unknown nonlinear dynamics by exploiting a learned forward dynamics model.
  • DAGGER Applications: DAGGER learned video-game steering from visual features of 2D images using a policy linear to those features and human demonstrations.A human expert demonstrated the correct steering for observed game images.

Inverse Reinforcement Learning

Inverse reinforcement learning recovers a reward function underlying an expert’s policy or demonstrations, but the task is ill-posed because multiple rewards can make the same policy optimal. Its methods address this ambiguity through objectives and iterative policy–reward optimization, with model-based, model-free, causal-entropy, adversarial, and active variants.

  • Definition: IRL seeks to recover the reward function an agent optimizes from behavioral measurements, sensory inputs, and a model of the physical environment.The recovered reward can provide a parsimonious description of desired behavior.
  • Identifiability: Because one policy can be optimal for multiple reward functions, IRL is ill-posed and requires additional objectives, including margin-based or maximum-entropy criteria.Maximum entropy became a popular objective, while maximizing the margin was prominent in earlier work.
  • Algorithms: Many IRL algorithms alternate between evaluating state-action visitation frequencies, updating a reward estimate through feature matching, and optimizing policy parameters with reinforcement learning.The reward objective makes demonstrations appear more optimal than the current policy before policy parameters are updated.
  • Algorithms: Model-based IRL uses known system dynamics to evaluate visitation frequencies, whereas model-free IRL commonly uses sampling-based methods.Different reinforcement learning and policy-search methods can optimize the policy under the recovered reward.
  • Entropy-based IRL: Maximum causal entropy makes action choices independent of future trajectory states, allowing IRL to incorporate stochastic dynamics during reward estimation.This approach addresses theoretical drawbacks of maximum-entropy IRL under stochastic dynamics.
  • Modern and active methods: GAIL constrains agent behavior to be approximately optimal under an unknown reward without explicitly recovering that reward, while active IRL models demonstration gathering as a POMDP.In active IRL, the hidden reward guides actions that gather information as well as accomplish other tasks.

Robotics

The robotics section frames behavioral cloning (BC) versus inverse reinforcement learning (IRL) as the central method-selection question, with the appropriate choice depending on the task setting and desired representation. It also identifies unresolved challenges in demonstrations, policy comparison, guarantees, and high-dimensional robotic motion.

  • BC vs IRL: BC and IRL are the two major imitation-learning classes, and choosing between them is the first question when applying imitation learning.
  • BC vs IRL: IRL is useful when imitation requires inferring expert intent, while reward functions can provide a more parsimonious description that supports generalization.Visual imitation without kinematic information may require recovering intent; recovered footstep-stability rewards can adapt planning across terrains, unlike merely learning footstep distributions.
  • Open questions: The section highlights open data challenges involving multiple experts, undesirable demonstrated motions, vision-only kinematics, third-person viewpoints, and reuse of related-task demonstrations.Methods generally work better with one expert, while detecting unnecessary motion, learning without embodiment information, third-person imitation, and leveraging prior demonstrations remain incompletely addressed.
  • Open questions: Selecting policy similarity measures remains unresolved, with alternatives including KL divergence, Euclidean distance, and Wasserstein divergence.Wasserstein divergence has improved GAN performance and inspired recent imitation-learning approaches.
  • Open questions: Many imitation-learning methods lack performance guarantees, although DMP stability and DAGGER’s low-error proof provide existing examples, while high-dimensional motion remains inefficient for differing embodiments.Robotics especially needs guarantees such as stability or convergence; humanoid robots often have over 50 joints, and high-dimensional input methods remain often limited to 2D images.
Loading 1811.06711v1…