An Algorithmic Perspective on Imitation Learning

1.1 Key successes in Imitation Learning

• ALVINN learned image-to-action driving and crossed North America autonomously; imitation learning later shifted from task-level planning to trajectory-level motion learning.Recent deep-network approaches also learn sequential tasks such as handwriting and initialize policies from demonstrations, often requiring large amounts of data.Source: §1.1 · pp. 11–12

1.2 Imitation Learning from the Point of View of Robotics

• Robotic imitation system design involves choosing who demonstrates, how behavior is recorded, what motions to extract, how to represent the policy, and how to learn it.Recording may use motion capture or teleoperation; recorded behavior may contain redundant information and unnecessary motions that require feature selection and segmentation.Source: §1.2 · p. 13
• The survey emphasizes algorithmic design choices, details existing methods, and an information-theoretic account of how they relate, while acknowledging that its coverage cannot be exhaustive.It aims to support both newcomers implementing methods and readers seeking theoretical understanding.Source: §1.2 · pp. 14, 16

1.3 Key Differences between Imitation Learning and Supervised Learning

• Imitation learning connects to structured prediction but adds policy structure, costly or infeasible resets, learner-dependent inputs, and adaptation when expert and learner embodiments differ.Limited demonstrations can leave learners in states absent from expert data, creating a source–target distribution shift.Source: §1.3 · p. 17, §1.4 · p. 18

1.4 Insights for Machine Learning and Robotics Research

• Robotics offers custom trajectory representations for constraints, while machine learning offers algorithms that may reduce the need for customization; both fields face costly data collection, rollouts, and embodiment gaps.Imitation learning also relates to reinforcement learning: good demonstrations provide prior knowledge, and cited work reports potentially exponential sample-complexity decreases versus trial-and-error learning.Source: §1.4 · pp. 18–19

1.5 Statistical Machine Learning Background

• The background formalizes demonstrations as trajectories and introduces state, context, horizon, Markov chains, and Markov decision processes as notation and concepts for understanding later algorithms.The article distinguishes expert and learner trajectory distributions and uses “action” and “control input” interchangeably.Source: §1.5.1 · pp. 20–21, §1.5.2 · p. 21, §1.5.3 · p. 21
• The background presents entropy, KL divergence, information and moment projections, and maximum entropy as tools for reasoning about distributions and policy learning.KL divergence is asymmetric; for multimodal distributions, M-projection averages modes while I-projection concentrates on one, and maximum entropy yields a log-linear distribution matching feature expectations.Source: §1.5.4 · p. 21, §1.5.5 · p. 22, §1.5.6 · p. 22

1.5.7 The Maximum Entropy Principle

• The Gaussian is a maximum-entropy distribution matching first- and second-order moments, while Maximum Causal Entropy provides a natural uncertainty notion for dynamical systems [Ziebart et al., 2013].Source: §1.5.7 · p. 24

1.5.8 Background: Reinforcement Learning

• Reinforcement learning seeks policies maximizing expected reward, with discounting for infinite horizons; state and action-value functions measure expected returns from states and state-action pairs.Source: §1.5.8 · pp. 24–25

1.6 Formulation of the Imitation Learning Problem

• Imitation learning seeks a policy that reproduces expert behavior from feature trajectories, which may represent robot states or raw-image pixels.The features need not be manually specified.Source: §1.6 · p. 25
• Task context can encode relevant conditions such as initial states or target positions, is typically fixed during execution, and may accompany an optional expert reward signal.Source: §1.6 · p. 25

1.6. Formulation of the Imitation Learning Problem

• A common optimization strategy uses demonstration trajectory-context pairs and optional rewards to align learner- and expert-induced feature distributions, with offline or online learning and context-based multitask settings.Source: §1.6 · p. 26
• A simulator or physical robot can evaluate policies, gather new data through interaction, and support iterative policy improvement toward the demonstrations.Source: §1.6 · p. 26

2.1 Design Choices for Imitation Learning Algorithms

• Reward access distinguishes imitation learning from reinforcement learning, while demonstrations can reduce exploration even when a score exists and some tasks are difficult to reward-design.Methods may initialize a policy from coarse demonstrations and refine it through reinforcement learning, or intermix learner actions with expert advice; with enough samples and representational power, this can recover a policy nearly as good as the expert and prevent errors from cascading.Source: §2.1 · pp. 27–28
• Other design choices include whether to encode behavior directly or through a learned cost, whether dynamics are needed, and how to select similarity measures, features, and policy representations.Features must express the desired behavior without excessive learning complexity, while more complex policy representations can require more data and training time.Source: §2.1 · pp. 29–30

2.2 Behavioral Cloning and Inverse Reinforcement Learning

• Behavioral cloning maps states or contexts directly to actions through supervised learning, whereas inverse reinforcement learning recovers a reward from demonstrations assumed approximately optimal under it.The choice depends on the most parsimonious description of the desired behavior: a valid learned mapping for behavioral cloning or a reward that represents the behavior appropriately for inverse reinforcement learning.Source: §2.2 · p. 31, §2.3 · p. 32

2.3. Model-Free and Model-Based Imitation Learning Methods

• Model-free methods avoid explicitly learning dynamics and suit systems with available controllers, while model-based methods use dynamics for feasible planning but can be difficult when contact is hard to model.Underactuated systems often require dynamics-aware planning, although some recent inverse-reinforcement-learning methods learn skills through iterative rollouts without an explicit dynamics model.Source: §2.3 · pp. 32–34, Table 2.1 · p. 33

2.4.1 Trajectories in Fully Observable Settings; 2.4.2 Trajectories in Partially Observable Settings

• Fully observable demonstrations can provide state-control trajectories, while demonstrations may instead omit expert controls or provide only partial observations of system state.The partial-observation formulation represents observations as y_t = f_o(x_t), with a linear special case y_t = H_t x_t.Source: §2.4.1 · p. 36, §2.4.2 · p. 36

2.4.3 Differences in observability between the expert and the learner

• Expert and learner observability can differ, making demonstrations suboptimal or causing the learner to infer the wrong policy from expert behavior.For example, an expert may route around an obstacle unseen by the learner, leading the learner to imitate that detour even when no obstacle is present.Source: §2.4.3 · pp. 37–38

2.5.1 Levels of Policy Abstraction

• Policy abstraction ranges from task-level option sequences to context-conditioned trajectories and state-to-action policies; trajectory learning is often model-free, while action-state inverse reinforcement learning is often model-based.Task-level abstraction can break complex tasks into simpler movements, while commercially available position/velocity controllers make model-free trajectory planning suitable for many manipulators.Source: §2.5.1 · pp. 38–40

2.5.2 Hierarchical vs Monolithic Policies

• A policy using one abstraction level is monolithic; learning complex tasks with such policies may require complex representations such as neural networks.The passage notes that numerous methods have been developed for monolithic policies.Source: §2.5.2 · pp. 40–41
• Hierarchical policies sequence lower-level primitive behaviors with an upper-level planner, allowing simpler low-level representations but making simultaneous learning of all levels nontrivial.Both behavior cloning and inverse reinforcement learning methods learn hierarchical policies.Source: §2.5.2 · p. 41

2.5.3 Feedback vs Open-Loop/Feedback-Free Policies

• Feedback policies update actions from environmental observations, whereas open-loop policies act from the initial input; one system can use both at different levels.Trajectory planners may also update trajectories online, as in Schulman et al. [2013]; Abbeel et al. [2010] combine open-loop trajectory planning with a feedback controller.Source: §2.5.3 · pp. 41–42

2.5.4 Stationary vs. Non-Stationary Policies

• Stationary policies depend only on current state, while non-stationary policies depend on time; trajectory movements often need the latter, whereas repeated state-based actions suit the former.Examples include matching a robot arm’s demonstrated speed across a trajectory versus steering a racing car back from the road edge regardless of when it occurs.Source: §2.5.4 · p. 42, §2.5.4.1 · p. 43

2.5.4.2 Deterministic Policy

• A deterministic policy selects a unique trajectory or control input for a given initial state or current state and context.Determinism is a special case of stochasticity.Source: §2.5.4 · p. 42, §2.5.4.2 · p. 43

2.5.4.3 Stochastic Policy

• A stochastic policy draws actions from a conditional distribution, modeling expert variability and providing uncertainty useful for policy-parameter exploration.Model-based behavior cloning and inverse reinforcement learning methods also use stochastic policies while learning system dynamics iteratively.Source: §2.5.4.3 · p. 44

2.6 Behavior Descriptors

• Behavior matching can target state-action distributions, expected trajectory features for long-horizon behavior, or trajectory-feature distributions that capture higher moments as well as means.Expected trajectory features reduce noise relative to individual trajectories.Source: §2.6.1 · p. 45, §2.6.2 · p. 45, §2.6.3 · p. 46

2.7.1 Information Theoretic Understanding of Imitation Learning Algorithms for Trajectory Learning

• For trajectory learning, maximum likelihood is an M-projection; with exponential-family policies it is dual to maximum entropy under feature constraints, while bounded constraint accuracy yields regularized likelihood.The regularized objective is equivalent to a maximum a-posteriori estimate with a prior on dual parameters.Source: §2.7.1 · pp. 49–50

2.7.2 Information Theoretic Understanding of Imitation Learning Algorithms in Action-State Space

• In action-state space, M-projection reduces to supervised matching of expert state-action distributions using demonstrations, but may fail; I-projection is intractable, though DAGGER may better fit learner-distributed data.The passages note that no existing imitation-learning method exactly performs I-projection, because evaluating the expert’s action likelihood is difficult.Source: §2.7.2 · pp. 51–52

Behavioral Cloning

• Behavioral cloning directly maps contexts to trajectories or states to actions, and can efficiently reproduce behavior when that mapping is the most parsimonious representation.The demonstrations provide the data for learning this policy as a supervised regression problem.Source: Behavioral Cloning · p. 53, §3.1 · pp. 54–55

3.1 Problem Statement

• Robotic imitation learning targets both an upper-level controller that plans a desired trajectory and a lower-level controller that produces inputs to follow it.The upper-level plan depends on context and/or observations, while the lower-level controller uses feedback.Source: §3.1 · p. 53, Figure 3.1 · p. 54
• For trajectory learning, a policy generates a trajectory for a context; for action-state learning, it generates a control input for a state and/or context from expert demonstrations.The demonstrations are represented as context-trajectory pairs or state-control pairs, and the policy is learned by regression.Source: §3.1 · pp. 54–55

3.2 Design Choices for Behavioral Cloning

• BC design requires selecting a surrogate loss and a regression method that is expressive enough to represent behavior yet simple enough for efficient training.The regression method should be chosen together with the surrogate loss for efficient learning.Source: §3.2 · pp. 55–56

3.2.1.1 Quadratic Loss Function

• The quadratic loss is a common surrogate, and minimizing it is equivalent to maximizing expected log likelihood under a Gaussian assumption.DMP [Schaal et al., 2004, Ijspeert et al., 2013] and ProMP [Paraschos et al., 2013, Maeda et al., 2016] learn trajectory representations by minimizing quadratic loss.Source: §3.2.1.1 · pp. 56–57
• A weighted quadratic loss can be used when an appropriate weight is known; Mahalanobis distance uses a covariance matrix as the weight.The cited literature uses this distance [Rozo et al., 2016, Osa et al., 2017a].Source: §3.2.1.1 · p. 57

3.2.1.2 ℓ1-Loss Function

• The ℓ1-loss is another commonly used regression loss, also known as absolute loss or least absolute deviations regression.The passage identifies minimizing ℓ1-loss with least absolute deviations regression.Source: §3.2.1.2 · p. 57

3.2. Design Choices for Behavioral Cloning

• ℓ1-loss is more robust to outliers than ℓ2-loss and yields a sparse, potentially computationally efficient solution.The passage attributes this robustness to ℓ1 minimizing the median rather than the mean, which can be influenced by a few large outliers.Source: §3.2 · p. 58

3.2.1.3 Log Loss Function

• Log loss is cross-entropy and, in binary classification, minimizing it maximizes expected Bernoulli log likelihood, equivalent to logistic-regression maximum likelihood.Source: §3.2.1.3 · pp. 58–59

3.2.1.4 Hinge Loss Function

• Hinge loss, used for maximum-margin classifiers such as SVMs, costs zero for correct classifications, grows linearly for wrong ones, and is convex.Although it is discontinuous at the hinge location, the passage notes that optimization solutions exist in practice and convex optimizers can optimize it efficiently.Source: §3.2.1.4 · p. 59

3.2.1.5 Kullback-Leibler Divergence

• KL divergence quantifies differences between probability distributions and is useful when learning stochastic policies.Source: §3.2.1.5 · p. 59

3.3. Model-Free and Model-Based Behavioral Cloning Methods

• KL divergence is asymmetric, and behavioral cloning methods such as use it as a loss function.Source: §3.3 · p. 60

3.2.2 Choice of Regression Methods for Behavioral Cloning

• Behavioral cloning model choice involves balancing simple, easy-to-train linear regression against neural networks’ nonlinear capacity and greater data requirements.Bishop [2006] discusses choosing a model with appropriate complexity; imitation learning also cannot be reduced to simple supervised learning in many applications.Source: §3.2.2 · p. 60

Model-Free and Model-Based Behavioral Cloning Methods

• Model-free BC reproduces expert behavior without estimating system dynamics or recovering reward, and is often simpler to implement without iterative learning.For trajectory learning, however, it does not ensure feasibility and is difficult to apply to underactuated systems with limited reachable states.Source: §3.3 · p. 60
• Model-free methods favor simplicity, while model-based methods support feasible trajectories in underactuated systems at the cost of difficult forward-model learning and iteration.The distinction is between reproducing behavior without a dynamics model and using dynamics information to plan.Source: §3.3 · pp. 60–61
• Model-based BC uses system dynamics to learn a policy and can plan feasible trajectories close to the expert’s behavior, including for underactuated robots.Learning a forward model is non-trivial, and model-based methods often require time-consuming iterative learning.Source: §3.3 · p. 61

3.4.1 Model-Free Behavioral Cloning as Supervised Learning

• In ALVINN, Pomerleau [1988] mapped camera images to steering angles, but sequential errors shifted the learner into unseen states; Ross and Bagnell [2010], Ross et al. [2011] address this through interaction.Demonstration states may violate the i.i.d. assumption of supervised learning, and learners need examples of recovery from mistakes.Source: §3.4.1 · pp. 62–63

3.4.2.1 Recent Successes of Imitation Leaning with Neural Networks

• Supervised neural networks can learn policies from expert-generated demonstrations, and recent applications include Go, handwriting, natural language, and image captioning.AlphaGo learned a competitive Go policy from supervised imitation despite the state space being too large for demonstrations to cover, then improved it with reinforcement learning.Source: §3.4.2 · p. 63, §3.4.2.1 · pp. 63–64, Figure 3.2 · p. 64

3.4.2.2 Learning with Recurrent Neural Networks

• Recurrent neural networks learn demonstrated sequences across language, image descriptions, handwriting, and speech, with architectures and regularization adapted to each task.Wen et al. [2015] use skip connections to soften vanishing gradients and dropout to reduce co-adaptation and overfitting; Chung et al. [2015] use hidden random variables to model variability across correlated time steps.Source: §3.4.2.2 · pp. 65–66, Table 3.3 · p. 65

3.4.3 Teacher-Student Interaction during Behavioral Cloning

• Because learner errors can cascade into states absent from demonstrations, practical imitation learning must obtain corrections for relevant learner-induced states rather than cover every possible situation.One approach alternates policy updates with requests for additional demonstrations under the current state distribution [Ross and Bagnell, 2010, Ross et al., 2011, Bagnell, 2015].Source: §3.4.3 · pp. 66–67
• Interactive behavioral cloning addresses state-distribution mismatch by gathering additional expert guidance while the learner acts, rather than relying only on fixed demonstrations.The section also frames imitation learning as structured prediction and describes confidence-triggered corrections and DAGGER’s iterative data aggregation.Source: §3.4.3 · p. 67, §3.4.3.2 · p. 68, §3.4.3.3 · p. 71

3.4.3.1 Reduction of Structured Prediction to Iterative Learning of Simple Classification

• SEARN reduces structured prediction to iterative cost-sensitive classification, while also showing that structured prediction can be reduced to imitation learning [Daumé III et al., 2009].Its classifiers generate new cost-sensitive samples from predictions and iteratively update the policy; physical resets and frequent expert demonstrations can make implementation burdensome.Source: §3.4.3.1 · pp. 67–68

3.4.3.2 Confidence-Based Approach

• Chernova and Veloso [2009] request expert demonstrations when a learner’s confidence in selecting an action primitive falls below a threshold, and add expert corrections of incorrect actions to training.This aims to learn under the state distribution induced by the learner’s policy.Source: §3.4.3.2 · p. 68

3.4.3 Data Aggregation Approach: DAGGER

• DAGGER collects expert labels on learner-induced states, reducing the training data needed for satisfactory performance and often improving asymptotic performance; no-regret learners can provide its guarantees.It mixes expert and learner policies to gather data, aggregates datasets, and updates the policy iteratively.Source: §3.4.3.3 · pp. 70–71

3.5 Model-Free Behavioral Cloning for Learning Trajectories

• With near-full actuation and an available low-level controller, model-free behavioral cloning can learn task trajectories without estimating system dynamics.The paper describes this as a dominant approach to robotic manipulator trajectory planning.Source: §3.5 · p. 72

3.5.1 Trajectory Representation

• Trajectory representation determines the parameterized space for demonstrations, so it should be parsimonious, physically feasible, and suited to the application.Regression can map contexts to trajectories, but DMPs and SEDS explicitly ensure smooth convergence to a goal; dimensionality reduction is another available learning tool.Source: §3.5.1 · pp. 72–73

3.5.1.1 Keyframe/Via-Point Based Approaches

• Keyframe or via-point approaches represent a task trajectory as discrete states, which can be adapted to humanoid motion and music-synchronized dance.Nakaoka et al. [2007] extracted keyframes from tracked expert demonstrations and modified them for humanoid dynamics; Okamoto et al. [2014] learned music-motion correspondence across rhythms.Source: §3.5.1.1 · pp. 73–74

3.5.1.2 Representation with Hidden Markov Models

• HMMs model motion phases and kinematic observations probabilistically, supporting motion recognition, clustering, and segmentation, but their discrete states limit smooth trajectory planning.When state counts are low, recognition can work well, though too few states may fail to reproduce motion; continuous state-specific Gaussian models and HSMMs address some limitations.Source: §3.5.1.2 · pp. 74–75

3.5.1.3 Dynamic Movement Primitives

• DMPs combine nonlinear forcing with attractor dynamics to represent nonlinear motion while maintaining stability, continuity, and smooth convergence to a goal.A phase variable enables speed modulation, and variants extend DMPs to obstacle avoidance, compliant or coupled motion, and generalization across contexts; DMP adaptation is also equivalent to a particular Hilbert-norm minimization [Dragan et al., 2015], as used in CHOMP [Zucker et al., 2013].Source: §3.5.1.3 · pp. 76, 77, 79, 80

3.5.1.4 Probabilistic Movement Primitives

• ProMPs represent stochastic demonstrations as a distribution over trajectories, and conditioning that distribution on via-points enables modulation of positions or velocities.Unlike deterministic DMPs, ProMPs capture variability across demonstrations; temporal modulation uses a phase variable.Source: §3.5.1.4 · pp. 80, 82

3.5.1.5 Trajectory Representation with Time-Invariant Dynamical Systems

• Time-invariant dynamical systems represent motion with a single nonlinear dynamics model and can be globally asymptotically stable at a target under a sufficient condition.SEDS uses a constrained GMM to learn the dynamics; the representation models correlations across multiple degrees of freedom but cannot represent time-varying behavior.Source: §3.5.1.5 · pp. 83–84

3.5.2 Comparison of Trajectory Representations

• SEDS guarantees stable attraction but cannot represent time-varying behavior or unequal input and output dimensions; DMPs and ProMPs trade stability against stochastic and multidimensional modeling.DMPs generalize point-to-point motion to new start and goal positions and guarantee BIBO stability, while ProMPs can encode trajectory stochasticity and multidimensional coordination but do not guarantee stability. SEDS provides Lyapunov global asymptotic stability for target attraction; choosing among representations requires matching model complexity to the desired behavior, and benchmarking metrics remain needed.Source: §3.5.1.5 · p. 84, §3.5.2 · pp. 85–86, Table 3.4 · p. 85

3.5.3 Generalization of Demonstrated Trajectories

• Trajectory generalization uses parameter conditioning or scene warping, with different methods trading flexibility against stability, stochasticity, and feature requirements.DMPs adapt to new starts and goals, ProMPs condition on starts, goals, or via-points, and SEDS generalizes to new states; Gaussian conditioning generally needs fixed-length informative features. Non-rigid registration transfers trajectories by transforming task-space end-effector paths into joint space, can extend to force profiles, and requires approximately invariant dynamics between scenes; selecting demonstrations can be costly.Source: §3.5.3 · pp. 86–88

3.5.4 Information Theoretic Understanding of Model-Free BC

• Model-free behavioral cloning with Gaussian-noise maximum likelihood is equivalent to minimizing KL divergence from the expert trajectory distribution to the policy model.This interpretation applies to trajectory representations such as DMP, ProMP, and SEDS; learned skills may fail under covariate shift when test conditions differ substantially from demonstrations.Source: §3.5.4 · pp. 90–91

3.5.5 Time Alignment of Multiple Demonstrations

• Because demonstrations often differ in execution speed, time-dependent representations may require alignment; van den Berg et al. [2010] estimate a reference trajectory and time maps with EM.The E-step uses a Kalman smoother, while the M-step updates time mappings and covariance weights; DTW is used to update the mappings.Source: §3.5.5 · pp. 91–93

3.5.6.1 Learning Coupled Movements with DMPs

• DMP coupling models perceptual dependencies, but its coupling factors require reinforcement learning because demonstrations alone do not provide the needed deviations.Kober et al. [2008] learn the factors through trial and error.Source: §3.5.6.1 · p. 94

3.5.6.2 Learning Coupled Movements with Gaussian Conditioning

• Gaussian conditioning predicts unobserved coupled motion from partial observations; Interaction ProMP represents human-robot movements with correlated weight vectors and handles noisy observations.Amor et al. [2014] model two agents’ motion-parameter correlations, while Maeda et al. [2016] estimate unobserved movements from observed ones through conditional distributions.Source: §3.5.6.2 · pp. 95–96

3.5.6.3 Learning Coupled Movements with Time-Invariant Dynamical Systems

• Coupled Dynamical Systems use SEDS-based distributions to coordinate master and slave movements with stable behavior, including arm-finger and eye-arm coordination.The framework models movement distributions and repeatedly plans the master motion, estimates the slave state, and plans the slave motion toward the goal [Shukla and Billard, 2012, Kim et al., 2014, Lukic et al., 2014].Source: §3.5.6.3 · p. 97

3.5.7 Incremental Trajectory Learning

• Human demonstrations may be suboptimal, so corrective actions can improve learner performance through incremental trajectory refinement.Calinon and Billard [2007] initialize GMMs from demonstrated trajectories and update the trajectory-distribution model through kinesthetic teaching; their method does not address task context or generalization to new situations.Source: §3.5.7 · pp. 97–98
• Ewerton et al. [2016] incrementally learn a joint distribution over ProMP weights and context, then plan a trajectory conditioned on a new context.Human intervention supplies executed trajectories and contexts used to update the distribution.Source: §3.5.7 · pp. 98–99
• Maeda et al. [2017] combine DMPs and GPs to generalize trajectories to new scenes, request demonstrations under high prediction uncertainty, and ensure convergence to the desired point.Source: §3.5.7 · p. 99
• Kronander et al. [2015] incrementally modify a time-invariant dynamical system through local scaling and rotation, modeling nonlinear local dynamics with a GP when additional demonstrations arrive.The framework is not limited to a particular regression method and applies to movements representable as vector fields.Source: §3.5.7 · pp. 99–100

3.5.8 Combining Multiple Expert Policies

• Mixture of experts mixes multiple expert policies using weights, whereas products of experts multiply policies; Mülling et al. [2013] mixed learned DMP policies for table-tennis strikes and improved performance with reinforcement learning.Source: §3.5.8 · pp. 100–101
• Products of experts have not been popular in imitation learning, making their use to combine expert policies a proposed future direction.MOSAIC [Haruno et al., 2001] instead combines modules of local forward and inverse dynamics models to determine control inputs.Source: §3.5.8 · p. 101

3.6.1 Segmentation and Clustering for Task-Level Planning

• Demonstrations may contain multiple primitive motions, making trajectory segmentation and clustering necessary but often time-consuming when done manually.The section reviews imitation-learning methods for automating these steps, while leaving the underlying clustering theory out of scope.Source: §3.6.1 · pp. 101–102
• Segmentation methods range from online HMM and change-point approaches to BP-AR-HMMs used to learn primitive motion sequences in robotics.Kohlmorgen and Lemm [2001] use distances between nearby data windows; Kulić et al. [2008] cluster motion segments into a tree, and Konidaris et al. [2011] extend change-point inference to skill trees. Niekum et al. [2014] use BP-AR-HMMs for primitive sequences, linking imitation-learning advances to methods such as Fearnhead and Liu [2007] and Fox et al. [2009].Source: §3.6.1 · p. 102

3.6.2 Learning a Sequence of Primitive Motions

• Skill sequences can be represented as trees or primitive-transition graphs, while HMM variants model probabilistic transitions and task phase.Konidaris et al. [2011] merge similar skills across segmented chains into a skill tree. Manschitz et al. [2015] use SVMs to select the next primitive, and Kroemer et al. [2014, 2015] use STARHMMs to represent task phases and sequences of DMPs.Source: §3.6.2 · pp. 103–104, Figure 3.9 · p. 103, Figure 3.10 · p. 104
• Niekum et al. [2014] learn an FSA over BP-AR-HMM-segmented primitives, then plan new tasks with it and refine execution through expert kinesthetic corrections.Their incremental framework is intended for settings where demonstrations may be insufficient; experts can interrupt autonomous execution to correct a planned motion.Source: §3.6.2 · pp. 104–105
• Takano and Nakamura [2015] learn a mutual motion-language model that can generate sentences from motions or motion sequences from sentences.HMMs represent primitive motions, while learned probabilities link motion symbols and annotated sentences; an n-gram language model arranges words grammatically.Source: §3.6.2 · pp. 105–107

3.7.1 Model-Based Behavioral Cloning Methods with Forward Dynamics Models

• Embodiment differences or underactuated velocities can make demonstrations infeasible; model-based BC adjusts trajectories using learned forward dynamics, with methods progressing from LWR/GMR to GPs and deep nets for images.Gaussian Processes can propagate input uncertainty over multi-step predictions, but their computational cost makes them unsuitable for high-dimensional data.Source: §3.7.1 · pp. 108–109, Table 3.6 · p. 109

3.7.1.1 Imitation with a Gaussian Mixture Forward Model

• Grimes and Rao [2009] use GMR for one-step forward prediction and recursively predict learner trajectories, selecting actions to maximize posterior likelihood; repeated execution improves the model.The planned actions account for observed demonstrations and learner states feasible under kinematic and dynamic constraints.Source: §3.7.1.1 · p. 111

3.7.1.2 Imitation with a Gaussian Process Forward Model

• GP forward models propagate input uncertainty across predictions; Englert et al. [2013] iteratively minimize trajectory-distribution KL divergence, matching its first and second moments, with analytic gradients enabling efficient learning.The trajectory-distribution objective compares expert and learner behaviors.Source: §3.7.1.2 · pp. 112–114

3.7.2 Imitation Learning through Iterative Learning Control

• Without a forward model, iterative learning control updates a target trajectory using execution error; it is simple but does not generalize across targets, while iLQR can learn trajectory-following controllers for systems with uncertain dynamics.The target update is ˆτ ← ˆτ − α(τ − τ_d); forward- and inverse-dynamics learning is challenging for systems that are not fully controllable.Source: §3.7.2 · pp. 114–115

3.7.3 Information Theoretic Understandings of Model-Based Behavioral Cloning Methods

• The information-theoretic view interprets iterative policy evaluation as matching learner and expert expected features, relating policy learning to their induced trajectory distributions.Figure 3.12 illustrates this process as matching expected features, Eq[φ] ≃ Ep[φ].Source: §3.7.3 · p. 115, Figure 3.12 · p. 115

• Under the Gaussian trajectory-distribution assumption, feature-expectation matching is an M-projection onto the maximum-entropy distribution manifold.The passage identifies Gaussian distributions as maximum-entropy distributions, following Park and Bera [2009].Source: §3.8 · p. 116

3.8 Robot Applications with Model-Free BC Methods

• Model-free BC has been applied to acrobatic helicopter flight and robotic surgery, using task-specific controllers and learning methods [Abbeel et al., 2010; Osa et al., 2017b].The helicopter example used iterative LQR, while the surgical knot-tying example used a standard PD controller.Source: §3.8 · p. 116

3.8.1 Learning to Hit a Ball with DMP

• DMPs learned human-demonstrated tennis swings that generalized to different target positions and kinesthetically taught rhythmic ball-paddling motions [Ijspeert et al., 2002b; Kober and Peters, 2009].The tennis swing was reproduced on a 30-DoF humanoid using inverse dynamics; ball-paddling used a seven-DoF Barrett WAM arm and rhythmic DMPs.Source: §3.8.1 · pp. 116–117, Figure 3.13 · p. 117

3.8.2 Learning Hand-Over Tasks with ProMPs

• ProMPs modeled correlated human-robot motion for hand-over tasks in which a KUKA LWR gives a plate or screw to a human operator [Maeda et al., 2016].The operator’s motion was optically tracked, and tasks were demonstrated 13–20 times.Source: §3.8.2 · p. 117

3.8. Robot Applications with Model-Free BC Methods

• Maeda et al. [2016] used interaction ProMPs to plan reactive robot motions conditioned on observed human movement across collaborative tasks.The model learns correlations between human and robot motion, representing robot motion as a conditional distribution given the human’s movement.Source: §3.8 · p. 118, Figure 3.14 · p. 118
• Lioutikov et al. [2017] proposed probabilistic trajectory segmentation and sequences of ProMP movement primitives for table tennis, writing, and chair-assembly tasks.Source: §3.8 · p. 118

3.8.3 Learning to Tie a Knot by Modeling the Trajectory Distribution with Gaussian Processes

• In bimanual surgical tasks demonstrated 9–20 times under varied contexts, learned trajectories could be updated in real time.The experiments emulated knot-tying and soft-tissue cutting.Source: Figure 3.15 · p. 119, §3.8.3 · p. 119
• Osa et al. [2017b] modeled demonstrated surgical trajectories as a context-conditioned Gaussian Process, enabling real-time generalization and trajectory-based contact-force planning.The task’s looping trajectory requires preserving its overall topological shape; Osa et al. [2014] used Algorithm 6 to normalize time alignment across demonstrations.Source: §3.8.3 · p. 119

• Acrobatic helicopter flights were learned from expert demonstrations despite the system’s highly nonlinear dynamics.Source: Figure 3.16 · p. 120
• An iterative LQR controller executes the trajectory learned from the helicopter demonstrations.Source: Figure 3.16 · p. 120
• The application demonstrates learning autonomous helicopter maneuvers from expert demonstrations.Source: Figure 3.16 · p. 120

3.9.1 Learning Acrobatic Helicopter Flights

• To learn time-dependent flight trajectories and control nonlinear helicopter dynamics, Abbeel et al. [2010] used EM-like temporal alignment normalization and an iterative LQR controller.Source: §3.9.1 · p. 120
• Combining model-based BC with iterative controller learning transferred expert-level acrobatic skills to an RC helicopter performing flips, rolls, loops, hurricanes, and auto-rotation landings.The maneuvers also included chaos and tic-toc.Source: §3.9.1 · p. 121

3.9.2 Learning to Hit a Ball with an Underactuated Robot

• For underactuated ball hitting, Englert et al. [2013] learned trajectories and a task controller from kinesthetic demonstrations on the compliant BioRobTM manipulator.The method learned a Gaussian Process forward model and used PILCO to train a policy reproducing demonstrated trajectories.Source: §3.9.2 · p. 121, Figure 3.17 · p. 121

3.9. Robot Applications with Model-Based BC Methods

• Model-based behavioral cloning can learn robot-specific controllers that are robust to the correspondence problem and use learned forward dynamics for underactuated robots with unknown nonlinear dynamics.Obtaining a policy that reproduces the expert trajectory requires an iterative learning process.Source: §3.9 · p. 122

3.9.3 Learning to Control with DAGGER

• DAGGER learned video-game steering from image features and human demonstrations, and trained a UAV controller that avoided trees during autonomous flights in a real forest [Ross et al., 2011] [Ross et al., 2013].The UAV controller was trained with a small set of human demonstrations.Source: §3.9.3 · p. 122, Figure 3.18 · p. 122
• DAGGER’s online expert queries are essential when early errors lead learners into unseen states where demonstrations do not specify how to act [Ross et al., 2011] [Ross et al., 2013].The learner requests demonstrations for states induced by its own policy.Source: §3.9.3 · pp. 122–123, Figure 3.18 · p. 122

4.1 Problem Statement

• IRL infers an agent’s reward from behavior, sensory inputs, and an environment and body model, but is ill-posed because multiple rewards can make one policy optimal.Proposed uniqueness objectives include maximizing the margin between policies or maximizing entropy.Source: Inverse Reinforcement Learning · p. 124, §4.1 · p. 125

4.1. Problem Statement

• Many IRL methods alternate between updating a reward estimate and solving a reinforcement-learning problem, matching state-action visitation frequencies to those from demonstrations.Model-based methods evaluate frequencies using system dynamics; model-free methods commonly estimate them through sampling.Source: §4.1 · pp. 125–126

4.2 Model-Based and Model-Free Inverse Reinforcement Learning Methods

• Model-based IRL uses known dynamics and can be more data-efficient, while model-free IRL handles nonlinear dynamics through sampling but may require many costly trajectories.Model-based methods can be challenging when nonlinear dynamics are difficult to estimate.Source: §4.2 · p. 127, Table 4.1 · p. 128

4.3 Design Choices for Inverse Reinforcement Learning Methods

• IRL design must choose a uniqueness objective and reward representation: maximum entropy is a popular recent objective, while nonlinear rewards offer expressiveness but are more challenging to learn.Margin maximization was popular in early IRL studies, and reward functions should be as parsimonious as possible while remaining sufficiently expressive.Source: §4.3 · pp. 127–129, Table 4.2 · p. 128

4.4.1 Feature Expectation Matching

• Feature expectation matching learns a policy close to demonstrations by maximizing its difference from alternatives, but matching expected feature counts alone can leave multiple policies indistinguishable.Abbeel and Ng [2004] formulate the margin objective as a quadratic program; additional conditions are needed because multiple policies may match the same feature counts.Source: §4.4.1 · p. 131

4.4.2 Maximum Margin Planning

• Maximum margin planning finds costs that make demonstrated trajectories cheaper than alternatives by a margin; LEARCH extends this framework to efficiently learn nonlinear costs.MMP relies on efficient MDP solvers that produce deterministic optimal policies, which can be limiting for robotic systems with large configuration spaces requiring stochastic policies or planning approximations.Source: §4.4.2 · pp. 131–133

4.4.3.1 Maximum Entropy Inverse Reinforcement Learning

• Maximum entropy IRL selects a high-entropy policy distribution matching expert feature expectations, but stochastic dynamics introduce a bias and standard methods assume known transitions.Maximum causal entropy IRL [Ziebart, 2010] addresses the stochastic-dynamics drawback.Source: §4.4.3.1 · pp. 133–135

4.4.3.2 Maximum Causal Entropy Inverse Reinforcement Learning

• Maximum causal entropy IRL makes action choices independent of future states, removing dynamics-driven “bonus entropy” that can otherwise favor highly stochastic regions.This approach incorporates stochastic dynamics into reward estimation and avoids policies that simply target areas with high environmental stochasticity.Source: §4.4.3.2 · p. 136

4.4.3 Maximum Entropy IRL Extensions

• Maximum causal entropy conditions actions only on states observed so far, while IRL from failed demonstrations favors trajectories with features dissimilar to failed examples [Ziebart, 2010; Shiarlis et al., 2016].The failed-demonstration method gradually decreases its regularization parameter, first emphasizing successful examples and then failed ones.Source: §4.4.3.2 · p. 137, §4.4.3.3 · pp. 137–138

4.4.4.1 Linearly-Solvable MDPs

• Linearly-solvable MDP IRL estimates a value function rather than a reward and avoids repeatedly solving an MDP, but learned values may be harder to approximate or transfer across dynamics [Dvijotham and Todorov, 2010].The method models transitions as passive dynamics plus policy-specific active dynamics and estimates the value function by unconstrained convex maximum-likelihood optimization.Source: §4.4.4.1 · pp. 139–140

4.4.4.2 IRL Methods Based on a Bayesian Framework

• Bayesian IRL treats expert actions as evidence about reward functions; methods estimate posterior means with MCMC or use maximum-a-posteriori inference with gradient optimization [Ramachandran and Amir, 2007; Choi and Kim, 2011b].Choi and Kim also suggest that maximum-margin and maximum-entropy IRL can be interpreted in a Bayesian framework.Source: §4.4.4.2 · pp. 140–141

4.4.5 Learning Nonlinear Reward Functions

• Nonlinear reward IRL includes boosting ensembles, backprop-trained deep costs, and GPIRL’s nonlinear feature rewards [Ratliff et al., 2006b; Grubb and Bagnell, 2010; Levine et al., 2011].GPIRL’s manually chosen feature coordinates affect reward coverage and computational demands, including covariance-matrix inversion.Source: §4.4.5.1 · p. 141, §4.4.5.2 · p. 142, §4.4.5.3 · p. 142

4.4.6 Guided Cost Learning

• Guided cost learning alternates sampled-trajectory cost updates with policy optimization, using constrained LQR and locally linearized dynamics for smooth continuous trajectories without an explicit dynamics model [Finn et al., 2016b].Its cost-learning and optimization roles also parallel the discriminator and generator in GANs.Source: §4.4.6 · pp. 143–145

4.5.1 Relative Entropy Inverse Reinforcement Learning

• Relative-entropy IRL minimizes divergence from a baseline trajectory distribution using importance-sampled feature expectations, enabling learning with unknown dynamics and linear rewards.The method uses a sub-gradient solver; its expected feature counts are estimated through sampling.Source: §4.5.1 · p. 145, §4.5 · p. 146

4.5.2 Generative Adversarial Imitation Learning

• GAIL trains a policy to reproduce expert behavior and a discriminator to distinguish learner from expert trajectories, matching behavior without recovering a reward function [Ho and Ermon, 2016].The method constrains behavior to be approximately optimal under an unknown reward; it is included here despite not recovering that reward.Source: §4.5.2 · p. 147

4.6 Interpretation of IRL with the Maximum Entropy Principle

• Maximum-entropy IRL iteratively updates rewards and policies using new samples, while choosing maximum-entropy trajectory distributions among those meeting reward constraints; related dynamic programs also arise in economics.The paper identifies connections among structural estimation, inverse optimal control, and inverse reinforcement learning as warranting further cross-fertilization.Source: §4.4.3.4 · pp. 138–139, §4.6 · pp. 148–149
• Under a uniform trajectory prior, maximizing entropy is equivalent to minimizing KL divergence; alternative priors can be incorporated as a log-prior feature.For physical systems, methods use passive dynamics as the prior or approximate dynamics through sampled trajectories; relative-entropy IRL uses feature matching and importance sampling.Source: §4.6 · p. 149, §4.7 · p. 150

4.7.1 IRL from Partially Observable Demonstrations

• For partially observed demonstrations, hMDP models observation probabilities, while MRFs smooth noisy features across similar states and occlusion methods either omit missing data or infer it.Kitani et al. [2012] extend maximum-entropy IRL with observation probabilities; Bogert and Doshi [2014] omit occluded states and actions, whereas Bogert et al. [2016] treat missing data as hidden variables and use EM.Source: §4.7.1 · p. 151, §4.7 · p. 152

4.7.2 IRL with Incomplete Expert Observations

• When the expert has incomplete observations, IRL requires a POMDP model; learning from a given policy is simpler than learning from trajectories, and benchmark demonstrations remain simple due to computational difficulty.Choi and Kim [2011a] extend classical IRL to both settings.Source: §4.7.2 · pp. 152–153

4.7.3 Active Inverse Reinforcement Learning as a POMDP

• Active IRL models reward as a hidden quantity in a POMDP, enabling actions to gather reward information while pursuing task objectives, but its computational cost limits robotic applications.POMDPs offer a principled model and avoid the exploration-exploitation dilemma, making active IRL an avenue for future work.Source: §4.7.3 · p. 153

4.7.4 Cooperative Inverse Reinforcement Learning

• CIRL jointly optimizes human teaching and robot learning in a POMDP, allowing a human to accept a suboptimal task reward to convey more information to the robot.Exact optimal solutions are hard to obtain, and the framework has been demonstrated in simple simulations; teaching optimally may require the human to reason about the robot’s learning as well as the task.Source: §4.7.4 · pp. 153–154

4.8.1 Learning to Drive a Car in a Simulator

• In driving simulation, IRL imitated different demonstrated driving styles, while MMP and LEARCH learned costs that yielded paths increasingly similar to examples.LEARCH uses learned costs with classical planning methods; MMP and LEARCH were also applied to quadruped footstep planning.Source: §4.8.1 · p. 155, §4.8.2 · pp. 155, 157

4.8.3 Learning Motion Planning with Deep Guided-Cost Learning

• Guided cost learning used a neural-network reward and vision-based features to learn manipulation tasks requiring nonlinear rewards under unknown dynamics.Finn et al. [2016b] demonstrated the method on housekeeping tasks, with demonstrations recorded through kinesthetic teaching using a PR2 robot.Source: §4.8.3 · p. 157

4.8.4 Learning a Ball-in-a-Cup task with Relative Entropy Inverse Reinforcement Learning

• For an underactuated ball-in-a-cup task with difficult-to-estimate dynamics, model-free RE-IRL learned the demonstrated motion in simulation from 17 human demonstrations.The motions were recorded with a 3D motion-capture system.Source: §4.8.4 · p. 158

5.1 Behavioral Cloning vs Inverse Reinforcement Learning

• BC and IRL can be equivalent under maximum-entropy assumptions, but IRL adds complexity; reward recovery is useful when intent or generalization across terrains matters, while trajectory distributions can suffice for manipulation.For manipulation, relevant scene features can make reward recovery difficult; for quadruped footstep planning, a reward can support adaptive choices across terrains.Source: §5.1 · pp. 160–161

5.2.1 Problems Related to Demonstrated Data

• Demonstration challenges include combining multiple experts, filtering undesirable motions, learning without embodiment information, and reusing prior demonstrations; each remains an open problem or direction.Methods for multiple experts are less effective than for one expert; reinforcement learning can improve policies from imperfect demonstrations [Kober et al., 2013, Mnih et al., 2015, Silver et al., 2016].Source: §5.2.1 · p. 161, How to deal with undesirable motions… · p. 161, How to learn from raw sensory inputs… · p. 162

5.2.2 Open Questions Related to Design Choices

• Open design questions include which policy-similarity measures and instruction types to use; exploring measures such as Wasserstein divergence and using varied human feedback remain under-investigated.Wasserstein divergence has improved GAN performance and inspired imitation-learning approaches [Arjovsky et al., 2017, Goodfellow et al., 2014, Ho and Ermon, 2016, Finn et al., 2016a]; some methods use multiple instruction types [Jain et al., 2015].Source: §5.2.2 · p. 163
• Explicitly incorporating available prior knowledge may reduce required demonstrations, while fusing redundant sensors may enable more robust, adaptive behavior; trajectory-representation selection also lacks a definite answer.Imitation methods also need ways to improve tasks humans cannot demonstrate, including tasks beyond human physical capability.Source: §5.2.2 · pp. 163–164

5.2.3 Problems Related to Algorithms

• Skill generalization to complex, high-dimensional conditions remains challenging, and performance guarantees such as stability or convergence are absent for many methods.Some distribution-based methods condition on start or end positions, but may not scale to high-dimensional conditions; guarantees are especially important in robotics.Source: §5.2.3 · pp. 164–165

How to scale up with respect to the number of dimensions?

• Scaling remains open for high-dimensional sensory inputs and actions, globally optimal solutions in continuous spaces, and multi-agent imitation beyond simple settings.Humanoid robots may have over 50 joints, while current high-dimensional input methods are often limited to 2D images; multi-agent results have not migrated to large-scale robot applications.Source: How to scale up with respect to the… · pp. 165–166

How to perform incremental/active learning in IRL?

• Although active-learning approaches to IRL can incrementally improve an unsatisfactory policy, such methods remain insufficiently investigated.The practical need arises when an initial demonstration dataset fails to produce satisfactory performance.Source: How to perform incremental/active… · p. 166

5.2.4 Performance Evaluation

• Imitation learning lacks widely accepted benchmark problems, established evaluation methods, and large-scale benchmarks for comparing approaches.Existing benchmarking efforts have not provided a clear way to compare performance between methods.Source: How to establish benchmark problems… · p. 166, What metric should be used to evaluate… · p. 167