A Survey of Imitation Learning Methods, Environments and Metrics

1 Introduction

• Imitation learning lets agents acquire skills by observing a teacher, using examples that indicate successful behavior rather than searching through trial and error.The approach is also described as supporting more human-centric learning from a proficient source.Source: §1 · p. 1
• The field has expanded across methods and applications, including robotics, game-playing, and natural language processing.Work has introduced ensemble, adversarial, and self-supervised approaches, as well as exploration and latent-action remapping.Source: §1 · p. 2
• The survey presents taxonomies for methods, environments, and metrics to address a lack of standardisation that makes meaningful comparisons challenging.It also discusses literature problems, challenges, and future directions.Source: §1 · p. 2

2 Review preliminaries

• The review starts from Hussein et al. and Zheng et al. [121], applying inclusion guidelines to filter the snowballed literature.The guidelines focus on imitation-learning or inverse-reinforcement-learning research and published peer-reviewed work, with a stated preprint exception.Source: §2 · pp. 2–3
• The authors organize methods around predominant training techniques, offering a more detailed classification of emerging trends than broad prior categories.Their approach acknowledges and builds on earlier methodological classifications, including Hussein et al. and Zheng et al. [121].Source: §2 · p. 3
• The survey introduces environment and metric taxonomies based on each environment’s role in learning and each metric’s meaning.Earlier surveys discussed environments by domain and metrics without classifying them or providing in-depth metric analysis.Source: §2 · p. 3

3 Background

• A teacher provides behavior for an agent to approximate, and may be human or computational without necessarily being an expert.The teacher is defined as an agent whose behavior imitation learning seeks to approximate.Source: §3 · pp. 3–4
• An MDP models states, actions, transitions, rewards, and discounting, while imitation learning seeks a policy mapping states to actions.The environment supplies a state, the agent predicts an action, and the transition dynamics generate the next state.Source: §3 · pp. 3–4
• A trajectory is an ordered sequence of state-action pairs, and the collected trajectory data is the learner’s only information for learning the task.Too few samples can leave the agent without prior guidance in encountered states, while corrupted data can produce behavior unlike the teacher’s.Source: §3 · p. 5
• Demonstrations pair states with teacher actions, but their lists need not preserve trajectory order and may not represent the environment adequately.Their representativeness matters because sparse or corrupted samples can leave the agent without guidance in states it encounters.Source: §3 · p. 5
• Experiences use state-action pairs from the learner’s own trajectories, unlike demonstrations, which record the teacher’s actions for states.Experiences may include environmental reward, but optimizing the policy with that reward would depart from the imitation-learning paradigm described here.Source: §3 · pp. 5–6
• Imitation from observation uses successive teacher states without action labels, reducing dataset requirements but risking ambiguity and inadequate transition coverage.The text notes that non-injective transitions can map different state-action pairs to the same state, and that assessing dataset coverage may be impossible.Source: §3 · p. 6
• Non-stationary policies map state sequences to actions and suit motor trajectories, while most imitation-learning work focuses on stationary policies using present information.Non-stationary policies can compound errors and are difficult to adapt to unseen scenarios; most work also focuses on low-level actions.Source: §3 · pp. 6–7

4 Imitation Learning Methods

• Model-based approaches learn an environment model, while model-free methods learn without one and can access more diverse states through trial and error.The text describes a trade-off between model-based time and sample efficiency and model-free robustness to unseen states; online methods assume access to a teacher policy, which is seldom available.Source: §4 · p. 7
• The proposed taxonomy distinguishes behavioural-cloning and adversarial model-free methods from dynamics and hybrid model-based methods.The authors describe the usual model-free/model-based classification as coarse and say their taxonomy classifies newer trends in more detail.Source: §4 · pp. 7–8
• 50 publications proposed new methods; adversarial learning was most common with 15, followed by dynamic methods with 14, behavioural cloning with 11, hybrid methods with 8, and online methods with 2.The authors suggest hybrid approaches and dynamic models may be gaining popularity, while noting that hybrid methods offer a better efficiency-effectiveness trade-off.Source: §4 · p. 8

4.1 Behavioural Cloning

• Behavioural cloning reduces imitation to supervised learning that predicts likely actions from demonstrated state-action pairs.The policy aims to approximate the teacher’s trajectory.Source: §4.1 · p. 8
• For complex dynamics or many possible actions, sample requirements are approximately 100 times those for classic control tasks.Policies also struggle to generalise from known trajectories to unseen states, while collecting labelled demonstrations can be costly.Source: §4.1 · p. 8
• Behavioural cloning can bootstrap later learning: Lynch et al. [64] use it before planning, while Daftry et al. use it before reinforcement learning for seasonal adaptation.The approach supports offline learning before direct environment interaction.Source: §4.1 · pp. 8–9
• When research focuses on factors outside learning, behavioural cloning is often paired with mechanisms such as attention or domain shift [16] [18] [28].The imitation paradigm could be replaced with adaptations, although labelled demonstrations remain costly.Source: §4.1 · p. 9

4.2 Dynamics Model methods

• Model-based imitation learns environmental dynamics through online play, using inverse models to infer actions or forward models to predict states.These models are proposed for tasks without direct teacher action information.Source: §4.2 · p. 9

4.2.1 Inverse Dynamics Models.

• Inverse dynamics retrieves likely actions from state transitions, and Nair et al. [73] use current and goal states to guide sequential robot rope manipulation.Their experiments use image-matrix states.Source: §4.2.1 · p. 9
• Behavioural Cloning from Observation uses an inverse model to create self-supervised action labels from transitions sampled by a random policy, then trains the policy with behavioural cloning.The method does not require teacher actions or a teacher-provided goal state; later work targets performance, stability, sample efficiency, and other domains [70] [38] [78] [84].Source: §4.2.1 · p. 9
• The first imitation-learning exploration mechanism [38] samples model outputs with a softmax distribution, favoring actions with higher predicted values while retaining exploration.The method applies exploration and refines sampling to improve on Torabi et al.’s approach.Source: §4.2.1 · p. 9
• Some inverse-dynamics approaches alternate reinforcement learning with imitation: Pavse et al. use rewards and demonstrations to seek task reward, while Gavenski et al. use goal-aware sampling and a limited replay buffer.The buffer aims to reduce the magnitude of updates from experience.Source: §4.2.1 · pp. 9–10

4.2.2 Forward Dynamics Models.

• Forward models predict next states from actions or temporal information; latent actions can represent transitions such as hitting a wall, but these models are less popular because next-state prediction is harder and added sequence information may reduce generalisation.Edwards et al. remap learned latent actions to environmental actions; forward-model approaches are described as less popular than inverse models.Source: §4.2.2 · p. 10

4.3 Adversarial learning methods

• Adversarial learning and inverse reinforcement learning create artificial rewards from demonstrations, which policies optimise using reinforcement learning rather than supervised action prediction.Most surveyed work uses adversarial reward construction [17] [57] [79], while earlier work also used other inverse reinforcement learning techniques [35] [94].Source: §4.3 · p. 10
• Ho and Ermon’s adversarial approach [44] uses a generative policy and discriminator to make student trajectories resemble teacher trajectories.It assumes teacher actions are available during training and can encounter local minima requiring prolonged environmental interactions.Source: §4.3 · pp. 10–11
• Observation-based adversarial learning discriminates between teacher and agent trajectories to derive a reward, improving performance in dynamic environments and reducing the cost of learning new policies.Later work also applies convolutional discriminators to vision tasks.Source: §4.3 · p. 11
• Chang et al. [17] use a noise-conditioned normalising flow to reduce seeking behaviour, while multi-agent methods include cooperative, decentralised, zero-sum, and strategy-conditioned settings [94].The flow adds noise during training but not evaluation.Source: §4.3 · p. 11
• Other settings include third-person teacher views, planning over behavioural-cloning and adversarial action predictions, and shared policy–generative-model weights for self-driving.These examples extend the methods across observation perspectives and application settings.Source: §4.3 · pp. 11–12

4.4 Hybrid approaches

• Combining dynamics models with adversarial learning provides temporal signals and environment knowledge that the survey says can create more robust policies and improve efficiency, with some methods requiring one trajectory.These methods use discriminators while drawing on dynamics-model approaches [37] [51] [59] [70].Source: §4.4 · p. 12
• Wang et al. use a variational autoencoder to increase diversity with relatively few demonstrations and learn new trajectories in one shot.A temporal neural network produces a non-stationary policy; optimisation does not rely on the forward dynamics model.Source: §4.4 · p. 12
• For learning from demonstrations, Baram et al. report earlier teacher behaviours and fewer teacher samples, while Yu et al. study partial Atari demonstrations of less than a full episode.The latter uses a variational autoencoder for a forward model and a discriminator for an artificial reward.Source: §4.4 · p. 12
• For observation-based learning, Zhu et al. switch between covering and seeking modes using ten trajectories, but assume unique state-action pairs per next state; Kidambi et al. [51] assume identical episode starts.The survey notes the first assumption is invalid in many momentum-rich environments and the second limits generalisation as state sizes grow; ensemble prediction adds cost, later addressed by Monteiro et al.Source: §4.4 · p. 12
• Liu et al. [59] translate between teacher and student contexts with learned encoders, avoiding model fine-tuning when context changes; Gangwani et al. [37] perform better in their setting than other methods but remain far from the teacher.The cross-context approach requires training a new encoder, while the survey says the divergent-trajectory setting needs further research.Source: §4.4 · pp. 12–13
• Hybrid methods have become more popular, and some achieve teacher results with ten or fewer trajectories, though exploration and cross-domain contexts remain uncommon research directions.The survey identifies these mechanisms as areas where further research is needed.Source: §4.4 · p. 13

• Online imitation learning accesses the teacher’s information in real time instead of using a prerecorded demonstration dataset.Source: §4.5 · p. 13
• Designing a reward function or policy function can be challenging in online imitation learning, and direct teacher access limits its use.The survey describes it as less commonly researched than other areas.Source: §4.5 · p. 13
• Real-time teacher interaction or changing conditions make online imitation learning more suitable.Source: §4.5 · p. 13
• Follow-The-Leader sequentially minimizes cumulative loss or maximizes reward; Follow-The-Regularised-Leader adds regularization to address noisy demonstrations and oscillations.The cited work reports better policy results after regularization.Source: §4.5 · p. 13

• Newer learning methods combine dynamics-model and adversarial mechanisms, while recent work also emphasizes learning from observation and efficiency.Source: §4.6 · p. 13
• The reviewed literature contains few imitation-learning methods for online learning and multi-agent systems.Source: §4.6 · pp. 13–14
• Online learning requires teacher access, while multi-agent settings have highly dynamic environments and potentially vast state combinations.These challenges can require substantial data or assumptions about the environment that may not hold across cases.Source: §4.6 · pp. 13–14
• The authors argue that methods tailored to online-learning and multi-agent challenges need further research and development.Source: §4.6 · p. 14

5 Imitation Learning Environments

• The proposed taxonomy groups environments as validation, precision, or sequential according to the agent’s intent rather than domain alone.It also records domain- and task-specific characteristics to help researchers judge suitability; environments support fairer comparisons by giving methods shared dynamics and demonstrations.Source: §5 · p. 14

5.1 Validation environments

• Validation environments offer relatively straightforward tasks, but relying on them alone can let agents overimitate rather than learn the underlying task.Limited variation in initialization can let agents match teacher states; the survey notes that even Google simulation suites with diverse initializations remain vulnerable to this behavior.Source: §5.1 · pp. 15–16

5.2 Precision environments

• Precision environments test accurate action with little dependence on long-term consequences, including robotic manipulation and obstacle navigation.Physical robots are often used for final evaluation but are costly; simulated held-out settings can test whether an agent learned the task beyond imitation [78].Source: §5.2 · p. 16

5.3 Sequential environments

• Sequential environments test actions with lasting consequences and future planning, using settings such as autonomous-driving simulations and video games.Different simulation settings can hinder comparisons, while diverse levels can assess generalization; the survey describes combining sequential and validation environments as ideal.Source: §5.3 · pp. 17–18

5.4 Environments Discussion

• Six of 66 environments account for more than 80% of use, while 42 appear only once, which the authors identify as evidence of insufficient experimental protocols.The six most common are Walker2D, MuJoCo robotic arm simulation, Hopper, CartPole, MountainCar, and HalfCheetah; five are classified as validation environments.Source: §5.4 · p. 18

6 Imitation Learning Metrics

• Metrics are grouped as quantitative or qualitative and by context—behaviour, domain, or model—because the same measure can convey different information across tasks.The taxonomy addresses context that a simple quantitative-versus-qualitative distinction misses.Source: §6 · pp. 18–19, Figure 10 · p. 18

6.1 Behaviour metrics

• Reward-based behaviour metrics are common: accumulated reward sums episode rewards, average episodic reward aggregates episodes, and performance normalizes rewards against random and teacher policies.A single episode can mislead about generalization, and performance can misrepresent agent quality when the teacher is not optimal or rewards are skewed.Source: §6.1 · pp. 19–20
• Distance metrics compare agent and teacher properties, while success rate records goal attainment; questionnaires and Turing tests assess human-likeness but require substantial effort.Trajectory measures may mislead when early action differences compound, and qualitative measures are less common in recent work partly because of their cost.Source: §6.1 · pp. 20–21

6.2 Domain metrics

• Domain metrics capture task-specific properties, including driving safety measures and robotic task time or failure rate, but failure counts alone do not explain why failures occurred.The review also notes domain-specific scores such as coins collected or goals scored, which should be paired with behaviour- or model-specific metrics.Source: §6.2 · pp. 21–22

6.3 Model metrics

• Model metrics such as accuracy can mislead about task performance; saliency maps and dimensionality-reduction methods offer intuition but have interpretability and reliability limits.The authors recommend treating model metrics as intuition rather than concrete proof and combining them with behaviour- and domain-specific measures.Source: §6.3 · pp. 22–23

6.4 Evaluation protocol

• Evaluation commonly trains a policy on teacher demonstrations and reports metrics over 5–100 episodes, but seed selection can enable unfair comparisons when teacher and agent share seeds.Other protocols vary demonstration quantity to study sample efficiency or teacher optimality to study robustness; comparisons often include behavioural cloning or category-specific baselines.Source: §6.4 · p. 23

6.5 Metrics Discussion

• 109 uses of quantitative metrics across 41 metrics contrast with 10 uses of 9 qualitative metrics; protocols often omit evaluation beyond optimal performance, and proposed adjustments will not solve all issues.The review reports increased use of performance, which compares policies against random and teacher agents; reward remains the most common quantitative metric.Source: §6.5 · p. 24

7.1 Imitation Learning Evaluation

• Testing can expose states absent from demonstrations, while customized teacher datasets and baselines make results difficult to compare; success should mean accomplishing the task, not merely copying actions.The trajectory example shows that conflicting demonstrations can produce loops, and that test initializations can reveal behavior not represented in training data.Source: §7.1 · p. 25

7.2 Expert optimality feasibility

• Proficient teachers may be non-optimal, their demonstrations may omit safety or fairness goals, and researchers should distinguish expert policies from acceptable teacher policies.The survey argues that optimality can be difficult to assess in complex settings and that the appropriate demonstrations depend on the task.Source: §7.2 · p. 26

7.3 Expert Necessary Requirements

• Reliable expert status requires necessary conditions, but continuous state-action spaces, partial or faulty observations, and incomplete state coverage can prevent guarantees of optimality.The authors caution that these conditions are too restrictive to serve as the sole criteria for evaluating an approach in practical applications.Source: §7.3 · p. 27

7.4 Challenges and Future Directions

• Future work should address learning safely from potentially unsafe teachers, assess effectiveness alongside sample efficiency, and pursue generalization and transfer.The survey defines effectiveness as learning optimally or near-optimally from non-optimal teachers and describes adaptability through generalization and transfer.Source: §7.4 · pp. 27–28

8 Conclusions

• The survey classifies methods by emerging trends while retaining the traditional model-free/model-based distinction.The authors present the classification as a way to identify opportunities, not replace the established dichotomy.Source: §8 · p. 28
• The environment taxonomy distinguishes validation, precision, and sequential purposes to help researchers choose evaluation environments.Sequential environments are those where some actions can influence the choice of future actions.Source: §8 · p. 28
• The metric review groups evaluation measures into behaviour, domain, and model categories, with qualitative and quantitative refinements.Whether varied metrics can be aggregated to rank or compare methods remains an open question.Source: §8 · p. 29
• The survey finds evaluation lacks standardization, making it difficult to compare imitation-learning methods uniformly.The authors describe this as the first survey, to their knowledge, to systematically review methods, environments, and metrics in this way.Source: §8 · p. 29
• The survey identifies behavior beyond conventional environmental metrics as an area for improvement and points to work on more human-like behavior.It names reinforcement learning from human feedback as a relevant research field.Source: §8 · p. 29

A Validation environments

• Table 2 lists validation-environment information and groups benchmark environments by suite.The accompanying text says it includes Russell and Norvig’s environment attributes.Source: Table 2 · p. 35
• Table 3 lists precision-environment information, grouping benchmark environments by suite.Source: Table 3 · p. 35
• Table 4 lists sequential-environment information and defines RAM as vector representations of values stored in memory.Source: Table 4 · p. 35, A Validation environments · p. 35