Source-linked AI summary
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges
Maryam Zare, Parham M. Kebria, Abbas Khosravi, Saeid Nahavandi
TL;DR
Complex, unstructured environments make manual behavior programming and task-specific reward design difficult, motivating learning from expert demonstrations. The paper surveys IL’s main approaches and recent developments, synthesizes their strengths and limitations, and identifies challenges and directions for future research. Its supported conclusion is that IL research spans diverse methods and applications while requiring substantial, diverse demonstrations for generalizable real-world policies.
Problem
Complex and unstructured environments make manually programming behavior or specifying task-specific reward functions difficult, motivating learning from expert demonstrations.
Method
The paper provides a comprehensive survey organized around behavioral cloning, inverse reinforcement learning, adversarial imitation learning, imitation from observation, and real-world challenges.
Results
The survey consolidates IL research, reviews strengths and limitations across approaches, and identifies gaps, challenges, and opportunities for future work.
Takeaways & Limitations
IL research offers diverse approaches for learning autonomous behavior from demonstrations, but generalizable real-world policies require diverse and large-scale demonstrations.
Takeaways & Limitations
Behavioral cloning can suffer causal misidentification under distributional shift, while inverse reinforcement learning faces ambiguity because one policy may correspond to infinitely many reward functions.
Abstract
from arXiv · showhide
In recent years, the development of robotics and artificial intelligence (AI) systems has been nothing short of remarkable. As these systems continue to evolve, they are being utilized in increasingly complex and unstructured environments, such as autonomous driving, aerial robotics, and natural language processing. As a consequence, programming their behaviors manually or defining their behavior through reward functions (as done in reinforcement learning (RL)) has become exceedingly difficult. This is because such environments require a high degree of flexibility and adaptability, making it challenging to specify an optimal set of rules or reward signals that can account for all possible situations. In such environments, learning from an expert's behavior through imitation is often more appealing. This is where imitation learning (IL) comes into play - a process where desired behavior is learned by imitating an expert's behavior, which is provided through demonstrations. This paper aims to provide an introduction to IL and an overview of its underlying assumptions and approaches. It also offers a detailed description of recent advances and emerging areas of research in the field. Additionally, the paper discusses how researchers have addressed common challenges associated with IL and provides potential directions for future research. Overall, the goal of the paper is to provide a comprehensive guide to the growing field of IL in robotics and AI.
I. INTRODUCTION
Imitation learning (IL) addresses the difficulty of manually programming behavior or designing task-specific rewards by learning from expert demonstrations. This survey reviews IL’s major approaches, recent developments, applications, challenges, and future directions.
- Motivation: IL learns desired behavior from expert demonstrations, reducing reliance on manually specified rules or task-specific reward functions.The approach maps observations to actions and is intended for autonomous behavior in complex environments.
- Motivation: Manual programming requires time, coding expertise, and detailed rules about actions and operating environments.
- Survey scope: The survey situates IL research alongside earlier surveys focused on humanoid robots, algorithms, process design, and autonomous driving.
- Survey scope: It aims to provide an updated overview of a rapidly evolving field with diverse applications and new algorithms, techniques, and use cases.
- Contribution: The survey seeks to consolidate research, identify gaps and challenges, and make IL concepts more accessible to researchers from related fields.
- Survey scope: The paper organizes IL around behavioral cloning, inverse reinforcement learning, adversarial imitation learning, imitation from observation, and real-world challenges.
II. BEHAVIORAL CLONING
Behavioral cloning learns expert actions from demonstrations as a supervised mapping, but covariate shift can expose the agent to states outside expert support. The survey discusses interactive, reward-based, and constrained strategies for improving robustness, alongside causal and shortcut-related limitations.
- Behavioral cloning: Behavioral cloning treats imitation as supervised learning by mapping environment states or observations to corresponding expert actions.Demonstrations provide state-action pairs used to train a function that generates actions for new states.
- Behavioral cloning: BC requires no knowledge of environment dynamics and is computationally efficient because it uses a well-studied supervised-learning formulation.
- Behavioral cloning: Covariate shift occurs because training states come from the expert policy, whereas testing states are induced by the learner’s own actions.When the agent leaves demonstrated states, BC does not know how to recover; this is especially hazardous in safety-critical settings such as driving.
- Covariate-shift mitigation: Interactive IL addresses train-test mismatch by querying an online expert, but frequent DAgger queries can burden experts and produce inaccurate or delayed feedback.Human-gated methods let experts choose interventions, while robot-gated methods allow agents to request intervention when needed.
- Covariate-shift mitigation: Reward-based methods estimate the support of the expert occupancy measure and use RL to encourage the agent to remain within that support without an online expert.Constrained IL instead restricts agents to demonstrated regions without relying on interactive experts or RL, supporting applications with safety constraints.
- Behavioral-cloning limitations: BC can suffer causal misidentification because supervised learning may rely on nuisance correlates instead of the underlying causes of expert actions.Distributional shift and sequential dependencies can intensify this problem, motivating causal-graph mappings and targeted interventions.
III. INVERSE REINFORCEMENT LEARNING
Inverse reinforcement learning infers a reward function from expert demonstrations and then optimizes it with reinforcement learning. The survey organizes IRL methods around computational cost, reward ambiguity, representation choices, and scalability.
- IRL infers the reward underlying demonstrations, assuming they come from an expert acting optimally, then trains a policy by optimizing that reward with RL.
- IRL agents interact with the environment to estimate rewards, but iterative reward estimation and policy training can be resource-intensive, unsafe, and sample-inefficient.
- A policy can be optimal for infinitely many reward functions, so IRL methods introduce additional structure through maximum-margin, maximum-entropy, or Bayesian formulations.
- Feature-based IRL can use manually designed or learned representations, but linear reward functions may be overly simplistic for complex tasks and raw sensory data.
- Classical Bayesian IRL does not scale well because each posterior sample requires solving an entire MDP; preference-based likelihoods and variational inference avoid that requirement.
- Many IRL algorithms assume known transition dynamics or an expert policy, whereas estimating them from samples can introduce errors into the recovered reward function.
IV. ADVERSARIAL IMITATION LEARNING
Adversarial imitation learning addresses IRL scalability by matching agent and expert trajectories through an agent–discriminator game. GAIL derives rewards from discriminator confusion, while later methods target stability, efficiency, and robustness.
- AIL searches for a policy without fully solving an RL subproblem at every iteration by having an agent imitate expert trajectories against a discriminator.
- GAIL trains a discriminator to distinguish expert from agent trajectories and uses its confusion as a reward encouraging expert-like behavior.
- AIL methods match expert and agent state-action distributions using divergences such as Shannon-Jensen or Kullback-Leibler divergence.
- Wasserstein distance through its dual formulation can improve training stability, motivating its use in several AIL methods.
- Min-max adversarial training can suffer from vanishing gradients and convergence failure, prompting alternatives such as primal Wasserstein IL.
V. IMITATION FROM OBSERVATION
Imitation from observation removes the requirement for expert action labels, enabling learning from videos and demonstrations across differing viewpoints, embodiments, and domains. The survey describes translation, representation-learning, inverse-dynamics, adversarial, and cross-domain approaches, alongside their assumptions and limitations.
- Imitation from observation learns without expert actions, making Internet videos and demonstrations from agents with different embodiments available for training.
- Context-aware translation converts expert demonstrations into the robot’s viewpoint and uses predicted observations to define a reward for RL optimization.
- The translation approach assumes time-aligned demonstrations, and its applicability is limited because such alignment is rare in real-world settings.
- Time-contrastive networks learn embeddings that bring simultaneous frames from different viewpoints together while separating visually similar frames from distant times.
- BCO infers missing expert actions with an inverse dynamics model before applying behavioral cloning, reducing the post-demonstration interactions required by earlier RL-based methods.
- GAIfO matches expert and agent state-transition distributions, handling time-unaligned demonstrations while addressing covariate shift through adversarial learning.
- Observation-based methods also include likelihood-based transition matching, cross-domain state maps, self-supervised video representations, and pseudo-labeled navigation learning.
A. Imperfect Demonstrations
Imperfect demonstrations are common because high-quality expert data is difficult to obtain and human demonstrators make mistakes. The survey reviews methods that weight, filter, or model demonstration quality instead of discarding non-optimal data.
- Optimal demonstrations are a common IL assumption, but collecting enough high-quality human demonstrations is difficult because demonstrations can contain mistakes.
- Discarding non-optimal demonstrations is often impractical, motivating methods that learn from imperfect data without requiring complete screening.
- 2IWIL estimates confidence for unlabeled demonstrations before reweighted GAIL, whereas IC-GAIL directly matches occupancies using confidence-scored and unlabeled data.
- Weighted behavioral cloning reuses the previous policy to reweight noisy action samples, but it converges to the optimal policy only when optimal demonstrations are the majority.
- Other approaches estimate demonstration weights automatically, regularize distribution matching with imperfect data, or infer state-dependent demonstrator expertise.
B. Domain Discrepancies
IL must address discrepancies between expert and agent domains, including differences in state-action spaces, viewpoint, embodiment, and environment dynamics. Recent approaches learn domain-invariant representations, task progress, or robust policies to transfer demonstrations across these discrepancies.
- Assuming identical expert and agent state-action spaces simplifies correspondence design but restricts IL to demonstrations from the agent’s domain.
- Cross-domain IL methods learn mappings or encodings between state-action spaces, often using paired and time-aligned demonstrations before policy learning.
- Viewpoint-agnostic imitation uses adversarial domain confusion, while goal-driven imitation emphasizes task progress over fine-grained structural matching.
- Robust-dynamics IL trains policies across continuously varying environment dynamics using only a few sampled environments during demonstrations and policy interaction.
- XIRL learns embodiment-invariant task representations by using temporal cycle consistency to identify key moments and define rewards from distance to a goal state.
- Gromov-Wasserstein distances align expert and agent state-action occupancies through isometric transformations without explicit shared latent spaces or proxy tasks.
- In Fig. 9, blue denotes policy-sample flow and green denotes expert-demonstration flow in IL under varying environment dynamics.
VII. OPPORTUNITIES AND FUTURE WORK
The survey categorizes IL methods, reviews their developments and challenges, and identifies future opportunities including state-only imitation and cross-domain learning. It emphasizes that diverse, large-scale demonstrations and robustness to domain differences remain important for real-world generalization.
- The survey organizes IL into behavioral cloning and inverse reinforcement learning, discusses adversarial imitation learning, and introduces IfO for state-only demonstrations.
- A major cross-category challenge is collecting diverse, large-scale demonstrations needed to train policies that generalize to real-world applications.
- Online videos can expand demonstration resources but contain demonstrators with varying levels of expertise.
- Future IL research should address differences in dynamics, viewpoint, and embodiment while learning from imperfect demonstrations and extracting useful information across domains.