Source-linked AI summary
Generative Adversarial Imitation Learning
Jonathan Ho, Stefano Ermon
TL;DR
Imitation learning traditionally recovers a cost function before learning actions, making it indirect and potentially expensive. This paper directly learns policies through a generative-adversarial framework, outperforming competing model-free methods on complex, high-dimensional control tasks.
Problem
Inverse reinforcement learning can be expensive and learns a cost function rather than directly specifying how the learner should act.
Method
The paper characterizes the policy induced by inverse reinforcement learning and instantiates the resulting direct policy-learning framework as generative adversarial imitation learning.
Results
The method outperformed competing approaches by a wide margin across complex, high-dimensional physics-based control tasks and achieved exact expert performance on Humanoid across tested dataset sizes.
Takeaways & Limitations
Matching expert and learned occupancy measures with Jensen-Shannon divergence can imitate expert policies exactly, unlike linear apprenticeship-learning algorithms.
Takeaways & Limitations
Although generally sample efficient with expert data, the method is not particularly sample efficient in environment interaction during training.
Abstract
from arXiv · showhide
Consider learning a policy from example expert behavior, without interaction with the expert or access to reinforcement signal. One approach is to recover the expert's cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning following inverse reinforcement learning. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks, from which we derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments.
1 Introduction
The section motivates directly learning an imitation policy from fixed expert trajectories without reinforcement signals, avoiding behavioral cloning’s compounding error and IRL’s costly, indirect procedure. It introduces a framework that bypasses intermediate IRL and yields a generative-adversarial, model-free algorithm that outperforms competing methods on complex, high-dimensional control tasks.
- Problem setting: Imitation learning here uses only expert trajectory samples, with no further expert queries during training and no reinforcement signal.The setting concerns learning to perform a task from expert demonstrations under these information constraints.
- Existing approaches: Behavioral cloning can suffer compounding error from covariate shift, while IRL avoids this issue by prioritizing entire trajectories.Behavioral cloning therefore tends to require large amounts of data, whereas IRL learns a cost function over trajectories rather than single-timestep decisions.
- Limitations of IRL: IRL algorithms can be extremely expensive because they require reinforcement learning in an inner loop, and learned costs do not directly specify how to act.The section identifies direct policy learning as a way to address both computational cost and IRL’s indirectness.
- Proposed framework: The proposed framework characterizes the policy produced by maximum causal entropy IRL and directly learns policies from data without an intermediate IRL step.This framework is introduced to explicitly tell the learner how to act.
- Algorithm and evaluation: The instantiated model-free algorithm uses generative adversarial training to fit expert state-action distributions and outperforms competing methods on complex, high-dimensional physics-based control tasks.The evaluation covers varying amounts of expert data.
2 Background
The paper formulates imitation learning in a γ-discounted infinite-horizon setting with finite state and action spaces. It adopts maximum causal entropy IRL, which learns costs making expert behavior preferable and uses reinforcement learning to recover high-entropy policies minimizing cumulative cost.
- Preliminaries: The formal setup uses finite state and action spaces in a γ-discounted infinite-horizon environment.The restriction avoids technical machinery concerning compactness of certain function sets.
- Preliminaries: Policy expectations are taken over trajectories generated from an initial distribution, policy actions, and environment transitions.Empirical expectations over sampled trajectories are denoted ˆEτ, and the expert policy is consistently written as πE.
- Inverse reinforcement learning: Maximum causal entropy IRL fits a cost function c ∈ C to rationalize a provided expert policy πE.In practice, πE is represented by trajectories sampled by executing it in the environment.
- Inverse reinforcement learning: The learned cost assigns low cost to expert behavior and high cost to other policies, enabling recovery of the expert through reinforcement learning.The expert’s expected cost is estimated from sampled trajectories.
- Inverse reinforcement learning: The associated reinforcement-learning procedure maps a cost function to high-entropy policies that minimize expected cumulative cost.The policy entropy is defined as the γ-discounted causal entropy H(π) ≜ Eπ[−log π(a|s)].
3 Characterizing the induced optimal policy
This section characterizes the policy obtained by running reinforcement learning on a ψ-regularized IRL cost as an occupancy-measure optimization relative to the expert. In the unregularized constant-ψ case, the induced policy exactly matches the expert’s occupancy measure, with IRL and RL forming dual and primal optimization steps.
- Regularization: A convex cost regularizer ψ controls overfitting when IRL uses the full class of costs RS×A = {c : S × A → R}.The section notes that convexity is not particularly restrictive and includes effective regularizers used for robotic manipulation.
- Occupancy measures: Valid occupancy measures form a convex feasible set with a one-to-one correspondence to policies, enabling policy optimization to be expressed as convex optimization over state-action visitation distributions.Occupancy measures represent encountered state-action distributions and uniquely determine policies.
- Policy characterization: RL ◦ IRLψ(πE) solves minπ∈Π −H(π) + ψ∗(ρπ − ρπE), making imitation an occupancy-measure matching problem shaped by ψ∗.The characterization follows from a saddle-point relationship in which IRL finds the cost coordinate and RL recovers the policy coordinate.
- Unregularized case: For constant ψ, the learned policy’s occupancy measure exactly equals the expert’s: ρ˜π = ρπE.With no cost regularization, the recovered policy exactly matches the expert’s occupancy measure.
- Primal-dual interpretation: IRL is the dual of occupancy-measure matching, with the recovered cost as the dual optimum and the induced policy as the primal optimum.Strong duality connects cost recovery to optimizing the occupancy-measure primal problem after RL is run on the recovered cost.
4 Practical occupancy measure matching
Exact occupancy-measure matching is impractical with finite expert samples because unseen state-action occupancies are zero, motivating smooth penalties for occupancy differences. The resulting framework encompasses entropy-regularized apprenticeship learning, which scales with policy approximation but may fail to imitate when its cost class is too restrictive.
- Practical occupancy measure matching: Finite expert samples make exact occupancy matching impractical in large environments because unseen state-action occupancies are exactly zero.Exact matching would force the learned policy never to visit unseen states.
- Practical occupancy measure matching: The framework relaxes exact matching by modifying ψ so its conjugate smoothly penalizes differences between learned and expert occupancy measures.This relaxation is motivated by the goal of developing imitation learning algorithms suitable for large environments.
- Entropy-regularized apprenticeship learning: With suitable ψ, entropy-regularized apprenticeship learning is equivalent to reinforcement learning after inverse reinforcement learning with ψ = δ_C, constraining the recovered cost to C.Scaling C by α scales policy entropy regularization, and α → ∞ recovers the original apprenticeship objective.
- Pros and cons of apprenticeship learning: Restrictive cost classes can prevent expert-like policies, while apprenticeship learning with such classes can scale to large state and action spaces using policy function approximation.The limitation is especially relevant to linear subspaces unless their basis functions are carefully designed.
- Apprenticeship learning optimization: The scalable apprenticeship algorithm alternates fitting a cost from current-policy trajectories with a TRPO update based on its policy-gradient estimate.The TRPO step constrains policy change by average KL divergence to reduce divergence from noisy gradient estimates.
5 Generative adversarial imitation learning
Generative adversarial imitation learning introduces a data-adaptive cost regularizer whose optimization minimizes Jensen–Shannon divergence between learner and expert occupancy measures. A practical model-free algorithm alternates discriminator updates with TRPO policy steps to solve this objective in large environments.
- Regularizer: The proposed regularizer combines exact occupancy-measure matching with tractability in large environments.The constant regularizer is intractable in large environments, whereas indicator regularizers require careful tuning and cannot exactly match occupancy measures.
- Regularizer: ψGA averages over expert data and heavily penalizes cost functions assigning large costs to expert state-action pairs.It places low penalty on cost functions assigning negative cost to expert pairs, while costs close to zero receive heavy penalties.
- Occupancy matching: The discriminator objective equals the optimal negative log loss for distinguishing learner from expert state-action pairs and corresponds, up to a constant shift, to Jensen–Shannon divergence.The divergence is defined between the learner and expert occupancy distributions and is a squared metric between distributions.
- Occupancy matching: Equation (15) finds a policy whose occupancy measure minimizes Jensen–Shannon divergence to the expert’s, enabling exact imitation of expert policies.Unlike linear apprenticeship learning algorithms, the objective minimizes a true metric between occupancy measures.
- Algorithm: The practical algorithm parameterizes the policy and discriminator, then alternates Adam discriminator ascent with TRPO policy descent on the adversarial objective.The TRPO step limits how much the policy changes, while the algorithm targets a saddle point for model-free imitation in large environments.
6 Experiments
Algorithm 1 was evaluated against behavioral cloning, FEM, and GTAL on nine physics-based control tasks using matched neural architectures and environment interaction. It generally outperformed the baselines, especially on high-dimensional MuJoCo tasks, while achieving exact expert performance on Humanoid across all tested dataset sizes.
- Evaluation setup: The evaluation covered 9 physics-based control tasks, from cartpole, acrobot, and mountain car to high-dimensional 3D humanoid locomotion.Non-classic tasks were simulated with MuJoCo.
- Evaluation setup: Algorithm 1 was compared with behavioral cloning, feature expectation matching, and game-theoretic apprenticeship learning.FEM used Clinear, while GTAL used Cconvex.
- Evaluation setup: All methods trained policies with the same two-hidden-layer, 100-unit neural architecture, while FEM, GTAL, and Algorithm 1 received equal environment interaction.Networks used tanh nonlinearities and random initialization at each trial.
- Results: On classic control tasks, behavioral cloning was less expert-data-efficient, whereas FEM and GTAL usually reached near-expert performance across dataset sizes.Algorithm 1 always outperformed behavioral cloning, FEM, and GTAL on these tasks.
- Results: On MuJoCo tasks, Algorithm 1 almost always reached at least 70% of expert performance and nearly always dominated the baselines.FEM and GTAL performed poorly on Ant, while behavioral cloning required sufficient data on several tasks.
- Results: Humanoid was the clearest case: behavioral cloning stayed below 60%, while Algorithm 1 achieved exact expert performance for every tested dataset size.Behavioral cloning reached satisfactory performance with enough data on HalfCheetah, Hopper, Walker, and Ant.
7 Discussion and outlook
The method is sample efficient with respect to expert data but requires substantial environment interaction during training. Its model-free, random-exploration design trades off the sample efficiency and expert guidance available to alternative approaches.
- Sample efficiency: The method is generally sample efficient in expert data but not particularly sample efficient in training-time environment interaction.Samples needed to estimate the imitation-objective gradient were comparable to those required for TRPO to train expert policies from reinforcement signals.
- Model-free limitation: As a model-free method, it generally requires more environment interaction than model-based methods such as guided cost learning.Guided cost learning inherits sample efficiency from guided policy search but requires a well-approximated model based on iteratively fitted time-varying linear dynamics.
- Expert interaction: Unlike expert-interactive methods such as DAgger, the approach randomly explores to identify actions that move the policy occupancy measure closer to the expert’s.The discussion suggests that combining well-chosen environment models with expert interaction could be advantageous.
A Proofs · A.1 Proofs for Section 3
The proofs establish strict concavity of the entropy-like function via the log-sum inequality and Proposition 3.1. They then use saddle-point arguments, minimax duality, and strict convexity to prove equality of the relevant occupancy measures and policies.
- A.1 Proofs for Section 3: The log-sum inequality yields concavity of ¯H for convex combinations of occupancy measures.The argument applies pointwise over states and actions before summing.
- A.1 Proofs for Section 3: Equality in the concavity inequality holds exactly when the corresponding policies, πρ and πρ′, are identical.This follows from the equality condition in the log-sum inequality.
- A.1 Proofs for Section 3: Proposition 3.1 strengthens the equality condition, showing that ¯H is strictly concave because equal policies imply equal occupancy measures.Thus equality holds if and only if ρ = ρ′.
- A.1 Proofs for Section 3: The proof of Proposition 3.2 compares πA and ˜π through their occupancy measures, ρA and ˜ρ, using a function ¯L over D × R^S×A.The construction is explicitly based on saddle-point properties.
- A.1 Proofs for Section 3: Convexity of D, −¯H, and ψ, together with concavity in the second argument, permits application of minimax duality to ¯L.D is compact and convex, while R^S×A is convex.
- A.1 Proofs for Section 3: The relevant optimal pair forms a saddle point of ¯L, providing the equalities needed to compare ρA and ˜ρ.This conclusion follows from the cited equations defining the saddle-point conditions.
- A.1 Proofs for Section 3: Strict convexity of ¯L(·, c) forces ρA = ˜ρ, and uniqueness of policies associated with occupancy measures then gives πA = ˜π.Strict convexity is supplied by Lemma 3.1, while policy uniqueness follows from Proposition 3.1.
A.2 Proofs for Section 5
The proofs establish that strictly decreasing convex classification losses induce valid cost regularizers whose imitation objectives minimize corresponding f-divergences between occupancy measures. They specialize this construction to the logistic loss and derive the causal-entropy policy-gradient identity.
- Loss-to-regularizer construction: Strictly decreasing convex surrogate losses induce cost regularizers whose dual objective equals the minimum expected classification risk between policy and expert occupancy measures.This connects binary state-action classification losses to imitation-learning objectives through cost regularization.
- Loss-to-regularizer construction: The construction can generate imitation-learning algorithms minimizing any f-divergence induced by a strictly decreasing convex surrogate loss.Jensen-Shannon divergence is identified as a special case of the f-divergences arising from this correspondence.
- Proposition A.1: For every strictly decreasing convex loss, the resulting regularizer is closed, proper, and convex, and reinforcement learning after inverse reinforcement learning recovers the entropy-regularized minimum-risk policy.This is the main conclusion of Proposition A.1.
- Logistic-loss corollary: The logistic loss φ(x) = log(1 + e^−x) yields the ψGA regularizer, whose optimal expected risk is, up to a constant, the Jensen-Shannon divergence.The proof derives the discriminator maximization over state-action occupancy measures using the sigmoid range (0, 1).
- Causal-entropy gradient: The causal-entropy gradient reduces to the standard policy-gradient formula for the fixed cost c_log(s, a) ≜ −log πθ(a|s).The derivation uses the entropy-related action-value Qlog and shows that the second term vanishes.
B Environments and detailed results
Experiments use OpenAI Gym environments spanning observation and action spaces with specified dimensions or finite cardinalities. The study reports interaction budgets, variance-reduction procedures, and policy results summarized from 50 trajectories.
- Environments: Experiments use OpenAI Gym environments listed with observation and action-space dimensions or finite cardinalities.Table 1 provides the environment names, version numbers, and space sizes.
- Experimental setup: FEM, GTAL, and the proposed algorithm use the environment-interaction amounts reported in Table 2.The passage identifies Table 2 as the source of these interaction quantities.
- Experimental setup: Value functions matching policy-network architectures and generalized advantage estimation reduce gradient variance for FEM, GTAL, and the proposed algorithm.The reported GAE settings are γ = .995 and λ = .97.
- Results: Exact experimental results are reported in Table 3, with means and standard deviations computed over 50 trajectories.The passage states that these summary statistics are based on 50 trajectories.