Source-linked AI summary
Model-Free Imitation Learning with Policy Optimization
Jonathan Ho, Jayesh K. Gupta, Stefano Ermon
TL;DR
Imitation learning must recover desirable behavior from demonstrations despite unknown costs, while many existing approaches repeatedly solve planning or reinforcement-learning problems. The paper directly optimizes parameterized stochastic policies with model-free policy-gradient methods, including a trust-region variant. It demonstrates high-dimensional continuous control performance and local optimality guarantees, while noting high variance and the absence of expert interaction or reinforcement signals as limitations.
Problem
Imitation learning must learn behavior from expert demonstrations when the cost function is unknown, but existing methods can require repeated planning or reinforcement-learning solves that limit scalability.
Method
The paper directly optimizes parameterized stochastic policies using model-free policy gradients, including a trust-region method, while fitting cost functions locally.
Results
The methods train neural-network policies from scratch with over 600 continuous features and identify locally optimal solutions without repeatedly solving inner planning problems.
Takeaways & Limitations
Policy optimization can achieve at least 98% of LPAL performance with similar sample complexity and faster training in the reported large-gridworld experiment.
Takeaways & Limitations
The policy-gradient estimator can have extremely high variance, and the approach does not use expert interaction or reinforcement signals that may improve sample efficiency.
Abstract
from arXiv · showhide
In imitation learning, an agent learns how to behave in an environment with an unknown cost function by mimicking expert demonstrations. Existing imitation learning algorithms typically involve solving a sequence of planning or reinforcement learning problems. Such algorithms are therefore not directly applicable to large, high-dimensional environments, and their performance can significantly degrade if the planning problems are not solved to optimality. Under the apprenticeship learning formalism, we develop alternative model-free algorithms for finding a parameterized stochastic policy that performs at least as well as an expert policy on an unknown cost function, based on sample trajectories from the expert. Our approach, based on policy gradients, scales to large continuous environments with guaranteed convergence to local minima.
1. Introduction
Imitation learning replaces difficult cost-function design with expert demonstrations, but behavioral cloning can suffer cascading errors and IRL can be expensive because it repeatedly solves reinforcement-learning problems. The paper therefore directly learns parameterized policies through policy optimization, including model-free methods for high-dimensional observations.
- Imitation learning uses expert demonstrations to distinguish desirable from undesirable outcomes when designing a cost function is difficult.
- Behavioral cloning can accumulate small prediction errors, causing the learner to encounter states unlike those in its training data.This failure mode is associated with cascading errors and covariate shift.
- IRL avoids cascading errors by training on complete expert trajectories and assuming the expert is optimal for an unknown cost function.
- IRL can be extremely expensive in large domains because each iteration typically runs reinforcement learning to evaluate candidate cost functions.
- The paper directly learns policies from expert trajectories, developing policy-gradient methods for high-dimensional observations, including over 600 continuous features.
2. Preliminaries
The paper models an environment with states, actions, dynamics, and an initial-state distribution, while policies assign action probabilities by state. Discounted state visitation distributions then express expected policy cost through state-action occupancy.
- An environment consists of a state space, action space, dynamics model, and initial state distribution, with stationary stochastic policies specifying action probabilities.The framework assumes finite state and action spaces initially but is intended to extend to continuous spaces.
- A cost function maps each state-action pair to a real value, and the discounted state-action value function evaluates future costs under a policy.
- The expected cost of a policy is defined by averaging its state-value function over the initial-state distribution.
- The discounted state visitation distribution sums, over time, the discounted probabilities of reaching each state under the policy.
- State-action visitation is obtained by multiplying state visitation by the policy's action probability, allowing expected cost to be written as an occupancy-weighted cost.
3. Apprenticeship learning
Apprenticeship learning seeks a policy matching or outperforming an expert across an assumed class of cost functions. The paper reformulates this goal for direct optimization over parameterized stochastic policies rather than state-action visitation distributions.
- The learner seeks a policy whose expected true cost is no greater than the expert's, using trajectory samples from the expert.
- Apprenticeship learning assumes the unknown true cost belongs to a specified class and requires the learned policy to perform at least as well as the expert for every cost in that class.
- The discrepancy objective is positive when some cost makes the learner worse than the expert, and nonpositive when the learner satisfies the apprenticeship constraints.
- The framework requires a cost-function class and an optimization algorithm, with the paper focusing on optimization over parameterized stochastic policies.
- Feature expectation matching: Feature expectation matching represents linear costs through feature expectations and seeks a policy whose feature expectations approximately match the expert's.
- Existing apprenticeship methods use approaches including inverse reinforcement learning, multiplicative weights, and linear programming, often involving inner-loop reinforcement learning or visitation-frequency optimization.
- The proposed approach optimizes directly over policy parameters, avoiding a variable for every state-action pair and supporting large spaces while remaining general to cost classes.
4. Policy optimization for apprenticeship learning
The paper directly optimizes apprenticeship learning over smoothly parameterized stochastic policies using model-free policy gradients, including a trust-region variant for more stable updates. The approach alternates between identifying a cost that distinguishes the current policy from the expert and improving the policy, while addressing high variance and scalability to large neural policies.
- Policy optimization: Policy optimization directly solves the apprenticeship objective over a class of smoothly parameterized stochastic policies.The optimization is performed over policy parameters rather than by repeatedly solving a separate planning problem.
- Policy optimization: Gradient-based optimization alternates between finding a cost function attaining the current policy-expert discrepancy and improving the policy with respect to that cost.The cost defines a local reinforcement learning problem that supplies the learning signal for the next policy update.
- Policy gradient: IM-REINFORCE estimates the policy gradient from sampled rollouts and expert trajectories, then updates the policy parameters in the resulting gradient direction.The method uses an empirical cost estimate and discounted future cost sums to form the gradient estimate.
- Policy gradient: The IM-REINFORCE gradient estimator has extremely high variance, causing very slow convergence or divergence for reasonably large step sizes.This is identified as a broader difficulty of policy-gradient reinforcement learning, not only of apprenticeship learning.
- Trust-region policy optimization: Trust-region policy optimization is adapted to apprenticeship learning by constructing a majorizer whose KL-divergence cost is independent of the particular cost function.Replacing the maximum-KL term with an average-KL constraint yields a tractable trust-region subproblem.
- Trust-region policy optimization: IM-TRPO solves an empirically estimated trust-region subproblem with computational requirements not much greater than the corresponding TRPO subproblem.The paper states that this requires no significant additional difficulty over the computation used for IM-REINFORCE when the supremum over the cost class is easily computable.
5. Experiments
Across gridworld, planar navigation, varying-dimensional environments, and highway driving, the experiments evaluate model-free policy optimization against expert behavior and existing methods. The results show strong performance, favorable scaling, and human-like behavior from high-dimensional partial observations.
- Gridworlds: IM-REINFORCE achieved at least 98% of LPAL’s performance with similar sample complexity, while training faster on large gridworlds.For 65536 states, LPAL averaged 10 minutes with high variance, whereas IM-REINFORCE consistently took around 4 minutes.
- Continuous navigation: IM-TRPO consistently achieved zero excess cost in continuous planar navigation, matching optimal trajectories produced using costs learned by CIOC.The excess cost was measured against the expert by averaging over 100 rollouts, despite IM-TRPO being model-free and not using dynamics derivatives.
- Varying dimension: In varying-dimensional environments, IM-TRPO achieved nearly perfect imitation without significant performance degradation as observation dimensionality increased.IM-REINFORCE also learned, but was far outpaced by IM-TRPO.
- Varying dimension: IM-TRPO required 8–9 seconds per iteration, with no statistically significant difference from plain TRPO in the tested environment.This supported the claim that computing the empirical cost for each trust-region step added negligible overhead.
- Highway driving: Using high-dimensional egocentric observations, IM-TRPO learned highway-driving policies whose behavior qualitatively and quantitatively resembled demonstrations.The policies used partial observations rather than full environment state and achieved behavior comparable to CIOC, which used full state features and a full environment model.
6. Discussion and future work
The approach combines policy gradients with local cost-function fitting to train imitation policies in high-dimensional continuous environments without repeatedly solving inner planning problems. Its scope excludes expert interaction, reinforcement signals, and the design of broad cost-function classes.
- Discussion and future work: Policy gradients combined with local cost-function fitting can train neural-network imitation policies in high-dimensional, continuous environments.The demonstrated environments included observations with over 600 continuous features.
- Discussion and future work: The method identifies a locally optimal solution even when optimal planning is out of reach.This provides local optimality guarantees that competing methods may lack when their inner planning problems are only approximately solved.
- Discussion and future work: The approach does not use expert interaction or a reinforcement signal.When either resource is available, alternative approaches may be more sample efficient.
- Discussion and future work: The work focuses on policy optimization rather than designing appropriate cost-function classes.The authors identify nonlinear and larger distinguishing cost classes as an important direction for future work.
Supplementary Material
The supplementary material describes experiment-specific environments, policies, optimization procedures, data collection, and evaluation protocols for gridworld, Waterworld, and Highway tasks.
- Gridworld: Gridworld experiments used tabular softmax policies for IM-REINFORCE, value iteration for Q values, linear programming for LPAL, and lookup policies for behavioral cloning.Unseen states under behavioral cloning received a random action, and timing tests used a 4-core 3.6GHz Intel i7-4790 CPU.
- Waterworld: Waterworld experts were generated with TRPO, each expert produced 25 trajectory samples, and IM-REINFORCE and IM-TRPO ran for 100 imitation iterations.The trajectories were 500 timesteps long.
- Highway: Highway experiments ran IM-TRPO for 500 iterations, collecting 20000 state-action pairs per iteration through simulation.Policies were evaluated using the same measurements as Levine & Koltun, averaged over 50 rollouts.