Source-linked AI summary
Model-based Adversarial Imitation Learning
Nir Baram, Oron Anschel, Shie Mannor
TL;DR
Imitating an expert from demonstrations without its reward signal is challenging, and model-free adversarial imitation relies on high-variance gradient estimates. The paper uses a forward model to make adversarial imitation differentiable and reports the highest reward across dataset sizes, comparable to the expert.
Problem
The paper studies how to train a policy to imitate expert demonstrations without access to the expert’s reward signal.
Source: §1 · p. 1Method
The proposed adversarial imitation approach uses a forward model to make the system differentiable and derive policy gradients from the discriminator.
Source: §1 · p. 2, §3.2 · p. 5Results
MAIL achieves the highest total cumulative reward across dataset sizes while performing comparably to the expert.
Source: §4 · p. 6Takeaways & Limitations
The approach offers relatively few environment interactions and fewer hyperparameters than the model-free alternative, while using discriminator partial derivatives for policy gradients.
Source: §1 · p. 2, §5 · p. 8Changing state distributions can make training difficult, and the reported behavioral-cloning initialization and learning-rate restart attempts did not succeed significantly.
Source: §4.2 · p. 8
Abstract
from arXiv · showhide
Generative adversarial learning is a popular new approach to training generative models which has been proven successful for other related problems as well. The general idea is to maintain an oracle $D$ that discriminates between the expert's data distribution and that of the generative model $G$. The generative model is trained to capture the expert's distribution by maximizing the probability of $D$ misclassifying the data it generates. Overall, the system is \emph{differentiable} end-to-end and is trained using basic backpropagation. This type of learning was successfully applied to the problem of policy imitation in a model-free setup. However, a model-free approach does not allow the system to be differentiable, which requires the use of high-variance gradient estimations. In this paper we introduce the Model based Adversarial Imitation Learning (MAIL) algorithm. A model-based approach for the problem of adversarial imitation learning. We show how to use a forward model to make the system fully differentiable, which enables us to train policies using the (stochastic) gradient of $D$. Moreover, our approach requires relatively few environment interactions, and fewer hyper-parameters to tune. We test our method on the MuJoCo physics simulator and report initial results that surpass the current state-of-the-art.
1 Introduction
The section frames imitation learning as training a policy from expert demonstrations without the expert’s reward signal, then introduces a forward-model-based adversarial approach. It contrasts this approach with behavioral cloning and reward-recovery methods, and with model-free adversarial imitation.
1 Introduction
- The goal is to imitate an expert policy from its sample trajectories without access to the expert’s reward signal.Imitation is motivated by automation, distillation, and initialization needs.Source: §1 · p. 1
- Behavioral cloning learns p(a|s) from single-step state-action pairs, but ignoring environment dynamics makes it vulnerable to compounding errors and often demands substantial expert data.Its dense supervision avoids the credit-assignment difficulty of reinforcement learning.Source: §1 · p. 1
- Inverse reinforcement learning seeks a reward signal before standard reinforcement learning, but reward recovery is ill-posed and synthesizing an expert-like reward requires domain expertise.The second phase maximizes discounted expected return after a reward is obtained.Source: §1 · pp. 1–2
- Generative adversarial imitation uses a discriminator to distinguish expert data from generated data, and model-free GAIL applied this approach to imitation learning.GAIL was reported to alleviate sample-complexity and compounding-error problems associated with imitation learning.Source: §1 · p. 2
- A forward model makes the proposed adversarial imitation system differentiable, allowing policy gradients from the discriminator and requiring relatively few environment interactions and fewer hyperparameters.The algorithm processes entire trajectories to minimize the sum of discriminator probabilities along each path.Source: §1 · p. 2
2 Background
The background frames imitation learning as policy learning from demonstrations and explains how adversarial training uses a discriminator to compare generated and expert data. It also describes why model-free policy gradients are difficult to optimize.
2.1 Markov Decision Process
- An infinite-horizon discounted Markov decision process specifies states, actions, transitions, rewards, an initial-state distribution, and a discount factor.Source: §2.1 · p. 2
2.2 Imitation Learning
- A learner’s slight behavioral deviation can change its future state distribution from the expert’s, resulting in compounding errors.Source: §2.2 · p. 2
2.3 Generative Adversarial Learning
- In adversarial learning, a discriminator distinguishes expert data from generated data while the generator learns to fool it and approximate the expert distribution.Source: §2.3 · p. 3
- Because the policy affects the discriminator objective indirectly, model-free optimization uses likelihood-ratio gradients, whose REINFORCE estimates have high variance.Source: §2.3 · p. 3
- A forward model lets the system use discriminator partial derivatives for end-to-end backpropagation, and the model-based approach can reduce environment-interaction demands.Source: §2.3 · p. 4
3 Algorithm
The algorithm interprets the discriminator as reflecting both policy and state-distribution likelihood ratios. MAIL uses a reparameterized stochastic policy and a forward model to propagate discriminator gradients through actions and future states.
3 Algorithm
- The discriminator reflects both policy likelihood ratios and state-distribution likelihood ratios, capturing action behavior and the states induced by each policy.The paper interprets these as the discriminator's two questions: how likely a state is under expert versus learner, and how likely an action is under each policy given that state.Source: §3 · p. 4
3.1 Re-parametrization of distributions (for using ∇aD)
- Reparameterizing the stochastic policy permits derivatives through stochastic models and a Monte Carlo estimate of the expected discriminator gradient.For a Gaussian policy, the action is rewritten as μθ(s) + ξσθ(s), with ξ drawn from a standard normal distribution.Source: §3.1 · p. 5
3.2 Forward model (for using ∇sD)
- The forward model expresses the next state as s′ = f(s, a), allowing policy actions to affect discriminator gradients through state changes.Considering multi-step transitions accounts for future-state distribution errors in earlier policy decisions.Source: §3.2 · p. 5
3.3 MAIL Algorithm
- MAIL minimizes discriminator probabilities along a trajectory and computes the policy gradient recursively from the final step back to the initial step.The algorithm uses a forward model and multi-step transitions; its gradients are applied recursively across time.Source: §3.3 · pp. 5–6
4 Experiments
MAIL achieves the highest cumulative reward among the compared algorithms for every dataset size, with performance comparable to the expert. The experiments also describe a training challenge arising from changing data distributions and report that attempted remedies did not significantly succeed.
4 Experiments
- The experiments use Hopper and Walker tasks in MuJoCo, with TRPO as the expert and four demonstration datasets per task containing 25, 18, 11, or 4 trajectories.Each trajectory has length 1,000; the tasks involve complex second-order dynamics and direct torque control.Source: §4 · p. 6
- MAIL achieves the highest cumulative reward for every dataset size, with performance comparable to the expert.Table 1 compares total cumulative reward for BC, GAIL, and MAIL; BC and GAIL results are reported from Ho and Ermon (2016).Source: §4 · p. 6, Table 1 · p. 8
Algorithm 1 Model-based Adversarial Imitation Learning
- Algorithm 1 collects transitions in an experience buffer, trains the forward model and discriminator, then computes policy gradients backward through time.The gradient recursion proceeds from t = T to t = 0.Source: Algorithm 1 Model-based Adversarial… · p. 7
4.2 The Changing Distribution Problem
- Changing policy data distributions affect both the discriminator and forward model, and BC initialization or learning-rate resets did not significantly resolve the training difficulty.The authors note that further research is needed; early examples can strongly influence the final solution under a decreasing learning rate.Source: §4.2 · pp. 7–8
5 Discussion
The discussion presents MAIL's use of discriminator partial derivatives as its main advantage, while noting dependence on a learned forward model. It also reports training configuration findings and possible uses for discriminator information.
5 Discussion
- MAIL uses discriminator partial derivatives to calculate the policy gradient, but depends on a forward model whose inaccuracy can produce noisy gradients and impede convergence.The paper also notes relatively few environment interactions and fewer hyperparameters than the model-free approach, while warning that learning the forward model may be difficult in some problems.Source: §5 · p. 8
- The authors report that the discriminator should be about twice the policy's size and trained with a large, slowly decaying learning rate, while the policy should train about three times faster.They also report that adding noise to expert data helps convergence, especially with few expert examples.Source: §5 · p. 8
- Discriminator outputs indicate where the policy resembles the expert, which the authors suggest using to prioritize training examples or assess policy confidence at inference.These are proposed uses beyond the discriminator's traditional role.Source: §5 · p. 8