Sample-Efficient Imitation Learning via Generative Adversarial Nets

1 Introduction

• Deep reinforcement learning uses neural function approximators to reduce the need for hand-crafted features in continuous control tasks.The introduction presents this as reducing a customary preliminary engineering workload.Source: §1 · p. 1
• Reward shaping can demand extensive trial-and-error engineering and leave systems prone to premature convergence to local minima.Imitation learning avoids the preliminary step of hand-crafting a reinforcement signal by learning from expert demonstrations.Source: §1 · p. 1
• Behavioral cloning can become fragile when agents stray from expert trajectories, as compounding errors produce covariate shift (Ross et al., 2011).It fits demonstrations alone without gathering new simulation data, but its predictions worsen as the agent departs from those trajectories.Source: §1 · p. 1
• GAIL's sample inefficiency is attributed to on-policy learning, which discards transitions and relies on high-variance Monte Carlo estimates.Retaining past trajectories for repeated updates can instead produce destructively large policy updates.Source: §1 · p. 2
• The proposed method reuses past experiences in an off-policy procedure to substantially reduce interactions needed to learn imitation policies.The authors report consistent stability despite combining adversarial training and an actor-critic method.Source: §1 · p. 2

2 Related Work

• Behavioral cloning regresses on demonstrations, while apprenticeship learning recovers an assumed expert reward and then runs reinforcement learning.Inverse reinforcement learning can be computationally costly because it models the reward and solves the resulting reinforcement-learning problem each iteration.Source: §2 · p. 2
• GAIL learns a surrogate similarity reward and uses it in an inner-loop reinforcement-learning procedure, but leaves sample inefficiency unresolved.The paper identifies that limitation as the focus of its extension.Source: §2 · p. 2
• Unlike prior model-based gradient recovery, the method uses deterministic policy gradients to exploit discriminator gradients with respect to actions without gradient-estimation tricks.In the model-free setting, states remain stochastic nodes and block backward gradient flow.Source: §2 · p. 3
• The method gains sample efficiency without preprocessing demonstrations, unlike a similar approach that uses preliminary preprocessing.The authors report gains by orders of magnitude.Source: §2 · p. 3

3 Background

• The agent learns from expert trajectories without direct action supervision or an environmental reinforcement signal.The trajectories guide learning in the environment.Source: §3 · p. 3
• The paper models sequential interaction as an MDP with state and action spaces, initial-state distribution, transition dynamics, reward, and discount factor.Experiments use fully observable episodic MDPs, with the discount set to zero at episode termination.Source: §3 · p. 3
• A policy is parameterized by a neural network, and the paper distinguishes the reward-free MDP M− from M+, which adds the learned surrogate reward.Trajectories are sequences of state, action, reward, and next-state transitions.Source: §3 · p. 3
• GAIL trains a discriminator to distinguish policy and expert state-action pairs while the policy tries to fool it, using the resulting signal as a reward proxy.The background describes alternating discriminator and policy optimization steps.Source: §3 · pp. 3–4
• Actor-critic methods alternate policy evaluation, which estimates state-action values with a critic, and policy iteration, which optimizes the policy against that estimate.The critic can be trained with Monte Carlo estimation or temporal-difference learning.Source: §3 · p. 4

• A deterministic policy is trained with an off-policy actor-critic, while parameter perturbations and temporally correlated Ornstein-Uhlenbeck noise provide exploration.The adversarial reward is trained on samples from the parameter-perturbed policy, which the authors report did not cause severe divergence and seemed to help training.Source: §4 · pp. 4, 6
• Sam links adversarial reward-policy training with actor-critic policy-critic training, while sealing critic-to-reward gradients to prevent unintended optimization.The policy participates in both optimization problems, while the reward and critic provide signals for the desired behavior.Source: Figure 1 · p. 4, §4 · p. 4
• Training cycles through environment interaction, replay-buffer storage, reward updates on policy and replay samples mixed with expert data, and policy-critic updates from replay.The reward module is updated with equal mixtures of expert and agent state-action pairs, using both newly collected and replayed experience.Source: §4 · pp. 4–5
• The reward module acts as a discriminator; Sam uses the negative generator loss as synthetic reward and applies a discriminator-gradient penalty to improve stability.The reward is updated first on the newest policy mini-batch, then on replay-buffer mini-batches.Source: §4 · p. 5
• The critic combines one-step and n-step Bellman residuals with weight decay, and separate slowly tracking target networks help prevent divergence.Uniform replay is used rather than prioritized replay, which the authors report did not consistently yield well-behaved policies.Source: §4 · p. 5
• N-step returns improved sample complexity and stability, whereas prioritized replay prevented consistent learning of well-behaved policies.The authors conjecture that learned rewards amplify prioritized replay's tendency to oversample transitions with erroneous priorities.Source: §4 · p. 5
• Sam uses the critic-based policy gradient for its delayed-reward tasks; mixing in the direct reward gradient impaired reward propagation across timesteps.The direct reward estimate reflects immediate similarity, whereas TD-trained Q values propagate rewards along trajectories.Source: §4 · p. 6

5 Results

• Experiments cover InvertedPendulum, InvertedDoublePendulum, Reacher, Hopper, and Walker2d, with experts trained using PPO for 10M timesteps.The agents were trained in MuJoCo physics-based environments; demonstration quantities varied by environment.Source: §5 · p. 6
• One or two orders of magnitude fewer environment interactions let Sam attain asymptotic expert performance than GAIL.Figure 2 reports this comparison across continuous-control tasks.Source: Figure 2 · p. 7, §5 · p. 8
• Figure 2 plots episodic return against logarithmic environment interactions; results use four seeds, with expert-dataset return statistics also shown.Solid blue and green lines are mean returns, and filled areas indicate confidence intervals; comparisons used the same selected trajectories and random seeds.Source: Figure 2 · p. 7, §5 · p. 8
• Sam uses Temporal Difference policy evaluation and replays previously collected transitions, requiring only a couple of new transitions per iteration.GAIL instead uses full interaction traces for Monte Carlo estimates and discards experienced transitions after updates.Source: §5 · p. 8
• GAIL sometimes fails to reach demonstrator asymptotic performance, including in Reacher and InvertedDoublePendulum.Source: §5 · p. 8
• The interaction savings generally come with longer wall-clock training because Sam replays past experience.The authors state that reducing real-world interactions can be more desirable for safety and cost reasons.Source: §5 · p. 8

6 Conclusion

• Sam is introduced as a method that addresses GAIL’s required agent–environment interactions.Source: §6 · p. 8
• Sam reduces the interactions needed to learn expert-like policies by an order of magnitude, and sometimes more.Source: §6 · p. 8
• The authors identify the off-policy procedure as key to Sam’s interaction savings.Source: §6 · p. 8