PaperlayerView exact revision on arXiv

Source-linked AI summary

Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning

Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, Jonathan Tompson

arXiv:1809.02925v2cs.LGstat.ML

TL;DR

AIL methods face reward-function bias and high environment-interaction costs despite learning from few expert demonstrations. DAC combines learned absorbing-state rewards with off-policy discriminator and actor-critic training, and the authors report state-of-the-art performance with substantially fewer samples.

  • Problem

    AIL methods can use biased reward functions and require prohibitively many policy-environment interactions for real-world imitation learning.

  • Method

    DAC combines explicit terminal-state handling, an off-policy discriminator, and an off-policy actor-critic reinforcement-learning algorithm compatible with GAIL and AIRL.

  • Results

    DAC reaches state-of-the-art performance on several standard reinforcement-learning benchmarks and recovers expert policies from significantly fewer samples than recent GAIL work.

  • Takeaways & Limitations

    Learning absorbing-state rewards removes the need to hand-craft discriminator rewards for task properties, while off-policy training improves imitation-learning sample efficiency.

  • Takeaways & Limitations

    The authors found that omitting importance weights worked well in practice because density estimation and discriminator updates could be challenging and high-variance.

Abstract

from arXiv · show

We identify two issues with the family of algorithms based on the Adversarial Imitation Learning framework. The first problem is implicit bias present in the reward functions used in these algorithms. While these biases might work well for some environments, they can also lead to sub-optimal behavior in others. Secondly, even though these algorithms can learn from few expert demonstrations, they require a prohibitively large number of interactions with the environment in order to imitate the expert for many real-world applications. In order to address these issues, we propose a new algorithm called Discriminator-Actor-Critic that uses off-policy Reinforcement Learning to reduce policy-environment interaction sample complexity by an average factor of 10. Furthermore, since our reward function is designed to be unbiased, we can apply our algorithm to many problems without making any task-specific adjustments.

1 INTRODUCTION

Adversarial Imitation Learning can learn robust imitation policies from few demonstrations but may require many environment interactions and use reward functions with task-dependent bias. DAC addresses these issues with off-policy learning, explicit terminal handling, and an off-policy discriminator.

  • 1 INTRODUCTION: AIL algorithms combine expert demonstrations, an adversarial discriminator, and reinforcement learning to imitate expert actions.GAIL uses discriminator rewards with policy gradients, while AIRL modifies the GAIL discriminator to recover a reward.
  • 1 INTRODUCTION: 25 million policy frame transitions may be required for GAIL convergence, while replacing TRPO with PPO can reduce this to approximately 10 million.The remaining interaction cost is described as intractable for many robotics and real-world applications.
  • 1 INTRODUCTION: Reward functions in AIL can impose bias: strictly positive rewards hinder minimal-step solutions, whereas strictly negative rewards cannot represent survival bonuses.These choices may require prior knowledge of the true environment reward when applying GAIL or AIRL.
  • 1 INTRODUCTION: DAC combines explicit terminal-state handling, an off-policy discriminator, and an off-policy actor-critic algorithm, while remaining compatible with GAIL and AIRL.The method is presented as a new algorithm for addressing reward bias and interaction sample complexity.
  • 1 INTRODUCTION: DAC addresses discriminator-reward bias and reduces agent-environment interactions through an off-policy variant of AIL.The authors also report robustness to noisy, multi-modal, and constrained human demonstrations on non-trivial robotic tasks.

2 RELATED WORK

Related work frames imitation learning through Behavioral Cloning, Inverse Reinforcement Learning, occupancy matching, and adversarial methods. DAC is positioned as complementary to observation-based imitation approaches while targeting AIL reward and sampling issues.

  • 2 RELATED WORK: Behavioral Cloning and Inverse Reinforcement Learning are two broad approaches to imitation learning.IRL estimates a reward function and then applies reinforcement learning to recover the policy.
  • 2 RELATED WORK: GAIL treats imitation learning as occupancy matching and learns a policy from demonstrations without recovering the expert’s reward function.Later work extends the framework for stability, robustness, and connections to model-based imitation learning.
  • 2 RELATED WORK: Expert demonstrations and template trajectories have been used to initialize reinforcement-learning policies and guide exploration under sparse rewards.Prior work also studies off-policy learning from demonstrations as a way to accelerate reinforcement learning.
  • 2 RELATED WORK: DAC is most closely related to AIL algorithms such as GAIL and AIRL, while remaining complementary and compatible with an observation-based method that removes dependence on actions.The cited observation-based approach relies on weaker assumptions than state-action-state′ transition tuples.

3 BACKGROUND

The paper formulates imitation-learning environments as Markov Decision Processes and uses absorbing states to represent finite episodes. It introduces GAIL’s discriminator-based reward learning and motivates adapting the framework for off-policy reinforcement learning.

  • 3 BACKGROUND: The environment is modeled as an MDP containing state and action spaces, initial-state distribution, transition dynamics, reward function, and discount factor.These components are represented by the tuple (S, A, p(s), p(s′|s, a), r(s, a, s′), γ).
  • 3 BACKGROUND: Absorbing states extend the notation from continuing to episodic tasks by following terminal transitions, receiving zero reward, and transitioning to themselves.This lets returns be expressed using a common notation for finite and unbounded trajectories.
  • 3 BACKGROUND: Absorbing states are often implicitly assigned zero reward in imitation-learning and IRL algorithms, but DAC learns potentially non-zero absorbing-state rewards.The paper identifies proper absorbing-state handling as especially important when rewards are learned.
  • 3 BACKGROUND: GAIL trains a discriminator D(s, a) to distinguish expert transitions from transitions generated by the policy, producing a learned reward signal for imitation.The learned reward may not equal the true reward but can support matching the expert and policy occupancy measures.
  • 3 BACKGROUND: Because GAIL requires state-action pairs sampled from the learned policy, adapting it to off-policy reinforcement learning requires modifications to the standard formulation.DAC presents an off-policy discriminator formulation for this purpose.

4 DISCRIMINATOR-ACTOR-CRITIC

DAC addresses reward bias and sample inefficiency in adversarial imitation learning by unbiasing rewards, handling absorbing states explicitly, and using off-policy training. The approach combines learned terminal-state rewards with replay-buffer discriminator training and off-policy actor-critic reinforcement learning.

  • Algorithm overview: DAC combines reward-function unbiasing with an off-policy discriminator formulation of adversarial imitation learning.The algorithm is designed to work with both GAIL and AIRL frameworks.
  • Reward bias: Positive GAIL rewards can create an implicit survival bonus, while negative rewards cannot emulate survival bonuses and may require task-specific priors.These biases can make performance depend on the chosen reward form rather than robust reward recovery.
  • Reward bias: In the illustrative MDP, the GAIL reward may prefer a looping policy over the expert’s shortest path unless γ and task-dependent rewards satisfy a restrictive inequality.With γ = 0.99, the loopy trajectory is disfavored only if r(s1, a1→2) < 0.0199 · r(s2, a2→g), but the discriminator values given do not satisfy this condition.
  • Reward bias: The alternative reward log(D(s, a)) penalizes every step and can cause collapse in environments with a survival bonus, while AIRL can produce sub-optimal policies when terminal rewards are fixed at zero.These examples motivate a reward formulation that does not require manually selecting task-specific reward biases.
  • Unbiasing reward functions: DAC learns rewards for absorbing states and represents them with an indicator dimension so the discriminator can distinguish desirable episode termination.Terminal returns can also be derived analytically, but the text reports that this alternative was much less stable.
  • Addressing sample inefficiency: To reduce sample inefficiency, DAC trains the discriminator from replay-buffer transitions and uses GAIL rewards with TD3, updating rewards when transitions are sampled.Replay-buffer training matches the mixture of policy distributions represented in the buffer; importance sampling is omitted because density estimation is difficult and discriminator updates can have high variance.

5 EXPERIMENTS

Experiments evaluate DAC on MuJoCo benchmarks and PyBullet robotic tasks with human demonstrations. Results show improved sample efficiency, reward-bias effects from discriminator rewards, and the importance of absorbing-state handling.

  • Experimental setup: DAC was evaluated on MuJoCo continuous-control benchmarks and newly defined PyBullet robotic tasks using human demonstrations.The robotic tasks involve a Kuka IIWA arm and address reaching or pushing blocks.
  • Experimental setup: DAC uses gradient penalties to stabilize the discriminator, particularly when off-policy training can cause overfitting.The discriminator is regularized with gradient penalties.
  • 5.1 OFF POLICY DAC ALGORITHM: DAC reaches comparable expected reward with orders of magnitude fewer environment steps and outperforms GAIL within a 1 million step threshold.On the top-left plot, DAC is described as an order of magnitude more sample efficient than TRPO- and PPO-based GAIL.
  • 5.2 REWARD BIAS: A fixed, untrained GAIL discriminator outputting 0.5 achieved around 1000 episode reward on Hopper, approximately one third of expert performance.This occurred without reward learning or expert demonstrations and surpassed behavioral cloning.
  • 5.2 REWARD BIAS: Without absorbing-state handling, the GAIL reward −log(1 − D(s, a)) completely failed to recover the expert from 600 unsampled expert trajectories.DAC instead quickly imitated noisy and potentially sub-optimal human demonstrations.
  • 5.2 REWARD BIAS: AIRL experiments compare learning versus not learning absorbing-state rewards using the Fu et al. discriminator structure with TD3.The experiments address incorrect per-frame penalties and early termination.

6 CONCLUSION

The conclusion presents DAC as addressing reward bias and policy-transition sample inefficiency in adversarial imitation learning. Its approach combines learned absorbing-state rewards with off-policy discriminator and reinforcement-learning training.

  • 6 CONCLUSION: DAC addresses reward bias by learning rewards for absorbing states instead of hand-crafting task-specific discriminator rewards.This is intended to address biases that can impose prior knowledge or prevent imitation of the optimal expert.
  • 6 CONCLUSION: DAC improves sample efficiency through off-policy discriminator training and an off-policy reinforcement-learning algorithm.The conclusion reports significantly fewer samples than recent GAIL work.
  • 6 CONCLUSION: DAC reaches state-of-the-art imitation-learning performance on several standard reinforcement-learning benchmarks.The conclusion also reports recovery of the expert policy with significantly fewer samples than recent GAIL work.

A DAC ALGORITHM

The DAC algorithm organizes expert and policy experience in replay buffers, handles terminal transitions with absorbing states, updates a discriminator, and trains the policy with TD3.

  • Algorithm 1: DAC initializes and maintains replay-buffer data for expert and policy trajectories.The algorithm takes an expert replay buffer as input and updates a policy replay buffer during training.
  • Algorithm 1: Expert rollouts are wrapped with absorbing states before being used by the algorithm.Terminal expert transitions are extended with an absorbing-state transition when the final state is terminal.
  • Algorithm 1: The discriminator is updated using GAN with gradient penalty during each training cycle.The supplied algorithm passage specifies a GAN+GP discriminator update.
  • Algorithm 1: The policy is updated with TD3 using the current reward estimate.The algorithm repeatedly uses the current reward estimate before updating πθ with TD3.

B SUPPLEMENTARY RESULTS ON MUJOCO ENVIRONMENTS

Figure 8 compares algorithms using the same number of expert demonstrations, with normalized reward scaled from random-policy to expert-policy performance.

  • Figure 8: Figure 8 compares different algorithms under the same number of expert demonstrations.The figure’s y-axis is normalized reward, where 0 denotes a random policy and 1 denotes an expert policy.

C KUKA-IIWA SIMULATED ENVIRONMENT

Figure 9 depicts the simulated Kuka-IIWA environment and how human demonstrations are recorded for Kuka-Reach and Kuka-PushNext tasks.

  • Kuka-IIWA environment: A VR headset and 6DOF controller let a human participant control the robot end-effector pose to record expert demonstrations.The setup is used to collect demonstrations in the Kuka-IIWA environment.
  • Kuka-Reach: Kuka-Reach requires moving the gripper to one of three blocks, with the task encoded one-hot in the state.The task identity is represented in the state by a 1-hot encoding.
  • Kuka-PushNext: Kuka-PushNext requires using the gripper to push one block next to another.The task specifies a block-to-block pushing objective.
Loading 1809.02925v2…