InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations

1 Introduction

• Reinforcement learning optimizes a predefined reward, which can be difficult to design for complex settings such as autonomous driving.Driving may require balancing safety, comfort, and efficiency.Source: §1 · p. 1
• GAIL imitates expert trajectories by jointly training a stochastic policy and a discriminator to distinguish generated from expert behavior.Imitation learning instead learns directly from expert demonstrations, without requiring an explicit reward signal.Source: §1 · p. 1
• Human demonstrations vary across experts and can reflect latent factors such as skill and habits that the environment does not explicitly capture.The paper aims to discover and disentangle these factors without supervision, analogously to learning interpretable factors in image generation.Source: §1 · p. 1
• The method extends GAIL with mutual-information maximization so high-level latent variables can control low-level actions and represent distinct behaviors.It also extends GAIL to raw-pixel inputs and demonstrates the approach in TORCS highway driving.Source: §1 · p. 2

2.1 Preliminaries

• An infinite-horizon discounted MDP is specified by state and action spaces, transitions, reward, initial-state distribution, and discount factor.The expert policy is available through demonstrations consisting of state-action sequences.Source: §2.1 · p. 2

2.2 Imitation learning

• Behavior cloning learns from expert state-action pairs but can generalize poorly due to compounding errors and covariate shift; apprenticeship learning recovers a reward but nests an expensive RL problem.Apprenticeship learning can use its learned reward to score trajectories.Source: §2.2 · p. 2

2.3 Generative Adversarial Imitation Learning

• GAIL matches generated and expert state-action distributions using a discriminator, avoiding direct reward estimation while optimizing the policy with RL techniques.Its objective uses Jensen-Shannon divergence and causal-entropy regularization; training alternates discriminator updates with TRPO policy updates because the environment is treated as a black box.Source: §2.3 · p. 3

3 Interpretable Imitation Learning through Visual Inputs

• InfoGAIL targets unsupervised discovery of salient behavioral variation and policies that produce trajectories corresponding to those factors, using visual inputs as the only external perceptual information.The motivation includes variation across experts and distinct decisions by the same expert.Source: §3 · p. 3

3.1 Interpretable Imitation Learning

• GAIL alone lacks an incentive to separate behaviors in mixed-expert demonstrations, so InfoGAIL conditions the policy on a latent code and regularizes mutual information with trajectories.A variational lower bound uses an approximate posterior; this encourages the latent code to represent salient trajectory factors while adversarial training imitates expert behavior.Source: §3.1 · pp. 3–4

3.2 Reward Augmentation

• Collision and off-road penalties significantly improve the learned driving policy’s average rollout distance by adding a surrogate reward alongside imitation.The surrogate state-based reward expresses a bias about desired behavior; the paper frames this as a hybrid of imitation and reinforcement learning.Source: §3.2 · p. 5

3.3 Improved Optimization

• For high-dimensional inputs such as images, the method uses Wasserstein GAN training to address vanishing gradients and mode collapse in complex trajectory distributions.The training procedure also uses variance-reduction techniques and separate discriminator and posterior networks because WGAN training constraints would interfere with an expressive shared posterior network.Source: §3.3 · p. 5

4.2 Utilizing Raw Visual Inputs via Transfer Learning

• The driving policy used raw visual inputs, with ImageNet-pretrained ResNet features transferred to the task because visual inputs make policy learning sample-intensive.The policy also received auxiliary information, including velocity, the two previous actions, and car damage.Source: §4 · p. 5, §4.2 · p. 6, §4.3 · p. 6

4.1 Learning to Distinguish Trajectories

• In the synthetic 2D task, InfoGAIL distinguished and imitated all three trajectory modes, while behavior cloning deviated and GAIL failed to separate them.The demonstrations were unlabeled, and the methods used the same architecture; only InfoGAIL regularized the latent code.Source: §4.1 · p. 6

4.3 Network Structure

• The policy combines convolutional visual features with auxiliary information and, for InfoGAIL, the latent code; the discriminator scores image, auxiliary, and action inputs, while Q estimates the latent code.Q uses a softmax output for discrete latent variables or a factored Gaussian for continuous ones.Source: §4.3 · p. 7

4.4 Interpretable Imitation Learning from Visual Demonstrations

• The driving experiments targeted two modes in each task: inside- versus outside-lane turns and left- versus right-side passing, using 80 expert trajectories per task.The two-state latent code had a uniform prior; a constant per-step reward encouraged the car to stay alive.Source: §4.4 · p. 7
• InfoGAIL separated the two driving modes, correctly classifying over 81% of pass state-action pairs and almost all turn pairs; average rollout distance rose during training.Original GAIL learned pass trajectories as a whole and did not distinguish the two modes.Source: §4.4 · pp. 7–8

4.5 Ablation Experiments

• Removing the replay buffer significantly worsened average rollout distance, while removing reward augmentation or WGANs yielded performance slightly below the expert.The authors attribute the replay-buffer result to increased variance in gradient estimation.Source: §4.5 · p. 8

5 Related work

• Mediated-perception systems first extract scene information, while behavior-reflex systems map visual inputs directly to driving actions.The paper notes that many current autonomous-driving methods use the mediated approach and hand-crafted features.Source: §5 · p. 9
• A prior visual imitation method uses behavior cloning and is described as generalizing poorly to lane-changing or vehicle-passing tasks, unlike this paper’s GAIL approach.Another TORCS approach uses DAgger, which queries reference policies during training.Source: §5 · p. 9
• Many end-to-end driving methods rely on LIDAR-like distance measurements, whereas this paper’s policy requires only raw visual information as external input.Source: §5 · p. 9
• Prior work uses pre-trained networks to learn dense reward functions, transferring over the critic rather than the policy as in this paper’s approach.The authors suggest combining those rewards with their approach through reward augmentation.Source: §5 · p. 9

6 Conclusion

• The method imitates complex behaviors while identifying salient latent factors of variation in demonstrations.Source: §6 · p. 9
• Discovering those factors requires no direct supervision beyond expert demonstrations, and training uses standard policy optimization algorithms.Source: §6 · p. 9
• In TORCS, the method automatically distinguishes certain human driving behaviors and learns a policy that can imitate and even outperform human experts using visual information as its sole external input.Source: §6 · p. 9

A Details for Experiment in Section 4.1

• The 2D synthetic experiment uses 10-dimensional observations and 2-dimensional actions, with InfoGAIL initialized from behavior cloning.Source: A Details for Experiment in Section 4.1 · p. 12
• Training uses Wasserstein GANs, and the experiment omits replay buffers because rollouts are cheap in this setting.Source: A Details for Experiment in Section 4.1 · p. 12
• Figure 4 depicts the network architecture for the 2D synthetic environment.Source: Figure 4 · p. 12

B Architectural Details For TORCS Experiment

• Lower-resolution discriminator inputs improve training speed without performance loss because high-level action inference does not require fine-grained image details.Source: B Architectural Details For TORCS… · p. 12
• Figure 5 depicts the network architecture for the driving experiments.Source: Figure 5 · p. 12

• The algorithm takes expert trajectories and initial policy, discriminator, and posterior parameters, and returns a learned policy.Source: C Algorithm Outline for InfoGAIL · p. 13
• Each iteration samples a latent code, rolls out the policy with that code fixed, adds trajectories to a replay buffer, and samples generated and expert batches.Source: C Algorithm Outline for InfoGAIL · p. 13
• The algorithm clips discriminator weights to [−0.01, 0.01] and updates posterior parameters by descending gradients.Source: C Algorithm Outline for InfoGAIL · p. 13
• It takes a TRPO policy step using an objective specified with or without reward augmentation.Source: C Algorithm Outline for InfoGAIL · p. 13

• Figure 6 presents visual inputs for the InfoGAIL policies in the pass and turn experiments.Source: Figure 6 · p. 13
• The latent code controls the behavior of the policies shown.Source: Figure 6 · p. 13
• The figure illustrates learned policies under different latent codes.Source: §D.1 · p. 13, Figure 6 · p. 13

D.2 InfoGAIL Policies for Ablation Experiments

• The agent is designed to drive around 80 km/h, so its control must be very precise.Source: §D.2 · p. 13
• During the ablation rollout, the policy passes several other cars.Source: Figure 7 · p. 14

D.3 Failure Case for Reward Augmentation

• Excessively high reward for avoiding collision causes the agent to repeatedly turn to increase friction and slow down dramatically.Source: §D.3 · p. 14