PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning from Imperfect Demonstrations

Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, Trevor Darrell

arXiv:1802.05313v2cs.AIcs.LGstat.ML

TL;DR

Existing approaches struggle to combine reinforcement learning with noisy demonstrations because supervised imitation and reward-based learning use divergent losses. The paper proposes Normalized Actor-Critic, which uses a unified normalized-Q objective for demonstrations and environmental interaction. NAC robustly outperforms prior methods across driving tasks while tolerating imperfect demonstrations.

  • Problem

    Existing methods combine supervised learning from demonstrations with reinforcement learning from rewards, but their divergent losses are difficult to jointly optimize and can be sensitive to noisy data.

  • Method

    Normalized Actor-Critic uses a unified reinforcement-learning objective and normalizes the Q-function to reduce values for actions unseen in demonstrations.

  • Results

    NAC outperforms previous approaches on driving tasks with modest demonstrations while tolerating significant demonstration noise.

  • Takeaways & Limitations

    NAC can learn from imperfect demonstrations and then be refined with rewards by continuing the same algorithm on on-policy environmental data.

Abstract

from arXiv · show

Robust real-world learning should benefit from both demonstrations and interactions with the environment. Current approaches to learning from demonstration and reward perform supervised learning on expert demonstration data and use reinforcement learning to further improve performance based on the reward received from the environment. These tasks have divergent losses which are difficult to jointly optimize and such methods can be very sensitive to noisy demonstrations. We propose a unified reinforcement learning algorithm, Normalized Actor-Critic (NAC), that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data. NAC learns an initial policy network from demonstrations and refines the policy in the environment, surpassing the demonstrator's performance. Crucially, both learning from demonstration and interactive refinement use the same objective, unlike prior approaches that combine distinct supervised and reinforcement losses. This makes NAC robust to suboptimal demonstration data since the method is not forced to mimic all of the examples in the dataset. We show that our unified reinforcement learning algorithm can learn robustly and outperform existing baselines when evaluated on several realistic driving games.

1. Introduction

The paper addresses inefficient and fragile reinforcement learning by proposing NAC, a unified objective that learns from imperfect demonstrations and environmental rewards. NAC is evaluated across several environments and outperforms prior approaches while tolerating noisy demonstrations.

  • RL often requires many environment interactions, and initial performance can be nearly random or dangerous in settings such as autonomous driving.
  • Prior demonstration-based approaches commonly assume near-optimal, relatively small-scale, noise-free demonstrations or combine separate supervised and reinforcement losses.
  • NAC uses one unified reinforcement-learning loss for both offline demonstrations and online environmental experience, without assuming demonstration optimality.
  • The method is evaluated in Minecraft, Torcs, and GTA V using tabular or neural-network Q-functions, including raw image input.
  • NAC outperforms previous approaches on driving tasks with modest demonstrations while tolerating significant demonstration noise and using rewards rather than simple imitation.
  • The paper presents NAC as a unified objective that outperforms methods including those with explicit supervised imitation losses.

2. Preliminaries

The preliminaries formulate reinforcement learning as policy optimization in a Markov decision process and introduce maximum-entropy reinforcement learning. Soft Q-learning and policy-gradient formulations incorporate entropy, bootstrapped Q-values, and baselines.

  • The reinforcement-learning problem is modeled as a Markov decision process with states, actions, rewards, transitions, and a discount factor.
  • Standard reinforcement learning seeks a policy that maximizes future discounted reward.
  • Maximum Entropy Reinforcement Learning: Maximum-entropy policy learning augments rewards with entropy, optimizing discounted future reward and action-distribution entropy over the trajectory.
  • Maximum Entropy Reinforcement Learning: The entropy weighting term α balances the importance of entropy, which can improve exploration in multimodal problems and connect Q-learning with actor-critic methods.
  • Soft Q-Learning and Policy Gradient: The policy-gradient formulation includes a term based on the gradient of log policy weighted by the Q estimate minus a baseline, plus an entropy gradient.
  • Soft Q-Learning and Policy Gradient: Soft Q-learning uses a bootstrapped estimate ˆQ(s, a) = R(s, a) + γV_Q(s′), where V_Q is computed from Q_θ.

3. Robust Learning from Demonstration and Reward

NAC is a unified reinforcement-learning approach that applies normalized actor-critic updates to demonstrations and environment interactions. Its Q-function normalization suppresses unsupported actions, helping the method learn from biased or noisy demonstrations without an auxiliary imitation loss.

  • Standard off-policy methods can fail on demonstration-only data because sampled transitions do not reveal why demonstrated actions are preferable to alternatives.Q-learning may assign high values to demonstrated actions without assigning low values to unobserved alternatives.
  • NAC normalizes the Q-function over actions, reducing Q-values for actions not observed in demonstrations and favoring demonstrated behavior when contrary evidence is absent.
  • NAC applies normalized actor-critic updates to both off-policy demonstrations and in-environment transitions under one learning objective.The method derives its updates from a soft policy-gradient objective and parameterizes policy and value functions through Q.
  • The extra −∇θVQ(s) term creates the normalization effect by preventing Q-values from increasing for actions absent from demonstrations.This term distinguishes NAC’s actor update from the soft Q-learning update.
  • Importance sampling formally corrects the demonstration-policy mismatch, but empirical evaluation found that omitting its weights produced better final performance.The final algorithm therefore does not use importance sampling.
  • NAC is less sensitive to noisy demonstrations because negative demonstrated rewards decrease Q(s, a) while increasing VQ(s), producing reverse normalization.

4. Related Work

Prior work includes maximum-entropy and off-policy reinforcement-learning methods, imitation-based learning, and approaches combining demonstration losses with reinforcement-learning objectives. NAC differs by using one objective across demonstrations and environments and by learning from imperfect demonstrations rather than simply copying them.

  • Most maximum-entropy reinforcement-learning methods address environment learning rather than learning from demonstrations.
  • Unlike methods derived from Bellman-error objectives, NAC is derived from policy gradient, which directly optimizes future accumulated reward.
  • NAC uses one objective across demonstrations and environments and outperforms alternatives including PCL and DQfD.
  • Prior demonstration-learning methods commonly assume near-perfect demonstrations or aim to copy demonstrated behavior.
  • NAC uses rewards and actions in demonstrations to distinguish good from poor behavior, whereas DQfD combines imitation and Q-learning losses and is more sensitive to noisy demonstrations.
  • Off-policy learning and learning from demonstration are distinct problems because many off-policy convergence guarantees assume infinitely many visits to every state-action pair.

5. Results

The experiments evaluate NAC against demonstration-learning and reinforcement-learning baselines in Toy Minecraft, Torcs, and GTA V. Results are reported for grid-world and visually complex driving environments using tabular or neural-network Q-functions.

  • The evaluation tests whether NAC benefits from rewards, tolerates ill-behaved demonstrations, and learns meaningful behavior from limited demonstrations.
  • Experiments cover Toy Minecraft, Torcs, and Grand Theft Auto V, spanning discrete grid-world inputs and raw image inputs.
  • Figure 3 plots average total rewards against training iterations over 10 random seeds, comparing demonstration-only learning with subsequent environment interaction.
  • DQfD combines hinge and temporal-difference losses during demonstration learning and environment fine-tuning.
  • The baseline suite includes Q-learning, soft Q-learning, behavior cloning with Q-learning, importance-weighted NAC, and PCL.
  • PCL-R failed when trained from scratch in the visually complex environment, so the experiments stabilized PCL with a target network.

5.3. Experiments on Toy Minecraft

The toy Minecraft experiment uses a tabular Q-function and suboptimal demonstrations to compare NAC with methods incorporating supervised loss. During environmental refinement, NAC finds the optimal path while DQfD remains on the demonstrated suboptimal path.

  • The experiment models state as the agent’s location and uses a tabular Q-function to expose differences from supervised-loss methods.
  • The demonstrations provide a longer suboptimal path, although the shorter path has higher discounted reward.
  • During demonstration-only learning, both DQfD and NAC learn the suboptimal path because neither accesses the environment.
  • During environmental fine-tuning, NAC finds the optimal path while DQfD remains on the suboptimal one.The passage attributes DQfD’s behavior to its imitation loss preventing deviation from the original solution.

5.4. Comparison to Other Methods

Across comparisons using trained-agent demonstrations and realistic driving environments, NAC performs strongly on demonstrations and improves rapidly during interaction. It outperforms DQfD in GTA V and avoids the initial-performance weaknesses of methods trained without demonstrations.

  • The comparison uses 300k transitions collected by executing a trained Q-learning expert policy with 0.01 random-action probability.
  • NAC performs better than other methods on demonstrations and reaches peak performance faster during fine-tuning.DQfD behaves similarly but at lower performance, while behavior cloning drops substantially during environmental interaction.
  • Q-learning and soft Q-learning without demonstrations begin with low performance, while PCL-R without demonstrations fails to learn from scratch.
  • In GTA V, NAC outperforms DQfD both on demonstrations and inside the environment.

5.5. Learning from Human Demonstrations

Human demonstrations introduce diverse, potentially suboptimal driving behavior that challenges methods designed around imitation. NAC outperforms behavior cloning and quickly surpasses DQfD, while remaining robust to corrupted demonstrations.

  • Three non-expert players provided human TORCS demonstrations by playing for 3 hours each.Human actions vary across players and within an individual, such as slowing before a U-turn or choosing different intersection actions.
  • Behavior cloning performs much worse than NAC and DQfD on human demonstrations.
  • DQfD initially outperforms NAC but is quickly surpassed, possibly because supervised hinge loss is harmful with suboptimal demonstrations.
  • With 30% imperfect data, NAC outperforms DQfD and behavior cloning without cloning suboptimal behaviors.The passage states that 50% and 80% imperfect-data results are similar and appear in the appendix.

5.7. Effects of Demonstration Size

NAC remains competitive with supervised methods when demonstrations are scarce and outperforms baselines with sufficient data. It also performs best under the less informative speed2 reward, whereas DQfD degenerates severely.

  • Effects of Demonstration Size: With 10k transitions, or 30 minutes of experience, NAC performs on par with supervised methods.
  • Effects of Demonstration Size: With 300k transitions, NAC outperforms the baselines by a large margin.
  • Effects of Demonstration Size: The demonstration-size experiments compare 10k, 150k, and 300k transitions collected by a trained agent.
  • Effects of Reward Choice: NAC performs best at convergence with the speed2 reward, while DQfD suffers severe performance degeneration.The speed2 reward maximizes squared speed regardless of direction and requires learning the future effects of off-road driving and collisions.

6. Conclusion

NAC provides a unified approach for learning from demonstrations and rewards, remains robust to suboptimal demonstrations, and supports reward-based fine-tuning through continued interaction.

  • NAC unifies learning from reward and demonstrations in one reinforcement learning algorithm.
  • The algorithm is robust to potentially suboptimal demonstration data.
  • Agents can be fine-tuned with rewards after demonstration training by continuing the same algorithm on on-policy data.
  • NAC preserves and improves behaviors learned from demonstrations during reward-driven environmental interaction.

7. Appendix

The appendix specifies NAC’s Q-based actor-critic parameterization, update mechanics, implementation settings, and additional experiments on demonstration quantity and noise.

  • Normalized Actor-Critic with Q Parametrization: NAC computes Q(s, a) with a neural network, then derives πQ(a|s) and VQ(s) through fixed mathematical transforms.This Q-parametrization restricts policy and value outputs to pairs satisfying their specified relationship.
  • Normalized Actor-Critic with Q Parametrization: The actor update maximizes expected future reward using an entropy-regularized policy gradient and a VQ(s) baseline.The critic target can use bootstrapping with R(s, a) + γVQ(s′), with VQ(s′) computed from a target network.
  • Normalized Actor-Critic with Q Parametrization: The critic update uses bootstrapped Q estimates based on immediate reward and discounted next-state value.The appendix states that the target-network value is used in practice.
  • Experiment Details: The implementation uses a 1-million-step replay buffer, 10K-step target updates, γ = 0.99, gradient clipping at 10, and four-frame inputs.The learning rate is annealed from 1e-4 to 5e-5 during the first tenth of training, then held constant.
  • Experiment Details: With 150k and 300k demonstration transitions, NAC performs best with many demonstrations and is comparable to supervised methods with fewer.
  • Experiment Details: With 50% and 80% imperfect actions, NAC is reported to remain highly robust to noisy demonstrations.
Loading 1802.05313v2…