PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Imitation Learning

Junhyuk Oh, Yijie Guo, Satinder Singh, Honglak Lee

arXiv:1806.05635v1cs.LGcs.AIstat.ML

TL;DR

Reinforcement learning must balance exploiting known rewards with exploring new behaviors, yet agents may fail to exploit useful past experiences that could lead to deeper exploration. The paper proposes Self-Imitation Learning, which replays and imitates an agent’s past good decisions, and finds improvements across hard-exploration Atari and MuJoCo tasks. Its results also show that exploitation and exploration can be complementary, while performance may depend on controlling the degree of self-imitation.

  • Problem

    The paper studies whether exploiting an agent’s past experiences, especially good experiences, can improve learning and indirectly lead to deeper exploration.

  • Method

    Self-Imitation Learning is an off-policy actor-critic algorithm that stores past episodes and imitates replayed state-action pairs when their returns exceed current value estimates.

  • Results

    SIL improves A2C on hard-exploration Atari games, is competitive with count-based exploration methods, improves A2C across 49 Atari games, and improves PPO on MuJoCo tasks.

  • Takeaways & Limitations

    Exploiting past good experiences can help drive deep exploration, and SIL can complement exploration methods across discrete and continuous-control tasks.

  • Takeaways & Limitations

    SIL cannot learn a good policy when training never produces a positive reward, and excessive early exploitation can leave it stuck at a sub-optimal policy.

Abstract

from arXiv · show

This paper proposes Self-Imitation Learning (SIL), a simple off-policy actor-critic algorithm that learns to reproduce the agent's past good decisions. This algorithm is designed to verify our hypothesis that exploiting past good experiences can indirectly drive deep exploration. Our empirical results show that SIL significantly improves advantage actor-critic (A2C) on several hard exploration Atari games and is competitive to the state-of-the-art count-based exploration methods. We also show that SIL improves proximal policy optimization (PPO) on MuJoCo tasks.

1. Introduction

The paper examines whether exploiting past good experiences can improve reinforcement learning and indirectly support deeper exploration. It introduces Self-Imitation Learning and reports gains across hard-exploration Atari games and MuJoCo control tasks.

  • Motivation and contributions: Many agents obtain proximal rewards such as Montezuma’s key but fail to exploit those experiences enough to discover distal rewards such as opening the door.SIL instead reproduces key-collecting behavior and explores onward from the resulting state.
  • Motivation and contributions: Self-Imitation Learning (SIL) learns to imitate the agent’s own past good decisions using experiences stored in a replay buffer.It uses past episodes when their returns exceed the agent’s value estimates.
  • Motivation and contributions: SIL combined with A2C is competitive with state-of-the-art count-based exploration methods on several hard-exploration Atari games.The paper also reports improved overall A2C performance across 49 Atari games.
  • Motivation and contributions: SIL also improves PPO on MuJoCo continuous-control tasks, supporting its applicability across actor-critic architectures.The method is presented as generally applicable rather than limited to A2C.

2. Related Work

Prior work improves exploration through curiosity, uncertainty, state visitation, or replay, while this paper emphasizes exploiting an agent’s own past good experiences. SIL differs from related approaches through its actor-critic compatibility and lack of importance sampling.

  • Exploration: Exploration methods commonly use curiosity, uncertainty, or state-visitation signals, whereas SIL focuses on exploiting past good experiences for exploration.The related-work discussion places SIL as a complementary perspective on the exploration–exploitation problem.
  • Episodic control: Episodic-control methods repeat actions associated with the best past outcomes but can be slow at test time and generalize poorly because their policies are non-parametric.MFEC and NEC extend this idea to complex domains.
  • Experience replay: Prioritized experience replay samples experiences according to temporal-difference error, while SIL prioritizes experiences using full episode rewards.These replay advances have mainly focused on value-based methods and are not easily applicable to actor-critic architectures.
  • Experience replay for actor-critic: Importance-sampling approaches may benefit less from old experience when past and current policies differ substantially.The paper contrasts these methods with SIL, which avoids importance sampling and applies to discrete and continuous control.
  • Learning from imperfect demonstrations: SIL treats the agent’s own experiences as imperfect demonstrations without requiring expert demonstrations.This distinguishes it from prior learning-from-demonstrations approaches.

3. Self-Imitation Learning

SIL augments actor-critic learning with replayed state-action-return tuples from past episodes. Its off-policy objective updates the policy and value function using only experiences whose returns exceed current value estimates.

  • Self-Imitation Learning: SIL stores state-action pairs and cumulative returns from past episodes in a replay buffer.The replay data are represented as D = {(s_t, a_t, R_t)}.
  • Self-Imitation Learning: The return R_t is the discounted sum of future rewards, using discount factor γ.This return supplies the off-policy learning target for replayed experiences.
  • Self-Imitation Learning: The policy loss imitates a replayed action only when its return exceeds the current value estimate, R > V_θ(s).Otherwise, the state-action pair contributes no policy update; the value loss moves the estimate toward R.
  • Prioritized Replay: Prioritized replay increases sampling of experiences with positive return–value gaps, making valid SIL samples more frequent.Sampling probability is proportional to (R − V_θ(s))^+.
  • Advantage Actor-Critic with SIL: A2C+SIL performs an on-policy A2C update and M replay-based SIL updates, without importance sampling.The same framework can be combined with any actor-critic method.

4. Theoretical Justification

The paper justifies SIL by relating its objective to lower-bound soft Q-learning in an entropy-regularized RL framework. This connection explains how off-policy updates can move the actor and critic toward better policies and values.

  • Theoretical Justification: SIL is presented as an implementation of lower-bound-soft-Q-learning under the entropy-regularized RL framework.The paper introduces entropy-regularized RL, lower-bound soft Q-learning, and an equivalence between the two objectives.
  • Entropy-Regularized Reinforcement Learning: Entropy-regularized RL optimizes discounted rewards together with a policy-entropy bonus, favoring diverse actions from the same state.The entropy weight is represented by α, which is nonnegative.
  • Lower Bound Soft Q-Learning: Lower-bound soft Q-learning uses behavior-policy state-action-return trajectories to update Qθ toward observed lower bounds of the optimal soft Q-value.The method updates Q-values only when the observed return exceeds the current estimate.
  • Lower Bound Soft Q-Learning: The lower-bound update excludes state-action pairs when R ≤ Qθ(s, a), using the positive-part operator to retain only higher-return experiences.For deterministic environments, the lower-bound argument gives Q∗(s, a) ≥ R > Qθ(s, a) for retained samples.
  • Connection between SIL and Lower Bound Soft Q-Learning: By substituting the actor-critic parameterization into the lower-bound objective, the paper derives an actor-critic gradient estimator connected to SIL.The derivation starts from parameterized forms of the soft value and policy functions.
  • Connection between SIL and Lower Bound Soft Q-Learning: As α →0, SIL becomes a form of lower-bound-soft-Q-learning without explicitly optimizing entropy bonus reward, directly updating policy and value toward optimal counterparts.The paper contrasts SIL’s off-policy direct updates with A2C’s on-policy updates toward the learner policy’s expected return.

5. Experiment

The experiments test whether Self-Imitation Learning supports exploration, complements count-based exploration, improves actor-critic performance across tasks, and identifies its limitations. Results show benefits on sequential-reward domains, hard-exploration Atari games, 49-game Atari performance, and MuJoCo, while excessive or reward-dependent exploitation can fail.

  • Experimental questions: The experiments evaluate SIL for exploration, complementarity with count-based exploration, broad task performance, failure conditions, compatibility with ACER, and PPO-based continuous control.The Atari setup used four SIL updates per on-policy actor-critic update; the gridworld included a count-based exploration bonus and A2C+SIL+EXP.
  • Key-Door-Treasure: A2C+SIL and A2C+SIL+EXP learn most quickly in Key-Door-Treasure, where sequential rewards make finding the treasure difficult.A2C tends to remain at a sub-optimal door-opening policy, while exploration bonuses help A2C collect treasure more often.
  • Hard Exploration Atari Games: SIL exploits occasional positive rewards from random exploration to learn better policies and drive deeper exploration on hard-exploration Atari games.A2C+SIL outperforms A2C on six hard-exploration games; A2C fails to learn better-than-random behavior except on Hero, while SIL reaches human-level performance on Hero and Freeway.
  • Hard Exploration Atari Games: A2C+SIL achieves better results on 6 out of 7 hard-exploration games without an explicit exploration bonus.The comparison includes count-based exploration actor-critic agents such as A3C-CTS, Reactor-PixelCNN, and SimHash.
  • Limitations and stability: SIL can fail without positive experiences, while excessive early exploitation can cause sub-optimal policies; reducing SIL updates or later-stage objective weight resolves the latter issue.On Venture, SIL never receives a positive reward; on games including James Bond and Star Gunner, excessive early exploitation can hurt performance.
  • MuJoCo: SIL improves PPO on Swimmer, Walker2d, and Ant, supporting applicability across actor-critic architectures and task types.MuJoCo learning curves include standard and delayed-reward tasks, with curves averaged over 10 random seeds.

6. Conclusion

The paper concludes that Self-Imitation Learning reproduces past good experiences and helps across hard-exploration and continuous-control tasks. It argues that appropriately balancing exploitation and exploration is an important direction for future work.

  • Self-Imitation Learning reproduces the agent’s past good experiences and is helpful on hard-exploration and continuous-control tasks.
  • Properly exploiting past experiences can drive deep exploration, while self-imitation and exploration methods can complement each other.
  • The relative importance of exploitation and exploration may vary across learning stages.
  • Balancing exploration and exploitation when collecting and learning from experiences is identified as an important future research direction.

A. Hyperparameters

The reported hyperparameters include prioritized-replay settings for Atari and separate SIL configurations for Atari and MuJoCo.

  • Table 3 specifies the hyperparameters used by A2C+SIL on Atari games.
  • The prioritization exponent is 0.6, with bias correction set to 0.1 for hard-exploration experiments and 0.4 for overall evaluation.
  • Table 4 specifies the hyperparameters used by PPO+SIL on MuJoCo.

B. Performance on Atari Games

The Atari performance evaluation reports results with 30 random no-op actions after 50M training steps, while Figure 6 presents learning curves across 49 games.

  • Performance was evaluated on 49 Atari games after 50M training steps, using 30 random no-op actions.
  • Figure 6 presents learning curves on 49 Atari games.
Loading 1806.05635v1…