PaperlayerView exact revision on arXiv

Source-linked AI summary

The Primacy Bias in Deep Reinforcement Learning

Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, Aaron Courville

arXiv:2205.07802v1cs.LGcs.AIstat.ML

TL;DR

Deep RL agents can overfit early interactions and fail to use later evidence, damaging continued learning. The paper analyzes this primacy bias and proposes periodically resetting part of the agent while preserving replay experience. Experiments across diverse environments and algorithms show that resetting consistently improves expected returns and can support higher replay ratios and longer n-step targets.

  • Problem

    Deep RL agents may overfit early interactions and ignore subsequent evidence, while replay-based design choices can amplify this primacy bias.

    Source: §1 · pp. 1–2

  • Method

    The paper periodically re-initializes the last layers of neural networks while maintaining the replay buffer.

    Source: §4 · p. 3, §5.3 · p. 5

  • Results

    Resetting consistently improves performance across discrete-action ALE and continuous-action DeepMind Control Suite benchmarks and enables higher replay ratios and longer n-step targets.

    Source: §1 · p. 2, §5.5 · p. 7

  • Takeaways & Limitations

    Resetting acts as regularization against early-data overfitting and addresses optimization challenges across diverse environments and algorithms.

    Source: §5.5 · p. 7, §8 · p. 8

    Reset periodicity is a hyperparameter practitioners must choose, and the paper focuses on empirical investigation without theoretical guarantees.

    Source: §7 · p. 8

Abstract

from arXiv · show

This work identifies a common flaw of deep reinforcement learning (RL) algorithms: a tendency to rely on early interactions and ignore useful evidence encountered later. Because of training on progressively growing datasets, deep RL agents incur a risk of overfitting to earlier experiences, negatively affecting the rest of the learning process. Inspired by cognitive science, we refer to this effect as the primacy bias. Through a series of experiments, we dissect the algorithmic aspects of deep RL that exacerbate this bias. We then propose a simple yet generally-applicable mechanism that tackles the primacy bias by periodically resetting a part of the agent. We apply this mechanism to algorithms in both discrete (Atari 100k) and continuous action (DeepMind Control Suite) domains, consistently improving their performance.

1. Introduction

Deep RL agents can overfit early interactions, ignore later evidence, and suffer compounding learning damage. The paper investigates causes and proposes periodic partial resets, reporting improvements across strong baselines.

  • Experience replay and high replay ratios can expose agents disproportionately to initial samples and exacerbate early-interaction overfitting.Source: §1 · p. 1
  • Periodic re-initialization of an agent’s last neural-network layers while preserving replay-buffer experience is proposed to alleviate the bias.Source: §1 · pp. 1–2
  • Resetting consistently improves performance on discrete-action ALE and continuous-action DeepMind Control Suite benchmarks without additional computational costs.Source: §1 · p. 2
  • The primacy bias is a tendency to overfit early interactions and ignore subsequent experiences, harming future decision making.Source: §1 · pp. 1–2
  • The paper’s contributions include demonstrating the bias, examining plausible causes, proposing resets, and empirically showing qualitative and quantitative performance improvements.Source: §1 · p. 2

2. Preliminaries

The paper formulates reinforcement learning as interaction within a Markov decision process and describes value learning, replay buffers, replay ratios, and n-step targets. Deep RL uses neural-network function approximators.

  • In the MDP formulation, an agent observes states, selects actions, receives rewards, and transitions according to a state-transition distribution.Source: §2 · p. 2
  • Temporal-difference learning updates value estimates toward bootstrapped targets involving immediate rewards and successor-state values.Source: §2 · p. 2
  • Replay buffers improve sample efficiency by reusing data, while replay ratio controls how often stored experiences are resampled.Source: §2 · p. 2
  • Higher replay ratios can overfit existing data, whereas n-step targets trade statistical bias against variance in value estimates.Source: §2 · p. 2
  • The paper considers deep RL methods that model value functions and, when needed, policies with neural-network function approximators.Source: §2 · p. 2

3. The Primacy Bias

Experiments show that excessive reliance on early data can make deep RL learning unrecoverable, even though later data may remain useful. The primacy bias reflects failure to learn from adequate experience rather than failure to collect it.

  • The primacy bias describes deep RL agents being disproportionately affected by initial training phases through overfitting to early experiences.Source: §3 · p. 2
  • The phenomenon has multiple roots and negative effects, all connected to improper learning from early data.Source: §3 · p. 2

3.1. Heavy Priming Causes Unrecoverable Overfitting

  • High reliance on early data creates a trade-off between exploiting initial experience for sample efficiency and risking compounded learning damage.Source: §3.1 · pp. 2–3
  • Heavy priming on the first 100 transitions leaves SAC unable to recover after nearly one million new transitions.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2. Experiences of Primed Agents are Sufficient

  • With 9 updates per step, a failing SAC agent’s replay buffer enables a freshly initialized agent to improve rapidly toward optimal task performance.Source: §3.2 · p. 3, Figure 2 · p. 3
  • The results indicate that primacy bias is a failure to learn from adequate collected data, rather than a failure to collect proper data.Source: §3.2 · p. 3

4. Have You Tried Resetting It?

The paper introduces resetting as a simple technique for mitigating overfitting to early data. It periodically re-initializes selected final network layers while preserving replay-buffer experience.

  • Resetting periodically re-initializes the parameters of an agent’s last few neural-network layers while preserving its replay buffer.Source: §4 · p. 3
  • The next section evaluates resetting both quantitatively and qualitatively as a method for addressing overfitting to early data.Source: §4 · p. 3

5. Experiments

Experiments test resets across algorithms, domains, and training choices, finding consistent performance gains and fast recovery while preserving replay data. Resets also mitigate TD failures and alter which hyperparameter settings perform best.

5.1. Setup

  • The experiments evaluate resets across discrete and continuous control, using SPR, SAC, and DrQ with varied replay buffers and network architectures.The study covers Atari 100k and DeepMind Control Suite, including raw images, dense features, prioritized replay, random sampling, and different network depths.Source: §5.1 · p. 4, §5.2 · p. 4

5.2. Resets Consistently Improve Performance

  • Resets consistently improve performance across diverse environments and algorithms without additional computational costs.Aggregated results report benefits across action spaces, input types, replay-buffer configurations, and network depths; SPR gains are comparable to prior algorithmic advancements.Source: §5.2 · p. 4, §5.5 · p. 7

5.3. Learning Dynamics of Agents with Resets

  • Preserving the replay buffer lets agents recover quickly after resets, while susceptible tasks can surpass their pre-reset performance.In cheetah-run, where the baseline already succeeds, resets provide little benefit and cause temporary performance spikes; in hopper-hop and humanoid-run, performance improves beyond prior levels.Source: §5.3 · p. 5

5.4. Elements Behind the Success of Resets

  • Resets yield larger gains at higher replay ratios, including over 40% for SPR at four updates per step and over 100% for SAC at 32 updates per step.They also enable reasonable performance at extreme SAC replay ratios of 128 and 256, where learning is barely possible without resets.Source: §5.4 · p. 5
  • Larger n-step targets increase reset benefits, reaching up to 40% for SPR at n = 20 and 50–60% for SAC at increased n values.The results align with the paper’s account that higher target variance makes agents more susceptible to primacy bias.Source: §5.4 · p. 6
  • Resets address TD collapse and divergence by giving agents another chance to find stable solutions, while the best reset scope depends on the algorithm.DrQ with resets learns near-optimal policies in representative failure cases; SAC can reset entirely, whereas SPR performs best when resetting only its final layer.Source: §5.4 · pp. 6–7

6. Related Work

The paper situates primacy bias in deep RL within research on memorization, overfitting, forgetting, optimization, and cognitive science. It adds empirical evidence that artificial agents exhibit a human-associated bias and shows that resetting complements existing mitigation approaches.

  • Primacy bias connects deep RL to memorization, optimization, and cognitive science, while representing a special form of overfitting.Source: §6 · p. 7

Overfitting in RL

  • Existing RL work links value-function overfitting and lost expressivity during bootstrapping to possible amplification of the influence of first data points.Source: §6 · pp. 7–8

Related mechanisms

  • Prior reset, sampling, and forgetting mechanisms address related problems, but SPR still benefits from resets despite already using prioritized experience replay.Source: §6 · pp. 7–8

Cognitive science

  • Human learning studies describe primacy bias as forming generalizations from initial facts and paying less attention to later ones.Source: §6 · p. 8
  • The paper provides evidence that artificial agents exhibit a primacy bias noted in humans, despite different learning conditions.Source: §6 · p. 8

7. Future Work and Limitations

The paper identifies empirical and practical boundaries for resetting: it lacks a theoretical account, requires choosing reset periodicity, and can briefly reduce performance.

  • The paper is limited to empirical investigation and leaves theoretical understanding and guarantees for learning with resets as future work.Source: §7 · p. 8
  • Reset periodicity remains a practitioner-selected hyperparameter, although feedback-based or metalearned strategies could improve performance.Source: §7 · p. 8
  • Brief performance collapses after resetting may be undesirable for regret minimization.Source: §7 · p. 8

8. Conclusion

The paper identifies primacy bias as harmful early-experience overfitting and proposes periodic partial resets as a simple remedy. Evidence across domains and algorithms suggests resets are effective and broadly applicable, while highlighting the need to study deep RL's combined learning dynamics.

  • Primacy bias is a damaging tendency of artificial agents to overfit early experiences.Source: §8 · p. 8
  • Resetting part of the agent is proposed as a simple solution for mitigating this form of overfitting.Source: §8 · p. 8
  • Experimental evidence across domains and algorithms suggests that resetting is effective and generally applicable in RL.Source: §8 · p. 8
  • The findings point to the importance of studying the interaction between reinforcement learning and deep learning rather than treating neural representation as merely technical.Source: §8 · p. 8
  • Resets unlock training regimes unavailable without them and open possibilities for improving understanding and performance of deep RL algorithms.Source: §8 · p. 8

A. Experimental Details

The experiments use open-source JAX implementations of SAC, DrQ, and SPR, with specified evaluation seeds and benchmark configurations. They cover DeepMind Control Suite tasks and 26 Atari 100k tasks, using different observation modalities and replay schemes.

B. Ablations

Controlled ablations show that reset effectiveness depends on what is reset, how deeply, and how often, while retaining replay-buffer experience is essential. Across algorithms, resetting all relevant networks is generally the most robust choice, and resets outperform standard regularizers in the tested settings.

Replay buffer

  • Retaining the replay buffer is essential: resetting it alongside the last layers makes DrQ learn almost from scratch.The results suggest knowledge retention matters more than retaining learned behavior for mitigating primacy bias.Source: B. Ablations · p. 12

Reset depth

  • Reset depth matters: DrQ’s last-layer resets are slightly inferior to resetting the full three-layer head, while SPR’s last-layer resets produce a slightly higher IQM than two-layer-head resets.Resetting the whole SPR network noticeably damages performance.Source: B. Ablations · p. 13, Figure 10 · p. 14, Figure 13 · p. 15

Which networks to reset

  • Resetting all DrQ networks is generally the most robust option, although resetting the critic has the predominant effect in most environments.The three groups are the actor, critic, and target critic.Source: B. Ablations · p. 13, Figure 11 · p. 14

Optimizer state

  • Resetting optimizer statistics has almost no effect when weights are preserved, whereas resetting parameters while keeping the optimizer delivers results almost identical to standard resets.The rapid recovery of Adam’s moment estimates is offered as a possible explanation.Source: B. Ablations · pp. 12–13, Figure 9 · p. 13

Reset structure

  • Random subnetwork resetting is on par or worse than the standard scheme, which re-initializes designated network layers.Each weight in the random subnetwork condition is re-initialized with probability 0.5.Source: B. Ablations · p. 13, Figure 10 · p. 14

Number of resets

  • A single reset contributes most to mitigating primacy bias, but continual resetting is required in some environments to reach standard-reset performance.The best number of resets varies across environments.Source: B. Ablations · p. 13, Figure 12 · p. 14

Other regularizers

  • L2 regularization and dropout are compared with resets, with the experiments testing whether resets provide superior performance under standard training.The comparison uses 10 runs and the evaluation tasks reported in Tables 3 and 4.Source: B. Ablations · pp. 13, 15, Table 5 · p. 15

C. Per-Environment and Additional Results

Additional results examine resets across replay ratios, n-step targets, and individual environments in SAC, SPR, and DrQ. Resets remain effective in regimes associated with overfitting, including extreme replay ratios where no-reset SAC achieves near-zero returns.

Additional training regimes

Replay-ratio robustness

Per-environment results

Related papers

Loading 2205.07802v1…