Source-linked AI summary
Loss of Plasticity in Continual Deep Reinforcement Learning
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, Marlos C. Machado
TL;DR
The paper asks whether canonical value-based deep RL agents retain their ability to learn under continual non-stationarity. It studies agents cycling through Atari games, analyzes weights, gradients, and activations, and evaluates CReLU as a mitigation. The experiments identify activation collapse and show that CReLU maintains or improves performance on successive visits, while catastrophic forgetting remains unresolved.
Problem
Canonical deep RL methods are often developed for mostly stationary settings, leaving their continual-learning behavior under changing environments insufficiently characterized.
Method
The paper cycles a single Rainbow agent through Atari games without resets, analyzes weights, gradients, and activations, and evaluates CReLU activations as a mitigation.
Results
The experiments show loss of plasticity with activation collapse, while replacing ReLUs with CReLUs mitigates this loss and maintains or improves performance on repeated game visits.
Takeaways & Limitations
CReLU facilitates continual learning in the changing Atari setting, but it does not resolve catastrophic forgetting or restore uninterrupted-learning performance.
Takeaways & Limitations
CReLU does not address forgetting, and its improvements remain modest relative to an agent learning uninterrupted on one game.
Abstract
from arXiv · showhide
The ability to learn continually is essential in a complex and changing world. In this paper, we characterize the behavior of canonical value-based deep reinforcement learning (RL) approaches under varying degrees of non-stationarity. In particular, we demonstrate that deep RL agents lose their ability to learn good policies when they cycle through a sequence of Atari 2600 games. This phenomenon is alluded to in prior work under various guises -- e.g., loss of plasticity, implicit under-parameterization, primacy bias, and capacity loss. We investigate this phenomenon closely at scale and analyze how the weights, gradients, and activations change over time in several experiments with varying dimensions (e.g., similarity between games, number of games, number of frames per game), with some experiments spanning 50 days and 2 billion environment interactions. Our analysis shows that the activation footprint of the network becomes sparser, contributing to the diminishing gradients. We investigate a remarkably simple mitigation strategy -- Concatenated ReLUs (CReLUs) activation function -- and demonstrate its effectiveness in facilitating continual learning in a changing environment.
1 INTRODUCTION
Deep RL is often developed for stationary settings, but changing environments require agents to keep adapting. The paper frames loss of plasticity, capacity loss, and primacy bias as related manifestations of continual learners exhausting their ability to learn.
- Stationary policies can become ineffective when deployed systems encounter changing conditions and additional data.
- Neural networks trained on incrementally arriving or distribution-shifted data can lose their ability to learn altogether.
- Deep RL agents can lose capacity while estimating changing value functions and overfit early experiences, producing related continual-learning failures.
- The paper studies catastrophic loss of plasticity in Rainbow under changing Atari games and analyzes weights, gradients, and activations at scales reaching 2 billion interactions over 50 days.
2 PRELIMINARIES
The paper studies value-based deep RL agents that continually interact with environments and estimates policies from learned value functions. Its Atari setup cycles one agent through games without resetting, revealing deteriorating performance on repeated visits.
- DQN continually updates a neural-network estimate of future reward from replayed experience and gradient-based weight updates.
- Rainbow extends DQN with components including distributional RL, noisy exploration networks, prioritized replay, and dueling networks.
- The empirical study uses the Arcade Learning Environment but replaces separate game training with one network playing a sequence of games without resets.
3 DEMONSTRATING LOSS OF PLASTICITY
The S-ALE benchmark evaluates Rainbow while it repeatedly switches among Atari games or game modes without resetting learned state. Rainbow progressively loses learning ability and performance across visits, with degradation also appearing in DQN and in some same-game mode sequences.
- 3.1 ADAPTING THE ALE FOR CONTINUAL LEARNING: S-ALE cycles through fixed game sequences while preserving network weights and replay contents between switches, enabling repeated visits under continual learning.Each game receives a fixed interaction budget, and experiments can exceed replay-buffer capacity by a large margin.
- 3.2 LEARNING PERFORMANCE IN S-ALE: Rainbow’s learning slows on repeated visits and its end-of-visit policy typically worsens, eventually showing no improvement.This pattern is observed when cycling through five Atari games with 20M frames per visit.
- 3.2 LEARNING PERFORMANCE IN S-ALE: The reported failures are demonstrative rather than universal because the experiments used limited runs and cannot establish that such failures always occur.Some experiments nevertheless spanned substantial durations and interaction budgets.
- 3.2 LEARNING PERFORMANCE IN S-ALE: Rainbow performs worse over time than an idealized reset agent that relearns each game from a fresh state.The reset comparison uses first-visit performance as a proxy rather than an actually executed reset agent.
- 3.2 LEARNING PERFORMANCE IN S-ALE: The degradation persists across different numbers of games and visit durations, and DQN shows the same phenomenon when cycling through ten games.The authors report similar patterns with 10M and 50M frames per visit and with DQN.
- 3.3 VARYING THE DEGREE OF NON-STATIONARITY: LOOPING THROUGH GAME MODES: Changing modes within one game produces heterogeneous outcomes: Breakout degrades across visits, whereas Freeway recovers faster and Space Invaders generally remains strong.In Breakout, the largest losses occur in modes with bigger changes in dynamics, suggesting task similarity is related to plasticity loss but is difficult to characterize beforehand.
4 CHARACTERIZING LOSS OF PLASTICITY
Rainbow progressively loses plasticity during continual training: repeated visits produce poorer performance, smaller weight updates, and diminishing gradients as network activations collapse.
- 4 CHARACTERIZING LOSS OF PLASTICITY: Successive visits reduce Rainbow’s performance, weight change, gradient magnitude, and number of active units.The analysis links fewer active ReLU units to reduced incoming-weight updates through the chain rule.
- 4.1 WEIGHT CHANGE DIMINISHES OVER TIME: The analysis compares normalized weight change, halfway-visit loss, aggregated gradient norms, and activation ℓ0 norms across continual and scratch training.Weight changes are measured through the first half of each visit, normalized to the first visit, and aggregated across layers.
- 4.1 WEIGHT CHANGE DIMINISHES OVER TIME: 20% of first-visit weight change remains by Alien’s tenth continual visit, versus 75% for the scratch agent.The continual agent’s loss grows large while its weight change sharply diminishes; the scratch agent’s weight change remains substantially larger.
- 4.3 ACTIVATION COLLAPSE: Activation collapse in Rainbow’s value and advantage networks leaves less than 1% of units producing non-zero values, hindering adaptation.The convolutional stack does not collapse in the same way, while the value and advantage streams become highly sparse.
5 MITIGATING LOSS OF PLASTICITY WITH CONCATENATED RELUS
The paper uses CReLU activations to prevent the activation collapse underlying Rainbow’s loss of plasticity. CReLU maintains continual performance across repeated Atari games and Breakout modes without improving conventional single-game learning.
- 5 MITIGATING LOSS OF PLASTICITY WITH CONCATENATED RELUS: CReLU concatenates ReLU(x) and ReLU(−x), ensuring one output is typically nonzero for each input and preventing activation collapse.Replacing ReLU doubles activations while keeping the number of parameters in the current layer unchanged, so effective capacity must be controlled.
- 5 MITIGATING LOSS OF PLASTICITY WITH CONCATENATED RELUS: Rainbow-CReLU maintains or improves performance on successive visits across all five S-ALE games, matching or exceeding reset-agent performance.The result holds with invariant input dimension; the invariant-output-dimension variant also maintains plasticity with fewer parameters.
- 5 MITIGATING LOSS OF PLASTICITY WITH CONCATENATED RELUS: CReLU maintains or improves performance across all 10 sequential Breakout game modes, while showing no significant improvement for Space Invaders where Rainbow did not substantially lose plasticity.The result supports a benefit in settings where loss of plasticity is present rather than a universal performance increase.
- 5 MITIGATING LOSS OF PLASTICITY WITH CONCATENATED RELUS: CReLU and ReLU produce comparable performance when Rainbow learns from scratch on one game for 100M frames.This comparison indicates the continual-learning benefit is not explained by general superiority in conventional single-game training.
- 5 MITIGATING LOSS OF PLASTICITY WITH CONCATENATED RELUS: In Alien, about half of CReLU activations remain nonzero, gradients stay relatively large, and weights continue changing through ten visits.The relative ℓ1 gradient norm declines slowly, while the ℓ0 norm does not diminish relative to the first visit.
6 CATASTROPHIC FORGETTING: AN UNRESOLVED CHALLENGE IN CONTINUAL LEARNING
The continual Rainbow agent forgets what it learned whenever it revisits a game, while CReLUs do not resolve this retention problem.
- Rainbow shows no retention when revisiting games or modes, so successive visits do not preserve previously learned performance.
- CReLUs can produce modest improvement across visits, but remain below uninterrupted single-game learning with equivalent experience.
- The gap between continual Rainbow and an idealized uninterrupted learner motivates methods addressing both catastrophic forgetting and loss of plasticity.
7 CONCLUSIONS AND FUTURE WORK
The paper finds that canonical value-based deep RL methods perform poorly in continual learning, identifies activation collapse as a mechanism, and shows that CReLUs mitigate loss of plasticity.
- Canonical value-based deep RL methods in DQN and Rainbow are not able to perform well in continual learning problems.
- In non-stationary Atari sequences, deep RL algorithms can lose their ability to update neural-network weights, a phenomenon called loss of plasticity.
- Activation collapse leaves only a tiny fraction of network units producing non-zero values, inhibiting adaptation.
- Replacing ReLUs with CReLUs mitigates loss of plasticity, but does not resolve catastrophic interference or enable effective reuse of past experience.
A APPENDIX
The appendix reports continual-learning comparisons across game-switch intervals, algorithms, game modes, network sizes, and activation functions.
- Rainbow is evaluated on repeating ten-game sequences with 10M, 20M, and 50M frames between game switches.The corresponding cycles restart every 100M, 200M, and 500M frames.
- DQN is evaluated on repeating ten-game sequences with 20M and 50M frames per game.The plotted cycles restart every 200M and 500M frames, respectively.
- Loss of plasticity is reported as absent in Freeway and Space Invaders sequences of game modes.The cited experiments use 20M frames per mode and repeat the mode sequences over 160M or 200M frames.
- CReLUs are compared across networks of different sizes, separating activation-function effects from parameter-count differences.
- The CReLU Rainbow agent consistently returns to the final performance reached during the first iteration when cycling across multiple games.
- A combined comparison reports catastrophic forgetting, loss of plasticity, and CReLU performance using the data from the ten-game experiment.