Source-linked AI summary
The Dormant Neuron Phenomenon in Deep Reinforcement Learning
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, Utku Evci
TL;DR
Deep RL agents can accumulate dormant neurons that reduce network expressivity and impair adaptation during training. The paper analyzes this phenomenon across algorithms and domains, then introduces ReDo, which periodically recycles dormant neurons. ReDo maintains network utilization and experiments suggest improved performance, while dormant neurons are reduced but not eliminated.
Problem
Deep RL networks can lose expressivity and ability to fit new targets over time, raising whether their parameters are used to full potential.
Method
The paper tracks low-activation neurons across RL training and proposes ReDo, which periodically recycles dormant neurons to maintain network capacity.
Results
ReDo consistently maintains network capacity throughout training and experiments suggest that it improves agent performance across evaluated algorithms and domains.
Takeaways & Limitations
Dormant-neuron analysis links reduced network utilization with reduced expressivity and motivates RL-specific optimization techniques.
Takeaways & Limitations
ReDo significantly reduces but does not completely eliminate dormant neurons, and its dormancy threshold is a hyperparameter.
Abstract
from arXiv · showhide
In this work we identify the dormant neuron phenomenon in deep reinforcement learning, where an agent's network suffers from an increasing number of inactive neurons, thereby affecting network expressivity. We demonstrate the presence of this phenomenon across a variety of algorithms and environments, and highlight its effect on learning. To address this issue, we propose a simple and effective method (ReDo) that Recycles Dormant neurons throughout training. Our experiments demonstrate that ReDo maintains the expressive power of networks by reducing the number of dormant neurons and results in improved performance.
1. Introduction
Deep RL networks can lose expressivity as training progresses because dormant neurons accumulate, especially under more frequent updates. The paper identifies this phenomenon across settings and proposes ReDo to recycle dormant neurons and improve network utilization and performance.
- Motivation: Prior layer-reset methods improve performance but can erase previously learned knowledge and require recovery updates.
- The dormant neuron phenomenon: Deep RL agents increasingly develop dormant neurons during training, reducing network expressivity and learning ability.Dormancy is associated with low activations and worsens over time.
- Evidence across RL settings: The dormant neuron phenomenon appears in DQN and DrQ(ϵ) on Atari and SAC on MuJoCo.
- ReDo: Recycling Dormant neurons (ReDo) recycles inactive neurons to an active state without sacrificing previously learned knowledge.The method explicitly limits the spread of dormant neurons during training.
- Results: ReDo maximizes network utilization and improves performance across the evaluated deep RL settings.
2. Background
Deep RL represents value functions with neural networks trained from replayed environment transitions using temporal-difference updates. Replay ratio and non-stationary data are central training choices that affect sample efficiency, stability, and performance.
- Value-function learning: Deep RL represents the Q-function with a neural network whose parameters are updated from replay-buffer batches of environment transitions.
- Value-function learning: The temporal-difference objective compares Qθ(s, a) with a bootstrap target computed using a delayed target network.
- Replay ratio: Replay ratio is the number of gradient updates per environment step and substantially affects agent performance.
- Replay ratio: Higher replay ratios can improve sample efficiency through more parameter updates, but can also cause training instability and decreased performance.
- Non-stationary training: Online RL involves input and target non-stationarity because policies change collected data distributions and learning targets.
3. The Dormant Neuron Phenomenon
Deep RL agents increasingly under-utilize their networks as dormant neurons accumulate during training. The phenomenon is linked to changing targets and replay intensity, and dormant neurons are associated with reduced adaptability, while ReDo is evaluated as a mitigation.
- Definition: Dormant neurons are neurons with persistently low activations, and the phenomenon is defined by their number increasing steadily throughout training.This increasing inactivity indicates that the network is not using its capacity to its full potential, with under-utilization worsening over time.
- Evidence across RL: The percentage of dormant neurons steadily increases during DQN training and appears across algorithms and environments.The reported evaluations include DQN and analyses across additional deep RL settings.
- Potential causes: Changing targets are implicated as a primary source: dormant neurons decrease with fixed targets but increase with non-stationary targets, including sharp rises when labels change.The phenomenon also remains in offline RL with fixed training data, whereas fixed random targets reduce dormant neurons.
- Persistence and impact: Dormant neurons tend to remain dormant, while pruning them during training does not affect agent performance.The overlap of dormant-neuron sets increases throughout training, suggesting limited reactivation.
- Learning effects: Higher replay ratios increase dormancy and correlate with performance drops, while pretrained networks with many dormant neurons have less ability to fit new fixed targets.The replay-ratio relationship is reported as a correlation, and the pretrained-network comparison uses a randomly initialized baseline.
- Mitigation: ReDo recycles dormant neurons to reduce network under-utilization and maintain expressivity without sacrificing previously learned knowledge.The method is evaluated for reducing dormant neurons and improving performance in DQN.
4. Recycling Dormant Neurons (ReDo)
ReDo addresses dormant neurons by periodically recycling low-activation units during training, reducing dormancy while preserving learned behavior. The method uses reinitialization choices designed to maintain network utilization with minimal output disruption.
- Method: ReDo periodically identifies τ-dormant neurons and reinitializes their incoming weights while zeroing their outgoing weights.The incoming weights use the original weight distribution, and small τ values only slightly change network output.
- Effectiveness: ReDo reduces dormant neurons and improves agent performance during training.Figure 9 reports both effects together.
- Design alternatives: Alternative connection-scaling and initialization strategies perform similarly to or worse than using the initial weight distribution.These alternatives include scaling incoming connections and random outgoing-connection initialization.
- Activation functions: Lowering the activation-function dependence does not eliminate the dormant-neuron problem.Experiments with a different activation function produced only a mild decrease in dormant neurons, while the phenomenon remained.
5. Empirical Evaluations
Empirical evaluations across agents, architectures, environments, and replay settings show that ReDo improves performance while reducing dormancy. Its benefits persist under higher replay ratios and exceed those of wider networks, resets, and weight decay in the reported comparisons.
- Replay ratio: ReDo avoids performance collapse as DQN replay ratios increase and benefits from higher replay ratios.DQN is evaluated at replay ratios 0.25, 0.5, 1, and 2 over 10M frames.
- Replay ratio: ReDo mitigates high-replay-ratio performance collapse with n-step returns and improves performance across all tested values.The n-step setting uses n = 3.
- Architectures: ReDo improves performance at higher replay ratios with both the default DQN CNN and the IMPALA ResNet architecture.The same trend is reported when varying architectures.
- Agents: ReDo improves performance for DrQ(ϵ) at higher replay ratios on Atari 100K.The evaluated replay ratios are 1, 2, 4, and 8 over 400K steps.
- Network width: Wider networks have at most a mild performance effect, while ReDo substantially reduces dormancy across widths.Dormant-neuron percentages remain similar across the tested widths.
- Related methods: ReDo outperforms periodic resets and weight decay for DQN, while it is the only compared method without performance degradation for SAC in the reported MuJoCo environments.Weight decay is comparable to periodic resets in the DQN comparison.
6. Related Work
Related work connects this study to research on changing network capacity, generalization, reinitialization, sparse topologies, and network growth in reinforcement learning. These strands motivate examining how training dynamics affect effective capacity.
- Function approximators in RL: Prior RL studies report that network expressivity or capacity changes during training and can affect performance.This work focuses on understanding those changes through dormant neurons.
- Function approximators in RL: Sparse-network studies show that up to 90% of weights can be removed with minimal performance loss.The result suggests RL agents may not fully use their available capacity.
- Generalization in RL: RL generalization research addresses overfitting through regularization, ensembles, data augmentation, and weight reinitialization.These methods target overfitting to environments or early training samples.
- Generalization in RL: Resetting network parameters in RL can improve performance, but recovering performance after each reset requires many gradient updates.Related work also resets smallest-utility features for continual learning.
- Neural network growing: Network-growing methods add, split, or merge neurons using gradient-based or feature-based strategies.These approaches span dense, sparse, and online-learning settings.
7. Discussion and Conclusion
The paper concludes that dormant neurons emerge across RL settings, reduce effective expressivity, and can be addressed by periodic recycling. ReDo improves utilization and performance, but its threshold and recycling design remain open for refinement.
- Discussion and Conclusion: Dormant neurons increase during RL training across algorithms and domains and are associated with reduced expressivity and inability to adapt to new tasks.The conclusion identifies this as the dormant neuron phenomenon.
- Discussion and Conclusion: Periodic recycling maintains network utilization and can improve performance while integrating easily with existing RL algorithms.The paper suggests ReDo may support scaling RL networks in a sample-efficient manner.
- Limitations and future work: ReDo reduces dormant neurons significantly but does not completely eliminate them.The authors identify improved initialization and optimization of recycled capacity as future directions.
- Limitations and future work: The dormancy threshold requires tuning, and an adaptive threshold could improve performance further.The paper also calls for studying task complexity, network capacity, and dormancy together.
- Discussion and Conclusion: The observed capacity-utilization dynamics differ from supervised learning and motivate RL-specific optimization research.The conclusion links this difference to the need for further study of RL architectures and topologies.
- Societal impact: The work is primarily academic but may aid development of more capable autonomous agents, requiring attention to possible negative societal impacts.The authors do not identify direct negative societal impacts from their contributions.
A. Experimental Details
Experiments evaluate DQN, DrQ(ϵ), and SAC across Atari, MuJoCo, and CIFAR-10 settings, using established implementations and aggregated IQM evaluation. Additional analyses vary replay ratio, ReDo hyperparameters, and supervised or regression conditions.
- DQN is evaluated on 17 Arcade Learning Environment games, while SAC is evaluated on four MuJoCo environments.
- Experiments build on Dopamine for DQN and DrQ(ϵ), and TF-Agents and related code for SAC.
- IQM aggregates normalized scores after discarding the bottom and top 25% across runs and games.
- Replay-ratio studies use {0.25, 0.5, 1, 2} for DQN and {1, 2, 4, 8} for DrQ(ϵ).
- The study also examines CIFAR-10 supervised learning, DQN-derived regression inputs, and ReDo recycling-period and dormancy-threshold grids.
B. The Dormant Neuron Phenomenon in Different Domains
The dormant neuron phenomenon appears in SAC actor and critic networks on MuJoCo, as well as during DrQ(ϵ) and DQN training. Pruning dormant neurons does not affect SAC performance, while recycling dormant capacity can help smaller networks.
- In DQN, higher replay ratios consistently increase dormant neurons across games and correlate with performance drops; DrQ(ϵ) shows dormancy as training steps increase.
- Dormant neurons increase over time in SAC actor and critic networks on MuJoCo environments.
- Pruning dormant SAC neurons every 200K steps does not affect performance, indicating little contribution to the learning process.
- On Ant-v2, halving or quartering actor and critic widths produces some gains from recycling dormant capacity.
- The design analysis compares recycling strategies and examines how dormant capacity relates to sample efficiency and network expressivity.
C.1. Effect of Activation Function
The activation-function analysis examines why ReLU may produce dormant neurons and compares ReLU with leaky ReLU. ReDo still addresses performance degradation, while connection-reinitialization choices affect outcomes.
- ReLU has zero gradients in its negative domain, so neurons whose pre-activations remain negative can stay inactive.
- Leaky ReLU slightly decreases dormant neurons but does not mitigate the issue, while ReDo overcomes the performance drop in both activation settings.
- Randomly reinitializing outgoing connections performs worse than ReDo’s zero initialization because random weights change network outputs.
- Scaling incoming weights by the average norm of non-dormant neurons performs similarly to random incoming-weight initialization.
- The identified percentage of dormant neurons is approximately unchanged across batch sizes 32, 64, 256, and 1024.
C.4. Comparison with Continual Backprop
The study compares ReDo’s activation-based recycling metric with Continual Backprop’s utility metric under a fixed recycling schedule. Both achieve similar results, but the latter’s running-average saliency estimates require additional resources and remain insufficiently analyzed.
- ReDo’s activation-based metric and Continual Backprop’s utility metric achieve similar results under a fixed recycling schedule.
- Continual Backprop calculates neuron scores every iteration and uses a running average to estimate neuron saliency.
- The running-average approach requires additional storage and computing, while the effect of more accurate saliency estimates needs further analysis.
C.5. Effect of Recycling the Dormant Capacity
ReDo recycles dormant neurons to preserve learned representations and improve performance under higher replay ratios. Across DQN experiments, it reduces environment interactions while maintaining performance and increases representation expressivity.
- Improving Sample Efficiency: ReDo with RR = 1 performs very close to RR = 0.25 while using four times fewer environment steps; DQN with RR = 1 suffers a performance drop.The comparison uses fixed numbers of gradient steps across DQN agents evaluated on 17 Atari games.
- Improving Networks’ expressivity: Recycling dormant neurons increases the effective rank of learned penultimate-layer representations in DQN trained on DemonAttack.The analysis averages results over 5 seeds after 10M frames and uses effective rank as a measure of expressivity.
- Training Curves: Training curves compare DQN and DrQ(ϵ) across high-replay-ratio and default settings for their respective CNN architectures.DQN uses RR = 1 and RR = 0.25, while DrQ(ϵ) uses RR = 4 and RR = 1.