PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Stabilizing Deep Q-Learning via Controlled Bootstrapping and Regulated Value Dynamics

Bozhou Chen, Yongyi Wang, Hanyu Liu, Xionghui Yang, Wenxin Li

arXiv:2608.16182v1cs.LGcs.AI

TL;DR

Deep Q-learning remains unstable because interacting effects in recursive value learning are not fully captured by maximization-bias explanations. This paper analyzes these mechanisms and derives stabilization principles, producing competitive Atari-100K and Procgen performance with improved training stability.

  • Problem

    Deep Q-learning instability is not fully explained by maximization bias, while interacting effects of bootstrapping, regression noise, and representations remain typically studied in isolation.

  • Method

    The paper analyzes operator bias, estimator sensitivity, and parameter imbalance, then integrates controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation.

  • Results

    On Atari-100K and Procgen, the resulting method achieves competitive performance while improving training stability.

  • Takeaways & Limitations

    The findings frame deep Q-learning instability as an interacting multistage phenomenon rather than an isolated maximization-bias problem.

  • Takeaways & Limitations

    Advantages are less pronounced on sparse-reward, long-horizon Procgen tasks where exploration efficiency and global planning matter, because the method primarily targets value instability.

Abstract

from arXiv · show

Deep Q-learning (DQL) has achieved remarkable empirical success in reinforcement learning, yet its training process remains notoriously unstable. Existing studies often attribute instability to isolated factors such as overestimation bias or representation learning issues, lacking a unified understanding of how different sources of instability interact during recursive value estimation. In this work, we provide a systematic analysis of instability in deep Q-learning from three complementary perspectives: operator-level bias in Bellman bootstrapping, estimator-level sensitivity of greedy action selection to regression noise, and parameter-dynamics imbalance under aggressive data reuse. We identify a reward-triggered self-reinforcing trap and characteristic parameter spike dynamics, then derive stabilization principles for controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation. Experiments on Atari-100K and Procgen demonstrate competitive performance and improved training stability.

1 INTRODUCTION

The paper presents a unified account of deep Q-learning instability as interacting feedback mechanisms in bootstrapping, value estimation, and parameter evolution. It derives coordinated stabilization principles and implements them through controlled bootstrapping, ensemble quantile regression, and parameter regulation, with experiments on Atari-100K and Procgen showing strong performance and improved robustness.

  • Operator-level instability: The self-reinforcing trap couples reward-driven value amplification with action-conditioned representation generalization, repeatedly selecting the same action in bootstrap targets.This feedback loop continuously amplifies that action’s value and produces biased bootstrap dynamics.
  • Estimator-level instability: When action gaps are small, unbiased regression noise can change greedy decisions and alter collected experience, propagating instability through the control loop.The resulting estimator-level instability arises because greedy selection depends on relative value differences.
  • Parameter-dynamics instability: Under high replay ratios, a small subset of parameters may grow disproportionately large, contributing to lost network plasticity; the spike ratio monitors this instability.The paper identifies parameter spike dynamics as a practical diagnostic of imbalanced parameter evolution.
  • Stabilization framework: The proposed framework coordinates controlled bootstrapping, ensemble quantile regression, and parameter regulation, achieving competitive performance and improved training robustness on Atari-100K and Procgen.These components instantiate stabilization principles for recursive amplification, decision variance, and parameter plasticity.
  • Unified instability analysis: Instability arises from three interacting mechanisms: operator-level bootstrapping bias, estimator-level sensitivity to regression noise, and parameter-dynamics imbalance.These mechanisms jointly shape learning dynamics through recursive value updates rather than operating in isolation.

2 RELATED WORK

Prior work stabilizes deep Q-learning through bias-controlled bootstrap targets, distributional and ensemble estimation, data-efficient representation learning, and optimization or plasticity interventions. These approaches improve robustness, but recursive bootstrapping, regression-noise effects on decisions, and training-dynamics instability remain incompletely characterized.

  • Bootstrap bias control: Double Q-learning and Double DQN reduce maximization overestimation by decoupling action selection from value evaluation with separate estimators or networks.Under function approximation, maximization bias can accumulate through recursive bootstrapping.
  • Bootstrap bias control: REM and REDQ control bootstrap bias or variance through ensemble aggregation, convex combinations of Q-heads, and randomized subsampling.These methods target value-learning stability under high update-to-data ratios.
  • Distributional and ensemble estimation: C51, QR-DQN, IQN, and related distributional methods model return distributions or quantiles to capture richer uncertainty than expected values alone.C51 uses categorical fixed-support representations, whereas QR-DQN and IQN use quantile-based parameterizations.
  • Distributional and ensemble estimation: Bootstrapped DQN, REM, SUNRISE, and SPQR improve estimation or exploration through multiple predictors, uncertainty-weighted backups, and mechanisms addressing ensemble collapse.Their primary emphasis is prediction accuracy or uncertainty estimation rather than regression noise affecting decision and interaction dynamics.
  • Data-efficient reinforcement learning: CURL, SPR, SR-SPR, DrQ, BBF, SGF, Drama, IRIS, and STORM improve data efficiency through representation learning, regularization, auxiliary objectives, or augmentation.High replay ratios and aggressive update-to-data strategies also improve sample efficiency but can amplify bootstrap errors and estimation-noise sensitivity.
  • Optimization dynamics and plasticity: Target networks, Huber loss, spectral normalization, architectural modifications, parameter resets, Shrink-and-Perturb, and ReDo address oscillations, unstable optimization, or loss of network plasticity.Reported phenomena include primacy bias, dormant neurons, and rank collapse during prolonged bootstrapped training.

3 ANALYSIS OF INSTABILITY MECHANISMS

Deep Q-learning instability arises from coupled mechanisms across Bellman bootstrapping, noisy greedy action selection, and parameter dynamics rather than from a single isolated source. The analysis identifies reward-triggered feedback amplification, action-selection errors under small gaps, and replay-induced parameter skew as interacting pathways.

  • Overview: Instability emerges from recursive coupling among biased bootstrap targets, noisy value estimates, control decisions, experience collection, and evolving parameters.The analysis treats these mechanisms across operator-, estimator-, and parameter-dynamics levels.
  • Operator-Level Instability: Positive rewards can amplify Q(s, a), propagate across nearby states through action-conditioned generalization, and bias the bootstrap maximization toward reselecting a.This produces a self-reinforcing loop specific to transitions with r > 0; zero-reward transitions do not create the same systematic preference.
  • Operator-Level Instability: The reward-triggered amplification is bounded under repeated replay of one transition and converges to a finite limit determined by reward scale and discount factor.This bound applies in the idealized case where the same reward-bearing transition is repeatedly replayed and the maximizer persistently selects the original action.
  • Estimator-Level Instability: Unbiased regression noise can still destabilize greedy action selection when the action gap is small, causing lower-return trajectories and slower learning.The issue is decision instability rather than systematic value bias, because the behavior policy determines the distribution of newly collected data.
  • Parameter-Dynamics Instability: As replay ratio grows, repeated reuse of past experiences amplifies the current data distribution and drives certain weights to dominate while others receive weaker updates.The resulting parameter distribution becomes increasingly skewed.
  • Parameter-Dynamics Instability: Parameter-dynamics instability can reduce representational plasticity without numerical value divergence, making adaptation slower as the data distribution shifts with policy improvement.A skewed parameter configuration may constrain adaptation to changing experience distributions.

4 STABILIZATION PRINCIPLES FOR VALUE LEARN-

This section derives three stabilization principles for deep Q-learning from interacting instability mechanisms: controlled bootstrapping, variance-aware value estimation, and parameter-dynamics regulation. Together, these principles form a controlled optimization procedure for stabilizing value learning.

  • Stabilization principles: Instability reflects interactions among recursive bootstrapping bias, greedy action-selection sensitivity to regression noise, and parameter-dynamics imbalance under non-stationary sampling.These mechanisms can amplify value estimates, alter control decisions, or distort parameter evolution through recursive value learning.
  • Controlled bootstrapping: Controlled bootstrapping regulates recursive Bellman updates and targets reward-triggered self-reinforcing traps that repeatedly select reward-producing actions.A transition replayed repeatedly can raise both Q(s, a) and Q(s′, a), while the imposed constraint disrupts this amplification pathway.
  • Variance-aware value estimation: Variance-aware estimation combines quantile regression with ensemble aggregation to reduce intra-model and intermodel variance and improve greedy action-selection reliability.This addresses instability caused when even unbiased regression noise changes greedy actions under small action gaps.
  • Parameter-dynamics regulation: Parameter-dynamics regulation monitors spike ratios and resets affected parameters when layer-wise extreme-value dominance exceeds a predefined threshold, restoring distributional balance and network plasticity.The mechanism addresses parameter-distribution distortion caused by high replay ratios.
  • Controlled bootstrapping: Cross-model action selection and value evaluation decouple estimation noise, substantially reducing upward bias induced by the max operator when noises are weakly correlated.Independent initialization and separate bootstrap targets encourage weak correlation between ensemble members.

5 ALGORITHM INSTANTIATION

Algorithm 1 instantiates the stabilization principles as a controlled value-learning procedure combining ensemble-averaged ϵ-greedy interaction, prioritized replay, reward-triggered bootstrap-action decoupling, cross-model bootstrapping, and quantile-regression updates.

  • Interaction and replay: The agent uses ϵ-greedy exploration, selecting random actions with probability ϵ and otherwise acting greedily according to the ensemble-averaged value ¯Q(s, a).Transitions are executed, observed, and stored in replay before learning begins.
  • Controlled bootstrapping: Learning samples prioritized minibatches from replay and determines bootstrap actions separately for each ensemble member using its own target network.Each member forms scalar action values from its quantile outputs before selecting bootstrap actions.
  • Controlled bootstrapping: For transitions with r(b) > 0, the reward-bearing decoupling rule masks the self-reinforcing candidate during bootstrap-action selection.This implements the reward-triggered decoupling mechanism within the per-member target-action procedure.
  • Ensemble quantile regression: The procedure samples a random permutation of ensemble members for cross-model bootstrapping, computes target quantiles, and updates parameters using quantile regression loss.The quantile regression update follows Eq. (19).
  • Parameter regulation: Layer-wise spike ratios are periodically monitored as part of the training process.This operationalizes spike-based parameter regulation during optimization.

30 end

The method decouples greedy action selection from value evaluation across ensemble members using a random permutation, then stabilizes updates with quantile regression, synchronized target networks, and spike-triggered parameter resets.

  • Controlled bootstrapping: A random permutation π over ensemble indices decouples bootstrap action selection from value evaluation across models.Member i evaluates the action selected by another member π(i).
  • Quantile estimation: For each quantile index j′, member i constructs target quantiles using the permuted selector’s action and its own target network, then minimizes Eq. (19).The update uses the quantile regression objective described for Lines 19–24.
  • Parameter regulation: Target networks are periodically synchronized, while layer-wise spike ratios detect over-adaptation and trigger parameter resets when exceeding a predefined threshold.The spike ratio is monitored using Line 26 and Eq. (11).

6 EXPERIMENTS

Experiments evaluate the proposed stabilization principles on Atari-100K and Procgen, combining benchmark results with controlled analyses of ensemble size, action-decoupling constraints, replay ratios, and parameter regulation. The method achieves strong aggregate performance, while results show that stabilization benefits and data-reuse settings depend on environment characteristics.

  • Atari-100K: On Atari-100K, our method achieves the highest IQM and Median HNS, plus the most human-level and per-game best results under 100k interaction steps.The benchmark contains 26 Atari 2600 games and corresponds to 400k frames with frame skip 4.
  • Procgen: On Procgen, our method outperforms the baseline on most environments, with especially pronounced gains under high visual and structural variability.The results suggest that stabilizing recursive value learning benefits representation robustness under procedural diversity.
  • Controlled ablations: Larger ensembles generally improve performance and training stability, with normalized final performance increasing as N_e rises from 2 to 16 in most cases.Alien, Amidar, and BankHeist show faster growth and higher final returns, whereas Breakout is less monotonic.
  • Controlled ablations: The Action-Decoupling Constraint improves performance clearly on dense-reward Alien, while differences are smaller or reversed on sparser-reward environments.The constraint mitigates self-reinforcing amplification of overestimated values, whereas moderate amplification can strengthen weak learning signals.
  • Controlled ablations: Spike ratios are highest in post-encoder MLP layers and increase with replay ratio, while parameter resets occur more often early and during volatile value growth.Convolutional encoder layers remain comparatively stable, and reset benefits vary across environments.
  • Controlled ablations: Replay-ratio effects are non-monotonic: moderate reuse can improve performance, but excessively large ratios degrade learning beyond a task-dependent threshold.BankHeist benefits consistently from higher replay ratios, while excessive reuse can rapidly amplify post-encoder spikes between monitoring and reset events.

7 CONCLUSION AND FUTURE WORK … C.3 Network Architecture

The paper frames deep Q-learning instability as an interacting systems-level phenomenon and derives stabilization principles for bootstrapping, estimation, and parameter evolution. Appendices detail spike-ratio analysis, bounded Bellman updates, Atari settings, replay sampling, and network architecture.

  • 7 CONCLUSION AND FUTURE WORK: Instability arises from interacting operator-level bootstrapping bias, estimator-level greedy-selection sensitivity, and parameter-dynamics imbalance under aggressive data reuse.The conclusion presents these mechanisms as a unified perspective on recursive value learning.
  • 7 CONCLUSION AND FUTURE WORK: The analysis identifies a reward-triggered self-reinforcing bootstrap trap and parameter spike dynamics linked to gradual loss of network plasticity.The spike ratio is introduced as a practical diagnostic for parameter dynamics.
  • 7 CONCLUSION AND FUTURE WORK: The resulting algorithm combines controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation, achieving competitive performance on Atari-100K and Procgen.These components instantiate stabilization principles for target construction, value-estimation variance, and balanced parameter evolution.
  • APPENDIX A SPIKE RATIO DYNAMICS: The appendix examines layer-wise spike-ratio evolution under replay ratios rr ∈{1, 2, 4} using the Atari-100K protocol.Spike ratios are computed periodically to analyze how data reuse affects parameter dynamics during learning.
  • APPENDIX B BOUNDED BELLMAN UPDATE: The bounded Bellman-update appendix derives a greedy-rollout identity that uses G_t as a trajectory-wise bound for Q(s_t, a_t) under the current greedy policy.The recursion follows greedy actions and closes when the episode terminates without future continuation.
  • APPENDIX C ATARI EXPERIMENT DETAILS: Atari experiments use a total interaction budget of 100k environment steps and follow the Atari-100K evaluation protocol.Implementation details are provided for the main Atari experiments.
  • C.1 Environment Setup: Observations are grayscale, resized to 84 × 84, stacked over four frames, and trained with frame skipping set to 4.The environment wrapper also applies up to 30 initial no-op actions and treats life loss as terminal during training.

C.4 Training Procedure … D.3 Network Architecture

The appendix specifies the training, distributional ensemble, parameter-reset, implementation, and evaluation procedures for Atari-100K and Procgen. Procgen uses standardized generalization environments and a residual encoder adapted to 64 × 64 RGB observations.

  • C.4 Training Procedure: The agent trains for 100, 000 environment steps with ϵ-greedy exploration, annealed from 1.0 to 0.01 over the first 2000 steps; learning then performs rr updates per environment step.Optimization uses Adam with learning rate 1 × 10−4, Polyak coefficient τ = 0.005, and gradient clipping at norm 10.
  • C.5 Distributional Value Learning: The value function uses quantile regression with the quantile Huber loss and K = 51 quantile atoms, constructing targets from the target network and mean-quantile greedy actions.Quantile fractions follow the standard QR-DQN formulation.
  • C.6 Ensemble Learning: The method employs Ne = 16 independently estimating value networks, averaging quantile values across both quantile and ensemble dimensions for action selection.Each ensemble member estimates quantile value distributions for all actions.
  • C.7 Parameter Reset Mechanism: Training monitors each layer’s spike ratio, reinitializing layers that exceed the threshold while clearing their optimizer states and synchronizing the target network.The spike ratio is defined as SpikeRatio = max |θ| / Quantile0.99(|θ|).
  • C.8 Hardware and Implementation: All experiments use a single NVIDIA RTX 5090 GPU with a PyTorch implementation running on Linux with CUDA acceleration.Reported results use the same hardware configuration.
  • C.9 Hyperparameters; C.10 Experimental Results: Atari experiments use the hyperparameters summarized in Table 4, while detailed Atari benchmark comparisons are provided in Table 5.The supplied passages identify these tables but do not provide their numerical contents.
  • APPENDIX D PROCGEN EXPERIMENT DETAILS; D.1 Environment Setup: Procgen evaluation follows the generalization protocol, training on fixed levels and evaluating on unseen levels; training uses the first 200 levels and evaluation begins at level 200.All environments use easy difficulty, 64 × 64 RGB observations, and 128 parallel training instances.

D.4 Training Procedure · D.5 Replay · D.6 Ensemble Distributional Value Learning

Training runs for 25 million environment steps with replay-buffer mini-batch updates, quantile-regression optimization, and an ensemble of 16 distributional value networks. Exploration, replay sampling, and action selection are specified through linear ε annealing, uniform batches, and averaged ensemble quantiles.

  • D.4 Training Procedure: 25 million environment steps define the total training horizon.Training proceeds for a fixed budget of 25 million environment steps.
  • D.4 Training Procedure: Exploration uses ε-greedy action selection, with the rate annealed linearly from 1.0 to 0.05 over the first 5 million environment steps.The annealing schedule covers the initial 5 million environment steps.
  • D.4 Training Procedure: Training starts after replay-buffer warm-up and uses mini-batches sampled from the replay buffer for gradient updates.The passage specifies both an initial warm-up period and replay-based mini-batch updates.
  • D.4 Training Procedure: Quantile regression loss is optimized with Adam for parameter updates.The optimization objective and optimizer are specified directly.
  • D.5 Replay: The replay buffer stores up to 100, 000 transitions, while uniformly sampled training mini-batches contain 256 transitions.Both the buffer capacity and batch size are fixed implementation choices.
  • D.6 Ensemble Distributional Value Learning: The method uses an ensemble of Ne = 16 value networks, each independently estimating the quantile value distribution for every action.Each ensemble member produces its own action-conditional quantile distribution.
  • D.6 Ensemble Distributional Value Learning: Final action-value estimates average predicted quantiles across both ensemble and quantile dimensions during action selection.The averaging spans the ensemble dimension and the quantile dimension.

D.7 Hardware and Implementation

Procgen experiments use PyTorch on a single NVIDIA RTX 5090 GPU, with vectorized CUDA-accelerated environments to improve training throughput.

  • Hardware and Implementation: Procgen experiments run in PyTorch on a single NVIDIA RTX 5090 GPU, using vectorized environments with CUDA acceleration to improve training throughput.The implementation combines PyTorch, single-GPU execution, vectorized environments, and CUDA acceleration.

D.8 Detailed Per-Environment Results

This section reports raw test scores for each Procgen environment under the generalization protocol and visualizes per-environment training performance throughout learning.

  • D.8 Detailed Per-Environment Results: Table 7 reports raw test scores for each Procgen environment under the generalization protocol.These scores provide a comprehensive view of performance across different tasks.
  • D.8 Detailed Per-Environment Results: Figure 10 shows per-environment training performance curves across the entire learning process, plotted against environment interaction steps.Each subplot corresponds to a different Procgen environment.
Loading 2608.16182v1…