PaperlayerView exact revision on arXiv

Source-linked AI summary

SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning

Kimin Lee, Michael Laskin, Aravind Srinivas, Pieter Abbeel

arXiv:2007.04938v4cs.LGcs.AIstat.ML

TL;DR

Off-policy RL can suffer from unstable Q-learning and difficult exploration–exploitation trade-offs. SUNRISE combines uncertainty-weighted Bellman backups, UCB action selection, and bootstrap-based ensemble diversity. Across continuous and discrete control tasks, it consistently improves existing off-policy methods and outperforms state-of-the-art RL algorithms.

  • Problem

    Off-policy RL faces Q-learning error propagation and the challenge of balancing exploration with exploitation.

  • Method

    SUNRISE combines ensemble-based weighted Bellman backups, UCB-based action selection, and bootstrap with random initialization across independent agents.

  • Results

    SUNRISE consistently improves SAC and Rainbow DQN and outperforms state-of-the-art RL algorithms across continuous and discrete control tasks in low- and high-dimensional environments.

  • Takeaways & Limitations

    Ensemble uncertainty weighting and UCB exploration can be integrated as a unified approach for improving existing off-policy RL algorithms.

  • Takeaways & Limitations

    SUNRISE adds computation overhead, requiring N inferences for weighted backups and 2N inferences for actors and critics, although it is parallelizable.

Abstract

from arXiv · show

Off-policy deep reinforcement learning (RL) has been successful in a range of challenging domains. However, standard off-policy RL algorithms can suffer from several issues, such as instability in Q-learning and balancing exploration and exploitation. To mitigate these issues, we present SUNRISE, a simple unified ensemble method, which is compatible with various off-policy RL algorithms. SUNRISE integrates two key ingredients: (a) ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble, and (b) an inference method that selects actions using the highest upper-confidence bounds for efficient exploration. By enforcing the diversity between agents using Bootstrap with random initialization, we show that these different ideas are largely orthogonal and can be fruitfully integrated, together further improving the performance of existing off-policy RL algorithms, such as Soft Actor-Critic and Rainbow DQN, for both continuous and discrete control tasks on both low-dimensional and high-dimensional environments. Our training code is available at https://github.com/pokaxpoka/sunrise.

1. Introduction

SUNRISE addresses error propagation in Q-learning and the exploration–exploitation trade-off with a unified ensemble method for off-policy RL. It combines uncertainty-weighted Bellman backups, UCB-based action selection, and bootstrap-induced agent diversity, improving SAC and Rainbow DQN across control settings.

  • Off-policy RL remains challenged by Bellman-backup error propagation in Q-learning and the difficulty of balancing exploration with exploitation.
  • SUNRISE reweights sample transitions using uncertainty estimates from a Q-ensemble to improve Q-update signal-to-noise and stabilize learning.
  • The method selects actions with the highest upper-confidence bounds, using ensemble variance to encourage visits to unseen state-action pairs.
  • Bootstrap with random initialization enforces diversity between agents, while the ensemble components remain largely complementary.
  • SUNRISE consistently improves SAC and Rainbow DQN on continuous and discrete control benchmarks, including noisy-reward environments.

2. Related work

Related work uses ensembles to stabilize value learning, improve exploration, and reduce model errors. SUNRISE differs by combining ensemble uncertainty with weighted Bellman backups and UCB exploration for off-policy RL.

  • Rainbow DQN and SAC improve sample efficiency in discrete and continuous tasks by reusing past experiences and combining established techniques.
  • Stabilizing Q-learning: Double Q-learning and related methods use multiple value estimators to address overestimation and instability in Q-learning.
  • Stabilizing Q-learning: SUNRISE proposes ensemble-based uncertainty estimates to provide more stable, higher-signal-to-noise Bellman backups.
  • Ensemble methods in RL: Ensembles have also been used in model-based RL to reduce dynamics-model errors and generate imagined experiences for policy learning.
  • Exploration in RL: Existing exploration methods include maximum entropy, bonus rewards, and randomization, but may focus agents on irrelevant environmental aspects.

3. Background

The paper frames RL as discounted-return maximization and describes SAC as an off-policy maximum-entropy actor–critic method. Figure 1 illustrates independent ensemble agents and their uncertainty estimates.

  • Reinforcement learning: At each timestep, an RL agent maps a state to an action, receives a reward, and transitions to a new state.
  • Reinforcement learning: The discounted return sums future rewards with discount factor γ, and RL maximizes its expected value.
  • Soft Actor-Critic: SAC is an off-policy actor–critic method that maximizes a weighted combination of reward and policy entropy.
  • Figure 1 depicts N independent agents sharing one replay buffer and shows ensemble mean and variance over ten regression models.
  • Soft Actor-Critic: SAC alternates between soft policy evaluation and soft policy improvement, using a replay buffer and delayed parameters for Q-function updates.

4. SUNRISE

SUNRISE builds an ensemble of off-policy agents whose uncertainty estimates weight Bellman backups and guide UCB exploration. Bootstrap with random initialization supplies diversity for integrating these components.

  • SUNRISE is designed to combine ensemble techniques with modern off-policy algorithms such as SAC and Rainbow DQN.
  • Ensemble construction: Each ensemble member has its own soft Q-function, policy, and delayed target Q-function, with no shared parameters between agents.
  • Weighted Bellman backups: Errors in target Q-functions propagate through Bellman backups and can produce inconsistent and unstable Q-function convergence.
  • Weighted Bellman backups: The weighted backup uses a confidence weight based on the empirical standard deviation of target Q-functions, down-weighting high-variance transitions.
  • Bootstrap with random initialization: Bootstrap with random initialization creates agent diversity through randomized parameters and different training samples.
  • UCB exploration: UCB exploration adds ensemble standard deviation as a bonus and, in continuous action spaces, selects among N candidate actions sampled from ensemble policies.

5. Experimental results

The experiments evaluate SUNRISE across continuous and discrete control tasks, testing its weighted Bellman backups, UCB exploration, ensemble size, and comparison with larger single agents. SUNRISE consistently improves off-policy baselines across low- and high-dimensional environments, while weighted backups help under noisy rewards and UCB improves sparse-reward learning.

  • Experimental setup: SUNRISE is evaluated with SAC on continuous-control benchmarks and Rainbow DQN on Atari discrete-control tasks.The study covers OpenAI Gym, DeepMind Control Suite, and Atari games.
  • Comparative evaluation: SUNRISE consistently improves SAC on OpenAI Gym and RAD on DeepMind Control Suite, including environments with high-dimensional image observations.It outperforms model-based methods on all OpenAI Gym environments except Ant and SlimHumanoid-ET, and outperforms pixel-based RL methods in almost all DeepMind Control Suite environments.
  • Comparative evaluation: SUNRISE improves Rainbow DQN in almost all Atari environments and outperforms CURL and SimPLe on 11 of 26 games.These results support SUNRISE as a general approach across continuous and discrete control tasks.
  • Ablation study: Weighted Bellman backups improve sample efficiency and asymptotic performance under noisy rewards, with larger gains in complex environments.The weighted method outperforms SAC and DisCor on noisy-reward OpenAI Gym tasks, while target-Q errors are characterized by the confidence weight.
  • Ablation study: Under large reward noise, proposed weighted backups outperform random weighting and remain effective without UCB exploration.The comparison uses SlimHumanoid-ET with Gaussian reward noise z ∼ N(0, 5).
  • Ablation study: UCB exploration improves sample efficiency on sparse-reward environments, while SUNRISE gains cannot be reproduced by simply increasing a single agent’s parameters or updates.Increasing ensemble size improves performance but saturates around N = 5, motivating five agents in the experiments.

6. Discussion

DisCor’s weighted Bellman backups address slow learning from naive backups, but estimating cumulative Bellman errors can itself introduce approximation error. SUNRISE instead uses ensemble uncertainty and incurs ensemble computation overhead, while remaining parallelizable.

  • DisCor’s weighted Bellman backups target slow learning caused by error propagation in naive Bellman backups.They induce an optimal data distribution to make steady learning progress.
  • DisCor trains an error model to approximate cumulative discounted Bellman errors for weighting its backups.
  • Approximation error in estimating cumulative Bellman errors limits DisCor’s ability to handle error propagation.
  • SUNRISE replaces cumulative-error estimation with ensemble uncertainty for its weighted Bellman backups.The approach relies on ensembles producing calibrated variance estimates on unseen samples.
  • With N ensemble agents, SUNRISE requires N× inferences for weighted Bellman backups and 2N× inferences for actors and critics.The authors note that this overhead can be mitigated through parallelization.
  • Increasing updates or parameters alone does not reproduce SUNRISE’s gains.

7. Conclusion

SUNRISE unifies ensemble-based weighted Bellman backups, bootstrap diversity, and UCB exploration for modern off-policy RL. It is designed to stabilize learning, improve exploration, and apply across SAC and Rainbow DQN.

  • 7. Conclusion: SUNRISE combines weighted Bellman backups, bootstrap with random initialization, and UCB exploration in a unified off-policy RL framework.The framework is intended to work with algorithms including SAC and Rainbow DQN.
  • 7. Conclusion: The Rainbow implementation uses UCB action selection and bootstrap-weighted Q-function updates, while the SAC version can use random policy inference.Random inference selects one policy uniformly for an episode.
  • 7. Conclusion: Bootstrap with random initialization promotes ensemble diversity through distinct initial parameters and different training samples for each Q-function.Replay transitions receive Bernoulli bootstrap masks that scale each agent’s objective.
  • 7. Conclusion: Weighted backups reweight target Q-values using ensemble uncertainty, down-weighting transitions with high target variance.The confidence weight is bounded in [0.5, 1.0].
  • 7. Conclusion: UCB exploration selects actions using ensemble mean plus an uncertainty bonus, encouraging visits to unseen state-action pairs.For evaluation, the method instead selects actions maximizing the ensemble mean.

C. Implementation details for toy regression tasks

The toy regression experiment tests whether bootstrap ensembles provide useful uncertainty estimates under a nonlinear noisy-data setting.

  • C. Implementation details for toy regression tasks: The experiment generates twenty samples from y = x3 + ϵ with ϵ ∼ N(0, 3^2) and trains ten bootstrap ensembles.Each regression model is a fully connected network with two hidden layers and 50 rectified linear units per layer.
  • C. Implementation details for toy regression tasks: Each ensemble uses bootstrap masks drawn from a Bernoulli distribution to train regression networks on resampled data.The passage specifies bootstrap masking as the ensemble construction procedure.

D. Experimental setups and results: OpenAI Gym

OpenAI Gym experiments evaluate SUNRISE with SAC across four complex continuous-control environments. SUNRISE consistently improves SAC, while larger ensembles help until performance saturates near five agents.

  • E. Experimental setups and results: OpenAI Gym: The study evaluates SAC combined with SUNRISE on four complex OpenAI Gym environments.The implementation uses publicly released RLkit code without changing SAC hyperparameters or architectures.
  • E. Experimental setups and results: OpenAI Gym: SUNRISE consistently improves SAC performance by a large margin across the evaluated environments.The learning-curve comparison is reported across all environments.
  • E. Experimental setups and results: OpenAI Gym: Increasing ensemble size improves performance, but the improvement saturates around N = 5.The ensemble-size comparison uses learning curves across all environments.

E. Experimental setups and results: noisy reward

The supplied passages identify DisCor as a comparison method for addressing Q-learning error propagation and provide figure captions for learning-curve analyses. They do not report noisy-reward results directly.

  • E. Experimental setups and results: noisy reward: DisCor addresses Q-learning error propagation by learning a cumulative Bellman-error model and using it to reweight Bellman backups.The supplied implementation uses SAC with DisCor and sets the temperature to T = 10.
  • E. Experimental setups and results: noisy reward: Figure 4 compares SUNRISE with a single agent using learning curves whose lines and shaded regions summarize four-run means and standard deviations.The caption does not state a noisy-reward outcome.
  • E. Experimental setups and results: noisy reward: Figure 5 varies ensemble size N and reports learning-curve means and standard deviations across four runs.The supplied caption specifies the encoding but not the resulting comparison.

F. Experimental setups and results: DeepMind Control Suite

On DeepMind Control Suite, SUNRISE is evaluated with RAD and SAC. It produces small but consistent gains over RAD and more significant gains over SAC.

  • Training details: The RAD and SUNRISE experiments use shared actor-critic encoders for image observations and select the weighted-backup temperature from T ∈{10, 100}.The implementation follows a publicly released RAD repository and uses the same encoder architecture as prior work.
  • Learning curves: RAD combined with SUNRISE achieves small but consistent gains because RAD is already near optimal, leaving limited room for improvement.The comparison covers all environments shown in Figure 6(g-l).
  • Learning curves: SAC combined with SUNRISE shows more significant gains than the corresponding RAD combination on the DeepMind Control Suite environments.The paper uses Figure 6(a-f) to make the SAC improvement more apparent.

G. Experimental setups and results: Atari games

The Atari experiments combine sample-efficient Rainbow DQN with SUNRISE and evaluate learning curves across all environments. The setup trains five ensemble agents without modifying Rainbow's hyperparameters or architectures.

  • Training details: The Atari evaluation combines sample-efficient Rainbow DQN with SUNRISE using the publicly released Rainbow implementation without modifying hyperparameters or architectures.The setup selects T ∈{10, 40}, β ∈{0.5, 1.0}, and λ ∈{1, 10}.
  • Training details: SUNRISE's Atari setup trains five ensemble agents and tunes the weighted-backup temperature, Bernoulli mean, and penalty parameter over specified candidate sets.The candidate sets are T ∈{10, 40}, β ∈{0.5, 1.0}, and λ ∈{1, 10}.
  • Training details: Unlike Bootstrapped DQN, each Atari ensemble agent does not share its encoder.This is explicitly noted as an implementation distinction from Bootstrapped DQN.
Loading 2007.04938v4…