PaperlayerView exact revision on arXiv

Source-linked AI summary

Dropout Q-Functions for Doubly Efficient Reinforcement Learning

Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, Yoshimasa Tsuruoka

arXiv:2110.02034v2cs.LGcs.AI

TL;DR

RL methods need better sample efficiency, but REDQ’s large Q-function ensemble makes it computationally expensive. The paper proposes DroQ, a REDQ variant using a small ensemble of dropout Q-functions with layer normalization, and reports comparable sample efficiency with substantially better computational efficiency.

  • Problem

    REDQ achieves high sample efficiency with a large Q-function ensemble but is less computationally efficient than non-ensemble methods such as SAC.

  • Method

    DroQ uses a small ensemble of Q-functions equipped with dropout connections and layer normalization as a REDQ variant.

  • Results

    DroQ achieves sample efficiency comparable to REDQ, runs more than two times faster than REDQ, and has process times comparable to SAC.

  • Takeaways & Limitations

    DroQ is doubly efficient in the reported experiments, combining REDQ-level sample efficiency with substantially improved computational and memory efficiency.

  • Takeaways & Limitations

    The computational-efficiency experiment keeps the number of hidden-layer units invariant across environments.

Abstract

from arXiv · show

Randomized ensembled double Q-learning (REDQ) (Chen et al., 2021b) has recently achieved state-of-the-art sample efficiency on continuous-action reinforcement learning benchmarks. This superior sample efficiency is made possible by using a large Q-function ensemble. However, REDQ is much less computationally efficient than non-ensemble counterparts such as Soft Actor-Critic (SAC) (Haarnoja et al., 2018a). To make REDQ more computationally efficient, we propose a method of improving computational efficiency called DroQ, which is a variant of REDQ that uses a small ensemble of dropout Q-functions. Our dropout Q-functions are simple Q-functions equipped with dropout connection and layer normalization. Despite its simplicity of implementation, our experimental results indicate that DroQ is doubly (sample and computationally) efficient. It achieved comparable sample efficiency with REDQ, much better computational efficiency than REDQ, and comparable computational efficiency with that of SAC.

1 INTRODUCTION

RL methods often require millions of interactions, making sample efficiency important for practical applications. REDQ improves sample efficiency with high update-to-data ratios and large Q-function ensembles, but DroQ targets REDQ’s computational cost while retaining comparable sample efficiency.

  • Motivation: Millions of training samples are generally required for RL tasks, creating a severe obstacle to practical applications with limited resources.The paper motivates improved sample efficiency for real-world environments and settings without simulators.
  • High update-to-data ratios: High update-to-data ratios can improve sample efficiency by sufficiently training Q-functions within relatively few environment interactions.MBPO is cited as using a UTD ratio of 20–40, compared with SAC’s ratio of 1.
  • REDQ: REDQ combines a high update-to-data ratio with a large Q-function ensemble to suppress estimation bias and achieve high sample efficiency.The paper reports REDQ’s sample efficiency as equal to or better than MBPO on the cited benchmark comparison.
  • Computational-efficiency gap: REDQ remains less computationally efficient than non-ensemble methods such as SAC because its large ensemble requires more computation and memory.This matters for lightweight on-device applications and rapid reinforcement-learning development cycles.
  • DroQ: DroQ uses a small ensemble of dropout Q-functions with dropout and layer normalization to improve computational efficiency while maintaining sample efficiency comparable to REDQ.The authors report more than a twofold computational-efficiency improvement over REDQ.
  • Additional contributions: DroQ also contributes a simple implementation and a successful demonstration of dropout in high update-to-data-ratio settings.The method can be implemented by adding readily available dropout and layer-normalization functions to REDQ or SAC Q-functions.

2 PRELIMINARIES

The paper frames RL as sequential state-action-reward interaction and focuses on maximum-entropy objectives. REDQ improves sample efficiency through high update-to-data ratios and ensembles that reduce the resulting overestimation bias, although the ensemble increases computational cost.

  • RL setting: RL models an agent interacting with an environment by observing states, selecting actions, and receiving rewards and subsequent states.The paper uses r, s, a, s′, and a′ as shorthand for reward, state, action, next state, and next action.
  • Maximum-entropy RL: Maximum-entropy RL seeks a policy that maximizes expected return together with an entropy bonus.The temperature α balances exploitation and exploration and affects policy stochasticity.
  • REDQ components: REDQ uses a high update-to-data ratio, meaning many agent updates are performed relative to actual environment interactions.This promotes sufficient Q-function training within a few interactions but increases overestimation bias.
  • REDQ components: REDQ uses a randomly selected subset of an independently initialized Q-function ensemble to calculate the target and reduce overestimation bias.The cited configuration uses a large ensemble of N = 10 and a small target subset of M = 2.
  • Computational trade-off: Using a large Q-function ensemble reduces bias and improves sample efficiency but makes REDQ computationally intensive.The paper therefore motivates reducing the ensemble size.

3 INJECTING MODEL UNCERTAINTY INTO TARGET WITH DROPOUT Q-FUNCTIONS

DroQ replaces REDQ’s large ensemble with a small ensemble of dropout Q-functions that inject model uncertainty into target values. Dropout and layer normalization preserve the target-estimation mechanism while reducing the number of Q-function updates and computational cost.

  • Method: DroQ replaces REDQ’s large Q-function ensemble with a small ensemble of dropout Q-functions.The proposed dropout Q-functions combine dropout and layer normalization, and the ensemble can be smaller than REDQ’s.
  • REDQ uncertainty injection: In REDQ, randomly selecting Q-functions for the target approximates an expected target value under model-parameter uncertainty.The approximation uses a proposal distribution based on resampling ensemble parameters and a one-sample average.
  • Dropout Q-functions: DroQ uses dropout connections and layer normalization to construct Q-functions whose target-value approximation injects model uncertainty.Layer normalization is applied after dropout for more effective use of dropout.
  • Target construction: The target value is approximated using M independently initialized and trained dropout Q-functions and their minimum Q-value.This uses a dropout-based proposal distribution in place of REDQ’s ensemble-resampling distribution.
  • Algorithm: DroQ modifies REDQ by using M dropout Q-functions for target calculation, Q-function updates, and policy updates instead of N Q-functions.The method uses M ≤ N, with M much smaller than N in practice.
  • Computational efficiency: Using fewer Q-functions makes DroQ more computationally efficient by reducing the number of Q-function updates.The paper gives M = 2 and N = 10 as an example configuration from REDQ.

4 EXPERIMENTS

Experiments compare DroQ with REDQ, SAC, DUVN, and ablations on MuJoCo tasks, assessing sample efficiency, estimation bias, computational cost, memory, and component effects. DroQ matches REDQ's sample efficiency while substantially reducing process time, parameters, and memory.

  • Experimental setup: Experiments used Hopper, Walker2d, Ant, and Humanoid, comparing REDQ, SAC, DroQ, and DUVN with average return and normalized estimation bias.Process time, parameter count, and bottleneck memory were evaluated separately on hardware with two Intel Xeon CPUs and one NVIDIA Tesla K80.
  • Sample efficiency and bias reduction: DroQ and REDQ achieved almost the same sample efficiency overall, with both generally improving earlier than SAC and DUVN.Their learning curves overlapped in Walker2d and Ant; REDQ was slightly better in Humanoid, while DroQ was better in Hopper.
  • Sample efficiency and bias reduction: DroQ and REDQ kept value-estimation bias closer to zero than SAC and DUVN across all evaluated environments.
  • Computational efficiency: More than two times faster per update, DroQ required 800–900 msec versus REDQ's 2200–2300 msec.The results indicate that Q-function learning dominates overall process time, making compact Q-function sets important for speed.
  • Computational efficiency: About one-fifth the parameters and about one-third the bottleneck memory of REDQ, DroQ was substantially more memory efficient.DroQ used the same number of Q-functions as SAC, so their parameter counts were equal.
  • Ablation study: DroQ significantly outperformed -DO, -LN, and -DO-LN in average return and bias reduction, especially in complex Ant and Humanoid environments.The ablations removed dropout, layer normalization, or both, indicating a strong synergistic effect between the two components.

5 RELATED WORK

Related work covers ensemble methods, transition-model ensembles, dropout-based uncertainty, and normalization in reinforcement learning. DroQ differs by combining multiple dropout Q-functions, layer normalization, and a high-UTD-ratio setting.

  • Comparison framework: The related-work comparison classifies studies by model-uncertainty type, uncertainty-injection method, and reinforcement-learning setting.
  • Ensembles: Prior reinforcement-learning studies use ensembles of Q-functions to represent model uncertainty and ensembles of transition models in model-based methods.
  • Dropout Q-functions: Earlier dropout Q-function methods used a single dropout Q-function for target calculation, whereas DroQ uses multiple dropout Q-functions and their minimum output.DroQ also applies layer normalization and targets the more challenging high-UTD-ratio setting.
  • Normalization in reinforcement learning: Normalization methods, including batch normalization and layer normalization, have previously been introduced into reinforcement-learning algorithms.

6 CONCLUSION

DroQ combines a small ensemble of dropout Q-functions with layer normalization, achieving comparable sample efficiency to REDQ while improving computational and memory efficiency. Ablations indicate that the combination is especially beneficial in complex environments.

  • DroQ uses a small ensemble of Q-functions equipped with dropout connections and layer normalization.
  • DroQ significantly improves computational and memory efficiency over REDQ while achieving comparable sample efficiency.
  • Using both dropout and layer normalization has a synergistic effect on sample efficiency, especially in complex environments such as Humanoid.
  • Figure 4 compares average return and estimation bias across DroQ dropout rates.
  • Figure 5 evaluates DroQ without layer normalization across different dropout rates.

A.3 SIN-DROQ: DROQ VARIANT USING A SINGLE DROPOUT Q-FUNCTION

Sin-DroQ replaces DroQ’s multiple dropout Q-functions with repeated evaluations of a single dropout Q-function. Its lower average return indicates that multiple dropout Q-functions are preferable.

  • Sin-DroQ uses a single dropout Q-function to calculate the target through M repeated evaluations.
  • Dropout causes the single Q-function’s output to differ across evaluations.
  • The remaining components of Sin-DroQ are the same as DroQ.
  • Sin-DroQ achieved lower average return than DroQ, favoring multiple dropout Q-functions.
  • Figure 6 compares average return and estimation bias for DroQ and Sin-DroQ.

B REDQ WITH DIFFERENT ENSEMBLE SIZE N

Compared with REDQ variants using different ensemble sizes, DroQ generally achieves stronger returns and comparable or better estimation bias while using less computation and memory than larger REDQ ensembles.

  • REDQ with ensemble size N is denoted REDQN, such as REDQ5 for an ensemble of five.
  • DroQ was superior to REDQ2–5 in overall average return and somewhat better than REDQ5 across all environments.
  • DroQ’s estimation bias was significantly better than REDQ2 and comparable with REDQ3–10.
  • DroQ ran as fast as REDQ3 and 1.4 to 1.5 times faster than REDQ5.
  • DroQ was less memory intensive than REDQ3–10.
  • Table 5 reports process times, while Tables 6 and 7 report parameter counts and bottleneck memory consumption.

C ADDITIONAL ABLATION STUDY OF DROQ

Ablations show that dropout in both target and current Q-functions is important for DroQ’s performance, while layer normalization complements dropout by reducing instability, especially in complex environments.

  • Using dropout for both target and current Q-functions is effective in the ablation study.
  • Removing dropout from target Q-functions significantly worsened average return in Ant.
  • Removing dropout from current Q-functions significantly worsened average return in Humanoid.
  • Variants without dropout in target Q-functions were significantly worse than DroQ in all environments.
  • The benefit of using dropout for both current and target Q-functions also holds in high-UTD settings.
  • Layer normalization and dropout have a synergistic effect, especially in Ant and Humanoid.
  • Without layer normalization, dropout produces larger Q-function-loss oscillations and corresponding gradient-variance oscillations.

E EFFECTS OF OTHER NORMALIZATION METHODS

The study compares batch normalization, group normalization, and layer normalization without variance re-scaling as alternatives to layer normalization in DroQ variants. Group normalization works when combined with dropout, whereas batch normalization and layer normalization without variance re-scaling perform poorly or destabilize Q-function learning.

  • E EFFECTS OF OTHER NORMALIZATION METHODS: The experiments compare batch normalization, group normalization, and layer normalization without variance re-scaling in six DroQ variants.The variants replace layer normalization alone or alongside dropout.
  • E EFFECTS OF OTHER NORMALIZATION METHODS: Batch normalization fails to significantly improve average return or estimation bias and produces unstable Q-function learning in Humanoid and Ant.These findings are reported for +BN and +DO+BN.
  • E EFFECTS OF OTHER NORMALIZATION METHODS: Group normalization improves average return and estimation bias when combined with dropout, while group normalization alone does not.The comparison concerns +DO+GN versus +GN in Humanoid and Ant.
  • E EFFECTS OF OTHER NORMALIZATION METHODS: Layer normalization without variance re-scaling does not significantly improve average return or estimation bias, especially in Humanoid, and yields less stable Q-function learning.The compared variants are +LNwoVR and +DO+LNwoVR.
  • E EFFECTS OF OTHER NORMALIZATION METHODS: Layer normalization normalizes all input elements jointly, whereas group normalization divides them into two groups and normalizes each group separately.Both methods use means and variances, but group normalization computes them within subgroups.

F RELATION BETWEEN (1) ENSEMBLE SIZE AND (2) THE EFFECT OF LAYER

The additional experiment varies the number of dropout Q-functions and tests the roles of layer normalization and dropout. Larger ensembles improve sample efficiency in complex environments, while the most effective normalization combination depends on ensemble size.

  • F RELATION BETWEEN (1) ENSEMBLE SIZE AND (2) THE EFFECT OF LAYER: DroQN varies the ensemble size N by bootstrapping a subset M of N dropout Q-functions.It is a DroQ variant and can also be viewed as REDQ using N dropout Q-functions.
  • F RELATION BETWEEN (1) ENSEMBLE SIZE AND (2) THE EFFECT OF LAYER: The experiments compare DroQN and ablations without layer normalization, or without both layer normalization and dropout, for N = 2, 3, 5, 10.The latter ablation is identical to REDQN.
  • F RELATION BETWEEN (1) ENSEMBLE SIZE AND (2) THE EFFECT OF LAYER: In Ant and Humanoid, increasing ensemble size improves sample efficiency, with Humanoid showing monotonic improvement for N = 2, 3, 5, 10.In Ant, DroQ5–10 are more sample efficient than DroQ2–3.
  • F RELATION BETWEEN (1) ENSEMBLE SIZE AND (2) THE EFFECT OF LAYER: For small ensembles, layer normalization and dropout work synergistically, while for N ≥5 the effect of layer normalization becomes dominant.DroQ2–3 outperform versions without dropout, whereas layer-normalized DroQ5–10 outperform versions without layer normalization.
  • F RELATION BETWEEN (1) ENSEMBLE SIZE AND (2) THE EFFECT OF LAYER: The authors suggest using layer normalization with dropout for small ensembles and layer normalization alone for large ensembles.This recommendation follows the ensemble-size ablation results.

I EXPERIMENTS ON THE ORIGINAL REDQ CODEBASE

Experiments replicated on the original REDQ codebase evaluate computational efficiency with wallclock time and retuned dropout rates. The replication includes the main comparison and ablations across normalization, dropout, and dropout-rate settings.

  • I EXPERIMENTS ON THE ORIGINAL REDQ CODEBASE: The replication experiments were implemented on the original REDQ codebase to reproduce results from the main content.The original main experiments used the SAC codebase.
  • I EXPERIMENTS ON THE ORIGINAL REDQ CODEBASE: Computational efficiency is evaluated by plotting wallclock time required to complete a specified number of environment interactions.This replaces the standard-deviation-of-bias plot used in the corresponding main experiment.
  • I EXPERIMENTS ON THE ORIGINAL REDQ CODEBASE: The replication covers the main comparison and the ablations corresponding to normalization methods, ensemble settings, and dropout rates.Figures 14, 15, and 16 replicate results from Sections 4.1, 4.3, and Appendix A.1.
  • I EXPERIMENTS ON THE ORIGINAL REDQ CODEBASE: Dropout rates were retuned separately for each environment in the replicated experiments.The retuned values are listed in Table 9.
Loading 2110.02034v2…