Source-linked AI summary
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
Xinyue Chen, Che Wang, Zijian Zhou, Keith Ross
TL;DR
Model-based methods achieve strong continuous-action sample efficiency using high UTD ratios, raising whether comparable performance is possible without a model. REDQ addresses this with a model-free combination of high UTD, Q-function ensembles, and randomized target minimization, matching or exceeding MBPO on MuJoCo. The results indicate that models may not be necessary for high sample efficiency on this benchmark.
Problem
Model-based methods had higher sample efficiency than previous model-free methods in continuous-action benchmarks, motivating whether high performance was possible without a model.
Method
REDQ combines a UTD ratio much greater than one, an ensemble of Q-functions, and minimization over a random subset of ensemble Q-functions in the target.
Results
REDQ attains sample-efficiency performance as good as or better than state-of-the-art model-based algorithms on the MuJoCo benchmark.
Takeaways & Limitations
For the MuJoCo benchmark, models may not be necessary to achieve high sample efficiency with continuous-action model-free DRL.
Takeaways & Limitations
The theoretical analysis assumes identically distributed approximation errors across Q-functions for each fixed state-action pair.
Abstract
from arXiv · showhide
Using a high Update-To-Data (UTD) ratio, model-based methods have recently achieved much higher sample efficiency than previous model-free methods for continuous-action DRL benchmarks. In this paper, we introduce a simple model-free algorithm, Randomized Ensembled Double Q-Learning (REDQ), and show that its performance is just as good as, if not better than, a state-of-the-art model-based algorithm for the MuJoCo benchmark. Moreover, REDQ can achieve this performance using fewer parameters than the model-based method, and with less wall-clock run time. REDQ has three carefully integrated ingredients which allow it to achieve its high performance: (i) a UTD ratio >> 1; (ii) an ensemble of Q functions; (iii) in-target minimization across a random subset of Q functions from the ensemble. Through carefully designed experiments, we provide a detailed analysis of REDQ and related model-free algorithms. To our knowledge, REDQ is the first successful model-free DRL algorithm for continuous-action spaces using a UTD ratio >> 1.
1 INTRODUCTION
REDQ asks whether a model-free algorithm using a high UTD ratio can match model-based sample efficiency, and introduces integrated mechanisms to make that possible.
- Motivation: Model-based methods achieve higher sample efficiency in continuous-action domains partly by using UTD ratios far above one.MBPO uses a UTD ratio of 20–40, whereas SAC uses 1.
- Main result: REDQ matches or exceeds MBPO while using fewer parameters and less wall-clock time.The paper presents this as evidence that high MuJoCo performance can be achieved without a learned model.
- Approach: REDQ is a model-free algorithm that performs updates using real data while combining a high UTD ratio, Q-function ensembles, and randomized in-target minimization.Unlike MBPO, REDQ uses no model-generated rollouts.
- Analysis: Experiments analyze Q-function bias and show that ensemble minimization reduces bias variability to close to zero for most training, even at very high UTD.The analysis also examines how the random-subset size controls average bias.
- Evaluation: The comparisons use a shared codebase for algorithmic evaluations, except that MBPO uses its authors’ code.The authors provide open-source code to support fair and reproducible comparisons.
2 RANDOMIZED ENSEMBLED DOUBLE Q-LEARNING (REDQ)
REDQ combines high-UTD model-free learning with randomized ensembles and target minimization, then evaluates the method against SAC and MBPO on challenging MuJoCo environments.
- Background: MBPO combines real and model-generated data, using UTD ratios above one to improve sample efficiency over model-free baselines.MBPO is introduced as a state-of-the-art model-based comparison method.
- REDQ design: REDQ uses an ensemble of independently initialized Q-functions, a shared target, and minimization over a random subset to reduce variance and over-estimation bias.Its UTD ratio G is much greater than one.
- REDQ design: REDQ separately controls Q-estimate variance and over-estimation bias through the ensemble size N and subset size M.This differs from Maxmin Q-learning, which minimizes over the full ensemble.
- Experimental results: REDQ and MBPO learn much faster than SAC, with REDQ somewhat better overall and stronger results particularly for Hopper.Across environments, REDQ is reported as 1.4x better than MBPO halfway through training and 1.1x better at the end.
3 WHY DOES REDQ SUCCEED WHEREAS OTHERS FAIL?
REDQ succeeds at high UTD because it combines ensemble Q functions with in-target minimization, stabilizing bias while allowing its average to be controlled. The analysis separates how ensemble size and subset size affect variance and expected bias.
- Empirical bias behavior: REDQ learns faster than SAC-20 and AVG while maintaining a much lower normalized standard deviation of bias for most training.Its bias is also a small, near-constant under-estimation for most of training.
- Empirical bias behavior: SAC-20’s high UTD produces fluctuating average bias, high non-uniform bias, and sensitivity to random initial conditions.These effects are especially associated with poor performance in Ant and Humanoid.
- Mechanism: The ensemble and in-target minimization are both critical for maintaining stable, near-uniform bias under high UTD ratios.AVG and SAC-20 each have only one of these components, while random subset selection can further improve REDQ.
- Theoretical analysis: Positive expected post-update bias indicates over-estimation accumulation, whereas negative expected post-update bias indicates under-estimation accumulation.The framework interprets the sign of post-update bias as the direction of accumulated estimation error.
- Theoretical analysis: Increasing M controls expected post-update bias, while increasing N reduces ensemble-average variance without changing that expected bias.This separates control of bias from control of variance, unlike Maxmin Q-learning where M=N.
- Theoretical analysis: The weighted variant’s target variance goes to zero as N increases, while finite N or unweighted REDQ retains target variance.The paper notes that some target variance may benefit overfitting reduction or exploration.
4 REDQ VARIANTS AND ABLATIONS
REDQ’s ablations show that ensemble size, in-target minimization, and target-selection mechanisms materially affect bias stability and learning performance. REDQ-OFE further improves REDQ on challenging Ant and Humanoid tasks.
- Ensemble size: Increasing ensemble size generally stabilizes average bias, lowers bias standard deviation, and strengthens performance; N = 5 already helps under high UTD.The ablation evaluates N = 2, 3, 5, 10, and 15 on Ant.
- In-target minimization: Increasing M lowers average bias, but excessively large M creates conservative estimates with large negative bias that hinder learning.Non-integer M values randomly alternate between neighboring subset sizes.
- In-target minimization: M = 2 provides the best overall performance by balancing small negative average bias with consistently small bias standard deviation.
- Target computation variants: Maxmin becomes unstable in continuous-action MuJoCo settings when larger ensembles accumulate highly negative bias, whereas REDQ and MinPair perform best similarly.Maxmin’s behavior contrasts with its reported usefulness in small finite-action Atari environments.
- Auxiliary feature learning: REDQ-OFE achieves 7x SAC’s sample efficiency for reaching 5000 on Ant and Humanoid, and exceeds MBPO by 3.12x at 150K and 1.26x at 300K data.Online feature extraction helps more on the challenging environments than on Hopper and Walker2d.
5 RELATED WORK
Related work addresses Q-learning bias, ensemble-based value estimation, and model-based learning with controlled model-generated data. REDQ builds on these directions while targeting high-UTD continuous-action learning.
- Q-learning bias: Double Q-Learning was introduced to address maximization bias and generally produces underestimation bias in the tabular setting.
- Ensembles: Ensemble methods have been used to reduce variance, enforce Bellman consistency through random mixtures, and control overestimation with Maxmin Q-learning.
- Model-based learning: Model-based methods such as MBPO combine model ensembles with controlled rollout horizons, while later work adds advanced sampling, bidirectional models, or model backpropagation.
6 CONCLUSION
The paper concludes that REDQ is a simple model-free algorithm achieving MuJoCo sample efficiency comparable to or better than state-of-the-art model-based methods. Its experiments explain its success under high UTD, and REDQ-OFE accelerates learning on difficult environments.
- Conclusion: REDQ achieves sample efficiency as good as or better than state-of-the-art model-based algorithms on the MuJoCo benchmark.The authors state that models may not be necessary for high sample efficiency on this benchmark.
- Conclusion: Experiments explain why REDQ succeeds when other model-free algorithms with high UTD ratios fail.
- Conclusion: Combining REDQ with OFE enables extremely fast learning on the challenging Ant and Humanoid environments.
A.2 PROOF OF THEOREM 1
The appendix establishes distributional properties of randomized subset minima and uses them to analyze REDQ’s bias and convergence. Under standard conditions, the tabular algorithm converges almost surely to the optimal Q function.
- Lemma 1: For i.i.d. variables, the sequence of sample minima decreases monotonically and converges almost surely to the lower endpoint of the distribution.
- Theorem 1: Minima over two equally sized random subsets have identical distributions when ensemble Q-values are i.i.d. across functions and actions.
- Theorem 1: As the subset size M grows, the target converges almost surely to the maximum across actions of the lower endpoints of the Q-value distributions.
- Convergence: The tabular weighted version of REDQ converges with probability 1 to the optimal Q function under standard Robbins-Monro step-size conditions.
- Convergence: The randomized tabular REDQ case also converges with probability 1 because its associated operator is an l∞ contraction and the stochastic approximation conditions apply.
B HYPERPARAMETERS AND IMPLEMENTATION DETAILS
REDQ experiments use a fixed ensemble and high UTD ratio, with initialization and implementation choices designed for fair comparison and efficient calculation.
- Implementation settings: REDQ uses an ensemble size N of 10 and a UTD ratio G of 20 across four MuJoCo environments.Most SAC-related hyperparameters match MBPO’s settings to ensure fair comparisons.
- Initialization: Training begins with 5000 randomly collected data points before parameter updates.Using a high UTD ratio immediately with very little data was found to cause SAC-20 to diverge.
- REDQ-OFE: REDQ-OFE first collects 20,000 random data points and pre-trains OFENet for 100,000 updates before joint training.The OFENet implementation omits batch-normalization layers and uses recommended hyperparameters.
- Weighted REDQ: The appendix describes a weighted REDQ variant that averages over all size-M subsets rather than sampling one random subset.The subsequent implementation reorders indices and uses an identity to avoid summing over all N choose M subsets.
C SAMPLE EFFICIENCY COMPARISON FOR REDQ, SAC AND MBPO
REDQ is substantially more sample efficient than SAC and achieves stronger reported performance than both SAC and MBPO across the evaluated MuJoCo environments.
- Sample efficiency: REDQ reaches specified performance levels with 5.0x the average sample efficiency of SAC across four environments.The reported environment-specific gains are 8x on Hopper, 5x on Ant, and 3.7x on Humanoid; including OFE raises the average to 6.4x.
- Performance comparison: At 150K and 300K steps, REDQ reaches 4.5x and 2.1x the average SAC performance, respectively.Including REDQ-OFE raises these ratios to 5.5x and 2.3x.
- Performance comparison: At 150K and 300K steps, REDQ reaches 1.4x and 1.1x the average MBPO performance, respectively.Including REDQ-OFE raises the reported 150K comparison to 1.8x MBPO performance.
- Evaluation measures: Table 2 reports data collected at specified performance levels, while Table 3 reports performance at specified data amounts.These tables support complementary comparisons of sample efficiency and fixed-data performance.
D NUMBER OF PARAMETERS COMPARISON
REDQ uses fewer parameters than MBPO across all four environments, including when OFENet is added in the reported Ant and Humanoid comparisons.
- REDQ versus MBPO: REDQ uses between 26% and 70% as many parameters as MBPO, depending on the environment.The comparison covers all four evaluated environments.
- REDQ-OFE versus MBPO: With OFENet, REDQ uses 80% and 35% as many parameters as MBPO on Ant and Humanoid, respectively.The reported Humanoid result combines stronger performance with fewer parameters.
- Parameter accounting: REDQ keeps the same network structure and ensemble size across the four environments.Parameter-count differences arise from environment-specific observation and action dimensions affecting network input and output layers.
E ADDITIONAL RESULTS FOR REDQ, SAC-20, AND AVG
Additional experiments examine Q-bias variability, policy delay, and UTD ratio, showing that REDQ maintains stronger performance and more controlled bias under high UTD.
- Q-bias analysis: Across four environments, REDQ has stronger performance and lower Q-bias standard deviation than SAC-20 and AVG.AVG or SAC-20 can have average bias slightly closer to zero in individual environments, but their bias standard deviation is consistently higher.
- Q-bias analysis: With UTD ratio 20, Figure 5 compares normalized Q-bias mean and standard deviation for REDQ, AVG, and SAC.The figure evaluates all three algorithms under the same UTD ratio.
- Policy delay: Policy delay consistently lowers and stabilizes bias and bias standard deviation, with a smaller effect on REDQ than on SAC.SAC always improves with policy delay, whereas REDQ improves in Hopper and Humanoid but becomes slightly worse in Walker2d and Ant.
- UTD ratio: In Ant, REDQ performs best at UTD 20, while SAC becomes much worse at UTD 20.The comparison covers UTD ratios 1, 5, 10, and 20.
H ADDITIONAL RESULTS FOR WEIGHTED VARIANT
Weighted and REDQ have similar Q-bias statistics, but REDQ generally performs better and is more robust across the four MuJoCo environments.
- Weighted and REDQ show similar average bias and standard deviation of bias across all four environments.
- Weighted performs worse than REDQ on Ant and Hopper, similarly on Humanoid, and slightly better on Walker2d.
- Overall, REDQ has stronger performance and greater robustness than Weighted.
- Conservative Q estimates can contribute to pessimistic underexploration, motivating exploration methods that do not rely on optimistic estimates.