Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics

Arsenii Kuznetsov; Pavel Shvechikov; Alexander Grishin; Dmitry Vetrov

Source-linked AI summary

Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics

Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, Dmitry Vetrov

arXiv:2005.04269v1May 8, 2020cs.LGcs.AIstat.ML

TL;DR

Accurate Q-function estimation is difficult in off-policy continuous control because approximation errors can accumulate and produce overestimation. TQC addresses this with distributional critics, truncated return distributions, and ensembling. It improves performance across the standard benchmark suite, while incurring computational overhead and leaving the link between aleatoric uncertainty and overestimation insufficiently understood.

Problem
Off-policy continuous-control learning needs accurate Q-function approximation, but approximation errors can accumulate and overestimation control through minima is coarse.
Method
TQC combines distributional critic representations, truncation of top return-distribution atoms, and ensembling to control overestimation.
Results
TQC improves performance on all environments in the standard MuJoCo-powered OpenAI Gym benchmark suite, with up to 30% improvement on some environments.
Takeaways & Limitations
Distributional representations and truncation provide fine control over under- and overestimation, while ensembling adds performance improvement.
Takeaways & Limitations
The connection between aleatoric uncertainty and overestimation remains insufficiently understood.

Abstract

from arXiv · show

The overestimation bias is one of the major impediments to accurate off-policy learning. This paper investigates a novel way to alleviate the overestimation bias in a continuous control setting. Our method---Truncated Quantile Critics, TQC,---blends three ideas: distributional representation of a critic, truncation of critics prediction, and ensembling of multiple critics. Distributional representation and truncation allow for arbitrary granular overestimation control, while ensembling provides additional score improvements. TQC outperforms the current state of the art on all environments from the continuous control benchmark suite, demonstrating 25% improvement on the most challenging Humanoid environment.

1. Introduction

Accurate Q-function approximation is central to stable, effective off-policy learning, yet continuous-control methods face overestimation and coarse min-based correction. TQC combines distributional critics, truncation, and ensembling to provide finer control and strong benchmark performance.

Motivation: Approximation errors matter because Q-functions provide temporal-difference targets and guide action selection or policy optimization.In off-policy learning, errors can propagate backward through episodes and accumulate over training.
Limitations of prior methods: Continuous-control methods can remain overestimated in highly variable state-action regions, while taking minima over more critics increases underestimation.The cited approach also ignores all estimates except the minimum, reducing the ensemble's informational use.
TQC: TQC combines distributional critic representations, truncation of predicted return distributions, and ensembling.The method adapts quantile-based distributional critics to continuous control and models return quantiles conditioned on state and action.
TQC: Truncating top return-distribution atoms enables fine-grained control over under- and overestimation by varying the number of dropped atoms.Typically, around 8% of atoms are dropped, and truncation can outperform multi-approximator methods even with one approximator.
TQC: Ensembling is decoupled from overestimation control and supplies additional performance improvement.The truncation operation itself does not constrain the number of approximators.
Evaluation: TQC improves performance across all environments in the standard MuJoCo-powered OpenAI Gym continuous-control benchmark suite.The introduction reports up to 30% improvement on some environments and twice the previous state-of-the-art running speed on Humanoid; the method adds computational overhead.

2. Background

The section introduces the MDP, SAC, and distributional reinforcement learning foundations used by TQC. It contrasts modeling return distributions with modeling expected returns and describes quantile-based approximation.

Soft Actor Critic: SAC is an off-policy actor-critic algorithm that augments rewards with policy entropy to encourage stochasticity.
Soft Actor Critic: The soft Q-function is learned by minimizing a soft Bellman residual using a temporal difference target.
Soft Actor Critic: SAC dynamically adjusts the entropy temperature α toward a target entropy, usually set to HT = −dim A.
Distributional Reinforcement Learning: Distributional reinforcement learning models the return random variable Zπ(s, a) rather than only its expectation, the Q-function.
Distributional Reinforcement Learning: QR-DQN represents the return distribution with quantile locations learned by quantile regression at fixed fractions τm.
Distributional Reinforcement Learning: The quantile critic parameters are optimized against the distributional Bellman target by minimizing an averaged 1-Wasserstein distance, using a Huber quantile loss.

3. Truncated Quantile Critics, TQC

TQC combines distributional critics, truncation, and ensembling to control overestimation in continuous control. It truncates pooled target atoms during value learning while using nontruncated estimates for policy optimization.

3.1. Overview: TQC decomposes expected returns into distributional atoms, whose number controls the precision of the return-distribution approximation.
3.1. Overview: TQC controls overestimation by dropping the largest atom locations and averaging the remaining atoms to estimate the Q-value.Varying the total atoms and dropped atoms balances underestimation against overestimation.
3.1. Overview: Higher return variance produces a lower truncated Q-value estimate, accounting for inflated overestimation through right-tail removal.
3.2. Computation of the target distribution: TQC pools distributions from N critics, sorts the mixture atoms, and removes the largest locations before forming the target distribution.
3.2. Computation of the target distribution: TQC truncates target distributions during value learning but uses nontruncated critic estimates for policy optimization to avoid double truncation.
3.2. Computation of the target distribution: The critics are trained against the truncated temporal difference target distribution using quantile regression and a 1-Wasserstein objective.
3.3. Algorithm: The practical algorithm alternates environment interaction, replay-buffer sampling, entropy-temperature updates, policy updates, critic updates, and target-network averaging.

4. Experiments

The experiments evaluate TQC for overestimation-bias correction in a simple continuous-action MDP and against state-of-the-art methods on MuJoCo environments. TQC achieves low approximation bias and variance and consistently stronger benchmark performance.

4.2. Comparative Evaluation: TQC is compared with SAC, TrulyPPO, and TD3 on MuJoCo environments implemented in OpenAI Gym.The evaluation reports learning curves and seed-return statistics over extended training ranges for several environments.
4.1. Single state MDP: TQC is evaluated against bias-correction methods in a single-state, one-dimensional continuous-action infinite-horizon MDP.The experiment trains Q-networks or Z-networks from scratch and compares their approximation of the true Q-function.
4.1. Single state MDP: The study measures signed Q-value discrepancy, robust-average bias, and robust-average variance across 100 seeds.The robust average truncates 10% from each tail, while TQC varies the number of dropped quantiles and AVG and MIN vary the number of networks.
4.2. Comparative Evaluation: TQC can achieve the lowest Q-function approximation variance and smallest bias among the evaluated competitors.Bias and variance also correlate well with policy performance in the experiment.
4.2. Comparative Evaluation: TQC performs consistently better than the competitors and improves on the maximal published score in four of five environments.The reported benchmark results average 10 seeds, with each seed performance averaged over its last 100 evaluations.

5. Ablation study

The ablation study isolates TQC's design choices, including quantile distributions, truncation, pooling, and ensembling, across Humanoid and Walker2d. Quantile representations, truncation, and ensembling improve relevant outcomes, while performance saturates with larger ensembles and is relatively insensitive to atom count above 10.

5.1. Design choices evaluation: The ablation compares incremental modifications from SAC to TQC on Humanoid and Walker2d, including network size, quantile networks, truncation, pooling, and ensembling.The study averages metrics over four seeds and uses two networks for modifications before the final ensembling comparison.
5.1. Design choices evaluation: Quantile distributional networks improve performance in both environments, with the most notable improvement on Humanoid.Increasing network size alone does not necessarily improve SAC, although some improvement is visible on Walker2d.
5.1. Design choices evaluation: Truncation stabilizes Walker2d results and appears to reduce seed variance on Humanoid, while the preferred pooling-truncation order differs by environment.TQC is the winner among the compared variants on Humanoid, whereas truncate-pooling performs better on Walker2d.
5.1. Design choices evaluation: Ensembling consistently improves results on both Humanoid and Walker2d.The study explicitly compares TQC with two and five Z-networks to illustrate the effect of ensembling.
5.2. Sensitivity to hyperparameters: Truncation is essential, with an optimal dropped-atom count of d = 2 or d = 3 when each network has M = 25 atoms.The experiment varies d from 0 to 5 and drops dN atoms in total.
5.2. Sensitivity to hyperparameters: For M ≥15, learning curves are indistinguishable, whereas performance improvement from ensembling saturates at approximately N = 3.A single network is consistently inferior to larger ensembles, while very small M, such as 10, is an exception to the atom-count insensitivity.
5.2. Sensitivity to hyperparameters: Distributional networks and ensembling incur additional computational overhead.The paper quantifies this overhead for different methods in its timing table.

6. Related work

TQC differs from prior overestimation-control methods by using quantile representations to control bias, rather than selecting or weighting multiple approximators. Its design combines distributional critics, truncation, and optional ensembling.

Distributional methods: Prior work applies distributional methods to learn quantile values or quantile fractions, including in continuous-control settings.The paper distinguishes its objective—overestimation-bias control—from exploration methods based on quantile value functions.
Overestimation bias: Overestimation bias is studied across statistics, economics, and reinforcement learning, where it is also called max-operator bias or the winner’s curse.Statistical approaches include maximum and double estimators, with later work addressing their respective biases.
Ensembling and minimization: Continuous-control methods such as TD3 reduce overestimation by taking the minimum across approximators, extending the minimization idea used in related ensemble methods.The paper notes that increasing ensemble size can intensify underestimation.
TQC distinction: TQC controls overestimation through quantile representation instead of choosing between or weighting multiple approximators, while using ensembling for further performance improvement.The method therefore supports bias control with a single approximator and separates that control from ensemble multiplicity.

7. Conclusion and Future Work

The paper proposes controlling overestimation through aleatoric uncertainty using distributional representations, truncation, and ensembling. Simulations and benchmark results support fine-grained bias control, while the uncertainty connection remains insufficiently understood.

Conclusion: TQC combines distributional representations, distribution truncation, and ensembling to control overestimation bias using aleatoric uncertainty.These are presented as the method’s three essential ideas.
Conclusion: Simulations show low expected approximation-error variance and fine control over underestimation and overestimation.The benchmark findings suggest distributional representations may help control overestimation bias.
Future work: The connection between aleatoric uncertainty and overestimation remains insufficiently understood and is identified as a direction for future work.The paper explicitly frames further investigation of this connection as future research.

A. Experimental setting

The experimental setting documents the MuJoCo and environment versions used, the critic-network output size, and the hyperparameter tables for TQC.

Software environment: MuJoCo 2.0 with Gym versions through v0.15.4 can invalidate comparisons because Gym nullifies state components corresponding to contact forces.The paper cautions that this makes results incomparable to previous works.
Software environment: The experiments use MuJoCo 1.5 and v3 versions of the environments.Versions of the other packages are listed in the distributed Conda environment file.
Network architecture: Critic networks are fully connected, with the final layer output size equal to the number of atoms M.This output dimension represents the distributional critic’s atom count.
Hyperparameters: Tables 5 and 6 report general hyperparameter values and environment-dependent TQC hyperparameters, respectively.The supplied passages identify the tables and their scopes but do not provide their entries.

C. Toy experiment setting

The toy experiment uses a one-state, one-dimensional stochastic-reward MDP to evaluate bias-correction methods under controlled action and data-generation procedures. Policies are evaluated greedily from learned value approximations.

Environment: The toy environment is an infinite-horizon MDP with one state, actions in [−1, 1], stochastic rewards, and noise standard deviation σ = 0.25.The mean reward is a cosine with slowly increasing amplitude.
Environment: The reward function has three local maxima, with the global optimum near a∗≈0.31; the optimal policy always selects that action.The other local maxima occur near the left end and at the right endpoint.
Training setup: Q-networks or Z-networks use two hidden layers of size 50 and train for 3000 iterations on a replay buffer of size 50.The toy experiment samples rewards over an action grid of size 50.
Evaluation: The experiment varies bias-control parameters, including dropped TQC quantiles d = M −k from specified values out of 25 atoms, and reports robust averages over 100 seeds.Reported quantities include the mean and variance of the signed approximation discrepancy over a dense action grid.
Data and evaluation: The implicit deterministic policy is obtained by taking the argmax over a dense action grid with step Δa = 0.001.This avoids policy-optimization subtleties when assessing Q-function approximation quality.
Data and evaluation: Each dataset samples one reward per action from a uniform grid, and the greedy policy is defined implicitly by the learned value function.The policy does not interact with the environment during this evaluation.

E. Additional experimental results

The additional experiments vary the number of atoms per critic and the number of critic networks in TQC. Results are smoothed over a 100-step window with ± standard deviation plotted.

Figure 9 varies the number of critic networks N for TQC with M = 25 atoms per critic and d = 2 dropped atoms per critic.
The plotted experiments use smoothing with a window of 100 and display ± standard deviation.
Figure 10 varies the number of atoms per critic M for TQC with N = 5 critics and d = 2 dropped atoms per critic.

E.3. Removed atoms stats

The removed-atom statistics show that truncation is uneven across critics after pooling, but no single critic consistently overestimates across the entire state-action space. Increasing the number of critics under SAC’s minimum operation instead produces more underestimation, without improving published MuJoCo performance.

Removed atoms distribution: 65/35% for N = 2 and 35/25/18/13/9% for N = 5 describe sorted, replay-averaged proportions of dropped atoms across critics.TQC drops atoms with the largest locations after pooling atoms from multiple Z-networks.
Removed atoms distribution: Without sorting, replay-averaged dropped-atom proportions are almost perfectly equal across critics.
Removed atoms distribution: The results suggest that critics overestimate more in different state-action-space regions rather than one critic systematically overestimating everywhere.
Removed atoms distribution: Figures 11 and 12 use N = 2 and N = 5 critics with M = 25 and d = 2 dropped atoms per critic.Figure 11 sorts proportions before averaging, whereas Figure 12 averages them without sorting.
Comparison with SAC: Increasing the number of networks under SAC’s minimum operation produces more underestimation.
Comparison with SAC: For MuJoCo benchmarks, coarse overestimation control did not improve published performance with either regular or increased network sizes.The comparison varies critic-network counts under the Clipped Double Q-learning estimate for SAC.