PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors

Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Bo Cheng

arXiv:2001.02811v3cs.LGcs.AIeess.SY

TL;DR

Q-value overestimations from estimation errors can reduce RL policy performance. The paper proposes DSPI and DSAC, which learn continuous return distributions within off-policy maximum-entropy actor-critic RL; DSAC outperforms or matches nine baselines across MuJoCo tasks while reducing overestimation bias.

  • Problem

    Estimation errors in reinforcement learning can produce Q-value overestimations that lead to suboptimal policies.

  • Method

    DSAC embeds continuous state-action return-distribution learning into maximum-entropy soft policy iteration and implements it as an off-policy deep actor-critic method.

  • Results

    DSAC outperforms or matches nine baseline algorithms across MuJoCo tasks and reduces overestimation bias versus Single-Q SAC by 10.53%, 5.76%, 926.09%, and 1.89% on four tasks.

  • Takeaways & Limitations

    Learning return distributions can mitigate Q-value overestimations without introducing an additional value or policy network.

Abstract

from arXiv · show

In reinforcement learning (RL), function approximation errors are known to easily lead to the Q-value overestimations, thus greatly reducing policy performance. This paper presents a distributional soft actor-critic (DSAC) algorithm, which is an off-policy RL method for continuous control setting, to improve the policy performance by mitigating Q-value overestimations. We first discover in theory that learning a distribution function of state-action returns can effectively mitigate Q-value overestimations because it is capable of adaptively adjusting the update stepsize of the Q-value function. Then, a distributional soft policy iteration (DSPI) framework is developed by embedding the return distribution function into maximum entropy RL. Finally, we present a deep off-policy actor-critic variant of DSPI, called DSAC, which directly learns a continuous return distribution by keeping the variance of the state-action returns within a reasonable range to address exploding and vanishing gradient problems. We evaluate DSAC on the suite of MuJoCo continuous control tasks, achieving the state-of-the-art performance.

I. INTRODUCTION

The paper targets Q-value overestimations that can produce suboptimal policies and proposes DSAC, an off-policy continuous-control algorithm integrating return distributions with maximum-entropy RL.

  • Motivation: Function approximation and other estimation errors can create upward Q-value bias, which may be amplified by temporal-difference learning into suboptimal policy updates.The max operator favors overestimated values, and subsequent-state estimates propagate the bias.
  • Related Work: Existing Double Q-learning methods reduce overestimation by separating action selection from evaluation but are limited to discrete action spaces.Double DQN uses DQN’s target network as the second Q-network.
  • Contribution: DSAC embeds a continuous state-action return distribution into maximum-entropy RL through the distributional soft policy iteration framework.The framework extends soft policy evaluation and improvement to distributional learning.
  • Contribution: Return distribution learning can adaptively adjust Q-value update stepsizes, unlike methods that directly learn expected state-action returns.This mechanism is proposed to mitigate Q-value overestimations without relying on multiple value networks.
  • Method Setting: The proposed method combines off-policy learning, maximum-entropy exploration, and an actor-critic architecture for continuous control.The maximum-entropy objective balances expected future return with policy entropy through temperature parameter α.
  • Related Work: Distributional RL models return distributions whose expectations are value functions, extending beyond methods that select actions from Q-networks.Prior distributional methods commonly used discrete return distributions and often targeted discrete or low-dimensional action spaces.

C. Distributional Soft Policy Iteration

DSPI extends maximum-entropy policy iteration by learning distributions of soft state-action returns rather than only their expectations, with distributional updates based on distribution distance.

  • Distributional Soft Policy Iteration: DSPI defines soft state-action returns as random variables because transitions, rewards, and policies can be stochastic.The framework therefore models the distribution of soft returns conditioned on each state-action pair.
  • Distributional Soft Policy Iteration: The distributional value function maps each state-action pair to a probability distribution over soft state-action returns.This replaces modeling only the expected return Qπ(s, a) with direct return-distribution learning.
  • Distributional Soft Policy Iteration: The distributional Bellman operator provides the soft policy-evaluation update, while policy improvement proceeds within the maximum-entropy framework.DSPI alternates these evaluation and improvement steps.
  • Properties: DSPI is reported to converge to the optimal return distribution and policy while improving the maximum-entropy objective.The convergence result combines prior distributional policy-iteration analysis with the paper’s DSPI result.
  • Implementation: Practical implementations update return distributions using a distance metric, with KL divergence used by many distributional RL algorithms.The paper notes that the target return-distribution variance is unknown for direct implementation of the theoretical objective.
  • Overestimation Analysis: The analysis studies how return-distribution learning affects Q-value bias, with the entropy coefficient set to zero for this section.It derives function-approximation bias and analyzes distributional Q-estimate bias.

A. Overestimation in Q-learning

Q-learning overestimates values because noisy estimates interact with the max operator, and temporal-difference updates can amplify this bias. Modeling returns as distributions makes the update stepsize depend on estimated uncertainty, reducing overestimation as distribution variance grows.

  • Sources of overestimation: The max operator over noisy Q-estimates creates an upward bias because it favors overestimated values.The analysis assumes estimation errors can arise from system noise or function approximation.
  • Sources of overestimation: Temporal-difference learning propagates and exaggerates overestimation errors, potentially producing suboptimal policy updates.Each state’s estimate is updated using the estimate of a subsequent state.
  • Distributional mechanism: A Gaussian return distribution uses its standard deviation to adaptively adjust the update stepsize of the Q-value mean.The return distribution is modeled as Zθ,ψ(·|s, a) = N(Qθ(s, a), σψ(s, a)^2).
  • Assumptions: The distributional analysis assumes Gaussian return and target distributions, with the target mean matching the standard Q-learning target.The Gaussian assumption may be approximate even when the actual distributions are not strictly Gaussian.
  • Distributional mechanism: The Q-value update stepsize decreases squarely as return-distribution standard deviation increases, mitigating overestimation.The distributional overestimation error is inversely proportional to σψ(s, a)^2.
  • Distributional mechanism: Repeated distributional updates increase standard deviation in regions with high target variance and random Q-estimation errors.Because distributional overestimation is inversely proportional to variance, this can mitigate errors from task randomness and approximation.

V. DISTRIBUTIONAL SOFT ACTOR-CRITIC

DSAC derives continuous return-distribution learning rules within the DSPI framework and replaces SAC’s clipped double Q-learning with distributional learning. Its Gaussian value model and variance controls stabilize training while retaining replay-buffer and target-network mechanisms.

  • Algorithm: DSAC replaces SAC’s clipped double Q-learning with continuous return-distribution learning derived from DSPI.The method parameterizes a distributional value function and a stochastic policy.
  • Algorithm: The state-action return distribution and policy are modeled as Gaussian functions whose means and covariances are produced by neural networks.The parameters θ and φ represent the value-distribution and policy networks, respectively.
  • Distributional policy evaluation: The return-distribution objective is trained under KL divergence using replay-buffer transitions and target distribution and policy parameters.Target networks use slow-moving updates to stabilize learning and evaluate targets.
  • Distributional policy evaluation: Because σtarget is unknown in practice, DSAC derives an equivalent objective before updating the return distribution.The derivation addresses the inability to apply the original target objective directly.
  • Variance stabilization: Gaussian distributional gradients can explode as σθ approaches zero or vanish as σθ approaches infinity.DSAC therefore constrains the standard deviation to a reasonable range.
  • Variance stabilization: DSAC sets σmin = 1 and clips the target-related quantity to control the return-distribution variance.The clipping operation restricts values to a specified interval [A, B].

2) Distributional Soft Policy Improvement:

The policy-improvement component learns a stochastic policy by maximizing the soft objective, using reparameterized Gaussian actions and log-likelihoods for entropy-aware optimization.

  • Policy update: The policy is learned by directly maximizing a parameterized version of the maximum-entropy objective.The objective uses the soft Q-value and policy log-likelihood.
  • Policy update: Policy log-likelihoods can be computed for both unbounded actions and actions constrained to finite intervals.The bounded-action calculation is provided through the paper’s appendix.
  • Policy update: DSAC uses the reparameterization trick because it reduces gradient-estimation variance.The paper contrasts it with alternatives such as the log-derivative method.
  • Policy update: Gaussian actions are represented as deterministic functions of an auxiliary fixed-distribution variable and policy parameters.The construction uses the Gaussian mean, standard deviation, and a Hadamard product.

3) Pseudo-code:

DSAC alternates distributional value updates with policy, temperature, and target-network updates, using replay-buffer samples. Its distributed PABAL implementation asynchronously coordinates buffers, actors, and learners, and evaluates against multiple continuous-control baselines.

  • Pseudo-code: Every m iterations, DSAC updates the policy, adjusts temperature α, and updates target networks after the distributional value update.The paper motivates less frequent policy updates through two-timescale updates and depicts value output guiding policy updates.
  • Pseudo-code: DSAC initializes value, policy, temperature, target-network parameters, learning rates, and an iteration index before repeating updates.The algorithm begins with θ, φ, α, θ′, and φ′, then sets k = 0.
  • Pseudo-code: Each iteration stores a transition in buffer B, samples N transitions, and updates the soft return distribution using JZ(θ).The value update is written as θ ← θ − βZ∇θJZ(θ).
  • Architecture: PABAL distributes buffers, actors, and learners across workers, asynchronously sending experience and gradients through shared parameters.Actors send experience to buffers, buffers send sampled experience to learners, and learners update shared value and policy functions.
  • Benchmarks and baselines: The evaluation uses unmodified MuJoCo continuous-control tasks and compares DSAC with mainstream and distribution-learning-related off-policy baselines.Baselines include DDPG, TRPO, PPO, D4PG, TD3, SAC, TD4, Double-Q SAC, and Single-Q SAC; all off-policy algorithms use PABAL.

C. Results

DSAC matches or outperforms the baselines across the continuous-control benchmarks while reducing Q-value overestimation. Its performance gains accompany comparable time efficiency, though clipping-boundary selection remains sensitive to the return range.

  • Performance: DSAC outperforms or matches all baseline algorithms across benchmark tasks in final performance.Evaluations used five runs, with returns averaged over five episodes without exploration noise.
  • Q-value Estimation Accuracy: 10.53%, 5.76%, 926.09%, and 1.89% reductions in overestimation bias occur on Humanoid-v2, Ant-v2, Walker2d-v2, and HalfCheetah-v2, respectively, versus Single-Q SAC.SAC and Double-Q SAC instead exhibit underestimations during learning, while DSAC uses no additional value or policy network.
  • Time Efficiency: DSAC’s average wall-clock time per 1000 iterations is comparable to DDPG and lower than SAC, TD3, and Double-Q SAC.The reported comparison concerns time efficiency on Ant-v2.
  • Ablation Studies: The minimum standard deviation σmin and clipping boundary b stabilize continuous distributional value learning against exploding and vanishing gradients.The clipping boundary generally improves performance, while σmin is set to 1 based on the paper’s analysis.
  • Ablation Studies: Final performance is sensitive to b because values that are too small reduce return-distribution accuracy, whereas values that are too large fail to constrain σθ(s, a) effectively.Selecting b according to the state-action return range limits DSAC’s flexibility in practical applications.

APPENDIX A PROOF OF CONVERGENCE OF DISTRIBUTIONAL SOFT

The appendix establishes convergence for distributional soft policy iteration by combining distributional soft policy evaluation with soft policy improvement. Under finite actions and bounded rewards, the process converges to an optimal maximum-entropy policy.

  • Distributional Soft Policy Iteration: Distributional soft policy iteration alternates distributional soft policy evaluation and soft policy improvement.The framework uses the learned return distribution to obtain Q-values before applying policy improvement.
  • Distributional Soft Policy Evaluation: The distributional soft Bellman operator is a γ-contraction, so repeated distributional soft policy evaluation converges to the unique fixed point Zπ.The operator maps soft return distributions to distributions and converges under the stated contraction measure.
  • Soft Policy Improvement: Soft policy improvement produces a policy whose Q-values are no lower than those of the previous policy for every state-action pair.The monotonicity result is stated as Qπnew(s, a) ≥ Qπold(s, a).
  • Distributional Soft Policy Iteration: Under finite action spaces and bounded rewards, distributional soft policy iteration converges to a policy π∗ whose Q-values dominate those of every policy.The theorem states Qπ∗(s, a) ≥ Qπ(s, a) for all policies and state-action pairs.

APPENDIX B DERIVATIONS

The derivations analyze how random Q-value errors affect the return-distribution update and its standard deviation. They also formulate the KL-divergence loss used to train the soft state-action return distribution.

  • Error Analysis: The appendix expresses the update perturbation δ using the assumption that the random Q-value error is independent of the state-action pair.This independence assumption allows δ to be further expressed from the preceding update equations.
  • Variance Derivation: The post-update standard deviation is linearized around ψ using Taylor expansion.The derivation then evaluates the expected post-update standard deviation.
  • Variance Derivation: When the estimated Q-value has converged to the target, the derivation specializes the post-update variance expression to the ideal case.The condition is ˜Q(s, a) = ˜y.
  • Distribution Learning: The soft state-action return distribution is trained with a KL-divergence loss, with c independent of θ.The constant c does not affect optimization with respect to θ.

C. Probability Density of the Bounded Actions

This section handles bounded continuous actions by transforming samples from an unbounded Gaussian policy into the allowed action interval. It also describes reparameterized return learning and percentile-based objectives.

  • Bounded Actions: Samples from the unbounded Gaussian policy are projected into the finite interval [a_min, a_max] to satisfy action constraints.The transformation uses an element-wise tanh and a Hadamard product.
  • Bounded Actions: The transformed action’s probability density is computed to account for the action constraint.The density follows from the bounded-action transformation.
  • Return Reparameterization: When Qθ(s, a) is not explicit in θ, the random return Z(s, a) is reparameterized for gradient-based learning.The reparameterization supports optimization of the return-distribution model.
  • Risk-Sensitive Objectives: The return distribution can support maximizing an ith percentile rather than only its expected value.A smaller percentile index is suggested for risk-aware policy learning, with gradients approximated using the reparameterization trick.

APPENDIX C EXPERIMENTAL DETAILS

The experimental appendix describes the MuJoCo task setting and the Double-Q SAC implementation used for comparison. The algorithm uses paired soft Q-value and policy networks, replay-buffer updates, temperature adjustment, and target-network updates.

  • Task Setting: Humanoid-v2 uses a 376-dimensional state and 17 joint-torque actions for three-dimensional bipedal locomotion.Its reward penalizes energy-costly actions and mission failure.
  • Double-Q SAC: Double-Q SAC maintains two soft Q-value functions and two policies, with each policy updated using its corresponding Q-value function.Separate target soft Q-value functions are also used to calculate update targets.
  • Algorithm Specification: The appendix identifies Algorithm 2 as the pseudo-code specification for Double-Q SAC.Initialization includes Q-value parameters, policy parameters, temperature, target parameters, learning rates, and the iteration index.
  • Optimization Procedure: Training stores transitions in a replay buffer, samples minibatches, and updates the soft Q-value parameters.The update uses the objective JQ for both Q-value functions.
  • Optimization Procedure: The policy parameters and temperature are updated during training, followed by target-network updates until convergence.The temperature α is adjusted with its own learning rate, while policy updates occur according to the specified update schedule.

C. Single-Q SAC Algorithm

The section describes Single-Q SAC updates for soft Q-values and policies, then introduces a distributional return approach with target networks for stabilized learning.

  • Single-Q SAC updates: Single-Q SAC computes its soft Q-value target from reward, discounted target Q-value, and an entropy term under the target policy.The target action is sampled from the target policy.
  • Single-Q SAC updates: The soft Q-value is trained by minimizing its objective, while the policy is learned by maximizing a parameterized objective.
  • Single-Q SAC algorithm: Single-Q SAC initializes online and target parameters, learning rates, temperature, and a target-update coefficient before replay-buffer training.
  • Distributional variant: The distributional variant parameterizes a state-action return distribution and deterministic policy, using target networks to stabilize learning.Its return distribution is trained by minimizing an objective.
  • Distributional variant: The distributional policy is learned by maximizing expected return, and its pseudocode and algorithm hyperparameters are provided separately.
Loading 2001.02811v3…