PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Actor-Critic Algorithms and Applications

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, Sergey Levine

arXiv:1812.05905v2cs.LGcs.AIcs.ROstat.ML

TL;DR

Deep RL remains limited by high sample complexity and sensitivity to hyperparameters. This paper develops Soft Actor-Critic with maximum-entropy learning and automatic temperature adjustment, matching or exceeding state-of-the-art methods while supporting real-world robotic tasks.

  • Problem

    Model-free deep RL remains hindered by high sample complexity and brittleness to hyperparameters, limiting evidence for reliable real-world deployment.

  • Method

    SAC combines off-policy actor-critic learning with maximum-entropy objectives, soft policy iteration, and constrained automatic temperature tuning.

  • Results

    SAC matches or exceeds state-of-the-art model-free deep RL methods without environment-specific hyperparameter tuning and performs robustly on real-world locomotion and dexterous manipulation.

  • Takeaways & Limitations

    SAC is a promising candidate for sample-efficient and robust learning in real-world robotic tasks.

  • Takeaways & Limitations

    The formulation targets continuous-action MDPs, while the entropy discussion assumes fully observed MDPs.

Abstract

from arXiv · show

Model-free deep reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. However, these methods typically suffer from two major challenges: high sample complexity and brittleness to hyperparameters. Both of these challenges limit the applicability of such methods to real-world domains. In this paper, we describe Soft Actor-Critic (SAC), our recently introduced off-policy actor-critic algorithm based on the maximum entropy RL framework. In this framework, the actor aims to simultaneously maximize expected return and entropy. That is, to succeed at the task while acting as randomly as possible. We extend SAC to incorporate a number of modifications that accelerate training and improve stability with respect to the hyperparameters, including a constrained formulation that automatically tunes the temperature hyperparameter. We systematically evaluate SAC on a range of benchmark tasks, as well as real-world challenging tasks such as locomotion for a quadrupedal robot and robotic manipulation with a dexterous hand. With these improvements, SAC achieves state-of-the-art performance, outperforming prior on-policy and off-policy methods in sample-efficiency and asymptotic performance. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving similar performance across different random seeds. These results suggest that SAC is a promising candidate for learning in real-world robotics tasks.

1 Introduction

The introduction frames deep RL as promising for challenging decision-making and control tasks but identifies poor sample efficiency and hyperparameter brittleness as barriers to real-world adoption. It motivates SAC as a maximum-entropy algorithm designed to address these limitations while building on prior work.

  • Deep RL combines reinforcement learning with neural networks to automate challenging decision-making and control tasks, but real-world adoption remains limited by major challenges.
  • On-policy methods require newly collected samples for nearly every policy update, making training increasingly expensive as task complexity grows.The number of gradient steps and samples needed for an effective policy increases with task complexity.
  • Off-policy algorithms seek greater sample efficiency by reusing past experience, a strategy that is relatively straightforward with Q-learning-based methods.
  • SAC is introduced as an off-policy actor-critic algorithm based on the maximum-entropy reinforcement learning framework.
  • The introduction notes that the original SAC formulation can suffer from brittleness to hyperparameters.

2 Related Work

The paper situates SAC within maximum-entropy reinforcement learning, actor-critic methods, and off-policy learning. Prior work connects these ideas to improved exploration, robustness, and sample efficiency across reinforcement learning and control.

  • Maximum-Entropy Reinforcement Learning: Maximum-entropy reinforcement learning maximizes both expected return and policy entropy, generalizing the standard objective while improving exploration and robustness.The standard objective is recovered in the zero-temperature limit; maximum-entropy policies are robust to model and estimation errors and can improve exploration.
  • SAC Ingredients: SAC combines separate policy and value networks, off-policy data reuse, and entropy maximization within an actor-critic architecture.These ingredients target efficiency, stability, and exploration.
  • Off-Policy Actor-Critic Methods: Prior methods improve sample efficiency through off-policy samples and higher-order variance reduction, but fully off-policy algorithms attain better efficiency.The cited efforts seek to retain robustness while increasing sample efficiency.
  • Off-Policy Actor-Critic Methods: DDPG uses a Q-function estimator for off-policy learning and a deterministic actor that maximizes the estimated Q-function.It can be viewed as both a deterministic actor-critic algorithm and an approximate Q-learning algorithm.
  • Related Policy Optimization: Maximum-entropy reinforcement learning has been applied to inverse reinforcement learning and optimal control, while MPO optimizes the standard RL objective through expectation maximization.MPO uses a probabilistic formulation of policy optimization.

3 Preliminaries

This section formulates reinforcement learning in continuous state and action spaces and contrasts standard expected-return optimization with maximum-entropy reinforcement learning. It explains how the temperature parameter controls stochasticity and motivates a soft actor-critic policy-iteration formulation.

  • MDP formulation: The reinforcement learning problem is policy search in an MDP with continuous state space S and action space A.The MDP is specified by (S, A, p, r), with p representing the next-state probability density given the current state and action.
  • RL objectives: Standard reinforcement learning seeks a policy π(a_t|s_t) that maximizes the expected sum of rewards.
  • RL objectives: Maximum-entropy reinforcement learning augments the standard objective with an entropy term, encouraging the policy to maximize entropy at each visited state.
  • RL objectives: The temperature parameter α sets the relative importance of entropy versus reward and therefore controls the stochasticity of the optimal policy.The conventional objective is recovered as α →0.
  • Maximum-entropy advantages: Maximum entropy objectives encourage broader exploration and allow policies to represent multiple modes of near-optimal behavior.When multiple actions are equally attractive, the policy assigns equal probability mass to them.
  • Soft actor-critic formulation: The paper develops soft actor-critic through policy iteration by evaluating the Q-function of the current policy rather than directly solving for the optimal Q-function.

4 From Soft Policy Iteration to Soft Actor-Critic

Soft Actor-Critic is derived from maximum-entropy soft policy iteration, whose alternating evaluation and improvement steps converge to the optimal policy within a restricted policy class. For continuous domains, the method is approximated with stochastic-gradient updates to parameterized soft Q-function and policy networks.

  • Soft Policy Iteration: Soft policy iteration alternates soft policy evaluation and improvement to learn optimal maximum-entropy policies in a tabular setting.The derivation provides theoretical analysis and convergence guarantees before extension to continuous settings.
  • Soft Policy Evaluation: Repeated application of the soft Bellman backup operator converges to a policy’s soft Q-function from any initial function when the action space is finite.The sequence Q_k converges to the soft Q-function as k →∞.
  • Soft Policy Improvement: The policy improvement step updates toward the exponential of the new soft Q-function and projects onto a tractable policy set, guaranteeing higher soft value.The restricted set Π may represent a parameterized family such as Gaussians.
  • Soft Policy Iteration: Alternating soft evaluation and improvement provably converges to the optimal maximum-entropy policy among policies in Π, but exact execution is limited to tabular cases.Continuous domains therefore require function approximation.
  • Continuous-Domain Approximation: For continuous domains, SAC replaces exact convergence of evaluation and improvement with stochastic-gradient optimization of parameterized soft Q-function and policy networks.The soft Q-function is trained using the soft Bellman residual, while the policy is learned by minimizing an expected KL-divergence.
  • Continuous-Domain Approximation: A target soft Q-function updated by exponentially moving-average weights stabilizes training, while reparameterization provides an unbiased, lower-variance policy-gradient estimator for tractable stochastic policies.The estimator extends DDPG-style policy gradients to any tractable stochastic policy.

5 Automating Entropy Adjustment for Maximum Entropy RL

SAC automates temperature selection by treating entropy as a minimum expected-entropy constraint while maximizing expected return. The resulting dual formulation yields recursive policy and temperature updates, forming the algorithm’s core in practice with function approximators and stochastic gradient descent.

  • Constrained objective: SAC replaces manual temperature tuning with a constrained objective that maximizes expected return subject to a minimum expected entropy H.The temperature is task-dependent, while the constrained formulation automates its adjustment.
  • Constrained objective: For fully observed MDPs, the entropy constraint is expected to be tight because return-optimal policies are deterministic.No upper entropy bound is needed under this assumption.
  • Dual optimization: Dualizing the entropy constraint produces a maximum-entropy policy whose temperature is the dual variable α_t, optimized recursively through time.The derivation proceeds backward using approximate dynamic programming and soft Q-function updates.
  • Algorithm: The policy and soft Q-function updates together with the dual-temperature solution constitute SAC’s core, but practical optimization uses function approximators and stochastic gradient descent.The algorithm also adjusts the temperature and updates target network weights during gradient steps.

6 Practical Algorithm

The practical SAC algorithm uses twin soft Q-functions and their minimum to reduce positive bias during policy and value updates. It alternates off-policy replay-based updates with automatic temperature learning through a dual objective.

  • Twin Q-functions: Twin soft Q-functions are trained independently, and their minimum is used for stochastic and policy gradients to mitigate positive bias.This design follows the clipped double-Q approach proposed by Fujimoto et al. (2018).
  • Automatic temperature tuning: The temperature α is learned by minimizing the dual objective through an approximate, truncated dual-gradient-descent procedure.The procedure alternates updates to primal variables with gradient steps on the dual variable.
  • Off-policy training: The algorithm alternates environment interaction with stochastic-gradient updates using batches sampled from a replay pool.Off-policy replay is feasible because both value estimators and the policy can be trained entirely on off-policy data.

7 Experiments

Experiments show that SAC is consistently effective on challenging continuous-control benchmarks and real-world robotic tasks. It combines strong sample efficiency and asymptotic performance with robust generalization to unseen terrains and end-to-end manipulation from raw images.

  • Continuous-control benchmarks: SAC performs consistently across continuous-control tasks and outperforms on-policy and off-policy methods on the most challenging benchmarks.The evaluation compares SAC with DDPG, PPO, and SQL across OpenAI Gym and Humanoid tasks.
  • Continuous-control benchmarks: SQL learns all tasks but is slower than SAC and achieves worse asymptotic performance, while SAC exceeds prior state-of-the-art results in sample efficiency and final performance.The experiments also indicate that SAC’s automatic temperature tuning improves robustness, though the supplied passage truncates the specific finding.
  • Real-world locomotion: A Minitaur policy trained only on flat terrain generalizes to varied terrains and obstacles without additional learning.The passage attributes this robustness to entropy maximization during training.
  • Dexterous manipulation: 300k environment interaction steps, equivalent to 20 hours, are required to learn dexterous valve rotation directly from raw RGB images.The task is learned end-to-end in the real world without simulation or pretraining.

8 Conclusion · Appendix

SAC is presented as an off-policy maximum-entropy deep RL algorithm combining sample-efficient learning, entropy maximization, and stability. The paper derives convergent soft policy iteration, validates SAC against established model-free methods without environment-specific hyperparameter tuning, and demonstrates real-world robotic applicability.

  • 8 Conclusion: SAC is an off-policy maximum-entropy deep reinforcement learning algorithm designed for sample-efficient and stable learning.
  • 8 Conclusion: Theoretical results derive soft policy iteration and show convergence to the optimal policy.
  • 8 Conclusion: These results yield a practical soft actor-critic algorithm for training deep neural network policies.
  • 8 Conclusion: Image-based valve rotation requires inferring the valve position from 32x32 pixel observations alongside finger joint positions and velocities.
  • 8 Conclusion: Learning the valve rotation task takes about 3 hours without images and about 20 hours from images.Without images, the valve initially points away from the target; with images, its initial position is sampled uniformly at random.
  • 8 Conclusion: SAC matches or exceeds state-of-the-art model-free deep RL methods, including off-policy TD3 and on-policy PPO.This performance is achieved without environment-specific hyperparameter tuning.
  • 8 Conclusion: Real-world experiments show SAC is robust and sample efficient enough for robotic tasks learned directly in the real world.Demonstrated tasks include locomotion and dexterous manipulation.

A Infinite Horizon Discounted Maximum Entropy Objective

The infinite-horizon objective maximizes discounted expected reward and entropy for future states originating from every state-action tuple, weighted by its probability under the current policy. Discounted policy gradients may instead optimize average reward because they discount rewards but not the state distribution, using discounting to reduce variance.

  • Discounted policy gradients typically do not optimize the true discounted objective because they discount rewards but not the state distribution, instead optimizing average reward while reducing variance.
  • The objective maximizes discounted expected reward and entropy for future states originating from every state-action tuple, weighted by its probability ρπ under the current policy.

B Proofs · B.1 Lemma 1 · B.2 Lemma 2

The proofs establish convergence of soft policy evaluation and monotonic improvement under soft policy updates. Together, Lemmas 1 and 2 support iterative convergence to the improved soft Q-value when the action space is finite.

  • B.1 Lemma 1: Lemma 1 states that repeatedly applying the soft Bellman backup operator converges to the soft Q-value of policy π.The iteration is defined by Qk+1 = TπQk as k →∞.
  • B.1 Lemma 1: The proof rewrites the update using an entropy-augmented reward that adds expected next-state policy entropy to the original reward.The augmented reward is rπ(st, at) ≜ r(st, at) + Est+1∼p[H(π(·|st+1))].
  • B.1 Lemma 1: Standard policy-evaluation convergence results then apply, with |A| < ∞ ensuring that the entropy-augmented reward is bounded.The proof cites Sutton & Barto (1998) for the convergence result.
  • B.2 Lemma 2: Lemma 2 states that optimizing the Equation 4 minimization problem produces πnew whose soft Q-value is at least that of πold for every state-action pair.The guarantee is Qπnew(st, at) ≥ Qπold(st, at) for all (st, at) ∈ S × A with |A| < ∞.
  • B.2 Lemma 2: The policy-improvement proof defines πold, its corresponding soft values, and πnew through the stated optimization construction.The proof compares the soft state-action and state values associated with πold.
  • B.2 Lemma 2: Because πold remains an admissible choice, the optimized objective for πnew cannot exceed the objective obtained by πold.This yields Jπold(πnew(·|st)) ≤ Jπold(πold(·|st)).
  • B.2 Lemma 2: The resulting expectation inequality compares πnew and πold under the terms log π(at|st), Qπold(st, at), and log Zπold(st).The inequality is evaluated under the respective action distributions.
  • B.2 Lemma 2: Since Zπold depends only on the state, the inequality simplifies; repeated soft Bellman expansion and Lemma 1 then establish convergence to Qπnew.The proof uses the soft Bellman equation and the bound in Equation 23.

B.3 Theorem 1 · C Enforcing Action Bounds

Soft policy iteration converges to an optimal policy under finite action spaces, while SAC enforces finite action bounds by squashing Gaussian samples with tanh and correcting likelihoods via change of variables. The transformation yields a simple log-likelihood correction because its Jacobian is diagonal.

  • B.3 Theorem 1: Soft policy evaluation and improvement converge to a policy π∗ satisfying Qπ∗(st, at) ≥ Qπ(st, at) for every π and state-action pair.Theorem 1 assumes a finite action space, |A| < ∞.
  • B.3 Theorem 1: The sequence Qπi increases monotonically during repeated soft policy iteration.This follows from Lemma 2 in the proof of Theorem 1.
  • B.3 Theorem 1: Qπ is bounded above because both rewards and entropy are bounded, so the increasing sequence converges to a policy π∗.The proof then establishes that this limiting policy is optimal.
  • B.3 Theorem 1: At convergence, π∗ maximizes the soft objective over all policies, yielding Qπ∗(st, at) > Qπ(st, at) for competing policies.The proof uses the iterative argument from Lemma 2 to establish strict improvement for π ≠ π∗.
  • C Enforcing Action Bounds: SAC begins with an unbounded Gaussian action distribution and applies elementwise tanh to Gaussian samples to constrain actions to a finite interval.If u ∈ R^D has infinite-support density μ(u|s), the bounded action is a = tanh(u).
  • C Enforcing Action Bounds: The likelihood of each bounded action is computed using the change-of-variables formula under the invertible squashing transformation.This preserves a tractable density for the transformed actions.
  • C Enforcing Action Bounds: The Jacobian da/du = diag(1 − tanh^2(u)) is diagonal, giving the transformed log-likelihood a simple form.The coordinate-wise structure follows because tanh is applied elementwise; ui denotes the ith component of u.
Loading 1812.05905v2…