PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine

arXiv:1801.01290v2cs.LGcs.AIstat.ML

TL;DR

Model-free deep RL is limited by high sample complexity and brittle convergence, motivating an efficient and stable approach for continuous control. Soft actor-critic combines off-policy actor-critic learning with maximum-entropy objectives and outperforms prior methods while remaining stable across random seeds.

  • Problem

    Model-free deep RL remains limited by high sample complexity and brittle hyperparameter-sensitive convergence, restricting its applicability to real-world tasks.

  • Method

    Soft actor-critic combines off-policy actor-critic learning, entropy maximization, and stochastic policy optimization for continuous control.

  • Results

    SAC outperforms state-of-the-art model-free deep RL methods, exceeds DDPG in sample efficiency, and performs consistently across random seeds.

  • Takeaways & Limitations

    The results support stochastic, entropy-maximizing reinforcement learning as a promising approach for improved robustness and stability.

  • Takeaways & Limitations

    Reward scaling remains the only hyperparameter requiring tuning in practice, with poor scaling impairing exploration and learning.

Abstract

from arXiv · show

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.

1. Introduction

The introduction identifies sample complexity and brittle hyperparameter sensitivity as major barriers to real-world deep RL. It presents Soft Actor-Critic (SAC) as an off-policy maximum-entropy actor-critic algorithm designed for efficient, stable learning in continuous control.

  • Challenges: Model-free deep RL methods are hindered by expensive sample complexity and brittleness to hyperparameter choices.Even relatively simple tasks can require millions of data-collection steps, while learning rates and exploration settings require careful tuning.
  • Sample efficiency: On-policy algorithms such as TRPO, PPO, and A3C require new samples for each gradient step, making learning increasingly expensive as task complexity grows.Off-policy methods aim to reuse past experience to improve sample efficiency.
  • Approach: The paper explores an efficient and stable model-free deep RL algorithm for continuous state and action spaces using maximum entropy reinforcement learning.The maximum entropy framework augments the standard maximum-reward objective with an entropy-maximization term.
  • Contribution: SAC is an off-policy maximum-entropy actor-critic algorithm intended to provide sample-efficient learning and stability on complex, high-dimensional tasks.The method extends to the 21-action-dimensional Humanoid benchmark, where DDPG typically struggles to obtain good results.

2. Related Work

The related work situates soft actor-critic at the intersection of actor-critic policy optimization, off-policy learning for sample efficiency, and maximum entropy reinforcement learning for stability and exploration. Prior work spans joint actor-critic optimization, fully off-policy methods such as DDPG, and maximum-entropy connections between Q-learning and policy gradients.

  • Actor-Critic and Off-Policy Methods: Soft actor-critic combines separate policy and value networks, off-policy data reuse, and entropy maximization for efficiency, stability, and exploration.These are presented as the algorithm’s three key ingredients.
  • Actor-Critic and Off-Policy Methods: Actor-critic methods derive from alternating policy evaluation and improvement, but large-scale problems typically optimize actor and critic jointly without convergence.The policy is the actor, while the value function is the critic.
  • Actor-Critic and Off-Policy Methods: Efforts to improve sample efficiency while retaining robustness use off-policy samples and higher-order variance reduction, while fully off-policy algorithms achieve better efficiency.DDPG is identified as a popular off-policy actor-critic method and a deep variant of deterministic policy gradient.
  • Maximum Entropy Reinforcement Learning: Maximum entropy reinforcement learning maximizes both expected return and expected policy entropy, with applications including inverse reinforcement learning, optimal control, and guided policy search.Guided policy search uses the maximum entropy distribution to guide learning toward high-reward regions.
  • Maximum Entropy Reinforcement Learning: Recent maximum-entropy work connects Q-learning and policy gradients, with model-free approaches approximating the maximum entropy distribution using Gaussian or sampling-network methods.Most prior model-free works assume discrete actions; Nachum et al. approximate the distribution with a Gaussian, while Haarnoja et al. use a trained sampling network.

3. Preliminaries

The paper studies policy learning in continuous-action, infinite-horizon MDPs and adopts maximum-entropy reinforcement learning, which augments reward maximization with entropy. It motivates a soft actor-critic formulation that evaluates the current policy and updates it off-policy.

  • Continuous-action MDP: The setting is an infinite-horizon MDP with continuous state and action spaces, unknown state transitions, and bounded rewards.The MDP is defined by (S, A, p, r), with rewards bounded in [rmin, rmax].
  • Maximum-entropy objective: Maximum-entropy reinforcement learning augments the expected reward objective with an entropy term, favoring stochastic policies.The temperature parameter α controls the entropy term’s relative importance and the optimal policy’s stochasticity.
  • Maximum-entropy objective: As α →0, the maximum-entropy objective recovers the conventional maximum expected reward objective.The temperature can be subsumed into the reward by scaling.
  • Maximum-entropy objective: The entropy objective encourages broader exploration, supports multiple modes of near-optimal behavior, and can improve exploration.When multiple actions are equally attractive, the policy assigns them equal probability mass.
  • Soft actor-critic formulation: Soft actor-critic uses policy iteration to evaluate the current policy’s Q-function and update the policy through an off-policy gradient update.This contrasts with prior approaches that directly solve for the optimal Q-function and recover the policy from it.

4. From Soft Policy Iteration to Soft Actor-Critic

Soft Actor-Critic is derived from maximum-entropy soft policy iteration, alternating policy evaluation and improvement with convergence guarantees in the tabular setting. For continuous domains, the method approximates these steps with neural networks, stochastic gradients, replay-buffer data, and variance- and bias-reduction techniques.

  • Soft Policy Iteration: Soft policy evaluation repeatedly applies a modified Bellman backup to converge to a policy’s soft Q-value.The convergence guarantee holds for finite action spaces and arbitrary initial Q-function mappings.
  • Soft Policy Iteration: Soft policy improvement updates toward the exponential of the new Q-function and projects onto a tractable policy class, yielding nondecreasing soft value.The projected policy satisfies Q^πnew(st, at) ≥ Q^πold(st, at) for all state-action pairs under the finite-action assumption.
  • Soft Policy Iteration: Alternating soft evaluation and improvement converges to the optimal maximum-entropy policy within the restricted policy set Π.Theorem 1 states convergence from any π ∈ Π to π* whose Q-value is at least that of every policy in Π.
  • Continuous-Domain Approximation: For continuous domains, SAC replaces exact iteration with stochastic-gradient optimization of parameterized value, Q-function, and policy networks.The practical algorithm uses a replay buffer, a target value network for stability, and simultaneous optimization rather than running evaluation and improvement to convergence.
  • Continuous-Domain Approximation: The reparameterization trick provides a lower-variance policy-gradient estimator for differentiable neural-network Q-functions and tractable stochastic policies.The resulting unbiased estimator extends DDPG-style policy gradients to any tractable stochastic policy.
  • Continuous-Domain Approximation: Two independently trained Q-functions and their minimum mitigate positive bias in policy improvement, while off-policy replay remains feasible for value and policy training.The method can use replay-buffer data because both value estimators and the policy can be trained entirely off-policy.

5. Experiments

Experiments compare SAC with established off-policy and on-policy methods across challenging continuous-control tasks, finding strong performance and stability, while component analyses identify stochasticity, deterministic evaluation, reward scaling, and target updates as important factors.

  • Benchmark comparisons: SAC is compared with DDPG, PPO, and soft Q-learning on challenging continuous-control tasks from OpenAI Gym and rllab.The evaluation targets sample complexity and stability relative to prior off-policy and on-policy deep RL algorithms.
  • Benchmark comparisons: SAC matches baseline methods on easier tasks and outperforms them on harder tasks in learning speed and final performance.DDPG fails to progress on Ant-v1, Humanoid-v1, and Humanoid (rllab), while SAC learns considerably faster than PPO.
  • Component analysis: Stochastic policies with entropy maximization substantially stabilize training compared with a deterministic SAC variant across random seeds.The deterministic variant exhibits much higher variability between seeds, especially on the Humanoid (rllab) benchmark.
  • Component analysis: Deterministic evaluation using the mean action can produce higher returns than stochastic evaluation, although the mean action is not generally optimal for maximum return.SAC trains with an entropy objective, so the final deterministic evaluation action differs from the training objective.
  • Hyperparameter sensitivity: Reward scaling controls SAC’s exploration–exploitation balance and is the only hyperparameter found to require tuning in practice.Small reward magnitudes make the policy nearly uniform, whereas large magnitudes make it nearly deterministic and can cause poor local minima.
  • Hyperparameter sensitivity: SAC updates a separate target value network using an exponentially moving average with smoothing constant τ to improve stability.τ = 1 corresponds to hard updates, while τ = 0 leaves the target network unchanged.

6. Conclusion · A. Maximum Entropy Objective

The paper presents SAC as an off-policy maximum-entropy deep RL algorithm designed for sample-efficient, stable learning, with theoretical convergence and empirical performance advantages. The maximum-entropy objective combines discounted future rewards and entropy across state-action tuples weighted by their policy probability.

  • 6. Conclusion: SAC combines off-policy learning, maximum-entropy reinforcement learning, sample efficiency, and stability.The conclusion identifies these as the algorithm’s central properties.
  • 6. Conclusion: Soft policy iteration is derived theoretically and shown to converge to the optimal policy.This result provides the basis for formulating SAC.
  • 6. Conclusion: The resulting SAC algorithm empirically outperforms state-of-the-art model-free deep RL methods, including off-policy DDPG.The supplied conclusion passage names DDPG as an included comparison.
  • A. Maximum Entropy Objective: The discounted maximum-entropy objective is complicated because policy-gradient methods typically discount rewards but not the state distribution.The passage contrasts the usual discounted-policy-gradient treatment with the true discounted objective.
  • A. Maximum Entropy Objective: Under this convention, discounted policy gradients optimize average reward, while discounting serves to reduce variance.The passage attributes this interpretation to Thomas (2014).
  • A. Maximum Entropy Objective: The objective maximizes discounted expected reward and entropy for future states originating from every state-action tuple (s_t, a_t).Each tuple is weighted by its probability ρ^π under the current policy.

B. Proofs · B.1. Lemma 1

Lemma 1 establishes that repeated application of the soft Bellman backup operator converges to the soft Q-value of a policy. The proof rewrites the update using an entropy-augmented reward and invokes standard policy-evaluation convergence results under finite action spaces.

  • B.1. Lemma 1: The sequence Q_k converges to the soft Q-value of π as k →∞ under repeated soft Bellman backups.Given Q_0: S × A →R and |A| < ∞, define Q_{k+1} = T_πQ_k.
  • B.1. Lemma 1: The proof defines an entropy-augmented reward by adding the expected next-state policy entropy to the original reward.The reward is r_π(s_t, a_t) ≜ r(s_t, a_t) + E_{s_{t+1}∼p}[H(π(·|s_{t+1}))].
  • B.1. Lemma 1: After rewriting the update rule, the proof applies standard convergence results for policy evaluation.The cited convergence results are from Sutton & Barto (1998).
  • B.1. Lemma 1: The finite-action assumption |A| < ∞ ensures that the entropy-augmented reward is bounded.This boundedness condition supports applying the standard policy-evaluation convergence results.

B.2. Lemma 2

Lemma 2 establishes soft policy improvement: optimizing Equation 4 produces a policy whose soft action-value function is no worse than the old policy’s for every state-action pair when the action space is finite. The proof uses the old policy as a feasible choice, simplifies the resulting inequality, and invokes repeated soft Bellman expansions plus Lemma 1 for convergence.

  • Lemma 2: For finite action spaces, the policy optimizer in Equation 4 satisfies Qπnew(st, at) ≥ Qπold(st, at) for all (st, at) ∈ S × A.This is the stated soft policy improvement guarantee.
  • Proof: Because πold is feasible, choosing πnew as the optimizer cannot increase Jπold relative to choosing πold.The proof then expresses this comparison as an expectation inequality involving log π, Qπold, and log Zπold.
  • Proof: The expectation inequality simplifies because the partition function Zπold depends only on the state.This removes the partition-function term from the action-dependent comparison.
  • Proof: Repeated soft Bellman expansions establish the value bound, and convergence to Qπnew follows from Lemma 1.The proof applies the soft Bellman equation on the right-hand side together with the bound in Equation 18.

B.3. Theorem 1 · C. Enforcing Action Bounds · D. Hyperparameters

Theorem 1 establishes convergence of repeated soft policy evaluation and improvement to an optimal policy under finite action spaces. SAC enforces bounded actions by applying tanh to Gaussian samples, while the paper separately specifies common and environment-specific hyperparameters.

  • B.3. Theorem 1: Repeated soft policy evaluation and improvement converge to a policy π∗ satisfying Qπ∗(st, at) ≥ Qπ(st, at) for every π ∈ Π.The guarantee assumes a finite action space, |A| < ∞.
  • B.3. Theorem 1: The proof establishes convergence because Qπi increases monotonically and is bounded above when rewards and entropy are bounded.At convergence, the resulting policy is shown to be optimal relative to every other policy in Π.
  • C. Enforcing Action Bounds: SAC samples actions from an unbounded Gaussian and applies elementwise tanh to obtain actions bounded to a finite interval.The change-of-variables formula is used to compute likelihoods after squashing.
  • D. Hyperparameters: Table 1 lists common SAC parameters used for the comparative evaluation in Figure 1 and Figure 4, while Table 2 lists tuned reward scales for each environment.The text distinguishes shared evaluation parameters from environment-specific reward-scale tuning.
  • D. Hyperparameters: The paper includes a table of common SAC hyperparameters for the evaluation.This is identified as Table 1, titled “SAC Hyperparameters.”
  • D. Hyperparameters: The paper includes a table of environment-specific SAC parameters.This is identified as Table 2, titled “SAC Environment Specific Parameters.”

E. Additional Baseline Results

Additional baseline experiments compare SAC with Trust-PCL and evaluate hard-target-update and deterministic SAC variants. Trust-PCL fails on most tasks within the allotted environment steps, while the variants alter SAC’s target-update or policy assumptions.

  • Trust-PCL baseline: Trust-PCL fails to solve most tasks within the given number of environment steps, although it can eventually solve easier tasks with longer training.Figure 4 compares SAC with Trust-PCL and reports this limitation of Trust-PCL.
  • SAC variants: The hard-target-update SAC variant copies value-network weights every 1000 iterations instead of using an exponentially smoothed average.This variant is shown in Figure 4 as distinct from standard SAC’s target-network update.
  • SAC variants: The deterministic ablation uses a deterministic policy with fixed Gaussian exploration noise and removes the value function and entropy terms from the actor and critic updates.These changes define the deterministic SAC ablation evaluated in Figure 4.
Loading 1801.01290v2…