Source-linked AI summary
Soft Actor-Critic for Discrete Action Settings
Petros Christodoulou
TL;DR
Poor sample efficiency remains a major reinforcement-learning concern, and SAC does not apply to discrete actions. The paper derives SAC-Discrete and evaluates it on Atari, finding competitive sample efficiency against the model-free state-of-the-art even without hyperparameter tuning.
Problem
SAC provides strong sample efficiency for continuous actions but is not applicable to discrete-action domains, which are common in reinforcement learning.
Method
The paper derives SAC-Discrete by adapting SAC’s objectives and optimization process to discrete action distributions, then tests it on Atari games.
Results
SAC-Discrete achieves a better score than Rainbow in 10 out of 20 Atari games and is roughly competitive with the model-free state-of-the-art in sample efficiency.
Takeaways & Limitations
SAC-Discrete extends SAC to discrete actions and performs competitively on the Atari suite even without hyperparameter tuning.
Takeaways & Limitations
The comparison favors Rainbow through substantial hyperparameter tuning while SAC-Discrete was untuned, so SAC-Discrete’s relative performance could improve with tuning.
Abstract
from arXiv · showhide
Soft Actor-Critic is a state-of-the-art reinforcement learning algorithm for continuous action settings that is not applicable to discrete action settings. Many important settings involve discrete actions, however, and so here we derive an alternative version of the Soft Actor-Critic algorithm that is applicable to discrete action settings. We then show that, even without any hyperparameter tuning, it is competitive with the tuned model-free state-of-the-art on a selection of games from the Atari suite.
1 Introduction
Reinforcement learning has advanced across several domains, but poor sample efficiency remains a major barrier to real-world adoption. This paper adapts SAC from continuous to discrete action settings and evaluates the resulting algorithm on Atari.
- Motivation: Poor sample efficiency remains a dominant concern limiting widespread reinforcement-learning adoption in real-world domains.The paper notes successful applications in board games, video games, and robot tasks despite this concern.
- Motivation: SAC improves sample efficiency in continuous-action settings but is not applicable when actions are discrete.
- Contribution: The paper derives a discrete-action SAC variant and tests whether it is competitive with the model-free state-of-the-art on Atari.
- Organization: The paper first reviews continuous-action SAC, then derives the discrete-action changes, and finally evaluates the algorithm on Atari.
2 Soft Actor-Critic
SAC maximizes a maximum-entropy objective through alternating soft policy evaluation and improvement. Its continuous-action implementation uses neural parameterizations, replay-buffer training, reparameterization, temperature adjustment, and twin soft Q-networks.
- Maximum-entropy objective: SAC maximizes expected rewards while weighting policy entropy through the temperature parameter α.The objective balances reward and entropy, with α controlling their relative importance.
- Soft policy iteration: Soft policy iteration alternates between evaluating the current policy and improving it within the maximum-entropy framework.
- Policy evaluation: Policy evaluation estimates soft values using Bellman backups in tabular settings or neural soft Q-functions trained on replay-buffer experiences.Continuous-state training minimizes the soft Bellman residual using target-network and Monte Carlo estimates.
- Policy improvement: Policy improvement moves toward the exponential of the soft Q-function and projects back into a tractable parameterized policy family using KL divergence.
- Continuous-action implementation: Continuous-action SAC parameterizes Gaussian policies and uses reparameterization to backpropagate through expectations over stochastic actions.The policy network outputs a mean and covariance, while noise is combined with its output to form actions.
- Guarantee: In the tabular setting, alternating policy evaluation and improvement is proved to converge to the optimal policy.
- Additional mechanisms: The temperature can be learned from a Monte Carlo estimate, and twin soft Q-networks use their minimum output to combat value overestimation.
3 Soft Actor-Critic for Discrete Action Settings (SAC-Discrete)
SAC-Discrete retains SAC’s objectives while replacing continuous-action machinery with exact discrete-action calculations. The resulting algorithm uses action-wise Q-values, categorical policies, direct expectations, and no reparameterization trick.
- Derivation: The three SAC objectives remain valid for discrete actions, but optimizing them requires five process changes.
- Q-function: The discrete-action soft Q-function outputs values for every possible action, changing its mapping from Q:S×A→R to Q:S→R^|A|.
- Policy: The policy directly outputs an action probability distribution through a final softmax layer instead of mean and covariance parameters.
- Exact expectations: Because the complete discrete action distribution is available, SAC-Discrete computes expectations directly rather than using Monte Carlo estimates for the Q-function and temperature losses.The paper states that this should reduce estimator variance.
- Policy optimization: Direct access to the action distribution also removes the need for the reparameterization trick in the policy objective.
- Algorithm: Algorithm 1 combines these changes into SAC-Discrete, with local and target Q-networks, a policy, a replay buffer, and gradient updates.
- Evaluation: Figure 1 compares SAC-Discrete with Rainbow across 20 Atari games after 100,000 training steps using five random seeds.
4 Results
SAC-Discrete was evaluated against tuned Rainbow on 20 Atari games under a 100,000-step sample-efficiency regime, without hyperparameter tuning. It was roughly competitive overall, outperforming Rainbow on half the games despite wide variation in performance.
- SAC-Discrete was run for 100,000 steps on 20 Atari games using five random seeds and compared with Rainbow.The games were selected a priori to estimate relative performance across the 49-game Atari suite.
- SAC-Discrete used untuned hyperparameters, whereas the compared Rainbow results came from substantial hyperparameter tuning.The authors note that SAC performance might improve with additional tuning.
- 10 out of 20 games favored SAC-Discrete over Rainbow.Median performance was -1%, with a maximum of +4330% and a minimum of -99%.
- Overall, the authors consider SAC-Discrete roughly competitive with the model-free state-of-the-art on Atari in sample efficiency.
5 Conclusion
The paper derives SAC-Discrete to extend Soft Actor-Critic from continuous to discrete action settings and finds it competitive with the model-free state-of-the-art on Atari without hyperparameter tuning.
- SAC-Discrete extends Soft Actor-Critic, previously inapplicable to discrete actions, to discrete action settings.The paper also provides a Python implementation through the project’s GitHub repository.
- SAC-Discrete performs competitively with the model-free state-of-the-art on the Atari suite without hyperparameter tuning.
A SAC and Rainbow Atari Results
Table 1 reports SAC-Discrete and Rainbow results across 20 Atari games, alongside SAC variability across five seeds and purely random-agent scores.
- Table 1 compares SAC and Rainbow results for 20 Atari games.
- SAC results are means over five random seeds, with standard deviations shown in brackets.
- The table includes purely random-agent scores as a benchmark, while Rainbow results come from Kaiser et al. (2019).
B SAC-Discrete Hyperparameters
Table 2 lists the hyperparameters used to obtain the SAC-Discrete results.
- Table 2 presents the hyperparameters used for the SAC-Discrete results.