PaperlayerView exact revision on arXiv

Source-linked AI summary

Average-Reward Soft Actor-Critic

Jacob Adamczyk, Volodymyr Makarenko, Stas Tiomkin, Rahul V. Kulkarni

arXiv:2501.09080v2cs.LGcs.AI

TL;DR

Deep RL lacked function-approximation algorithms combining average reward and entropy regularization. This paper extends SAC to that setting and reports favorable performance against average-reward baselines, with advantages also reported over discounted counterparts.

  • Problem

    Deep RL had few average-reward algorithms, and no function-approximation algorithm for the entropy-regularized average-reward setting was available.

    Source: §1 · pp. 2–3

  • Method

    The paper extends SAC to entropy-regularized average-reward RL, interleaving policy evaluation and improvement and adding a centering baseline and adaptive reset cost.

    Source: §4.2 · pp. 8–9, §6 · p. 10

  • Results

    ASAC performs well against ARO-DDPG, ATRPO, and APO; those baselines fail to solve Walker, Ant, and Humanoid meaningfully.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The authors report favorable stability, convergence speed, and asymptotic performance relative to discounted counterparts, with an implementation that uses existing codebases.

    Source: §6 · p. 10

    The proposed algorithm's sample complexity and convergence properties remain topics for future study.

    Source: §7 · p. 11

Abstract

from arXiv · show

The average-reward formulation of reinforcement learning (RL) has drawn increased interest in recent years for its ability to solve temporally-extended problems without relying on discounting. Meanwhile, in the discounted setting, algorithms with entropy regularization have been developed, leading to improvements over deterministic methods. Despite the distinct benefits of these approaches, deep RL algorithms for the entropy-regularized average-reward objective have not been developed. While policy-gradient based approaches have recently been presented for the average-reward literature, the corresponding actor-critic framework remains less explored. In this paper, we introduce an average-reward soft actor-critic algorithm to address these gaps in the field. We validate our method by comparing with existing average-reward algorithms on standard RL benchmarks, achieving superior performance for the average-reward criterion.

1 Introduction

Discounting can require problem-specific tuning and constrain long-term behavior, while entropy regularization supports naturally stochastic policies. The paper addresses the limited availability of deep algorithms combining these objectives by extending SAC to average-reward RL.

1 Introduction

  • Discounted RL requires tuning γ, and its horizon can limit optimization of long-term behaviors.The motivating Swimmer-v5 example shows SAC can solve the task after careful tuning, but is sensitive to γ.Source: §1 · p. 2, Figure 1 · p. 3
  • Average-reward RL is presented as an alternative for continuing tasks, but deep methods have mainly used policy-gradient approaches and remain limited.The average-reward Bellman operators lack the contraction property.Source: §1 · pp. 2–3
  • Entropy regularization yields naturally stochastic policies, which the paper motivates as useful under model, perception, or control errors.The objective rewards policies with lower relative entropy with respect to a prior.Source: §1 · p. 3
  • The paper proposes an average-reward extension of SAC to address the lack of function-approximator algorithms combining average reward and entropy regularization.The implementation is described as requiring minimal changes to common codebases.Source: §1 · p. 3

2 Preliminaries

The preliminaries define the MDP ingredients and average-reward objective, then introduce entropy regularization relative to a prior policy. They also state the assumptions used for the average-reward setting.

2 Preliminaries

  • An MDP is specified by state and action spaces, rewards, transition dynamics, and an initial-state distribution.The paper assumes fully observable MDPs, where state and observation are synonymous.Source: §2 · pp. 3–4
  • MaxEnt is the uniform-prior special case; the paper also considers general priors, whose choice can encode inductive biases.With a uniform prior, low relative entropy is equivalent to high Shannon entropy.Source: §2 · pp. 3–4
  • The stated average-reward assumptions are that every stationary policy induces a communicating Markov chain and that rewards are bounded.Source: §2 · p. 4
  • The average-reward objective seeks a policy maximizing expected reward-rate over trajectories generated by the dynamics, policy, and initial-state distribution.The state-action-dependent contribution is called the differential bias function.Source: §2 · p. 4
  • An ERAR MDP adds a prior policy and inverse temperature β, and optimizes reward-rate with a relative-entropy regularizer.The paper assumes the prior has support across the action space so the KL divergence remains finite.Source: §2 · p. 4

3 Prior Work

Prior deep average-reward work extended policy-gradient algorithms, while entropy-regularized methods were established mainly in the discounted setting. The paper builds on theory combining the objectives and targets their deep-RL intersection.

3 Prior Work

  • Prior deep average-reward studies reported superior performance on long-term Mujoco tasks compared with discounted counterparts.Source: §3 · p. 5
  • Deep average-reward work extended TRPO, PPO, and DDPG, with reported experimental successes against their discounted counterparts or earlier average-reward methods.The DDPG extension, ARO-DDPG, also included finite-time convergence analysis.Source: §3 · p. 5
  • Entropy-regularized methods were developed for discounted RL, where the entropy bonus supports naturally stochastic policies and SAC became a state-of-the-art solution method.Source: §3 · p. 5
  • Earlier theory and experiments laid groundwork for combining entropy regularization with average reward, while deep-RL work on that combination remained limited.The paper leverages those results and introduces new theoretical results.Source: §3 · pp. 5–6

4 Proposed Algorithm

ASAC extends soft actor-critic to the entropy-regularized average-reward setting, where policy improvement is evaluated through reward-rate. The section develops its policy-improvement theory and implementation.

4.1 Theory

  • The average-reward policy value is θπ, so soft policy improvement constructs π′ from differential values to guarantee nondecreasing entropy-regularized reward-rate.The theorem gives θπ′ ≥ θπ, with equality only at convergence.Source: §4 · p. 6, §4.1 · p. 7
  • For a fixed policy with calculated θπ, policy evaluation converges to its entropy-regularized differential value using a backup of reward minus θπ plus next-state value.The backup is the average-reward Bellman equation for Qπ.Source: §4.1 · pp. 6–7
  • ASAC learns fastest and reaches the best asymptotic performance against ARO-DDPG, ATRPO, and APO on continuous-control benchmarks.Figure 2 averages each curve over 20 random seeds and shows standard errors as shaded regions.Source: Figure 2 · p. 8

4.2 Implementation

  • Training interleaves stochastic-approximation policy evaluation and improvement, using a KL actor loss, critic MSE against the Bellman target, and a learned reward-rate estimate.The actor update follows the policy-improvement theorem; the critic evaluates actions sampled from the current actor.Source: §4.2 · pp. 8–9
  • Two critics use their minimum Q-value to reduce estimation bias and improved experimental performance, though the average-reward theory for this choice remains understudied.The authors identify the effect of estimation bias as future work.Source: §4.2 · p. 9
  • ASAC centers differential value by subtracting Q(0,0) and uses a rolling batch mean of nonterminal rewards as its adaptive reset cost.The centering choice adds no hyperparameter; the reset-cost method addresses tasks with terminating states and varying reward scales.Source: §4.2 · p. 9

5 Experiments

The experiments evaluate ASAC on six version-5 Gymnasium MuJoCo locomotion tasks and compare it with existing average-reward algorithms. ASAC performs well against both off-policy and on-policy baselines, which fail to solve some environments meaningfully.

5 Experiments

  • The evaluation compares average returns over 10 episodes on HalfCheetah, Ant, Swimmer, Hopper, Walker2d, and Humanoid, all version 5.Baseline hyperparameters came from their papers; ASAC used SAC defaults to reduce tuning costs.Source: §5 · p. 9
  • ASAC performs well against off-policy ARO-DDPG and on-policy ATRPO and APO; those baselines fail to solve Walker, Ant, and Humanoid meaningfully.The authors present these results in Figure 2 and describe ASAC as an effective algorithm for the average-reward setting.Source: §5 · p. 10

6 Discussion

The discussion motivates average-reward RL as an alternative to discounting, whose discount factor requires a problem-specific choice. The paper presents ASAC as an entropy-regularized average-reward approach and reports favorable comparisons with discounted counterparts.

6 Discussion

  • A discount factor γ is a required hyperparameter, trading off long-term rewards against computational efficiency, which typically scales polynomially with H = (1 − γ)^−1.The authors note that choosing γ often lacks a principled problem-specific basis.Source: §6 · p. 10
  • For discounted solutions to approximate the average-reward optimal policy, the horizon must exceed the induced Markov chain’s mixing time, which can be difficult to estimate.The authors say principled discounting can therefore be computationally expensive.Source: §6 · p. 10
  • The paper reports low Swimmer evaluation rewards for SAC at γ = 0.99 relative to the optimal case, while average-reward algorithms perform well.The authors note that a sufficiently large γ is often chosen without knowing the relevant problem-dependent timescale.Source: §6 · p. 10
  • The authors present ASAC as a function-approximation solution for entropy-regularized average-reward MDPs in continuous state and action spaces, with favorable stability, convergence speed, and asymptotic performance comparisons to discounted counterparts.They describe its implementation as straightforward and extendable through existing codebases.Source: §6 · p. 10

7 Future Work

The authors identify several potential extensions to ASAC and call for future study of its sample complexity and convergence properties.

7 Future Work

  • Future extensions include automated temperature adjustment, TD(n), REDQ, DrQ, methods to combat estimation bias, dueling architectures, and prioritized replay sampling for θ.The authors describe automated temperature adjustment as a straightforward extension and say replay sampling could benefit θ calculation.Source: §7 · pp. 10–11
  • The sample complexity and convergence properties of the proposed algorithm remain topics for future study.The authors characterize entropy-regularized average reward as a direction for further research and real-world application.Source: §7 · p. 11

8 Proofs

The proofs use the stationary-distribution property to relate expectations under successive states and establish an entropy-regularized average-reward policy improvement result. The section also includes a comparison of ASAC with discounted SAC, which optimizes a different objective.

8 Proofs

  • The ERAR backup lemma states that the differential value Qπ satisfies a backup equation for a fixed evaluation policy and prior policy.The supplied passage introduces the lemma but does not provide the equation’s full terms.Source: §8 · p. 16
  • Stationarity of dπ′ makes the expected next-state value equal the expected current-state value under dπ′.The proof identifies dπ′ as an eigenvector of the transition operator induced by the policy and dynamics.Source: §8 · p. 17
  • The policy-improvement theorem states that an admissible updated policy achieves at least the current policy’s entropy-regularized reward-rate, with equality only at convergence.The stated condition is absolute continuity of π with respect to π0.Source: §8 · p. 17
  • Figure 3 reports that ASAC outperforms discounted SAC on the tested environments, while noting that the methods optimize different objectives and use updated reward versions.SAC optimizes discounted return; ASAC optimizes average reward.Source: Figure 3 · p. 18

9 Implementation Details

ASAC uses SAC-style defaults, with pseudocode specifying replay-buffer updates, critic handling, penalty adaptation, and policy and target updates. The authors report no tuning beyond the stated defaults for ASAC.

9 Implementation Details

  • SAC finetuning took approximately 3,000 GPU hours across environments, with runs requiring roughly 1–10 GB of RAM.Runs used RTX-series and A100 GPUs; critic gradients were clipped to a maximum norm of 10.Source: §9 · p. 18, §9.1 · p. 18

9.1 Hyperparameters

  • ASAC uses batch size 256, replay buffer 1,000,000, two 256-unit hidden layers, and Adam learning rates 10^-4, 5 × 10^-4, and 5 × 10^-3 for actor, critic, and reward-rate.Polyak coefficient is 0.005; β is 5 except 20 for Swimmer and Humanoid, and the critic gradient norm is clipped at 10.Source: §9.1 · p. 18
  • The authors did not tune ASAC beyond its defaults, while noting that ARO-DDPG’s replay-buffer size was sensitive and further environment-dependent tuning remains future work.They suggest broader hyperparameter sweeps and sensitivity analysis to assess robustness and maximize performance across environments.Source: §9.1 · p. 18
  • The pseudocode initializes policy, twin Q-functions, target parameters, optimizers, and replay buffer, then samples actions and stores transitions before gradient updates.The replay buffer holds state, action, reward, and next-state tuples.Source: §9.1 · p. 19
  • Target Q-functions are shifted to pass through the origin, and the critic uses the pointwise minimum as its pessimistic estimate.The pseudocode also specifies policy and target-network updates.Source: §9.1 · p. 19
  • When an episode terminates, the algorithm applies an adaptive penalty, updating its value toward a baseline penalty scaled by p0.The stated update uses τ to interpolate the penalty estimate.Source: §9.1 · p. 19
Loading 2501.09080v2…