PaperlayerView exact revision on arXiv

Source-linked AI summary

An analysis of Mirror-Descent Soft Actor-Critic

Denis Zorba, Michal Valko

arXiv:2609.35466v1cs.LGmath.OC

TL;DR

Practical SAC uses only a few actor steps toward an evolving target, while its squashed-Gaussian actor and non-convex objective make theoretical analysis challenging. The paper proves population-level convergence guarantees for a formalised SAC algorithm with Gibbs and mirror-descent targets. Its analysis links actor tracking to target drift and shows that the mirror-descent step size can control that drift.

  • Problem

    SAC’s practical squashed-Gaussian actor and few-step reverse-KL updates make theoretical analysis challenging because regularity hypotheses used in policy-gradient analyses can fail.

    Source: §1 · p. 2

  • Method

    The paper analyzes a population-level SAC formalisation with a fixed-covariance squashed-Gaussian actor, linear mean, critic oracle, and a single projected gradient step toward an evolving target.

    Source: §1 · p. 2, §7 · p. 5

  • Results

    Theorem 1 establishes a value-function optimality-gap bound, while smaller mirror-descent λ makes consecutive targets closer and lets the actor track them more accurately.

    Source: §8 · pp. 7–8

  • Takeaways & Limitations

    Mirror-descent step size λ directly controls target drift and actor tracking, and the paper reports theoretical advantages over the classical Gibbs target.

    Source: §8 · p. 8, §9 · p. 9

    The analysis is population-level, leaving sampling effects open, and it does not model explicit critic dynamics.

    Source: §9 · pp. 9–10

Abstract

from arXiv · show

Soft Actor-Critic (SAC) is widely used for entropy-regularised reinforcement learning with continuous action spaces, and practical implementations perform only a few actor steps towards an evolving target. In this work, we prove convergence guarantees when the target policy arises from policy mirror descent and compare it with the classical Gibbs target. We derive sufficient conditions for the strong convexity and smoothness of the actor objective, characterised by the curvature of the $Q$-function estimate through the Legendre differential operator, and establish an $\mathcal{O}\!\left(N^{-\frac{1}{5}}\right)$ best-iterate finite-time convergence rate up to actor and critic approximation errors. Moreover, the mirror-descent step size $\lambda$ directly controls the target drift and hence actor tracking error, whereas the analogous Gibbs bound contains a non-vanishing tracking term.

1. Introduction

SAC provides practical entropy-regularised policy improvement for continuous-action reinforcement learning, but its commonly used actor and update procedure present obstacles to theoretical analysis. The paper studies guarantees for a population-level SAC formalisation with evolving mirror-descent or Gibbs targets.

1. Introduction

  • In continuous-action model-free RL, a critic estimates an advantage or Q-function from transitions and is used to update the actor policy.Actor updates may use policy gradients or proximal methods that control changes between successive policies.Source: §1 · p. 1
  • SAC uses reverse-KL descent toward an exponential-Q Gibbs target, whose actor-independent normalizer disappears from the gradient.This provides a practical policy-improvement procedure without requiring the intractable normalisation constant.Source: §1 · pp. 1–2
  • SAC’s practical squashed-Gaussian actor and few-step reverse-KL updates can violate regularity assumptions used in policy-gradient analyses.The tanh transformation makes policy log-densities unbounded near action-space boundaries, and the actor objective is generally non-convex.Source: §1 · p. 2
  • The paper gives population-level guarantees for a fixed-covariance squashed-Gaussian actor with linear mean and a critic oracle, considering mirror-descent and classical Gibbs targets.It studies a single projected gradient step toward an evolving target policy.Source: §1 · p. 2

2. Related works

Related work covers entropy-regularised policy optimisation and actor-critic theory. The paper situates its analysis among policy-space mirror-descent results and studies of practical parameterised updates and population-level convergence.

2. Related works

  • Entropy-regularised policy mirror descent has shown exponential convergence to the optimal policy with a known critic, with extensions to sufficiently accurate critics in tabular and general settings.The cited results include, (Labbi et al., 2026), and.Source: §2 · p. 2
  • Some work approximates policy-space mirror-descent updates by solving the minimisation sub-optimally under different assumptions.The cited approaches include (Sherman et al., 2025; Alfano et al., 2023; Yuan et al., 2023).Source: §2 · p. 2
  • Actor-critic theory spans stochastic-approximation analyses, sample-complexity studies in tabular and function-approximation settings, and population-level convergence and stability analyses.The cited literature includes, and population-level work.Source: §2 · pp. 2–3

3. Contributions •

The paper formalises SAC with a squashed-Gaussian actor and studies conditions for convexity, smoothness, convergence, and actor tracking under Gibbs and mirror-descent targets.

3. Contributions •

  • The reverse-KL actor objective is generally non-convex, but sufficient conditions guarantee strong convexity on any Euclidean ball of radius R > 0.The conditions depend on critic curvature quantified by the Legendre differential operator and on entropy regularisation τ > 0.Source: §3 · p. 3
  • The analysis formalises the SAC policy as the tanh push-forward of a Gaussian with parameterised mean and considers both Gibbs and mirror-descent targets.The mirror-descent update uses step size or penalty λ.Source: §3 · p. 3
  • Under an εcrit-optimal critic-oracle assumption, the paper proves global finite-time best-iterate convergence bounds up to approximation errors and bounds actor tracking error for both targets.For mirror descent, λ controls tracking error and trades off policy improvement against the actor’s ability to follow the target; the Gibbs bound has a non-vanishing contribution determined by τ.Source: §3 · p. 3

4. Entropy Regularised MDPs

The section sets up an infinite-horizon discounted MDP with continuous actions in a bounded cube and introduces regularised value, entropy, and occupation-measure notation.

4. Entropy Regularised MDPs

  • The MDP has an arbitrary Polish state space, action space A := (−1, 1)^M, bounded cost, transition kernel P, and discount factor γ ∈ (0, 1).The action dimension M is at least one.Source: §4 · p. 3
  • The setup uses regularisation parameter τ > 0 and defines normalised Lebesgue measure on the action space.The regularised value is associated with trajectories generated by the policy and transition kernel; the entropy is finite only for measures absolutely continuous with respect to Lebesgue measure.Source: §4 · p. 3
  • The section introduces notation for the policy occupation measure and refers to the appendix for its definition and the entropy-regularised Bellman principle.The occupation measure is indexed by policy π and initial state distribution ρ.Source: §4 · p. 3

5. Soft Actor Critic and Mirror Descent

SAC projects an exponentiated critic onto a policy class using KL divergence, while policy mirror descent penalizes large changes from the previous policy. The mirror-descent target interpolates between the previous policy and a full soft policy-improvement update.

5. Soft Actor Critic and Mirror Descent

  • SAC learns a critic and projects its exponential onto a policy class using KL divergence to obtain a new actor.The critic is learned through temporal-difference or Q-learning methods.Source: §5 · p. 4
  • A fully solved update guarantees policy improvement in the tabular setting, but the guarantee requires exact critics and a finite action space.The optimal policy need not belong to the chosen policy class.Source: §5 · p. 4
  • Policy mirror descent penalizes large changes from the previous policy through a relative-entropy term weighted by λ.The update addresses the possibility that a new policy differs drastically from the old one.Source: §5 · p. 4
  • The mirror-descent policy geometrically interpolates between the old policy and full soft improvement, coinciding with soft improvement when λ = 1/τ.Replacing the classical Gibbs target with the mirror-descent target leaves the gradient steps unaffected by the intractable normalisation constant, and this approach has shown promising empirical results.Source: §5 · p. 4

6. Policy Class

The policy class consists of tanh-transformed Gaussian policies with a feature-based linear mean and fixed covariance. The transformation yields bounded entropy even though policy log densities are unbounded near the action-space boundary.

6. Policy Class

  • A push-forward measure is the probability measure induced by applying a measurable map to another probability measure.The paper uses this construction to define its squashed Gaussian policies.Source: §6 · pp. 4–5
  • The policy class is the push-forward under tanh of Gaussian kernels with fixed covariance and a feature-based linear mean.The mean is x(s)⊤w, with w in the Euclidean ball B(R).Source: §6 · pp. 4–5
  • Near the action-space boundary, log densities are unbounded, but the policy entropy remains bounded for parameters in B(R).This bounded entropy can remedy the obstacle to directly applying tools that require bounded log densities.Source: §6 · pp. 4–5
  • The analysis uses tanh for tractability and practical prevalence, while extending to other sufficiently regular bijections from R^M to (−1, 1)^M.Source: §6 · p. 5

7. Algorithm

The algorithm performs one projected gradient step per critic estimate toward an evolving policy mirror-descent target, while the actor tracks that target. Its analysis isolates actor updates using a critic oracle and imposes replay-distribution and feature non-degeneracy conditions.

7. Algorithm

  • The replay-buffer state law is an exponentially weighted mixture of preceding state occupancies, initialized at the initial distribution ρ.Source: §7 · p. 5
  • Each iteration takes one gradient step on the actor objective toward the mirror-descent target, then projects parameters onto B(R).Targets are updated recursively from the previous target, while the actor tracks them.Source: §7 · p. 5
  • The critic-oracle assumption bounds critic approximation error and lets the analysis isolate Gaussian-policy training without jointly modeling critic updates.It allows twice-differentiable Q-function parametrisations, including smooth neural networks.Source: §7 · p. 6
  • The concentrability condition controls distribution shift between the optimal discounted occupancy measure and the initial state distribution.It requires square-integrability of the optimal occupancy density under the initial distribution, weaker than a uniform bound or conditions on every training policy.Source: §7 · p. 6
  • The actor feature covariance under the initial state distribution must be positive definite, ensuring every actor-parameter direction is represented.Source: §7 · p. 6

8. Main results

The results connect critic-curvature conditions to actor-objective convexity and convergence bounds. Mirror-descent targets have drift that can be controlled through λ, unlike the classical Gibbs target’s non-vanishing tracking term.

8. Main results

  • The guarantees account for bias when squashed Gaussians cannot represent target policies; faster-rate routes involving critic dynamics or interior minimisers are not pursued.The authors leave those routes aside to retain generality for every R > 0.Source: §8 · pp. 6, 9
  • Theorem 1 gives a value-function optimality-gap bound for Algorithm 1, with convergence depending on actor tracking and progress controlled by λ.The actor must track an evolving target while performing one gradient step per iteration.Source: §8 · p. 7
  • Theorem 2 gives sufficient curvature conditions for strong convexity of the mirror-descent actor objective, with the Legendre differential operator expressing relevant curvature.For M = 1, the operator reduces to the classical Legendre differential operator; the entropy curvature must dominate adverse critic curvature.Source: §8 · pp. 7–8
  • Under the stated step-size condition, the tracking bound separates actor optimisation error from target drift, which λ directly controls for mirror-descent targets.Smaller λ slows target movement and improves tracking, but also slows the underlying mirror-descent progress.Source: §8 · p. 8
  • The classical Gibbs target bound contains a non-vanishing tracking term, whereas mirror descent provides a step-size mechanism to slow target evolution.Thus, the Gibbs bound does not guarantee arbitrarily close tracking even when the replay distribution changes slowly.Source: §8 · p. 8
  • With tuned λ and h, Theorem 3 establishes finite-time convergence up to approximation errors for any projection radius R > 0.The authors note that faster-rate approaches were not pursued in order to retain generality for every R > 0.Source: §8 · pp. 8–9

9. Conclusion and limitations

The paper establishes convergence guarantees for a mathematical formalisation of Soft Actor-Critic and identifies theoretical advantages of mirror-descent targets over the classical Gibbs target. Its analysis is limited to the population setting, and the effects of sampling and explicit critic dynamics remain open.

9. Conclusion and limitations

  • The work proves convergence guarantees for a mathematical formalisation of Soft Actor-Critic and demonstrates theoretical advantages of mirror-descent targets over the classical Gibbs measure.The conclusion points to simplified settings in Figures 1 and 2.Source: §9 · p. 9
  • The analysis is performed only in the population setting, leaving the effect of sampling open.Source: §9 · p. 9
  • Explicit critic dynamics are not incorporated and remain a challenging direction for future work.Source: §9 · p. 10

Appendix D. Proof of Theorem 2

The proof establishes strong convexity and smoothness of the actor objective under conditions on critic curvature, with the same constants applying to the Gibbs target.

Appendix D. Proof of Theorem 2

  • The actor objective is strongly convex and smooth on the parameter ball under the stated critic-curvature conditions.The proof bounds the objective Hessian from below and above.Source: Appendix D · pp. 23–24
  • The same strong-convexity and smoothness constants apply when the target is the classical Gibbs policy.Source: Appendix D · p. 24

Appendix F. Proof of Lemma 2

These appendices establish supporting bounds used in the convergence analysis, including a finite-time convergence result and controls on the actor update.

Appendix F. Proof of Lemma 2

  • The proof of Lemma 2 uses projected-gradient optimality and smoothness to control the actor iterate relative to the minimizer.Source: Appendix F · p. 26

Appendix G. Proof of Theorem 3

  • Under the stated step-size conditions, Theorem 3 combines Theorem 1 and Lemma 2 to obtain a finite-time convergence bound.Source: Appendix G · p. 28

I.1. Proof of Lemma 5

  • Lemma 5 supplies an entropy bound for the squashed Gaussian policy used in the subsequent estimates.Source: §I.1 · p. 30

I.2. Proof of Theorem 5

  • Under Assumption 1 and 0 < τλ < 1, Theorem 5 asserts a finite nonnegative constant CKL such that its stated conclusion holds for every n ∈ N and s ∈ S.Source: §I.2 · p. 31

I.3. Proof of Lemma 7

  • Lemma 7 derives a bound under its stated density condition.Source: §I.3 · p. 32

I.4. Proof of Lemma 8

  • Lemma 8 establishes uniform boundedness of the actor gradient over iterations and the parameter ball.Source: §I.4 · p. 34

Appendix J. Implementation details

The implementation details describe a simplified MDP setup designed to validate the theory, with exact critics used to isolate critic-curvature and actor-tracking effects.

Appendix J. Implementation details

  • The validation MDP uses quadratic costs in the action and transitions affine in the action, so its critics can be solved exactly.Source: Appendix J · p. 35
  • Figure 1 compares training when the curvature condition is satisfied versus violated, using perturbed critics with the same approximation error.The critics are constructed by perturbing an exactly solved critic.Source: Appendix J · p. 36
  • Figure 2 varies state and action dimensions with exact critics to isolate how proximal step size λ affects actor tracking error.The experiments use actor step size h = 0.01 and replay weight χ = 1.Source: Appendix J · p. 36
Loading 2609.35466v1…