PaperlayerView exact revision on arXiv

Source-linked AI summary

Equivalence Between Policy Gradients and Soft Q-Learning

John Schulman, Xi Chen, Pieter Abbeel

arXiv:1704.06440v4cs.LG

TL;DR

The paper addresses why inaccurate Q-values can nevertheless support effective Q-learning by studying its relationship with policy gradients. It proves an expected-gradient equivalence between soft Q-learning and policy gradients, connects damped Q-learning to natural policy gradients, and finds matching practical dynamics on Atari.

  • Problem

    The paper asks why Q-learning can work despite empirically inaccurate estimated Q-values, focusing on whether it implements policy-gradient updates.

  • Method

    The paper analyzes entropy-regularized reinforcement learning, comparing soft Q-learning gradients with policy-gradient and natural-policy-gradient formulations.

  • Results

    Soft Q-learning is equivalent to policy gradients in expected gradients, and a constructed Q-learning method closely matches A3C learning dynamics on Atari without a target network or ε-greedy schedule.

  • Takeaways & Limitations

    The equivalence holds in a practically relevant regime, supporting entropy-regularized Q-learning and policy gradients as closely related update rules.

  • Takeaways & Limitations

    The policy-gradient approximations discussed do not account for how actions affect entropy at future timesteps, so they are not proper estimators of the entropy-augmented return.

Abstract

from arXiv · show

Two of the leading approaches for model-free reinforcement learning are policy gradient methods and $Q$-learning methods. $Q$-learning methods can be effective and sample-efficient when they work, however, it is not well-understood why they work, since empirically, the $Q$-values they estimate are very inaccurate. A partial explanation may be that $Q$-learning methods are secretly implementing policy gradient updates: we show that there is a precise equivalence between $Q$-learning and policy gradient methods in the setting of entropy-regularized reinforcement learning, that "soft" (entropy-regularized) $Q$-learning is exactly equivalent to a policy gradient method. We also point out a connection between $Q$-learning methods and natural policy gradient methods. Experimentally, we explore the entropy-regularized versions of $Q$-learning and policy gradients, and we find them to perform as well as (or slightly better than) the standard variants on the Atari benchmark. We also show that the equivalence holds in practical settings by constructing a $Q$-learning method that closely matches the learning dynamics of A3C without using a target network or $ε$-greedy exploration schedule.

1 Introduction

The paper studies the close relationship between policy-gradient and Q-learning updates under entropy regularization. It extends prior informal or tabular connections by proving an exact expected-gradient equivalence under appropriate conditions.

  • Policy-gradient and Q-learning methods both reinforce actions followed by high returns, increasing action probability or action value respectively.
  • Entropy regularization makes the connection closer because the optimal policy is proportional to exp(Q(s, a)), linking policy optimization to the optimal Q-function.
  • The paper proves that, under appropriate conditions, n-step Q-learning and n-step policy-gradient losses have equal gradients, including value-function fitting.
  • The paper uses a KL-divergence penalty rather than an entropy bonus, a more general formulation that is often called an entropy term in the text.

2 Bandit Setting

The bandit analysis introduces entropy-regularized objectives and shows how value-based learning with a Boltzmann policy connects to policy-gradient learning. The squared-error Q-function gradient decomposes into a policy-gradient component and a value-fitting component.

  • The entropy-regularized bandit objective combines expected reward with a KL-divergence penalty weighted by temperature τ.The objective uses a reference policy and temperature parameter τ.
  • The optimal policy for known expected rewards is the Boltzmann policy, obtained by minimizing the KL term after introducing a normalizing constant.The policy weights actions according to exponentiated expected reward relative to the reference policy.
  • The policy-based approach updates the policy by stochastic gradient ascent, while the value-based approach learns an action-value function with a squared-error loss.
  • The Boltzmann policy is defined from the learned action-value function and serves as the policy used for the subsequent gradient comparison.
  • The squared-error gradient decomposes into the Boltzmann policy gradient plus a squared-error gradient fitting the value function to the entropy-augmented expected reward.

3 Entropy-Regularized Reinforcement Learning

The paper extends entropy-regularized value functions, backup operators, and policy gradients from bandits to MDPs. It defines soft Q-learning and shows how its gradients relate to policy gradients, while identifying limits of approximate entropy-return estimators.

  • Entropy-regularized returns subtract discounted KL-divergence penalties between the current and reference policies at each timestep.The formulation includes discount factor γ and temperature τ; a uniform reference policy recovers a standard entropy bonus up to a constant.
  • The state-value and Q-functions are redefined to incorporate KL penalties, with the Q-function omitting the action-independent first penalty term.
  • Entropy regularization replaces the deterministic greedy policy with a stochastic Boltzmann policy proportional to the reference policy times exp(Q(s, a)/τ).
  • The entropy-regularized Bellman operators compute one-step expected rewards, KL penalties, and discounted successor values for V and Q.
  • Soft Q-learning minimizes least-squares losses against entropy-regularized backup targets, with n-step targets requiring actions sampled from the Boltzmann policy for unbiasedness.
  • The policy-gradient estimators must include entropy terms in returns for proper estimation; common approximations ignore delayed effects of actions on future entropy penalties.

4 Soft Q-learning Gradient Equals Policy Gradient

The soft Q-learning squared-error gradient equals a policy gradient plus a value-function fitting gradient, without assumptions about Q-function parameterization. The equivalent policy-gradient method scales the policy-gradient term by τ relative to value-function error.

  • Gradient equivalence: Soft Q-learning’s squared-error gradient equals a policy gradient plus a value-function fitting gradient.The result holds without assuming a particular parameterization of Qθ.
  • Gradient equivalence: The policy component arises from the entropy-regularized Boltzmann policy defined from Qθ.The paper defines πθ and Vθ as functions of the parameterized Q-function.
  • Gradient decomposition: The derivation rewrites the soft Q-learning objective using the n-step backup and temporal-difference accumulation before separating gradient terms.The calculation swaps gradient and expectation, substitutes the Q-function representation, and replaces the backup target.
  • Gradient decomposition: The separated terms identify a policy-gradient contribution and a value-function-gradient contribution.The policy term includes the entropy-regularization KL component, while the remaining term is labeled the value-function gradient.
  • Coefficient choice: The equivalent policy-gradient method multiplies the policy gradient by τ relative to the value-function error.Equivalently, the value-function error has coefficient τ^-1, larger than typically used in practice.

5 Soft Q-learning and Natural Policy Gradients

The paper gives a second-order, coordinate-free connection between soft Q-learning and natural policy gradients. Standard Q-learning produces a different regression target, while a damped Q-learning update exactly matches the natural-gradient least-squares problem.

  • Natural policy gradient: Natural policy gradients are presented through a least-squares regression involving the Fisher information matrix and policy-gradient estimate.The natural gradient is F^-1g, with F the average Fisher information matrix and g a policy-gradient estimate.
  • Natural policy gradient: The least-squares problem approximates changes in log policy probabilities using advantage estimates and a parameter update.To first order, log πθ − log πθold is approximated by the policy-gradient feature vector dotted with the update.
  • Entropy regularization: The natural-gradient equivalence includes an entropy-regularization KL-divergence gradient term.The corresponding least-squares formulation incorporates the entropy-regularized objective.
  • Q-learning regression: Soft Q-learning regresses log πθ toward log πθold + Δt/τ + KL[πθold, π](st), which is not equivalent to the natural policy-gradient loss.This regression interpretation follows from the Q-learning update under the stated value-function assumption.
  • Damped Q-learning: A damped Q-learning update interpolating between old and backed-up values exactly matches the entropy-regularized natural-policy-gradient least-squares problem.The interpolation uses ˆQt = (1−ε)Qθold(st,at) + εˆQt, yielding a natural-gradient step.

6 Experiments

Experiments compare entropy-regularized and standard objectives on Atari, then test whether the theoretical policy-gradient/soft-Q-learning equivalence persists under practical sampling and architecture choices.

  • Experimental questions: The experiments evaluate entropy-regularized policy gradients and Q-learning against naive or standard variants on Atari.They examine proper versus naive entropy handling, soft versus hard Q-learning, and matched policy-gradient and Q-learning updates.
  • 6.1 A2C on Atari: Naive vs Proper Entropy Bonuses: The proper entropy-regularized policy-gradient version performs the same as or possibly better than the naive version.The proper algorithm reused hyperparameters tuned for the naive algorithm.
  • 6.2 DQN on Atari: Standard vs Soft: Soft Q-learning with an entropy bonus and τ = 0.1 seems slightly better than standard DQN, whereas the KL-bonus version performs worse.The authors suggest the benefit may come from the positive constant added to rewards, which encourages longer episodes.
  • 6.2 DQN on Atari: Standard vs Soft: Five-step Q-learning improves performance on Pong and Q-bert but worsens it on other games, matching the pattern found with n-step policy gradients.All conditions use the same epsilon-greedy exploration; only the backup equation and loss definition differ.
  • 6.3 Entropy Regularized PG vs Online Q-Learning on Atari: With A2C-matched gradient scaling, policy-gradient and Q-learning curves are almost identical in most games.The result indicates essentially the same learning dynamics despite gradients being estimated from few samples, and the Q-learning method remains stable without a target network or epsilon schedule.

7 Related Work

Related work connects policy-based and value-based methods through entropy regularization, but differs in decomposition, off-policy consistency, and high-dimensional action-space techniques.

  • Connections between policy and value methods: Prior work also observes that policy-based and value-based updates become closely connected with entropy regularization.The cited approaches motivate or develop related links between policy gradients, Q-learning, and SARSA.
  • O’Donoghue et al.: O’Donoghue et al. decompose Q into policy and value components and derive a Bellman-error gradient resembling a policy-gradient update.Their PGQ algorithm mixes the resulting updates.
  • Nachum et al.: Nachum et al. develop an off-policy entropy-regularized method using a multi-step consistency equation valid when actions come from a different policy.Their objective jointly minimizes a squared error in policy and value parameters, resembling Bellman residual minimization.
  • Haarnoja et al.: Haarnoja et al. address high-dimensional action spaces where Boltzmann policies are multimodal and difficult to sample from in closed form.They use Stein Variational Gradient Descent to derive a procedure that jointly updates the relevant quantities.

8 Conclusion

The paper establishes an expected-gradient equivalence between soft Q-learning and policy gradients with value fitting, relates damped Q-learning to natural policy gradients, and validates the connection on Atari.

  • Conclusion: Soft Q-learning is equivalent to a policy-gradient method with value-function fitting in entropy-regularized reinforcement learning, in terms of expected gradients.The paper also analyzes damped Q-learning as a natural policy-gradient method from a second-order perspective.
  • Conclusion: Experiments show that the analyzed entropy-regularized formulation works on Atari and that the equivalence holds in a practically relevant regime.The empirical conclusion covers both benchmark performance and matched learning dynamics.
Loading 1704.06440v4…