Combining policy gradient and Q-learning

1 INTRODUCTION

• Model-free reinforcement learning commonly uses action-value fitting or policy gradients; Q-learning can use replay-buffer data, while vanilla actor-critic methods are on-policy.Q-learning targets values for the optimal policy, whereas SARSA fits values to the current policy.Source: §1 · p. 1
• Empirical examples indicate that PGQL improves data efficiency and stability.The tabular policy also stays close to the policy without the Q-learning update when regularization is small.Source: §1 · p. 2
• PGQL derives Q-value estimates from the policy at a regularized policy-gradient fixed point, then refines them using off-policy data and Q-learning.The paper motivates this update by showing that the induced Q-values have a small Bellman residual when regularization is small.Source: §1 · p. 2

1.1 PRIOR WORK

• The paper relates TD-actor-critic to expected-SARSA with Boltzmann exploration and advantage/value-decomposed Q-values; its Q-learning-style extension resembles the dueling architecture.It also frames regularized actor-critic methods as action-value learning.Source: §1.1 · p. 2, §1 · p. 2

2 REINFORCEMENT LEARNING

• An action-value is expected discounted return after taking an action and following a policy; the Bellman operators have unique fixed points, approached by value iteration.The section also defines state value and advantage as the action-value difference from state value.Source: §2 · pp. 2–3

2.1 ACTION-VALUE LEARNING

• Q-learning updates action-values toward the optimal Bellman target, while SARSA uses the current-policy target; epsilon-greedy and Boltzmann policies support exploration.Boltzmann exploration applies a softmax over Q-values with a temperature.Source: §2.1 · p. 3

2.2 POLICY GRADIENT

• Policy gradients improve a parameterized policy by gradient ascent, using state-action expectations; this paper uses the non-discounted state distribution and entropy regularization.Entropy regularization discourages deterministic policies and supports continued exploration.Source: §2.2 · pp. 3–4

3.1 TABULAR CASE

• At a tabular fixed point, regularized policy gradient yields a softmax over its policy-induced advantage, enabling Q-value estimation from policy probabilities.Action preferences equal Q-values scaled by 1/α, up to an additive per-state constant.Source: §3.1 · p. 4

3.2 GENERAL CASE

• With function approximation, the fixed-point condition makes Q-estimation error orthogonal to policy log-gradient directions; a good architecture may make the policy-to-Q relation approximate.Source: §3.2 · p. 5

3.3 CONNECTION TO ACTION-VALUE METHODS

• Actor-critic and action-value fitting updates coincide under matched initialization and critic errors when the fitting measure is replaced by the current policy, modulo the stated state-distribution qualification.Regularized policy-gradient methods can also be interpreted as advantage-function learning; in the tabular case, the centered action preferences equal the advantage values.Source: §3.3 · pp. 5–6

3.4 BELLMAN RESIDUAL

• As α decreases, tabular policy-induced Q-values have Bellman residual tending to zero, and Q^π_α converges to Q*.The section bounds the residual using f_α(x) ≤ αe^-1.Source: §3.4 · p. 6

4 PGQL

• PGQL adds a Q-learning update to regularized policy gradient, using Q-values estimated from the policy to reduce the Bellman residual.The update is motivated by the small Bellman residual at the regularized policy-gradient fixed point when the regularization penalty is sufficiently small.Source: §4 · pp. 6–7

4.1 PGQL UPDATE

• The combined update weights policy-gradient and Q-learning terms with η, ranging from entropy-regularized policy gradient at 0 to a batch Q-learning variant at 1.Intermediate values blend the updates; the two terms trade off consistency with the critic and optimality over time.Source: §4.1 · p. 7

4.2 PRACTICAL IMPLEMENTATION

• In the online scheme, agents write experience to shared replay memory, which a separate learner samples for Q-learning updates on the policy parameters.Replay uses data from older policies, making this slightly off-policy; it can also reduce correlated-data problems and act as a regularizer against moving too far from the Bellman equation.Source: §4.2 · pp. 7–8

4.3 MODIFIED FIXED POINT

• In the tabular case, as α decreases, the policy-estimated and induced Q-values approach one another, while the limiting Q-values remain optimal.The authors conclude that the auxiliary update does not perturb the policy very much in this limit.Source: §4.3 · p. 8

5.1 GRID WORLD

• In the grid world, PGQL applied a TD-actor-critic update followed by a replay-buffer Q-learning update, using α = 0.001 and γ = 0.95.The comparison used a 4 by 6 grid world with a reward of 1 at the terminal square and zero otherwise.Source: §5.1 · pp. 8–9
• With step size 1 in the deterministic grid world, pure actor-critic failed at larger step sizes, while PGQL and Q-learning tolerated some increase, possibly through replay stabilization.The step size was set to 1; the authors suggest replay may explain the other methods' tolerance of some larger values.Source: §5.1 · p. 9
• At matched data exposure in the grid world, PGQL had higher performance than both vanilla methods at practically every point, indicating greater data efficiency.Performance traces show true expected policy performance from the start state.Source: §5.1 · pp. 8–9

5.2 ATARI

• For Atari, PGQL augmented an A3C-matched network with a parameterless Q-value layer and one replay-based Q-learning update after each policy-gradient update.Each actor-learner kept the last 100k transitions; training ran for 50 million agent steps (200 million Atari frames).Source: §5.2 · p. 9
• PGQL performed best in 34 games, compared with 7 for A3C and 10 for Q-learning, and was the worst performer in only one game.Two or more methods tied in six games.Source: §5.2 · p. 10
• In Atari games where PGQL underperformed, it often showed better early data efficiency before performance saturated or collapsed; the authors hypothesize local optima or early-data overfit.They suggest those cases might improve with hyperparameter tuning.Source: §5.2 · p. 10
• Across Atari, PGQL had the highest mean and median normalized scores in both random-start and human-start evaluations, with median scores above the 100% human threshold.Scores are reported as percentages of human normalized score.Source: Table 1 · p. 10, Table 2 · p. 10, §5.2 · p. 10

• For small regularization, the Bellman residual of Q-values induced by a regularized policy-gradient fixed point must be small.Source: §6 · p. 11
• PGQL adds an auxiliary update related to that Bellman residual and performs it off-policy using stored experience.Source: §6 · p. 11
• The authors report improved data efficiency and stability versus actor-critic or Q-learning alone, and Atari performance exceeding both A3C and Q-learning.Source: §6 · p. 11

A PGQL BELLMAN RESIDUAL

• The proof first shows that the Bellman residual of the approximate Q-values converges to zero by bounding it using the policy Bellman residual and the difference between approximate and induced Q-values.The intermediate bounds combine the contraction factor γ with the residual term scaled by η.Source: A PGQL BELLMAN RESIDUAL · p. 14
• A final bound relates the induced Q-values’ Bellman residual to the approximate Q-values’ residual and their difference, yielding the stated zero-residual limit.The conclusion follows from this bound and the two preceding results.Source: A PGQL BELLMAN RESIDUAL · p. 14
• As α decreases, the Bellman residual of PGQL’s induced Q-values converges to zero, matching vanilla regularized policy gradient’s guarantee.The result is stated for the tabular case.Source: A PGQL BELLMAN RESIDUAL · p. 14

B ATARI SCORES

• Table 3 reports normalized scores for the Atari suite from random starts, expressed as percentages of human normalized score.Source: Table 3 · p. 15