PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the Gap Between Value and Policy Based Reinforcement Learning

Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans

arXiv:1702.08892v3cs.AIcs.LGstat.ML

TL;DR

The paper addresses the difficulty of combining policy-based stability with value-based data efficiency under deep function approximation. It derives entropy-regularized softmax path consistency and uses it to build PCL and Unified PCL for on- and off-policy learning. Experiments report that PCL significantly outperforms strong actor-critic and Q-learning baselines across several benchmarks.

  • Problem

    Deep RL lacks a well-understood way to combine policy-based stability with value-based data efficiency while avoiding off-policy instability under function approximation.

  • Method

    The paper derives softmax consistency between entropy-regularized policies and values along action sequences, then minimizes multi-step consistency errors using PCL and a unified policy-value model.

  • Results

    PCL significantly outperforms strong actor-critic and Q-learning baselines across several benchmarks, while Unified PCL is competitive or better on more difficult tasks.

  • Takeaways & Limitations

    Softmax path consistency provides an RL framework that connects actor-critic and Q-learning perspectives and can use a single model for policy and value estimates.

  • Takeaways & Limitations

    In stochastic settings, Monte Carlo estimation of the squared inconsistency is biased relative to the true squared inconsistency.

Abstract

from arXiv · show

We establish a new connection between value and policy based reinforcement learning (RL) based on a relationship between softmax temporal value consistency and policy optimality under entropy regularization. Specifically, we show that softmax consistent action values correspond to optimal entropy regularized policy probabilities along any action sequence, regardless of provenance. From this observation, we develop a new RL algorithm, Path Consistency Learning (PCL), that minimizes a notion of soft consistency error along multi-step action sequences extracted from both on- and off-policy traces. We examine the behavior of PCL in different scenarios and show that PCL can be interpreted as generalizing both actor-critic and Q-learning algorithms. We subsequently deepen the relationship by showing how a single model can be used to represent both a policy and the corresponding softmax state values, eliminating the need for a separate critic. The experimental evaluation demonstrates that PCL significantly outperforms strong actor-critic and Q-learning baselines across several benchmarks.

1 Introduction

Model-free RL seeks effective behavior through reward-driven interaction, but deep function approximation makes it difficult to combine policy-based stability with value-based data efficiency. The paper uses entropy-regularized softmax consistency to develop stable off-policy learning and unify policy and value representations.

  • Motivation: Model-free RL optimizes expected discounted reward through trial-and-error interaction with a black-box environment.Applications include games, robotics, and marketing.
  • Policy-based methods: Policy-based methods directly optimize the target and remain stable under function approximation, but rollout-based gradients can have extreme variance.Actor-critic methods reduce variance by using value approximators.
  • Value-based methods: Value-based methods learn from arbitrary trajectories and can exploit expert data, but off-policy learning is unstable with function approximation and may require extensive tuning.Their off-policy nature makes them more sample efficient than on-policy methods.
  • Motivation: Existing off-policy actor-critic methods improve data efficiency but retain theoretical difficulties and can require specialized development and delicate tuning.
  • Contribution: The paper relates entropy-regularized policy optimization to softmax value consistency, yielding stable off-policy learning and a single model that can represent both policy and value.These observations motivate the proposed training objective and unified actor-critic representation.

2 Notation & Background

The paper formalizes neural policies interacting with an environment and reviews hard-max Bellman consistency underlying Q-learning. It also notes that the main presentation assumes deterministic dynamics, while the theory extends to stochastic settings.

  • RL setup: An agent samples actions from a neural-network policy, receives scalar rewards, and transitions to subsequent states.The policy is a parametric distribution over a finite action set.
  • Assumption: The main presentation assumes deterministic rewards and transitions for clarity, while the appendix treats stochastic dynamics without changing the desired properties or algorithms.
  • Q-learning background: Q-learning uses hard-max Bellman backups to bootstrap action values from successor action values for off-policy temporal-difference control.
  • Optimal control: Under the standard objective, the optimal policy is a one-hot distribution selecting an action with maximal return.

3 Softmax Temporal Consistency

Discounted entropy regularization produces softmax temporal consistency: optimal policies become Boltzmann distributions, and optimal values follow log-sum-exp backups. These consistencies support asynchronous backups with tabular convergence to a unique optimal fixed point.

  • Entropy regularization: Entropy regularization encourages exploration and helps prevent early convergence to sub-optimal policies.The regularized objective combines expected reward with discounted entropy weighted by temperature τ.
  • Soft optimal policy: For τ > 0, the entropy-regularized optimal policy is a non-one-hot Boltzmann distribution over actions based on reward and successor-state values.
  • Soft value backup: The soft optimal state value is obtained through a softmax, or log-sum-exp, backup over successor values.As τ approaches zero, the softmax formulation recovers hard-max state values.
  • Algorithmic implication: The softmax consistency can be used for asynchronous one-step backups, which converge in the tabular case to a unique fixed point representing optimal values.
  • Relation to prior work: The paper distinguishes its contribution from prior softmax Q-value work by developing training principles beyond single-step Q-learning variants.

4 Consistency Between Optimal Value & Policy

The paper establishes that entropy-regularized optimal policies and state values satisfy a softmax consistency relation for individual actions and arbitrary action sequences. Its converse shows that satisfying this relation identifies the optimal policy and values, motivating path-wise learning objectives.

  • Core connection: The paper observes that exp{V∗(s)/τ} normalizes the optimal policy and links state values to optimal action probabilities.
  • One-step consistency: For τ > 0, optimal values and policy probabilities satisfy V∗(s) − γV∗(s′) = r(s, a) − τ log π∗(a | s).
  • Multi-step consistency: The one-step relation extends to any action sequence by relating endpoint values to rewards and optimal log-probabilities along the trajectory.Intermediate state values cancel when the one-step relations are summed.
  • Converse result: The converse theorem states that a policy and value function satisfying the consistency property everywhere equal the optimal policy and state values.
  • Learning objective: This theorem motivates learning parameterized policy and value estimates by minimizing one-step and multi-step path-wise consistency discrepancies.

5 Path Consistency Learning (PCL)

PCL minimizes soft consistency errors over multi-step trajectories from both on-policy and replay data. Its unified variant represents policy and value with one model, while the path-consistency framework connects actor-critic and Q-learning regimes.

  • The algorithm trains a policy πθ and state-value function Vφ by driving the consistency error toward zero across sub-trajectories.
  • PCL minimizes squared soft consistency error over d-length sub-trajectories collected from on-policy and off-policy traces.Updates use trajectories sampled from the current policy and episodes drawn from a replay buffer.
  • In stochastic settings, Monte Carlo estimation of the squared inconsistency is biased because the expectation lies inside the true squared objective.
  • 5.1 Unified Path Consistency Learning (Unified PCL): Unified PCL combines the policy and value function into a single action-value model while optimizing the same PCL objective.The unified model estimates both state values and policy probabilities.
  • 5.2 Connections to Actor-Critic and Q-learning: As τ → 0 without replay, PCL recovers a slight variation of A2C, while replay data extends the framework beyond A2C’s on-policy setting.
  • 5.2 Connections to Actor-Critic and Q-learning: Setting d = 1 in Unified PCL yields a form of soft Q-learning, whereas multi-step path consistency provides a theoretically sound extension beyond one-step hard-max consistency.

6 Related Work

Prior work established related softmax and temporal-consistency ideas, but generally used single-step algorithms or lacked the paper’s explicit entropy-regularized connection. PCL instead emphasizes multi-step consistency on both on- and off-policy data, and experiments report performance matching or exceeding the cited baselines.

  • Earlier work noted connections between softmax Q-values and entropy-regularized policies but did not state the paper’s single- and multi-step consistencies over all action choices.
  • Multi-step hard-max Q-learning has empirical success but lacks theoretical justification because rewards after non-optimal actions do not relate to hard-max Q-values.
  • Across the reported tasks, PCL is comparable to A3C on some tasks, outperforms A3C on more challenging tasks, and exceeds DQN performance.Figure 1 plots average reward over training iterations across five runs, or ten for Synthetic Tree.
  • Related softmax, Boltzmann, and mellowmax methods primarily optimize single-step consistency and do not present the same clear link to entropy-regularized expected reward.
  • Unlike many multi-step off-policy methods using importance-sampling or eligibility-trace corrections, PCL defines an unbiased consistency for an entire trajectory applicable to on- and off-policy data.

7 Experiments

The experiments compare PCL and Unified PCL with A3C and prioritized double Q-learning across multiple tasks. PCL matches or beats the baselines, while expert trajectories substantially improve performance.

  • Baseline comparisons: PCL consistently matches or beats A3C and prioritized double Q-learning across several evaluated tasks.The evaluation uses average reward over training iterations and compares both proposed algorithms with strong actor-critic and Q-learning baselines.
  • Experimental protocol: The experimental setup and task details are provided in the appendix.The reported comparisons use multiple randomly seeded runs and average reward over training iterations.
  • Baseline comparisons: On harder tasks, the performance gap between PCL and A3C becomes more pronounced, while prioritized DQN performs worse across the experiments.The larger gaps occur on Synthetic Tree, DuplicatedInput, ReversedAddition, ReversedAddition3, and Hard ReversedAddition.
  • Unified PCL: Unified PCL is slightly detrimental on simpler tasks but competitive with or better than PCL on more difficult tasks.Unified PCL uses a single model for both values and policy; Figure 2 reports the same qualitative pattern.
  • Expert trajectories: Adding 10 expert trajectories to the replay buffer significantly improves PCL performance despite a mini-batch size of 400.The expert trajectories are retained with maximal priority, and Unified PCL shows a similar lift.

8 Conclusion

The conclusion presents PCL as a path-consistency method connecting actor-critic and Q-learning, with Unified PCL combining policy and value representations. Both methods support on- and off-policy data while using multi-step consistencies.

  • Conclusion: Entropy regularization induces single-step and multi-step softmax consistency between optimal policies and state values.This consistency motivates the path-wise learning objective used by PCL.
  • Conclusion: PCL jointly learns separate policy and state-value models while minimizing a temporal consistency error.This gives PCL similarities to actor-critic and Q-learning without reducing it to either method.
  • Conclusion: Unified PCL combines policy and value estimation in a single model, removing the usual distinction between actor and critic.The unified parameterization represents both the policy and corresponding softmax state values.
  • Conclusion: PCL and Unified PCL use both on-policy and off-policy trajectories and can exploit multi-step consistencies.This contrasts with standard policy-based methods and value-based algorithms as characterized in the conclusion.
  • Conclusion: The paper reports significant improvement over baseline methods in empirical evaluations.The implementation and experimental setup are described through Algorithm 1 and the accompanying experimental section.

B.2 Algorithmic Tasks

The algorithmic-task evaluation uses six OpenAI Gym tasks ordered roughly by difficulty, plus a curriculum-free hard variant. Agents operate on character or digit grids and are implemented with recurrent neural networks.

  • Task suite: The six algorithmic tasks are Copy, DuplicatedInput, RepeatCopy, Reverse, ReversedAddition, and ReversedAddition3.They are listed in rough order of difficulty and require copying, transforming, or adding sequences.
  • Task suite: ReversedAddition and ReversedAddition3 require summing two or three ternary numbers observed in little-endian grid representations.The tasks differ in whether the input contains two or three numbers.
  • Task suite: Hard ReversedAddition has the same goal as ReversedAddition but removes its implicit curriculum.It is included to evaluate performance without curriculum assistance.
  • Model and setup: The agents use LSTM recurrent neural networks with hidden dimension 128 on these environments.The same task setup includes extensive searches over discount, replay, critic, and entropy-related hyperparameters.
  • Model and setup: Experiments were implemented using TensorFlow.The appendix provides the broader experimental setup and task details.

C Proofs

The proofs extend path-consistency results from a simple setting to stochastic infinite-horizon environments. They establish a unique entropy-regularized value fixed point, its optimality, and the policy consistency results.

  • Proof strategy: The theoretical development begins with one-shot decision making before addressing the general infinite-horizon setting.The initial results support the proof of the broader path-consistency claims.
  • Proof strategy: The main-body deterministic-dynamics assumption is only for clarity; the appendix treats the more general stochastic setting.The authors state that the restriction is unnecessary for the main results.
  • Stochastic infinite horizon: The softmax Bellman operator is shown to have a unique fixed point through contraction under the infinity norm.The proof then identifies this fixed point with the optimal entropy-regularized expected-return value.
  • Stochastic infinite horizon: The proofs construct a policy from the fixed-point value and establish its consistency and optimality through the KKT conditions.The resulting value equals the optimal entropy-regularized value denoted V †.

C.1 Basic results for one-shot entropy regularized optimization

The section characterizes entropy-regularized one-shot optimization through softmax values and softmax-induced policy probabilities. Mutual consistency across values and action probabilities is equivalent to optimality, and the softmax operator is a contraction.

  • Definitions: The softmax-induced probability vector maps any finite action-value vector to a probability distribution, while the softmax value incorporates entropy.The paper defines f_τ as the gradient of F_τ and writes the entropy-regularized objective using H(π).
  • One-shot optimization: KKT conditions uniquely yield π = f_τ(q) and v = F_τ(q), establishing the unique optimizer and its corresponding value.The objective is strictly concave, so the KKT solution is the unique global maximizer.
  • Value-policy relationship: For an optimal action-value vector, the optimal state value equals the entropy-regularized value of the softmax policy.This identifies the softmax state value, action values, and policy probabilities as mutually linked quantities.
  • Optimality and consistency: Consistency holds for every action rather than only in expectation, and achieving this mutual consistency is equivalent to optimality.The stronger condition connects action-level probabilities, state values, and action values directly.
  • Operator properties: The softmax Bellman operator is an infinity-norm contraction, supporting convergence properties for entropy-regularized value updates.The contraction result is stated as a consequence of the corresponding softmax inequality.
  • Stochastic extension: For stochastic environments with finite action and discrete state spaces, the entropy-regularized value is a fixed point of the on-policy Bellman operator, which is itself a contraction.Repeated on-policy backups converge to the policy’s entropy-regularized value.

C.3 Proof of main optimality claims for off-policy softmax updates

The section proves that the softmax Bellman fixed point is the optimal entropy-regularized value and that action-level consistency characterizes the corresponding optimal policy. These results extend from deterministic presentation to stochastic environments.

  • Fixed-point existence: For γ < 1, the softmax Bellman operator has a unique fixed point.The proof uses its infinity-norm contraction property and the fixed-point theorem.
  • Policy optimality: The policy defined from the fixed-point value achieves that value as its entropy-regularized expected return.Lemma 18 establishes V ∗ = ˜V π∗ for the policy constructed from V ∗.
  • Main optimality theorem: The fixed point of the softmax Bellman operator equals the optimal value function.Theorem 19 derives equality by comparing the fixed point with all policy returns and the constructed policy’s return.
  • Consistency: The optimal state value and policy satisfy V ∗(s) = r(s, a) + γE[V ∗(s′)] − τ log π∗(a|s) for every state and action.This is the paper’s optimality-implies-consistency result.
  • Consistency: Conversely, satisfying the consistency equation for every state and action implies both the optimal value function and the optimal policy.Corollary 21 establishes V = V ∗ and π = π∗ under the stated condition.
  • Stochastic extension: The main deterministic result is presented as a special case of the more general stochastic-environment development.The appendix explicitly states that the deterministic assumption simplifies presentation rather than limiting the proof’s general setting.
Loading 1702.08892v3…