PaperlayerView exact revision on arXiv

Source-linked AI summary

Sample Efficient Actor-Critic with Experience Replay

Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, Nando de Freitas

arXiv:1611.01224v2cs.LG

TL;DR

Actor-critic methods need greater stability and sample efficiency across discrete and continuous action spaces, especially as environment simulation becomes expensive. The paper introduces ACER, combining experience replay with corrected importance sampling, Retrace, and scalable trust-region optimization. ACER nearly matches the best prioritized-replay deep Q-networks on Atari and outperforms A3C on Atari and continuous-control problems.

  • Problem

    The paper addresses the challenge of designing stable, sample-efficient actor-critic methods that apply to both continuous and discrete action spaces.

  • Method

    ACER combines actor-critic learning with experience replay, Retrace, truncated importance sampling with bias correction, stochastic dueling networks, and efficient trust-region policy optimization.

  • Results

    ACER nearly matches prioritized-replay deep Q-networks on Atari and substantially outperforms A3C in sample efficiency on Atari and continuous-control domains.

  • Takeaways & Limitations

    ACER matches the best known methods on Atari and outperforms popular techniques on several continuous-control problems.

  • Takeaways & Limitations

    Small values of λ can reduce variance, but occasional large importance weights may still cause instability.

Abstract

from arXiv · show

This paper presents an actor-critic deep reinforcement learning agent with experience replay that is stable, sample efficient, and performs remarkably well on challenging environments, including the discrete 57-game Atari domain and several continuous control problems. To achieve this, the paper introduces several innovations, including truncated importance sampling with bias correction, stochastic dueling network architectures, and a new trust region policy optimization method.

1 INTRODUCTION

The paper targets stable, sample-efficient actor-critic learning across discrete and continuous action spaces. ACER combines experience replay with several innovations and reports strong Atari and continuous-control performance.

  • Motivation: Increasing simulation costs make reducing environment samples central to training agents in realistic environments.Each agent action incurs an expensive simulation step.
  • Motivation: Deep Q-learning is sample efficient but has deterministic optimal policies and costly greedy action selection for large action spaces.These limitations motivate an actor-critic alternative.
  • Motivation: Existing actor-critic methods applicable to both continuous and discrete domains, including A3C, are sample inefficient.Many other policy-gradient methods are restricted to continuous domains or specific tasks.
  • Contribution: ACER nearly matches prioritized-replay deep Q-networks on Atari and substantially outperforms A3C in sample efficiency on Atari and continuous-control domains.The paper presents this as addressing the stable, sample-efficient actor-critic challenge at scale.
  • Contribution: ACER introduces truncated importance sampling with bias correction, stochastic dueling network architectures, and efficient trust region policy optimization.Its design also builds on deep neural networks, variance reduction, Retrace, and parallel RL training.
  • Contribution: The paper proves that the Retrace operator can be rewritten using the proposed truncated importance sampling with bias correction technique.This provides a theoretical connection between the proposed estimator and Retrace.

2 BACKGROUND AND PROBLEM SETUP

The paper formulates an agent that maps observed states to actions and rewards while maximizing discounted return. Actor-critic methods estimate value information to construct policy updates, with A3C providing the on-policy reference that ACER extends off-policy.

  • Problem setup: At each discrete time step, the agent observes a state, selects an action from a discrete or continuous action space, and receives a reward.The state is an n_x-dimensional vector, with discrete and continuous actions treated in later sections.
  • Problem setup: The objective is to maximize expected discounted return, with γ controlling the relative importance of immediate and future rewards.The paper defines state-action and state value functions with respect to the policy.
  • Value functions: Q^π measures expected return after a state-action pair, while V^π averages action values under policy π.These value functions are defined over trajectories and policy-generated actions.
  • Value functions: The advantage A^π(x_t,a_t)=Q^π(x_t,a_t)−V^π(x_t) measures an action’s value relative to the state value.Its expectation over actions under π is zero.
  • Policy gradients: Policy-gradient updates can use advantage, action value, return, or temporal-difference residual estimates without introducing bias, though their variances differ.Neural-network approximation can introduce additional approximation errors and biases.
  • Policy gradients: A3C uses a single trajectory sample and k-step returns to trade off estimator variance and bias.Its value estimate acts as a policy-gradient baseline for variance reduction.
  • ACER setup: ACER is presented as the off-policy counterpart of A3C and retains A3C’s engineering innovations, including efficient parallel CPU computation.It uses one deep network to estimate both policy and value functions, with shared parameters in most components.

3 DISCRETE ACTOR CRITIC WITH EXPERIENCE REPLAY

The discrete ACER agent uses replay with off-policy corrections to improve sample efficiency while controlling estimator variance and instability. It combines Retrace value estimation, bias-corrected importance truncation, and a scalable trust-region update.

  • Off-policy learning: Experience replay improves actor-critic sample efficiency but makes variance and stability of off-policy estimators difficult to control.Importance sampling is unbiased but can involve products of many unbounded weights.
  • Off-policy learning: Marginal importance sampling avoids products of trajectory-level weights, but estimating Q^π and controlling large marginal weights remain challenges.The marginal formulation is expected to have lower variance than trajectory-level importance sampling.
  • Off-policy learning: Small λ can reduce variance, yet occasional large importance weights can still cause instability.This motivates replacing the preceding estimator with Retrace and additional truncation corrections.
  • ACER components: ACER adopts Retrace to estimate Q^π and proposes importance-weight truncation with a correction term for the off-policy actor-critic.The method also introduces an efficient trust-region scheme, with further innovations for continuous actions.
  • Retrace estimation: Retrace recursively combines rewards, truncated importance weights, value differences, and the next-state value estimate.Its truncated weight is min(c,ρ_t), and the algorithm is proven to converge tabularly to the target-policy value function for any behavior policy.
  • Network: The discrete network uses two heads to output action values and the policy, with the state value derived by taking the policy-weighted expectation of action values.This representation modifies the A3C network by outputting a Q vector instead of a scalar value.
  • Retrace estimation: Multi-step Retrace returns estimate Q^π for the policy gradient and can reduce gradient-estimation bias while accelerating critic learning.The same return serves as the critic’s mean-squared-error target.
  • Bias correction: Truncating importance weights bounds gradient-estimator variance, while the correction term preserves unbiasedness for actions whose ratios exceed c.The truncated term is at most c and the correction contribution is bounded by 1.

4 RESULTS ON ATARI

ACER is evaluated on 57 Atari games using a fixed algorithm, network architecture, and hyper-parameters, with replay ratios varied to assess data efficiency. Replay makes ACER more sample efficient than A3C and can match leading DQN performance without necessarily increasing wall-clock cost.

  • Evaluation setup: ACER is evaluated on 57 Atari games from raw pixels and rewards using one algorithm and network architecture with fixed hyper-parameters.The setup uses 16 actor-learner threads on a single machine without GPUs.
  • Evaluation metric: The Atari metric is the cumulative maximum median human-normalized score over all 57 games, computed from scores averaged over the past 1 million frames.Human and random scores are normalized to 1 and 0, respectively.
  • Evaluation setup: Figure 1 compares replay ratios of 0, 1, 4, and 8, with solid and dashed curves denoting ACER with and without trust region updating.The figure also includes original DQN and Prioritized Replay agents as comparison methods.
  • Results: Replay significantly increases ACER’s data efficiency, and sufficient replay enables performance matching the best DQN agents.Off-policy ACER is much more sample efficient than its on-policy counterpart, A3C.
  • Results: ACER performs similarly to A3C when measured by wall-clock time, so improved data efficiency need not compromise computation time.ACER with a replay ratio of 4 is identified as an appealing alternative to Prioritized DQN or A3C.

5 CONTINUOUS ACTOR CRITIC WITH EXPERIENCE REPLAY

The continuous-action extension addresses Retrace’s difficulty in producing both Q and V estimates by introducing Stochastic Dueling Networks and a new value target. It combines these representations with truncated importance sampling, trust-region updating, and Gaussian policies for continuous control.

  • Motivation: Retrace requires both Q and V estimates, but integrating over Q to derive V is difficult in continuous action spaces.Importance sampling can estimate V from Q, but the resulting estimator has high variance.
  • Stochastic Dueling Networks: Stochastic Dueling Networks estimate stochastic Qπ and deterministic Vπ off-policy while maintaining consistency between the two estimates.The architecture samples actions from πθ(·|xt) and uses the stochastic Q estimate to provide an error signal for updating V.
  • Stochastic Dueling Networks: The method adds a novel target for estimating Vπ, derived using truncation and bias correction.This target complements the SDN representation for value estimation.
  • Trust-region updating: For continuous actions, ACER uses Gaussian policies with fixed diagonal covariance and mean φθ(x), then applies the trust-region update scheme.The continuous-space formulation uses a policy gradient with respect to φ and a suitable continuous-action gradient specification.
  • Continuous-control evaluation: Figure 3 presents screenshots of the continuous-control tasks above performance curves, where ACER outperforms other methods and gains are clear on higher-dimensional tasks.The figure caption names humanoid, cheetah, walker, and fish as higher-dimensionality tasks.

6 RESULTS ON MUJOCO

ACER is tested on six MuJoCo continuous-control tasks against on-policy, truncated-importance-sampling, and trust-region baselines. It substantially outperforms the baselines, with especially large gains in higher-dimensional action spaces, while ablations identify Retrace, SDNs, and trust-region updating as critical components.

  • Tasks and baselines: The six MuJoCo tasks span action dimensionalities from 1D cartpole and 3D reacher to 21D humanoid.The benchmark also includes 6D cheetah, 5D fish, and 6D walker.
  • Tasks and baselines: The comparison includes A3C, Trust-A3C, truncated-importance-sampling baselines, Trust-TIS, TIS, and ACER.All five setups are implemented in the asynchronous A3C framework.
  • Experimental setup: Continuous-action ACER is entirely off-policy, with replay memories added to threads for repeated updates from simulator-generated experiences.The implementation uses replay memories of 5,000 frames and updates every 50 steps when replay is used.
  • Experimental setup: The reported curves show the mean and standard deviation of the best 5 out of 30 hyper-parameter settings.Hyper-parameter sensitivity analyses are provided in Appendix Figures 5 and 6.
  • Results: ACER outperforms A3C and truncated-importance-sampling baselines by a very significant margin across continuous-control tasks.The proposed trust-region optimization method also produces huge improvements over the baselines.
  • Results: Trust-region gains are larger in continuous control because high-dimensional continuous-action policies are harder to optimize than small discrete-action policies.Even after trust-region improvements, ACER outperforms all other methods, especially in higher dimensions.
  • Ablation analysis: Removing Retrace or off-policy correction, SDNs, or trust-region updating causes clear performance deterioration in the ablation analysis.Removing truncation with bias correction did not alter results in the reported Fish and Walker cases.

7 THEORETICAL ANALYSIS

The analysis interprets Retrace as truncated importance sampling with bias correction and establishes the resulting operator’s contraction and equivalence to Retrace. It also shows that the operator interpolates between Bellman evaluation and importance sampling.

  • Retrace is interpreted as an application of the paper’s importance-weight truncation and bias-correction technique in the policy-gradient setting.
  • The proposed operator B is a contraction with unique fixed point Qπ and is equivalent to Retrace.Its contraction bound is ∥BQ − Qπ∥∞ ≤ γ∥Q − Qπ∥∞.
  • The alternative derivation provides a different proof that Retrace is a contraction operator.
  • When c = 0, B equals the Bellman operator Tπ; when c = ∞, B recovers importance sampling.
  • The analysis motivates Retrace as a bridge between Bellman evaluation and importance sampling through the truncation parameter c.

8 CONCLUDING REMARKS

The paper concludes that ACER provides a stable off-policy actor-critic framework for both continuous and discrete action spaces by combining established RL advances with three proposed innovations. It reports Atari performance matching the best known methods and outperforming popular techniques on several continuous-control tasks.

  • ACER is a stable off-policy actor-critic method that scales to both continuous and discrete action spaces.
  • The method integrates truncated importance sampling with bias correction, stochastic dueling networks, and efficient trust region policy optimization.
  • ACER matches the performance of the best known methods on Atari and outperforms popular techniques on several continuous-control problems.
  • The efficient trust region optimization method performs remarkably well in continuous domains.The authors suggest it could be useful in other deep-learning domains where training stabilization is difficult.

B Q(λ) WITH OFF-POLICY CORRECTIONS

This section contrasts Q(λ) with off-policy corrections and Retrace, emphasizing their stability conditions and return utilization. It also states the finite-state, finite-action assumptions used in the theoretical analysis.

  • Q(λ) with off-policy corrections can estimate Qπ from trajectories generated under a behavior policy μ.
  • Qopc is a contraction only when the target and behavior policies are close, making it less stable and unsafe for policy evaluation.
  • Qopc can use returns more efficiently because its traces are not cut by truncated importance weights.
  • Qopc leads to faster learning in the continuous-control experiments.
  • The theoretical proposition assumes a Markov decision process with finite state and action spaces and discount factor γ ∈ [0, 1).
  • The operator B becomes importance sampling when c = ∞.

D DERIVATION OF V target

The supplied passages describe continuous-control evaluation settings and baseline trust-region updates, alongside truncation and bias-correction transformations used to derive the V target. The experiments use MuJoCo tasks with fixed episode length and discount factor.

  • The truncation and bias-correction trick is applied to derive identities used in constructing the V target.
  • Because Qπ is unavailable, the derivation replaces it with Monte Carlo Qret and the current neural-network approximation Qθv.
  • The continuous-control experiments use MuJoCo, 500-step episodes, and discount factor γ = 0.99.
  • The benchmark includes Cartpole swingup, Reacher3, Cheetah, Fish, Walker, and Humanoid control tasks.
  • The baselines include separate policy and value updates, while Trust-TIS is modified according to the paper’s trust-region update.

E.3 SENSITIVITY ANALYSIS

ACER’s final performance is evaluated across 30 randomly searched learning rates and trust-region constraints in continuous-control tasks. Overall, ACER is not very sensitive to these hyper-parameters, although larger δ and learning rates can hurt performance in higher-dimensional action spaces.

  • Sensitivity analysis: ACER’s final performance is assessed against learning rates and trust-region constraints δ across continuous-control tasks.Figures 5 and 6 show results for 30 randomly searched settings, with each learning rate paired with a different δ and vice versa.
  • Sensitivity analysis: ACER is not very sensitive to the hyper-parameters overall.
  • Sensitivity analysis: In higher-dimensional action spaces, larger δ values tend to hurt final performance, whereas smaller δ values do not seem to adversely affect it.
  • Sensitivity analysis: In higher-dimensional action spaces, smaller learning rates perform well while bigger learning rates tend to hurt final performance.

E.4 EXPERIMENTAL SETUP OF ABLATION ANALYSIS

The ablation analysis removes one component at a time from the continuous-control experimental setup. It separately evaluates off-policy correction, stochastic dueling networks, and truncated importance sampling with bias correction.

  • E.4 EXPERIMENTAL SETUP OF ABLATION ANALYSIS: The ablation analysis uses the continuous-control experimental setup while removing one component at a time.
  • E.4 EXPERIMENTAL SETUP OF ABLATION ANALYSIS: Replacing Retrace/Q(λ) with recursively expressed importance-sampling estimates evaluates the effectiveness of off-policy correction.The replacement uses the recursive estimate Rt = rt + ρt+1Rt+1.
  • E.4 EXPERIMENTAL SETUP OF ABLATION ANALYSIS: Replacing stochastic dueling networks with separate state-value and Q-value networks evaluates their contribution.The analysis contrasts a naive state-value update with an alternative update described as having markedly lower variance.
  • E.4 EXPERIMENTAL SETUP OF ABLATION ANALYSIS: Setting c to ∞ replaces truncated importance sampling with pure importance sampling to evaluate truncation and bias correction.
Loading 1611.01224v2…