Source-linked AI summary
Experience Replay with Likelihood-free Importance Weights
Samarth Sinha, Jiaming Song, Animesh Garg, Stefano Ermon
TL;DR
Prioritized replay for actor-critic learning can favor large TD errors even when those errors occur on states the current policy rarely visits. This paper weights replay experiences by their likelihood under the current policy’s stationary distribution, estimating density ratios from replay buffers. The approach achieves superior sample complexity to other weighted replay methods on the reported SAC and TD3 tasks.
Problem
In actor-critic methods, prioritizing large TD errors can favor states rarely visited by the current policy instead of correcting errors on frequently encountered states.
Source: §1 · p. 1Method
Results
LFIW outperforms other weighted replay approaches in sample complexity on 4 of 5 SAC tasks and all TD3 tasks.
Source: §1 · p. 2Takeaways & Limitations
The approach can be combined with replay-based actor-critic methods, and the reported SAC and TD3 results show superior sample complexity on most evaluated tasks.
Source: §7 · p. 8, §6.1 · p. 8Correcting policy updates with the weights yielded little marginal improvement and was not included in the comparison.
Source: §4 · p. 6
Abstract
from arXiv · showhide
The use of past experiences to accelerate temporal difference (TD) learning of value functions, or experience replay, is a key component in deep reinforcement learning. Prioritization or reweighting of important experiences has shown to improve performance of TD learning algorithms.In this work, we propose to reweight experiences based on their likelihood under the stationary distribution of the current policy. Using the corresponding reweighted TD objective, we implicitly encourage small approximation errors on the value function over frequently encountered states. We use a likelihood-free density ratio estimator over the replay buffer to assign the prioritization weights. We apply the proposed approach empirically on two competitive methods, Soft Actor Critic (SAC) and Twin Delayed Deep Deterministic policy gradient (TD3) -- over a suite of OpenAI gym tasks and achieve superior sample complexity compared to other baseline approaches.
1 Introduction
The introduction motivates prioritizing replay data according to how often the current policy encounters it, rather than relying on large TD errors. It proposes likelihood-free importance weights and evaluates them with SAC and TD3.
1 Introduction
- Actor-critic methods should prioritize correcting TD errors on frequently encountered states, rather than large errors on states rarely visited by the current policy.The paper motivates this distinction because actor-critic methods learn the Q-function induced by the current policy.Source: §1 · p. 1
- The method weights replay experiences by their likelihood under the current policy’s stationary distribution, estimated with a likelihood-free density-ratio classifier.It uses fast and slow replay buffers to represent near-on-policy and additional off-policy experiences.Source: §1 · pp. 1–2
- LFIW outperforms other weighted replay approaches in sample complexity on 4 of 5 SAC tasks and all TD3 tasks.The evaluation uses OpenAI Gym tasks based on the MuJoCo simulator.Source: §1 · p. 2
2 Preliminaries
The preliminaries define the MDP and value-function setting for actor-critic learning. They describe TD learning as fitting a Q-function to Bellman targets using replay data, and introduce prioritized replay as a way to reweight that data.
2 Preliminaries
- An MDP is specified by state and action spaces, transition dynamics, rewards, a discount factor, and an initial-state distribution.The objective is to learn a stationary policy that maximizes expected cumulative reward.Source: §2 · p. 2
- For a fixed policy, the paper defines its state-action value function Qπ and the corresponding value and advantage functions.Actor-critic methods update a parameterized stochastic policy using these quantities.Source: §2 · p. 2
- TD learning fits a Q-network to Bellman targets estimated from on-policy or off-policy experiences, including data in a replay buffer.Prioritized experience replay assigns higher weights to transitions with larger TD errors.Source: §2 · pp. 2–3
3 Prioritized Experience Replay based on Stationary Distributions
The section argues that replay prioritization should match the current policy’s stationary state-action distribution. This choice aligns TD errors with policy-relevant states and preserves a Bellman contraction property that other weighting distributions do not guarantee.
3 Prioritized Experience Replay based on Stationary Distributions
- The proposed priority distribution is dπ, the stationary distribution of state-action pairs under the current policy.The paper argues that errors on frequently encountered pairs matter more for policy updates.Source: §3 · p. 3
3.1 Policy-dependent Norms for Bellman Backup
- The Bellman operator is a γ-contraction under the ℓ2 norm weighted by dπ, but this contraction is not guaranteed under other distributions.The section contrasts this policy-dependent distance with the policy-independent supremum norm, which can be dominated by a rarely sampled pair.Source: §3.1 · pp. 3–4
3.2 TD Learning based on dπ
- Large Q-function errors on high-frequency state-action pairs can make actor-critic policy updates worse than updates using the true Qπ.Minimizing the TD objective under dπ gives more emphasis to TD errors on frequently encountered pairs.Source: §3.2 · p. 4
- In a five-state chain MDP with p = 0.8, dπ prioritization took 120 steps on average to reduce expected error below 1, versus 182 steps for TD-error prioritization.The simulation compares dπ prioritization with uniform sampling and TD-error prioritization, and reports faster convergence, especially in early iterations.Source: §3.2 · p. 4
4 Likelihood-free Importance Weighting over Replay Buffers
The method estimates replay-buffer importance weights without requiring policy likelihoods, using fast and slow buffers to approximate on-policy and off-policy distributions. It applies the resulting weights to Q-function updates while leaving the rest of the actor-critic algorithm unchanged.
4 Likelihood-free Importance Weighting over Replay Buffers
- Likelihood-free density-ratio estimation is used because the replay buffer mixes trajectories from different policies, making direct importance ratios difficult to estimate.The approach estimates ratios from samples rather than requiring likelihoods.Source: §4 · p. 5
- The fast replay buffer contains recent, approximately on-policy experiences, while the larger slow buffer contains a mixture of off-policy trajectories.The slow buffer has broader coverage of transition dynamics but is less on-policy; the estimated ratios are self-normalized with temperature T.Source: §4 · p. 5
- Estimated density ratios weight Q-function updates, while policy-gradient and value-network updates remain unchanged.The method is applied with SAC and TD3 and does not require likelihood estimates from the policy.Source: §4 · pp. 5–6
- Correcting policy updates with the weights yielded little marginal improvement and was not included in the comparison.The authors distinguish this choice from applying the weights to Q-function updates.Source: §4 · pp. 5–6
5 Related Work
Prior work uses replay for sample-efficient learning and prioritizes experiences in several ways. This paper places its approach among methods that reweight state-action sampling for TD value updates, using likelihood-free density ratios rather than requiring the policy distribution.
Related work
- Replay supports off-policy sample efficiency, and this paper addresses importance weighting for sampling state-action pairs during value updates rather than debiasing longer-trace TD targets.The latter is a distinct use of importance weights for TD(λ).Source: §5 · p. 6
- The paper distinguishes its focus on value-update sampling from importance weighting that corrects off-policy TD targets for longer traces.Replay also supports policy updates and TD-target evaluation, beyond the value-update use addressed here.Source: §5 · p. 6
- Prioritized Sweeping selects updates by value changes, PER [32] emphasizes large TD errors, and ReF-ER [28] removes experiences that differ substantially from current-policy choices.The proposed approach is similar to ReF-ER in encouraging on-policy behavior but does not require knowledge of the policy distribution.Source: §5 · p. 6
- Prior analyses study TD convergence under a policy’s stationary distribution, while this paper provides a practical deep-RL approach using likelihood-free density-ratio estimation.Unlike cited IL, IRL, and OPE uses, its weights target TD learning with function approximation rather than advantage estimation or reward-estimation bias reduction.Source: §5 · p. 6
6 Experiments
The experiments compare LFIW with uniform replay and TD-error prioritization on SAC and TD3 across Mujoco continuous-control tasks. LFIW improves sample complexity and final performance on most tasks, while additional analyses examine its weights and Q-value estimates.
6 Experiments
- The evaluation combines LFIW with SAC and TD3 and compares it with uniform replay and TD-error prioritization on five Mujoco tasks.The tasks are HalfCheetah, Walker, Ant, Hopper, and Humanoid; Humanoid is excluded for TD3 because the original TD3 algorithm fails to learn successfully.Source: §6 · p. 6, §6.1 · p. 7
6.1 Evaluation
- The evaluation includes learning curves and a table of maximum performance by environment timestep, with the table reporting means and standard deviations over five random seeds.Curves show mean evaluation returns and one standard deviation; evaluations are performed every 1000 timesteps.Source: Table 1 · p. 7, §6.1 · p. 7
- LFIW outperforms baselines in sample complexity and final performance on most tasks, with similar performance on the rest; noticeable gains occur on HalfCheetah, Ant, and Humanoid with SAC and HalfCheetah, Walker, and Ant with TD3.The SAC and TD3 learning-curve figures report evaluation returns; shaded regions show standard deviation across five trials. TD3 Humanoid is omitted because the original algorithm fails to learn successfully.Source: Figure 2 · p. 7, Figure 3 · p. 7, §6.1 · p. 8
- PER and ERE do not perform very favorably against uniform sampling, and combining PER with LFIW had little initial success.The authors suggest PER is designed for Q-learning, where learning the max Q-function is the objective.Source: §6.1 · p. 8
- On SAC Hopper, LFIW initially rises faster but plateaus near baseline performance from about 1200 episodes, when the slow buffer contains mostly near-optimal-policy experiences.SAC reaches near-optimal performance around 200 episodes, so the authors report little LFIW improvement when the slow buffer is already near on-policy.Source: §6.1 · p. 8
6.2 Additional analyses
- SAC+LFIW is reported insensitive to changes in temperature, fast-buffer size, and density-estimator hidden-unit count.These are the hyperparameters varied in the appendix analyses.Source: §6.2 · p. 8
- wψ discriminates 5M-step SAC-policy experiences from a mixture of 1M-to-4M-step policy experiences, achieving 87.3% precision and 73.1% accuracy; its importance weights tend to favor on-policy data and move replay closer to on-policy experiences.Source: §6.2 · p. 8
- SAC+LFIW Q estimates have Pearson and Spearman correlations of 0.74 and 0.42 with Monte Carlo ground truth, versus 0.41 and 0.11 for SAC.The comparison uses 20 trajectories sampled from each policy; higher correlations are better.Source: §6.2 · p. 8
7 Conclusion
The paper concludes that likelihood-free importance weighting prioritizes replay experiences to bring the buffer closer to on-policy data, and reports superior sample complexity on most challenging Mujoco environments. The algorithm estimates density ratios from fast and slow replay buffers before weighting value updates.
7 Conclusion
- The method reweights replay-buffer experiences toward on-policy data using estimated density ratios against the stationary policy distribution.The ratios are estimated from fast and slow replay buffers reflecting on-policy and off-policy experiences, respectively.Source: §7 · p. 8
- The authors identify larger-scale discrete environments, improved density-ratio representations, and prioritization with learned dynamics as future directions.The latter is proposed as a possible aid to planning in model-based reinforcement learning.Source: §7 · p. 8
A Algorithm
- The algorithm updates fast and slow buffers with new transitions, estimates weights after a threshold, and applies weighted TD updates to the Q-function.When the threshold is not met, it uses unit weights; policy and any value-network updates follow the base algorithm.Source: A Algorithm · p. 12
B Proofs
The proof establishes that the Bellman operator is a γ-contraction under a d-weighted norm exactly when d is the stationary distribution dπ. It uses Jensen’s inequality for one direction and a counterexample for the other.
B Proofs
- The Bellman operator is a γ-contraction under the ∥·∥d norm if and only if d = dπ almost everywhere.Source: B Proofs · p. 12
- When d = dπ, stationarity gives d = d′, and Jensen’s inequality establishes the contraction direction.Source: B Proofs · p. 12
- For d different from dπ, the proof constructs a perturbation Q0 + ϵ that makes h(Q) − g(Q) positive, contradicting contraction.Source: B Proofs · pp. 12–13
C.1 Setup on Chain MDP
The chain MDP experiment uses stochastic policies to ensure a stationary distribution, then simulates different weighting schemes with normalized tabular TD updates.
C.1 Setup on Chain MDP
- Because the chain MDP has deterministic transitions, the policy is made stochastic to ensure a stationary distribution exists.Source: §C.1 · p. 13
- Weights for uniform, TD-error, or dπ prioritization are normalized to mean 1, keeping TD updates per epoch equal across methods.Source: §C.1 · p. 13
C.2 Ablation studies
The ablations examine temperature, fast replay-buffer size, and classifier width. Performance is generally stable across tested settings, with weaker sample efficiency at T = 1 and early gains for the smaller classifier.
Replay buffer sizes |Df|
- Performance is relatively stable for |Df| = 1,000–100,000, with a small drop at |Df| = 100,000.The largest tested fast buffer spans 100 episodes, while |Ds| = 10^6.Source: §C.2 · p. 14
Hidden units of wψ
- 128-unit hidden layers perform better initially, while the 256- and 512-unit configurations catch up around 1,000 episodes.Source: §C.2 · p. 14
Temperature T
- Except at T = 1, tested temperatures have similar sample efficiency on Walker2d-v2; Humanoid-v2 likewise performs significantly worse at T = 1.Source: §C.2 · p. 14, Figure 4 · p. 15, Figure 5 · p. 15