PaperlayerView exact revision on arXiv

Source-linked AI summary

Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?

Alessandro Montenegro, Riccardo Venturelli, Marco Mussi, Matteo Papini, Alberto Maria Metelli

arXiv:2610.01399v1cs.LG

TL;DR

PPO’s on-policy data are discarded after a few updates, limiting sample efficiency. This paper studies recent-sample reuse within PPO using two importance-weighting variants and derives policy improvement lower bounds. It finds moderate reuse improves final performance across diverse control tasks, while excessive reuse can be harmful.

  • Problem

    PPO’s on-policy data are discarded after a few updates, limiting sample efficiency.

    Source: §1 · p. 2

  • Method

    The paper studies PPO variants that reuse samples from recent iterations with uniform or balance-heuristic-corrected importance weights and derives policy improvement lower bounds.

    Source: §1 · pp. 2–3

  • Results

    Moderate reuse consistently improves final performance across diverse control tasks, while excessive reuse can be harmful and neither weighting variant consistently dominates.

    Source: §5 · p. 10

  • Takeaways & Limitations

    ωPPO-U and ωPPO-BH exploit past data differently, so their relative benefit depends on task and hyperparameters.

    Source: §4 · pp. 9–10

    The analysis permits γ=1 only when the horizon is finite.

    Source: §2 · p. 3

Abstract

from arXiv · show

Among on-policy deep reinforcement learning methods, Proximal Policy Optimization (PPO) has become the de facto standard, due to its consistently strong empirical performance across diverse application domains. However, on-policy methods are inherently sample inefficient: fresh data collected under the current policy is used for just a few updates before being discarded. Off-policy methods avoid this inefficiency via experience replay, achieving notable sample efficiency gains, but at the cost of training instabilities or extensive tuning. This motivated the rise of hybrid strategies that augment PPO with off-policy data reuse. Existing sample-reuse variants of PPO demonstrated improved sample efficiency over vanilla PPO, yet a systematic study of when reuse helps, in which scenarios, and to what extent remains missing. In this work, we study the effectiveness of sample reuse in PPO by instantiating two variants within a multiple importance weighting framework. Both retain the core PPO mechanics, reusing only samples from a window of recent iterations, thereby isolating the effect of data reuse from other factors. The variants, termed wPPO-U and wPPO-BH, employ vanilla importance weights or balance-heuristic-corrected ones, respectively. For both, we derive policy improvement lower bounds providing theoretical grounding for their respective losses. We use them to empirically study when and how data reuse improves sample efficiency or final performance of PPO across continuous control tasks.

1 Introduction

On-policy methods offer stable training and theoretical guarantees but discard data after a few updates, limiting sample efficiency. This motivates a controlled study of reusing recent samples in PPO while preserving its core mechanics.

1 Introduction

  • On-policy methods discard collected trajectories after a few updates, limiting sample efficiency despite stable training and theoretical guarantees.Off-policy methods reuse past experience and can be sample-efficient, but may require extensive tuning and be computationally intensive.Source: §1 · p. 2
  • Prior PPO reuse methods often add critic re-evaluation or heuristics, while a systematic study isolating when reuse helps remains missing.The described methods include GePPO, SARD-PPO, PTR-PPO, Off-Policy PPO, HP3O, and RPPO.Source: §1 · p. 2
  • The paper studies reuse within multiple importance weighting, retaining samples from recent iterations to isolate reuse from other changes.Its variants use uniform or balance-heuristic-corrected importance weights.Source: §1 · pp. 2–3
  • On-policy and off-policy are not strictly binary: PPO is conventionally on-policy because it does not reuse data across policy updates.PPO can still take multiple gradient steps on one batch, causing mild distribution shift within an update cycle.Source: §1 · p. 2
  • The study derives policy improvement lower bounds for both reuse variants and compares them with PPO under controlled MuJoCo settings.It examines convergence speed, final performance, window size, and the two weighting schemes.Source: §1 · p. 3

2 Preliminaries

The preliminaries set up the MDP and policy-optimization notation, then introduce PPO’s trust-region motivation and multiple importance weighting for combining samples from behavioral policies.

2 Preliminaries

  • An MDP specifies state and action spaces, transitions, rewards, an initial-state distribution, discount factor, and horizon.The text also defines trajectories and their discounted returns.Source: §2 · p. 3
  • Policy optimization seeks policy parameters that maximize expected return, evaluated through discounted state and state-action occupancies and value functions.The policy is stochastic and parameterized.Source: §2 · p. 3
  • The setup allows a discount factor of 1 only when the horizon is finite.Source: §2 · p. 3
  • PPO replaces a trust-region constraint with a clipping heuristic, while its practical loss uses critic-based advantage estimates and fits the critic on returns.The cited policy-improvement bound becomes tighter as the target policy approaches the behavioral policy in total variation distance.Source: §2 · p. 4
  • Multiple importance weighting estimates a target-distribution expectation from samples drawn under several behavioral distributions using partitioned weights.Uniform weighting gives samples equal weight; the balance heuristic is presented as a variance-efficient alternative.Source: §2 · p. 4

3 Sample Reuse in PPO

The section extends PPO reuse across a window of recent policies and develops uniform and balance-heuristic-corrected variants. Their lower bounds account for stale advantages and indicate that policies in the reuse window should remain close.

3 Sample Reuse in PPO

  • Cross-iteration reuse retains datasets from up to ω recent iterations, sampling equally across the window rather than using only the latest behavior policy.Past samples retain advantages and returns computed by the critic available when they were collected.Source: §3 · p. 5

3.1 Cross-Iteration Uniform Sample Reuse in PPO

  • ωPPO-U uniformly weights the composite buffer with vanilla importance weights; past samples add actor updates, while critic updates use only the latest dataset.It deliberately keeps stale advantages instead of re-evaluating past data under the current critic.Source: §3.1 · p. 5
  • The ωPPO-U lower bound includes an uncontrollable stale-advantage penalty, suggesting policies in the reuse window should remain close.A separate penalty requires the target policy to stay close to behavioral policies; PPO clipping partially enforces this.Source: §3.1 · pp. 5–6

3.2 Cross-Iteration Balance-Heuristic-Corrected Sample Reuse in PPO

  • ωPPO-BH replaces vanilla weights with ratios against the uniform mixture of policies in the reuse window.It also uses stale advantages and restricts critic fitting to the latest dataset.Source: §3.2 · p. 6
  • BH ratios are bounded by the window size and can clip some fresh samples immediately, while tending to clip a smaller fraction of older samples than ωPPO-U.Its trust region is centered on the policy mixture, and computing the loss requires evaluating samples under all policies in the window.Source: §3.2 · p. 6
  • The ωPPO-BH lower bound has penalties non-decreasing with window size that actor updates cannot control, favoring close policies within the window.The target policy is instead constrained relative to the mixture policy.Source: §3.2 · pp. 6–7

4 Experiments

The experiments compare recent-sample reuse with PPO under fixed-mini-batch-size and fixed-mini-batch-count settings. Reuse generally improves final performance at moderate window sizes, while larger windows can reduce its benefits and neither weighting scheme consistently dominates.

Fixed NB

  • With fixed mini-batch counts, reuse benefits all four environments, most visibly at ω = 4, while ω = 8 yields diminishing returns.All methods perform the same number of updates per iteration, and each update sees the same amount of fresh data.Source: Table 1 · p. 8, Figure 2 · p. 8, §4 · p. 8

Fixed B

  • At ω = 4, both reuse variants consistently outperform PPO in final performance, while PPO with matched extra epochs collapses in almost every setting.The comparison indicates that the reuse benefit is not merely due to performing more updates.Source: §4 · p. 7, Table 1 · p. 8

ωPPO-U vs. ωPPO-BH

  • Neither weighting scheme consistently dominates: ωPPO-BH clips fewer older samples, while ωPPO-U fully uses fresh samples but clips more older ones.Clip diagnostics show lower pre-optimization clipping for older windows under ωPPO-BH, whereas ωPPO-U's clip fraction grows with window index.Source: §4 · pp. 8–9, Figure 3 · p. 9

Main Takeaways

  • Reuse is most effective at moderate window sizes; larger windows can dilute fresh data and increase stale-advantage and policy-diversity effects.In the fixed-B Hopper-v5 case, the authors conjecture these factors hinder training at ω = 8.Source: §4 · pp. 8–10

5 Conclusion

The paper studies recent-sample reuse in PPO while preserving its core mechanics, proposing uniform- and balance-heuristic-weighted variants and deriving policy improvement lower bounds for both. It concludes that moderate reuse improves final performance across diverse control tasks, whereas excessive reuse can be harmful.

5 Conclusion

  • The study isolates data reuse in PPO using ωPPO-U and ωPPO-BH, and derives policy improvement lower bounds for both variants.The bounds reveal penalties uncontrollable by actor updates, suggesting that policies in the reuse window should remain close.Source: §5 · p. 10
  • Moderate data reuse consistently improves final performance across diverse control tasks, but excessive reuse can be harmful.Neither variant consistently dominates; ωPPO-BH may exploit older samples better but can clip some fresh data.Source: §5 · p. 10
  • The analysis should be extended to other deep on-policy methods and should isolate the contributions of advantage re-evaluation, modified clipping, and other reuse heuristics.Source: §5 · p. 10

A Algorithmic Details: PPO and Data Collection

The appendix outlines practical PPO data collection and optimization details, including fixed-length rollouts, advantage estimation, and the joint actor–critic loss. It also describes how rollout termination and truncation are handled.

A Algorithmic Details: PPO and Data Collection

  • The practical PPO setup uses a common implementation and collects fixed-length rollouts, bootstrapping from the last state when a rollout truncates without reaching an absorbing state.A termination flag distinguishes absorbing-state termination from truncation.Source: A Algorithmic Details: PPO and Data… · p. 15, §A.1 · p. 15

A.1 On-Policy Data Collection

  • On-policy data collection stores state-action pairs, rewards, value estimates, and action probabilities, then computes advantages and return targets using GAE.GAE forms an exponentially weighted sum of TD errors; λ controls its bias–variance trade-off, and the sum is truncated at the rollout boundary.Source: §A.1 · pp. 15–16

A.2 PPO Details

  • PPO details define a joint actor–critic loss and describe sample-based optimization using batch advantages and return-to-go estimates.The appendix then gives parameter updates for both actor and critic.Source: §A.2 · pp. 16–17

B Proofs

The proof appendix presents policy improvement lower-bound results, including a bound for a pair of policies and a final bound for the reuse setting. The supplied passages identify these results but do not expose their full formulas.

Policy improvement bounds

  • Lemma B.2 is identified as a policy improvement lower bound for a pair of policies.Source: B Proofs · p. 17

Final bound

  • The proof develops a bound for a set of policies under a distribution over that set, then gives a final policy improvement bound for the reuse setting.The supplied passages identify the theorem and final bound but do not state their full expressions.Source: B Proofs · pp. 19, 20, 25

C Experimental Details

The appendix documents the shared experimental setup and implementation, then defines three comparisons designed to isolate different effects of reusing recent data in PPO.

C Experimental Details

  • Experiments use six MuJoCo continuous-control environments and compare reuse variants against tuned PPO hyperparameters, with Hopper-v5 tuned manually.The evaluation uses 10 shared-seed runs, deterministic-policy returns every 10,000 environment steps, and 50 episodes per evaluation.Source: C Experimental Details · p. 25
  • Reuse variants retain the most recent ω rollouts, sample balanced mini-batches from the buffer, and use per-policy or balance-heuristic-corrected ratios in the actor loss.ωPPO-BH also evaluates stored samples under past policies to form its mixture denominator; other implementation details are shared with PPO.Source: C Experimental Details · p. 25
  • Distribution-shift diagnostics track clipping frequency and normalized effective sample size for each reuse window before optimization.The diagnostics are averaged across iterations and reported as means with 95% confidence intervals over 10 runs.Source: C Experimental Details · pp. 26–27

D Additional Experiments

  • The appendix's broader campaign is designed to assess when and how sample reuse benefits PPO under fair comparisons.It includes update accounting for both PPO and the reuse variants.Source: D Additional Experiments · pp. 27–28
  • The appendix compares fixed mini-batch size, fixed mini-batch count, and reduced data collection to isolate update-count, batch-composition, and fresh-data effects.The reduced-collection setting keeps PPO's batch and update counts but collects only H/ω new steps per iteration.Source: D Additional Experiments · p. 28

D.1 Fixed Mini-Batch Size and Increased PPO Epochs Settings

With fixed mini-batch size, reuse often improves final performance, but its effects depend on the environment and window size. Update-matched PPO controls show that gains are not simply due to more gradient updates.

D.1 Fixed Mini-Batch Size and Increased PPO Epochs Settings

  • Reacher-v5 improves with reuse from ω=2, with most gains at ω=4; ωPPO-BH performs best across windows, while update-matched PPO becomes unstable at larger windows.The passage attributes the gains to additional data, not just additional updates.Source: §D.1 · p. 29
  • In Swimmer-v5, reuse at ω=4 and ω=8 reaches higher final performance while update-matched PPO saturates early, without faster convergence as the window grows.At ω=2, improvement is mild and its advantage over the update-matched baseline is not clearly separable.Source: §D.1 · p. 29
  • In Hopper-v5, reuse helps at ω=2 and ω=4, but both variants degrade significantly at ω=8, showing that excessive reuse can harm performance.The authors conjecture that many updates, stale advantages, and a low fresh-sample share compound at ω=8.Source: §D.1 · p. 29
  • Reuse improves final performance in most tested cases, but does not always accelerate convergence; neither variant consistently dominates, and the best window depends on the environment.The comparison includes a PPO baseline with epochs increased to match the reuse variants' update count.Source: §D.1 · p. 29, Table 4 · p. 30
  • In Walker2d-v5, reuse is beneficial especially at ω=2 and ω=4, while update-matched PPO collapses at ω=4 and ω=8.The passage reports no clear advantage for balance-heuristic correction over uniform weighting.Source: §D.1 · p. 30
  • In HalfCheetah-v5, reuse yields higher final performance at every tested window, with little additional gain from ω=4 to ω=8.The reuse variants perform comparably; update-matched PPO learns faster initially but finishes lower and is less stable.Source: §D.1 · p. 30
  • In Ant-v5, reuse does not clearly improve final performance, but at ω=4 and ω=8 it reaches that level sooner than update-matched PPO, which diverges.At ω=2, the update-matched baseline has comparable convergence speed without instability.Source: §D.1 · p. 30

D.2 Fixed Mini-Batch Count Setting

With fixed mini-batch count, reuse improves performance across the tested environments, typically with the strongest trade-off at ω=4. Ant-v5 is a notable exception for convergence speed and has only marginal final gains.

D.2 Fixed Mini-Batch Count Setting

  • In Reacher-v5, larger windows improve convergence speed and final performance, with ωPPO-BH slightly better; all methods perform the same number of updates.This supports gains beyond simply increasing gradient steps.Source: §D.2 · pp. 31–32
  • With fixed mini-batch count, reuse benefits PPO in both convergence speed and final performance; ωPPO-BH often has higher final returns, and ω=4 offers the best trade-off.Larger windows provide diminishing returns across the environments.Source: §D.2 · pp. 30–31, Table 5 · p. 32
  • In the other environments, gains are marginal at ω=2 and most evident at ω=4; at ω=8, Hopper-v5's collapse disappears, though early convergence is hindered.The authors conjecture that stale advantages from older policies contribute to the slower early learning with large windows.Source: §D.2 · p. 32
  • In Ant-v5, reuse does not appear to improve convergence speed and yields only a marginal final-performance gain; the PPO confidence interval slightly overlaps the best variant.ωPPO-BH at ω=4 is the best-performing variant in this environment.Source: §D.2 · p. 32

D.3 Reduced Data Collection Setting

When reuse variants collect only H/ω new samples per iteration, the tested reuse strategy does not help overall; the authors connect this outcome to reduced fresh data for the critic.

D.3 Reduced Data Collection Setting

  • The authors conjecture that less fresh data impairs critic quality and advantage estimates; at ω=8, some tasks have only 64 fresh samples per iteration.For Hopper-v5 at ω=2, both reuse variants still collect 1,024 fresh samples per iteration and retain performance.Source: §D.3 · p. 34

D.4 ωPPO-U vs. ωPPO-BH

The diagnostic comparison tracks clipping and effective sample size (ESS) across reuse windows before optimization. ωPPO-BH has higher, more uniform ESS than ωPPO-U, but can clip some fresh transitions before an update.

D.4 ωPPO-U vs. ωPPO-BH

  • Clip fraction measures how often clipping is active, while normalized ESS measures importance-weight efficiency and falls as the sampling and current-policy distributions diverge.The diagnostics are averaged over training and grouped by window index; they are evaluated before optimization to capture pre-existing distribution shift.Source: §D.4 · pp. 34–36
  • Under ωPPO-U, older windows show higher pre-optimization clip fractions and monotonically decreasing ESS as their importance ratios move farther from 1.Older samples have already been used in previous updates, increasing their distributional shift from the current policy.Source: §D.4 · p. 36
  • ωPPO-BH maintains substantially higher and more uniform ESS across window indices than ωPPO-U, potentially yielding more reliable gradients from older samples.Its balance-heuristic ratios can also place some fresh transitions outside the clipping range before any gradient update.Source: §D.4 · p. 36

D.5 Ablation: Training the Critic on Dk vs. on Dpωkq

The critic-fitting ablation compares training on the full composite buffer with training on the current dataset across three environments and comparison settings. Full-buffer critic fitting generally offers no clear advantage or worsens learning, with task- and setting-specific exceptions.

D.5 Ablation: Training the Critic on Dk vs. on Dpωkq

  • The ablation compares full-composite-buffer versus current-dataset critic fitting in Reacher-v5, Swimmer-v5, and Hopper-v5 across three comparison settings.Table 7 reports averages of the last 10 final-performance values, with 95% confidence intervals across 10 runs.Source: §D.5 · p. 37, Table 7 · p. 39
  • Full-buffer critic fitting could be treated as another hyperparameter when seeking the best configuration, but outcomes depend on the task, starting hyperparameters, and comparison setting.Source: §D.5 · p. 37
  • In the fixed B setting, full-buffer fitting is nearly identical at ω = 2, then worsens or leaves performance unchanged as ω increases; Reacher-v5 shows only a negligible final-performance gain.The Reacher-v5 exception is particularly noted for ωPPO-BH.Source: §D.5 · p. 37
  • In the fixed NB setting, Reacher-v5 is comparable across critic-fitting strategies, Swimmer-v5 training is hindered at ω ∈ {4, 8}, and Hopper-v5 improves with full-buffer fitting at every window size..Source: §D.5 · pp. 37–38
  • In the H/ω steps setting, critic-fitting choice has no discernible effect across window sizes, with all methods performing poorly.Source: §D.5 · p. 38
  • As the reuse window grows, full-buffer critic fitting generally has no clear advantage or worsens learning, except in Reacher-v5 across settings and Hopper-v5 in the fixed NB setting.The authors attribute possible harm to bias from stale returns and retain current-dataset critic fitting for the controlled study.Source: §D.5 · p. 38
Loading 2610.01399v1…