Source-linked AI summary
Generalized Proximal Policy Optimization with Sample Reuse
James Queeney, Ioannis Ch. Paschalidis, Christos G. Cassandras
TL;DR
Real-world reinforcement learning needs both stable improvement and efficient use of expensive data, but on-policy stability and off-policy sample reuse introduce different trade-offs. The paper develops GePPO by extending policy-improvement guarantees to off-policy samples and connecting them to PPO clipping. GePPO improves PPO’s sample efficiency and is supported by theoretical and empirical evidence on continuous-control tasks.
Problem
Real-world decision making requires stable training performance and efficient use of limited data, while existing methods often emphasize one goal over the other.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
GePPO offers a more reliable approach to sample reuse than standard off-policy algorithms without approximate policy-improvement guarantees.
Source: §9 · p. 9Because GePPO relies on PPO’s clipping heuristic, its policy-improvement guarantees are only approximate.
Source: §9 · p. 10
Abstract
from arXiv · showhide
In real-world decision making tasks, it is critical for data-driven reinforcement learning methods to be both stable and sample efficient. On-policy methods typically generate reliable policy improvement throughout training, while off-policy methods make more efficient use of data through sample reuse. In this work, we combine the theoretically supported stability benefits of on-policy algorithms with the sample efficiency of off-policy algorithms. We develop policy improvement guarantees that are suitable for the off-policy setting, and connect these bounds to the clipping mechanism used in Proximal Policy Optimization. This motivates an off-policy version of the popular algorithm that we call Generalized Proximal Policy Optimization with Sample Reuse. We demonstrate both theoretically and empirically that our algorithm delivers improved performance by effectively balancing the competing goals of stability and sample efficiency.
1 Introduction
The paper targets the tension between stable policy improvement and efficient reuse of limited reinforcement-learning data. It proposes GePPO, an off-policy PPO variant supported by generalized improvement guarantees and evaluated on continuous-control tasks.
1 Introduction
- Off-policy methods improve data efficiency through replay buffers but distribution shift can invalidate on-policy guarantees and destabilize training.Existing methods may therefore require implementation tricks and extensive hyperparameter tuning.Source: §1 · p. 2
- GePPO combines PPO’s theoretically supported stability with off-policy sample reuse to address the competing demands of real-world decision making.Real-world data collection can be difficult and expensive, while stable performance is required throughout training.Source: §1 · pp. 1–2
- The paper extends policy-improvement guarantees to off-policy data from all recent policies.This provides the theoretical basis for reusing samples beyond the current policy.Source: §1 · p. 2
- The authors connect PPO clipping to the penalty term in the generalized lower bound and use that connection to motivate generalized clipping and adaptive learning rates.The adaptive learning-rate method is designed to connect the theoretical penalty with practical updates.Source: §1 · p. 2
- Theoretical evidence and experiments on high-dimensional continuous-control tasks in OpenAI Gym’s MuJoCo environments demonstrate strong performance while balancing stability and sample efficiency.The reported evaluation covers high-dimensional continuous-control tasks rather than real-world deployments.Source: §1 · p. 2
2 Related work
Related work develops policy-improvement methods through trust regions or clipping and improves sample efficiency through replay-based off-policy learning. GePPO differs by directly accounting for off-policy distribution shift within policy-improvement guarantees.
On-policy policy improvement methods
- TRPO and PPO use policy-improvement theory with different constraints: TRPO enforces a KL-divergence trust region, whereas PPO clips the probability ratio.Both methods are presented as on-policy approaches grounded in approximate policy improvement.Source: §2 · p. 2
- Prior analyses modify PPO clipping using KL-divergence trust regions, while this work relates clipping directly to total variation distance and uses adaptive learning rates for a similar control goal.The related methods include rollback operations that keep probability ratios close to the clipping range.Source: §2 · p. 2
Sample efficiency with off-policy data
- DDPG, TD3, and SAC improve sample efficiency by replaying data from prior policies, but they are not motivated by policy-improvement guarantees and do not directly control off-policy bias.Their common mechanism is storing samples in a replay buffer for policy updates.Source: §2 · p. 2
- Hybrid policy-gradient methods seek to balance on-policy variance and off-policy bias, with some relating bias to KL divergence or controlling it through trust regions.These approaches are related to GePPO’s penalty term, which can be bounded using KL divergence.Source: §2 · pp. 2–3
- Heuristic attempts to add off-policy data to PPO do not account for the distribution shift that invalidates PPO’s theoretical support.GePPO is distinguished by incorporating that shift into its principled policy-improvement treatment.Source: §2 · p. 3
3 Preliminaries
The preliminaries define the reinforcement-learning setting and policy-improvement framework underlying PPO. PPO uses clipped probability ratios to approximate a lower-bound objective, but its stable improvement comes with substantial on-policy sampling requirements.
Reinforcement learning framework
- The paper formulates an infinite-horizon discounted MDP with states, actions, transitions, rewards, an initial-state distribution, and discount rate.Policies induce discounted state and state-action visitation distributions used in subsequent objectives and bounds.Source: §3 · p. 3
- The policy objective maximizes expected discounted rewards, with value, action-value, and advantage functions defining the quantities used for policy improvement.The advantage function is the action value minus the state value.Source: §3 · p. 3
Policy improvement lower bound
- The policy-improvement lower bound decomposes into a surrogate objective and penalty term that can guarantee improvement when the next policy maximizes the bound.For the standard bound, expectations can be approximated using samples generated by the current policy.Source: §3 · p. 3
Proximal Policy Optimization
- PPO maximizes the surrogate objective while constraining the next policy through clipped probability ratios, making the clipped objective a lower bound to the surrogate objective.The ratio π(a|s)/πk(a|s) is discouraged from leaving [1 − ϵ, 1 + ϵ], and the objective is optimized with minibatch stochastic gradient ascent.Source: §3 · p. 3
- PPO provides stable improvement for sufficiently small learning rates and sufficiently many samples, but high variance and current-policy data collection can make it sample intensive.Large sample requirements are needed for the empirical objective to accurately estimate the true objective.Source: §3 · p. 4
4 Generalized policy improvement lower bound
The paper generalizes the policy-improvement lower bound so it can use samples from multiple recent policies. The resulting penalty connects off-policy reuse to controlling policy distances across consecutive updates, providing a basis for extending PPO.
4 Generalized policy improvement lower bound
- The generalized lower bound uses samples from the last M policies instead of requiring data exclusively from the current policy πk.This addresses the distribution shift that otherwise invalidates the standard on-policy bound.Source: §4 · p. 4
- Theorem 1 establishes a lower bound for any distribution ν over M prior policies and any future policy π.The theorem is obtained by combining policy-improvement bounds associated with the prior policies.Source: §4 · p. 4
- The generalized bound remains a lower bound between the current and future policies while allowing visitation distributions, probability ratios, and total variation terms to depend on prior policies.Setting M = 1 recovers the standard policy-improvement lower bound.Source: §4 · p. 4
- The penalty can be controlled by managing expected total variation distances between the future policy and recent policies, including distances between consecutive prior policies.The decomposition shows why controlling consecutive policy changes can control expected performance loss at each update.Source: §4 · p. 4
5 Clipping mechanism
The paper interprets PPO clipping as an approximate controller of a total variation distance penalty, then generalizes that mechanism to off-policy data and links it to adaptive learning rates.
5 Clipping mechanism
- PPO clipping approximately bounds the expected total variation distance between the current and future policies by ϵ/2.It removes the incentive for the probability ratio to exceed the clipping range at sampled state-action pairs.Source: §5 · p. 5
- The off-policy policy-improvement penalty can likewise be controlled through an appropriate generalized clipping mechanism.Lemma 3 provides the expected total variation distance used to select this mechanism.Source: §5 · p. 5
- For prior-policy samples, generalized clipping centers each probability ratio at πk(a|s)/πk−i(a|s), while current-policy samples recover standard PPO clipping.This preserves the PPO mechanism when i = 0.Source: §5 · p. 5
- Clipping only approximately controls the penalty when policy-update learning rates are sufficiently small.Large initial gradient steps can move ratios outside the clipping range, and changing sensitivity may require time-varying learning rates.Source: §5 · p. 6
- The adaptive method decreases learning rates when estimated total variation exceeds ϵ/2 and increases them when it is substantially below ϵ/2.This prevents large unstable updates while accelerating learning when updates are too small.Source: §5 · p. 6
6 Algorithm
GePPO extends PPO’s clipped objective to samples from recent policies and updates it with minibatch stochastic gradient ascent plus an adaptive learning rate.
6 Algorithm
- GePPO approximates its generalized objective using samples collected from each of the last M policies.The empirical objective is approximately optimized with minibatch stochastic gradient ascent.Source: §6 · p. 6
- The algorithm updates its learning rate at every iteration using the adaptive method tied to the policy-improvement penalty.This gives GePPO a principled route to improving PPO sample efficiency while retaining approximate policy-improvement guarantees.Source: §6 · p. 6
7 Sample efficiency analysis
The analysis characterizes how GePPO trades update size against reused data, showing improved stability–sample-efficiency trade-offs under uniform policy weights.
7 Sample efficiency analysis
- GePPO must make smaller policy updates than PPO in total variation distance when it uses prior-policy samples.The additional data increase the batch size for objective estimation, enabling more frequent updates and faster, more stable learning.Source: §7 · p. 6
- GePPO reuses n samples from each of πk−i for i = 0, . . . , M −1 and optimizes the resulting empirical objective with minibatch stochastic gradient ascent.The algorithm uses adaptive learning-rate parameters η, α, and β alongside the reused samples.Source: §7 · p. 7
end
- With M = B, GePPO increases the policy’s total variation change by 2B/(B+1) while using the same samples per update as PPO.This is the first stability–sample-efficiency operating point established by Theorem 2.Source: end · p. 7
- With M = 2B −1, GePPO increases samples per update by (2B−1)/B while preserving PPO’s total variation change.This is the alternative operating point established by Theorem 3.Source: end · p. 7
- For uniform policy weights, GePPO improves PPO’s stability–sample-efficiency trade-off for every B ≤M ≤2B −1.The benefit increases with B, the number of batches needed to estimate the true objective accurately.Source: end · p. 7
8 Experiments
Experiments on six MuJoCo locomotion tasks compare GePPO with PPO, focusing on final performance, sample efficiency, and policy-update stability. GePPO improves performance and sample use while adaptive learning rates control instability across network choices.
8 Experiments
- The experiments use six MuJoCo locomotion environments, five random seeds, and one million training steps per experiment.The evaluated tasks are Swimmer-v3, Hopper-v3, HalfCheetah-v3, Walker2d-v3, Ant-v3, and Humanoid-v3.Source: §8 · pp. 8–9
- GePPO improves final performance across all evaluated MuJoCo environments and improves average training performance by 8%–65% over PPO.The comparison covers six continuous-control locomotion tasks, with Table 1 and Figure 1 reporting the evaluation scope.Source: §8 · pp. 8–9, Table 1 · p. 8
- GePPO reaches PPO’s final performance using 15%–77% fewer samples, demonstrating improved sample efficiency despite PPO’s stable default small-batch training.Source: §8 · p. 9
- GePPO keeps consecutive-policy total variation distance near its clipping target through adaptive learning rates, whereas PPO’s update magnitude increases during training.Source: §8 · p. 9
- On Walker2d-v3, PPO’s total variation change is almost 40% above its target after one million steps, and wide networks exacerbate this instability.With wide networks, PPO performance declines over time because excessively large policy updates render clipping ineffective; GePPO remains stable under the same implementation change.Source: §8 · p. 9, Figure 2 · p. 9
9 Conclusion
The conclusion presents GePPO as a theoretically supported way to reuse off-policy samples in PPO, while the appendix derives the off-policy improvement bounds and their policy-update consequences. These results connect clipping, total variation penalties, and sample reuse, but the method retains approximation and scope limitations.
9 Conclusion
- GePPO incorporates off-policy samples into PPO using a novel off-policy policy-improvement lower bound and is presented as a more reliable approach to sample reuse.The conclusion frames this as a step toward stable, sample-efficient reinforcement learning for high-stakes decision making.Source: §9 · p. 9
- The method’s guarantees are approximate because clipping is heuristic, the penalty constant may be too large for practical guarantees, and PPO may not be the best stability–sample-efficiency starting point.Source: §9 · p. 10
A.1 Proof of Theorem 1
- Theorem 1 combines lower bounds from the last M policies using arbitrary policy weights, under support-containment assumptions for the future policy.The proof extends the bound to an arbitrary reference policy using state-visitation total variation bounds and derives the result by convex combination.Source: §A.1 · pp. 12–13
A.2 Proof of Lemma 2
- The proofs of Lemmas 2 and 3 use total variation definitions and support-containment assumptions to relate policy and reference-policy ratios.Source: §A.2 · p. 13, §A.3 · p. 14
A.4 Proof of Lemma 4
- The proof compares PPO’s and GePPO’s approximations of the policy-improvement penalty and concludes that their worst-case expected performance loss at every update is the same.For GePPO, the penalty is bounded using the triangle inequality for total variation distances, with each expected distance approximately bounded by ϵ_GePPO/2.Source: §A.4 · p. 14
A.5 Proof of Theorem 2
- With M = B and uniform weights, GePPO makes B updates per N samples and increases total policy change by 2B/(B+1) relative to PPO.The comparison uses one PPO update per N = Bn collected samples versus B GePPO updates using reused data.Source: §A.5 · p. 15
A.6 Proof of Theorem 3
- With M = 2B − 1, GePPO uses (2B − 1)n samples per update, increasing each update’s sample size by (2B − 1)/B while preserving PPO’s overall total variation change.Source: §A.6 · p. 15
B Optimal policy weights
GePPO optimizes non-uniform policy weights to improve either policy-change magnitude or effective sample size under constraints matching PPO’s sample size or stability. The resulting convex programs can be solved once before training and yield improvements over uniform-weight baselines.
B Optimal policy weights
- For non-uniform weights, effective sample size is smaller than the true sample count Mn unless ν is uniform.Effective sample size is defined as the number of uniformly weighted samples producing the same variance level.Source: B Optimal policy weights · p. 15
- Theorem 4 optimizes policy weights to maximize GePPO’s total variation change while maintaining PPO’s effective sample size N = Bn.The optimization minimizes the denominator governing total variation change while constraining effective sample size to at least N = Bn and ensuring ν is a distribution.Source: B Optimal policy weights · pp. 15–16
- With B = 2, Figure 3 compares Theorem 4 and Theorem 5’s optimal weights against the corresponding uniform weights from Theorems 2 and 3.The left panel compares Theorems 4 and 2; the right panel compares Theorems 5 and 3.Source: Figure 3 · p. 16
- Theorem 4 increases total variation distance relative to Theorem 2, while Theorem 5 increases effective sample size relative to Theorem 3.The uniform weights in Theorems 2 and 3 are feasible solutions to the respective optimization problems.Source: B Optimal policy weights · pp. 16–17
- Theorem 5 optimizes policy weights to maximize effective sample size while maintaining PPO’s total variation change throughout training.The optimization requires M̄ ≥ 2B − 1 and constrains the total variation change to equal ϵPPO/2.Source: B Optimal policy weights · p. 16
- The optimal policy weights can be computed efficiently by solving the convex optimization problems once at the beginning of training.The optimized weights determine both the optimal number of prior policies M and the corresponding ν when M̄ is chosen sufficiently large.Source: B Optimal policy weights · p. 17
C Implementation details
The experiments use PPO-derived neural-network settings and configure GePPO for off-policy estimation, adaptive learning rates, and controlled policy changes. Results compare training performance and total variation changes across MuJoCo tasks.
C Implementation details
- GePPO uses Gaussian policies and separate value networks, each with two hidden layers of 64 tanh units, standardized observations, and minibatch stochastic-gradient updates.The policy mean and value function use separate networks, while the policy standard deviation is state-independent.Source: C Implementation details · p. 17
- With PPO batch size N = 2,048 and GePPO minimum batch size n = 1,024, the experiments use B = 2, calculate weights via Theorem 5, and set ϵGePPO = 0.1.The PPO clipping parameter is ϵPPO = 0.2; the adaptive-learning-rate factors α and β are listed in Table 2.Source: C Implementation details · p. 18
- GePPO uses V-trace to correct the bias of multi-step advantage estimates computed from prior-policy data, with truncated importance sampling controlling variance.The experiments use truncation parameter c̄ = 1.0 and then form a weighted average of corrected multi-step estimates as in GAE.Source: C Implementation details · p. 18
- Off-policy advantage standardization centers the starting point of policy updates, generalizing PPO’s standardization where initial probability ratios equal one.The relevant quantity has zero expectation under samples from prior policies.Source: C Implementation details · p. 18
- GePPO takes approximately 4 hours per environment and seed versus 3 hours for PPO because it performs twice as many updates, while improving performance for fixed samples and wall-clock time on average.These timings use unoptimized code and the stated Windows, CPU, RAM, and GPU setup.Source: C Implementation details · p. 18
- Figure 4 reports training performance and Figure 5 reports average total variation change per update across MuJoCo tasks using the default two-layer 64-unit policy network.Both figures use shading equal to half a standard error and include PPO-Adapt as a comparison variant; Figure 5 also marks PPO and GePPO target distances.Source: Figure 4 · p. 19, Figure 5 · p. 19
D Additional experimental results
Additional experiments evaluate GePPO across MuJoCo tasks with default and wide policy networks, comparing it with PPO and PPO-Adapt. GePPO improves average performance, while adaptive learning rates help stabilize policy updates, especially for wide networks.
Default policy network
- GePPO improves average performance throughout training by 31% versus PPO and 32% versus PPO-Adapt with the default policy network.These experiments use the default two-hidden-layer policy network with 64 units per layer.Source: D Additional experimental results · pp. 19–20
- With the default network, PPO’s policy-update change increases throughout training, while adaptive learning rates keep it near the clipping-determined target.PPO-Adapt has comparable performance in this well-tuned setting, whereas GePPO outperforms it, indicating that sample reuse contributes materially to performance.Source: D Additional experimental results · pp. 19–20
Wide policy network
- GePPO improves average performance throughout training by 63% versus PPO and 34% versus PPO-Adapt with the wide policy network.The wide network has two hidden layers of 400 and 300 units, respectively; Figures 6 and 7 report performance and policy-update changes across MuJoCo tasks.Source: Figure 6 · p. 20, Figure 7 · p. 20, D Additional experimental results · p. 21
- With the wide network, PPO’s clipping becomes ineffective as probability ratios grow more sensitive to updates, producing large policy updates and poor or declining performance.PPO-Adapt controls this instability and improves performance throughout training, except on Humanoid-v3, where PPO performs better.Source: D Additional experimental results · p. 21
Interpretation
- Principled sample reuse drives performance gains, while adaptive learning rates provide stability robust to hyperparameter settings.Without adaptation, strong performance remains possible but requires careful learning-rate tuning across environment, network, training length, and other settings.Source: D Additional experimental results · p. 21