PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Exploration and Loss of Plasticity in On-Policy Reinforcement Learning

Andrea Vasco Grieco, Evelyn D'Elia, Edoardo Caldarelli, Raffaello Camoriano

arXiv:2610.36309v1cs.LGcs.RO

TL;DR

The paper addresses limited understanding of LoP in PPO with Gaussian policies and examines policy standard deviation σ as a possible link. It combines theoretical bounds with benchmark and robotic experiments, reporting that σ collapse accompanies LoP-related metric changes and adaptation effects.

  • Problem

    The causes and geometric conditions of LoP remain unclear, and prior PPO-specific study of LoP focused on discrete actions.

    Source: §1 · p. 1

  • Method

    The paper derives and numerically checks bounds connecting σ to LoP metrics in PPO with Gaussian policies, then evaluates the connection in benchmarks and robotic task adaptation.

    Source: §1 · p. 2

  • Results

    As σ →0, PPO gradient norm and Hessian sharpness diverge while Hessian approximate rank collapses, and experiments including robotic deployment provide evidence linking σ collapse to LoP.

    Source: §5 · p. 8

  • Takeaways & Limitations

    In PPO with Gaussian policies, σ is a contributor to observed LoP and offers an explanation for the effectiveness of some existing mitigation techniques.

    Source: §2 · p. 2, §1 · p. 2

    The paper leaves whether policy entropy affects LoP in other state-of-the-art RL methods and how it relates to catastrophic forgetting for future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

Deep reinforcement learning (RL) algorithms have been observed to lose the ability to adaptively learn from new data, a phenomenon termed Loss of Plasticity (LoP). While many metrics have been shown to correlate with it, the causal mechanisms and the geometric conditions from which it can arise have not been clearly identified. We analyze LoP in Proximal Policy Optimization with isotropic Gaussian policies, the state-of-the-art formulation for many continuous action spaces in RL. We report multiple metrics which characterize the effect that the policy standard deviation $\sigma$ has on LoP, and we also suggest related explanations for the effectiveness of widely adopted LoP mitigation techniques. We observe a significant effect of $\sigma$ on the extent of LoP, and accompany this with both theoretical and experimental supporting arguments. Our experimental validation consists of LoP benchmark evaluations against standard baselines as well as real-world policy deployment on a 7-DoF robotic arm.

1 INTRODUCTION

PPO is a widely used on-policy method, but its theory is limited, especially for Gaussian policies in continuous action spaces. The paper studies whether policy standard deviation σ links to Loss of Plasticity (LoP), alongside existing explanations and mitigation strategies.

1 INTRODUCTION

  • PPO theory remains limited; prior analysis notes that gradients can scale inversely with σ, producing large updates and instability.Existing theoretical work largely studies vanilla policy-gradient methods, including increased gradient variance as σ →0.Source: §1 · p. 1
  • LoP is diminished ability to learn new objectives after training, with proposed explanations including representation richness, loss-landscape sharpening, dormant neurons, and gradient norms.Prior PPO-specific LoP work focused on discrete actions.Source: §1 · p. 1
  • Prior mitigation strategies include changing Adam moment coefficients or partially resetting optimizer parameters, while stronger entropy regularization has improved adaptation under non-stationarity.A separate proposal describes LoP manifolds that can trap network weights under gradient descent.Source: §1 · pp. 1–2

Contributions

  • The paper formally and experimentally links policy standard deviation σ to LoP in PPO with Gaussian policies, including benchmark and real-world robotic task evaluations.It reports that as σ →0, gradient norm and Hessian sharpness diverge while Hessian approximate rank collapses.Source: §1 · p. 2
  • The paper focuses on PPO with Gaussian policies because LoP has no consensus explanation and proposed causes have counter-examples.Its stated contribution is to examine σ as a link among LoP metrics and existing mitigation techniques.Source: §1 · p. 2

2 BACKGROUND

The background defines the episodic decision setting and PPO formulation used in the paper, then frames LoP as diminished ability to fit new objectives. It identifies gradient and Hessian measures as candidate correlates that the paper relates to σ collapse.

Markov Decision Processes

  • The episodic MDP is specified by state and action spaces, horizon, transition probabilities, reward function, and initial-state distribution.The action space has dimensionality n.Source: §2 · p. 2
  • Discounted cumulative reward defines the value function as expected return from a state and the state-action value as expected return from a state-action pair.Both expectations are under the policy.Source: §2 · p. 2
  • The policy assumption is a deep isotropic Gaussian with neural-network mean μ_ϕ(s) and standard deviation σ, sampling actions from N(μ_ϕ(s), σ^2I).The policy parameters are θ := (ϕ, σ).Source: §2 · p. 2

Proximal Policy Optimization

  • PPO optimizes policy parameters using the previous policy’s density ratio and an estimated advantage; the paper uses generalized advantage estimation.The cited passages define the ratio and advantage estimate but do not include the full objective.Source: §2 · p. 2

Probability measures

  • A policy induces distributions over states and state-action pairs across a finite horizon, and the analysis assumes an event condition for gradients to be defined almost everywhere.The supplied passages state the event condition only partially.Source: §2 · p. 2

Loss of Plasticity

  • LoP is diminished ability of neural networks to fit new objectives after training on a different goal; proposed correlates include gradient magnitude, Hessian sharpness, and Hessian stable rank.The paper focuses on linking these factors to collapse of σ in PPO with Gaussian policies.Source: §2 · p. 2

3 THEORETICAL ANALYSIS

The theoretical analysis links shrinking Gaussian-policy standard deviation σ to three LoP-associated geometric changes: diverging gradient norms and Hessian sharpness, and collapsing Hessian approximate rank. It also derives implications for learning-rate scaling and optimizer behavior.

3.1 Divergence of the gradient norm

  • As σ →0, the expected minibatch gradient norm is bounded below at scale σ^-1 times the expected squared Frobenius norm of the policy-mean Jacobian, when σ_old ≈σ.The analysis also notes that reduced exploration can account for behavior typically associated with LoP.Source: §3.1 · p. 3

3.2 Divergence of the sharpness

  • As σ →0, the largest Hessian eigenvalue diverges proportionally to σ^-2 in expectation under the stated conditions.When σ ≈σ_old, C_θ = o(σ^-2), and lim inf χ_θ > 0, the sharpness is expected to scale as σ^-2.Source: §3.2 · pp. 3–4

3.3 Collapse of the Hessian approximate rank

  • As σ →0, at most n × N empirical-Hessian eigenvalues scale as 1/σ^2, reducing the empirical matrix rank for action dimension n and batch size N.The result connects the rank reduction to the Hessian approximate-rank analysis.Source: §3.3 · p. 4

3.4 Results interpretation and consequences

  • The gradient and sharpness results motivate the heuristic learning-rate scaling η_t = η_0 σ^2 / ∥ḡ(B)∥ for SGD, without gradient normalization for Adam or RMSProp.The same scaling was proposed in Safe Policy Gradients; the analysis also motivates why setting Adam’s β1 = β2 reduces step size and why update-normalizing optimizers can avoid effective step-size blow-up if σ collapses smoothly enough.Source: §3.4 · p. 4

4 EXPERIMENTS

Experiments validate the theoretical predictions, examine σ’s role in LoP mitigation and task adaptation, and test policy transfer to a real robot. Across these settings, results link σ management to reward collapse and improved adaptation.

4.1 Theoretical validation

  • The fitted exponents are slightly above α1 = 1 and near α2 = 2, consistent with the theoretical bounds; κ̃θ and K̃θ neither diverge nor collapse to zero.Mean R2 is 0.6409 for α1 and 0.8728 for α2.Source: §4.1 · p. 5, Figure 1 · p. 6, Figure 2 · p. 6

4 EXPERIMENTS

  • The experiments test σ’s effect on adaptation using non-stationary Slippery Ant and a simulated joint-lock perturbation, followed by deployment on a 7-DoF Franka Emika Panda.The theoretical validation also uses Isaac-Reach-Franka-v0 policies with periodic σ resets.Source: §4.2 · p. 5, §4 · p. 5, §4.3 · p. 6

4.2 The unreported role of σ in a common benchmark

  • Slippery Ant results associate σ collapse with reward collapse; L2 and L2+CBP show stronger σ regularization and no reward collapse.Reducing the learning rate avoids asymptotic collapse but does not restore adaptation; matching σ’s evolution to L2 yields comparable results.Source: §4.2 · p. 6, Figure 3 · p. 7, Figure 4 · p. 7
  • Matching σ to L2 yields comparable deterministic-policy rewards, while CBP underperforms by 20%; L2 and L2+CBP show no collapse.The results identify σ as a confounding factor in Slippery Ant comparisons.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3.1 Results in simulation

  • Resetting σ or strengthening entropy regularization provides a strong advantage in deterministic-policy rewards after the simulated joint-lock perturbation.The evaluation uses 30 seeds.Source: Figure 6 · p. 7, §4.3.1 · p. 7

4.3.2 Deployment on the real robot

  • Real-world joint-lock tests show a statistically significant reward increase from approximately -0.01 to 0.045, with lower reward variability after σ reset.The comparison uses selected median-performing seeds and 50 target points per method.Source: §4.3.2 · p. 8, Figure 8 · p. 8

5 CONCLUSIONS

The paper connects Gaussian-policy standard deviation σ to Loss of Plasticity (LoP), combining theoretical bounds with numerical and experimental evidence. It identifies future questions about policy entropy beyond the studied methods and its relationship to catastrophic forgetting.

5 CONCLUSIONS

  • The derived bounds show that as σ →0, the gradient norm and Hessian sharpness diverge while the Hessian approximate rank collapses.The paper reports numerical verification of these results.Source: §5 · p. 8
  • Experimental evidence, including a real-world robotic task, supports that LoP can be caused by σ →0.Source: §5 · p. 8
  • Future work will test whether policy entropy affects LoP in other state-of-the-art RL methods and examine its relationship with catastrophic forgetting.Source: §5 · p. 8

A Useful results

The appendix presents derivations for the gradient and Hessian of PPO’s clipped loss, alongside a lemma used in the proofs.

A Useful results

  • The appendix provides an explicit derivation of the gradient and Hessian of L(θ), including CLIP and a lemma used in the proofs.Source: A Useful results · p. 12

A.1 Gradient of the clipped loss

  • The gradient derivation defines V(ε) using policy density ratios and advantage signs, and assumes actions are sampled from the old Gaussian policy during PPO updates.Source: §A.1 · pp. 12–13

A.2 Hessian of the clipped loss

  • The Hessian derivation expresses the density-ratio-weighted advantage term using the policy log-density gradient and Hessian.Source: §A.2 · p. 13

A.3 A useful lemma

  • A useful lemma for the theorem proofs applies Paley–Zygmund’s inequality under bounded first and second moments of a nonnegative random variable.Source: §A.3 · p. 14

B PROOF OF THEOREM 3.1

The proof of Theorem 3.1 uses assumptions on Gaussian policies and the clipped objective to bound minibatch gradient behavior. It invokes a lemma and i.i.d. minibatch samples to obtain the desired result.

B PROOF OF THEOREM 3.1

  • The proof assumes Gaussian policies, a finite ratio, and that the clipped objective and policy log-gradient are defined almost everywhere with well-defined expectations.Source: B PROOF OF THEOREM 3.1 · p. 15
  • Conditioning on the state yields a normally distributed auxiliary variable, which is used in direct calculations for the required inequality.Source: B PROOF OF THEOREM 3.1 · p. 15
  • The proof combines the intermediate result with Lemma A.2 and applies it to the minibatch gradient norm to obtain the theorem’s desired result.Source: B PROOF OF THEOREM 3.1 · p. 16

C PROOF OF THEOREM 3.2

The proof of Theorem 3.2 uses a Gaussian projection lemma and bounds involving Hessian norms and eigenvalues. It averages over the auxiliary variable and states the resulting inequality.

C PROOF OF THEOREM 3.2

  • Lemma C.1 states that projecting an isotropic Gaussian vector onto a unit vector produces a normally distributed scalar.Source: C PROOF OF THEOREM 3.2 · p. 16
  • It bounds the expected operator norm of the Hessian using the stated norm inequality and assumptions, then averages over states to obtain the desired inequality.Source: C PROOF OF THEOREM 3.2 · pp. 17–18
  • The proof uses the Rayleigh quotient and the projected Gaussian to relate eigenvalues of Gϕ(s, ξ) to a scalar Gaussian calculation.Source: C PROOF OF THEOREM 3.2 · p. 17

D PROOF OF THEOREM 3.4

The proof shows how the eigenvalue and singular-value behavior of the Hessian relates to M1 as σ approaches zero, and bounds the number of rapidly scaling eigenvalues and the stable rank.

D PROOF OF THEOREM 3.4

  • The Hessian eigenvalues scale with 1/σ^2 precisely for nonzero eigenvalues of M1, which is symmetric and nonzero and therefore has at least one such eigenvalue.Source: D PROOF OF THEOREM 3.4 · p. 18
  • As σ → 0, the i-th singular value of ¯Hθ(B) is asymptotic to 1/(Nσ²) times the i-th singular value of M1.Source: D PROOF OF THEOREM 3.4 · p. 19

E IMPLEMENTATION DETAILS

E.1 Training setup for Slippery Ant

The Slippery Ant setup modifies the Mujoco Ant task with periodically resampled friction and specifies its reward, observation and action spaces, and PPO variants.

E.1 Training setup for Slippery Ant

  • Every 2 million steps, Slippery Ant resamples friction log-uniformly from [0.02, 2.00] while the quadruped learns to walk forward.The task has been used to study loss of plasticity in deep RL [9].Source: §E.1 · p. 19
  • The 111-dimensional observation contains 27 body-position and relative-angle values plus 84 external-force values, and the 8-dimensional action specifies joint torques.Source: §E.1 · p. 20
  • The evaluated optimizer variants are standard PPO, tuned Adam, tuned Adam with L2, and tuned Adam with L2 plus continual backpropagation.The supplied passage specifies the optimizer coefficients and regularization settings for these variants.Source: §E.1 · p. 20

E.2 Training setup for Isaac-Reach-Franka-v0

Isaac-Reach-Franka-v0 trains a policy to move a robotic arm’s end-effector to target poses while limiting abrupt changes, using parallel simulation and a 7-dimensional continuous action space.

E.2 Training setup for Isaac-Reach-Franka-v0

  • The inverse-kinematics task trains in 4096 parallel environments, with each iteration aggregating 24 timesteps per environment.An iteration therefore comprises 98,304 timesteps, and training applies eight epochs of four mini-batches at iteration end for completed environments.Source: §E.2 · pp. 20–21
  • The reward function includes end-effector position and orientation targets and a penalty related to current joint velocity.Source: §E.2 · p. 21
  • The target pose and orientation are resampled uniformly every 4 seconds, while curriculum weights kl2 and kvel change after 187 iterations.The weights change from 10^-4 to 10^-3 and 5 × 10^-3, respectively.Source: §E.2 · p. 21
  • The policy observes 32 values combining position, velocity, target pose, and previous action, and outputs desired joint positions in a 7-dimensional continuous action space.The desired joint positions are interpreted as angles; the resulting torques use stiffness Kp and damping Kd.Source: §E.2 · p. 21

F ADDITIONAL RESULTS

In Isaac-Reach-Franka-v0, periodically restarting σ significantly affects gradient norm and Hessian sharpness. Cycle-wise fits are consistent with exponents near 1.2 and 2, respectively, while fitted intercepts also rise during training.

F ADDITIONAL RESULTS

  • In Isaac-Reach-Franka-v0, restarting standard deviation σ significantly affects gradient norm and Hessian largest eigenvalue (sharpness), suggesting σ is the main factor affecting these metrics.Source: F ADDITIONAL RESULTS · p. 22
  • The fitted gradient-norm exponent is initially slightly higher before hovering near 1.2, consistent with Theorem 3.1’s exponent-1 bound.Source: F ADDITIONAL RESULTS · p. 22, Table 6 · p. 23
  • The fitted Hessian-sharpness exponent hovers near 2, consistent with Theorem 3.2, while β2 increases tenfold over training.The authors note that factors beyond σ may contribute to this parameter.Source: F ADDITIONAL RESULTS · p. 22, Table 7 · p. 23
  • The gradient-norm exponent hovers near 1.2 and the Hessian-sharpness exponent near 2 across the reported fits, while fitted β1 nearly doubles and β2 increases tenfold.The authors report large R^2 values for both log-linear fits; the β increases suggest additional factors beyond σ.Source: F ADDITIONAL RESULTS · p. 22, Table 6 · p. 23, Table 7 · p. 23
Loading 2610.36309v1…