Robust Policy Optimization in Deep Reinforcement Learning

Problem

• A fixed parametric distribution may inadequately represent different tasks, while reducing variance can hamper exploration throughout training.The paper therefore focuses on both action-distribution expressiveness and exploration.Source: §1 · p. 1

Motivation

• Policy-gradient agents can lose entropy during training, reducing stochasticity and potentially causing premature convergence to suboptimal policies.This is especially concerning in high-dimensional environments, where exploration and representative data collection are difficult.Source: §1 · pp. 1–2

Method

• RPO adds a uniform random perturbation z ∼ U(−α, α) to the Gaussian mean before sampling the action, producing a flatter distribution with broader sample spread.The perturbed mean is µ′ = µ + z, and the action is sampled as a ∼ N(µ′, σ).Source: §1 · p. 3
• The perturbation is generic for parameterized distributions using means and standard deviations, and its state- and parameter-independent randomness helps maintain stochasticity.The paper demonstrates the approach with Gaussian, Laplace, and Gumbel distributions.Source: §1 · p. 3

Results

• Compared with entropy regularization, RPO performs better in most environments and remains improving or stable with more training data, whereas some entropy coefficients worsen performance.The paper also reports that standard PPO can lose performance with extended training in CartPole and BipedalWalker.Source: §1 · p. 3
• RPO consistently outperforms PPO, entropy regularization, Laplace and Gumbel distributions, and data-augmentation methods across 18 tasks from four RL benchmarks.The benchmarks are DeepMind Control, OpenAI Gym, PyBullet, and Nvidia IsaacGym.Source: §1 · p. 3

• An MDP comprises states, actions, transition probabilities, and rewards, while the agent seeks a policy maximizing cumulative reward.Source: §2 · p. 4

Policy Gradient

• PPO directly optimizes cumulative future return with a neural-network policy, using Gaussian parameters for continuous actions and sampling actions from the resulting distribution.Source: §2 · p. 4

Entropy Measure

• Policy entropy measures action uncertainty and stochasticity; higher entropy permits broader state exploration during learning.Source: §2 · p. 4

Action Representation

• Parameterized action distributions provide exploration through variance, but a fixed distribution may not suit every task and shrinking variance can hamper exploration.Source: §2 · p. 4

• Standard PPO becomes less random as training progresses, motivating a method that maintains exploration after the policy has learned.Source: §3 · pp. 4–5
• RPO perturbs Gaussian mean µ with z sampled from U(−α, α), then evaluates actions under N(µ′, σ), where µ′ = µ + z.Source: §3 · p. 5
• The state-independent uniform perturbation can preserve stochasticity even as the state-conditioned Gaussian becomes more confident.Source: §3 · p. 5
• With µ = 0.0, σ = 1.0, and z ∼ U(−3.0, 3.0), Perturb Gaussian samples are less centered than standard-Gaussian samples and should have higher entropy.Source: §3 · p. 5
• The perturbation mechanism also applies to Laplace and Gumbel distributions, although Gaussian results are reported by default.Source: §3 · p. 5

4.1 SETUP

• The evaluation spans diverse continuous-control tasks from DeepMind Control, OpenAI Gym, PyBullet, and IsaacGym, including low- and high-dimensional environments and fast GPU simulation.Experiments compare RPO with PPO, multiple action distributions, entropy coefficients, and data-augmentation methods, generally using 10 random seeds.Source: §4.1 · p. 6, §4.2 · p. 6

4.2.1 COMPARISON WITH PPO

• RPO consistently improves over PPO across DeepMind Control, OpenAI Gym, PyBullet, and IsaacGym tasks, including cases where PPO plateaus, fails to learn, or degrades with training.RPO maintains improvement over extended training, while PPO often stops improving after roughly 2M timesteps or deteriorates.Source: §4.2.1 · pp. 6–8, Figure 2 · p. 7
• RPO increases entropy early and maintains a threshold, whereas PPO entropy generally decreases, accompanying RPO’s continued exploration and performance gains.This pattern appears across DeepMind Control, Gym, PyBullet, and IsaacGym comparisons.Source: §4.2.1 · pp. 6–8

4.2.2 COMPARISON WITH ENTROPY REGULARIZATION

• RPO outperforms PPO and entropy-coefficient baselines without using entropy-coefficient hyperparameters, while poorly tuned coefficients can worsen performance.Coefficients above 0.05 can produce unbounded entropy increases and poorer returns in the reported setup.Source: §4.2.2 · pp. 8–9, Figure 6 · p. 9

4.2.3 COMPARISON WITH OTHER DISTRIBUTIONS

• Action-distribution choice affects task performance: Laplace substantially outperforms Gaussian on Pendulum-v1, while Gumbel performs similarly to Gaussian; RPO improves the base distributions overall.RPO also produces higher policy entropy throughout training in the reported distribution comparisons.Source: §4.2.3 · p. 9

4.2.4 COMPARISON WITH DATA AUGMENTATION

• Data augmentation can increase entropy but may not improve return, requires environment-specific knowledge, and adds processing time; RPO’s alpha controls the entropy-performance trade-off.Values between 0.1 and 3 often perform better, whereas very small or very large alpha values can hamper performance; experiments use α = 0.5.Source: §4.2.4 · p. 9, §4.2.5 · p. 12, Figure 9 · p. 12
• RPO achieves higher mean episodic return than PPO, RAD, and DRAC in most DeepMind Control environments after 8M timesteps.Some data-augmentation baselines perform worse than base PPO, while RPO requires no environment-specific domain knowledge or additional preprocessing.Source: §4.2.4 · p. 9, Table 1 · p. 10

RELATED APPROACHES

• Entropy regularization can improve convergence, but empirical studies report difficulty selecting its coefficient and little performance gain in some environments.Source: §5 · p. 12

ACTION DISTRIBUTIONS

• Gaussian policies are common for continuous actions, while RPO proposes a perturbed Gaussian distribution and reports advantages over Gaussian, Laplace, and Gumbel distributions.Source: §5 · p. 12

DATA AUGMENTATION

• Data augmentation can raise policy entropy and improve exploration, but its suitable transformation is environment-specific and requires domain knowledge.Source: §5 · p. 12

IMPLEMENTATION IMPROVEMENTS

• RPO is complementary to established policy-optimization improvements, including advantage estimation, advantage normalization, and clipped policy and value objectives.The baselines also use these implementation improvements for fair comparison.Source: §5 · p. 12

• RPO perturbs the action distribution, encourages high-entropy actions, and consistently improves performance over PPO, entropy regularization, alternative distributions, and data augmentation across continuous-control tasks.Experiments cover DeepMind Control, OpenAI Gym, Pybullet, and IsaacGym.Source: §6 · p. 13

Entropy regularization

• RPO is compared with entropy regularization using multiple coefficient values, with the comparisons reported in Figures 10 and 11.Source: §A.1 · p. 14, Figure 10 · p. 15, Figure 11 · p. 16

Data augmentation

• RPO is compared with RAD and DRAC data augmentation methods using return curves and entropy measurements.RAD and DRAC use PPO as their base algorithm, while RPO does not use data augmentation.Source: §A.1 · p. 14, Figure 12 · p. 17

DeepMind Control results

• RPO achieves higher episodic returns than PPO across DeepMind Control environments, including settings where PPO fails to learn useful behavior.PPO often stops improving after around 2M timesteps, whereas RPO continues improving throughout training.Source: §A.1 · p. 14, Figure 13 · p. 17

Action distributions

• Across Gym and Pybullet action-distribution comparisons, perturbing the distribution with a Uniform component gives RPO higher policy entropy and potentially improved exploration.Source: §A.1 · p. 18, Figure 15 · p. 19
• DeepMind Control experiments also compare different action distributions and report entropy results for those environments.Source: §A.1 · p. 18, Figure 16 · p. 20

Ablation and implementation

• The experiments include an ablation over the α values controlling the Uniform-distribution component in RPO.The shared experimental setup uses PPO implementation hyperparameters across agents and environments for fair comparison.Source: §A.1 · pp. 14, 18, Table 2 · p. 17, Figure 17 · p. 21