Source-linked AI summary
Parameter Space Noise for Exploration
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, Marcin Andrychowicz
TL;DR
Deep RL commonly explores by injecting action noise, but this can be less consistent than perturbing policy parameters. The paper combines parameter-space noise with DQN, DDPG, and TRPO, finding improved exploration and performance across discrete and continuous tasks. Parameter-space noise is especially effective in sparse-reward settings, while benefits are not guaranteed generally.
Problem
Deep RL needs effective exploration beyond action-space noise, particularly because sparse rewards and premature local-optimum convergence make exploration difficult.
Method
The paper adds Gaussian parameter perturbations to off-the-shelf DQN, DDPG, and TRPO policies, sampling a fixed perturbed policy per episode and adapting its noise scale.
Results
Parameter-space noise often outperforms action-space noise, supports earlier learning, and outperforms evolutionary strategies on 15 of 21 Atari games despite 25 times less data.
Takeaways & Limitations
Parameter-space noise is a viable alternative to action-space noise and can solve environments with very sparse rewards.
Takeaways & Limitations
Exploration improvements are not guaranteed generally, so the benefit of parameter-space noise must be evaluated case by case.
Abstract
from arXiv · showhide
Deep reinforcement learning (RL) methods generally engage in exploratory behavior through noise injection in the action space. An alternative is to add noise directly to the agent's parameters, which can lead to more consistent exploration and a richer set of behaviors. Methods such as evolutionary strategies use parameter perturbations, but discard all temporal structure in the process and require significantly more samples. Combining parameter noise with traditional RL methods allows to combine the best of both worlds. We demonstrate that both off- and on-policy methods benefit from this approach through experimental comparison of DQN, DDPG, and TRPO on high-dimensional discrete action environments as well as continuous control tasks. Our results show that RL with parameter noise learns more efficiently than traditional RL with action space noise and evolutionary strategies individually.
1 INTRODUCTION
Exploration is a central deep-RL challenge because agents can prematurely converge to local optima, while efficient exploration is difficult and often requires complex auxiliary structures. Parameter-space noise offers an alternative explored across multiple RL settings and algorithms.
- Exploration aims to prevent deep-RL agents from prematurely converging to local optima.
- Efficient exploration is difficult because the reward function does not directly guide it.
- Existing exploration methods often add complex structures, including counting tables, density models, learned dynamics, or self-supervised curiosity.
- Parameter noise is investigated with DQN, DDPG, and TRPO across high-dimensional discrete environments and continuous-control tasks.
- The experiments indicate that parameter noise outperforms traditional action-space-noise baselines, especially when rewards are extremely sparse.
2 BACKGROUND
The paper frames RL through Markov decision processes and distinguishes off-policy algorithms, which learn from arbitrary-policy data, from on-policy methods, which follow the currently updated policy. It introduces DQN, DDPG, and TRPO as representative algorithms.
- A Markov decision process specifies states, actions, initial-state distribution, rewards, transitions, a time horizon, and a discount factor.
- The agent’s objective is to maximize expected discounted return over trajectories generated by its policy.
- Off-policy methods: Off-policy methods learn from data collected by arbitrary policies; this paper uses DQN and DDPG as examples.
- Off-policy methods: DQN estimates the optimal Q-value function with a deep network and typically uses ε-greedy or Boltzmann action-space noise for exploration.
- Off-policy methods: DDPG is an actor-critic method for continuous actions whose exploration uses uncorrelated or correlated action-space noise.
- On-policy methods: On-policy methods update function approximators according to the currently followed policy, including TRPO’s constrained policy-distribution update.
3 PARAMETER SPACE NOISE FOR EXPLORATION
Parameter-space noise perturbs a policy’s parameters at episode start and keeps the resulting policy fixed during the rollout, producing state-dependent, temporally consistent exploration. The method includes adaptive scaling and separate off-policy and on-policy integrations.
- Policies are parameterized functions πθ, with θ representing the neural-network parameter vector.
- Structured exploration samples eθ = θ + N(0, σ2I) once per episode and keeps the perturbed policy fixed throughout the rollout.
- Unlike action noise, parameter perturbations make repeated visits to the same state produce the same action under a fixed perturbed policy.
- Layer normalization enables a common perturbation scale across network layers despite differing activation magnitudes.
- Adaptive scaling relates parameter-noise magnitude to induced action-space variation and adjusts it around a threshold.
- For off-policy learning, only the exploratory policy is perturbed while the unperturbed network is trained on replayed data.
- For on-policy learning, parameter noise is incorporated through an adapted policy gradient with fixed isotropic covariance whose scale is adjusted adaptively.
4 EXPERIMENTS
The experiments compare adaptive parameter space noise with action-space noise and other exploration strategies across discrete Atari, continuous-control, and sparse-reward tasks. Parameter space noise often improves exploration and learning, but its benefits are task-dependent and not guaranteed.
- Experimental setup: The experiments test parameter space noise with DQN, DDPG, and TRPO across high-dimensional discrete-action and continuous-control environments.Comparisons include action-space noise, no noise, and other exploration strategies.
- Discrete-action environments: Parameter space noise often outperforms action-space noise on Atari games requiring consistent behavior and usually starts learning sooner.It performs comparably on the remaining games, and a double-headed DQN comparison indicates the architectural change is not responsible for the improvement.
- Continuous control environments: On HalfCheetah, parameter space noise achieves significantly higher returns by escaping the local optimum reached by other exploration schemes; on two other environments, performance is comparable.The local behavior involves flipping onto the back and wiggling forward.
- Continuous control environments: In Walker2D, adding parameter noise decreases performance variance between seeds, indicating aid in escaping local optima.The result is reported for TRPO and concerns variability across random seeds.
- Sparse-reward environments: In sparse-reward tasks, parameter space noise is the only tested DDPG exploration strategy that learns successful policies on SparseCartpoleSwingup and SparseMountainCar.All DDPG configurations fail on SwimmerGather, while SparseDoublePendulum is easy to solve even without noise.
- Comparison with evolution strategies: With 25 times less data, DQN using parameter space noise outperforms evolution strategies on 15 of 21 Atari games.DQN trains on 40 M frames, whereas the cited ES results use 1 000 M frames.
5 RELATED WORK
Prior work explored parameter perturbations, evolutionary strategies, and temporally consistent exploration, but this paper extends parameter-space noise to modern deep RL across broader settings.
- Earlier policy-gradient work found parameter perturbations generally outperformed random exploration, but used relatively low-dimensional policies and state spaces.
- This work evaluates parameter-space noise with on- and off-policy methods, high-dimensional policies, and environments with large state spaces.
- Evolution strategies can train high-dimensional policies from pixels, but generally disregard temporal structure in exploration.
- Bootstrapped DQN promotes directed exploration by selecting one network head per episode, whereas this approach perturbs network parameters directly.
6 CONCLUSION
The conclusion presents parameter-space noise as a simple alternative to action-space noise, applicable across DQN, DDPG, and TRPO settings and especially useful for sparse rewards.
- Parameter-space noise is proposed as a replacement for ϵ-greedy and additive Gaussian action noise in contemporary deep RL.
- Experiments combine parameter perturbations with DQN, DDPG, and TRPO across high-dimensional discrete environments and continuous control tasks.
- The evaluation uses OpenAI Gym and modified rllab environments, including sparse tasks with reward conditions tied to specific achievements.
- Figure 6 depicts a simple, scalable environment designed to test exploratory behavior.
- Adaptive parameter-space noise DQN is compared with bootstrapped DQN and ϵ-greedy DQN using a single perturbed head and adaptive scaling near δ ≈0.05.
B PARAMETER SPACE NOISE FOR ON-POLICY METHODS
For on-policy methods, the paper formulates policy-gradient learning with Gaussian parameter perturbations and uses a variance-reducing baseline.
- Policy-gradient methods optimize expected trajectory return for stochastic policies whose parameters are sampled from a Gaussian distribution.
- The likelihood-ratio expansion supports subtracting a variance-reducing baseline b_i.
- The parameter covariance is set to Σ := σ2I and rescaled using the proposed adaptation method.
C ADAPTIVE SCALING
Adaptive scaling addresses the difficulty of choosing a parameter-noise scale by varying σ_k over time and relating it to induced action-space variation. The paper uses policy-distance measures tailored to DQN, DDPG, and TRPO.
- Parameter-noise scale σ_k is adapted over time because network architecture and learning progress affect parameter sensitivity.
- The update heuristic compares perturbed and unperturbed policies in action space using distance d, rescaling factor α, and threshold δ.The experiments use α = 1.01; the distance and δ depend on the policy representation.
- C.1 A DISTANCE MEASURE FOR DQN: For DQN, probabilistic softmax policies avoid treating equal policy behavior as different merely because all Q-values shift by a constant.The resulting formulation normalizes Q-values and avoids the pitfall of using a direct Q-function norm.
- C.1 A DISTANCE MEASURE FOR DQN: DQN compares policies with KL divergence, which can be related to ε-greedy noise through the action-space divergence between greedy and ε-greedy policies.This relation supports fair comparison without choosing an additional δ hyperparameter.
- For DDPG, parameter perturbations are calibrated against additive Gaussian action noise, while TRPO constrains perturbed policies through a natural-step trust region.DDPG sets δ := σ to match the standard deviation of regular Gaussian action noise; TRPO uses conjugate gradients and line search.
D ADDITIONAL RESULTS ON ALE
Additional ALE results report median DQN returns across all 21 Atari games and compare DQN exploration variants with Evolution Strategies under different training budgets.
- Figure 7 plots median DQN returns for all ALE environments over training steps.
- Table 1 compares final ES performance after 1 000 M frames with DQN variants after 40 M frames, using evaluation without exploration.DQN values are median returns across three seeds, while ES values come from Salimans et al. (2017).
E ADDITIONAL RESULTS ON CONTINUOUS CONTROL WITH SHAPED REWARDS
Additional continuous-control results with dense rewards show noisy performance in two inverted-pendulum environments, while adaptive parameter noise is especially stable in InvertedDoublePendulum.
- Figure 8 plots median DDPG returns for all evaluated dense-reward environments over epochs.
- Adaptive parameter space noise achieves the most stable performance on InvertedDoublePendulum despite noisy results in both inverted-pendulum tasks.Overall performance is comparable to other exploration approaches, while removing both action and parameter noise does not achieve comparable results.
F ADDITIONAL RESULTS ON CONTINUOUS CONTROL WITH SPARSE REWARDS
Additional sparse-reward continuous-control results evaluate TRPO with curvature-scaled parameter noise across three environments. The reported results indicate more consistent learning on these challenging tasks.
- Adding parameter space noise aids more consistent learning on challenging sparse-reward environments.The comparison uses a TRPO baseline with action noise from a Gaussian policy whose variance is learned.
- Figure 9 plots median TRPO returns for three sparse-reward environments over epochs.