Source-linked AI summary
Noisy Networks for Exploration
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, Shane Legg
TL;DR
Deep reinforcement learning needs exploration strategies that go beyond local random perturbations and that can work with neural-network agents. NoisyNet injects learned parametric noise into network weights, replacing conventional exploration heuristics across DQN, Dueling, and A3C. Across Atari games, the resulting agents achieve substantial performance gains, including super-human performance in some cases, with little computational overhead.
Problem
Conventional exploration uses local dithering, while optimism and intrinsic-reward approaches have limitations with complex neural-network function approximators or exploration-generalisation separation.
Method
NoisyNet learns perturbations and their variances in network weights through reinforcement-learning gradients, using consistent sampled noise to drive exploration.
Results
Across 57 Atari games, NoisyNet versions of DQN, Dueling, and A3C improve performance over baseline agents, with some games reaching super-human performance.
Takeaways & Limitations
Learned weight noise provides contextual exploration that can replace entropy bonuses and ϵ-greedy heuristics while requiring little additional computation or hyperparameter tuning.
Takeaways & Limitations
For A3C, unbiased gradients require the network noise to remain consistent throughout each rollout.
Abstract
from arXiv · showhide
We introduce NoisyNet, a deep reinforcement learning agent with parametric noise added to its weights, and show that the induced stochasticity of the agent's policy can be used to aid efficient exploration. The parameters of the noise are learned with gradient descent along with the remaining network weights. NoisyNet is straightforward to implement and adds little computational overhead. We find that replacing the conventional exploration heuristics for A3C, DQN and dueling agents (entropy reward and $ε$-greedy respectively) with NoisyNet yields substantially higher scores for a wide range of Atari games, in some cases advancing the agent from sub to super-human performance.
1 INTRODUCTION
NoisyNet addresses limitations of conventional exploration heuristics by learning perturbations to network weights, enabling consistent, state-dependent policy changes. The approach applies across deep RL architectures and improves Atari performance with little additional computational cost.
- Motivation: Conventional exploration methods such as ϵ-greedy and entropy regularisation rely on local random policy perturbations that may not produce large-scale behavioural patterns.These heuristics induce novel behaviour through random perturbations, but the passage notes that such local dithering can be inefficient in many environments.
- Motivation: Optimism-based methods often do not extend easily to neural networks, while intrinsic-reward methods separate exploration from generalisation and depend on reward weighting.The cited limitations concern applicability to complex function approximators, data efficiency, and the choice of intrinsic-reward weighting.
- NoisyNet: NoisyNet learns perturbations and their variances directly in network weights, producing consistent, potentially complex, state-dependent policy changes across multiple time steps.Noise variance is learned with gradients from the reinforcement-learning loss alongside the other agent parameters.
- NoisyNet: NoisyNet functions as a randomised neural-network value function without requiring duplicated network sections, while remaining applicable to policy-gradient methods such as A3C.Its affine noise parameterisation doubles linear-layer parameters but typically leaves computational cost dominated by weight-by-activation multiplications.
- Results: Across 57 Atari games, NoisyNet-DQN and NoisyNet-Dueling achieve striking gains over baselines, while NoisyNet-A3C also improves performance without significant extra computational cost.The adaptations require fewer hyperparameters to tune than the corresponding conventional exploration approaches.
2 BACKGROUND
The background frames deep reinforcement learning through MDPs, value functions, and neural-network loss minimisation. It then describes DQN, Dueling, and A3C as representative algorithms with distinct value-based and policy-gradient formulations.
- 2.1 MARKOV DECISION PROCESSES AND REINFORCEMENT LEARNING: An MDP is defined by state space X, action space A, reward function R, transition kernel P, and discount factor γ.A stochastic policy maps each state to a distribution over available actions.
- 2.1 MARKOV DECISION PROCESSES AND REINFORCEMENT LEARNING: Qπ(x, a) is the expected γ-discounted cumulative reward obtained by executing policy π from state x after taking action a.The optimal action-value function is defined by selecting the policy with the highest value.
- 2.1 MARKOV DECISION PROCESSES AND REINFORCEMENT LEARNING: The value function Vπ(x) averages Qπ(x, a) over actions sampled from policy π and represents expected discounted return from state x.This connects action values to the policy-level return used in reinforcement learning.
- 2.2 DEEP REINFORCEMENT LEARNING: Deep RL uses neural networks to approximate RL functions and trains their parameters by minimising a loss L(θ).The background lists DQN, Dueling, A3C, policy-optimisation, deterministic-policy, and distributional methods as examples.
- 2.2 DEEP REINFORCEMENT LEARNING: DQN estimates the optimal action-value function from replay-buffer transitions using a periodically updated target network and ϵ-greedy action selection.Dueling extends DQN with separate value and advantage subnetworks sharing a convolutional encoder and uses a double-DQN update.
- 2.2 DEEP REINFORCEMENT LEARNING: A3C directly learns a policy and its value function with a policy-gradient loss that combines policy, value, and entropy terms.Its advantage uses observed rewards and value estimates, while λ balances policy loss against value-function loss.
3 NOISYNETS FOR REINFORCEMENT LEARNING
NoisyNet uses learned, parametric perturbations of neural-network weights to drive exploration, with noise sampled during optimization and held fixed between updates. The approach is adapted across DQN, Dueling, and A3C by replacing conventional exploration mechanisms with noisy network parameters.
- Noisy networks: NoisyNet represents neural-network weights and biases as learnable means and scales applied to sampled noise.The noisy linear layer replaces weights and biases with affine transformations using learnable µ and σ parameters and random variables ε.
- Noisy networks: The noisy-network loss is an expectation over parameter noise, optimized with gradients estimated from a single Monte Carlo noise sample per step.The parameters of the noise distribution are optimized jointly through the expected loss.
- Noise distributions: Factorised Gaussian noise reduces random-number-generation cost by using separate noise variables for inputs and outputs instead of one per weight.Independent Gaussian noise uses one independent entry per weight, whereas factorised Gaussian noise uses one noise variable per input and output.
- Exploration: Noise levels are automatically tuned during training, allowing the agent to adjust the amount of parameter perturbation used for exploration.The noise scale is learned rather than manually tuned across tasks.
- DQN and Dueling: NoisyNet-DQN and NoisyNet-Dueling remove ε-greedy exploration, greedily selecting actions from randomized value functions with parameters resampled around optimization steps.The current noisy parameter sample is held fixed across each replay batch and resampled before each action because optimization occurs at every action step.
- A3C: NoisyNet-A3C removes the entropy bonus and samples noisy policy parameters after optimization, while keeping noise fixed throughout each rollout for unbiased gradients.A3C uses independent Gaussian noise and samples actions from the resulting current policy.
4 RESULTS
Across 57 Atari games, NoisyNet variants were evaluated against baseline DQN, Dueling, and A3C agents using human-normalised performance. NoisyNet improved aggregate scores and generally produced stronger performance throughout learning, while its learned noise evolved differently across games and layers.
- Evaluation setup: 57 Atari games were used to compare NoisyNet agents with baseline DQN, Dueling, and A3C agents relying on conventional exploration methods.The baseline comparisons used the corresponding original network architectures and hyperparameters.
- Aggregate performance: NoisyNet improved median human-normalised scores for all three agents by 18 to 48 percentage points.The smallest increase occurred for A3C and the largest for DQN.
- Aggregate performance: NoisyNet also significantly improved mean human-normalised scores for DQN, Dueling, and A3C.Table 1 reports comparisons using both median and mean human-normalised scores.
- Learning performance: NoisyNet agents outperformed their corresponding baselines on most games and maintained superior performance throughout the learning process.The improvement was especially significant for NoisyNet-DQN and NoisyNet-Dueling, with some games showing an order-of-magnitude improvement over the vanilla agent.
- Learning in noisy layers: The average noise parameter ¯Σ decreased in the final layer across all five examined games but decreased in the penultimate layer for only two games.Across the remaining three games, penultimate-layer ¯Σ increased; trajectories also varied across games and seeds.
- Learning in noisy layers: NoisyNet-DQN did not necessarily converge to a deterministic solution, suggesting that its exploration strategy is problem-specific rather than fixed.The authors interpret ¯Σ as a measure of noisy-layer stochasticity.
5 CONCLUSION
NoisyNet is presented as a general exploration method that improves performance across Atari games and multiple deep reinforcement learning architectures. Its exploration benefits may be entangled with optimization effects, while the approach automatically tunes noise and can extend beyond the evaluated baselines.
- NoisyNet-DQN, NoisyNet-Dueling, and NoisyNet-A3C achieve super-human performance in Beam Rider, Asteroids, and Freeway, respectively.
- Further analysis is needed to disentangle exploration effects from optimization effects because NoisyNet modifies the networks’ cost functions.
- The reinforcement learning algorithm automatically tunes the amount of injected noise, eliminating hyperparameter tuning required by entropy bonuses and ε-greedy exploration.
- NoisyNet can be applied to deep reinforcement learning algorithms trained with gradient descent, including DDPG, TRPO, and distributional RL.
A NOISYNET-A3C IMPLEMENTATION DETAILS
NoisyNet-A3C replaces the policy and value heads’ linear layers with noisy layers while estimating returns from roll-outs. To preserve a consistent on-policy return estimate, the network noise remains fixed throughout each roll-out.
- A3C directly parameterizes its policy and updates policy parameters by gradient ascent on the expected return.
- For each roll-out state, A3C samples actions from the policy and computes an approximation of the policy gradient using rewards and a bootstrapped value.
- A3C trains its value head by minimizing squared error between estimated returns and predicted values, then updates parameters after each roll-out.
- The original A3C policy update can include an entropy term that encourages exploration by favoring policies uniform over actions.
- NoisyNet-A3C replaces the policy and value heads’ linear layers with noisy layers and estimates returns using the resulting noisy network.
- The same noise is held fixed across each roll-out so the on-policy network produces a consistent estimate of the current policy’s return.
B NOISY LINEAR LAYER
A noisy linear layer retains the standard linear-layer computation while perturbing its weights and bias with learned scales and sampled noise. Its output is computed from the perturbed parameters.
- A noisy linear layer is presented as a graphical representation of the layer used by NoisyNet.
- The learnable parameters are μ_w, μ_b, σ_w, and σ_b, while ε_w and ε_b are noise variables that may use factorized or non-factorized sampling.
- The noisy weights and bias are formed as w = μ_w + σ_w ⊙ε_w and b = μ_b + σ_b ⊙ε_b.
- After perturbation, the layer produces its output with the usual affine computation y = wx + b.
C.1 NOISYNET-DQN AND NOISYNET-DUELING
The NoisyNet-DQN and NoisyNet-Dueling procedures use noisy action-value networks within replay-based training. Their implementation samples transitions, handles replay-buffer capacity, and applies gradient updates using target-network machinery.
- NoisyNet-DQN and NoisyNet-Dueling take an environment, network parameters, target parameters, replay-buffer settings, and network noise variables as inputs.
- Each interaction samples a next state and reward, then stores the transition in the replay buffer.
- When the replay buffer exceeds its configured size, the oldest transition is deleted.
- Training samples a minibatch of transitions from a replay distribution that may be uniform or prioritized.
- The algorithm performs a gradient step using squared error between the target action value and the noisy network’s predicted action value.
- The NoisyNet-A3C procedure uses global policy and value parameters, shared counters, thread-specific counters, roll-out sizes, and noise variables.
- Each A3C thread obtains a state, samples network noise, collects actions and rewards, and receives subsequent states until termination or the training limit.
- The procedure continues until the shared counter exceeds the maximum training limit.
D COMPARISON BETWEEN NOISYNET-A3C (FACTORISED AND NON-FACTORISED NOISE) AND A3C
Figure 5 compares factorised and non-factorised NoisyNet-A3C with the baseline using median human normalised learning curves.
- Figure 5 compares factorised and non-factorised NoisyNet-A3C against the baseline using median human normalised score.
E LEARNING CURVES AND RAW SCORES
This section presents learning-curve comparisons and raw scores across Atari games for baseline agents and their NoisyNet counterparts.
- Table 3 reports raw scores across all games with random starts.
- Learning curves compare DQN with NoisyNet-DQN across all Atari games.
- Learning curves compare Duelling with NoisyNet-Dueling across all Atari games.
- Learning curves compare A3C with NoisyNet-A3C across all Atari games.