Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective

1. Introduction

• Deep reinforcement learning data are non-stationary and highly correlated, and updates may not form a gradient vector field; reported issues include narrow good hyper-parameter ranges and ineffective regularisation.The cited examples of ineffective regularisation include Batch Normalisation, dropout, and L2.Source: §1 · p. 1
• SN can lead to performance gains that rival other algorithmic improvements collated in RAINBOW.The paper also provides code to support reproducibility.Source: §1 · p. 2
• SN’s effect is argued to be primarily optimisation-related: changing parameter updates without changing the represented function can recover SN’s performance, with smoothness effects secondary.The ablations identify a small correlation between smoothness and performance.Source: §1 · p. 2
• Large-scale experiments show that SN can extend Adam’s range of usable hyper-parameters.Source: §1 · p. 2

2.1. Value based deep reinforcement learning

• DQN learns a state-action value function by minimising temporal-difference error, with held-back parameters in the bootstrapped target and replay-buffer sampling to mitigate, but not enforce, i.i.d. and stationarity assumptions.Source: §2.1 · p. 2
• C51 predicts a categorical distribution of expected returns, while RAINBOW combines it with replay prioritisation, n-step returns, double Q-learning, duelling value estimates, and noisy nets.Source: §2.1 · p. 2
• Adaptive optimisers such as RMSProp and Adam are described as closely linked with recent deep reinforcement learning success.Source: §2.1 · p. 2

2.2. Spectral Normalisation

• SN controls a function’s Lipschitz constant by normalising a linear layer’s largest singular value; for a composition, the Lipschitz bound is the product of the component bounds.The paper uses the relation between the Jacobian’s largest singular value and the Lipschitz constant to characterise local network smoothness.Source: §2.2 · p. 2

3. Methods

• One power-iteration step per forward pass approximates the spectral radius, avoiding the prohibitive cost of computing it at every training step.Source: §3 · pp. 2–3
• The method hard-projects weights as Ŵ = W / max(λ, ρ), using λ = 1 unless otherwise specified; convolutional layers use convolution and transposed-convolution operations.Source: §3 · p. 3

3.1. Relaxations of Spectral Normalisation

• Constraining all layers negatively affected performance, consistent with the possibility that optimal Q-functions can be non-smooth when nearby states have very different outcomes.Source: §3.1 · p. 3
• Normalisation is applied to only a few layers, preferably those with the most weights; in classic DQN architectures this is typically the layer before the output.The paper uses SN[-1] for the output layer and SN[-2] for the penultimate layer.Source: §3.1 · p. 3

4. Spectral Normalisation on the Atari suite

• The Atari evaluation uses random starts and reports mean and median Human Normalised Score; DQN-Adam layer comparisons are averaged over 54 games.The experiments investigate SN’s dual role in smoothing the function and altering optimisation dynamics.Source: Table 1 · p. 3, Figure 2 · p. 3, §4 · p. 3
• Normalising only C51’s penultimate layer (SN[-2]) raises its performance beyond RAINBOW on the Atari suite.The comparison is reported with Fig. 1 and Table 1.Source: §4 · p. 3
• Applying SN to any hidden layer significantly improves the DQN baseline and closes the performance gap with the authors’ C51 implementation.The normalised variants reused the baseline hyper-parameters rather than being fine-tuned.Source: §4 · p. 4
• Across 54 games, normalising at most one layer improves on the baseline in most cases, with exceptions on 3/54 DQN games and 11/54 C51 games.The authors note that mean and median Human Normalised Score can be dominated by large scores.Source: §4 · p. 4

5. Analysis of results

• The analysis combines large-scale architectural and optimiser experiments with MinAtar evaluations, generally averaging normalized scores over four games.The large-scale study considers six model sizes and 14,000 DQN agents; MinAtar is used for many experiments because the required ALE study would be prohibitive.Source: §5 · p. 4, §5.1 · p. 4

5.1. Spectral Normalisation decreases hyper-parameter sensitivity

• SN extends Adam’s usable hyper-parameter range and supports continued learning as MinAtar difficulty rises, while the baseline plateaus.The figure reports normalized scores for four games across ten seeds; normalising too many layers can reduce capacity and harm learning.Source: §5.1 · pp. 4–5, Figure 4 · p. 5

5.2. Smoothness and performance are weakly correlated

• SN does not generally produce smoother networks, and performance’s relation to smoothness varies by game; gradient penalty and batch normalisation generally hurt or do not improve performance.The reported relation ranges from strong dependence in Space Invaders to no correlation in Breakout.Source: §5.2 · pp. 5–6, Figure 5 · p. 5

5.3. Limitations of 1-Lipschitz networks and practical considerations

• In depth-5 networks, normalising more than half the layers degrades behaviour, consistent with evidence that strict 1-Lipschitz constraints reduce the range of functions ReLU networks can express.The authors present selective SN as consistent with relaxing the 1-Lipschitz constraint.Source: §5.3 · p. 6

5.4. Spectral Normalisation has primarily an optimisation effect

• SN adapts optimisation-step size and direction as weights grow; the authors suggest that, with Adam, it may compensate for late-training curvature estimates from expected squared gradients.They also propose MULEPS, which multiplies Adam’s epsilon term by the product of spectral radii, as a scheduler with similar expected behaviour to DIVGRAD.Source: §5.4 · pp. 6, 8
• Gradient-scaling schedulers recover SN’s performance across MinAtar models, and DIVGRAD can improve performance where SN fails with all hidden-layer radii.The authors report that data-dependent optimiser alterations recover and sometimes surpass SN’s performance boost.Source: Figure 6 · p. 6, §5.4 · pp. 6–7, Figure 7 · p. 7

• Earlier SN studies focused on regularisation for adversarial robustness, GAN stability, uncertainty estimation, and model-based RL; its optimisation effect in DRL had not been considered.Source: §6 · p. 8
• Weight Normalisation (WN) shares SN’s hard-projection approach, but the authors’ experiments support the view that its Frobenius norm encourages feature-rank loss, potentially explaining a performance difference.The cited prior work reports WN gains over a DQN baseline on a subset of ALE environments.Source: §6 · p. 8
• Prior analyses of Batch Normalisation and WN derive an effective learning rate and show weight norms interact with adaptive-algorithm damping; this paper identifies a similar damping scaling while preserving learning rate.Source: §6 · p. 8
• The authors suggest SN’s implicit scheduling of Adam’s damping term contributes to reported gains, connecting it to curvature methods that schedule Tikhonov damping.Their study suggests the empirical-Fisher curvature approximation in DRL is deficient and reports benefits from monotonically increasing damping.Source: §6 · p. 8
• Other work discusses adverse interactions between adaptive supervised-learning optimisers and TD(0), while proposed RL optimisers are restricted to linear estimators or show no empirical advantage.Source: §6 · p. 8

• Across many architectures and settings, ablations distinguish SN’s smoothness and optimisation effects and find that its performance boost is primarily associated with changed optimisation dynamics.Source: §7 · p. 8
• Moving the spectral norm into Adam’s update step yields explicit schedulers that recover SN’s performance.Source: §7 · p. 8
• The findings suggest potential gains from designing better adapting optimisers for deep reinforcement learning.Source: §7 · p. 8

A.1. Computing the spectral norm

• Power iteration estimates each selected layer’s dominant singular value using stored left and right singular vectors and two matrix-vector multiplications per normalised layer during each forward pass.Convolutional layers replace these multiplications with convolutional and transposed-convolutional operations; inference adds no computational cost.Source: §A.1 · p. 12
• On a subset of Atari games, dropping the Jacobian from power iteration caused no observed performance loss.The tested derivative was whether to backpropagate through the power-iteration step when computing gradients.Source: §A.1 · p. 12

A.2. Computing the norm of the Jacobians

• Network smoothness was measured indirectly by the maximum Euclidean norm of input Jacobians across thousands of on-policy states per network.The authors used the Euclidean norm rather than the operator norm.Source: §A.2 · p. 12

Game selection

• Freeway was excluded from the five-game MinAtar set because all agents performed essentially the same there.The remaining games were Asterix, Breakout, Seaquest, and Space Invaders.Source: B. MinAtar experiments · p. 13

Network architecture

• MinAtar experiments varied convolutional-network depth or width while retaining two linear layers; convolutional layers used 3-by-3 kernels, stride 1, and ReLU activations.Width changes affected convolutional channels and the penultimate linear layer; convolutional layers were identically scaled.Source: B. MinAtar experiments · p. 13

General hyper-parameter settings

• A small grid search around Young and Tian’s published initial values set shared MinAtar hyperparameters, with experiment-specific deviations noted separately.Source: B. MinAtar experiments · p. 13, Table 2 · p. 13

MinAtar Normalised Score

• MinAtar Normalised Score averages performance across games by scaling each agent’s score between random play and the highest baseline score recorded in the experiments.The maximum and random scores used for the calculation are listed in Table 3.Source: B. MinAtar experiments · p. 13, Table 3 · p. 13

B.1. Large optimiser hyper-parameter sweep

• SN generally outperforms the baseline, especially among higher-performing optimiser configurations.The sweep also found that normalising one layer did not degrade baseline performance and allowed larger learning rates with Adam.Source: §B.1 · pp. 13–14, Figure 9 · p. 14

B.2. Effect on model capacity

• The capacity study evaluated 12 model sizes across three depths and four widths, applying SN to various layer subsets.Each game–architecture combination was trained with 10 seeds, and scores were averaged over four MinAtar games.Source: §B.2 · p. 14

B.3. Smoothness and performance are weakly correlated

• Normalising a layer subset can improve performance even when the network is less smooth than the baseline.The figure plots each seed’s maximum performance against its Jacobian norm, illustrating that smoothness and performance do not consistently move together.Source: §B.3 · p. 14, Figure 14 · p. 15

B.4. Other regularisation methods

• Relaxing the 1-Lipschitz target recovers good performance when all layers are normalised, while GP and BN do not recover SN performance.The relaxed setup required approximating spectral radii for all layers and adding a hyperparameter per layer, so the authors did not pursue it further.Source: Figure 10 · p. 15, §B.4 · p. 15, Figure 16 · p. 15

B.5. Adaptability to changing dynamics

• With SN on a single hidden layer, agents surpass the baseline and continue improving through 15M training steps, unlike the baseline’s usual plateau.The authors interpret the extended learning curves as support for SN yielding a better-adapting optimiser.Source: §B.5 · p. 16, Figure 11 · p. 17

B.6. Spectral Schedulers

• DIVOUT behaves similarly to SN, while MULEPS and DIVGRAD converge even when all hidden layers are normalised.The comparisons use four estimator architectures and vary which layers’ spectral radii are used by SN or the schedulers.Source: §B.6 · p. 16, Figure 18 · p. 19

C.1. Evaluation protocols on Atari

• Atari agents were evaluated every 250K training steps for up to 500K frames, with validation epsilon set to 0.001 for C51 and DQN-Adam and 0.05 for DQN-RMSprop.Episodes were limited to 108K steps, and null-op starts induced stochasticity during training and evaluation.Source: §C.1 · p. 20

C.3. DQN - RMSprop

• DQN-RMSprop shows minimal benefit from SN, unlike the reported improvement for DQN-Adam.The authors leave the interaction between SN and RMSprop for future work.Source: §C.3 · p. 20

C.4. Effective rank

• SN preserves feature rank during training, whereas the baseline uses fewer feature dimensions as training progresses.Effective rank is approximated by the number of leading singular values capturing 99% of the feature matrix’s variance; the matrix is the input to the final linear layer.Source: §C.4 · pp. 20–21, Figure 21 · p. 21