PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Deeper Deep Reinforcement Learning with Spectral Normalization

Johan Bjorck, Carla P. Gomes, Kilian Q. Weinberger

arXiv:2106.01151v4cs.LG

TL;DR

RL commonly relies on small networks, leaving it unclear whether larger models fail because of limited data or another constraint. The paper studies modern actor-critic architectures, identifies instability from gradients through deep critics, and uses spectral normalization to stabilize them. The resulting larger models can outperform state-of-the-art methods across continuous-control tasks, while the study remains limited to SAC- and DDPG-based agents and related settings.

  • Problem

    RL typically uses small networks despite larger architectures improving performance in computer vision and NLP, and the hypothesis that RL’s limited data causes large-model overfitting remains largely untested.

    Source: §1 · p. 1

  • Method

    The paper evaluates larger actor-critic architectures with normalization and skip connections, then applies spectral normalization to stabilize training by smoothing critic networks.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    Spectral normalization enables stable training of larger modern networks that can outperform state-of-the-art methods on a large set of continuous-control tasks.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Network architecture can be competitive with algorithmic innovations in reinforcement learning, making model scaling a potentially important source of improvements.

    Source: §7 · pp. 9–10

    The study evaluates only SAC- and DDPG-based agents, leaving the broader applicability of large architectures and smoothing methods across RL algorithms unresolved.

    Source: §7 · pp. 9–10

Abstract

from arXiv · show

In computer vision and natural language processing, innovations in model architecture that increase model capacity have reliably translated into gains in performance. In stark contrast with this trend, state-of-the-art reinforcement learning (RL) algorithms often use small MLPs, and gains in performance typically originate from algorithmic innovations. It is natural to hypothesize that small datasets in RL necessitate simple models to avoid overfitting; however, this hypothesis is untested. In this paper we investigate how RL agents are affected by exchanging the small MLPs with larger modern networks with skip connections and normalization, focusing specifically on actor-critic algorithms. We empirically verify that naively adopting such architectures leads to instabilities and poor performance, likely contributing to the popularity of simple models in practice. However, we show that dataset size is not the limiting factor, and instead argue that instability from taking gradients through the critic is the culprit. We demonstrate that spectral normalization (SN) can mitigate this issue and enable stable training with large modern architectures. After smoothing with SN, larger models yield significant performance improvements -- suggesting that more "easy" gains may be had by focusing on model architectures in addition to algorithmic innovations.

1 Introduction

The paper tests whether RL’s reliance on small networks reflects overfitting or instability, finding that spectral normalization can make larger modern architectures effective.

Motivation

  • Unlike computer vision and NLP, RL commonly uses small feedforward networks while performance gains typically come from algorithmic innovations.Source: §1 · p. 1

Study design

  • The study examines high-capacity networks using normalization and skip connections in continuous-control actor-critic agents based on SAC and DDPG.Source: §1 · pp. 1–2

Problem and diagnosis

  • Large modern networks fail for two competitive actor-critic agents, exhibiting dramatic training instabilities that cast doubt on overfitting as the explanation.Source: §1 · pp. 1–2

Mechanism and solution

  • The paper argues that propagating actor gradients through deeper critics causes instability and proposes spectral normalization from GANs to smooth the critic.Source: §1 · p. 2

Contributions

  • Spectral normalization enables larger modern networks to train stably and improve state-of-the-art performance on difficult continuous-control tasks.The approach is presented as complementary to algorithmic innovation and computationally feasible for RL from pixels.Source: §1 · p. 2

2 Background

The background introduces continuous-control RL and explains how actor-critic agents use an actor to produce actions and a critic to evaluate them. Actor updates differentiate through the critic’s Q-values.

2.1 Reinforcement Learning

  • Continuous-control RL models environments as MDPs with states, actions, transitions, and rewards, seeking a policy that maximizes expected discounted reward.Source: §2.1 · p. 2

2.2 Actor-Critic Agents

  • Actor-critic methods combine an actor that maps state features to an action distribution with a critic that estimates Q-values for state-action pairs.Source: §2.2 · p. 2
  • SAC and DDPG are popular actor-critic methods that underpin many recent state-of-the-art continuous-control algorithms.Source: §2.2 · p. 2
  • The critic is trained from replay-buffer transitions using a soft Bellman residual and target-critic Q-value estimates.Source: §2.2 · p. 3
  • Actor gradients are obtained by differentiating the critic’s Q-value with respect to the actor parameters, so gradients propagate through the differentiable critic network.Source: §2.2 · p. 3

3 Motivating Experiments

The experiments test modern and deeper networks for pixel-based continuous-control actor-critic agents, finding that increased capacity often harms performance because training becomes unstable rather than overfit.

3.1 Experimental Setup

  • The evaluation uses pixel-based continuous control on 15 DeepMind Control Suite tasks, primarily with the DRQ SAC agent, and also considers augmentation-based DDPG.Source: §3.1 · p. 3

3.2 Testing Modern networks

  • Modern architectures with residual connections and normalization often decrease performance, sometimes catastrophically, although a handful of tasks improve.Source: Figure 1 · p. 3, §3.2 · p. 4

3.3 Testing Deeper Networks

  • Simply increasing MLP depth from two to four hidden layers improves some tasks but degrades many others, so performance does not improve monotonically.Source: §3.3 · p. 4, Figure 2 · p. 4

3.4 Is it overfitting?

  • Larger modern networks develop dramatically increasing critic losses instead of lower training losses, arguing against overfitting as the explanation for their poor performance.Source: §3.4 · p. 4, Figure 3 · p. 5
  • Large modern networks exhibit larger, growing actor and critic gradients with spikes, indicating that training instability rather than overfitting is the central problem.Source: §3.4 · p. 5, Figure 3 · p. 5

4 Stabilizing with Spectral Normalization

The paper analyzes critic smoothness through layer operator norms and proposes spectral normalization to stabilize gradients propagated into the actor. Bounding every layer’s spectral norm by one is hypothesized to enable stable deeper-network training.

  • The critic’s Lipschitz constant is bounded by the product of its layers’ operator norms, with each linear layer’s norm equal to its largest singular value.Source: §4 · p. 5
  • Because critic smoothness bounds its gradients, controlling layer spectral norms can bound gradients propagated through the critic into the actor.Source: §4 · pp. 5–6
  • Spectral normalization divides each layer’s weights by the largest singular value, approximated using power iteration with vectors u and v.Source: §4 · p. 6
  • Applying spectral normalization across all layers makes their spectral norms no larger than one, which the analysis predicts will stabilize deeper-network training.Source: §4 · p. 6

5 Experiments

Across SAC- and DDPG-based agents, spectral normalization stabilizes larger modern networks, enabling stronger performance than unsmoothed networks and competing agents at relatively modest training costs.

5.1 Smoothness Enables Larger Networks

  • Smoothed deep modern networks learn stably across tasks, whereas unsmoothed networks learn slowly or sometimes fail; performance generally improves with depth.The smoothed strategy narrowly loses only on walker walk, which the authors suggest may be a statistical outlier.Source: §5.1 · p. 6, Figure 4 · p. 7

5.2 Comparing to Other Agents

  • With spectral normalization, the modern-network SAC agent outperforms DRQ with its default MLP and Dreamer on most continuous-control tasks.The gains vary by task, and some sparse-reward tasks also improve.Source: §5.2 · p. 6, Table 2 · p. 8

5.3 Experiments with DDPG

  • For DRQv2, larger networks hurt performance without spectral normalization but often improve it when spectral normalization is applied.The DDPG experiments use selected dense- and sparse-reward locomotion and classical-control tasks, with results averaged over 10 seeds after 500,000 frames.Source: §5.3 · p. 7, Table 1 · p. 8

5.4 Performance Cost

  • Training memory rises roughly 17% when scaling the head networks, while image convolution remains the dominant memory and computation cost.During acting, memory increases by less than 200 MB in absolute terms and compute time changes little, making the architectural improvements relatively cheap for pixel-based RL.Source: §5.4 · p. 8

6 Related Work

Related work places spectral normalization within broader efforts to scale or smooth neural networks, while distinguishing this paper’s actor-critic setting from prior RL applications.

  • Prior RL work commonly uses simple feedforward networks, while RNNs and Transformers address sequential information and architectural methods such as dueling heads or double Q-learning target other improvements.The paper studies scaling rather than memory-based architectures or these algorithmic modifications.Source: §6 · pp. 8–9
  • Table 3’s related computational framing emphasizes that head-network scaling can be comparatively cheap because convolutional processing dominates training resources.The table compares four head architectures on Tesla V100 GPUs and reports small training costs relative to the shared convolutional backbone.Source: Table 3 · p. 9
  • This work differs from prior spectral-normalization studies by enabling larger networks in actor-critic algorithms where the actor takes gradients through the critic.The authors contrast this setting with DQN and note its similarity to GAN training, where spectral normalization was developed.Source: §6 · p. 9

7 Discussion

The discussion argues that network architecture is an underexamined source of RL progress, while noting compute, evaluation, algorithmic, and environmental boundaries on the conclusions.

The Importance of Architecture

  • Scaling network architecture can affect RL performance as dramatically as, or more than, algorithmic innovations, so architecture choices should not be treated as incidental.The authors suggest systematic architecture studies and note that additional compute might be offset through asynchronous training or acceleration methods.Source: §7 · pp. 9–10

Evaluation in RL

  • Because architectural changes can enable much larger networks, fair RL comparisons may need to report compute resources, especially for compute-intensive unsupervised and model-based methods.This concern follows from the paper’s demonstration that architectural modifications can substantially change achievable performance.Source: §7 · p. 9

Limitations

  • The study is limited to SAC- and DDPG-based agents and does not establish why large architectures are uncommon across RL algorithms generally.The authors also leave open whether alternative smoothing methods or other agents and environments would perform better.Source: §7 · pp. 9–10

Conclusion

  • The conclusion is that spectral normalization enables stable modern networks that can outperform state-of-the-art methods on many continuous-control tasks, making architecture competitive with algorithmic innovation.The paper frames this as demonstrating an important and feasible problem rather than solving scaling for every RL algorithm.Source: §7 · p. 10

A Appendix

The appendix examines instability indicators for modern networks and documents the associated gradient behavior, while also listing hyperparameters.

A Appendix

  • The modern network’s largest singular value grows dramatically without spectral normalization, unlike the MLP comparison.Figure 5 reports cartpole swingup results averaged across 5 runs.Source: Figure 5 · p. 15
  • Spectral normalization keeps gradients stable during training even for larger modern networks.The comparison is made with and without smoothing and refers readers to Figure 3.Source: Figure 6 · p. 15

B Hyperparameters list

C Network Details

The experiments use wide convolutional and transformer-style networks, then test whether depth improves performance when spectral normalization stabilizes training.

Baseline architecture

  • The baseline MLP uses a convolutional encoder followed by a 1024-wide, two-hidden-layer feedforward actor and critic.The input is an 84-by-84 tensor with frame-stacking from the last 3 steps.Source: C Network Details · p. 15

Modern architecture

  • The modern architecture uses transformer feedforward blocks with layer normalization, residual connections, width 1024, and intermediate width 2048.Dropout is not used, and endpoint linear layers adjust feature dimensions.Source: C Network Details · p. 15

Network depth

  • Deeper spectrally normalized networks improve performance for the 4-hidden-layer model on most of 15 DeepMind Control Suite tasks.Performance is averaged across ten seeds.Source: Figure 7 · p. 16

Depth ablation

  • Increasing the number of modern network blocks in both critic and actor appears to improve, or at least not reduce, performance on cartpole swingup sparse.This ablation averages performance across 5 seeds.Source: Figure 8 · p. 16

Spectral normalization

  • Spectral normalization is applied only to intermediate layers so the network need not remain 1-Lipschitz at its input and output.The stated motivation is preserving the ability to represent very large q-values.Source: C Network Details · p. 16

D Improvements

Deeper modern networks with spectral normalization outperform the compared baselines across many control tasks, while unsmoothed larger networks can perform poorly.

Architecture versus algorithms

  • The architecture change improves average performance by slightly more than 141, exceeding the less-than-40 improvement attributed to Dreamer-to-DRQ algorithmic changes.The passage also compares this with a 63-point CURL improvement and a smaller later improvement.Source: D Improvements · p. 16

Experimental controls

  • The authors rerun baselines to control implementation differences such as CUDA versions, code changes, GPUs, and seeds.These details can substantially affect reinforcement-learning results.Source: D Improvements · pp. 16–17

Benchmark comparison

  • Modern networks with spectral normalization beat Dreamer on 11 of 15 games and DRQ’s two-hidden-layer MLP on 13 of 15 games.The comparison uses Table 5 results across DeepMind Control Suite tasks.Source: Table 5 · p. 17, D Improvements · p. 17

Benchmark setup

  • The Table 5 comparisons use modern architecture and spectral normalization in SAC-based DRQ against default-MLP DRQ and Dreamer.The reported baseline numbers were obtained through communication with the authors and do not consistently use 10 seeds.Source: Table 5 · p. 17

DDPG results

  • For the DDPG-based agent, larger modern networks perform poorly without spectral normalization but become beneficial when spectral normalization is added.Results are reported after 1 million frames over 10 seeds.Source: Table 6 · p. 18

E Compute Resources

The experiments used high-end GPU infrastructure and required approximately one to two years of aggregate GPU time.

Hardware and software

Compute cost

Related papers

Loading 2106.01151v4…