PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity

Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, Jan Peters

arXiv:1902.05605v4cs.LGstat.ML

TL;DR

Sample-efficient Deep RL is important for physical systems, while high-UTD methods such as REDQ and DroQ incur greater computational cost. CrossQ removes target networks and uses BatchNorm, matching or surpassing state-of-the-art sample efficiency while reducing computation.

  • Problem

    Improving Deep RL sample efficiency matters for real physical systems, but high-UTD approaches increase computational cost.

    Source: §1 · pp. 1–2

  • Method

    CrossQ removes target networks and carefully applies BatchNorm, with wider critic layers providing an additional performance boost.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    CrossQ matches or surpasses state-of-the-art sample efficiency on continuous-control environments with state observations while being up to 4× faster in wallclock time.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CrossQ achieves its reported performance without critic ensembles, target networks, or high UTD ratios.

    Source: §1 · p. 2

    Evaluation focuses on standard continuous-control benchmarks; applying CrossQ to real robots and visual observations remains future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Sample efficiency is a crucial problem in deep reinforcement learning. Recent algorithms, such as REDQ and DroQ, found a way to improve the sample efficiency by increasing the update-to-data (UTD) ratio to 20 gradient update steps on the critic per environment sample. However, this comes at the expense of a greatly increased computational cost. To reduce this computational burden, we introduce CrossQ: A lightweight algorithm for continuous control tasks that makes careful use of Batch Normalization and removes target networks to surpass the current state-of-the-art in sample efficiency while maintaining a low UTD ratio of 1. Notably, CrossQ does not rely on advanced bias-reduction schemes used in current methods. CrossQ's contributions are threefold: (1) it matches or surpasses current state-of-the-art methods in terms of sample efficiency, (2) it substantially reduces the computational cost compared to REDQ and DroQ, (3) it is easy to implement, requiring just a few lines of code on top of SAC.

1 INTRODUCTION

Deep RL sample efficiency matters for physical systems, but improving it through high update-to-data ratios raises computational cost. CrossQ instead uses BatchNorm, removes target networks, and achieves strong sample efficiency with low computational cost.

1 INTRODUCTION

  • Figure 1 aggregates CrossQ performance across environments and reports greater sample and gradient-step efficiency at UTD = 1.Performance is normalized by each environment’s maximum REDQ performance.Source: Figure 1 · p. 1
  • REDQ and DroQ use ensembling or dropout-based bias reduction to support UTD ratios of 20, improving sample efficiency at greater wallclock-time and energy cost.The introduction motivates alternatives that preserve or improve sample efficiency with lower computational cost.Source: §1 · pp. 1–2
  • CrossQ combines target-network removal, carefully applied BatchNorm, and wider critic layers to improve training and returns, including at UTD = 1.The paper says BatchNorm variants stabilize training and improve sample efficiency; wider networks provide an additional performance boost.Source: §1 · p. 2
  • CrossQ matches or surpasses state-of-the-art sample efficiency for continuous control while being up to 4× faster in wallclock time, without critic ensembles, target networks, or high UTD ratios.The paper identifies these as its main contributions for model-free off-policy RL with state observations.Source: §1 · p. 2

2 BACKGROUND

Off-policy RL learns policies from environment interactions, and SAC is a popular maximum-entropy approach. Recent methods improve its sample efficiency by raising UTD, with REDQ and DroQ trading increased computation for high update rates.

2.1 OFF-POLICY REINFORCEMENT LEARNING AND SOFT ACTOR-CRITIC

  • SAC solves the maximum-entropy RL problem with neural-network Q functions and a policy, using the minimum of two Q estimates for temporal-difference targets.The clipped double-Q trick helps reduce potentially destabilizing overestimation bias.Source: §2.1 · p. 2

2.2 HIGH UPDATE-TO-DATA RATIOS, REDQ, AND DROQ

  • SAC performs one gradient-based optimization step per environment interaction, and later methods improved sample efficiency by increasing the update-to-data ratio.The section compares CrossQ with REDQ and DroQ as high-UTD methods.Source: §2.2 · p. 2
  • CrossQ’s critic-loss implementation adds BatchNorm, removes the target network, and concatenates current and next observations and actions for a joint forward pass.Combining the passes requires one CUDA call instead of two and grants a small speed-up.Source: Figure 2 · p. 3
  • REDQ stabilizes high UTD ratios up to 20 by expanding SAC’s two Q networks into an ensemble of 10 to address accumulated estimation bias.Simply raising SAC’s UTD ratio was reported to hurt performance.Source: §2.2 · p. 3
  • DroQ replaces REDQ’s large ensemble with a smaller dropout-equipped Q ensemble and Layer Normalization, reducing—but not eliminating—its computational expense at UTD 20.Its design uses Layer Normalization to stabilize training against dropout noise.Source: §2.2 · p. 3

3 THE CROSSQ ALGORITHM

CrossQ modifies SAC with UTD = 1 by removing target networks, carefully using BatchNorm, and widening critic layers. It reports sample efficiency comparable to or better than REDQ and DroQ, with substantially better performance on the more challenging Humanoid tasks.

3.1 DESIGN CHOICE 1: REMOVING TARGET NETWORKS

  • BatchNorm or bounded activations prevent critic divergence without target networks, whereas ReLU without normalization diverges.The authors identify normalization as making target networks redundant; carefully combining BatchNorm with target-network removal trains stably.Source: §3.1 · p. 4, §3.2 · p. 4

3.2 DESIGN CHOICE 2: USING BATCH NORMALIZATION

  • Removing target networks lets CrossQ concatenate replay-buffer and current-policy batches for one critic forward pass, giving BatchNorm moments from their mixed distributions.This addresses the mismatch between old behavior-policy actions and current-policy actions that makes BatchNorm use in the critic require care.Source: §3.2 · p. 4

3 THE CROSSQ ALGORITHM

  • With UTD = 1, CrossQ performs comparably to or better than REDQ and DroQ, and substantially better on the more challenging Humanoid tasks.The Figure 5 caption reports interquartile mean and 70% quantile intervals for episodic returns over 10 seeds; CrossQ’s changes also transfer to TD3.Source: Figure 5 · p. 5, §3.3 · p. 5

3.3 DESIGN CHOICE 3: WIDER CRITIC NETWORKS

  • Wider critic layers further boost performance and help CrossQ match or outperform REDQ and DroQ in sample efficiency.The authors attribute most gains to the first two design choices, with wider critics providing an additional boost.Source: §3.3 · p. 5

4 EXPERIMENTS AND ANALYSIS

Across six continuous-control environments, CrossQ matches or outperforms the best baseline and is substantially more computationally efficient. The experiments also examine Q-estimation bias and show how removing target networks and using BatchNorm affect learning and stability.

4 EXPERIMENTS AND ANALYSIS

  • The evaluation compares sample efficiency, computational efficiency, Q-function bias, and ablations across six MuJoCo continuous-control tasks, using 10 random seeds per task.The tasks are Hopper, Walker2d, Ant, HalfCheetah, Humanoid, and HumanoidStandup.Source: §4 · p. 5, §4.1 · p. 6

4.1 SAMPLE EFFICIENCY OF CROSSQ

  • CrossQ matches or outperforms the best baseline across the presented environments; on Humanoid and HumanoidStandup, CrossQ and CrossQ (TD3) substantially surpass all baselines.On Ant, REDQ performs better early, but CrossQ eventually matches it; CrossQ and REDQ overlap on Hopper, Walker, and HalfCheetah.Source: §4.1 · p. 6

4.2 COMPUTATIONAL EFFICIENCY OF CROSSQ

  • CrossQ requires 20× fewer gradient steps than REDQ and DroQ and achieves roughly 4× faster wallclock speeds.The computational-efficiency comparison is shown after 5 × 10^6 environment steps; the figure reports IQM and 70% quantile intervals over 10 seeds.Source: §4.2 · p. 6, Figure 6 · p. 6

4.3 EVALUATING Q FUNCTION ESTIMATION BIAS

  • CrossQ performs well despite generally higher Q-estimation bias, so the experiments find no clear relationship between estimation bias and performance.REDQ generally has the least bias, yet achieves comparable or worse returns than CrossQ.Source: §4.3 · p. 7, Figure 7 · p. 7

4.4.1 DISENTANGLING THE EFFECTS OF TARGET NETWORKS AND BATCHNORM

  • BatchNorm without target networks achieves the best aggregate performance; BatchNorm with target networks is inconsistent, while removing target networks alone yields only small improvements.The ablation uses tanh critic activations to permit divergence-free training without target networks.Source: §4.4.1 · p. 7, Figure 8 · p. 8

4.4.2 ABLATING THE DIFFERENT DESIGN CHOICES AND HYPERPARAMETERS

  • Removing BRN produces the worst overall performance, while replacing BatchNorm with LayerNorm degrades CrossQ roughly to the SAC baseline level.Reducing critic width to 256 reduces performance but still significantly outperforms SAC; widening SAC’s critic to 2048 alone does not help.Source: §4.4.2 · p. 8, Figure 9 · p. 8
  • BRN’s less noisy moving averages make late-training collapses less likely than with BN; higher momentums eliminate the observed collapses, and critic-only normalization is sufficient for strong performance.Including BatchNorm in both actor and critic performs slightly better than using it only in the critic.Source: §4.4.2 · p. 9, Figure 10 · p. 9

5 CONCLUSION & FUTURE WORK

CrossQ matches or exceeds REDQ and DroQ in sample efficiency on continuous-control tasks with state observations while using substantially less computation. The conclusion attributes this efficiency to removing target networks and adding BatchNorm, while noting open questions about Q-estimation bias and broader applications.

5 CONCLUSION & FUTURE WORK

  • CrossQ matches or exceeds REDQ and DroQ in sample efficiency on continuous-control environments with state observations while being multiple times more computationally efficient.The combined effect of removing target networks and adding BatchNorm makes CrossQ efficient.Source: §5 · p. 9
  • The relationship between Q-estimation bias and agent performance was not straightforward, and the authors call for more extensive bias analysis and theoretical study of BatchNorm in reinforcement learning.The authors also identify wider critic networks as a source of additional computation despite their performance boost.Source: §5 · p. 9
  • The study focuses on standard continuous-control benchmarks; future work could test CrossQ on real robots, with visual observations, and with image-based reinforcement-learning techniques.The authors also suggest investigating whether the computation cost of wider critics can be reduced.Source: §5 · p. 9

A.1 DEEPMIND CONTROL SUITE EXPERIMENTS

The appendix extends the main-paper experiments to DeepMind Control Suite tasks. Those experiments used five random seeds, with a lower learning rate and a CrossQ policy delay of 1.

A.1 DEEPMIND CONTROL SUITE EXPERIMENTS

  • For these tasks, all algorithms used a learning rate of 8 × 10−4, and CrossQ used a policy delay of 1.All other hyperparameters remained the same as in the main paper.Source: §A.1 · p. 13
  • The DeepMind Control Suite experiments were moved to the appendix because of space constraints.Source: §A.1 · p. 13
  • CrossQ’s good sample efficiency transfers to the presented DeepMind Control Suite tasks, with each experiment performed across five random seeds.Figure 11 presents the additional task results as an extension of Figure 5 in the main paper.Source: Figure 11 · p. 13, §A.1 · p. 13

A.2 HYPERPARAMETERS

The section presents the main-paper learning hyperparameters and notes how its moving-average momentum relates to the update rate used in BatchNorm papers.

A.2 HYPERPARAMETERS

A.3 WALLCLOCK TIME MEASUREMENT

Wallclock times were measured as pure training time through five million environment steps. CrossQ substantially reduces wallclock time relative to REDQ and DroQ, while wider critics incur a performance penalty relative to CrossQ (Small) and SAC.

A.3 WALLCLOCK TIME MEASUREMENT

  • Training times were averaged over four seeds and measured through 5 × 10^6 environment steps, excluding synchronous evaluation and logging overhead.Measurements used an Nvidia RTX 3090 Turbo and AMD EPYC 7453 CPU.Source: §A.3 · p. 14
  • CrossQ shows a substantial wallclock-time improvement over REDQ and DroQ in the reported comparison.The table reports hours across environments; measurement details are in Appendix 4.2.Source: Table 2 · p. 14
  • Compared with CrossQ (Small) and SAC, using wider critic networks comes with a performance penalty.Source: Table 2 · p. 14

A.4 EVOLVING ACTION DISTRIBUTIONS

Replay-buffer and current-policy actions remain distinct and drift during training, so BatchNorm requires care in off-policy TD learning.

A.4 EVOLVING ACTION DISTRIBUTIONS

  • Across 300,000 Walker2d steps, replay-buffer and policy action distributions were sampled every 60,000 steps for comparison.Each snapshot used 10,000 transitions and compared actions across action-space dimensions 4 and 5.Source: Figure 12 · p. 15
  • Replay-buffer and current-policy actions remain visibly different in their distributions, means, and standard deviations throughout training.The distributions continue to drift as training progresses.Source: Figure 12 · p. 15
  • This persistent discrepancy implies that BatchNorm must be used with care in off-policy TD learning.Source: Figure 12 · p. 15

A.4.1 ABLATING THE DIFFERENT DESIGN CHOICES AND HYPERPARAMETERS

The ablations show that CrossQ depends on BatchNorm and double Q networks, while several alternative settings fail to improve REDQ or DroQ. Exploratory tests find other routes to divergence-free training, but not a full account of stability.

A.4.1 ABLATING THE DIFFERENT DESIGN CHOICES AND HYPERPARAMETERS

  • Removing BatchNorm causes divergence, while LayerNorm learns more slowly and a single critic harms performance.LayerNorm trains stably without target networks; adding BatchNorm to SAC while reusing the live critic’s normalization moments in the target also fails.Source: Figure 13 · p. 16
  • CrossQ small is competitive in four of six environments; wider SAC does not improve, default Adam momentum degrades some results, and policy delay has little effect except on Ant.Source: Figure 13 · p. 16

A.5 REDQ AND DROQ ABLATIONS

A.6 EFFECT OF ACTIVATIONS AND NORMALIZERS ON LEARNING STABILITY

  • In exploratory SAC without target networks, tanh activations and feature normalizers avoid divergence, but these alternatives do not perform as strongly as BatchNorm.The paper notes that a thorough theoretical or experimental study of activation and normalization effects on stability is beyond its scope.Source: §A.6 · p. 18, Figure 16 · p. 18

A.7 NORMALIZED Q BIAS PLOTS

  • CrossQ generally has the same or larger Q estimation bias than REDQ yet matches or exceeds REDQ’s learning speed, especially on Humanoid.The experiments find no straightforward connection between normalized Q-function bias and learning performance.Source: Figure 17 · p. 19
Loading 1902.05605v4…