PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research

Johan S. Obando-Ceron, Pablo Samuel Castro

arXiv:2011.14826v2cs.LGcs.AI

TL;DR

Standard RL benchmarks help standardize evaluation, but their computational demands present a barrier for researchers without ample resources. This paper revisits Rainbow on smaller-scale environments and investigates its components and related choices. The authors reproduce its findings at a high level and report that small- and medium-scale environments support closer empirical analysis, while Rainbow’s component effects vary across environments.

  • Problem

    Standard benchmark evaluations can require substantial compute, while small-scale environments are used less often for evaluating new RL methods despite their low cost and training speed.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    The authors revisit Rainbow by comparing component additions to DQN and removals from Rainbow, and examine architecture, batch size, and related algorithmic choices.

    Source: §1 · pp. 1–2, §4.1 · p. 3

  • Results

    The authors report reproducing Hessel et al. (2018)’s findings at a high level, while finding that component effects vary across environments and that Rainbow still outperforms DQN in the compared environments.

    Source: §6.1 · p. 7, §7 · p. 8

  • Takeaways & Limitations

    The authors argue that small- and medium-scale environments support more critical analysis of algorithms and should be considered in research and review, without replacing large-scale benchmarks.

    Source: §7 · pp. 8–9

    M-DQN could not be integrated with C51 or double-DQN, so M-Rainbow is compared with Rainbow lacking distributional RL and double-DQN.

    Source: §5.3 · p. 6

Abstract

from arXiv · show

Since the introduction of DQN, a vast majority of reinforcement learning research has focused on reinforcement learning with deep neural networks as function approximators. New methods are typically evaluated on a set of environments that have now become standard, such as Atari 2600 games. While these benchmarks help standardize evaluation, their computational cost has the unfortunate side effect of widening the gap between those with ample access to computational resources, and those without. In this work we argue that, despite the community's emphasis on large-scale environments, the traditional small-scale environments can still yield valuable scientific insights and can help reduce the barriers to entry for underprivileged communities. To substantiate our claims, we empirically revisit the paper which introduced the Rainbow algorithm [Hessel et al., 2018] and present some new insights into the algorithms used by Rainbow.

1. Introduction

Standard RL benchmarks have helped standardize evaluation, but their computational demands can make it harder for underprivileged newcomers to contribute. The paper argues for rigorous small- and medium-scale empirical work and revisits Rainbow to test its findings and investigate algorithm–architecture interactions.

1. Introduction

  • Small-scale environments make broad hyperparameter sweeps and closer investigation of algorithmic nuances computationally feasible.The authors argue that such environments can also support reevaluation of empirical choices that have become “folk wisdom.”Source: §1 · p. 1
  • Standard benchmarks can establish a high computational threshold for recognition, while newcomers from underrepresented communities face substantial compute hurdles.The authors describe this concern as a barrier to entry, not a reason to sacrifice scientific rigour.Source: §1 · p. 1
  • The paper revisits Rainbow on small- and medium-sized tasks to test whether Hessel et al. (2018) would have reached the same conclusions at smaller scale.It also investigates interactions between algorithms and network architecture, distribution parameterization, bootstrapping methodology, and DQN’s base loss.Source: §1 · pp. 1–2

2. Preliminaries

The preliminaries define the MDP and value-function setting, then outline DQN and the enhancements combined in Rainbow. These components include alternative action selection, replay sampling, network structure, return targets, distributional value estimates, and exploration.

2. Preliminaries

  • An MDP specifies states, actions, rewards, transitions, and a discount factor; a policy induces a value function, with action values given by the Bellman recurrence.Q-learning seeks optimal policies by updating a parameterized Q function from observed transitions using temporal differences.Source: §2 · p. 2

2.1. DQN

  • DQN uses a feed-forward network with online and target copies, a replay buffer, and minibatches of 32 for its update.The target network is synced with the online weights at regular, less frequent intervals.Source: §2.1 · p. 2

2.2. Rainbow

  • Rainbow adds double Q-learning to mitigate overestimation and prioritized replay to sample transitions in proportion to temporal-difference error.Double Q-learning decouples action selection from maximization in the target bootstrap.Source: §2.2 · p. 2
  • Rainbow also combines dueling networks and multi-step targets with distributional RL’s 51-atom categorical return estimate and noisy linear layers for exploration.The distributional approach represents the return as a distribution rather than only a scalar value.Source: §2.2 · pp. 2–3

3. The cost of Rainbow

The authors argue that the compute demands of Rainbow’s Atari evaluation make the standard benchmark costly and difficult to reproduce. They ask whether its conclusions transfer to smaller tasks, non-ALE environments, and empirical research at small to medium scale.

3. The cost of Rainbow

  • 34,200 GPU hours (or 1425 days) were estimated for Rainbow’s convincing ALE evidence, excluding hyperparameter tuning.The estimate reflects 57 Atari games, roughly five days per game, and at least five independent runs for confidence bounds.Source: §3 · p. 3
  • The paper asks whether Hessel et al. (2018)’s conclusions would hold on smaller-scale experiments and whether their results generalize beyond ALE.A third question asks whether empirical RL research restricted to small- and medium-scale environments has scientific value.Source: §3 · p. 3
  • The authors note that MuJoCo tasks from the DM control suite take about two days on the same hardware, illustrating that the expense is not limited to ALE.The comparison is tied to the paper’s stated hardware context.Source: §3 · p. 3

4. Revisiting Rainbow

The authors compare Rainbow components by adding them to DQN and removing them from Rainbow on classic-control tasks and MinAtar. Component effects vary by environment, but their overall assessment favors combining all components, while identifying distributional RL effects that warrant further investigation.

4.1. Methodology

  • The evaluation adds each Rainbow component to DQN and removes each from Rainbow, sweeping learning rates using an implementation based on Dopamine.The implementation adds noisy networks, double DQN, and dueling networks to Dopamine’s “lite” Rainbow.Source: §4.1 · p. 3
  • Classic-control experiments use four OpenAI Gym environments, two-layer 512-unit MLPs, and CPU training; LunarLander finishes in under two hours.The environments are CartPole, Acrobot, LunarLander, and MountainCar.Source: §4.1 · p. 3
  • MinAtar experiments use miniaturized versions of five ALE games and a convolutional layer followed by a linear layer.Training each MinAtar agent takes approximately 12–14 hours on a P100 GPU.Source: §4.1 · p. 3

4.2. Empirical evaluation

  • Under constant hyperparameter settings, the study evaluates component additions and removals on classic-control environments and MinAtar.The comparisons are presented in Figures 1 and 2.Source: §4.2 · p. 3, Figure 1 · p. 4, Figure 2 · p. 4
  • Across environments, component effects vary, but combining all components produces a better overall agent; distributional RL can hurt DQN or help when removed from Rainbow.The reported examples include added distributional RL hurting Acrobot and Freeway, and its removal sometimes increasing performance on MountainCar and Seaquest.Source: §4.2 · p. 4, Figure 1 · p. 4, Figure 2 · p. 4

5. Beyond the Rainbow

The authors use small- and mid-scale environments to investigate Rainbow components and their interactions, finding that component effects vary across environments and agent variants.

5. Beyond the Rainbow

  • The authors leverage the low cost of small- and mid-scale environments to conduct a thorough investigation of Rainbow's algorithmic components.Source: §5 · p. 4

5.1. Examining network architectures and batch sizes

  • Two to three layers with at least 256 units generally perform best; aside from CartPole, algorithms are generally robust to architecture dimensions, and Rainbow is robust to batch size while DQN is suboptimal below 64.Source: §5.1 · p. 5

5.2. Examining distribution parameterizations

  • The study extends Rainbow's original C51 evaluation by testing newer return-distribution parameterizations and their interactions with other Rainbow components.Source: §5.2 · p. 5

Quantile Regression for Distributional RL

Implicit quantile networks

5.3. Munchausen Reinforcement Learning

  • In classic control, M-Rainbow underperforms its non-Munchausen counterpart while M-IRainbow can gain; in MinAtar, Munchausen agents have an advantage on Asterix, Breakout, and SpaceInvaders.M-Rainbow omits C51 and double-DQN because the authors could not successfully integrate either with M-DQN.Source: §5.3 · p. 6, Figure 5 · p. 7

5.4. Reevaluating the Huber loss

  • Across 60 Atari games, Adam+MSE overwhelmingly beat RMSProp+Huber; on all tested environments, MSE beat Huber, sometimes surpassing full Rainbow.With RMSProp, Huber tends to outperform MSE, which the authors say explains DQN's original Huber choice; their experiments used Adam, unlike original DQN.Source: §5.4 · pp. 6–7

6. Putting it all together

Rainbow outperforms DQN, though its variants differ; classic-control and MinAtar environments pose distinct challenges for reinforcement-learning research.

6.1. Rainbow flavours

  • Rainbow outperforms DQN, but differences among its variants invite further investigation.Figure 7 shows two environments; full comparisons are in the appendix.Source: §6.1 · p. 7, Figure 7 · p. 8

6.2. Environment properties

  • The classic-control and MinAtar environments pose varied challenges for RL research, with fuller analysis provided in the appendix.Source: §6.2 · p. 8

Classic control

  • CartPole probes optimizer stability; Noisy networks and MSE help its learning-rate sensitivity, while distributional RL appears needed for LunarLander and Noisy networks aid sparse-reward Acrobot and MountainCar.MountainCar appears more difficult than Acrobot.Source: Classic control · p. 8

MinAtar

  • Seaquest and Freeway suit exploration research because of partial observability and sparse rewards, while MinAtar permits convolutional-network research without ALE's prohibitive expense.Source: MinAtar · p. 8

7. Conclusion

With limited compute, the authors reproduce Rainbow's findings at a high level and show that small-scale environments support more thorough empirical analysis. They recommend valuing this work alongside large-scale benchmarks to help reduce barriers for underprivileged newcomers.

  • With limited compute, the study reproduced Hessel et al.'s findings at a high level; 100 classic-control trials tightened confidence intervals after 10 trials proved inconclusive.Source: §7 · p. 8
  • Small- and medium-scale environments support more critical analysis of algorithm performance, behaviors, and intricacies, while making many independent significance trials practical.Source: §7 · p. 8
  • The authors urge retaining large-scale benchmarks while valuing small-scale empirical work and not dismissing it, arguing this can reduce barriers for underprivileged newcomers.Source: §7 · pp. 8–9

APPENDICES: Revisiting Rainbow

The appendices describe the evaluated environments and Rainbow-related algorithmic components, and provide hyperparameter, runtime, and component-impact tables.

A. Environments

  • CartPole balances a pole, Acrobot swings a double-jointed pendulum, LunarLander attempts a landing, and MountainCar builds momentum to reach a hilltop.The environment descriptions specify each task's observations, actions, rewards, or termination conditions.Source: §A.1 · p. 12, §A.2 · p. 12, §A.3 · p. 12

A.5. MinAtar Environments

  • MinAtar offers five Atari analogues on a 10x10 grid, reducing representation complexity while retaining game mechanics as much as possible.Each state has 10x10xn channels for game-specific objects.Source: §A.5 · p. 13

B.1–B.3. Rainbow components

  • Double Q-learning decouples action selection from target evaluation; prioritized replay samples by temporal-difference error, while dueling networks separate value and action advantages.The dueling streams share initial convolutional layers.Source: §B.1 · p. 14, §B.2 · p. 14, §B.3 · p. 14

B.4–B.5. Multi-step and distributional RL

  • Multi-step learning uses trajectory rewards over an n-step horizon, while distributional RL estimates return distributions, including C51's categorical model with 51 atoms.The cited descriptions present the Bellman recurrence for value distributions and the C51 parameterization.Source: §B.4 · p. 14, §B.5 · p. 14

B.6–B.7. Quantile methods

  • Quantile regression approximates returns with a uniform mixture of quantile values, while implicit quantile networks learn a continuous quantile function from uniform samples.The quantile estimates use Huber quantile-regression loss.Source: §B.6 · p. 15, §B.7 · p. 15

B.8–B.9. Noisy nets and Munchausen RL

  • Noisy nets replace epsilon-greedy exploration with noisy linear layers, and Munchausen RL adds a scaled log-policy term to the immediate reward.The noisy-net description identifies factorised Gaussian noise in Hessel et al. (2018).Source: §B.8 · p. 15, §B.9 · p. 15

B.10. Loss functions

  • MSE penalizes large errors more heavily while smoothing gradients for smaller errors, whereas Huber loss is less sensitive to outliers.The appendix introduces both losses because the experiments compare them.Source: §B.10 · p. 15

C–F. Experimental reference tables

  • Appendix tables report classic-control and MinAtar hyperparameters, runtimes, and the impact or leading component for agents and environments.The component tables cover both classic-control environments and MinAtar games.Source: Table 1 · p. 16, Table 2 · p. 17, Table 3 · p. 17

G. Examining network architectures and batch sizes

The experiments vary network depth, units, and batch size for DQN and Rainbow, then compare optimizer–loss combinations. Adam with MSE worked best on classic-control and MinAtar environments.

G. Examining network architectures and batch sizes

H. Reevaluating the Huber loss, complete results

H.1. Classic control and MinAtar results

  • Adam+MSE worked best across the classic-control and MinAtar environments.Figure 13 presents the DQN evaluation of MSE in place of Huber loss.Source: §H.1 · p. 22, Figure 13 · p. 22

H.2. Atari results

The Atari comparison tests Adam and RMSProp with Huber and MSE losses. Across the reported results, Adam with MSE performed best, contrary to the standard RMSProp-plus-Huber practice.

H.2. Atari results

  • The Atari analysis compares Adam against RMSProp under each loss, and compares MSE against Huber under each optimizer.Figures 14–17 show these pairwise comparisons; Figure 19's shaded areas represent 75% confidence intervals.Source: Figure 14 · p. 22, Figure 15 · p. 23, Figure 16 · p. 23
  • Adam with MSE produced the best Atari results, unlike the standard practice of RMSProp with Huber loss.Figure 18 compares human-normalized mean and median scores; Figure 19 shows training curves for all 60 Atari games, averaged over five runs per combination.Source: §H.2 · p. 22, Figure 18 · p. 24, Figure 19 · p. 25

I. Rainbow flavours, full results

The section presents complete results comparing DQN with the different Rainbow variants across the environments.

I. Rainbow flavours, full results

  • Figure 20 compares DQN against the various Rainbow flavours across the environments.The passage introduces these as the complete results for the comparisons.Source: Figure 20 · p. 26, §I · p. 26
Loading 2011.14826v2…