Source-linked AI summary
Quantifying Generalization in Reinforcement Learning
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, John Schulman
TL;DR
Deep RL benchmarks often reuse environments for training and testing, limiting evidence about generalization. The paper addresses this with procedurally generated environments and CoinRun, finding that agents overfit large training sets while deeper architectures improve generalization.
Problem
Deep RL agents often overfit, while common benchmarks reuse training environments for testing and therefore provide limited evidence about generalization.
Method
The paper uses procedurally generated environments with distinct training and test sets, introducing CoinRun to quantify generalization.
Results
Agents require many training environments for good generalization, and deeper convolutional architectures improve generalization performance.
Takeaways & Limitations
CoinRun enables precise measurement of overfitting and evaluation of architectural and algorithmic decisions for more generalizable agents.
Takeaways & Limitations
The suitability of alternative recurrent architectures for generalization in memory-dependent environments remains unclear and is left for future work.
Abstract
from arXiv · showhide
In this paper, we investigate the problem of overfitting in deep reinforcement learning. Among the most common benchmarks in RL, it is customary to use the same environments for both training and testing. This practice offers relatively little insight into an agent's ability to generalize. We address this issue by using procedurally generated environments to construct distinct training and test sets. Most notably, we introduce a new environment called CoinRun, designed as a benchmark for generalization in RL. Using CoinRun, we find that agents overfit to surprisingly large training sets. We then show that deeper convolutional architectures improve generalization, as do methods traditionally found in supervised learning, including L2 regularization, dropout, data augmentation and batch normalization.
1. Introduction
Deep RL agents often specialize in training environments, while standard benchmarks commonly reuse those environments for evaluation. The paper addresses this gap by separating training and test environments and examining factors that affect generalization.
- Agents trained on some environments can fail when encountering new environments, indicating over-specialization to training conditions.
- Common RL benchmarks often train and evaluate on the same environments, providing limited evidence about generalization.
- The authors train agents on CoinRun, a procedurally generated environment designed to quantify generalization.
- The paper reports that good generalization requires many more training environments than prior RL transfer work used.
- CoinRun provides a generalization metric that serves as a signal for evaluating and iterating on agents.
- Convolutional architecture and regularization choices can significantly improve generalization performance.
2. Related Work
Prior RL studies address train-test conflation through distinct environments, altered game modes, parameter ranges, procedural generation, and stochasticity. Together, these works motivate experimental protocols that directly measure generalization.
- The Sonic Benchmark separates training and testing levels to address evaluation on the test set.
- Using Atari game modes, prior work finds that L2 regularization and dropout can help agents learn more generalizable features.
- Benchmarks using parameter ranges evaluate interpolation and extrapolation, while continuous-domain studies relate generalization to the number of training seeds.
- Procedurally generated levels reveal overfitting to particular training distributions and dependence on the level generators used during training.
- Gridworld studies report that agents can memorize specific levels and that sticky actions and random starts often fail to mitigate overfitting.
- The paper extends this line of work by relating overfitting to training-environment count and testing additional methods for reducing it.
- The cited studies motivate experimental protocols that directly address generalization in reinforcement learning.
3. Quantifying Generalization
CoinRun uses procedurally generated levels to compare performance on training and unseen test sets. Agents substantially overfit small fixed training sets, while performance is best with unbounded level generation.
- 3.1. The CoinRun Environment: CoinRun levels require an agent to collect a coin while avoiding stationary and moving obstacles, with death occurring upon collision.
- 3.1. The CoinRun Environment: Deterministic level generation from seeds provides an arbitrarily large and quantifiable supply of training data.
- 3.1. The CoinRun Environment: Train and test levels come from the same distribution, so the train-test performance gap measures overfitting without test-time fine-tuning.
- 3.2. CoinRun Generalization Curves: The experiment trains agents on sets ranging from 100 to 16,000 levels and on an unbounded set, with approximately 2M unique training levels encountered in the latter condition.
- 3.2. CoinRun Generalization Curves: Substantial overfitting occurs below 4,000 training levels, remains noticeable at 16,000 levels, and is lowest with an unbounded training set.
4. Evaluating Architectures
The paper evaluates convolutional architectures using both continually changing and fixed training levels. IMPALA-CNN generalizes better than Nature-CNN, while larger architectures improve results with diminishing returns beyond IMPALA-Large.
- Architecture comparison: IMPALA-CNN agents perform better at test time than the Nature-CNN baseline across training-set sizes.
- Evaluation settings: Unbounded training measures continual generalization, but faster learning in that setting does not always correlate with better fixed-set generalization.
- Architecture comparison: IMPALA-CNN generalizes better across CoinRun levels in both unbounded and fixed-500-level settings.
- Architecture comparison: IMPALA-Large uses five residual blocks instead of three and twice as many channels at each stage.
- Limitations: Further architectural tuning may improve generalization, but increasing network size beyond IMPALA-Large produces diminishing returns and can greatly increase training time.
5. Evaluating Regularization
With separate training and test environments, the paper evaluates regularization as a way to reduce overfitting in CoinRun. L2 regularization, dropout, data augmentation, batch normalization, and increased stochasticity improve generalization, while combining several regularizers yields only a slight additional gain.
- Dropout and L2 Regularization: L2 regularization and dropout noticeably reduce the generalization gap, with best settings of w = 10^-4 and p = 0.1.Dropout has a smaller impact than L2 regularization in these experiments.
- Data Augmentation: A Cutout-like augmentation masks multiple randomly colored rectangular regions in each observation and boosts performance in CoinRun.The method is presented as a simple augmentation scheme resembling domain randomization.
- Batch Normalization: Batch normalization after every convolutional layer offers a significant generalization performance boost over the baseline.Training workers use current-batch statistics, while test workers use a moving average.
- Stochasticity: Increasing either environmental or policy stochasticity improves generalization, and each method alone provides a similar boost.Training uses 512M timesteps because increased stochasticity slows training; the methods outperform previously mentioned regularization methods in these experiments.
- Combining Regularization Methods: Combining data augmentation, batch normalization, and L2 regularization yields slightly better test performance than any method individually.The small gain suggests these regularizers may address similar underlying causes; combining stochasticity with other regularization was unsuccessful for unknown reasons.
6. Additional Environments
The paper tests overfitting beyond CoinRun using CoinRun-Platforms and RandomMazes, where agents must explore partially observed environments and use memory. Both environments reveal substantial overfitting, while the authors leave alternative recurrent architectures for future work.
- CoinRun-Platforms: CoinRun-Platforms shows overfitting up to around 4000 training levels after 2B timesteps.Training performance increases with more levels past a threshold, which the authors attribute to an implicit curriculum in the generated-level distribution.
- RandomMazes: RandomMazes retains a sizeable generalization gap even when agents train on 20,000 levels.The environment uses randomly generated mazes with partial observation of a 9 × 9 local patch.
- Discussion: Both additional environments require recurrence and memory for optimal navigation, creating generalization challenges unlike those in CoinRun.The experiments use the original IMPALA-CNN followed by an LSTM because memory is necessary for optimal exploration.
- Discussion: Although LSTM agents eventually approach an optimal policy with sufficient data and training time, their relatively poor generalization leaves the suitability of other recurrent architectures unresolved.The paper identifies alternative recurrent architectures as a direction for future work.
7. Conclusion
The procedurally generated CoinRun environment exposes substantial overfitting to fixed training sets and provides a precise way to quantify it. This metric supports evaluating architectural and algorithmic choices, with lessons intended to extend to more complex settings.
- CoinRun precisely quantifies how agents overfit to a fixed training set.
- The resulting metric enables evaluation of key architectural and algorithmic decisions.
- The authors expect lessons from CoinRun to apply in more complex settings.
A. Level Generation and Environment Details
CoinRun procedurally generates levels by sampling difficulty and conditioning level structure on that setting. Agents receive centered RGB observations, with velocity represented through stacking, recurrence, or visual markers.
- CoinRun samples each level’s difficulty uniformly from settings 1 to 3.
- Difficulty controls sections, section dimensions, and obstacle frequency, creating a useful curriculum.
- Each timestep provides a 64 × 64 × 3 RGB observation centered on the agent.
- Velocity can be supplied through frame stacking, a recurrent model, or two visual markers encoding x and y velocity.
A.2. CoinRun-Platforms
CoinRun-Platforms uses RGB observations without painted velocity and draws levels from one distribution without explicit difficulty settings. It is substantially harder than CoinRun and requires far more training to approach convergence.
- CoinRun-Platforms uses 64 × 64 × 3 RGB observations and relies on its LSTM to encode velocity.
- Unlike CoinRun, CoinRun-Platforms draws all levels from one distribution without an explicit difficulty setting.
- CoinRun-Platforms is much more difficult than CoinRun.
- 6B timesteps were needed for training to converge at a mean score of 20 per level, whereas 2B timesteps were insufficient for full convergence.
- RandomMazes also uses 64 × 64 × 3 observations for consistency, although smaller observations showed similar overfitting.
C. Data Augmentation Screenshots
The appendix presents augmented CoinRun observations alongside baseline experiment settings for CoinRun, CoinRun-Platforms, and RandomMazes. The environments use mostly shared optimization hyperparameters, with differences in worker environments and LSTM usage.
- The appendix includes example observations augmented with a modified version of Cutout.
- Baseline experiments cover CoinRun, CoinRun-Platforms, and RandomMazes.
- CoinRun omits the LSTM, while CoinRun-Platforms and RandomMazes use it.
- All three baselines use γ=.999, λ=.95, 256-step rollouts, three epochs, eight minibatches, entropy bonus .01, and Adam learning rate 5 × 10^-4.
E. Performance
The performance section reports results for CoinRun, CoinRun-Platforms with IMPALA, and RandomMazes with IMPALA, using results aggregated across multiple seeds.
- Table 1 reports CoinRun results averaged across 5 seeds.
- Table 2 reports CoinRun-Platforms results for IMPALA across 3 seeds.
- Table 3 reports RandomMazes results for IMPALA across 3 seeds.