Source-linked AI summary
Gotta Learn Fast: A New Benchmark for Generalization in RL
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, John Schulman
TL;DR
RL benchmarks often fail to measure generalization across related tasks, motivating a Sonic-based benchmark with train/test splits for transfer and few-shot learning. The paper evaluates baseline algorithms and finds that Joint PPO transfers useful information to test levels, while overall performance remains far from the maximum and transfer gains leave room for improvement.
Problem
Existing RL benchmarks are not ideal for testing cross-task generalization because algorithms are often evaluated on environments used during training.
Method
The paper introduces a Sonic-based meta-learning benchmark with separate training and test environments and evaluates several RL baseline algorithms.
Results
Joint PPO's test performance is superior to Rainbow and roughly 100% better than regular PPO, indicating useful information transfer from training levels to test levels.
Takeaways & Limitations
The benchmark provides a setting for studying transfer, few-shot learning, and generalization across related RL environments.
Takeaways & Limitations
The benchmark may be insufficient for fully exploring meta-learning, transfer learning, and generalization in RL, and it omits boss fights.
Abstract
from arXiv · showhide
In this report, we present a new reinforcement learning (RL) benchmark based on the Sonic the Hedgehog (TM) video game franchise. This benchmark is intended to measure the performance of transfer learning and few-shot learning algorithms in the RL domain. We also present and evaluate some baseline algorithms on the new benchmark.
1 Motivation
The report targets a gap in reinforcement learning: existing methods can solve difficult tasks with sufficient reward and interaction, but benchmarks rarely test generalization across related tasks. It introduces a benchmark designed for cross-task and few-shot RL evaluation, with additional potential for studying exploration transfer.
- Deep RL can solve difficult, high-dimensional problems with a good reward function and unlimited interaction, but intelligence also requires learning across tasks.
- Popular RL benchmarks often evaluate agents on the same environments used for training, providing inadequate tests of generalization.
- The benchmark separates training and test environments to measure cross-task generalization similarly to train/test splits in supervised learning.
- It samples many similar tasks from one distribution, making it a meta-learning dataset for evaluating few-shot reinforcement learning.
- The benchmark also presents a challenging exploration problem in which exploration skills learned on some levels can be transferred to others.
2 Related Work
Prior RL transfer studies use synthetic tasks or the ALE, but these settings make comparison difficult or offer limited opportunity for large transfer gains. The proposed benchmark is intended to combine ALE-like difficulty with many related environments and a meta-learning role analogous to supervised datasets.
- Gym Retro extends emulator-based RL environments with a flexible interface intended to make adding many environments straightforward.
- The benchmark is intended to serve the same role for RL that Omniglot and Mini-ImageNet serve for supervised learning.
- Prior transfer studies reported faster learning or improved performance across Atari, Super Mario Bros., DeepMind Lab, and MuJoCo tasks.
- Synthetic-task evaluations are ad hoc, while ALE evaluations typically show small sample-complexity gains and may limit transfer because its games differ substantially.
- The proposed benchmark is designed to be difficult like ALE while containing many similar environments like well-crafted synthetic tasks.
3 The Sonic Benchmark
The Sonic benchmark uses Gym Retro to expose emulated games as RL environments and organizes related Sonic games into level-based tasks. It provides varied platforming and timing challenges across 58 save states, with test levels differing mainly in layout from training levels.
- 3.1 Gym Retro: Gym Retro exposes emulated games as Gym environments through the gym-retro Python package and libretro API.
- 3.1 Gym Retro: Each game dataset contains a ROM, save states, scenarios, and a data file describing game state and task conditions.
- 3.2 The Sonic Video Game: The benchmark uses three similar Sonic games with shared rules and controls but subtle differences, increasing the number of available environments.
- 3.2 The Sonic Video Game: A level is defined as a (ROM, zone, act) tuple; zones vary in textures and objects, while acts within a zone mainly differ in spatial layout.
- 3.2 The Sonic Video Game: Sonic levels include challenges involving doors, levers, projectiles, and precisely timed swings, requiring exploration, navigation, and timing.
- 3.2 The Sonic Video Game: The benchmark contains 58 save states from three games, each beginning a different level, and excludes incompatible acts or boss-only stages.
- 3.2 The Sonic Video Game: The test set is formed by selecting zones with multiple acts and holding out an act, so test levels mostly reuse training objects and textures with different layouts.
3.4 Frame Skip
The benchmark advances Sonic environments with frame skipping and stochastic action delays, while defining episodes through level completion, life loss, or a timestep limit. Actions use binary button combinations, and rewards track horizontal progress.
- 3.4 Frame Skip: Frame skip 4 makes each timestep represent roughly 1/15 of a second while retaining temporal resolution intended to support Sonic gameplay.
- 3.4 Frame Skip: Sticky frame skip delays each action by one frame with probability 0.25, applying the previous action instead to introduce stochasticity.
- 3.4 Frame Skip: Episodes end when the player completes a level, loses a life, or reaches 4500 timesteps, approximately five minutes of gameplay.
- 3.4 Frame Skip: Agents cannot use special APIs to reset episodes before one of the specified done conditions occurs.
- 3.4 Frame Skip: The benchmark omits boss fights and defines completion before them, focusing evaluation on exploration, navigation, and speed.
- 3.4 Frame Skip: At each timestep, agents choose binary button vectors, with eight essential combinations identified for Sonic and UP generally ignored.
3.8 Rewards
Sonic assigns reward from horizontal progress and level completion, while evaluation aggregates per-level mean scores across held-out levels. Because progress rewards can make necessary backward movement look harmful, the baselines preprocess rewards, and evaluation uses a fixed timestep limit to assess rapid transfer.
- Reward design: Reward combines normalized horizontal offset with a completion bonus, yielding 9000 at the predefined endpoint and up to 1000 for instant completion.The completion bonus decreases linearly to zero at 4500 timesteps, encouraging faster level completion.
- Reward design: The dense reward favors rightward movement, but successful trajectories may require prolonged backward movement that produces negative immediate reward.Figure 2 illustrates rightward, leftward, and recovery segments before cumulative reward surpasses its earlier level.
- Reward design: Baseline agents preprocess rewards so backward movement is not penalized, but the resulting signal does not indicate when or how to move backward.This preserves the challenge of discovering the appropriate backward maneuver rather than directly encoding it.
- Evaluation: Evaluation averages episode rewards over 1 million timesteps per test level, then averages those per-level mean scores into an aggregate metric.Test levels are played separately, preventing information flow between them during evaluation.
- Evaluation: The fixed timestep limit makes rapid transfer relevant because unlimited interaction provides less reason to rely on meta-learning or transfer learning.The benchmark chooses 1 million timesteps so modern RL algorithms can make some progress while retaining a limited-time evaluation setting.
4 Baselines
The benchmark evaluates baseline RL agents on held-out Sonic levels, including independent methods, a scripted reward-only method, and transfer through joint training followed by fine-tuning. Joint PPO transfers useful information from training levels, but the overall results leave substantial room for improvement.
- Baseline evaluation: The baselines include human players, methods that ignore the training set, and joint training followed by fine-tuning.Table 1 aggregates baseline test scores, while Figure 3 compares their learning curves.
- Rainbow: Rainbow is a DQN variant retained for strong performance on high-dimensional video-game environments, with a seven-action space and reward preprocessing based on maximum x-position.The preprocessing avoids punishing backtracking and provides a sizable performance boost.
- JERK: A Scripted Approach: JERK ignores observations, uses only rewards, and increasingly replays action sequences with high running-mean rewards.It achieves high benchmark rewards without deep learning and performs better than regular PPO.
- Joint PPO: Joint PPO trains one policy across all training levels, then uses that policy as the initialization for fine-tuning separately on each test level.The same evaluation procedure is retained aside from the initialization.
- Joint PPO: After roughly 50 million joint-training timesteps, training-set improvement no longer improves test performance, while zero-shot performance changes little after the first few million timesteps.This pattern is described as the point where the model starts to overfit.
- Transfer results: Joint PPO outperforms Rainbow and is roughly 100% better than regular PPO, indicating useful information transfer from training levels to test levels; Joint Rainbow does not analogously outperform Joint PPO.A Reptile variant yielded worse results, and the authors note that transfer gains remain close to learning-from-scratch results overall.
5 Discussion
The benchmark and baselines establish a foundation for studying transfer, few-shot learning, and generalization in RL, while leaving substantial room for improvement and raising important scope concerns.
- The benchmark and baseline algorithms are presented as a foundation for future research on transfer learning, few-shot learning, and generalization in RL.
- The best transfer-learning results are not much better than the best results from learning from scratch.
- Baseline scores remain below the designed maximum achievable range of 9000–10000.
- Benchmark limitations: The benchmark may be insufficient for fully exploring meta-learning, transfer learning, and generalization until its possible problems are tested further.
- Benchmark limitations: Sonic-specific hacks may outperform general meta-learning approaches, limiting how broadly benchmark performance reflects general methods.
- Generalization limitations: Exploration strategies effective in Sonic may not generalize beyond Sonic, while level mastery may reward few-shot memorization rather than other capabilities.
A Detailed Algorithm Descriptions
The appendix details JERK and Joint Rainbow, including their control parameters, trajectory-replay procedure, and distributed gradient-aggregation procedure.
- JERK: JERK uses an initial exploitation fraction β, jump and movement durations, jump probability, and an evaluation timestep limit.The experiments set β = 0.25, Jn = 4, Jp = 0.1, Rn = 100, and Ln = 70.
- JERK: JERK increasingly replays the best stored trajectory while updating its mean reward from subsequent episode outcomes.The replay probability is β + T/Tmax, and episodes are padded with no-ops when needed.
- JERK: When progress stalls, JERK reverses direction and continues periodic jumping before storing the highest-reward prefix from the episode.The stored trajectory ends at the timestep with the highest cumulative reward.
- Joint Rainbow: Joint Rainbow samples N transitions per worker, computes prioritized replay loss, averages gradients across workers, and updates parameters with Adam.The experiments set N = 256.
B Plots for Multiple Seeds
The appendix provides test learning curves for JERK, PPO, Rainbow, Joint PPO, and Joint Rainbow, with each algorithm evaluated across three random seeds.
- Test learning curves are reported for JERK, PPO, Rainbow, Joint PPO, and Joint Rainbow.
- Each algorithm’s test learning curves are based on three different random seeds.
C Scores on Test Set
The appendix lists detailed test-set evaluation results separately for Rainbow, JERK, PPO, Joint PPO, Joint Rainbow, and human players.
- Detailed evaluation results are provided for Rainbow, JERK, PPO, Joint PPO, Joint Rainbow, and humans.
D Scores on Training Set
Tables 8 and 9 report final performance for joint Rainbow and joint PPO models across environments, using the last 10 episodes and run-based standard-deviation error margins.
- Table 8 reports final performance for the joint Rainbow model across each environment.
- Both tables summarize performance over the last 10 episodes for each environment.
- Error margins use standard deviation across three runs for Rainbow and two runs for PPO.
- Table 9 reports final performance for the joint PPO model across each environment.