PaperlayerView exact revision on arXiv

Source-linked AI summary

PokeRL: Reinforcement Learning for Pokemon Red

Dheeraj Mudireddy, Sai Patibandla

arXiv:2604.10812v1cs.LG

TL;DR

Pokémon Red challenges reinforcement learning with long horizons, sparse rewards, partial observability, and unusual controls. PokeRL builds a modular PyBoy-based system with loop-aware environment engineering, spatial memory, anti-spam and anti-loop mechanisms, and hierarchical training for early-game tasks. The system reports reduced looping and more movement-focused action behavior while enabling meaningful behaviors across selected objectives.

  • Problem

    Pokémon Red combines long-horizon tasks with sparse rewards, hidden game variables, and non-standard movement and menu controls that make RL training brittle.

  • Method

    PokeRL uses a PyBoy-based environment with per-map visited masks, explicit handling of double-press movement, multi-layer anti-loop and anti-spam controls, and a curriculum over early-game tasks.

  • Results

    Movement actions rose to 68.2% of actions, while A/B combined fell to 24.3% and no-op dropped to 7.5% after graduated penalties and removing Start/Select.

  • Takeaways & Limitations

    Environment-side engineering improved the stability and interpretability of training for selected objectives including house exit, Pallet Town exploration, and the first rival battle.

  • Takeaways & Limitations

    The system relies on direct memory access and handcrafted rewards, remains computationally heavy and time-expensive, and is sensitive to hyperparameters.

Abstract

from arXiv · show

Pokemon Red is a long-horizon JRPG with sparse rewards, partial observability, and quirky control mechanics that make it a challenging benchmark for reinforcement learning. While recent work has shown that PPO agents can clear the first two gyms using heavy reward shaping and engineered observations, training remains brittle in practice, with agents often degenerating into action loops, menu spam, or unproductive wandering. In this paper, we present PokeRL, a modular system that trains deep reinforcement learning agents to complete early game tasks in Pokemon Red, including exiting the player's house, exploring Pallet Town to reach tall grass, and winning the first rival battle. Our main contributions are a loop-aware environment wrapper around the PyBoy emulator with map masking, a multi-layer anti-loop and anti-spam mechanism, and a dense hierarchical reward design. We argue that practical systems like PokeRL, which explicitly model failure modes such as loops and spam, are a necessary intermediate step between toy benchmarks and full Pokemon League champion agents. Code is available at https://github.com/reddheeraj/PokemonRL

I. INTRODUCTION

Pokémon Red combines long-horizon navigation, interaction, and combat with sparse rewards, hidden state, and unusual controls. PokeRL addresses these challenges through explicit environment modeling and a curriculum over early-game objectives.

  • Pokémon Red requires coordinated navigation, interaction, and turn-based combat across tens of thousands of timesteps.
  • Key rewards such as catching a Pokémon or winning a battle may arrive only after long action sequences.
  • The agent observes only the current 2D screen while crucial variables such as HP, map ID, and party status remain hidden.
  • Movement uses a double press, and several menu buttons are useless or harmful for learning.
  • Earlier PPO attempts encountered reward exploitation, button or no-op spam, spinning from mishandled movement, and unproductive local wandering.
  • PokeRL explicitly encodes loop detection, spam penalties, and spatial memory while training a curriculum from house exit to exploration and the first rival battle.

A. Why Pokemon Red?

Pokémon Red is a compact testbed combining overworld exploration, resource management, and turn-based combat. Its opening tasks expose recurrent RL failures involving loops, sparse rewards, spam, movement semantics, and missing exploration memory.

  • Pokémon Red embeds navigation, inventory and resource management, and turn-based combat within one game environment.
  • The opening curriculum targets house exit, reaching tall grass through Pallet Town and Route 1, and defeating the first rival.
  • PPO agents can exploit local cycles that provide small positive movement rewards without meaningful progress.
  • Sparse event rewards produced highly variable returns and policies that failed to improve.
  • Action distributions collapsed onto A, Start, or no-op, creating pointless menus or idling.
  • Without visited-state memory, agents repeatedly revisited explored areas; PokeRL was designed to address these five failure modes.

III. RELATED WORK

Prior work established Pokémon Red as a challenging deep-RL setting, while related projects explored intrinsic motivation, battle-only environments, and other Pokémon titles. PokeRL focuses on engineered anti-loop and anti-spam mechanisms for early-game sequences.

  • Intrinsic motivation and Go-Explore addressed sparse-reward exploration by rewarding novelty and maintaining state memories.
  • Earlier work used PPO, dense rewards, and a 72×80 visited-mask channel to progress as far as the second gym in Pokémon Red.
  • That work also showed that reward bonuses could be exploited and that dense shaping carries both benefits and risks.
  • Battle-only Pokémon environments offer cleaner rewards and smaller action spaces, enabling near-human-parity results with offline RL and language-model policies.
  • Community efforts also applied reinforcement learning to Pokémon Gold and proposed Pokémon benchmarks for speedrunning and competitive play.
  • PokeRL differs by emphasizing anti-loop and anti-spam mechanisms, per-map centered visited masks, and an early-game curriculum rather than full completion.

IV. SYSTEM OVERVIEW

PokeRL is a PyBoy-based Gymnasium environment with memory reading, reward computation, curriculum control, and a convolutional policy. Its observations combine stacked frames with per-map spatial memory, while its action interface models the game’s controls.

  • PokeRL extends a custom Gymnasium environment around PyBoy with modules for memory reading, reward computation, and curriculum control.
  • The CNN policy receives eight channels formed from four consecutive grayscale frames and four corresponding visited-mask frames.
  • Each map uses a 72×80 binary visited mask centered at the map-entry position so it remains aligned as the camera scrolls.
  • The action space contains seven actions: four directions, A, B, and no-op; Start and Select are removed because they open menus and invite spam.
  • Each movement action issues two press-release cycles, matching Pokémon Red’s double-press mechanic so one RL action corresponds to one grid step.

D. Memory Reader

PokeRL uses PyBoy memory reads to compute environment events and organizes learning into three task-specific sequences with dense hierarchical rewards.

  • RAM addresses are read through PyBoy to compute events such as entering tall grass or starting a battle, without exposing those values directly to the policy.
  • Hierarchical Rewards: The reward design has micro, meso, and macro levels, supplemented by mild penalties for idling, repeated presses, and loops.Negative rewards were tuned to remain between −0.02 and −0.2.
  • Curriculum: Training is split into three sequences: house exit, exploration to tall grass, and the first rival battle.Each sequence uses its own configuration and save state.
  • House Exit: House-exit training starts in Red’s upstairs bedroom and terminates after an outdoor map transition or a step-limit timeout.Rewards emphasize movement and the first map transitions.
  • Exploration to Grass: Exploration training starts outside Red’s front door and ends at tall grass, Professor Oak’s scripted event, or timeout, with rewards emphasizing coverage and reaching grass.
  • First Rival Battle: The battle sequence begins at a first rival battle with a fixed starter and rewards offensive moves, opponent fainting, and victory.

A. Anti-Loop System

PokeRL counters unproductive behavior with layered loop detection and graduated anti-spam penalties that reshape both trajectories and action choices.

  • The anti-loop mechanism combines position-visit penalties, sliding-window action-pattern detection, and repeated-position loop detection.Breaking a detected action pattern yields a small bonus, while repeated patterns receive negative reward.
  • Evaluation: Loop episodes are logged using position visits and action-pattern triggers, then compared across 1,000 episodes before and after the intervention.
  • Anti-Spam: Anti-spam training uses streak counters, a −0.1 penalty after three identical presses, a −0.2 penalty beyond five, and a diversity bonus.Start and Select are removed from the action space.
  • Action Distribution: Shannon entropy increased from 1.21 bits before anti-spam to 1.82 bits afterward, indicating a more uniform action distribution.

VII. EXPERIMENTAL SETUP

The experiments train PPO agents with Stable-Baselines3 across parallel environments and evaluate changes in action distributions under anti-spam controls.

  • PPO agents use Stable-Baselines3’s built-in CNN policy with learning rate 3 ∗ 10−4, gamma = 0.999, n steps = 2048, batch size = 128, and 10 epochs per update.
  • Four parallel environments are run with DummyVecEnv, with one instance optionally rendered for video inspection.
  • Each sequence and configuration receives multiple training sessions lasting several hundred thousand timesteps, with metrics logged to TensorBoard and analysis scripts.
  • Results: 68.2% of actions became movement after anti-spam, while A/B combined fell to 24.3% and no-op fell to 7.5%.Before controls, movement comprised 27.2% of actions.
  • Results: 1.21 bits increased to 1.82 bits in action-distribution entropy, about a ≈50% relative improvement in exploration efficiency.The comparison uses the reported pre- and post-anti-spam distributions.

B. Exploration with Per-Map Visited Mask

Adding a per-map visited mask improves exploration behavior compared with grayscale frames alone, increasing coverage and unique positions while reducing revisits.

  • The ablation compares grayscale-only observations with observations augmented by a per-map visited channel over 300k timesteps per variant.
  • Exploration Metrics: Average unique positions per episode increased from 34.2 to 48.1, a +40.6% change, with the visited mask.
  • Exploration Metrics: Exploration coverage in Pallet Town rose from 12% of tiles to 41%.
  • Exploration Metrics: Revisit ratio decreased from 4.8 to 3.1, indicating fewer redundant revisits.The reported differences are attributed to the policy learning to use the mask as explored-area memory.
  • Loop Evaluation: The loop-episode comparison classifies episodes using repeated tile visits or excessive action-pattern repetition.
  • Loop Evaluation: Loop episodes fell from 41.2% without anti-loop controls to 4.7% with them, while normal episodes rose from 58.8% to 95.3%.

D. Task-Level Performance

PokeRL achieves meaningful early-game behaviors while emphasizing environment design over absolute performance. Preliminary success rates reach about 65% for house exit, 60% for reaching tall grass, and 50% for winning the first rival battle, with important stability benefits from calibrated penalties and explicit game modeling.

  • The house-exit agent succeeds in roughly 65% of episodes after 150k timesteps.
  • The exploration agent reaches tall grass in about 60% of episodes by 500k timesteps.
  • The battle agent achieves roughly 50% rival win rates after 500k timesteps with a fixed reward structure.
  • Mild penalties preserve positive learning signals, whereas large negative rewards for spam or loops initially collapsed training.Scaling penalties down by an order of magnitude preserved the incentive structure without overwhelming positive rewards.
  • Encoding double-press movement prevents superficially active behavior that produces no positional change.The environment must represent the game’s rotation-then-step mechanic explicitly.
  • A visited-mask channel supplies spatial memory without recurrent networks, while curriculum decomposition enables faster iteration and clearer debugging.
  • The system remains limited by direct memory access, handcrafted rewards, computational cost, time requirements, and hyperparameter sensitivity.

X. FUTURE WORK

Future work would unify the independently trained early-game sequences and improve long-horizon exploration. Proposed directions include stronger state-memory methods and reducing reliance on manually shaped rewards.

  • A unified environment could require one agent to coordinate house exit, overworld navigation, and battle strategy in a single long episode.
  • Exploration could be improved by augmenting the visited mask with curiosity rewards or Go-Explore-style state archives.
  • Learned reward models, including inverse RL, are proposed as a way to reduce manual reward shaping.
  • PokeRL currently demonstrates meaningful behavior on selected objectives rather than full-game completion.
  • The paper frames environment engineering, anti-loop modules, hierarchical rewards, and visited masks as pragmatic tools for other long-horizon environments.
Loading 2604.10812v1…