Source-linked AI summary
PCGRL: Procedural Content Generation via Reinforcement Learning
Ahmed Khalifa, Philip Bontrager, Sam Earle, Julian Togelius
TL;DR
PCGRL addresses how to apply reinforcement learning to game-level generation when few or no training examples exist. It formulates sequential level design as an MDP with learned policies, testing three representations across three environments; all representations can succeed with suitable rewards and episode lengths, while generated artefacts differ. The approach shifts computational cost from inference to training and supports policy-based, iterative content generation.
Problem
The paper addresses how level generation can be formulated as a tractable reinforcement-learning problem, particularly when training data are unavailable or limited.
Method
The framework learns policies that iteratively modify randomly initialized two-dimensional levels through narrow, turtle, or wide MDP representations, with task-specific rewards and change limits.
Results
All three representations can succeed across Sokoban, simplified Zelda, and maze scenarios when reward schemes and episode lengths are appropriately chosen, producing different artefacts.
Takeaways & Limitations
PCGRL searches policy space rather than complete-content space, shifting most time consumption from inference to training and enabling fast generation after training.
Takeaways & Limitations
The approach requires goals that permit many possible levels, and the Sokoban results were constrained by model capacity and a limited solver.
Abstract
from arXiv · showhide
We investigate how reinforcement learning can be used to train level-designing agents. This represents a new approach to procedural content generation in games, where level design is framed as a game, and the content generator itself is learned. By seeing the design problem as a sequential task, we can use reinforcement learning to learn how to take the next action so that the expected final level quality is maximized. This approach can be used when few or no examples exist to train from, and the trained generator is very fast. We investigate three different ways of transforming two-dimensional level design problems into Markov decision processes and apply these to three game environments.
Introduction
This paper frames game-level generation as a reinforcement-learning problem, learning policies that choose sequential design actions to maximize final level quality. It contrasts policy-space search with content-space methods and evaluates three representations across three environments.
- Introduction: The paper asks how level generation can be formulated as a tractable reinforcement-learning problem.It defines observation spaces, action spaces, and reward schemes for existing RL algorithms.
- Introduction: PCGRL searches the space of content-generating policies rather than the space of complete game content.At each step, the policy selects the action expected to produce the highest final level quality.
- Introduction: Compared with search-based methods, trained RL generators can produce new levels faster at inference, while requiring a longer training phase.Compared with supervised learning, they do not require training data and may better support interactive generation.
- Introduction: The experiments cover Sokoban, simplified Zelda, and a maze environment, using narrow, turtle, and wide representations.All three representations can succeed across the three scenarios when reward schemes and episode lengths are chosen appropriately, with differences in generated artefacts.
Background
Prior PCG research mainly treated content generation as optimization or learning from examples, while reinforcement learning was rarely applied directly. Sequential, iterative generation provides a route to formulate level design as an MDP.
- Background: Reinforcement learning was rarely applied to PCG because it was unclear how to formulate level generation as an RL problem.Earlier PCG approaches commonly used optimization, supervised learning, or other machine-learning techniques.
- Background: The PCGRL framework is presented as a new application of reinforcement learning to game-level generation.The paper positions this as addressing the previously unclear mapping from level generation to RL.
- Background: Iterative generation modifies small parts of content sequentially, including platformer-level generation with Markov Chains and LSTMs.This sequential perspective supports incremental construction rather than generating a complete level at once.
- Background: Framing generation as an MDP lets an agent make small iterative changes while rewards evaluate their effect on level quality.Related work includes MDP formulations for 3D building generation and RL systems that design cities or game content.
PCGRL Framework
PCGRL turns level construction into an iterative MDP in which an agent observes and modifies a randomly initialized level, receives quality-based rewards, and stops under episode constraints. Its narrow, turtle, and wide representations control where and how the agent can edit tiles.
- PCGRL Framework: PCGRL initializes a level with random tiles and lets the agent make small changes whose rewards reflect progress toward a target goal.The generation process also needs a halting rule, such as a limit on iterations or a reached goal.
- PCGRL Framework: The framework separates game-specific information, state representation, and the percentage of content the agent may change.The problem module supplies goals and rewards; the representation module produces observations and applies actions; change percentage limits episode modifications.
- PCGRL Framework: The agent-environment loop maps an observation and action to a new state, while the problem module evaluates the resulting map-quality change and returns a reward.Episodes may terminate early through an end signal after extensive map changes.
- Representation: Narrow agents edit a predetermined location sequence, turtle agents move locally while editing, and wide agents directly control both location and tile type.These representations define different observation and action spaces and can also be combined into hybrids.
- Change Percentage: The change percentage limits how many tiles can be modified and regulates the agent’s greediness over short-term versus more optimal solutions.Restricting actions preserves influence from the initial random layout, supporting multiple distinct generated levels rather than one convergent optimum.
Experiments
The experiments evaluate PCGRL through three representations and game-design problems, using trained agents, randomized starts, and varying inference conditions.
- The framework is implemented as an OpenAI Gym interface and tested with Narrow, Turtle, and Wide representations across Binary, Zelda, and Sokoban problems.
- Binary levels must connect empty spaces and increase the longest shortest path by at least 20 tiles.
- Zelda levels require exactly one player, door, and key, with a reachable key-then-door path of at least 16 steps and enemies more than three tiles from the player.
- The change percentage controls how many tiles may be modified, and experiments fix it at 20% during training to encourage reaction to randomized starting states.
- Training uses PPO, while Narrow and Turtle encode location by translating the map so the relevant position is centered.
Results
Across the three environments, trained agents generally produced successful levels after changing at most 40% of tiles, while representations mainly shaped design style and efficiency. Narrow and Wide had task-specific weaknesses, including longer-path difficulty for Narrow and low Sokoban success for Wide.
- Overall success: At most 40% tile changes were sufficient for successful level design, despite training with a 20% change limit.Success was measured as the percentage of generated levels satisfying each game's goal criterion.
- Narrow representation: Narrow generated mostly good levels but rarely produced solution paths longer than 16 steps and remained undertrained after 100 million frames.The authors attribute this difficulty to random tile selection and the long waits needed to visit every tile.
- Sokoban: At 100% inference change percentage, Sokoban solvability was 86.7% for Narrow, 88.3% for Turtle, and 67.5% for Wide.Most generated Sokoban levels were easy, solvable in fewer than 18 steps; the authors suggest longer training and a more powerful solver.
- Change behavior: With no inference change bound, most agents still changed relatively few tiles, indicating they reacted to input maps rather than overwriting them.This analysis used a 100% change percentage during inference.
- Representation effects: Different representations did not consistently change success rates across problems, but they produced distinct design styles, especially for Binary.Binary agents achieved similar success percentages while generating stylistically different levels.
- Wide representation: Wide used changes most efficiently and stayed closest to the original pattern, but required learning a much larger action space.Its precise change control avoided ending episodes because of excessive changes.
Discussion
The paper frames PCGRL as searching policy space rather than content space, with content generated through iterative improvement. Its practical appeal is fast inference after training, while reward design and broader content coverage remain boundaries.
- Conceptual differences: PCGRL searches content-generator space and treats generation as iterative improvement, unlike approaches that search or sample complete content.The paradigm is presented as more scalable through automatic learning.
- Scope: The paper demonstrates the technique only for game levels, leaving its application to other content types as an intended extension.The authors state that the interfaces could be adopted for other PCG problems when a problem definition and reward function exist.
- Applications: The method’s incremental edits support mixed-initiative tools in which agents suggest localized improvements or act as design brushes.The proposed interaction fits a turn-taking paradigm between human users and trained agents.
- Practical trade-offs: PCGRL shifts most computational cost from runtime inference to development-time training, addressing the expense of search-based content evaluation.This trade-off is motivated by the high cost of search when simulation is part of the evaluation function.
- Limitations: PCGRL may support runtime content generation, but designing an appropriate reward scheme remains an unresolved challenge shared with search-based PCG.The reward scheme plays a role similar to the evaluation function in search-based approaches.
Conclusion
The paper proposes reinforcement-learning-based content generation through MDP representations and evaluates it across Binary, Zelda, and Sokoban. The approach generates many playable levels, but agents struggle to design hard levels; the framework also motivates several future RL-based and mixed-initiative extensions.
- Conclusion: The framework transforms content generation into MDPs through Narrow, Turtle, and Wide representations and trains PPO models across Binary, Zelda, and Sokoban.Each problem/representation combination receives a trained model.
- Conclusion: For Binary, all models performed well given enough changes, with the main difference being the style of generated content.The reported variation concerns generated-content style rather than the basic ability to perform well.
- Conclusion: For Zelda and Sokoban, agents generated many playable levels but struggled to design hard ones.The authors suggest stronger game solvers and longer training as possible ways to address this difficulty.
- Future work: The framework could support self-play, collaborative, hierarchical, and mixed-initiative agents for future content-generation tools.Mixed-initiative extensions would prevent agents from overwriting human-created content.