Source-linked AI summary
Increasing Generality in Machine Learning through Procedural Content Generation
Sebastian Risi, Julian Togelius
TL;DR
Machine-learning systems often overfit fixed training environments, limiting generalization when tasks or conditions change. This review surveys procedural content generation and its hybridization with machine learning as a toolbox for producing varied training data and environments, including domain randomization that supported simulated-to-real robotic transfer.
Problem
Machine-learning systems can overfit training examples or environments and generalize poorly when the task or environment changes.
Method
The review synthesizes PCG history and methods hybridized with machine learning, including data augmentation, domain randomization, learned representations, and search.
Results
PCG-based variation supported robust behavior across environments, including a simulated robot hand that manipulated a Rubik’s cube in the real world.
Takeaways & Limitations
PCG provides machine-learning researchers with tools for creating training situations that may encourage more general policies and support transfer across environments.
Takeaways & Limitations
The review identifies gradual environment complexification as an outstanding challenge for generating appropriately difficult problems throughout lifelong learning.
Abstract
from arXiv · showhide
Procedural Content Generation (PCG) refers to the practice, in videogames and other games, of generating content such as levels, quests, or characters algorithmically. Motivated by the need to make games replayable, as well as to reduce authoring burden, limit storage space requirements, and enable particular aesthetics, a large number of PCG methods have been devised by game developers. Additionally, researchers have explored adapting methods from machine learning, optimization, and constraint solving to PCG problems. Games have been widely used in AI research since the inception of the field, and in recent years have been used to develop and benchmark new machine learning algorithms. Through this practice, it has become more apparent that these algorithms are susceptible to overfitting. Often, an algorithm will not learn a general policy, but instead a policy that will only work for a particular version of a particular task with particular initial parameters. In response, researchers have begun exploring randomization of problem parameters to counteract such overfitting and to allow trained policies to more easily transfer from one environment to another, such as from a simulated robot to a robot in the real world. Here we review the large amount of existing work on PCG, which we believe has an important role to play in increasing the generality of machine learning methods. The main goal here is to present RL/AI with new tools from the PCG toolbox, and its secondary goal is to explain to game developers and researchers a way in which their work is relevant to AI research.
1 Introduction
PCG algorithmically creates game content for replayability, lower production and storage costs, and new game possibilities. The review connects PCG methods with machine learning’s generalization problem, arguing that procedurally varied training situations can support more general policies.
- 1 Introduction: PCG algorithmically creates levels, quests, maps, characters, or rules at runtime or design time, supporting replayability, lower costs, reduced storage, and new game types.It generates content rather than the game engine itself.
- 1 Introduction: Machine-learning systems often perform well on training environments but overfit and generalize poorly when tasks or environments change.The review identifies this as a significant problem, especially in deep reinforcement learning.
- 1 Introduction: PCG-based training creates additional data or environments so systems can learn general task properties rather than spurious features of fixed examples.Examples include data augmentation and randomized training environments.
- 1 Introduction: A robot hand trained in simulation manipulated a Rubik’s cube in the real world after training across varied environments.The example illustrates transfer from simulated training to real-world behavior.
- 1 Introduction: The review surveys PCG history and hybrid PCG–machine-learning methods to give ML researchers tools for generalization and game researchers perspectives from ML.It focuses especially on learning centered on agents in simulated physical environments.
2 Classic Procedural Content Generation
Classic PCG emerged in games to generate replayable, unpredictable, and space-efficient worlds. These systems range from fixed-time constructive generators to search-based methods that evaluate candidate content.
- 2 Classic Procedural Content Generation: Rogue generated new multi-level dungeons for every playthrough, establishing a design paradigm motivated by replayability and unpredictability.Its generation approach influenced the roguelike genre.
- 2 Classic Procedural Content Generation: Elite recreated thousands of planets from a random seed, allowing a large game world to fit within 64 kilobytes of memory.The world was regenerated when each star system was visited.
- 2 Classic Procedural Content Generation: Civilization and Minecraft use PCG to create new worlds for players to explore and contest at the start of each game.Other games apply PCG more peripherally, such as for sidequest generation.
- 2 Classic Procedural Content Generation: Constructive PCG generates content in fixed time without iteration or search, using techniques such as fractal noise and L-systems.These methods support content such as textures, heightmaps, vegetation, and cave systems.
- 2 Classic Procedural Content Generation: Table 1 contrasts hand-designed and learned representations with rule-based generation, resampling, evolutionary search, reinforcement-learning search, and gradient-based generation.It also distinguishes domain-randomization strategies by whether they sample or search a hand-coded representation.
3 “PCG” in Machine Learning: Data Augmentation and Domain Randomization
Machine-learning research uses PCG-like data augmentation and domain randomization to increase diversity, reduce overfitting, and support transfer across environments. Methods range from uniform sampling to guided or automatically expanded variation during training.
- 3 “PCG” in Machine Learning: Data Augmentation and Domain Randomization: Stylized ImageNet training increased a network’s shape bias, accuracy, and robustness compared with training on standard ImageNet images.The same network architecture was trained with images whose textures were deliberately altered.
- 3 “PCG” in Machine Learning: Data Augmentation and Domain Randomization: Domain randomization trains one policy across simulated environments with varied properties to improve robustness and facilitate transfer to the real world.Simulation can make training faster, cheaper, more scalable, and easier to supervise.
- 3 “PCG” in Machine Learning: Data Augmentation and Domain Randomization: Randomized simulation enabled object detection and quadrotor navigation in the real world without additional training.The examples varied visual conditions such as colors, textures, camera positions, lighting, and rendering settings.
- 3 “PCG” in Machine Learning: Data Augmentation and Domain Randomization: Uniform domain randomization samples each parameter within a fixed range, whereas guided randomization focuses variation on aspects that improve model generality.Guided methods aim to save computation by concentrating training on useful task variations.
- 3 “PCG” in Machine Learning: Data Augmentation and Domain Randomization: Automatic Domain Randomization increases environmental variation as performance improves, helping a simulated robot hand adapt to situations including real-world transfer.The Rubik’s-cube size range expanded from 5.7 cm initially to 5.47–6.13 cm later in training.
- 3 “PCG” in Machine Learning: Data Augmentation and Domain Randomization: The review argues that sophisticated PCG algorithms could improve ML generality beyond current domain-randomization methods.It highlights search-based PCG and learned PCG representations as promising directions.
4 AI-driven Procedural Content Generation
AI-driven PCG combines search, constraints, and machine learning to generate functional game content, while addressing the difficulty of producing artifacts that both resemble examples and remain playable.
- 4.1 Search-based PCG: Search-based PCG has generated levels, maps, and other specific artifacts more effectively than complete video games with rules.Applications include StarCraft maps, Super Mario Bros, Doom, and Angry Birds levels, as well as particle effects, character classes, and flowers.
- 4.1 Search-based PCG: Search-based PCG evaluates candidate artifacts with functions based on gameplay, direct observation, or learned estimates of player experience.The evaluation function assigns a number or vector representing artifact desirability.
- 4.1 Search-based PCG: Constraint-based PCG efficiently searches for artifacts satisfying explicit constraints, but is difficult to combine with simulation-based evaluation.Answer set programming is presented as an example of solver-based PCG.
- 4.2 PCG via machine learning: Functional content must satisfy algorithmically verifiable requirements such as playability, making it more demanding than pictorial content generation.A visually plausible level can still be unusable if, for example, it contains an impossible jump.
- 4.2 PCG via machine learning: Training GANs on functional artifacts does not guarantee functional outputs because models may learn local appearance patterns without long-range or counting-based constraints.Generated artifacts can look right while failing to function well; bootstrapping verified artifacts into the training set is one proposed countermeasure.
- 4.2 PCG via machine learning: Machine learning can provide a representation for search, allowing optimization within a learned content space, while reinforcement learning treats generation as sequential artifact modification guided by reward.Latent-space evolution illustrates the representation-and-search approach, whereas reinforcement learning shifts computational expense toward training.
5 Procedurally generated learning environments
Procedurally generated learning environments support curriculum construction and broader evaluation by varying task content and difficulty. Evidence spans generalization to unseen levels, robust game-playing policies, and performance gains from progressive difficulty.
- 5 Procedurally generated learning environments: PCG-based curricula can scaffold learning by automatically generating progressively more difficult environments rather than relying entirely on hand-designed training sequences.MCC co-evolves agents and environments, while POET uses solutions to one obstacle course as stepping stones for another.
- 5 Procedurally generated learning environments: Procedurally generated environments can increase reinforcement-learning generality by exposing agents to many varied levels and requiring strategies that transfer beyond training environments.Examples include unseen video-game levels, procedurally generated racing tracks, and environments demanding more sophisticated neural architectures.
- 5 Procedurally generated learning environments: Training on varied indoor and outdoor maps enabled Quake III Capture the Flag agents to learn policies robust to map and player-count variations.The agents also learned to track map locations through external memory systems.
- 5 Procedurally generated learning environments: Progressive PCG increased training performance with less data by changing level difficulty in response to agent performance.The approach is described as a response to the large number of levels required by earlier variety-based training methods.
- 5 Procedurally generated learning environments: Highly varied training environments show initial evidence of fostering recurrent-network meta-learning and adaptation to situations absent during training.The review identifies broader environment encodings as a potential route to further improving agent generality.
- 5 Procedurally generated learning environments: The reviewed approaches differ in how environments are produced: PPCG uses a rule-based generator, whereas POET and MCC use hand-designed representations with different evolutionary roles.The comparison is summarized as a distinction between evolving levels and generating them through rules.
6 Looking forward: Opportunities and Challenges
The review identifies opportunities for PCG to support more general, adaptive machine learning, while highlighting unresolved challenges in generating suitable environments and complete games. It proposes increasingly complex PCG-based benchmarks and more diverse training environments, but notes that current methods remain limited in complete-game generation, lifelong learning, and real-world transfer.
- 6.2 Generating Complete Games: Generating complete games remains substantially harder than generating particular content, with many results still unplayable or uninteresting because games are complex and difficult to evaluate.The review identifies complete games as including rules, topology, visuals, and other components.
- 6.3 Lifelong generation for lifelong learning: Lifelong learning requires environments that gradually grow or complexify so agents face appropriately difficult and varied challenges over long sequences of learning.The review presents mechanisms for generating the right problems at the right time as an outstanding research challenge.
- 6.4 New PCG-based RL benchmarks: PCG-based benchmarks should make environments and rewards nonstationary, increasing in complexity and adapting rules according to agents’ performance and learning capacity.The proposed benchmarks are intended to test adaptation both between trials and within trials.
- 6.5 From simulation to the real world: Procedurally generated environments can help transfer robot policies across the reality gap, but current approaches remain limited to laboratory settings and do not yet handle real-world messiness and task diversity.More diverse simulated environments are proposed as a way to improve transfer across a wider range of physical tasks.
- 6.5 From simulation to the real world: A central open challenge is creating large, diverse training environments that reduce the content gap between synthetic scenarios and the situations agents encounter in the real world.The review frames this as a challenge alongside developing more sophisticated machine learning models.