PaperlayerView exact revision on arXiv

Source-linked AI summary

Playable Game Generation

Mingyu Yang, Junyou Li, Zhongbin Fang, Sheng Chen, Yangbin Yu, Qiang Fu, Wei Yang, Deheng Ye

arXiv:2412.00887v1cs.AI

TL;DR

Playable game generation requires real-time interaction, sufficient visual quality, and accurate simulation of game mechanics, but existing approaches often lack real-time capability or mechanics accuracy. PlayGen combines game-data generation, an autoregressive diffusion model, and playability evaluation. On 2D and 3D games, it reports real-time interaction, adequate visual quality, and accurate mechanics sustained beyond 1000 frames on an NVIDIA RTX 2060.

  • Problem

    Existing game-generation approaches often lack real-time interaction or accurate simulation of interactive mechanics, both needed for playable games.

    Source: §1 · p. 2

  • Method

    PlayGen combines game-data generation, an autoregressive diffusion model, and an automated playability evaluation framework.

    Source: §1 · p. 3

  • Results

    PlayGen sustains real-time interaction, sufficient visual quality, and accurate mechanics over 1000 frames on Super Mario Bros and Doom using an NVIDIA RTX 2060 GPU.

    Source: §5 · p. 14

  • Takeaways & Limitations

    PlayGen demonstrates playable game generation across 2D Super Mario Bros and 3D Doom, with performance sustained over extended gameplay.

    Source: §5 · p. 14

    The visual-quality metrics may not fully reflect quality at long prediction lengths because environment randomness can make generated and ground-truth frames differ substantially.

    Source: §4.1.1 · p. 10

Abstract

from arXiv · show

In recent years, Artificial Intelligence Generated Content (AIGC) has advanced from text-to-image generation to text-to-video and multimodal video synthesis. However, generating playable games presents significant challenges due to the stringent requirements for real-time interaction, high visual quality, and accurate simulation of game mechanics. Existing approaches often fall short, either lacking real-time capabilities or failing to accurately simulate interactive mechanics. To tackle the playability issue, we propose a novel method called \emph{PlayGen}, which encompasses game data generation, an autoregressive DiT-based diffusion model, and a comprehensive playability-based evaluation framework. Validated on well-known 2D and 3D games, PlayGen achieves real-time interaction, ensures sufficient visual quality, and provides accurate interactive mechanics simulation. Notably, these results are sustained even after over 1000 frames of gameplay on an NVIDIA RTX 2060 GPU. Our code is publicly available: https://github.com/GreatX3/Playable-Game-Generation. Our playable demo generated by AI is: http://124.156.151.207.

1 Introduction

Playable game generation must sustain real-time interaction, sufficient visual quality, and accurate interactive mechanics, yet prior systems often lack real-time capability or faithful mechanics. PlayGen combines data generation, an autoregressive diffusion model, and automated playability evaluation; it is tested on 2D and 3D games.

1 Introduction

  • Prior game-generation systems either lack real-time interaction or fail to accurately simulate game interactions, leaving games not truly playable.Playability requires real-time response to player actions, sufficient visual quality, and precise mechanics simulation sustained over extended gameplay.Source: §1 · p. 2
  • Game generation must support arbitrary user actions while maintaining visual quality, basic game rules, and accurate mechanics, and playability evaluation remains underexplored.The paper frames accurate mechanics simulation as a more pressing unresolved challenge than visual quality or frame rate.Source: §1 · p. 2
  • PlayGen combines an autoregressive diffusion model trained on VAE-compressed latents with diverse, balanced interaction data and a playability evaluation method.The method also uses self-supervised long-tailed sample learning for rare interactions and an RNN-like architecture for extended gameplay.Source: §1 · p. 2
  • After over 1000 frames, PlayGen maintains accurate interactive mechanics while achieving real-time interaction on an NVIDIA RTX 2060 and adequate visual quality.With denoise sampling timesteps set to 4, interaction speed nearly doubles, with a 1.4% to 1.8% visual-quality decrease and a 0.2% reduction in mechanics accuracy.Source: §1 · p. 2
  • PlayGen is evaluated on Super Mario Bros and Doom, and the paper reports extensive testing on established 2D and 3D game benchmarks.The benchmarks test effectiveness and reliability.Source: §1 · pp. 2–3
  • PlayGen contributes an automated evaluation system for generation efficiency, visual quality, and the accuracy of simulated interactive mechanics.The system is intended to streamline evaluation.Source: §1 · p. 3

2 Related Work

Related work covers neural-network game simulation, conversion of media into playable games, and video generation. PlayGen is distinguished by its focus on sustained real-time interaction, accurate game mechanics, and automated playability evaluation.

Game Simulation with Neural Networks

  • GameNGen simulates Doom at 20 FPS but its data collection does not ensure coverage and balance, resulting in inaccurate interactive mechanics; PlayGen reports 20 FPS on Doom and Super Mario Bros on an RTX 2060 while maintaining mechanics accuracy over extended play.After 1000 frames, PlayGen's interactive-mechanics accuracy decreases by 0.2%.Source: §2 · p. 3

Media to Playable Games Conversion

  • Prior media-to-game conversion methods lack real-time interaction or require pre-input action sequences, while PlayGen offers sustained 20 FPS operation.Genie is limited to 32 frames, and its actions lack meaningful impact.Source: §2 · p. 3

Playablity Evaluation

  • PlayGen introduces an automated playability evaluation method covering generation efficiency, image quality, and interactive-mechanics accuracy, addressing a gap in related work that mostly relies on human evaluation.Source: §2 · p. 3

Video Generation

  • Video-generation work prioritizes resolution, length, and control-signal alignment, whereas PlayGen prioritizes real-time performance and accurate interactive mechanics.Video-generation methods often take several minutes or longer and typically use larger models and datasets.Source: §2 · p. 3

Related Work

  • PlayGen's overall framework comprises hybrid-agent data generation with cluster-based balancing, a VAE-and-DiT diffusion model, and VAM-based playability evaluation.The evaluation uses action-aware metrics to assess interactive-mechanics accuracy.Source: Figure 2 · p. 4

3 Method

PlayGen learns a game transition model from generated transition data, using data coverage and balancing alongside an autoregressive latent diffusion model. Its evaluation measures real-time interaction, visual quality, and action-execution accuracy.

3.2 Data Generation

  • Cluster-based sampling targets a balanced transition distribution by grouping samples using transition-characteristic features and selecting counts to approximate a target distribution.The method solves for non-negative cluster sample counts and then samples from each cluster.Source: §3.2 · pp. 4–6
  • Diverse collection combines random actions with optional expert or reinforcement-learning agents, recording observations, actions, and game information across episodes.Random exploration and agent-guided progress aim to cover game scenarios without requiring complex domain-specific rewards.Source: §3.2 · p. 5

3.3 Game-Generative Model

  • The game-generative model uses a VAE to encode observations into latents and an autoregressive latent diffusion model to predict subsequent game states while retaining past context.A hidden state summarizes current latent information and past memory; inference uses DDIM sampling, with four timesteps supporting 20 FPS on an NVIDIA RTX 2060 for Super Mario Bros and Doom.Source: §3.3 · pp. 6–7

3.4 Playability-Based Evaluation

  • A Valid Action Model recognizes actions from adjacent frames, and ActAcc measures agreement between recognized and executed actions to evaluate interactive mechanics.The VAM is trained on the same balanced dataset, using frames before and after the current frame as context.Source: §3.4 · p. 8
  • ProbDiff complements ActAcc by comparing predicted-action and ground-truth-action probabilities when different actions can yield identical frames.The authors use both metrics because unrelated or very low-quality frames can produce misleading ProbDiff scores, while ActAcc remains low when the VAM fails to recognize actions.Source: §3.4 · p. 8

4 Experiments

Experiments assess PlayGen’s playability across visual quality, interactive-mechanics accuracy, and interaction efficiency. The evaluation uses Super Mario Bros and Doom, with game-specific transition datasets for training.

4 Experiments

  • Experiments assess visual quality, interactive-mechanics accuracy, and interaction efficiency as the three playability dimensions.The experiments also analyze PlayGen’s components, action-aware metrics, and balanced data sampling.Source: §4 · p. 8
  • Validation uses the 2D game Super Mario Bros and the 3D game Doom, with 50M balanced transitions sampled from 236M collected frames and 200M from 900M, respectively.Each sample contains 200 timesteps, and training and test frames are 128 × 128 pixels.Source: §4 · p. 8
  • Inference receives only the first ground-truth frame and subsequent ground-truth actions as context.The action-aware metrics use a 32-frame sequence with eight preceding and eight succeeding context frames, so they theoretically require sequences longer than 17 frames.Source: §4 · p. 9

4.1 Quantitative Results

PlayGen meets the three playability criteria—real-time interaction, sufficient visual quality, and accurate mechanics simulation—and sustains these results beyond 1000 frames.

4.1 Quantitative Results

  • PlayGen sustains real-time interaction, sufficient visual quality, and accurate interactive-mechanics simulation even after over 1000 frames.Source: §4.1 · p. 9

4.1.1 Visual Quality

  • For prediction lengths under 128, PlayGen meets visual-quality thresholds on all four metrics for Super Mario Bros and PSNR/FID for Doom; Super Mario Bros PSNR exceeds 30.Source: §4.1.1 · pp. 9–10
  • At longer prediction lengths, visual-quality metrics worsen as prediction errors accumulate; randomness can make them poorly reflect quality, though the 1024-versus-128 video-metric difference is not substantial.Source: §4.1.1 · p. 10

4.1.2 Accuracy of Game Interactive Mechanics

  • Across both games and all tested lengths, AccAct > 0.789 and ProbDiff < 0.065; from 32 to 1024 frames, ActAcc reduction is <0.036 and ProbDiff reduction <0.009.Source: §4.1.2 · p. 11

4.1.3 Interaction Efficiency

  • On an NVIDIA RTX 2060, 4 denoise sampling timesteps reaches 20 FPS while meeting all playability criteria for Super Mario Bros; Doom misses strict LPIPS and FVD thresholds.Source: §4.1.3 · p. 11

4.2 Analysis

PlayGen improves playability through broader transition coverage, balanced data sampling, and learning rare transitions, while ActAcc and ProbDiff provide complementary evaluations of interactive mechanics. Balanced sampling improves reported metrics across Super Mario Bros and Doom.

4.2.1 How PlayGen Progressively Achieves Playability

  • Diverse data collection broadened transition coverage, yielding richer backgrounds and gameplay frames that better adhere to game mechanics.Without it, generated frames lacked elements such as clouds, monsters, and pipes, and showed an unreasonable brick sequence.Source: §4.2.1 · p. 11, Figure 4 · p. 12
  • Balanced data sampling corrected a left-jump action that otherwise produced an obstructed, unusually short jump.The authors attribute the failure to an imbalance in collected actions, with far less data for leftward movement than rightward movement.Source: §4.2.1 · pp. 11–12, Figure 4 · p. 12
  • Self-supervised long-tailed transition learning addressed the rare mushroom-consumption transition, allowing Mario to grow after consuming a mushroom.The authors present this as the remaining failure after common mechanics could be simulated.Source: §4.2.1 · p. 12, Figure 4 · p. 12

4.2.2 Rationality of Action-Aware Metrics

  • The action-aware metrics identified a bad case, with ActAcc = 0.3 described as very low and ProbDiff = 0.411 described as very high.In a separate case, generated frames were identical to ground-truth frames, and the metrics were ActAcc = 1 and ProbDiff = 0.Source: §4.2.2 · p. 14
  • When airborne actions did not affect subsequent frames, ActAcc was 0.6 despite good mechanics simulation, while ProbDiff remained low at 0.089.The cases support using ProbDiff to complement ActAcc, and the metrics’ assessments accorded with human visual assessments.Source: Figure 5 · p. 13, §4.2.2 · p. 14

4.2.3 The Impact of Balanced Data Sampling

  • 1%–46.1% across Super Mario Bros metrics and 1.9%–57.7% across Doom metrics improved with balanced data sampling.Table 3 compares models trained on original datasets of 236M and 900M frames with balanced datasets of 50M and 200M frames, respectively.Source: Table 3 · p. 13, §4.2.3 · p. 14

5 Discussion

PlayGen is validated on Super Mario Bros and Doom, with real-time interaction, visual quality, and interactive-mechanics accuracy sustained beyond 1,000 frames on an NVIDIA RTX 2060 GPU. The discussion identifies long-term memory precision as a limitation and outlines future work on quantitative ablations, more complex games, and prompt customization.

Discussion

  • 20 FPS, PSNR 33.81 for Super Mario Bros, and PSNR 23.81 for Doom are reported, with accurate mechanics sustained over 1,000 frames on an NVIDIA RTX 2060 GPU.The code, trained models, and an interactive demo are also made available.Source: §5 · p. 14
  • Long memories can become inaccurate: similar Doom paths may lead to the wrong room because the RNN structure retains memory length but not precise long-term memory.The authors describe distinguishing similar historical information as challenging.Source: §5 · p. 14
  • The authors plan quantitative ablations on data coverage, long-tailed transition learning, and model architecture because current playability analysis lacks quantitative results.Source: §5 · p. 14
  • Super Mario Bros and Doom are representative 2D and 3D scenarios, but their visual quality is relatively low and their interactive mechanics are simple.Future work would explore more complex mechanics and higher-resolution games akin to AAA games.Source: §5 · p. 14
  • PlayGen currently generates games frame by frame through actions; the authors aim to add prompt control by expanding the model and creating a prompt dataset.Source: §5 · pp. 14–15

A.1 Model Hyperparameter Setting

The appendix points to Table 4 for the VAE and LDM hyperparameters, including training, architecture, and latent-shape settings.

A.1 Model Hyperparameter Setting

  • Table 4 lists VAE and LDM hyperparameters, including batch size 32, 55M parameters, and learning rate 4.5e-6.Source: Table 4 · p. 18, §A.1 · p. 18
  • The listed architecture settings include 128 channels, channel multiplier [1, 2, 4], and 2 ResNet blocks.Source: §A.1 · p. 18
  • The table specifies latent shape 4×32×32, KL loss weight 1e-6, and perceptual loss weight 1.Source: §A.1 · p. 18

A.2 Data Collection

Data collection uses 200-timestep samples in both Super Mario Bros and Doom, with game-specific choices to broaden coverage or emphasize selected actions.

A.2 Data Collection

  • Super Mario Bros samples are truncated at 200 timesteps regardless of death or victory, and agents begin at random map positions and random small, large, or fire status.Source: §A.2 · p. 18
  • After dying or winning in Super Mario Bros, the agent respawns at the map's initial point in small status, following the game rules.Source: §A.2 · p. 18
  • Doom samples also span 200 timesteps and start at random map positions; each episode randomly selects a preferred action that is more likely to be chosen during that episode.Source: §A.2 · p. 18

B More Visualization Results

The appendix presents additional PlayGen visualizations for Super Mario Bros and Doom, including frames generated from an initial frame and supplied actions.

B More Visualization Results

Related papers

Loading 2412.00887v1…