PaperlayerView exact revision on arXiv

Source-linked AI summary

PlayTrain: An Efficient Reinforcement Learning Framework for LLM-Generated Adaptable JavaScript Games

Ryan Truong, Lance Ying, Samuel J. Gershman, Kazuki Irie

arXiv:2609.09059v2cs.LG

TL;DR

JavaScript has been considered unsuitable for RL because slow browser interactions bottleneck training. PlayTrain pairs LLM-generated, adaptable JS games with an efficient backend; its 24-game IMPALA/Nature-CNN suite averages 1.07 M agent-steps per second.

  • Problem

    JavaScript has been considered unsuitable for RL because slow browser-based interactions bottleneck practical training throughput, while existing environment variations have been limited.

    Source: §1 · p. 2

  • Method

    PlayTrain uses LLMs to generate and modify JavaScript games and provides an efficient backend for training agents on those games, which are also playable in browsers.

    Source: §1 · p. 3, §5 · p. 10

  • Results

    PlayTrain trains its 24-game suite at an average of 1.07 M agent-steps per second under IMPALA with the Nature-CNN encoder; its JS replicas outpace ALE on all eight shared games and ProcGen on 14 of 16.

    Source: §3.1 · p. 6

  • Takeaways & Limitations

    Researchers can adapt games or create novel test environments by changing visuals, physical parameters, mechanics, or game dynamics, typically through a natural-language prompt.

    Source: §1 · p. 3

    PlayTrain focuses on 2D games of a few hundred lines, and current LLMs cannot reliably generate full-fidelity modern console games.

    Source: §4 · p. 9

Abstract

from arXiv · show

While many video-game environments (VGEs) have played crucial roles in advancing reinforcement learning (RL), developing novel VGEs or modifying existing ones to support new features, has been a laborious process requiring extensive hand-coding. Here we present PlayTrain, an RL framework that combines the abilities of large language models (LLMs) to robustly generate JavaScript (JS) games from a minimal human prompt, and an efficient pipeline that can run any JS game in a standard 'gym' environment. Not only are recent LLMs particularly good at writing JS code, but the JS format also allows users to easily play generated VGEs, while PlayTrain enables us to train RL agents on the exact same games. We demonstrate multiple use cases of PlayTrain, including cloning well-known Atari and ProcGen games in simple JS, where PlayTrain trains pixel-based agents end-to-end at over 1M agent-decisions per second on a single GPU node; and creating modified versions thereof (e.g., that support novel test sets, procedural generation logics, or game dynamics). Through PlayTrain, we reimagine RL VGE development: all we need is a single JS file, generated and modified through an LLM. We discuss promising future RL research directions that PlayTrain unlocks.

1 INTRODUCTION

PlayTrain is designed to combine efficiency and conceptual complexity with adaptability and playability in RL environments. It uses LLM-generated JavaScript games to make environments easier to create, modify, play, and train on.

1 INTRODUCTION

  • A short game specification can yield a single JavaScript file that is immediately playable and ready for baseline RL training.The same format also lets developers and testers play and test generated games.Source: Figure 1 · p. 2, §1 · p. 2
  • JavaScript supports complex, varied game mechanics and lets generated environments run and be shared in a browser without extra package installation.The paper contrasts this design space with the fixed ROM set in ALE and parametrized generators in ProcGen.Source: §1 · p. 2
  • JavaScript's browser-based processing has been considered too slow for practical RL training because environmental interactions bottleneck steps-per-second throughput.PlayTrain proposes a framework to overcome this bottleneck and exceed established environment-efficiency benchmarks.Source: §1 · p. 2
  • PlayTrain addresses the engineering effort involved in creating flexible RL environments by supporting game edits and variants through natural-language prompts.Edits can change visuals, physical parameters, or mechanics, or transplant one game's dynamics into another.Source: §1 · p. 3

2 METHODS

PlayTrain combines LLM-based JavaScript environment generation with a backend that runs those games efficiently for RL. Its pipeline supports refinement and variants, while a shared-memory, multithreaded design moves rendered observations to the trainer.

2 METHODS

  • PlayTrain combines an LLM prompting pipeline for generating and modifying JavaScript games with a JavaScript-to-gym backend for RL training.A short specification and fixed prompt produce a self-contained JavaScript file; an automated pass checks Gymnasium API compliance, proper observations, and matching outputs across two instances on random actions.Source: §2 · p. 3, §2.1 · p. 3

2.1 GENERATING AND DEVELOPING JS ENVIRONMENTS USING AN LLM

  • Play-testing can reveal design discrepancies, after which natural-language feedback lets an LLM refine the JavaScript game; variants typically require a single prompt.The authors report that all environments generated in their experiments passed the automated validation requirements one-shot.Source: §2.1 · p. 3

2.2 EFFICIENT BACKEND TO SUPPORT JS ENVIRONMENTS FOR RL

  • PlayTrain removes the browser from RL execution and reduces the pipeline to an input channel and renderer.It runs JavaScript on QuickJS, intercepts p5 drawing calls in a C++ library, and uses a Rust rasterizer to write pixels directly into the trainer's observation buffer.Source: §2.2 · pp. 3–4
  • Environment instances run on threads in the trainer's process, allowing rendered frames to be written directly into the trainer's shared buffer.The paper reports that stepping scales almost linearly with available threads.Source: §2.2 · pp. 4–5, Figure 2 · p. 4
  • Games read keyboard inputs, and the backend defaults to eight discrete actions and 64×64 RGB observations, while allowing both to be modified.The paper also states that seeded game logic and deterministic rasterization make runs reproducible across machines, with the same game files used for training and human play.Source: §2.2 · p. 5

3 EXPERIMENTS

The experiments assess PlayTrain’s environment and training speed, its ability to create variants and new games, and human play on generated games. They report high training throughput, illustrate several kinds of game changes, and compare agent learning with human scores.

3.1 EFFICIENCY OF ENVIRONMENTS AND TRAINING

  • Per core, JS replicas average 12.62× the speed of ALE and 2.18× that of ProcGen; with a trainer attached, they average 2.25× ProcGen speed and are 5.81× faster than ALE.The per-core comparisons are geometric averages over the suites; two ProcGen clones are not faster than their originals.Source: §3.1 · p. 6
  • PlayTrain trains its 24-game suite at an average of 1.07 M agent-steps per second with IMPALA and Nature-CNN; 23 games exceed 1M steps per second.climber trains at 881,299 steps per second. Table 1 reports single-node throughput on four H100s and 92 CPU cores.Source: §3.1 · p. 6, Table 1 · p. 6

3.2 GENERATING VARIANTS AND NEW GAMES

  • Variants include parametric changes, structural world changes, and visual or thematic changes, illustrated by breakout.multiball, qbert.bigmap, flappy_bird.hoop, and frostbite.jungle.The paper notes that space limits the examples, while PlayTrain supports other changes such as introducing new actions.Source: §3.2 · pp. 7–8
  • From single prompts and a few natural-language feedback rounds, PlayTrain generates VVVVVV and Downwell from existing game concepts.VVVVVV features gravity-flipping; Downwell features downward-firing boots and descent through enemies and gems.Source: §3.2 · p. 8

3.3 HUMAN PLAY

  • On six of eight games, PPO or IMPALA reaches the human mean, with success varying by game; participants and agents play the same games on identical levels.PPO reaches the mean on flappy_bird at 1 M steps, and IMPALA does so on coinrun after 81 M; neither reaches it on caveflyer or VVVVVV. The comparison is illustrative, with deeper human–machine studies left for future work.Source: §3.3 · pp. 8–9, Figure 5 · p. 8
  • PlayTrain is the only framework in the qualitative comparison described as both fast to train on and easy to modify.Source: Table 2 · p. 9

4 DISCUSSION

PlayTrain’s scope is limited by current LLMs’ ability to generate game size and complexity, and its speed varies with per-frame drawing and game-logic work. The framework may support research requiring diverse environments, but human playtesting remains necessary to definitively evaluate newly created environments.

Scope Limitations

  • Current LLMs cannot reliably generate games at the scale of full-fidelity modern console titles, and PlayTrain focuses on 2D environments without full 3D support.The games in this paper are single JavaScript files of a few hundred lines.Source: §4 · p. 9

Speed Limitations

  • Drawing-heavy games can bottleneck on draw calls, while state-heavy game logic can also limit speed, as shown by dodgeball and climber.Climber is one of the two clones slower than its original; its steps are 18% drawing, while dodgeball’s are 10%.Source: §4 · p. 9

Further potential of PlayTrain

  • Methods bottlenecked by training-environment scale and diversity, including meta-RL, may benefit from PlayTrain’s ability to generate more environments.The passage identifies environment data, rather than the algorithm itself, as the bottleneck for some meta-RL methods.Source: §4 · p. 9
  • Prompt edits can expose new mutation variables or create environments for co-evolution and unsupervised environment design; newly created environments still require human test-playing for definitive evaluation.The discussion also identifies diverse environments with hard-exploration or memory/cognition challenges as possible research targets.Source: §4 · pp. 9–10

5 CONCLUSION

PlayTrain combines LLM-generated JavaScript environments with a high-throughput backend, while keeping the same games playable in a browser. The related-work discussion contrasts this approach with environment-generation and speed-focused methods that constrain editability, engine choice, or practical training throughput.

5 CONCLUSION

  • PlayTrain combines adaptable, complex, playable JavaScript environments with a backend reaching close to one million environment steps per second for classic RL agents.The same environments are browser-playable and may also suit cognitive science studies.Source: §5 · p. 10

A.1 RELATED WORK

  • Procedural generation creates environment variations along prespecified axes, while neural world models and LLM-generated code offer other environment-generation approaches.The cited world models generate pixel observations in response to actions; code-generation approaches include tasks for fixed simulators.Source: §A.1 · p. 15
  • PlayTrain’s environments remain inspectable, editable, and deterministic without being confined to a specific engine, unlike the cited neural-world-model and fixed-simulator approaches.The passage contrasts neural environments encoded in model weights with code environments confined to physics engines or configuration spaces.Source: §A.1 · p. 15
  • AI GameStore generates LLM-agent benchmark games but is not a training framework and runs far below practical RL throughput; PlayTrain addresses the speed gap.The games are generated from popular app-store titles.Source: §A.1 · p. 15
  • GPU-port and system-level speed approaches trade off environment flexibility, while PlayTrain uses standard JavaScript programs and a fast system architecture for pixel-observation training.The passage notes that game engines make mechanics and parameters difficult to modify, and system optimizations are mostly limited to existing symbolic-state environments.Source: §A.1 · p. 15

A.2 FURTHER DISCUSSIONS

PlayTrain’s potential includes scaling environment diversity, supporting co-evolution and direct comparisons among humans, RL agents, and VLMs, and facilitating novel environment research. Its backend also standardizes actions and observations across games while keeping configuration adjustable.

A.2 FURTHER DISCUSSIONS

  • Some meta-RL methods are bottlenecked by the scale and diversity of training environments rather than by the algorithm, a need PlayTrain may help address.The discussion cites learning-to-learn methods as examples.Source: §A.2 · p. 16
  • For POET-style co-evolution, a prompt edit can expose new mutation variables or create an environment before the search loop; mutation along those variables then costs the same as in POET.PlayTrain may also serve as a tool for continually generating diverse environments in open-ended learning.Source: §A.2 · p. 16
  • The same browser-based JavaScript games can be played by a VLM, an RL agent, or a human, making their scores directly comparable..Source: §A.2 · p. 16
  • Novel environments may support research on exploration, memory, and cognition, but definitive evaluation of new environments currently requires human test-playing.The discussion identifies development of appropriate environments as a longstanding difficulty for such research.Source: §A.2 · p. 16
  • PlayTrain’s pipeline can use multiple JavaScript files as environments grow more complex; a CraftAx port and first-person variation were built and tested against the C implementation.The variation changed observations by adding voxel rendering while retaining the underlying game state.Source: §A.2 · pp. 16–17

B PLAYTRAIN ACTION SPACE AND OBSERVATIONS

  • The backend’s default action space is Discrete(8): it maps an integer action to key states, then runs draw() to produce the next observation.A NOOP holds no keys but still runs draw() and returns a new observation.Source: B PLAYTRAIN ACTION SPACE AND… · p. 17
  • Users can configure action spaces, including continuous input, and adjust observation resolution, frame handling, color settings, environment count, workers, and seeding without changing game files.The default observation is a single 64×64 RGB frame with no frame stacking.Source: B PLAYTRAIN ACTION SPACE AND… · pp. 17–18

C IMPLEMENTATION DETAILS

  • The implementation section covers environment execution, browser and Node backends used as baselines, and IMPALA and PPO trainer construction..Source: C IMPLEMENTATION DETAILS · p. 18

C.1 ENVIRONMENT IMPLEMENTATION

PlayTrain’s backend combines an embedded JavaScript engine, a C++ p5 layer, and a Rust rasterizer, with ahead-of-time compilation and runtime optimizations. A step runs the game, rasterizes its drawing commands into the observation buffer, and obtains reward and termination from game state.

C.1 ENVIRONMENT IMPLEMENTATION

  • The backend compiles QuickJS, a C++ p5 layer, a Rust rasterizer, and frozen math functions; the four pieces form the frame-processing path.The default game step executes draw(), sends drawing commands to the p5 layer, writes pixels into the observation buffer, then reads reward as score delta and termination from game state.Source: Table 6 · p. 19, §C.1 · p. 20
  • QuickJS embeds directly as a C library and initializes cheaply, enabling the same-program design even though it is slower than Node’s V8 engine.QuickJS also avoids additional runtime machinery.Source: §C.1 · p. 19
  • Ahead-of-time compilation lets games load as compiled data instead of parsing JavaScript source; fresh or uncompiled games run in the interpreter until compilation finishes.The backend is also profile-guided and link-time optimized, and unchanged drawing commands can skip rasterization without changing frames or game files.Source: §C.1 · p. 19
  • QuickJS calls precompiled C++ drawing functions directly, which communicate with the Rust rasterizer through a C ABI.The Rust rasterizer draws directly into the default 64×64 observation buffer, avoiding a full-size render and downscale pass; browser-play rasterization is separately implemented in JavaScript.Source: §C.1 · p. 19

C.2 OTHER BACKEND IMPLEMENTATIONS

The three backends run the same game JavaScript while changing the layers beneath it. The Playwright version uses browser APIs and real p5.js, whereas Node/V8 uses a custom API and rasterizer to run outside a browser.

C.2 OTHER BACKEND IMPLEMENTATIONS

  • Playwright steps through page.evaluate and draw(), copies canvas pixels with getImageData, downsamples them, and returns frames to Node through base64 encoding.This backend uses Chromium and the actual p5.js package rather than the custom rasterizer or p5 C++ layer.Source: §C.2 · p. 20
  • Node/V8 replaces browser machinery with an API layer connected to a JavaScript rewrite of p5.js, whose drawing commands use a WebAssembly rasterizer to generate pixels.Frames then pass from Node to Python through a pipe and shared memory, with Python driving the steps.Source: §C.2 · p. 20
  • The game’s JavaScript is identical across all three backends, but the browser backend uses real p5.js and browser canvas while the others use different underlying layers.The three configurations are Playwright with a headless browser, browserless Node/V8, and QuickJS.Source: Table 7 · p. 20, §C.2 · p. 20

C.3 IMPLEMENTATION DETAILS OF TRAINERS

The trainer implementation combines centralized batched inference and double-buffered environment sampling for IMPALA, while PPO runs synchronously on one process and GPU. The setup assigns GPUs between IMPALA learning and inference according to the encoder.

C.3 IMPLEMENTATION DETAILS OF TRAINERS

  • IMPALA moved from shared-CPU actors, which were too slow for the environments, to centralized batched inference and workers with GPU policy copies and vectorized environments.Centralized inference separates actors from model execution but adds send, batch, forward, and reply latency; each worker then steps its environments in one call.Source: §C.3 · pp. 21–22
  • IMPALA continuously updates on worker-collected trajectories using V-trace, with GPUs split between learner and inference according to encoder.IMPALA-CNN uses two GPUs for DDP learning and two for inference; Nature-CNN uses one for learning and three for inference.Source: §C.3 · p. 22
  • PPO runs synchronously on one process and GPU, collecting 128 steps from each of 192 environments before optimizing 24,576 timesteps for three epochs over eight minibatches.Its configuration differs from IMPALA in reward clipping and discounting: PPO does not clip rewards and discounts at 0.999, while IMPALA clips to ±1 and discounts at 0.99.Source: §C.3 · p. 22, Table 8 · p. 22
  • PPO makes three gradient computations per frame versus IMPALA’s one, a plausible explanation for PPO’s greater sample efficiency and slower training on some environments.The authors describe this as a plausible explanation, not a demonstrated causal result.Source: §C.3 · p. 22
  • Double buffering lets one group of environments step while another group's inference runs, helping most on slower environments.Table 9 measures agent-steps/s under IMPALA with Nature-CNN and fifteen workers, holding the trainer and games fixed.Source: Figure 8 · p. 21, §C.3 · p. 22, Table 9 · p. 23

D FULL-SUITE LEARNING CURVES

Across the 24-game suite, PPO wins on most titles despite training more slowly, while IMPALA performs better on several games and receives no Freeway reward. The curves compare both trainers and encoders over 100M environment steps, with episode-completion caveats for some games.

D FULL-SUITE LEARNING CURVES

  • The full-suite curves cover 16 ProcGen and eight ALE games for PPO and IMPALA with IMPALA-CNN and Nature-CNN, over 100M steps and three seeds.Figure 9 shows IMPALA-CNN comparisons; Figures 10 and 11 show IMPALA and PPO with both encoders.Source: Figure 9 · p. 23, D FULL-SUITE LEARNING CURVES · pp. 23, 25, Figure 10 · p. 24
  • PPO curves begin at 0.5M steps, while IMPALA’s early curves for maze, heist, and freeway average only completed episodes until the 12.3M-step horizon.Those three games have no failure state, so episodes do not end before the horizon unless won.Source: Figure 9 · p. 23, D FULL-SUITE LEARNING CURVES · p. 25
  • PPO wins 17 of 24 games despite being the slowest arm, while IMPALA outperforms it on climber, coinrun, chaser, heist, and asteroids and receives no Freeway reward.IMPALA’s zero on Freeway holds across both encoders and all three seeds; PPO reaches returns of 12 to 13 on five of six seeds.Source: D FULL-SUITE LEARNING CURVES · p. 25

E.1 SETUP AND BASELINES

The throughput evaluation compares PlayTrain’s 24 ALE and ProcGen clones with their original environments in three measurement settings. The setup varies the timing window, thread configuration, and observation resolution by setting.

E.1 SETUP AND BASELINES

  • The evaluation measures throughput on eight ALE and 16 ProcGen clones against their originals in single-core, multi-thread, and multi-thread-with-trainer settings.Comparisons run in a single job on one node, include resets in timed regions, and use random actions when no trainer is attached.Source: §E.1 · p. 25
  • Single-core speed is completed steps divided by wall time after warmup, comparing PlayTrain with Gymnasium/ALE and ProcGen package baselines.PlayTrain is stepped from C; baselines are stepped from Python. PlayTrain and ProcGen use 64×64 RGB observations, while ALE emits native 210×160 observations.Source: §E.1 · pp. 25–26, Table 11 · p. 25
  • Multi-thread speed is measured over a fixed 12-second window, with scaling efficiency defined relative to the lowest thread count and 100% representing linear scaling.The comparison uses a tuned EnvPool configuration; the table caption distinguishes per-core comparisons against shipped ALE and ProcGen from thread-scaling comparisons against EnvPool.Source: Table 11 · p. 25, §E.1 · p. 26, Table 12 · p. 26
  • With a trainer attached, the evaluation reports median agent-steps/s across four timed windows, using the trainer’s own step counter.The thread-scaling suite’s geometric mean is reported in Table 13.Source: §E.1 · p. 26, Table 13 · p. 26

E.2 ENVIRONMENT IMPLEMENTATION DETAILS

The section explains why profiling is needed beyond counting drawing and logic operations, then describes an operation-level method for estimating game-step costs.

E.2 ENVIRONMENT IMPLEMENTATION DETAILS

  • Operation counts alone do not reveal which costs more: drawing or game logic, and games with few drawing commands can still be slower than games spending half a step drawing.The authors therefore consider profiling critical for understanding what makes an environment slow or fast.Source: §E.2 · p. 27
  • To isolate costs, generated PlayTrain files measure each operation separately, with draw() issuing up to 4096 copies at random positions or drawing pinned for logic measurements.For an actual game, operation call counts are multiplied by the Figure 12 costs; the total is divided by step latency, with leftover time attributed to game logic. Runs use one core of a Sapphire Rapids node without dirty-rectangle skipping.Source: §E.2 · p. 27
  • Table 14 reports each game’s p5 drawing-command count, percentage of the step spent drawing, and single-core environment speed.Source: Table 14 · p. 27

F GAME GENERATION

This section describes PlayTrain’s browser-based workflow for generating, refining, and forking JavaScript games using prompts and a shared RL-compatible template. It outlines prompt constraints for visual simplicity, technical compatibility, and learnability, including how to modify existing games while preserving their runtime contract.

F GAME GENERATION

  • The tester supports refinement and forking through targeted LLM edits: it selects code chunks, generates JSON edits, and saves timestamped snapshots; forks create new files and seek deliberate changes.If chunk selection or targeted editing fails, the interface performs a full rewrite; generated games must define setup, draw, getGameState, resetGame, and mulberry32.Source: F GAME GENERATION · p. 27, Figure 13 · p. 29, Figure 14 · p. 29

F.1 FROM PROMPT TO GAME

  • Generation uses a single Gemini 3.1 Pro call, prompted with a game name, reference link, actions, and concise mechanic; ProcGen clones instead provide original C++ source.These resources are optional, and the section presents separate generate, refine, and fork prompts.Source: §F.1 · p. 30
  • The shared template gives games abstract Discrete(8) actions and runtime-downscaled 64x64 RGB observations, with a required single-file game interface.Actions are interpreted by each game, while the agent receives pixels and rewards; the template also standardizes the RL state and reset interface.Source: §F.1 · pp. 31–32
  • Seeded RNG controls procedural layouts, spawns, and other randomized parameters but not core mechanics or rewards; fixed-timestep Matter.js games replay identically from identical initial conditions.The template requires resetGame(seed) to initialize the seeded RNG and generate the level.Source: §F.1 · pp. 32–34
  • Because agents see 64x64 RGB images, the template recommends visually distinct, sufficiently large entities and gameplay-only screens without text-based HUDs, menus, or cutscenes.It also recommends solid fills and distinct colors and shapes so game elements remain legible at the observation resolution.Source: §F.1 · p. 34
  • Generated games should use simple, gameplay-relevant shapes and colors, with one core mechanic, no multi-phase progression, and no complex inventory or state systems.The guidelines also exclude decorative effects, narrative elements, and other visual or gameplay flourishes.Source: §F.1 · pp. 34–35
  • Games should expose the core mechanic within 10 steps, offer an early reward within 20–50 random-play steps, and provide scoring opportunities every 50–100 steps.The rules also call for gradual difficulty and deaths that are possible but not instant, so agents can learn from mistakes within an episode.Source: §F.1 · p. 35
  • The generation contract requires a single JavaScript file without imports, DOM access, asynchronous code, images, audio, timers, or non-seeded randomness.It also specifies keyboard input, meaningful scoring, terminating episodes, and ProcGen-style visuals.Source: §F.1 · p. 35
  • Validation checks deterministic seeded frames, non-degenerate visuals, score changes under random actions, episode termination within 5,000 random steps, and error-free execution for 1,000 steps.The game-state interface must also return score, lives, and gameState.Source: §F.1 · p. 35
  • The examples demonstrate patching Downwell to alter enemies, scoring, and animations, and forking Frostbite into a jungle variant while retaining required functions, controls, seeded determinism, and game-state reporting.The Downwell patch prompt requires selected-chunk replacements; the Frostbite prompt allows changes to mechanics, rules, entities, visuals, and difficulty.Source: §F.1 · p. 36
Loading 2609.09059v2…