Source-linked AI summary
GUI Agents for Continual Game Generation
Yixu Huang, Bo Li, Na Li, Zhe Wang, Kaijie Chen, Haonan Ge, Qingyi Si, Yuanzhe Shen, Ruihan Yang, Guangjing Wang, Hongcheng Guo
TL;DR
Game generation needs playtesting because code and static inspection can miss interaction-level failures. The paper introduces PlaytestArena for GUI-based evaluation and Play2Code for sustained coding-and-playing refinement, achieving a 66.8% rubric pass-rate while remaining scoped to curated HTML-based games.
Problem
Game generators produce artifacts that may compile yet fail during play, while quality remains inaccessible to code-level inspection.
Method
PlaytestArena evaluates browser games with rubrics adjudicated by a GUI agent, while Play2Code loops a game agent and GUI agent through shared memory.
Results
66.8% rubric pass-rate improves over single-pass and agentic-coding baselines by 37.1 and 14.6 points respectively.
Takeaways & Limitations
Playtesting provides a usable optimization signal for code generation, with rubric scores rising monotonically within refinement trajectories.
Takeaways & Limitations
The current framework focuses on HTML-based games, uses a curated benchmark, and shares one memory schema across games.
Abstract
from arXiv · showhide
Generating a game is not the same as making one that can be played. Despite advances in code generation, existing approaches treat game generation as one-shot translation from prompt to artifact, leaving interaction-level failures undetected. We argue that evaluating and improving game generation requires a player, and study two roles for graphical user interface (GUI) agents in this process: (1) as an objective evaluator, for which we introduce PlaytestArena, a new evaluation environment that pairs 200 browser-based game generation tasks across eight genres with rubrics of expected in-play behaviors, adjudicated by a GUI agent that loads each build in a browser and plays it; and (2) as a subjective playtester, for which we propose Play2Code, where a game agent and a GUI agent operate in a sustained loop with shared memory, turning game generation into a dialogue between coding and playing. Our experiments show that even frontier models struggle to generate playable games directly, while Play2Code achieves a 66.8\% rubric pass-rate, improving over single-pass and agentic-coding baselines by 37.1 and 14.6 points respectively. Further analysis shows that GUI playtester feedback is more traceable than a human report, yet idiosyncratic in ways reminiscent of human testers, establishing game playtesting as a critical testbed for interactive code generation. Our project website is available at https://continual-game-generation.vercel.app/.
1 Introduction
Game quality cannot be established from code or static inspection alone because games must be played to reveal interaction-level failures. The paper therefore studies GUI agents as both evaluators and playtesters, introducing PlaytestArena and Play2Code to connect playing with generation.
- Games can compile and pass tests yet still fail during play through unresponsive inputs, missing visuals, or broken win conditions.
- Existing code generators produce game artifacts faster than humans can play them, while game quality remains inaccessible to code-level inspection.
- GUI agents evaluate games on the same rendered interface and input channels used by human playtesters.
- PlaytestArena pairs generation prompts with rubrics of expected in-play behaviors and uses a GUI agent to load, play, and adjudicate each build.
- Play2Code places a game agent and GUI agent in a sustained shared-memory loop, turning generation from one-shot translation into dialogue between coding and playing.
- 66.8% rubric pass-rate improves over single-pass and agentic-coding baselines by 37.1 and 14.6 points respectively.
2 Can GUI Agents Playtest?
GUI agents can play browser-based games by perceiving rendered states, reasoning about goals, and acting through keyboard and mouse inputs. The playtest evaluation spans diverse games and exposes where agent reliability differs across genres.
- Human playtesters assess a game through its presentational surface, recording working mechanics, breakdowns, and missing experiences.
- GUI agents perceive rendered interfaces, decide from those perceptions, and act through clicks and key presses like human players.
- A Snake session illustrates the observe–reason–act loop, with the agent updating its plan after each state transition.
- GUI agents clear the majority of approximately 120 game levels, with GPT-5.4 approaching the human reference at k ∈{5, 10, 20}.
- Their logs reflect rendered states, decisions connect observations to goals, and inputs produce observable game transitions.
- PlaytestArena covers 200 games across eight genres, with statistics tracking prompts and rubric dimensions.
3 What’s “Better”: PlaytestArena
PlaytestArena defines game-generation quality through observable in-play behavior rather than code alone, using browser-based tasks, human-authored rubrics, and GUI-agent adjudication. Its scores are validated against human judgments.
- PlaytestArena contains 200 browser-based game-generation tasks spanning eight genres, each paired with a prompt and rubric of expected behaviors.
- The rubric corpus contains 1,548 testable criteria, averaging 7.7 criteria per game across mechanics, controls, progression, and visual feedback.
- A GUI agent loads each generated game in a browser, plays through clicks and keys, and assigns pass-fail judgments for each rubric criterion.
- GUI judgments match human annotators on 84.2% of criteria, with game-level rankings near-identical to human-derived rankings.Agreement was κ = 0.64 at criterion level, while game-level correlations were ρ = 0.87 and r = 0.88.
4 Play2Code
Play2Code turns game generation into a sustained interaction between a coding agent and a GUI playtester. Shared runtime and memory let play observations guide successive revisions across rounds and tasks.
- Play2Code couples a Game Agent that writes or patches code with a GUI Agent that plays each build and reports observations through shared memory.Each round contains a generate-or-revise step followed by play and reporting.
- The Game Agent conditions revisions on the latest build, episode memory, and cross-task skill and world memories.
- The GUI Agent plays without code or rubric access, then converts its action and observation history into a play summary and actionable fixes.These artifacts are written into shared episode memory for the next revision round.
- The game agent treats GUI feedback as advice, selecting which fixes to implement before the next round.
5 Does Play Drive Code Generation?
Play2Code tests whether gameplay feedback improves generated games beyond direct synthesis and code-level inspection. It outperforms both baselines and improves scores across successive rounds, with gains depending on game complexity.
- 66.8% rubric pass-rate is achieved by Play2Code, compared with 29.7% for Direct LLM and 52.2% for OpenGame.The comparisons isolate the added value of GUI playtesting over single-pass generation and iterative code-level inspection.
- Play2Code scores increase monotonically from Round 1 to Round 3 as observed failures are translated into concrete fixes.Early rounds expose basic interaction failures, while later rounds address finer-grained issues such as visual effects.
- Moderate-complexity games benefit most from the iterative loop, showing steady improvement across all five rounds and the largest absolute gain.
- High-complexity games show markedly limited improvement, while low-complexity games converge by rounds 3–5.
6 How Far Are GUI Agents from Human Playtesters?
GUI agents remain unlike human playtesters in subjective experience, but they provide more traceable execution feedback and exhibit backbone-specific emphases reminiscent of human tester differences.
- Human playtesters uniquely provide felt difficulty, frustration, boredom, surprise, and tacit aesthetic judgments that current agents do not approximate.
- GUI agents therefore fall short of full human playtesting while still offering a distinct, structured feedback modality.
- GUI-agent feedback is more traceable because actions, observations, and diagnoses are logged rather than reconstructed from memory in a summary report.
- Different playtester backbones emphasize different feedback categories, including functionality, experience, or visual aesthetics.
- The authors interpret this divergence as agent-specific taste in what gets noticed, paralleling idiosyncrasies among human playtesters.
7 Related Work
Related work establishes strong code-generation and GUI-agent capabilities, while highlighting that conventional functional evaluations miss important properties of playable game interaction.
- Large language models can generate increasingly capable programs and complete interactive games from natural-language prompts.
- Existing game-generation evaluations commonly rely on functional correctness, which does not capture playability, responsiveness, or fidelity to intended interaction design.
- GUI agents use visual understanding and action planning to interpret screen observations and act according to environment state and task goals.
- Agent evolution uses iterative feedback, tool use, and multi-agent interaction, while much prior work emphasizes static rather than interactive environments.
8 Conclusions
The paper treats game quality as something established through play rather than code, using GUI-agent evaluation and continual refinement to improve generated games.
- PlaytestArena evaluates generated games by having a GUI agent load each build, play it, and score per-game rubrics from observation.
- Play2Code places the GUI agent inside generation, allowing game and GUI agents to co-evolve through shared memory across games.
- Rubric scores rise monotonically within each Play2Code refinement trajectory across three backbones.
- The findings position playtesting as both a richer evaluation signal and a usable optimization signal for code generation.
Limitations
The study is limited by its HTML-focused implementation, curated benchmark, shared memory design, and a feasibility testbed restricted to relatively simple browser games and controlled protocols.
- The current implementation focuses on HTML-based games, leaving native-engine and 3D titles for future work.
- The benchmark uses a curated set selected for quality control and does not cover the full diversity of real-world game scenarios.
- The current memory schema is shared across all games, whereas game-specific or genre-aware memory may yield further gains.
- The feasibility testbed contains 20 games, intentionally kept small for matched end-to-end comparison between GUI agents and human references.
- Its LLM-assisted games are simple by design, and its public games are limited to static HTML bundles without login, payment, or external API dependencies.
- Action and shooter genres are omitted because fast real-time dynamics introduce inference-latency and game-tick-rate confounds.
- The feasibility protocol decomposes games into discrete levels with unambiguous completion conditions and evaluates pass@k from repeated episodes under a five-minute timeout.
- Agents use three backbones, clean-slate level play, screenshots at 1280×720, and 20 independent episodes per backbone-level pair.
A.5 Per-Genre Results
GUI agents clear most game levels across genres, but performance varies by interaction demands: puzzle and card games are closest to human performance, while platformers remain hardest. Their playtest reports also reveal both grounded observations and recurring timing and UI-reading limitations.
- Genre performance: All three GUI backbones clear most levels across every genre, with the narrowest gap to humans on puzzle and card games.These genres reward deliberation more than reaction speed.
- Genre performance: Platformer games show the widest gap to humans because precise jump timing remains a bottleneck.The limitation is associated with real-time interaction demands.
- Qualitative behavior: GUI agents ground observations in the rendered surface, including object locations, interface changes, and failure states.Their logs reflect what appears on screen rather than what the agent expects to see.
- Qualitative behavior: Agents typically retry failed interactions once or twice before trying an alternative, helping distinguish missing mechanics from difficult-to-trigger mechanics.This distinction supports actionable fix lists for Play2Code.
- Limitations: The dominant failure modes are imprecise timing in real-time games and misreading small or low-contrast interface elements.These are described as mainly backbone-level limitations caused by inference latency.
C.1 Sample Construction
The feasibility study evaluates GUI playtesting on a stratified 32-game sample spanning eight genres and three generation methods. It measures both criterion-level agreement and game-level ranking consistency against human judgments.
- Sample design: The validation sample contains 32 generated games, evenly stratified across eight genres and the three compared generation methods.The design prevents agreement estimates from reflecting only easy or uniformly poor builds.
- Sample design: The sample yields 375 rubric criteria, averaging 11.7 criteria per game.Games contain between 9 and 15 criteria.
- Evaluation protocol: Agreement is evaluated with raw agreement and Cohen’s κ at the criterion level, plus Spearman’s ρ and Pearson’s r for game rankings.Game scores are the fraction of criteria each judge marks PASS.
- Results: GUI–human agreement is comparable to human–human agreement across all reported agreement and correlation measures.No individual human annotator is substantially closer to the GUI judge than to other humans.
- Agent protocol: The agents play browser games through a structured workflow of observation, starting, interaction, assessment, and memory capture.They use screenshots, game guidance, mouse and keyboard actions, and severity-tagged findings grounded in observed trajectories.
F Ablation Studies of Play2Code
Ablations show that Play2Code benefits from both layered memory and grounded GUI feedback, with the GUI Agent contributing more than any individual memory layer. Human-driven refinement still outperforms GUI-driven refinement, especially in later rounds.
- Memory ablations: Removing memory entirely causes the largest memory-related drop, while Skill Memory helps with recurring implementation pitfalls and World Memory provides smaller but consistent genre-transfer gains.Examples include platformer jump arcs, tower-defense scheduling, and underrepresented genres.
- GUI Agent ablation: Removing the GUI Agent causes a larger rubric-score drop than removing any single memory layer.Self-verification misses runtime failures and interaction bugs that appear during actual play, especially in action- and physics-heavy genres.
- Memory ablations: Each memory layer contributes independently, with Episode Memory providing the largest share, Skill Memory accumulating role-specific know-how, and World Memory transferring design knowledge across agents and genres.The full configuration combines Episode, Skill, and World Memory.
- Complexity categorization: Complexity tiers are descriptive and defined by average rubric-score gain across five Play2Code rounds, rather than by an independent difficulty measure.Games are split into High-complexity (∆≤10), Moderate-complexity (10 < ∆≤15), and Low-complexity (∆> 15) tiers.
- Human comparison: Human-driven refinement achieves higher rubric scores than GUI-agent-driven refinement across all rounds, with the largest difference in later rounds.Humans continue to surface subtle interaction failures that GUI agents tend to miss, while GUI agents track humans more closely early on without requiring human effort.