Source-linked AI summary
PlayCoder: Making LLM-Generated GUI Code Playable
Zhiyuan Peng, Wei Tao, Xin Yin, Chenhao Ying, Yuan Luo, Yiwen Guo
TL;DR
Conventional code-generation benchmarks and unit tests do not adequately capture the stateful interactions needed to assess GUI applications. PlayCoder combines repository-aware generation with behavioral-test-guided iterative repair, and its evaluation reports that it significantly outperforms baseline methods.
Problem
Existing code-generation benchmarks underrepresent GUI applications, and traditional unit tests do not capture behavioral correctness that requires interactive validation.
Source: §1 · p. 2, §2.1 · p. 4Method
PlayCoder uses a repository-aware generation agent and a repair agent that iteratively refines code based on behavioral-testing feedback.
Source: §4 · p. 12Results
PlayCoder’s collaborative framework significantly outperforms baseline methods in evaluation on multilingual GUI applications.
Source: §8 · p. 20Takeaways & Limitations
Repository-level GUI code generation remains challenging, even with prompting and agentic approaches.
Source: §3.6.2 · p. 12PlayCoder’s testing and validation are constrained by VLM limitations in recognizing fine-grained GUI elements and interpreting complex visual semantics.
Source: §7 · p. 20
Abstract
from arXiv · showhide
Large language models (LLMs) have achieved strong results in code generation, but their ability to generate GUI applications, especially games, remains insufficiently studied. Existing benchmarks mainly evaluate correctness through test cases, which are inadequate for GUI applications because these systems are interactive, event-driven, and require correct state transitions across sequences of user actions. Their evaluation therefore should consider interaction flows and UI logic rather than only pass/fail outcomes. To study this problem, we introduce PlayEval, a repository-aware benchmark built from 43 multilingual GUI applications in Python, TypeScript, and JavaScript. Unlike prior GUI benchmarks that are difficult to adapt to desktop environments, PlayEval covers six major GUI application categories and directly supports code-generation evaluation. We further propose Play@k, a metric that measures whether at least one of *k* generated candidates can be played end-to-end without logical errors. To support reliable evaluation, we develop PlayTester, an LLM-based agent that performs task-oriented GUI playthroughs and detects logic violations automatically. Experiments on 10 state-of-the-art code LLMs show that, despite high compilation rates, they achieve near-zero Play@3, revealing major weaknesses in generating logically correct GUI applications. To address this limitation, we present PlayCoder, a multi-agent, repository-aware framework that generates, evaluates, and iteratively repairs GUI application code in a closed loop. PlayCoder substantially improves both functional correctness and semantic alignment for open-source and closed-source models, reaching up to 38.1% Exec@3 and 20.3% Play@3. Case studies further show that it can uncover silent logic bugs missed by traditional metrics and fix them through targeted edits.
1 Introduction
GUI code generation requires evaluating interactive behavior and evolving application state, not just compilation or unit-test correctness. PlayEval, PlayTester, and PlayCoder address this gap through end-to-end playthroughs and iterative code repair.
1 Introduction
- GUI applications can compile and pass unit tests yet fail during interaction, while pixel-rendered interfaces and nondeterministic behavior make structural testing and comprehensive test coverage difficult.The passage’s Flappy Bird example shows a bird passing through obstacles without exceptions.Source: §1 · p. 2
- PlayEval covers 43 multilingual GUI applications across six categories, while Play@k measures whether generated code passes unit tests and can be played end-to-end without logical errors.Games exemplify interaction-heavy applications that rely on visual feedback, while desktop applications broaden the benchmark’s use cases.Source: §1 · p. 2
- With Claude-Sonnet-4, PlayCoder reaches 36.8% Exec@3 and 20.3% Play@3, and it achieves up to 20.2% higher Exec@3 and 11.0% higher Play@3 than baselines.With GPT-5-mini, PlayCoder achieves 26.8% Exec@3 and 9.8% Play@3, compared with DeepCode’s 17.9% Exec@3 and 6.4% Play@3.Source: §1 · p. 3
- PlayTester simulates user interactions to detect behavioral failures that unit tests miss and provides diagnostics for iterative code improvement across programming languages and platforms.The framework uses screenshots and mouse or keyboard interaction to explore interfaces and identify failures such as invisible text or unresponsive buttons.Source: §1 · p. 3
- PlayCoder combines repository-aware code generation by PlayDeveloper with PlayTester feedback and iterative repairs by PlayRefiner in successive test-and-repair cycles.The introduction reports that the top-performing model achieves 9.9% Play@3 and 18.6% Exec@3, while the weakest baseline scores below 1% Play@3.Source: §1 · p. 3
2 Motivation
GUI code can compile and run while failing during interaction, so reliable evaluation must test behavior across GUI playthroughs. The section motivates PlayEval and PlayTester, then presents a closed-loop multi-agent approach to generating and repairing GUI code.
2.1 A Motivating Example
- A Flappy Bird game generated by GPT-4o-mini lets the bird pass through pipes instead of ending on collision, a silent logic failure that compilation-only checks miss.The human-written version terminates the game on collision; the generated version runs without crashing but never ends.Source: §2.1 · p. 3, Figure 1 · p. 4
- Repository-aware methods and static analysis can miss interactive semantics, while Web and Mobile testers rely on DOMs or accessibility trees often absent from desktop GUIs.This is especially problematic for PyGame, which renders to a canvas; GUI correctness depends on event loops, state transitions, and temporal behavior.Source: §2.1 · p. 4
- 18.6% Exec@3 to 9.9% Play@3 for Claude-Sonnet-4 and 17.5% Exec@3 to 6.9% Play@3 for GPT-5 expose a gap between executability and behavior.Traditional unit-test metrics such as Pass@k miss GUI failures in collision detection, event handling, and stateful interactions; existing benchmarks underrepresent GUI tasks.Source: §2.1 · p. 4, §2.2 · p. 5
2.2 Key Ideas
- PlayEval combines repository scaffolds and behavioral specifications with hierarchical PlayTester testing and Play@k, which evaluates candidates that first pass unit tests.Its standardized interaction scripts target event handling, state management, and physics or animation to detect defects that unit tests can miss.Source: §2.2 · p. 5
- PlayCoder uses PlayTester as both behavioral evaluator and feedback source in a closed-loop multi-agent framework for repository-aware generation and targeted repair.PlayDeveloper retrieves repository context, PlayTester reports behavioral diagnostics, and PlayRefiner applies repairs; compilation errors, API misuse, and behavioral failures trigger different responses.Source: §2.2 · p. 5
3 Benchmark and Evaluation · 3.1 Benchmark Construction
PlayEval is a repository-aware benchmark for GUI code generation, built from 43 applications across Python, TypeScript, and JavaScript. Its evaluation progresses from execution and unit tests to interactive validation of GUI behavior.
3 Benchmark and Evaluation
- Experiments use three samples per problem, an unbiased estimator with k≤3, official baseline configurations, and five repetitions with standardized configurations.Source: §3 · p. 5
3.1 Benchmark Construction
- PlayEval is a repository-aware code-generation benchmark for GUI applications in Python, TypeScript, and JavaScript.Source: §3.1 · p. 6
3.1.1 Selection Criteria.
- Repository selection prioritized active development, community validation, complete GUI workflows, framework diversity, and clear structure; extracted functions focused on core behavior and used a 28-line minimum.The function-length threshold excluded docstrings and decorator lines; shorter functions were more likely to be helpers or accessors than core interactive behavior such as game loops and event handlers.Source: §3.1.1 · p. 6
- The 43 applications span six categories, from emulation and classic games to MMORPGs, game engines, standalone applications, and desktop widgets.The benchmark includes a complete Game Boy emulator, strategy and arcade games, and two TypeScript MMORPGs; Python, TypeScript, and JavaScript are represented.Source: §3.1.2 · p. 6
3.1.2 Dataset Composition.
- Standalone applications typically support dynamic resizing, whereas desktop widgets are often fixed-size components of larger user interfaces.Standalone applications include productivity, multimedia, and web-based apps as well as small-scale applications such as calculators.Source: §3.1.2 · p. 7
- Each evaluation instance combines an original function signature, an LLM-generated natural-language requirement, and repository context drawn from the same codebase.GPT-4o-mini generated requirements, and three developer experts manually verified a subset of about 10%; over 95.6% were rated high-quality. Original repository tests had 47.2% line coverage and 32.1% branch coverage.Source: §3.1.3 · p. 7
3.1.3 Benchmark Structure.
- Exec@k measures whether at least one of k samples executes without runtime errors, syntax errors, or import failures.Source: §3.1.4 · p. 7
3.1.4 Evaluation Workflow.
- Pass@k measures passing all provided unit tests, while Play@k measures correct behavior in live GUI playthroughs using game objectives or feature-covering interaction sequences.Because original projects lack sufficient tests, the benchmark generates broader suites covering unit, integration, functional, and edge-case tests; PlayTester conducts GUI interaction and validation.Source: §3.1.4 · p. 7
3.2 Baselines
The baselines cover state-of-the-art LLMs and LLM-based enhancement approaches, selected for breadth across model capabilities and representative code-generation strategies.
3.2 Baselines.
- Baseline selection prioritizes breadth of model capabilities and coverage of prompt-based and agentic enhancement strategies demonstrated for code generation.The model range includes general-purpose and coding-specialized LLMs from both open- and closed-source families.Source: §3.2 · p. 8
3.2.1 Basic LLMs.
- The ten SOTA LLM baselines span general-purpose and coding-specialized models, open- and closed-source families, and diverse architectures and scales.Closed-source models are GPT-5, GPT-5-mini, GPT-4o, GPT-4o-mini, Claude-Sonnet-4, Claude-Sonnet-3.7, and Grok-3-mini; open-source models are Qwen3-Coder (480B), GLM-4.5 (355B), and DeepSeek-V3 (671B).Source: §3.2.1 · p. 8
3.2.2 LLM-based Approaches.
- The enhancement baselines include prompt-based SCoT and repository-aware HCPCoder, alongside agentic frameworks MetaGPT and OpenManus and code-generation platform DeepCode.SCoT uses an 8-step structured chain-of-thought pipeline; HCPCoder prunes context and retrieves code examples and import patterns; MetaGPT simulates specialized software-development roles and was shown suitable for generating games such as 2048 and Snake.Source: §3.2.2 · pp. 8–9
3.3 PlayTester
PlayTester combines visual observation, GUI action execution, and vision-language test planning to evaluate GUI behavior through automated playthroughs.
3.3 PlayTester
- Three specialized components collaborate to validate GUI application behavior, with their prompts specified in.Source: §3.3 · p. 9
3.3.1 Visual Observer Module.
- VisualObserver captures screenshots, detects window and state changes, and caches recent frames to distinguish animations from static states.It captures screenshots one second after each action and supports Windows, macOS, and X11-based Linux, but not Wayland.Source: §3.3.1 · p. 9
3.3.2 Action Executor Module.
- ActionExecutor converts parsed structured LLM output into GUI operations, applies safety checks, and records execution history for debugging.Supported operations include clicks, typing, hotkeys, key presses, scrolling, waiting, and finishing with success or failure.Source: §3.3.2 · p. 9
3.3.3 Test Manager.
- The Test Manager uses vision-language models and distinct test-strategy templates for goal-driven game testing and coverage-driven non-game workflows.Behavioral verification is fully automated, and tests are reusable across consistent screen resolutions.Source: §3.3.3 · p. 9
3.3.4 Case Study.
- In a 2048 case study, PlayTester simultaneously advances gameplay and checks swipe responsiveness, tile merging, score updates, invalid moves, freezes, numerical consistency, and termination.Its exception-aware validation can reveal behavioral faults that unit-test-based evaluation would miss.Source: §3.3.4 · pp. 9–10, Figure 3 · p. 10
3.4 Evaluation Metrics
The section defines four metrics for code-generation evaluation: execution, test-case success, interactive gameplay correctness, and effectiveness relative to token cost. Exec@k and Play@k estimate the chance that at least one of the top-k samples meets its criterion, following Pass@k [10].
3.4 Evaluation Metrics
- Exec@k and Play@k estimate the probability that at least one of the top-k samples succeeds at execution or logically correct gameplay, respectively.For each problem, the estimator uses n samples and the number c_succeed meeting the criterion; the metrics are inspired by Pass@k [10].Source: §3.4 · p. 10
- Exec@k, Pass@k, and Play@k are evaluated in sequence, with each stage testing only samples that passed the preceding stage.Exec@k requires successful execution without syntax or compile errors; Pass@k requires passing all provided tests; Play@k requires correct GUI behavior under interactive testing, with crash constraints checked against logs and final-state screenshots.Source: §3.4 · p. 10
- Efficiency@k measures effectiveness relative to token cost, using average tokens per problem and total token usage across N problems.The passage specifies a 10^3 scaling factor to keep numerator and denominator on comparable scales; for k=1, the metric simplifies through a combinatorial formula based on successful samples and samples per problem.Source: §3.4 · pp. 10–11
3.5 Preliminary Study 1: PlayTester Effectiveness · 3.6 Preliminary Study 2: Method Performance
Human evaluation supports PlayTester’s reliability, while the evaluated LLMs and enhanced methods show substantial limitations on PlayEval. Performance declines as criteria shift from execution to interactive behavioral correctness, and existing enhancement methods do not close the cross-language gap.
3.5 Preliminary Study 1: PlayTester Effectiveness
- PlayTester’s manual validation found a 16% false-negative rate and a 5% false-positive rate, with substantial agreement with human evaluators (Krippendorff’s α=0.790; Kendall’s τb=0.795).Three experienced GUI software engineers independently evaluated 100 randomly selected successes and 100 failures identified by PlayTester.Source: §3.5 · p. 11
3.6 Preliminary Study 2: Method Performance
- The study uses PlayTester to assess state-of-the-art LLMs and LLM-based methods after establishing it as a reliable evaluation framework.This transition applies the validated framework to repository-level GUI code generation on PlayEval.Source: §3.6 · p. 11
- The study evaluates 10 state-of-the-art LLMs and five enhanced methods using Exec@k, Pass@k, and Play@k for executability, unit-test correctness, and interactive semantic correctness.Base models use few-shot prompting to generate complete repository-aware implementations; enhanced methods include SCoT, HCPCoder, MetaGPT, OpenManus, and DeepCode.Source: §3.6.1 · p. 11
3.6.2 Results.
- Across 10 LLMs and 5 enhanced methods, results decline as evaluation criteria become more stringent, and no enhanced method narrows the cross-language performance gap.The comparison covers three programming languages and all evaluation metrics; Table 2 reports means over five independent runs with 95% confidence intervals.Source: Table 2 · p. 11, §3.6.2 · p. 12
3.6.1 Experiment Setup.
- Table 2 reports PlayEval performance across three programming languages and all evaluation metrics as means over five runs with 95% confidence intervals.The intervals are calculated using the Student’s t-distribution.Source: Table 2 · p. 11, §3.6.2 · p. 12
3.6.2 Results.
- Claude-Sonnet-4 leads base LLMs across languages and evaluation metrics, yet the best TypeScript Play@3 scores remain in the single digits.The execution-to-behavior gap widens from JavaScript to Python to TypeScript, so execution success does not reliably predict behavioral validity.Source: §3.6.2 · p. 12
- Enhancement methods yield limited and inconsistent gains: SCoT improves marginally in Python and JavaScript, HCPCoder performs catastrophically, and multi-agent methods have mixed results.DeepCode achieves strong execution success but limited behavioral-validation improvement across all three languages.Source: §3.6.2 · p. 12
4 Our Approach: PlayCoder
PlayCoder is a repository-aware, multi-agent framework pairing PlayDeveloper for code generation with PlayRefiner for iterative repair based on behavioral testing feedback.
4.1 Multi-Agent Collaboration Workflow
- PlayDeveloper generates repository-aware GUI code from specifications using retrieved code patterns and module structures, then subjects it to automated behavioral testing.Testing combines application-state capture, vision-language interaction planning, and action execution to collect behavioral signals.Source: §4.1.1 · p. 12
4.1.1 Workflow Phases.
- PlayRefiner diagnoses execution traces and test feedback, synthesizes repository-aware patches, checks fixes, and repeats testing until criteria pass or the six-iteration limit is reached.The loop ends early when the application passes all per-sample behavioral checks enforced by PlayTester; checks include interactive semantics such as collision handling and state transitions.Source: §4.1.1 · p. 13
4.1.2 Trajectory Recording.
- Applications run in sandboxed environments with deterministic seeding and controlled timing, while standardized logs support reproducible comparisons and provide failure signals to Testing and APR agents..Source: §4.1.3 · p. 13
4.2 PlayDeveloper
- PlayDeveloper uses a modular tool architecture with repository search, file operations, shell execution, and conversation management, alongside few-shot requirement-to-code prompting.ContextSearchTool retrieves repository code examples and import patterns; the agent supports multiple LLM providers.Source: §4.2 · p. 13
4.3 PlayRefiner
- PlayRefiner performs automated program repair driven by behavioral feedback and execution traces, coordinating validation tools and iterative fixes until behavioral criteria are satisfied..Source: §4.3 · p. 13
4.3.1 Repair Tooling.
- PlayRefiner coordinates ContextSearcher for repository APIs and import patterns, Validator for syntax/AST checks and compile gating, and Executor for sandboxed runtime and behavioral signals..Source: §4.3.1 · pp. 13–14
4.3.2 Repair Workflow.
- The repair workflow aggregates compiler, runtime, and behavioral failures, proposes and atomically applies minimal context-guided patches, then validates and re-evaluates behavior within a fixed budget.The five phases are diagnosis, patch generation, patch application, build and runtime validation with behavioral re-evaluation, and iterative refinement until criteria are met or the budget is exhausted.Source: §4.3.2 · p. 14
5 Results
PlayCoder outperforms repository-aware baselines, while efficiency results, ablations, and case studies show the resource trade-offs, component contributions, and behavioral testing capabilities and limitations. The results report means over five independent runs with 95% confidence intervals.
5.1 RQ-1: How does PlayCoder perform compared to baselines?
- 11.6pp Exec@3 and 12.7pp Pass@3 over SCoT, and 14.0pp Exec@3 and 16.2pp Pass@3 over HCPCoder, are GPT-5-mini PlayCoder gains.PlayCoder outperforms all baselines across LLMs and languages, while behavioral validation remains difficult for baselines; JavaScript is comparable to or better than Python, whereas TypeScript is systematically lower. Claude-Sonnet-3.7 validation changed Play@3 rankings minimally (0.9%-3.7%).Source: §5.1 · p. 15, Table 3 · p. 15
5.2 RQ-2: How does the efficiency of PlayCoder compare with the baselines?
- 8.3% Play@1 with 164K tokens and 5,480 tokens per function gives PlayCoder the third-best cost-effectiveness among evaluated methods.The analysis attributes about 22% of PlayCoder’s token use to testing and 78% to generation and iterative patching; the RQ-2 answer reports highest Efficiency@k and Play@k performance.Source: §5.2 · p. 16, Table 4 · p. 16
5.3 RQ-3: How do different components and models affect PlayCoder’s effectiveness?
- 8.5pp lower Python Exec@3, 9.8pp lower JavaScript Exec@3, and 6.9pp lower TypeScript Exec@3 follow APR removal for GPT-5-mini.Removing GUI feedback reduces Play@3 by 4.5pp, 4.7pp, and 4.2pp across those languages, respectively; both APR and GUI feedback contribute to effectiveness, with consistent improvements across diverse LLM architectures.Source: Table 5 · p. 16, §5.3 · p. 17, Table 6 · p. 17
5.4 Case Study
- In the 2048 case study, PlayCoder uses visual analysis and adaptive testing to select gameplay actions that also validate game mechanics.It identified tile positions, built tests for merging, score computation, random tile generation, responsiveness, and boundary conditions, then used a rightward swipe to merge two 2-tiles while checking mechanics. PlayTester also detected and fixed invisible white-on-white tile numbers that standard unit tests missed.Source: §5.4 · pp. 17–18, Figure 5 · p. 18
- 4 of 43 PlayEval projects defeated all methods, including PlayCoder, under high-frequency or long-duration gameplay conditions.Failures occurred in fast-paced arcade games and real-time physics simulations, where discrete polling and inference latency could not meet tight timing demands.Source: §5.4 · p. 18
6 Related Work
Prior work advances LLM code generation through prompting, repository context, agents, and practical benchmarks, while GUI research spans automation, screen understanding, testing, and generation. PlayCoder is distinguished by feeding dynamic execution feedback into code generation and repair to detect and fix GUI logic errors that compile- and run-based checks miss.
6.1 Code Generation with LLMs
- Pre-training has strengthened code generation, while later work develops reusable prompting, repository-aware context methods, and multi-agent frameworks for coordinated software tasks.HCP [80] orders context by dependency and prunes non-essential detail, improving CrossCodeEval accuracy on five of six repository-trained code LLMs while increasing throughput; agentic examples include MetaGPT [22], OpenManus, and DeepCode.Source: §6.1 · p. 19
- Practical coding benchmarks evaluate realistic scenarios, class synthesis, or issue resolution, yet largely omit behavioral correctness for generated GUI applications and visual, interactive execution feedback.Examples include CoderEval [76], ClassEval, and SWE-Bench [26].Source: §6.1 · p. 19
6.2 GUI Interaction Understanding, Testing, and Generation
- GUI research spans rule-based exploration, model- and search-based testing, screen understanding, and generation, but often prioritizes structural coverage, visual defects, layouts, scripts, or action traces over semantic behavior.Examples include Android Monkey, Sapienz, Stoat, UIED, ScreenAI [8], pix2code [6], and Seq2Act.Source: §6.2 · pp. 19–20
- Dynamic execution feedback in the code-generation loop enables PlayCoder to detect and repair GUI logic errors, including game rule violations, that compile- and run-based checks miss.Unlike coverage- or crash-oriented testers and agents evaluated on existing GUIs, this approach targets end-to-end runtime logic in generated applications.Source: §6.2 · p. 20
7 Threats to Validity
PlayCoder’s evaluation is limited by underlying model capabilities and stochasticity, as well as screenshot-based testing, repository-scale retrieval, and benchmark scope. The authors mitigate stochasticity with five repetitions and address scope by selecting applications across domains and platforms.
External Threats
- External validity is constrained by VLMs’ difficulty recognizing fine-grained GUI elements and complex visual semantics, while LLM response variation persists even at zero temperature.The authors conduct all experiments with five repetitions to improve result stability and reliability; they expect future VLM advances to mitigate capability limits.Source: §7 · p. 20
Internal Threats
- Internal validity is limited because screenshot analysis may miss critical frames, repository retrieval may not scale, and the benchmark’s application and scenario coverage may be limited.The authors select applications spanning multiple domains and platforms to address benchmark scope.Source: §7 · p. 20
8 Conclusion
The paper identifies a gap in evaluating interactive GUI code and introduces PlayCoder, a multi-agent framework combining automated GUI testing with iterative program repair. Evaluation across multilingual applications shows that PlayCoder outperforms baseline methods in functional correctness and semantic alignment.
8 Conclusion
- Existing benchmarks miss behavioral failures in interactive applications that syntactically correct code and traditional unit tests may not reveal.Source: §8 · p. 20
- PlayCoder pairs PlayTester, which simulates user interactions and detects behavioral deviations, with PlayRefiner, which autonomously debugs and refines generated code.Source: §8 · p. 20
- Across Python, TypeScript, and JavaScript GUI applications, PlayCoder outperforms baseline methods in functional correctness and semantic alignment.Source: §8 · p. 20