Source-linked AI summary
SPRITE: From Static Mockups to Engine-Ready Game UI
Yunshu Bai, RuiHao Li, Hao Zhang, Chien Her Lim, Ming Yan, Mengtian Li
TL;DR
Game UI Screenshot-to-Code systems struggle with irregular geometry and deep hierarchies that do not map cleanly to web layouts. SPRITE combines VLM reasoning, 2D geometric grounding, and YAML-based scaffolding to generate editable Unity assets. Expert reviews rated visual fidelity and hierarchical logic strongly, while complex interaction remains a boundary.
Problem
Existing Screenshot-to-Code approaches struggle with game interfaces because web-oriented models assume rectangular layouts rather than irregular geometries, deep scene graphs, and overlapping sprites.
Method
SPRITE uses a training-free pipeline combining VLM semantic scaffolding, a hierarchical YAML intermediate representation, 2D geometric grounding, and LLM-generated Unity assets.
Results
8.5/10 Visual Fidelity and 8.0/10 Hierarchical Logic were achieved in expert review, compared with 7.0/10 Interaction Accuracy.
Takeaways & Limitations
SPRITE streamlines translation from static mockups to functional Unity assets, supporting novice prototyping and helping professionals focus on complex interaction logic.
Takeaways & Limitations
SPRITE remains limited for ambiguous overlapping translucent or diegetic elements and complex temporal logic such as drag-and-drop mechanics.
Abstract
from arXiv · showhide
Game UI implementation requires translating stylized mockups into interactive engine entities. However, current "Screenshot-to-Code" tools often struggle with the irregular geometries and deep visual hierarchies typical of game interfaces. To bridge this gap, we introduce SPRITE, a pipeline that transforms static screenshots into editable engine assets. By integrating Vision-Language Models (VLMs) with a structured YAML intermediate representation, SPRITE explicitly captures complex container relationships and non-rectangular layouts. We evaluated SPRITE against a curated Game UI benchmark and conducted expert reviews with professional developers to assess reconstruction fidelity and prototyping efficiency. Our findings demonstrate that SPRITE streamlines development by automating tedious coding and resolving complex nesting. By facilitating rapid in-engine iteration, SPRITE effectively blurs the boundaries between artistic design and technical implementation in game development. Project page: https://baiyunshu.github.io/sprite.github.io/
1 Introduction
Game UI implementation is bottlenecked by translating irregular, deeply hierarchical artistic designs into engine structures. SPRITE addresses this gap with a YAML-mediated, structure-aware pipeline evaluated through a professional benchmark and expert standards.
- Motivation: Game UIs use irregular geometries, diegetic elements, and deep visual hierarchies that create a persistent design-to-engine bottleneck.Developers must manually slice, measure, and reconstruct assets, which stifles iteration and can produce visual discrepancies.
- Motivation: Web-centric Screenshot-to-Code methods assume rectangular DOM elements and flow-based layouts, mismatching game engines’ absolute coordinates, scene graphs, and overlapping sprites.These assumptions make modest adaptations inadequate for game interfaces.
- Approach: SPRITE combines VLM semantic reasoning, 2D-model localization, and a YAML intermediate representation to reconstruct mockups as engine-native assets.The YAML scaffold preserves hierarchical consistency between visual design and proprietary engine formats.
- Contributions: The system is presented as a training-free refactoring pipeline and structure-preserving image-to-engine bridge for editable assets.Its benchmark targets layout accuracy and usability against expert standards.
- Contributions: SPRITE is named both as an acronym for Screenshot Parsing and Reconstruction of Interfaces via Training-free Engineering and after Unity’s 2D Sprite asset type.The naming connects the reconstruction pipeline to a fundamental engine graphic primitive.
2 Related Work
Prior UI-generation research separates visual fidelity from logical structure and is largely shaped by web-oriented data and layouts. SPRITE targets this gap with training-free alignment for game-specific visual and structural complexity.
- Visual and structural modeling: Visual-centric methods can produce high-fidelity aesthetics but flatten layouts, while logic-centric methods preserve structure at the cost of pixel-level fidelity.The related-work divide motivates approaches that jointly address appearance and engine-relevant hierarchy.
- Game-specific mismatch: Web-based MLLM tools lack the structural rigidity required by game interfaces.Game UI reconstruction therefore requires representations beyond standard web layouts.
- Data constraints: Public datasets such as RICO and PubLayNet provide flattened RGB screenshots rather than raw RGBA layers and complex occlusion relationships.These omissions limit direct applicability to game-rendering pipelines.
- Training-free alignment: SPRITE uses training-free alignment to bypass the need for large-scale domain-specific annotation while targeting engine-native structures.Its design responds to the data scarcity and alignment challenges identified in prior work.
3.1 Design Goals
SPRITE’s design goals prioritize semantic and structural fidelity, broad training-free generalization, and support for users with different technical expertise. These goals frame reconstruction as both an engine-structure problem and a workflow-access problem.
- DG1: Fidelity Beyond Pixels: SPRITE must recover the semantics of non-standard assets and reconstruct deep nested hierarchies into logically sound scene graphs.This goal extends beyond visual replication toward engine interaction readiness.
- DG2: Zero-Shot Generalizability: The training-free architecture aims to remain agnostic to visual styles and adapt across genres without costly domain-specific retraining.A coarse-to-fine pipeline and modular YAML representation provide the intended generalized bridge.
- DG3: Dual-Audience Empowerment: SPRITE is designed to support a spectrum of technical expertise within the game-development community.The dual-audience goal covers both novice access and professional augmentation.
- DG3: Dual-Audience Empowerment: For novices, the system lowers the technical barrier by enabling non-programmers to turn sketches into functional prototypes.This goal emphasizes accessible creation rather than manual engine implementation.
- DG3: Dual-Audience Empowerment: For professionals, SPRITE removes pixel-level slicing and layout scaffolding so experts can focus on interaction logic and creative iteration.The intended benefit is augmentation of existing workflows rather than replacement of expert work.
3.2 The SPRITE Pipeline
SPRITE decomposes screenshot reconstruction into semantic scaffolding, precision grounding, and engine-native synthesis. A schema-guided YAML representation establishes hierarchy before geometric extraction and executable code generation.
- Semantic Scaffolding: Semantic Scaffolding uses a VLM and a predefined YAML template to populate a nested scene graph of functional UI components.The template mirrors engine hierarchy, while an expert-persona prompt guides component identification.
- Semantic Scaffolding: YAML reduces token consumption by approximately 20–30% compared with JSON and mirrors Unity UXML through indentation-based nesting.These properties support denser layout processing and downstream structural alignment.
- Precision Grounding: Precision Grounding uses YAML-defined regions of interest and 2D foundation models to segment standalone alpha-channel assets and recover occluded backgrounds.The stage addresses VLM limitations in spatial precision while targeting engine-ready layer completeness.
- Engine-Native Synthesis: Engine-Native Synthesis translates calibrated YAML into executable UXML/USS, inferring basic affordances such as button hover states from semantic labels.The output combines a no-code gateway for novices with an editable layered scaffold for professionals.
3.3 Implementation Details
SPRITE is a modular, training-free pipeline that combines semantic parsing, geometric localization, asset cleanup, and code generation for engine-native UI reconstruction.
- SPRITE orchestrates state-of-the-art foundation models in a modular, training-free pipeline.
- Qwen3-VL performs coarse semantic parsing and component identification using the “UI Master Persona” system prompt.The prompt enforces functional decoupling, hierarchical parent fields, and visual cues for downstream models.
- GroundingDINO detects components from VLM cues, while SAM2 generates precise pixel-level masks.
- LaMa inpaints backgrounds to remove occlusion artifacts, and GPT-5 and Claude 4.5 Sonnet synthesize Unity UXML and USS from structured YAML.
4 Evaluation
SPRITE is evaluated on a diverse, production-level GAMEUI Benchmark and through qualitative baseline comparisons and expert review. The results indicate strong visual and structural reconstruction, while interaction accuracy remains comparatively lower because dynamic and ambiguous interactions exceed the current scope.
- 4.1 GAMEUI Benchmark: The GAMEUI Benchmark contains hundreds of interfaces spanning RPG, FPS, Strategy, and Casual genres and varied layout complexity.
- 4.1 GAMEUI Benchmark: Each benchmark entry includes Figma structural and interaction metadata, segmented sprites with text metadata, and hand-authored Unity UXML/USS gold standards.
- 4.1 GAMEUI Benchmark: SPRITE maintains structural integrity and generates engine-ready scaffolds across complex, non-standard game UI scenarios.
- 4.2 System Validation: SPRITE extracts pixel-perfect, logically grouped assets for irregular shapes, unlike bounding-box VLMs and fragmented pixel-level baselines.
- 4.2.2 Expert Review: 8.5/10 Visual Fidelity and 8.0/10 Hierarchical Logic were rated by three senior designers, while Interaction Accuracy scored 7.0/10.The ratings used a 10-point Likert scale from entirely unusable to production-ready.
- 4.2.2 Expert Review: SPRITE handles hover states, toggles, and simple clicks but struggles with overlapping translucent elements, abstract diegetic UIs, and complex temporal logic.These boundaries directly explain the lower interaction score reported in the expert review.
5 Discussion and Future Work
The discussion frames SPRITE’s future around expanding from static reconstruction toward dynamic interactivity, generative restyling, and human-AI workflow studies. These directions target broader UI functionality, controllable visual variation, and the changing division of labor in game development.
- From Static Visuals to Dynamic Interactivity: Future work will use Video-LLMs to extract temporal logic from gameplay clips or interaction recordings.Targets include animation curves, statemachine transitions, and conditional visual feedback loops.
- Disentanglement for Generative Design: SPRITE aims to disentangle functional hierarchy in UXML from visual presentation in USS for prompt-driven global restyling.
- GAMEUI Benchmark: The GAMEUI Benchmark’s diverse genres and layouts define the functional diversity and zero-shot generalizability targeted by SPRITE.
- Disentanglement for Generative Design: Decoupled structural and visual components could support rapid A/B testing and procedural UI variants without altering functional logic.
- Human-AI Collaboration and Workflow Cognition: Longitudinal studies will examine trust calibration, structural-refactoring breakdowns, and mixed-initiative co-creation in game-development workflows.
6 Conclusion
SPRITE automates translation from static mockups to functional Unity assets by combining semantic scaffolding, geometric precision, and code generation. The resulting workflow reduces manual reconstruction overhead while supporting different roles in rapid game creation and interaction development.
- SPRITE harmonizes VLM semantic scaffolding, 2D foundation-model precision, and LLM code generation to automate translation into functional Unity assets.
- SPRITE reduces manual asset-slicing and assembly overhead, lowering the technical barrier for novices and helping professionals focus on complex interaction logic.