Source-linked AI summary
SchemaGUI: A Schema-Driven Benchmark for Controllable GUI Generation Evaluation
Jiarui Dong, Yin Cai, Zhouhong Gu, Chenmou Wu, Ci Tao, Yiran Chen, Jialing Li, Xiaoran Shi, Juntao Zhang, Zhijun Fang
TL;DR
Reliable GUI evaluation is limited by uncontrolled coverage and noisy annotations. SchemaGUI generates paired instructions and deterministic references from parameterized schemas, then benchmarks five models across six bilingual scenarios. Results identify geometry, layout complexity, and thinking-mode efficiency as central evaluation findings.
Problem
GUI evaluation lacks datasets that are coverage-controlled, clean per sample, and scalable.
Method
SchemaGUI uses parameterized schemas to generate paired natural-language instructions and deterministic function-call references for controlled GUI evaluation.
Results
Larger models improve schema compliance and overall performance, but geometric layout control remains an important challenge, especially in complex layouts.
Takeaways & Limitations
The findings support geometry-aware, schema-constrained evaluation for diagnosing structured GUI generation ability.
Takeaways & Limitations
The benchmark focuses on static text-to-schema synthesis and may not capture dynamic interaction, multimodal refinement, or the visual diversity of real interfaces.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated strong potential in graphical user interface (GUI) generation, but reliable evaluation remains challenging due to uncontrolled data distributions, noisy annotations, and limited layout scenario coverage. To address this, we propose SchemaGUI, a template-based benchmark for controllable GUI generation evaluation. By synthesizing paired natural language instructions and deterministic function-call references from parameterized interface schemas, SchemaGUI can generate thousands of deterministically annotated tasks in seconds without human labeling. Based on 1,000 evaluated instances per scenario and language across six representative bilingual scenarios, we benchmark five mainstream models, including the Qwen3.5 family, Qwen3-Coder-30B, and DeepSeek-R1. Our extensive analysis reveals three key insights. First, precise geometric spatial control remains an important bottleneck; while scaling Qwen3.5 from 4B to 27B improves Schema Feasibility from 91.56% to 99.63%, the Geometry score improves more modestly (from 67.05% to 75.30%). Second, generation difficulty is highly sensitive to layout complexity, with current LLMs excelling at simple sequential arrangements but suffering severe coordinate drift in dense grids and multi-region compositions. Third, thinking mode increases token consumption while generally reducing GUI Score, particularly for smaller models.
1 Introduction
SchemaGUI addresses unreliable GUI evaluation by generating scalable, coverage-controlled tasks with deterministic annotations. Experiments show that geometric control, layout complexity, and thinking mode remain important challenges.
- Motivation: Existing GUI benchmarks suffer from uncontrolled distributions, limited scenario coverage, expensive annotation, and noisy or hallucinated references.These issues make large-scale model comparisons unreliable.
- Contribution: SchemaGUI generates paired natural-language instructions and deterministic component-wise annotations from adjustable interface templates.The framework covers six interface types and avoids human labeling and model-generated references.
- Contribution: More than 10,000 samples can be generated in seconds through template-derived annotations without relying on human labeling or model outputs.
- Findings: Scaling Qwen3.5 from 4B to 27B raises average Schema Feasibility from 91.56% to 99.63%, while Geometry rises from 67.05% to 75.30%.The Geometry improvement is more modest than the feasibility improvement.
- Findings: Models perform better on simple linear layouts but degrade substantially on dense components and multi-region combinations.The paper identifies precise geometric spatial control as an important challenge.
- Findings: Thinking mode reduces Qwen3.5-27B Overall GUI Score by 9.72 points in EN and 12.36 points in ZH while increasing response-token consumption.
2 Related Work
Related work progresses from screenshot-to-code generation toward structural, execution-oriented, and modular GUI synthesis. SchemaGUI responds to remaining layout-integrity issues with unified schema-constrained functional synthesis.
- GUI Generation: Early systems demonstrated end-to-end GUI reconstruction from screenshots or sketches, while later multimodal models target high-fidelity structured interfaces.
- Structural Constraints: End-to-end generation can cause layout drift or component overlap because it relies on stochastic generation without explicit structural constraints.
- Structural Constraints: Structural modeling methods impose hierarchy, grammar, preference, or specification constraints to improve layout consistency and generative stability.
- Structural Constraints: Geometric auditing addresses physical conflicts such as component overlap in predicted function calls.
- Execution-Oriented Methods: Execution-oriented and modular frameworks evaluate interactive workflows, task decomposition, or iterative tool use, but often emphasize localized tasks.
- SchemaGUI Positioning: SchemaGUI uses formal parameter-space modeling to govern the full layout and maintain systemic integrity beyond atomized methods.
3 Methodology
SchemaGUI formalizes GUI generation as schema-driven sequences of atomic function calls, then evaluates predictions against deterministic references across feasibility and five quality dimensions.
- Formalization: A GUI is represented as a structured collection of components generated through a sequence of atomic function calls.
- Pipeline: The benchmark samples instances from a schema-driven design space and constructs an aligned instruction, deterministic reference, and model prediction triplet.
- Functional Representation: Each function call specifies an operation primitive and parameters for component type, geometry, and content.
- Design Space: The design space exhaustively enumerates compatible combinations across actions, layout geometry, content attributes, and structural relations.
- Evaluation: Schema feasibility requires structurally valid and executable function-call sequences; infeasible predictions receive a GUI Score of zero.
- Evaluation: The GUI Score is the equally weighted mean of the five submetrics for schema-feasible predictions.
- Evaluation: Feasible predictions are compared with deterministic references using Geometry, Component F1, Text Match, Spatial Order, and Layout Harmony.
4 Experiment Setup
The experiments evaluate five models on controlled bilingual samples spanning six layout scenarios. Scores combine schema feasibility and five GUI-quality dimensions, alongside token-based efficiency measures.
- Models: Five models are evaluated across parameter scales and families, including Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B, Qwen3-Coder-30B-A3B-Instruct, and DeepSeek-R1.
- Languages: English and Chinese subsets share scenario definitions, function-call schemas, and layout controls while using language-specific instructions and content.
- Dataset: The benchmark retains 1,000 instances per scenario and language, yielding 6,000 instances per language and 12,000 bilingual instances per model.
- Scenarios: Scenarios range from sequential layouts in Login Form and Info Card to dense grids, coordinated components, and multi-region layouts.
- Reporting: Overall scores in Table 1 are macro-averages across six equally weighted scenarios, with EN and ZH reported separately.
- Metrics: GUI quality uses Geometry, Component F1, Text Match, Spatial Order, and Layout Harmony, while efficiency records Prompt Tokens and Response Tokens.
5 Analysis
Across scenarios and models, GUI generation is strongest for simple sequential layouts but remains limited by precise geometric control, dense compositions, and reasoning overhead.
- Overall Model Performance: Qwen3.5-27B achieves the strongest overall performance, with Overall GUI Scores of 84.61% in EN and 81.78% in ZH.Qwen3-Coder-30B ranks second in both settings.
- Performance Across GUI Scenarios: Login Form reaches the highest average GUI Score at 90.27%, followed by Music Player at 85.67% and Info Card at 83.25%.These scenarios use clearer sequential arrangements or functionally separated groups.
- Performance Across GUI Scenarios: Calculator records the lowest average GUI Score at 62.13%, with scores ranging from 33.25% to 75.54%.Its dense button grid requires consistent sizes, spacing, and row–column alignment; Image Editor and Doc Editor also score lower in multi-region layouts.
- The Impact of Reasoning: Thinking mode lowers Overall GUI Score for all three Qwen3.5 models, with decreases of 31.54 and 32.74 percentage points for Qwen3.5-4B in EN and ZH.The decreases shrink for Qwen3.5-9B and Qwen3.5-27B, but no model achieves an Overall improvement.
- The Impact of Reasoning: Calculator consumes the most response tokens at 3939.2 on average, whereas Login Form requires 1248.5.Qwen3.5-27B uses 360.7 tokens in EN and 364.7 in ZH while achieving the highest Overall GUI Scores.
6 Conclusion
SchemaGUI introduces a template-based benchmark for controllable GUI generation evaluation, producing paired natural-language instructions and deterministic function-call references across six GUI scenarios. Experiments show that larger models improve schema compliance and overall performance, but precise geometric layout control remains a central challenge.
- SchemaGUI is a template-based benchmark for controllable GUI generation evaluation.
- It produces paired natural-language instructions and deterministic function-call references across six GUI scenarios.
- Larger models improve schema compliance and overall performance, but geometric layout control remains an important challenge.
Limitations
SchemaGUI is limited to static structural synthesis and geometric layout planning, while dynamic interactions and multimodal design workflows remain unexplored.
- SchemaGUI focuses on static structural synthesis and geometric layout planning, leaving dynamic user interactions and sequential runtime workflows unexplored.
- The evaluation uses a text-to-schema setting rather than multimodal feedback loops combining textual intent, visual inspection, and iterative refinement.
Ethical Concern
SchemaGUI’s ethical risks are indirect because it evaluates generated GUI structure and layout rather than deployment readiness or user-facing safety. Practical use would require human review and additional validation.
- SchemaGUI uses programmatically generated samples and deterministic annotations without real interfaces, private screenshots, personal identifiers, or behavior logs.
- The benchmark evaluates structural correctness, geometric layout quality, and schema-following ability, not whether interfaces are safe, accessible, visually appropriate, or deployment-ready.
- Human review and additional validation would be necessary for practical GUI workflows to address misleading layouts, inaccessible designs, and unmet user needs.
- Template-based controllability may simplify real GUI design, so SchemaGUI is intended as a diagnostic tool rather than a replacement for human-centered design evaluation.
- AI tools may assist auxiliary tasks, but authors retain responsibility for the research’s correctness, validity, and ethical treatment.
A.1 Scenario Schemas and Prompt Construction
SchemaGUI constructs bilingual scenario templates and paired prompts and deterministic references from shared parameterized design instances. Its six scenarios vary in structural and geometric complexity while preserving functional schemas for controlled evaluation.
- Scenario Schemas: The six scenarios span compact grids, sequential structures, coordinated media controls, and multi-region editors with increasingly complex spatial relationships.
- Scenario Schemas: Parameterized templates vary window configuration, components, content, dimensions, spacing, alignment, and scenario-specific organization while preserving each scenario’s functional identity.
- Scenario Schemas: Table 4 summarizes controlled variation sources that generate multiple interface configurations within each consistent functional schema.
- Prompt Construction: Each design instance specifies the scenario, window and layout requirements, allowed functions, required components, and scenario-specific guidance for prompt construction.
- Prompt Construction: The same design instance feeds a verbalized prompt branch producing model sequence yL and a deterministic reference branch producing yT.
- Prompt Construction: Evaluation first checks schema feasibility, then compares the generated sequence yL with the deterministic reference yT using GUI Score.
A.2 Metric Implementation Details
SchemaGUI evaluates generated GUIs through deterministic references and complementary metrics covering feasibility, component recovery, text, spatial order, layout harmony, and geometry. These metrics distinguish executable outputs from outputs that also preserve content and visual organization.
- Deterministic reference sequences are generated from the same schema configuration and slot values used to construct each model instruction.
- Schema Feasibility: Schema Feasibility accepts only non-empty valid JSON function-call sequences with valid initialization, supported operations, positions, and positive dimensions.
- Component Matching: Component matching uses exact component identity first, then greedily matches same-type components by normalized center-position distance.
- Component F1: Component F1 measures multiset-level component recovery, penalizing both missing and redundant components through precision, recall, and their harmonic mean.
- Text Match: Text Match uses exact agreement between normalized component identities, rather than edit distance or semantic similarity.
- Spatial Order: Spatial Order reconstructs visual rows and left-to-right ordering, then scores the proportion of adjacent reference relations preserved after matching.
- Layout Harmony: Layout Harmony averages alignment, spacing, and proportion to assess internal visual organization beyond individual coordinates and sizes.
- Bilingual Evaluation: The bilingual settings compare instantiated English and Chinese instructions and text directly against deterministic references without modeling locale-specific conventions.
A.3 Geometry
Geometry evaluates whether matched GUI components retain consistent positions and sizes, complementing metrics that measure feasibility, content, and relative organization. The broader analysis shows that geometric fidelity remains sensitive to layout complexity and can fail despite schema-valid, recognizable outputs.
- Geometry Definition: Geometry lies within [0, 1] and combines coordinate-size consistency with penalties for unmatched components, out-of-bounds placement, and severe overlap.
- Bounding-Box Consistency: For each valid matched pair, normalized bounding-box deviation jointly measures position and size error relative to the reference window.
- Error Treatment: An unmatched reference component contributes a unit geometric error, while moderate coordinate or size deviations reduce the score continuously.
- Geometric Layout Failure: Schema-valid predictions can still exhibit grid collapse, out-of-bounds placement, overlap, or inconsistent dimensions, reducing usability despite recognizable content.
- Scenario Sensitivity: The calculator scenario requires button recovery together with row-column alignment, dimensions, spacing, and window containment.
- Generation Cost: Longer raw or clean outputs do not necessarily improve GUI quality, and concise format adherence can matter more than unconstrained reasoning.
- Human Evaluation: Automatic and human evaluations show high rank-level agreement, with Kendall’s τ = 0.80 and Spearman’s ρ = 0.90.