Source-linked AI summary
GEBench: Benchmarking Image Generation Models as GUI Environments
Haodong Li, Jingwei Wu, Quan Sun, Guopeng Li, Juanxi Tian, Huanyu Zhang, Yanlin Lai, Ruichuan An, Hongbo Peng, Yuhong Dai, Chenxi Li, Chunmei Qing, Jia Wang, Ziyang Meng, Zheng Ge, Xiangyu Zhang, Daxin Jiang
TL;DR
Existing benchmarks leave GUI-specific state transitions and temporal coherence underexplored. GEBench addresses this gap with a 700-sample, five-task benchmark and GE-Score, finding that models handle single-step transitions better than multi-step consistency and spatial grounding while exposing key bottlenecks.
Problem
Existing benchmarks prioritize general-domain visual fidelity or continuous transitions, leaving GUI-specific functional logic and state-transition consistency underexplored.
Method
GEBench evaluates image generation models as GUI environments using 700 samples, five task categories, and GE-Score’s five-dimensional rubric.
Results
Current models perform robustly on localized single-step transitions but struggle with long-term interaction consistency, spatial grounding, icon interpretation, and Chinese text rendering.
Takeaways & Limitations
GEBench identifies critical bottlenecks and directions for developing high-fidelity, temporally coherent generative GUI systems.
Abstract
from arXiv · showhide
Recent advancements in image generation models have enabled the prediction of future Graphical User Interface (GUI) states based on user instructions. However, existing benchmarks primarily focus on general domain visual fidelity, leaving the evaluation of state transitions and temporal coherence in GUI-specific contexts underexplored. To address this gap, we introduce GEBench, a comprehensive benchmark for evaluating dynamic interaction and temporal coherence in GUI generation. GEBench comprises 700 carefully curated samples spanning five task categories, covering both single-step interactions and multi-step trajectories across real-world and fictional scenarios, as well as grounding point localization. To support systematic evaluation, we propose GE-Score, a novel five-dimensional metric that assesses Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality. Extensive evaluations on current models indicate that while they perform well on single-step transitions, they struggle significantly with maintaining temporal coherence and spatial grounding over longer interaction sequences. Our findings identify icon interpretation, text rendering, and localization precision as critical bottlenecks. This work provides a foundation for systematic assessment and suggests promising directions for future research toward building high-fidelity generative GUI environments. The code is available at: https://github.com/stepfun-ai/GEBench.
1. Introduction
GEBench addresses the gap between general visual-fidelity benchmarks and the functional, discrete state transitions required for GUI interactions. It combines a five-dimensional evaluation framework with 700 samples and finds that current models remain weak in long-term consistency and spatial grounding.
- Existing benchmarks emphasize general-domain visual fidelity or continuous video transitions, leaving GUI-specific functional logic and discrete state-transition consistency underexplored.
- GUI interaction requires action-triggered interface jumps, precise coordinate grounding, icon recognition, text rendering, and logical continuity across changing visual elements.
- GEBench contains 700 samples across five task categories, aligning GUI image sequences with corresponding user instructions for multifaceted evaluation.
- GE-Score uses five VLM-guided rubrics to assess goal fulfillment, interaction logic, UI content consistency, structural integrity, and visual fidelity.
- Current models handle localized single-step transitions relatively well but show substantial weaknesses in long-term interaction consistency and precise spatial grounding.
- Icon interpretation, Chinese text rendering, and grounding-point localization are identified as bottlenecks associated with layout drift and logical inconsistencies.
2. Related Work
Related work traces GUI generation from rule-based layout reconstruction toward data-driven, reference-based image synthesis. Existing image benchmarks measure single-image visual fidelity, motivating evaluation methods that also capture logical coherence.
- GUI generation has shifted from heuristic structural mapping toward data-driven synthesis powered by multimodal large language models.
- Recent image-generation systems increasingly use style or structural references to improve spatial precision and identity consistency.
- Standard image benchmarks focus on single-image visual fidelity and text alignment, commonly using FID and CLIP score.
- These visual metrics provide an aesthetic-realism baseline but motivate incorporating logical coherence for more holistic evaluation.
3. GEBench
GEBench treats image generation models as interactive GUI environments that transform a current visual state and user instruction into a subsequent state. Its five task types and GE-Score jointly evaluate planning, grounding, consistency, interaction realism, and visual fidelity using curated interaction data.
- Benchmark Framework: A GUI environment translates user instructions or agent actions into visual feedback through software logic, making functional rules central to its structure.
- Benchmark Framework: GEBench supplies a current GUI image and user instruction, then evaluates the model-generated subsequent GUI state.
- Task Suites: GEBench contains 700 interaction sequences organized into five task categories for fine-grained capability evaluation.
- Task Suites: Single-step transition tests detailed instruction following, whereas multi-step planning generates a five-step trajectory and assesses long-horizon planning, temporal coherence, and UI-structure consistency.
- Task Suites: The task suite also covers fictional-app zero-shot generalization, rare real-app trajectories, and grounding-based generation.
- Scoring Rubric: GE-Score aggregates five complementary dimensions that capture functional accuracy, interaction realism, content consistency, UI plausibility, and visual quality.
- Scoring Rubric: Each dimension is scored from 0 to 5, and the aggregate score uses a linear normalization into the percentage range [0, 100].
- Data Construction: The dataset pipeline records mobile and desktop interactions, annotates actions as structured JSON, filters noisy samples, verifies sequences manually, and calibrates the final distribution.
4. Evaluation
GEBench evaluates 12 image generation models across task suites, revealing strong single-step performance but major weaknesses in multi-step planning and spatial grounding. VLM-based evaluations closely align with human judgments.
- Evaluation Setup: 12 models were evaluated, comprising 8 commercial and 4 open-source systems.
- Overall Performance: Commercial models, led by Nano Banana Pro, show more balanced task-suite profiles than open-source models.
- Multi-step Planning: Single-step transition scores exceed 80 points for leading models, but multi-step planning generally falls below 60 or even 10 points.
- Multi-step Planning: Multi-step failures arise from accumulated intermediate visual errors and difficulty reasoning about dependencies between successive visual states.
- Spatial Grounding: Nano Banana Pro achieves only 23.9% on the grounding GOAL score, while most other models fall below 10%.
- Evaluation Validity: VLM evaluations strongly correlate with human judgments, reaching overall Pearson r = 0.9892, with r = 0.9926 for Nano Banana Pro and r = 0.9833 for GPT-Image-1.
5. Discussion and Analysis
The analysis shows that visual quality can mask failures in GUI logic, especially as task complexity increases. Key bottlenecks involve text rendering, icon semantics, and precise spatial localization.
- Task Difficulty: Performance declines as task complexity rises, suggesting that strong single-step visual synthesis does not ensure deep GUI-mechanics understanding.
- Bottleneck Analysis: Text rendering failures produce deformed, overlapping, or semantically corrupted characters in dense layouts.
- Bottleneck Analysis: Icon interpretation errors sever interaction intent from visual affordances, disrupting subsequent state transitions.
- Bottleneck Analysis: Localization errors offset generated elements by dozens of pixels, while GOAL scores generally remain below 20%.
- Bottleneck Analysis: High visual fidelity does not guarantee functional viability because aesthetically strong GUIs may contain hallucinated widgets or illogical layouts.
6. Conclusion
GEBench reframes image-generation evaluation around GUI interaction logic rather than general visual fidelity. Its benchmark and GE-Score expose barriers to reliable, high-fidelity GUI simulation.
- Conclusion: GEBench introduces a systematic benchmark for evaluating image generation models as GUI environments.
- Conclusion: The benchmark shifts evaluation from general-domain visual fidelity toward GUI interaction logic and state behavior.
- Conclusion: GE-Score and a VLM-based evaluation pipeline identify critical barriers to high-fidelity GUI simulation.
A. Evaluation Framework
The GEBench framework uses a three-stage evaluation pipeline to assess image generation models across five GUI-generation task categories. Its overview diagram summarizes this process.
- Evaluation Framework: The evaluation framework operates through a systematic three-stage pipeline for benchmarking image generation models as GUI environments.
- Evaluation Framework: The image-generation stage produces visual outputs across five distinct task categories.
- Evaluation Framework: Figure A1 presents an overview of the process for evaluating generated GUI sequences.
B. Detailed Performance On GEBench Using Different Judges
GEBench reports detailed performance profiles for 12 models using three VLM judges across three evaluation runs. Scores cover all five GE-Score dimensions and are normalized to [0, 100] for consistent comparison.
- Evaluation setup: 12 evaluated models are assessed across three separate runs by Gemini-3-Pro-Native, GPT-4o, and Qwen3-vl-235b-a22b-thinking judges.The detailed results are organized in Tables A1–A3 by judging VLM.
- Cross-validation: The multi-judger evaluation enables cross-validation of model capabilities and examines the consistency of the GE-Score framework.Using multiple autonomous judges is presented as a robustness and objectivity measure.
- Score coverage: Each performance table reports scores for GOAL, LOGIC, CONS, UI, and QUAL across every GEBench task category.Together, these dimensions form the full GE-Score profile for each model and task.
- Score normalization: [0, 100] is the standard range used after linearly normalizing raw outputs from different VLM judges.Normalization addresses variance in the judges’ internal scoring scales.
C. Detailed Rubric on five tasks
GEBench uses fine-grained, task-specific rubrics as the foundation of its VLM-as-a-judge evaluation. The rubrics translate five GE-Score dimensions into explicit linguistic criteria across performance tiers.
- Rubric design: Fine-grained scoring rubrics are tailored to the specific requirements of different GUI generation tasks.They provide the standardized evaluation logic for the VLM-as-a-judge framework.
- Rubric design: Each rubric decomposes GOAL, LOGIC, CONS, UI, and QUAL into explicit descriptions for multiple performance tiers.The tiers provide concrete visual and functional benchmarks for assigning scores.
- Evaluation consistency: Concrete visual and functional benchmarks are intended to minimize subjective bias among VLM judges.The structured descriptions make score levels more explicit and comparable.
D. Detailed Rubric on five tasks
The detailed rubrics operationalize the five GE-Score dimensions across GEBench’s task types. They distinguish immediate instruction following, long-horizon coherence, fictional-app plausibility, and grounding alignment.
- Five GE-Score dimensions: Across tasks, the rubrics decompose GOAL, LOGIC, CONS, UI, and QUAL into explicit descriptions spanning performance tiers from incomplete to exceptional.This structure supplies concrete criteria for comparing functional and visual outcomes.
- Scoring framework: The detailed rubric framework bridges qualitative visual inspection and quantitative performance metrics.Its task-specific criteria provide the operational basis for converting visual and functional judgments into scores.
- Single-Step Transition: Single-Step Transition rubrics evaluate whether the generated GUI state matches the caption-described change.The goal criterion asks whether the change is achieved completely and unambiguously.
- Multi-Step Planning: Multi-Step Planning rubrics evaluate final global-goal achievement across five-step GUI trajectories.The task emphasizes accumulated errors, temporal coherence, and consistency across the trajectory.
- Zero-shot Virtual GUI: Zero-shot Virtual GUI rubrics assess the imaginative plausibility and structural integrity of non-existent applications.The rubric category is identified as Fiction-App/Real-App evaluation.
- Grounded Generation: Grounding rubrics evaluate whether generated images match the expected effect and align generated content with specified coordinates.The grounding task is defined around effect alignment and pixel-level coordinate alignment.