GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments

Lin Fu; Zheyuan Yang; Tianhui Zhang; Jinbiao Wei; Guo Gan; Boxu Liu; Yilun Zhao; Yu Rong

Source-linked AI summary

GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments

Lin Fu, Zheyuan Yang, Tianhui Zhang, Jinbiao Wei, Guo Gan, Boxu Liu, Yilun Zhao, Yu Rong

arXiv:2609.00048v1Aug 30, 2026cs.CLcs.AI

TL;DR

GUI world-model evaluations often test isolated next-screen prediction even though agents need reusable environments that preserve context across interactions. GUI-CC introduces offline reference-action and online agent-loop tracks to measure this property, and finds that plausible local screens do not reliably support multi-step task progress. The benchmark is currently focused on mobile GUIs and fixed probing-agent evaluation.

Problem
GUI world models are often evaluated as isolated next-screen predictors, leaving contextual consistency under repeated state reuse insufficiently tested.
Method
GUI-CC evaluates world models with offline reference-action rollouts and online closed-loop interaction, using metrics for transition fidelity, plausibility, contextual consistency, and task progress.
Results
Plausible single-step generation does not guarantee reliable rollout behavior; current models often lose task-relevant state or fail to support later actions, with the best offline model reaching only 16.7 reference-action progress.
Takeaways & Limitations
High visual plausibility and contextual-consistency scores do not reliably translate into executable task progress across autoregressive rollouts.
Takeaways & Limitations
The benchmark focuses on mobile GUI tasks and uses a single fixed probing agent for controlled comparisons.

Abstract

from arXiv · show

GUI world models are increasingly evaluated as one-step next-screen predictors, yet their intended use is often as multi-step environments for GUI agents. This mismatch leaves a key requirement under-tested: generated states must remain contextually consistent when they are repeatedly reused for future interaction. We introduce GUI-CC, a benchmark that evaluates contextual consistency of GUI world models as agent environments rather than isolated next-screen predictors. GUI-CC contains two complementary tracks: an offline reference-action track that rolls models along real mobile GUI trajectories, and an online agent-loop track that lets fixed probing agents interact with model-generated UIs. We construct 500 offline trajectory tasks from GUIOdyssey and 200 emulator-verified online tasks across 30 mobile apps. GUI-CC evaluates transition fidelity, transition plausibility, contextual consistency, and task progress. Experiments show that plausible single-step generation does not guarantee reliable environment simulation: current models often produce usable-looking screens while failing to preserve task-relevant context or support executable multi-step rollouts.

1 Introduction

GUI-CC addresses the gap between isolated next-screen evaluation and the need for GUI world models to support coherent, executable multi-step interaction. Its two-track benchmark evaluates contextual consistency and task progress, showing that plausible local predictions can still fail during long-horizon rollout.

Live environments support interactive execution but are costly to reset, while logged trajectories are scalable yet cannot simulate unseen actions.
GUI world models learn action-conditioned transitions as surrogate environments for simulated rollouts, model comparison, and planning.
Contextual consistency requires generated states to preserve app identity, navigation context, task-relevant entities, selected or created state, and action affordances across reuse.
GUI-CC combines an offline reference-action task with an online agent-loop task to test controlled multi-step executability and closed-loop task progress.
500 offline trajectory tasks and 200 online tasks across 30 mobile apps evaluate transition fidelity, plausibility, contextual consistency, and task progress.
Plausible single-step prediction does not guarantee reliable rollout behavior: models can lose task-relevant state, drift across apps or pages, and fail later actions.

2 Related Work

Prior GUI resources provide either costly live interaction or scalable but fixed demonstrations, while existing world-model evaluations largely emphasize isolated or scripted predictions. GUI-CC instead evaluates whether generated states remain coherent when repeatedly reused during agent rollouts.

Live GUI benchmarks evaluate execution in web, desktop, and mobile applications, but their reliance on live execution makes large-scale rollout costly and hard to reproduce.
Large-scale mobile interaction datasets provide screenshots, actions, and annotations from real trajectories, but they are demonstrations rather than learned environments.
GUI world models predict action consequences in graphical interfaces using textual, structured, image-level, or renderable-code state representations.
Existing evaluations score one-step fidelity, semantic prediction, scripted multi-step generation, or downstream planning utility rather than repeated state-reuse consistency.
GUI-CC distinguishes predicted-state feedback, agent-in-loop evaluation, direct world-model metrics, and consistency evaluation while measuring reference-action executability and context persistence.

3 GUI-CC Benchmark

GUI-CC evaluates learned GUI transitions as reusable environments through autoregressive rollouts, using complementary reference-action and agent-loop tracks. The benchmark operationalizes consistency through latent-state continuity, executable actions, and multi-step progress.

GUI-CC evaluates GUI world models as agent environments under multi-step action-conditioned state reuse, with Figure 2 summarizing its construction and track statistics.
3.1 Problem Formulation: A GUI world model replaces the environment transition with a learned function that generates each next observation from generated history and the current action.
3.1 Problem Formulation: Generated observations become later inputs, so rollout continuity must be reconstructed from the generated UIs without access to the latent environment state.
3.1 Problem Formulation: A rollout is contextually consistent when its observations admit a plausible latent-state sequence and every action remains executable across multiple reused states.
3.2 Benchmark Tracks: The offline track replays reference semantic actions without feeding reference screenshots into the rollout, whereas the online track lets a frozen probing agent act on generated UIs.
3.2 Benchmark Tracks: The benchmark constructs offline tasks from filtered, normalized, plausibility-verified GUIOdyssey trajectories and selects 500 tasks for quality and diversity.
3.2 Benchmark Tracks: The online track spans 30 mobile apps and uses curated, instantiated, emulator-verified tasks with ordered milestones and a human-derived step budget.
3.7 Evaluation Metrics: Transition plausibility measures action adherence, action identifiability, and GUI state usability to assess whether predicted updates remain usable for interaction.

4 Experiments

GUI-CC evaluates GUI world models as action-conditioned environments through offline and online multi-step settings. Results show that local visual plausibility and consistency scores do not reliably yield executable task progress.

Experimental Setup: The evaluation covers 12 GUI world models across two output modalities, producing 18 evaluated configurations per track.Code-based outputs are rendered into screenshots, while other models generate images directly.
Experimental Setup: Models predict each next UI from the current UI and semantic action, with optional variants conditioning on up to three previous observation-action pairs.GPT-5.5 serves as the fixed probing agent and VLM judge, while transition fidelity is measured only offline.
Main Results: 16.7 reference-action progress is achieved by the best offline model, making task progress the most difficult dimension across both tracks.Online results identify GPT-5.5 as strongest on milestone progress, while specialized GUI world models remain substantially lower.
Main Results: Best fine-grained element-alignment and layout-integrity scores remain below 20 despite relatively high embedding-based similarity.Models often preserve rough screen semantics while misplacing interaction-critical elements, text, controls, and layouts.
Main Results: Flux.2-dev achieves high usability in both tracks but near-zero task progress, showing that readable interfaces can lack correct action consequences.Plausible screens may fail to preserve the correct app or page state.
Main Results: History raises GPT-5.5's online overall score from 82.5 to 90.3, but Code2World's offline reference-action progress changes only from 9.1 to 9.5.The results indicate improved continuity without a proportional solution to long-horizon state maintenance or action-effect propagation.
Analysis: Automatic metrics provide scalable diagnostics but can miss fine-grained task-state errors or penalize alternative plausible and executable next states.Examples include wrong selections, stale queries, unpropagated saved states, and visually plausible but task-incorrect pages.
Analysis: Manual analysis finds missing world knowledge, error accumulation, and context inconsistency account for about 42%, 33%, and 25% of sampled failures, respectively.Autoregressive reuse amplifies early deviations, while context failures include lost app identity, queries, saved states, and selected objects.

5 Conclusion

GUI-CC extends GUI world-model evaluation from isolated next-screen fidelity to contextual consistency and task completion across interactions. Its results show that current models struggle to preserve task-relevant state during autoregressive rollout, motivating persistent state representations and stronger action-effect knowledge.

Conclusion: GUI-CC evaluates whether GUI world models preserve contextual consistency and task completion across interactions through online agent loops and offline multi-step diagnosis.The two tracks keep environment usability and scalable trajectory analysis explicit.
Conclusion: Current models capture parts of local action dynamics but struggle to preserve task-relevant state through autoregressive rollout.High visual plausibility and contextual-consistency scores do not reliably translate into task progress.
Conclusion: Limited gains from short histories suggest that future models need persistent state representations and stronger action-effect knowledge rather than visual memory alone.This conclusion follows from the observed gap between continuity metrics and task progress.

Limitations

GUI-CC is an initial mobile-focused benchmark for evaluating GUI world models as reusable agent environments. Future work should broaden environments, probing agents, state verification, and supported model interfaces.

GUI-CC currently focuses on mobile GUI tasks, where screenshots, action semantics, and emulator verification can be standardized at scale.
The online agent-loop track uses one fixed probing agent, limiting coverage of planning strength, recovery strategy, and prompting style.
Future evaluations should include agents with diverse planning, recovery, and prompting behaviors to assess support for varied interaction policies.
Future benchmark versions could add human-validated checks, structured UI-state probes, and executable verification for fine-grained persistent states.
GUI-CC could support richer model interfaces, including structured states, memory modules, simulator APIs, and hybrid code-image outputs, while retaining common rollout metrics.

B Evaluation Prompts

The appendix documents the prompts used for probing agents, world models, and VLM judges, including Code2World variants with and without observation-action history.

B Evaluation Prompts: VLM-judged evaluation includes GUI-CC metrics Suse, Scp, Srd, Srap, and Smp, alongside adapted Sad and Sid prompts for autoregressive rollouts.
B Evaluation Prompts: The online track uses a single-stage probing agent that selects one action per step through the computer tool using absolute screenshot coordinates.
B Evaluation Prompts: Its image rules interpret red circles as click or long-press targets and red arrows as scroll or swipe directions without rendering those cues.
B Evaluation Prompts: Structural rules require raw HTML with visible content wrapped in a specified root element and rendered within a fixed target container.
B Evaluation Prompts: The content-generation logic requires actions to produce corresponding UI results, semantic image placeholders instead of real URLs, and simple SVG or Unicode icons.
B Evaluation Prompts: The Code2World without-history prompt frames the model as a UI state-transition simulator predicting the next state from a screenshot and interaction.
B Evaluation Prompts: The Code2World history variant adds observation-action history for multi-step rollout prompting.

D Case Study

The case-study materials identify the benchmark’s evaluated world models and provide a table summarizing them.

Table 4 summarizes the world models evaluated in GUI-CC.The table covers proprietary APIs and open-source checkpoints across code/HTML-output and direct image-generation paradigms.

E Offline Data Construction Details

The appendix specifies the concrete rules underlying the offline trajectory pipeline described in subsection 3.4.

The appendix provides concrete rules for the offline trajectory pipeline described in subsection 3.4.

E.1 Filtering and Normalization

The filtering pipeline removes failed, overly short or long, and operation-dominated GUIOdyssey trajectories, then rejects any trajectory containing an implausible action–outcome transition.

Failed or interrupted episodes are excluded, retaining only trajectories whose final action marks successful task completion.
Trajectory length is restricted to 5–20 steps because shorter episodes provide little long-horizon consistency signal and longer ones add uncontrolled complexity.
Meaningless-operation-dominated trajectories are further removed using additional heuristic requirements.
Transitions are checked for outcomes that cannot plausibly result from the recorded action alone, such as skipping intermediate search or navigation states.
A trajectory is retained only when every transition passes the plausibility check; one flagged transition discards the entire trajectory.

E.3 Strongly State-Dependent Steps

Strongly state-dependent steps require information established across multiple earlier states, motivating history-aware rollout prompts and evaluations of persistence, dynamics, and progress.

E.3 Strongly State-Dependent Steps: A step is strongly state-dependent when correct execution relies on information from multiple earlier states rather than only the immediately preceding screen.
Prompting: History-aware prompts provide observation–action history while asking models to predict the next UI state after an interaction.
Evaluation: The usability judge evaluates only screenshot quality, so a clean but incorrect UI can score highly while action and task correctness are measured elsewhere.
Evaluation: The judge for state and context persistence evaluates a multi-step rollout using the task instruction, action sequence, and chronologically ordered predicted screenshots.
Evaluation: Action-controlled rollout dynamics are evaluated separately from whether a rollout continues to support the reference semantic action sequence.
Qualitative comparison: Without history, the model loses long-horizon contextual consistency, producing an incorrect trajectory and failed final UI in the watchlist task.
Qualitative comparison: Qualitative examples show distinct rollout failures: Code2World fails across most transitions, whereas GPT Image 2 succeeds early before later drift.