PaperlayerView exact revision on arXiv

Source-linked AI summary

GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments

Lin Fu, Zheyuan Yang, Tianhui Zhang, Jinbiao Wei, Guo Gan, Boxu Liu, Yilun Zhao, Yu Rong

arXiv:2609.00048v1cs.CLcs.AI

TL;DR

GUI world models are often assessed as isolated next-screen predictors, leaving their consistency under repeated use in multi-step agent interaction under-tested. GUI-CC benchmarks them as agent environments through offline reference-action trajectories and online probing-agent loops. Across both tracks, current models capture local action dynamics but still struggle to preserve task-relevant state through rollouts.

  • Problem

    Prior evaluations typically assess semantic or visual transition quality, scripted GUI generation, or isolated outputs, but do not directly test contextual coherence under repeated state reuse.

    Source: §1 · p. 1

  • Method

    GUI-CC evaluates GUI world models as agent environments through offline reference-action trajectories and online probing-agent loops.

    Source: §1 · pp. 1–2

  • Results

    Across both tracks, current models capture local action dynamics but struggle to preserve task-relevant state through rollouts; the best offline model reaches 16.7 reference-action progress.

    Source: §4.2 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    Current systems remain short of the contextual consistency needed for scalable agent training and evaluation.

    Source: §1 · p. 2

    The benchmark currently focuses on mobile GUI tasks and uses a single fixed probing agent in its online track.

    Source: Limitations · p. 9

Abstract

from arXiv · show

GUI world models are increasingly evaluated as one-step next-screen predictors, yet their intended use is often as multi-step environments for GUI agents. This mismatch leaves a key requirement under-tested: generated states must remain contextually consistent when they are repeatedly reused for future interaction. We introduce GUI-CC, a benchmark that evaluates contextual consistency of GUI world models as agent environments rather than isolated next-screen predictors. GUI-CC contains two complementary tracks: an offline reference-action track that rolls models along real mobile GUI trajectories, and an online agent-loop track that lets fixed probing agents interact with model-generated UIs. We construct 500 offline trajectory tasks from GUIOdyssey and 200 emulator-verified online tasks across 30 mobile apps. GUI-CC evaluates transition fidelity, transition plausibility, contextual consistency, and task progress. Experiments show that plausible single-step generation does not guarantee reliable environment simulation: current models often produce usable-looking screens while failing to preserve task-relevant context or support executable multi-step rollouts.

1 Introduction

GUI-CC evaluates GUI world models as multi-step agent environments, where generated states must preserve task-relevant context across repeated reuse. Its results show that plausible single-step predictions do not ensure reliable rollouts.

1 Introduction

  • Live environments are interactive but costly to reset and execute, while logged trajectories are scalable but fixed; GUI world models offer action-conditioned surrogate environments for simulated rollouts, comparison, and planning.Live benchmarks use task initialization and execution-based success checks (Zhou et al., 2024; Xie et al., 2024; Rawles et al., 2025), while mobile datasets provide logged demonstrations (Rawles et al., 2023; Li et al., 2024; Lu et al., 2025).Source: §1 · p. 1
  • Contextual consistency requires app identity, navigation, task-relevant entities, selected or created state, and action affordances to remain coherent when generated screens become inputs to later actions.Prior evaluations typically assess semantic or visual transitions, scripted generation, or isolated outputs rather than coherence under repeated state reuse.Source: §1 · p. 1
  • GUI-CC pairs offline reference-action rollouts with online probing-agent loops, covering 500 GUIOdyssey tasks and 200 emulator-verified tasks across 30 apps and 18 templates.It evaluates transition fidelity, transition plausibility, contextual consistency, and task progress to separate local prediction quality from trajectory-level environment utility.Source: §1 · pp. 1–2
  • Reasonable local transition scores do not ensure reliable rollouts: models lose task-relevant state, drift across apps or pages, or fail later reference actions; history helps in some cases, but task progress remains limited.The limitations are especially pronounced in longer rollouts and closed-loop agent interaction.Source: §1 · p. 2

2 Related Work

Related work covers live GUI interaction, trajectory datasets, several forms of GUI world models, and evaluations that do not directly test coherence under repeated state reuse.

GUI Agent Benchmarks and Interaction Datasets

GUI World Models

Evaluating GUI World Models

  • Existing evaluations score next-state fidelity, semantic prediction, scripted multi-step generation, or planning utility, but GUI-CC tests coherence when predicted states are repeatedly reused.Its offline track measures reference-action executability and context persistence; its agent-loop track measures milestone progress, loops, and stagnation. GEBench reports temporal coherence but does not test state-reuse consistency.Source: §2 · pp. 2–3, Table 1 · p. 3

3 GUI-CC Benchmark

GUI-CC benchmarks whether GUI world models can sustain multi-step, action-conditioned interaction, not just generate plausible next screens. It pairs offline reference-action rollouts with online agent-loop evaluation.

3.1 Problem Formulation

  • A rollout is contextually consistent when generated screens admit a plausible latent-state sequence and each action remains executable; GUI-CC evaluates this across repeated reuse.The model does not observe the environment’s latent state, so continuity must be reconstructed from generated UIs.Source: §3.1 · p. 3

3.2 Benchmark Tracks

  • The offline track fixes real-trajectory actions to isolate model behavior, while the online track lets a frozen probing agent react to generated screens in a closed loop.Offline reference screenshots score predictions but are not rollout inputs; online interaction ends at task completion, unusable output, or stagnation.Source: §3.2 · pp. 3–4

3.3 Action Representation

  • Actions use a semantic schema with descriptions and parameters rather than raw pixels, enabling offline executability scoring by whether the relevant affordance remains visible.The online track also includes terminate and answer actions; offline trajectories use five GUIOdyssey action types.Source: §3.3 · p. 4

3.4 Offline Reference-Action Track

  • The offline track selects 500 quality- and diversity-balanced tasks from GUIOdyssey (Lu et al., 2025), filtering and normalizing trajectories before plausibility checks.GPT-5.5 verifies recorded action-conditioned transitions and prioritizes trajectories with more strongly state-dependent steps, while selection balances apps, actions, task categories, steps, and interaction patterns.Source: §3.4 · pp. 4–5

3.5 Online Agent-Loop Track

  • The online track spans 30 mobile apps and uses app-grounded tasks with ordered milestones to assess task advancement and retention of context.Annotators curate templates, instantiate goals and initial states, and verify tasks in an Android emulator; step budgets are typically about twice human completion length.Source: §3.5 · p. 5

3.6 Quality Control

  • An independent second-pass audit checks trajectory coherence offline and task clarity, milestone observability, and step budgets online; unreliable tasks are revised or removed.VLM judges use fixed model versions and frozen prompts released with the benchmark.Source: §3.6 · p. 5

3.7 Evaluation Metrics

  • GUI-CC evaluates transition fidelity, transition plausibility, contextual consistency, and task progress, separating local prediction quality from multi-step coherence and executability.Fidelity covers global visual similarity and element/layout structure; plausibility covers action adherence, action identifiability, and GUI usability; consistency covers state/context persistence and rollout dynamics; progress measures reference-action or ordered-milestone completion.Source: §3.7 · pp. 5–6
  • Metrics use rubric-mapped VLM judgments or clamped encoder similarities, with step-level scores averaged across transitions and episodes and trajectory-level scores computed once per episode.Offline reference-action progress is the longest supported reference-action prefix divided by episode length.Source: §3.7 · p. 6

4 Experiments

Experiments find that current GUI world models often fail to sustain task progress across rollouts, even when generated screens look usable or history improves contextual consistency. The analysis identifies limits in automatic metrics and recurring rollout failures involving missing world knowledge, error accumulation, and context inconsistency.

4.1 Experimental Setup

  • The evaluation covers 12 GUI world models across two output modalities, yielding 18 configurations per track, with GPT-5.5 as the fixed online probing agent.Models predict action-conditioned next states as renderable HTML or images; scores are task averages on a 0–100 scale.Source: §4.1 · pp. 6–7

4.2 Main Results

  • 16.7 reference-action progress is the best offline result; online GPT-5.5 achieves the strongest milestone progress, while specialized GUI world models remain substantially lower.Task progress is the most difficult dimension across both tracks, and most offline rollouts quickly become unable to support later reference actions.Source: §4.2 · p. 7
  • Offline transition fidelity shows a gap between relatively high embedding similarity and fine-grained element alignment and layout integrity, whose best scores remain below 20.Models can preserve rough screen semantics while misplacing interaction-critical elements, text, controls, and layouts.Source: §4.2 · p. 7
  • High GUI usability can coexist with poor action grounding and low task progress; Flux.2-dev has high usability in both tracks but near-zero task progress.Plausible-looking interfaces may fail to reflect the action or preserve the correct app or page state.Source: §4.2 · pp. 7–8
  • History raises GPT-5.5's online overall score from 82.5 to 90.3, but Code2World's offline reference-action progress changes only from 9.1 to 9.5.History generally improves contextual-consistency metrics, including Code2World's offline state/context persistence and rollout dynamics, without proportionate progress gains.Source: §4.2 · p. 8

4.3 Analysis

  • Automatic metrics are scalable diagnostics but can miss fine-grained task-state errors or penalize a different yet plausible and executable next state.VLM judges may under-penalize errors when local visual continuity and layout seem reasonable; fidelity metrics compare against a single ground-truth next UI.Source: §4.3 · p. 8
  • Manual annotation of 200 failed rollouts attributes about 42% to missing world knowledge, 33% to error accumulation, and 25% to context inconsistency.Autoregressive reuse can amplify early deviations, while context failures include lost app identity, corrupted navigation history, forgotten text, and missing saved states.Source: §4.3 · p. 8

5 Conclusion

GUI-CC evaluates GUI world models as multi-step agent environments, combining online agent-loop probing with scalable offline trajectory diagnosis. The results reveal that plausible screens and contextual-consistency scores do not reliably yield task progress, pointing to a need for persistent state representations and stronger action-effect knowledge.

5 Conclusion

  • Current GUI world models capture some local action dynamics but often fail to preserve task-relevant state, and plausible, contextually consistent screens do not reliably support task progress.The findings expose a gap between convincing screens and executable environments.Source: §5 · p. 8
  • GUI-CC evaluates world models beyond isolated next-screen fidelity through online agent-loop tests of GUI usability and scalable offline multi-step trajectory diagnosis.Its two-track design makes contextual consistency and task completion across interactions central evaluation targets.Source: §5 · p. 8
  • Limited gains from short histories suggest that future models need persistent state representations and stronger action-effect knowledge, not visual memory alone.Closing this gap is described as essential for scalable GUI agents.Source: §5 · p. 8

Limitations · A Model Input Prompt Templates

GUI-CC is an initial mobile-GUI benchmark whose future development could broaden environments, agents, consistency checks, and model interfaces. The appendix lists prompt templates for the probing agent and HTML- and image-generation world models.

Limitations

  • The benchmark currently standardizes mobile GUI tasks; extending its contextual-consistency framework to web, desktop, and multi-device settings could cover richer persistent state.Examples include browser sessions, file systems, and cross-device context.Source: Limitations · p. 9
  • Using one fixed probing agent controls world-model comparisons, but agents with varied planning, recovery, and prompting could better assess support for diverse interaction policies.Source: Limitations · p. 9
  • VLM judges offer scalable diagnostics, while future assessments could add human-validated checks, structured UI-state probes, and executable verification of fine-grained task states.Examples include selected items, saved entities, stale queries, and persistent settings.Source: Limitations · p. 9
  • Future releases could support model-specific inputs and outputs, including structured state, memory modules, simulator APIs, and hybrid code-image outputs, under a common evaluation protocol.Source: Limitations · p. 9

A Model Input Prompt Templates

B Evaluation Prompts · C Model List

The appendix documents GUI-CC’s VLM judge prompts and the prompt templates for its online probing agent and Code2World rollouts. It notes adaptations to the rollout setting and action schema and says all judge prompts are released with the benchmark.

B Evaluation Prompts

  • GUI-CC introduces judge prompts for Suse, Scp, Srd, Srap, and Smp, shown in Figures 14–18, respectively.Source: B Evaluation Prompts · p. 11
  • The remaining VLM-judged metrics, Sele, Slay, Sad, and Sid, follow Code2World; Sad and Sid prompts are adapted for autoregressive rollouts and GUI-CC’s action schema.All judge prompts are released with the benchmark.Source: B Evaluation Prompts · p. 11
  • The online probing agent selects one action per step through the computer tool and emits absolute pixel coordinates on the observed screenshot.Source: Figure 3 · p. 12
  • The appendix includes a Code2World prompt template without history.Source: Figure 4 · p. 13
  • It also labels a Code2World-8B prompt template as using history and provides an additional system prompt for multi-step rollout.Source: B Evaluation Prompts · p. 14
  • Figure 5 presents a Code2World prompt template with observation-action history.Source: Figure 5 · p. 14

D Case Study

Table 4 lists the world models evaluated in the benchmark, spanning proprietary APIs and open-source checkpoints across code/HTML-output and direct image-generation paradigms.

D Case Study

  • The evaluated models cover proprietary APIs and open-source checkpoints, using code/HTML output or direct image generation.Table 4 summarizes the world models evaluated in the benchmark.Source: D Case Study · p. 15, Table 4 · p. 16

E Offline Data Construction Details

This appendix specifies the concrete rules used in the offline trajectory pipeline described in subsection 3.4.

E Offline Data Construction Details

E.1 Filtering and Normalization · E.2 Single-Step Plausibility Verification

The benchmark filters GUIOdyssey trajectories by completion, length, action composition, and device, then normalizes coordinates for consistent model inputs. A second stage rejects trajectories with any transition whose outcome is implausible given the recorded action.

E.1 Filtering and Normalization

  • Only successful GUIOdyssey episodes with 5–20 steps are retained, excluding failed or interrupted runs and limiting trajectory complexity.Source: §E.1 · p. 16
  • Tap, type, and long_press actions must exceed 50% of steps; no action may repeat more than three times consecutively.Source: §E.1 · p. 16
  • HOME and app-switching operations may account for at most 50% of steps, and tablet and foldable recordings are excluded.Source: §E.1 · p. 16
  • After filtering, normalized dataset coordinates are restored to pixel level so benchmarked models share a common input convention.Source: §E.1 · p. 16

E.2 Single-Step Plausibility Verification

  • Transitions are flagged when their outcomes cannot plausibly follow from the recorded action alone, such as skipping required search entry and submission.Source: §E.2 · p. 16
  • Tapping an app icon and landing on a deep in-app page instead of its entry screen is another implausible transition; one flagged transition discards the entire trajectory.Source: §E.2 · p. 16

E.3 Strongly State-Dependent Steps

Strongly state-dependent steps require information established across multiple earlier states, not just the immediately preceding screen. The examples show how missing history can break long-horizon context and task completion, while judge prompts assess persistence, rollout dynamics, and reference-action progress.

E.3 Strongly State-Dependent Steps

  • A step is strongly state-dependent when correct execution relies on information established by multiple earlier states rather than only the immediately preceding screen.Source: §E.3 · p. 16
  • The VLM judge prompts assess state and context persistence, action-controlled temporal dynamics, and whether rollouts continue to support reference semantic actions.The prompts use task instructions, action sequences, and chronological predicted screenshots; the reference-action prompt also receives reference screenshots for the current step and the next reference action when available.Source: §E.3 · pp. 24–26
  • Without history, the eBay Watchlist rollout loses long-horizon context and fails, while the history-conditioned rollout preserves task state and reaches the expected final UI.The task is to search for a vintage camera, add a listing to the Watchlist, and confirm it appears there.Source: §E.3 · p. 30, Figure 21 · p. 30
  • On a Yahoo Sports-to-Calendar task, Code2World fails in most transitions, whereas GPT Image 2 succeeds early but drifts later.The task is to find the next NBA game date and time and create a Calendar reminder.Source: §E.3 · p. 31, Figure 22 · p. 31

Related papers

Loading 2609.00048v1…