PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Visual Code Mobile World Models

Woosung Koh, Sungjun Han, Segyu Lee, Se-Young Yun, Jamin Shin

arXiv:2602.01576v2cs.LGcs.AIcs.CV

TL;DR

Existing mobile GUI world models either lose visual information through text representations or depend on slow, complex visual-generation pipelines. This paper introduces gWorld, which uses a single VLM to predict GUI next states as renderable web code and reports a new accuracy–model-size frontier, with scaling and downstream-policy gains. The approach is supported by evaluations across in-distribution and out-of-distribution benchmarks.

  • Problem

    Text-based world models sacrifice fine-grained GUI visual information, while visual models struggle with precise text and often require complex pipelines.

  • Method

    gWorld uses open-weight 8B and 32B VLMs to generate renderable web code for next GUI states, trained with automatically converted trajectory data and reasoning traces.

  • Results

    Across six benchmarks, gWorld 32B and 8B achieve the best and second-best IAcc., outperforming frontier open-weight models up to 50.25× larger and establishing a new accuracy–model-size pareto frontier.

  • Takeaways & Limitations

    Renderable-code world modeling preserves GUI structure and text fidelity while avoiding the complexity and latency of pixel-based pipelines, and stronger world modeling improves downstream policy performance.

  • Takeaways & Limitations

    Image-generation baselines can achieve high visual similarity yet fail to model mobile dynamics, reaching only 10.9 to 29.1% IAcc.

Abstract

from arXiv · show

Mobile Graphical User Interface (GUI) World Models (WMs) offer a promising path for improving mobile GUI agent performance at train- and inference-time. However, current approaches face a critical trade-off: text-based WMs sacrifice visual fidelity, while the inability of visual WMs in precise text rendering led to their reliance on slow, complex pipelines dependent on numerous external models. We propose a novel paradigm: visual world modeling via renderable code generation, where a single Vision-Language Model (VLM) predicts the next GUI state as executable web code that renders to pixels, rather than generating pixels directly. This combines the strengths of both approaches: VLMs retain their linguistic priors for precise text rendering while their pre-training on structured web code enables high-fidelity visual generation. We introduce gWorld (8B, 32B), the first open-weight visual mobile GUI WMs built on this paradigm, along with a data generation framework (gWorld) that automatically synthesizes code-based training data. In extensive evaluation across 4 in- and 2 out-of-distribution benchmarks, gWorld sets a new pareto frontier in accuracy versus model size, outperforming 8 frontier open-weight models over 50.25x larger. Further analyses show that (1) scaling training data via gWorld yields meaningful gains, (2) each component of our pipeline improves data quality, and (3) stronger world modeling improves downstream mobile GUI policy performance.

1. Introduction

Mobile GUI world models must preserve visual detail while modeling action-contingent transitions. gWorld addresses this trade-off with open-weight code-generating models and reports strong benchmark, scaling, ablation, and downstream-policy results.

  • Motivation: Text-based world models discard fine-grained GUI layout and visual attributes, while prior visual approaches use complex pipelines with substantial overhead and limited reproducibility.These limitations include losses in iconography, typography, and color, plus reliance on external OCR, VLM, diffusion, and text-filling components.
  • Approach: gWorld introduces open-weight 8B and 32B self-contained visual mobile GUI world models that predict next states through renderable web code.The paper presents code-based representation as an alternative to direct pixel generation and describes the models as the first of their kind for this setting.
  • Approach: Its data framework repurposes offline policy trajectories, converts pixel next states into renderable web code, and synthesizes look-ahead reasoning traces for supervised training.The resulting data targets both reasoning traces and code-based next-state prediction.
  • Results: Across six in- and out-of-distribution benchmarks, the models outperform 8 frontier open-weight image- and code-generation models up to 50.25× larger.The paper reports gWorld 32B and 8B as the best and second-best performers in IAcc. across all six benchmarks.
  • Results: Scaling datasets from 37K to 240K samples yields predictable gains, while ablations show that each method component contributes meaningfully.The dataset scaling analysis reports a power-law relationship, and the ablation results support the contribution of the pipeline components.
  • Results: Improved world-model performance translates to gains in downstream mobile GUI policy performance.The paper presents this relationship as an experimental result of integrating the world model into mobile GUI agents.

2. gWorld: Generative Visual Code World Modeling

gWorld formulates mobile GUI world modeling as predicting the next state with a generative model, using renderable web code rather than pixels. Its training framework converts policy trajectories into state-action next-state data, relabels visual targets as code, and adds look-ahead reasoning traces.

  • 2.1. Problem Setting: A world model represents the transition distribution pθ(Sₜ₊₁ | Sₜ, Aₜ) over next states conditioned on the current state and action.The formulation builds on a Markov Decision Process, where S and A denote state and action spaces.
  • 2.2. Motivation: Limitation of Generating Pixel-based Next State: Renderable web code addresses pixel-generation limitations by enabling VLMs to produce legible, contextually coherent GUI text and structured next states.The approach post-trains VLMs to generate the next GUI state as web code instead of directly generating pixels.
  • 2.4. World Model Training Data Generation: The data framework repurposes offline mobile-agent trajectories into state-action-conditioned next-state pairs matching the world-model objective.This transformation reduces each episode from T examples to T − 1 state transitions.
  • 2.4. World Model Training Data Generation: Next-state pixel supervision is automatically relabeled as renderable web code using a frontier image-to-web-code model.The resulting code targets are suitable for training a VLM that outputs text rather than pixels.
  • 2.4. World Model Training Data Generation: Free look-ahead reasoning traces use the ground-truth next state to align textual state-change reasoning with the generated web code.This decomposes world modeling into predicting state changes in natural language and converting that description into web code.
  • 2.4. World Model Training Data Generation: The generated dataset contains 260K samples derived from Android in the Wild, GUIOddyssey, AndroidControl, and Android Multi-annotation Expo trajectories.Gemini 3 Flash generates the data, while Qwen3 VL 8B and 32B serve as the base training models.

3. MWMBENCH: Comprehensive Mobile GUI World Modeling Benchmark

MWMBENCH evaluates mobile GUI world models in the native visual action space across diverse in-distribution and out-of-distribution settings. It compares gWorld with larger baselines and reports a new accuracy–model-size pareto frontier with low render-failure rates.

  • 3. MWMBENCH: Comprehensive Mobile GUI World Modeling Benchmark: MWMBENCH comprises state-action-next-state tuples from six data sources for systematic measurement of mobile GUI next-state prediction quality.The benchmark spans diverse applications, tasks, interaction patterns, and languages.
  • 3. MWMBENCH: Comprehensive Mobile GUI World Modeling Benchmark: Unlike text-only benchmarks, MWMBENCH evaluates world models in the native visual modality, preserving rich GUI details and semantics.Its action representation also retains original coordinate-space actions rather than converting them to text.
  • 3. MWMBENCH: Comprehensive Mobile GUI World Modeling Benchmark: MWMBENCH supports four in-distribution and two out-of-distribution evaluations for comprehensive assessment.The in-distribution sets are sampled from AitW, GUIO, AC, and AMEX, while AW and KA provide held-out OOD settings.
  • 3. MWMBENCH: Comprehensive Mobile GUI World Modeling Benchmark: gWorld 8B and 32B establish a new pareto frontier, consistently outperforming significantly larger image-generation and VLM baselines across ID and OOD benchmarks.The compared larger models include Llama 4 402B-A17B and Qwen3-VL 235B-A22B.
  • 3. MWMBENCH: Comprehensive Mobile GUI World Modeling Benchmark: <1% Render Fail accompanies +45.7% and +27.1% average Instruction Accuracy gains over Qwen3 VL 8B and 32B, respectively.These results are reported for the code-based gWorld approach in the main benchmark comparison.

4. Empirical Study

Across six benchmarks, gWorld achieves strong action-conditioned world-modeling performance, generalizes to out-of-distribution settings, and improves downstream mobile GUI policies. Analyses show that its gains reflect active state transformation, scalable data quality, and effective code-generation components.

  • World Modeling Results: gWorld 32B and 8B achieve the best and second-best IAcc. performance, respectively, across all six benchmarks.Their OOD performance does not significantly degrade compared with in-distribution settings.
  • World Modeling Results: gWorld 32B and 8B rank first in Similarity on all but two benchmarks, where Emu3.5 34B is highest.
  • Limitation of Image-gen Models: 10.9 to 29.1% IAcc. is achieved by image-generation baselines despite their high visual similarity scores.Their similarity can result from copying the input state rather than modeling the semantic transition.
  • Limitation of Image-gen Models: ρ = 0.92 for Emu3.5 34B and ρ = 0.74 for Qwen-Image-Edit 20B, compared with ρ ≈0.4 for gWorld, in correlations between transition similarity and output similarity.Emu3.5 produces near-zero similarity gains, whereas gWorld 32B shows substantial variance and actively predicts structural changes based on the action.
  • Data Scaling: 37K, 77K, 129K, and 240K training samples produce monotonic performance gains following a power law, with an average R2 of 0.948.The analysis supports performance projection as the dataset is expanded.
  • Human Evaluation: 77.5% and 61.4% pairwise win rates are observed for gWorld 32B and 8B, whose average human-evaluation ranks are 1.68 and 2.16.Automatic metrics show Spearman ρ = 0.806 and Kendall τ = 0.600 against human preferences.
  • Downstream Policy Performance: A 1.0 percentage point increase in world-modeling performance corresponds to a 0.49 percentage point improvement in downstream policy performance on average.This relationship is observed across gWorld 8B, Qwen3 VL 32B, and Qwen3 VL 8B with two backbone policies and K = 3.

5. Additional Discussion

The discussion compares gWorld with prior visual world-model pipelines and reports practical efficiency, benchmark, and application implications. It also identifies direct comparison limits and points to future work.

  • VIMO cannot be compared directly under the paper’s experimental settings because its diffusion model is not open-weight.The paper instead compares Similarity v1 scores on two splits where the experiments are directly comparable.
  • 81%, 81.9%, and 74% are the average Similarity v1 scores for gWorld 8B, gWorld 32B, and VIMO, respectively, on two MWMBENCH splits.
  • 0.3 seconds is the approximate wall-clock time for rendering and capturing each generated web-code state after a one-time 1-second browser launch.The rendering step can also be parallelized using more process threads.
  • 0.55s and 1.30s are gWorld’s per-state end-to-end latencies for the 8B and 32B models, versus 160s for VIMO.After removing VIMO’s reported 38s local model-execution time, its remaining 122s sequential overhead is still substantially slower than gWorld.
  • World models could support synthetic data scaling by simulating risky actions and expanding coverage of deep application states during mobile GUI-agent training.
  • The paper discusses its limitations and future directions in Appendix E.

Impact Statement

The work aims to advance mobile agents and world models while providing accessibility and open-research benefits. It also acknowledges dual-use risks associated with improved GUI automation.

  • Open-weight models requiring less compute than prior methods may democratize AI research and enhance digital accessibility for users with impairments.
  • Improved GUI automation carries dual-use risks, including automated fraud.

A. Further Details on our Method

The method converts offline mobile-agent trajectories into code-based, reasoning-augmented world-model training data. It uses image-to-code and look-ahead prompts to represent action-conditioned GUI transitions as renderable HTML.

  • World Model SFT Data Generation: Offline policy trajectories provide screenshot states, actions, rewards, and subsequent states for constructing the world-model dataset.
  • World Model SFT Data Generation: The pipeline iterates over each trajectory transition from S_t to S_t+1 before generating training examples.
  • World Model SFT Data Generation: Each next-state screenshot is synthetically relabeled as renderable code using the image-to-code prompt.
  • Look-Ahead Reasoning Synthesis: Free look-ahead reasoning is generated from the current screenshot and action, focusing on changes predictable before observing the ground-truth next state.
  • Mobile GUI Image to Code: The image-to-code prompt requires complete mobile-first HTML that preserves visible text, differentiable button functions, layout, hierarchy, styling, and interaction purposes.
  • Benchmark Construction: MWMBENCH includes four in-distribution and two out-of-distribution benchmarks for evaluating mobile GUI world models.

B.1. MWMBENCH-ANDROIDWORLD

MWMBENCH-ANDROIDWORLD is built from agent trajectories and balanced through visual deduplication, while KApps tests Korean-language and application-domain generalization. The appendix also specifies evaluation, training, and inference procedures.

  • MWMBENCH-ANDROIDWORLD: 686 transitions across 88 episodes and 18 applications comprise MWMBENCH-ANDROIDWORLD.The benchmark was collected by running M3A with Qwen3 VL 235B-A22B as the base policy on AndroidWorld.
  • Deduplication: Candidate duplicates are grouped by app-action pairs when both pre-action and post-action visual similarities exceed 0.997, then manually verified.
  • OOD Splits: AndroidWorld is dominated by productivity apps at 55.4%, whereas KApps distributes transitions more evenly across food and shopping, communication, and productivity domains.
  • Deduplication: 37% reduction removes redundant transitions, shrinking the dataset from 1,094 to 686 while retaining task-specific unique transitions.
  • OOD Splits: KApps is an entirely Korean-based OOD set, with Korean task goals and Korean text in 94.5% of transition screenshots.It contains popular Korean applications absent from the English-focused training data.
  • Training and Inference: Training freezes the vision encoder and updates only the language model and MLP projector because unfreezing did not meaningfully improve performance.
  • Evaluation: Instruction Accuracy evaluates action-conditioned next-state correctness, while Similarity measures cosine similarity between generated and ground-truth state embeddings.

C.1. Further Details on Scaling Analysis

The scaling analysis uses up to 240K training samples, while substantially more offline transitions remain available for future expansion.

  • 240K training samples are used currently, although up to 3.7 million samples are available for training.

C.2. Further Details on World Model-enhanced Policy Experiments

The policy experiments compare oracle selection, value estimation without a world model, and value estimation with world-model-predicted next states. The evaluation isolates action-selection ability while examining whether predicted future states improve action valuation.

  • The oracle policy receives the ground-truth action, current screenshot, goal, and action history to evaluate selection ability.
  • M3A generates K−1 alternatives, combines them with the ground-truth action, and selects among candidates.
  • The no-world-model value baseline scores each candidate from the current screenshot, action, reasoning, and goal, selecting the highest-confidence valid action.
  • The world-model variant predicts an after screenshot for each candidate and scores whether that predicted state shows progress toward the goal.
  • The evaluation uses coordinate-space actions and JSON judgments with validity labels and confidence scores.

D. Extended World Modeling Experiment Results

The extended results section organizes judge- and encoder-specific results, scaling analyses, inter-judge agreement, ranking consistency, and qualitative examples.

  • Results are organized across VLM-as-a-Judge choices, vision encoders, scaling analyses, judge agreement, model rankings, and qualitative examples.

E. Limitations and Future Work

The paper identifies limitations in data coverage, photorealistic rendering, and long-range temporal modeling, while outlining scaling, hybrid rendering, and memory augmentation as future directions.

  • Only 260K of 3.7 million available transitions are currently used, leaving data scaling as a future opportunity.
  • Web code has fundamental difficulty rendering photorealistic content such as complex natural imagery with high fidelity.
  • Future work may use hybrid techniques for visual failure cases and explicit working memory for long-term GUI dependencies.
  • The current single-frame Markov assumption does not explicitly model long-range dependencies such as shopping-basket state across pages.
Loading 2602.01576v2…