PaperlayerView exact revision on arXiv

Source-linked AI summary

Code2World: A GUI World Model via Renderable Code Generation

Yuhao Zheng, Li'an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, Kevin Qinghong Lin

arXiv:2602.09856v1cs.CVcs.AIcs.CLcs.HC

TL;DR

GUI world models need to predict action-conditioned visual states while retaining fine-grained structural control, a combination existing text- and pixel-based approaches do not simultaneously provide. Code2World generates renderable HTML, builds AndroidCode with visual-feedback revision, and trains with render-aware rewards; Code2World-8B achieves top-performing next UI prediction and improves downstream navigation, including +9.5% for Gemini-2.5-Flash on AndroidWorld.

  • Problem

    Existing GUI world models struggle to combine high visual fidelity with fine-grained structural controllability.

  • Method

    Code2World generates structured HTML for rendered next-state prediction, using AndroidCode and Render-Aware Reinforcement Learning to align visual fidelity with action consistency.

  • Results

    Code2World-8B achieves top-performing next UI prediction and boosts Gemini-2.5-Flash navigation success by +9.5% on AndroidWorld.

  • Takeaways & Limitations

    As a plug-and-play simulator, Code2World enhances downstream GUI-agent navigation and provides a sandbox for evaluating potentially irreversible actions before execution.

  • Takeaways & Limitations

    The paper acknowledges that hallucinated incorrect world-model predictions could introduce risks if agents rely on them for potentially irreversible actions.

Abstract

from arXiv · show

Autonomous GUI agents interact with environments by perceiving interfaces and executing actions. As a virtual sandbox, the GUI World model empowers agents with human-like foresight by enabling action-conditioned prediction. However, existing text- and pixel-based approaches struggle to simultaneously achieve high visual fidelity and fine-grained structural controllability. To this end, we propose Code2World, a vision-language coder that simulates the next visual state via renderable code generation. Specifically, to address the data scarcity problem, we construct AndroidCode by translating GUI trajectories into high-fidelity HTML and refining synthesized code through a visual-feedback revision mechanism, yielding a corpus of over 80K high-quality screen-action pairs. To adapt existing VLMs into code prediction, we first perform SFT as a cold start for format layout following, then further apply Render-Aware Reinforcement Learning which uses rendered outcome as the reward signal by enforcing visual semantic fidelity and action consistency. Extensive experiments demonstrate that Code2World-8B achieves the top-performing next UI prediction, rivaling the competitive GPT-5 and Gemini-3-Pro-Image. Notably, Code2World significantly enhances downstream navigation success rates in a flexible manner, boosting Gemini-2.5-Flash by +9.5% on AndroidWorld navigation. The code is available at https://github.com/AMAP-ML/Code2World.

1. Introduction

Code2World addresses the limits of text- and pixel-based GUI world models by generating renderable HTML that combines visual fidelity with structural controllability. It introduces AndroidCode, render-aware training, and evaluation showing strong next-UI prediction and downstream navigation gains.

  • Motivation: Existing GUI agents lack foresight into action consequences, making errors costly and especially risky for irreversible actions.A virtual sandbox can let agents simulate action-conditioned outcomes before execution.
  • Motivation: Text-based world models capture semantic intent but omit visual information, while pixel-based approaches struggle with structural controllability.The paper motivates a representation that preserves both visual and structural information.
  • Approach: Code2World predicts dynamic GUI transitions by synthesizing structured HTML and rendering it into the next visual state.Programmatic GUI code provides a native representation with deterministic rendering and structural control.
  • Approach: AndroidCode contains over 80K samples derived from AndroidControl trajectories, with visual-feedback revision used to improve synthesized HTML quality.The corpus grounds GUI states in high-fidelity structured code through automated synthesis and visual alignment checks.
  • Approach: Render-Aware Reinforcement Learning follows SFT and uses visual and action-consistency rewards to align generated code with rendered outcomes.The training strategy targets both visual reality and interaction dynamics.
  • Results: Code2World-8B achieves top-performing next UI prediction and boosts Gemini-2.5-Flash navigation success by +9.5% on AndroidWorld.The model also functions as a plug-and-play simulator for downstream GUI agents.

2. Related Work

Related work develops GUI agents through improved grounding, context management, reflection, and reinforcement learning, while GUI world models predict environment transitions in text or structural forms. Code2World instead optimizes a learnable simulator with render-aware reinforcement learning.

  • GUI Agents: Recent GUI-agent research improves long-horizon navigation through advanced grounding, efficient context management, and self-reflection mechanisms.These methods augment agents' deliberative capabilities.
  • GUI Agents: Reinforcement learning has been used to optimize GUI-agent decision logic through reasoning-enhanced fine-tuning and online policy optimization.These approaches optimize the agent policy in the actual environment.
  • Code2World: Code2World shifts reinforcement-learning optimization from the acting agent to the environment simulator through render-aware training.This distinguishes simulator optimization from RL methods that directly optimize agent policies.
  • GUI World Models: GUI world models internalize environment dynamics as virtual sandboxes, with existing approaches commonly predicting transitions through text or structural abstractions.Text-based methods include natural-language state differences and structural DOM updates.

3. Code2World

Code2World predicts next GUI states by generating structured HTML and deterministically rendering it, combining visual fidelity with structural controllability. It is trained on AndroidCode through supervised cold-start training followed by render-aware reinforcement learning with visual and action-consistency rewards.

  • 3. Code2World: Code2World predicts the next visual state by generating structured HTML and rendering it deterministically rather than predicting pixels directly.The model conditions generation on the current visual observation, executed action, and task goal.
  • 3.1. Training Data Synthesis: AndroidCode addresses scarce paired GUI trajectories and clean HTML by converting screenshots into constrained, self-contained HTML representations.The synthesis process standardizes dimensions, replaces unreliable assets with placeholders, and uses inline SVGs for icons.
  • 3.1. Training Data Synthesis: Visual-feedback revision renders synthesized code, compares it with the ground-truth screenshot using SigLIP, and selectively revises samples that fail alignment checks.Samples above the confidence threshold are retained immediately, while lower-scoring samples enter the revision process.
  • 3.2. Model Optimization: The training pipeline first uses SFT to learn valid HTML syntax and UI-layout mappings, then applies Render-Aware Reinforcement Learning to optimize rendered outcomes.The two stages use multimodal triplets containing the conditioning input, target HTML, and target screenshot.
  • 3.2. Model Optimization: The reinforcement-learning objective combines visual semantic fidelity and action consistency rewards, optimized with GRPO and a KL penalty relative to the SFT policy.The action reward checks whether the predicted state is a valid consequence of the executed action, helping penalize logically contradictory updates.
  • 4.2. Application for GUI Agent: Code2World enhances GUI-agent decisions through a “Propose, Simulate, Select” process that evaluates rendered futures before selecting an action.The scorer can use a VLM verifier to identify hallucinated or illogical plans and choose the candidate that best advances the task goal.

4. Evaluation and Application of Code2World

The evaluation tests whether Code2World produces visually faithful, logically valid next states and whether its simulations improve GUI-agent planning. It uses VLM-based assessment across functional and visual dimensions and integrates the model as a look-ahead simulator.

  • 4. Evaluation and Application of Code2World: The evaluation defines GUI world-model quality through precise next-UI prediction and effective enhancement of GUI-agent planning and decision-making.These requirements motivate separate simulation-quality and downstream-agent evaluations.
  • 4.1. Evaluation Protocol: The holistic protocol uses a unified VLM-as-a-Judge framework to assess GUI simulation across functional logic and visual quality.The design reflects the need to satisfy interaction rules and precise structural rendering simultaneously.
  • 4.1. Evaluation Protocol: Functional Logic measures whether generated states follow interaction rules and whether the executed action can be inferred from the visual state difference.The protocol uses Action Adherence and Action Identifiability to test logical validity and causal clarity.
  • 4.1. Evaluation Protocol: Visual Quality evaluates rendered-interface fidelity using semantic similarity measures plus specialized checks for element alignment and layout integrity.These metrics examine both fine-grained UI positioning and preservation of geometric containment.
  • 4.2. Application for GUI Agent: As a plug-and-play look-ahead simulator, Code2World has an agent propose candidate actions, simulate each resulting GUI state, and select the most promising action.A VLM-based scorer can judge which predicted state best advances the task goal and filter hallucinated or illogical plans.

5. Experiments

Experiments evaluate Code2World’s next-UI prediction, generalization, and downstream navigation enhancement across in-domain, out-of-distribution, offline, online, and ablation settings. Code2World shows strong visual and dynamic fidelity, improves agent performance, and benefits from combining semantic and action rewards.

  • 5.1. World Model Ability (RQ1): Code2World outperforms open-source baselines exceeding 10× its parameters across dynamic logic and visual quality, while rivaling GPT-5 and Gemini-3-Pro-Image.Large generalist VLMs lack UI-to-code alignment, whereas image-editing models struggle with interface dynamics.
  • 5.1. World Model Ability (RQ1): 92.73 Sad and 78.22 Sid on GUI Odyssey show robust dynamic-logic generalization despite visual-similarity declines in unseen cross-app environments.The benchmark tests robustness across diverse UI styles, domains, and device shifts.
  • 5.2. GUI Agent Enhancement (RQ2): Code2World accurately predicts both logical transitions and fine-grained visual details, including a temporal change from 5:46 to 5:47.Baselines either reproduce the initial state, produce chaotic layouts, or preserve the original geometry.
  • 5.2. GUI Agent Enhancement (RQ2): 5.23-point Grounding accuracy improvement for Qwen2.5-VL-7B demonstrates offline gains, while Mobile-Agent-v3 achieves the best Type and Grounding accuracy with Code2World.The simulator is applied to both a specialized GUI agent and a general MLLM.
  • 5.2. GUI Agent Enhancement (RQ2): Code2World consistently improves task success rates across evaluated models on AndroidWorld, enabling foresight for selecting advantageous actions during long-horizon interaction.AndroidWorld evaluates continuous action sequences across 116 tasks and 20 mobile applications.
  • 5.3. Ablation Study (RQ3): Combining semantic and action rewards achieves optimal performance because Rsem improves rendering but can leave functional logic stagnant, whereas Ract primarily improves dynamic logic.SFT provides the HTML syntax and layout foundation for both functional and visual improvements.
  • 5.3. Ablation Study (RQ3): 50.9% Success Rate for full Code2World improves Gemini-2.5-Flash by +9.5% over baseline, following gains from SFT, Rsem, and Ract.The baseline is 41.4%; the successive variants reach 47.5%, 49.2%, and 50.1%.

6. Conclusion

Code2World is a code-native GUI world model that predicts next UI states through renderable HTML, combining visual fidelity with structural controllability. It achieves strong next-UI prediction and improves downstream GUI-agent navigation.

  • Conclusion: Code2World shifts next UI prediction from raw pixel estimation to renderable HTML code generation.Its native representation combines high-fidelity visualization with fine-grained structural controllability.
  • Conclusion: +9.5% improvement boosts Gemini-2.5-Flash on AndroidWorld navigation.Code2World-8B functions as a plug-and-play simulator for downstream agents.

Impact Statement

Code2World is presented as a tool for digital inclusivity and safer GUI-agent development. Its sandbox can support evaluating irreversible actions before execution, but incorrect safety cues and misuse remain risks.

  • Impact Statement: Code2World may assist users with disabilities while reducing repetitive digital labor.The paper identifies enhanced digital inclusivity as its primary societal benefit.
  • Impact Statement: Its world-model sandbox allows agents to simulate potentially irreversible actions without executing them in the real world.Examples include financial transactions and data deletion, reducing risks from on-policy trial-and-error learning.
  • Impact Statement: Hallucinated safety cues could mislead agents into harmful actions, while automation could be misused for cyber-attacks or navigation spamming.The paper calls for robust verification mechanisms and responsible deployment.

A.1. Experimental Setup

Experiments run on a cluster with eight NVIDIA H20 GPUs, using Qwen3-VL-8B-Instruct as the Code2World backbone and standardized evaluation configurations.

  • A.1. Experimental Setup: Experiments use a computing cluster equipped with 8 NVIDIA H20 GPUs with 96GB of memory.The hardware configuration is reported for all experiments.
  • A.1. Experimental Setup: Qwen3-VL-8B-Instruct serves as the backbone, with SFT using LLaMA-Factory and RL using EasyR1.These frameworks support the two training stages.
  • A.1. Experimental Setup: VLM-as-a-Judge evaluations use Qwen3-VL-8B-Instruct at temperature 0.1 with a maximum generation limit of 1,024 tokens.SigLIP visual feature scores use the google/siglip-so400m-patch14-384 checkpoint.

A.2. Training Hyperparams

Training partitions AndroidCode between SFT and RL, then applies full-parameter SFT followed by GRPO-based render-aware reinforcement learning with long-context generation settings.

  • A.2. Training Hyperparams: 70% of AndroidCode supports Stage 1 SFT, while 30% supports Stage 2 RL.The subsets are disjoint and correspond to foundational policy learning and subsequent reward alignment.
  • A.2. Training Hyperparams: SFT fine-tunes the language model for 2 epochs while freezing the vision encoder and multimodal projector.DeepSpeed ZeRO-2 and Flash Attention improve memory efficiency and training throughput.
  • A.2. Training Hyperparams: GRPO-based RL generates G = 4 candidates per prompt to align the SFT model with render-aware rewards.The stage uses sampling temperature 1.0, rollout batch size 16, and a learning rate of 1×10^-6.

B. Training Data Construction

AndroidCode is synthesized from GUI screenshots into structured HTML, then iteratively refined through rendered visual feedback to produce a high-fidelity training corpus.

  • Constrained Initial Synthesis: GPT-5 converts each GUI screenshot into structured HTML under constrained initial-synthesis rules.The pipeline renders the generated code and compares it with the source screenshot.
  • Revision with Visual Feedback: Samples that remain below the quality threshold after Nmax revision attempts are discarded.This filtering step is intended to preserve corpus quality.
  • Revision with Visual Feedback: Visual-feedback revision repeatedly compares target and rendered screenshots, revises the HTML, and recomputes alignment until the threshold is met or attempts are exhausted.The revision loop uses the current HTML and rendered image as inputs to the multimodal coder.
  • Visual Prompting: Visual prompting marks target elements and motion so the model can associate static frames with the intended operation.The strategy grounds spatial attention on target elements and visualizes dynamic operation flow.
  • Instruction Expansion: A deterministic expansion engine turns abstract action primitives and coordinates into descriptive natural-language instructions.Scrolling and text entry are rewritten in terms of their expected effects and content injection.
  • Prompt Integration: The augmented visual hints and expanded descriptions are combined in a standardized prompt template for interface-dynamics simulation.The template is designed to keep instruction context consistent during prediction.

C. Evaluation Metrics of Next UI Prediction

The evaluation protocol measures GUI world models along functional logic and visual quality because generic image similarity does not capture action validity or structural layout.

  • Motivation: Generic image metrics such as SSIM and LPIPS miss deterministic interaction rules and precise DOM-aligned rendering.GUI evaluation must distinguish pixel-level texture from whether actions produce valid state changes and aligned elements.
  • Protocol Overview: The protocol uses four specialized metrics across Functional Logic and Visual Quality within a unified VLM-as-a-Judge framework.This provides a more granular comparison tailored to GUI world models.
  • Functional Logic: Action Adherence (Sad) checks whether the predicted next state is a logically valid consequence of the executed action.It penalizes contradictions such as clicking Back while remaining on the same page.
  • Functional Logic: Action Identifiability (Sid) measures whether the action causing a visual state change can be inferred from the transition.An inverse-dynamics VLM predicts the action type from the before-and-after visuals, with performance calculated as classification accuracy.
  • Visual Quality: Element Alignment (Sele) evaluates whether key interactive components occupy the correct relative coordinates in the generated screen.Semantic placeholders are tolerated when they match the ground-truth region spatially.
  • Visual Quality: Layout Integrity (Slay) penalizes global structural failures such as CSS collapse, overlapping containers, and misalignment.The visual judge compares generated output with the ground truth under specified criteria.

D. Prompt Template

The prompt template constrains HTML output, encodes action cues, and specifies VLM-based visual, action-consistency, and inverse-dynamics evaluation procedures.

  • 4. OUTPUT FORMAT / 6. OUTPUT FORMAT (Strict): The generation prompt requires raw HTML only, beginning with <!DOCTYPE html>, without Markdown, explanations, or conversational text.It also requires a single render-target container with fixed viewport dimensions.
  • Instruction-following System Prompt: Red circles encode click or long-press targets, while red arrows encode scroll or swipe direction and are excluded from the rendered HTML.An upward finger-motion arrow means content scrolls down.
  • Instruction-following System Prompt: The UI-generation prompt wraps all visible content in a fixed-size, relatively positioned render-target with hidden overflow.The body is not centered, and the container begins at coordinate (0,0).
  • D.3.2. ACTION CONSISTENCY System Prompt: Action Consistency judges whether the predicted transition directly reflects the user action, including clicks, text input, scrolling, and navigation.The inputs are the current screenshot, action description and data, and rendered predicted next state.
  • D.3.2. ACTION CONSISTENCY System Prompt: Placeholder equivalence prevents penalizing gray image substitutes when their position, size, and semantic label match the reference region.The same principle is applied when assessing visual similarity and layout fidelity.
  • D.4.2. ACTION IDENTIFIABILITY METRICS: Inverse Dynamics assigns one action category from the visual transition, while layout-fidelity evaluation compares reference and candidate images on element position, content, and structure.The inverse-dynamics categories include click, long press, scroll, input text, app opening, home, back, wait, and none.

E.1.1. CODE2WORLD

The examples show Code2World using predicted outcomes to avoid redundant actions, discover more direct routes, and support action selection across GUI tasks.

  • Examples: Code2World’s examples cover search, deletion, menu opening, text entry, traveler selection, swiping, and app launching.The examples present origin states, Code2World next states, and the associated actions.
  • AndroidWorld: In MarkorCreateNoteFromClipboard, previewing the next state prevents a repeated Save action after the note was already saved.The agent instead selects navigate back, avoiding an unnecessary loop.
  • AndroidWorld: In SystemWifiTurnOffVerify, Code2World favors opening Settings directly over scrolling to locate the settings application.The predicted post-launch interface makes the shorter route more apparent to the agent.
  • AndroidControl-High: In AndroidControl-High examples, the pipeline evaluates alternative actions and identifies outcomes that would produce no change.These include scrolling to the bottom or clicking inactive elements.
Loading 2602.09856v1…