PaperlayerView exact revision on arXiv

Source-linked AI summary

ViMo: A Generative Visual GUI World Model for App Agents

Dezhao Luo, Bohan Tang, Kang Li, Georgios Papoudakis, Jifei Song, Shaogang Gong, Jianye Hao, Jun Wang, Kun Shao

arXiv:2504.13936v2cs.HCcs.LGcs.SYeess.SY

TL;DR

App agents face long-horizon decisions, and existing world models often describe future GUIs in text without essential visual detail. ViMo predicts future GUI images by separating graphic generation from text generation using symbolic placeholders. Experiments report gains in world-model evaluation, App-agent action prediction, and online navigation.

  • Problem

    Existing world models typically describe future GUIs in language, which can omit visual details such as GUI-element locations and colours, while pixel-based image generation can distort small text.

    Source: §1 · pp. 1–2

  • Method

    ViMo replaces GUI text with symbolic placeholders, predicts the next GUI’s STR from the current STR and user action, then uses an LLM-based GUI-text predictor to generate text.

    Source: §1 · p. 2

  • Results

    ViMo achieved average relative improvements of 29.14% on automatic metrics and 182.74% in user studies over existing world models, plus a 14.07% relative gain in step-wise action prediction accuracy.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    ViMo generated visually plausible and functionally effective GUIs and improved App agents’ action prediction and online navigation performance.

    Source: §1 · p. 3, §5 · p. 10

    Text symbols that fail to appear as black-bordered, white-filled rectangular placeholders can become unrecognisable, and improving their representation remains future work.

    Source: D Limitation · p. 18

Abstract

from arXiv · show

App agents, which autonomously operate mobile Apps through Graphical User Interfaces (GUIs), have gained significant interest in real-world applications. Yet, they often struggle with long-horizon planning, failing to find the optimal actions for complex tasks with longer steps. To address this, world models are used to predict the next GUI observation based on user actions, enabling more effective agent planning. However, existing world models primarily focus on generating only textual descriptions, lacking essential visual details. To fill this gap, we propose ViMo, the first visual world model designed to generate future App observations as images. For the challenge of generating text in image patches, where even minor pixel errors can distort readability, we decompose GUI generation into graphic and text content generation. We propose a novel data representation, the Symbolic Text Representation~(STR) to overlay text content with symbolic placeholders while preserving graphics. With this design, ViMo employs a STR Predictor to predict future GUIs' graphics and a GUI-text Predictor for generating the corresponding text. Moreover, we deploy ViMo to enhance agent-focused tasks by predicting the outcome of different action options. Experiments show ViMo's ability to generate visually plausible and functionally effective GUIs that enable App agents to make more informed decisions.

1 Introduction

App agents struggle with long-horizon decisions, while existing world models often describe future GUIs textually and omit visual details. ViMo predicts future GUI images by separating graphic generation from text generation, and experiments report gains in world-model evaluation and App-agent tasks.

1 Introduction

  • Long-horizon App-agent decisions motivate visual world models because language-based future descriptions can omit GUI element locations and colours.Image-based generation also struggles with distortions in small text, where pixel accuracy matters.Source: §1 · pp. 1–2
  • ViMo uses Symbolic Text Representation (STR) placeholders to separate GUI graphics from text, with a STR Predictor generating layouts and a GUI-text Predictor generating corresponding text.This reframes text generation as predicting text-symbol locations, then generating text for each symbol.Source: §1 · p. 2
  • The evaluation covered generated-GUI quality, App-agent decision-making, online navigation, and zero-shot generalisation to unseen Apps.Generated GUIs were assessed for visual similarity, instructional accuracy, and action readiness using automatic metrics and user studies.Source: §1 · p. 2
  • ViMo achieved average relative improvements of 29.14% on automatic metrics and 182.74% in user studies over existing world models.The paper also reports a 14.07% relative gain in step-wise action prediction accuracy for App agents.Source: §1 · pp. 2–3
  • Online navigation task completion increased from 33.19% to 40.95%, a 7.76% improvement.The evaluation used real-time interactions in an online navigation setup.Source: §1 · pp. 2–3

2 Related Works

Related work describes App agents’ difficulty with long-horizon tasks and world models’ use in predicting future observations. Prior GUI-generation approaches work in pixel space, while ViMo separates pixel-rendered graphics from language-generated text.

2.1 App Agent

  • App agents struggle with long-horizon tasks; real-world emulators can simulate GUI changes but incur setup costs and safety risks, motivating world models as an alternative.App agents use language-based or multimodal methods to choose actions from GUI observations and goals.Source: §2.1 · p. 3

2.2 World Model

  • Website world models predict next observations as text, but mobile settings typically expose only GUIs, and text-only descriptions can omit precise visual details.The related work identifies a need for high-fidelity future GUI images.Source: §2.2 · p. 3

2.3 GUI Generation

  • Pixel-based GUI generation struggles to render small text accurately; ViMo instead renders graphics as pixels and generates text as language tokens.Prior work includes layout generation, scene-text generation, UI-Diffuser, and image-and-text-conditioned GUI generation.Source: §2.3 · p. 3

3 Method

Section 3 defines GUI world modeling as predicting a next screen from the current GUI and an action, then presents ViMo’s graphics/text generation approach and its use for agent action selection.

3.1 Problem Setup

  • A GUI world model predicts the next GUI image from the current image and a user action.The setup formalizes this as applying a world model to the current GUI and action.Source: §3.1 · p. 4

3.2 ViMo: Generative Visual GUI World Model

  • ViMo separates GUI graphics generation from text generation: it replaces text with symbols, predicts the next symbolic GUI, then generates and overlays text.The STR predictor uses pixel-based diffusion; the GUI-text predictor uses an LLM to generate text for the symbols.Source: §3.2 · p. 4

3.2.1 STR: Symbolic Text Representation

  • STR uses OCR to detect and mask GUI text while preserving static text, turning dynamic text generation into prediction of symbols’ locations and sizes.The authors report that LLM prediction of static text with complex spatial patterns is challenging.Source: §3.2.1 · pp. 4–5

3.2.2 STR Predictor

  • The STR predictor fine-tunes a pretrained Stable Diffusion model to predict the next STR, conditioned on the current STR and user action.Its denoising network is conditioned on the timestep, action, noisy latent, and current STR image condition.Source: §3.2.2 · p. 5

3.2.3 GUI-Text Predictor

  • The GUI-text predictor locates colored, bounded text symbols, assigns IDs, uses an LLM to infer context-appropriate text, then overlays it with styling based on size and surrounding colors.Text is positioned according to symbol coordinates to reconstruct the predicted GUI image.Source: §3.2.3 · p. 5

3.3 ViMo Enhanced App Agent

  • ViMo supports agent action selection by predicting each candidate’s next GUI, which an LLM-based selector evaluates before comparing the two highest-scoring actions.In over 70% of tasks, the top two action scores differed by 0.1 or less, motivating explicit comparison.Source: §3.3 · pp. 5–6

4 Experiments

Across App-agent evaluations, ViMo improved step accuracy, while GUI-quality evaluations favored it over comparison methods. Experiments also examined real-world transfer and ablations of text handling, action conditioning, and rollout length.

4.1 Dataset Summarisation

  • The STR dataset draws on Android Control and AITW, covering 19 Apps; experiments used a partial evaluation split of 57 episodes across those Apps.Source: §4.1 · pp. 6–7

4.2 World Model Ability

  • ViMo achieved the highest harmonic average across the three automatic GUI metrics, with a 29.14% average relative improvement; user studies also ranked it best.Source: §4.2 · p. 7
  • Qualitative examples contrast graphic changes with text generation: HTML-vision is more flexible in text but lacks concrete details, while IP2P* has plausible graphics but inflexible text.Source: §4.2 · pp. 7–8, Figure 3 · p. 7

4.3 World Model Enhanced App Agent

4.4 Real-world Applications

  • In real-world evaluations, ViMo achieved 47.6% in zero-shot evaluation on unseen Apps and improved AndroidWorld task accuracy by 7.76% over baseline; deployment requires a 16 GB GPU.Source: §4.4 · pp. 8–9, Table 4 · p. 8, Table 5 · p. 8

4.5 Ablation Study

  • Ablations found that preserving static text and conditioning on action instructions each contributed to performance improvements across the two tested App agents.Source: §4.5 · p. 9, Table 6 · p. 9
  • Two rollout iterations yielded the highest accuracy but increased computational cost; iteration 3 declined slightly, so one step was chosen as a practical performance-efficiency trade-off.Source: §4.5 · p. 9, Table 7 · p. 9

5 Conclusion

The conclusion presents ViMo as a visual GUI world model that separates graphic generation from text handling through STR. It reports improved agent action prediction alongside generalization to unseen Apps and online navigation.

5 Conclusion

  • ViMo predicts future App observations visually, offering a more realistic and concrete approach than language-based world models.Source: §5 · p. 9
  • STR overlays text with placeholders for location prediction and delegates text-content generation to an LLM.Source: §5 · pp. 9–10
  • ViMo generated visually plausible and functionally effective GUIs, with a 14.07% relative gain in step-wise action prediction accuracy.Source: §5 · p. 10
  • Real-world experiments demonstrated generalization to unseen Apps and robust performance in online navigation with real-time environment interaction.Source: §5 · p. 10

A Experimental Details

The GUI-text predictor detects text symbols in the predicted STR, uses an LLM to recover their text, and renders it within each symbol. Action selection scores candidate actions by validity and confidence, then uses an LLM to choose between the top two.

A.1 GUI-text Predictor

  • Black-pixel borders, rectangularity above 0.8, and more than 50% white pixels identify text symbols and assign each a unique ID.Black pixels use BGR values [0,0,0] to, and white pixels use [200,200,200] to [255,255,255].Source: §A.1 · p. 13
  • The LLM identifies unchanged symbols and reuses their previous GUI pixels, then infers each remaining symbol’s semantic role and exact text.The predictor overlays each symbol’s ID at its location in the STR before these steps.Source: §A.1 · p. 13
  • For rendering, the predictor estimates the background color at each symbol’s edge, selects black or white text for contrast, and fits the largest text size within the symbol.The background estimate is the average color along the symbol’s edge.Source: §A.1 · p. 13

A.2 Action Selection

  • Actions receive positive confidence scores when judged valid and negative confidence scores when judged invalid; an LLM then selects between the two highest-scoring actions.In over 70% of tasks, the top two scores differ by 0.1 or less.Source: §A.2 · p. 13

A.3 Data Collection

The STR dataset combines Android Control and AITW data, with additional manual collection for some apps, and covers 19 unique apps. Action commands were converted to instructions for AITW, and PaddleOCR was used for STR generation.

A.3 Data Collection

  • Combining Android Control and AITW data yielded 19 unique apps; additional samples were manually collected for the 13 Android Control apps with more than 50 samples.The Android Control filtering began with 15,274 episodes, of which 5,697 included “open_app”; these identified 758 apps, only 13 of which had more than 50 samples.Source: §A.3 · p. 14
  • For AITW, action commands were converted into action instructions, and PaddleOCR was used for STR generation.The dataset was split into Train, Validation, and Test sets.Source: §A.3 · p. 14

A.4 Evaluation

The evaluation assesses generated GUI quality through automatic metrics and user studies, and also describes agent-focused app categories and iterative prediction. It reports the evaluation scales and defines trajectory and randomness-table metrics, but these passages do not provide result values.

A.4 Evaluation

  • Automatic metrics assess instructional accuracy and action readiness, counting a generation successful when “success” appears for instructional accuracy and “yes” appears under ready for action.The user study included 80 generated samples (20 from each of four world models) and 70 participants; graphic-consistency scores were rated 1–5 and normalized to [0,1].Source: §A.4 · pp. 14–15
  • Table 10 defines T+L as the accuracy of the whole trajectory with length L.Source: Table 10 · p. 15
  • Table 11 compares three runs on a sampled test split with the full test split; it defines sh as the harmonic score and STD as the standard deviation across runs.The table uses sgc, sia, and sar, the same metrics as Table 1.Source: Table 11 · p. 15
  • The agent-focused evaluation groups apps as Leisure, Work, or System and tests repeated ViMo calls, using only the final output to guide candidate-action selection.Iteration number is defined as the number of times ViMo is called.Source: §A.4 · p. 15

B Additional Experimental Results

ViMo consistently outperforms existing world models across several App agents, and its trajectory accuracy remains higher as iterative predictions degrade. Additional experiments examine accumulated error, generation choices, generalization to novel GUI-action combinations, randomness, and inference-time/step-accuracy comparisons.

B Additional Experimental Results

  • ViMo consistently outperforms existing world models under M3A, T3A, APP-Agent, and Mobile-Agent-V2.Table 3 compares methods under M3A, and Table 9 reports additional agent comparisons.Source: B Additional Experimental Results · p. 15
  • ViMo’s accumulated generation error grows with iteration count, motivating comparisons against other world models over longer rollouts.Table 10 presents the trajectory-synthesis evaluation, where T+L denotes whole-trajectory accuracy at length L.Source: B Additional Experimental Results · p. 15
  • The generalization study tests novel combinations of a current GUI and user action, including actions outside an app’s usual context.Figure 5 depicts GUI generation conditioned on such a novel combination.Source: Figure 5 · p. 16
  • Although performance decreases with more iterations across all tested world models, ViMo significantly outperforms visual- and language-based alternatives in trajectory evaluation.The evaluation uses generated GUIs as agent observations and assesses success across L-step trajectories; simulated actions are judged against ground-truth actions.Source: B Additional Experimental Results · pp. 15–16
  • Across three runs, the randomness study found that random factors did not significantly affect ViMo’s performance or consistency; full-test results showed only minor differences from the randomly selected evaluation subset.Source: B Additional Experimental Results · p. 16
  • Table 12 compares models on inference time and step accuracy.Source: Table 12 · p. 17
  • Figure 6 shows false cases where text symbols are represented incorrectly and become unrecognizable as indicators of text location.Source: Figure 6 · p. 17
  • The qualitative ablation contrasts LLM-generated static text with preserving original text pixels, and shows action instructions align STR predictions with user intent better than action commands.Figure 4 presents the static-text alternatives and the two STR input formats.Source: Figure 4 · p. 16, B Additional Experimental Results · p. 17

C Practical Deployment

ViMo’s deployment profile reports its hardware and inference costs alongside an accuracy gain, and the accompanying figure shows current and generated GUIs for user actions.

C Practical Deployment

  • On a V100 GPU, STR image generation takes about 8 seconds and GUI-text prediction about 30 seconds; each request predicts GUIs for three actions.The minimum hardware requirement is a GPU with 16 GB of memory; total inference time per request is about 2 minutes, including model loading and communication overhead.Source: C Practical Deployment · p. 17
  • With an additional 2 minutes of inference time, ViMo achieves a 6.07% accuracy improvement.The passage describes Table 12 as comparing inference time and step accuracy with other world models.Source: C Practical Deployment · p. 17
  • Figure 7 places the action above the current GUI on the left and the generated GUI on the right.Source: Figure 7 · p. 18

D Limitation

ViMo’s failure cases include text symbols that do not match the intended rectangle-shaped placeholder, leaving them unrecognisable; improving the representation is future work.

D Limitation

  • Fig. 6 shows failures when text symbols are not represented as rectangle-shaped placeholders with a black border and white fill.Source: D Limitation · p. 18
  • When those placeholder features are absent, the text symbols become unrecognisable as text symbols.Source: D Limitation · p. 18
  • Improving the representation of text symbols remains a potential direction for future work.Source: D Limitation · p. 18

E Additional Visualisation

The appendix presents GUI-generation visualizations and safeguards, then documents prompts for symbol handling, action selection and evaluation, and a user study. The examples show current/action-conditioned next-GUI generation, while the prompts specify procedures for downstream tasks.

E Additional Visualisation

F Social Impact and Safeguards

  • The authors identify possible misuse, misinformation, and user manipulation risks, and require an additional user commitment to access the forthcoming checkpoint.They hope this requirement will alleviate potential misuse while benefiting further research.Source: F Social Impact and Safeguards · p. 19

G.1 Prompt to decide the text symbols to remain unchanged after the action

  • The unchanged-text prompt asks which masked text layouts keep the same position and context across current and next screenshots, excluding text changed by the action.Source: §G.1 · p. 20

G.2 Prompt to determine the semantic role of each text symbol

  • The semantic-role prompt groups masked boxes into functional windows and assigns each a role using its location, size, and visual context.Source: §G.2 · p. 21

G.3 Prompt to predict the exact text content for each symbol

  • The text-prediction prompt uses the current screenshot, next-screen layout, user instruction, and mask semantics to predict text for each masked area.Source: §G.3 · p. 21

G.4 Prompt to evaluate actions with a confidence score

  • The action-evaluation prompt judges whether a selected action advances the user’s goal, considering that the generated after-screen may not faithfully represent the real UI.Source: §G.4 · p. 22

G.5 Prompt to select the optimal actions among two highest-scoring actions

  • The action-selection prompt compares two candidate actions, using their rationales, task history, and hints such as whether scrolling suits the goal.Source: §G.5 · p. 22

G.6 Prompt to convert action commands into action instructions

  • The action-conversion prompt maps current and next screenshots, intent, and touch/lift coordinates to a plain-language action, returning empty output if they are identical or not one step apart.Source: §G.6 · p. 23

Related papers

Loading 2504.13936v2…