How Mobile World Model Guides GUI Agents?

1 Introduction

• Mobile agents lack foresight to anticipate action consequences, distinguish divergent outcomes of similar UI elements, and avoid irreversible or high-risk states.Predictions can serve as a simulator of interface transitions, but discontinuous screens, changing action spaces, and mixed visual, textual, and spatial signals make useful prediction formats unclear.Source: §1 · pp. 1–2

• RQ1: What Should Mobile World Models Predict?

• Code2Image offers strong in-distribution reconstruction, while text-based predictions are more robust online and out of distribution; diffusion images add visual context but have text-rendering errors and high inference cost.The study argues that format choice should reflect generation reliability, interpretability, deployment distribution, and efficiency, not visual fidelity alone.Source: • RQ3: Can GUI Agents Learn from… · p. 3

• RQ3: Can GUI Agents Learn from Mobile World Model Imagination?

• Mobile agents are often overconfident and low-entropy, limiting posterior critics’ candidate diversity; prior perception can expose action consequences before commitment, with benefits constrained by confidence, diversity, and inference budget.Test-time scaling can partly mitigate limited sampling when world-model feedback distinguishes candidate actions.Source: • RQ3: Can GUI Agents Learn from… · p. 3
• Training on imagined trajectories degrades AndroidControl performance but improves AndroidWorld success, suggesting the trajectories offer transferable interaction experience without preserving the source distribution.Their value depends on whether generated behaviors are useful for generalization.Source: • RQ3: Can GUI Agents Learn from… · p. 3
• The paper concludes that world models should be integrated into agent systems rather than treated as standalone next-state predictors.Visual fidelity, agent confidence, candidate diversity, inference budget, and trajectory distribution all constrain their usefulness.Source: • RQ3: Can GUI Agents Learn from… · p. 4

2.1 Prediction Formats of Mobile World Models

• Four formats predict the next state as a delta description, full textual description, renderable interface code, or synthesized screenshot.Delta text reduces redundancy when changes are small; full text handles larger changes but can obscure task-relevant details, while code preserves layout and diffusion adds screenshot context with text-rendering and latency costs.Source: §2.1 · p. 4

2.2 Transition-Level Data Construction

• GUI trajectories are converted into state-action-next-state triples, then deduplicated by similar start and next states plus matching action type.The similarity criterion uses DINOv2; deduplication aims to remove visual redundancy while preserving interaction semantics.Source: §2.2 · p. 4
• Render-code samples are discarded when their rendered screenshot falls below a preset similarity threshold against the ground-truth next state.The check uses the same resolution; HTML rendering may fail to reproduce photographic content, native widgets, or dynamic visual effects.Source: §2.2 · pp. 4–5
• A verifier keeps a transition only when its annotated action appears among five plausible actions generated from the current and next screenshots.This filter targets inconsistencies among action descriptions, coordinates, and observed state changes.Source: §2.2 · p. 5
• The corpus also removes homogeneous blank search pages, which can bias diffusion generation and HTML rendering toward black-and-white layouts.The paper says this removal improves diversity and reliability of the final training corpus.Source: §2.2 · p. 5

2.3 Data Statistics

• The corpus combines AITW, AMEX, AITZ, AndroidControl, and GUI-Odyssey data, but GUI-Odyssey is excluded from training and used as an OOD benchmark.The sources supply different annotation granularity, interface distributions, and trajectory horizons; the largest page-noise reduction occurs for AndroidControl and GUI-Odyssey.Source: §2.3 · p. 5

3.1 How Well Do Mobile World Models Predict? Next-state Reconstruction Quality

• MobileWorldModel-8B scores 8.07 on full descriptions and 12.53 on delta descriptions, and visual hints raise these to 8.37 and 12.74, respectively.Generic Qwen3-VL models score 5.83–5.97 on full descriptions and 9.25–9.87 on delta descriptions.Source: §3.1 · p. 6, Table 2 · p. 6
• MobileWorldModel-8B leads the image-based code-generation metrics, while diffusion models score far lower on text-sensitive than visual-similarity measures.Diffusion scores are 32.17 and 34.13 on text-sensitive metrics versus 82.19 and 66.94 on visual-similarity metrics.Source: §3.1 · p. 6, Table 3 · p. 6

3.2 How Do Modalities Affect Action Selection? Offline Step-Level Evaluation

• Code2Image raises Gemini 3-Flash from 46.39 to 71.63 overall Step Acc., matching PASS@3 and exceeding its base model by 25.24.Text-based judging can repair click coordinates while worsening action-type prediction; Code2Image improves both metrics for Gemini 3-Flash.Source: §3.2 · pp. 6–7
• Diffusion-image guidance lowers Qwen3-VL-8B overall Step Acc. from 66.61 to 62.05, while delta-text judging lowers Gemini 3-Flash overall Type Acc. from 77.21 to 64.15.The passage attributes these weaknesses to image text recognition and notes the 40-step diffusion denoising cost.Source: §3.2 · p. 7

3.3 How Do Modalities Affect Task Completion? Online End-to-End Evaluation

• Text feedback improves AndroidWorld success for Gemini 3-Flash and GPT-5.4, while Code2Image can reduce Qwen3-VL-32B performance and MA3.5 performance regresses with every tested format.The reported delta-text SRs are 50.6 to 66.6 for Gemini 3-Flash and 56.0 to 63.8 for GPT-5.4; with Qwen3-VL-8B, MA3.5 falls from 57.2 to 55.2, 51.9, or 51.1.Source: §3.3 · p. 8, RQ1: What should mobile world models… · p. 9
• Code2Image can mislead Qwen3-VL-32B on text-input and textedit tasks, where rendered futures miss fine-grained UI transitions.The passage connects guidance effectiveness to both world-model fidelity and the agent's ability to propose diverse executable actions.Source: §3.3 · p. 8

RQ1: What should mobile world models predict?

• Text predictions are more robust for online OOD execution, Code2Image favors in-distribution visual guidance, and diffusion images remain costly and text-rendering-prone.The section's modality comparison distinguishes reconstruction fidelity from usefulness in downstream decisions and task completion.Source: RQ1: What should mobile world models… · p. 9

4.1 Why Does Guidance Fail? Decision Confidence Analysis

• Qwen3-VL-8B mean entropy is 0.15 on AITZ and 0.08 on AndroidControl, versus 0.26 on MMBench and 0.34 on APIBench.Higher-entropy GUI steps are generally less accurate, and high-entropy steps have more erroneous action corrections.Source: §4.1 · p. 9, Figure 7 · p. 9

4.2 Which Interaction Mode Works Best? Posterior, Reflection, and Prior

• Full-text entropy-gated self-reflection reaches 61.26 overall Step Acc. versus 56.47 for Qwen-judged posterior selection, at about one-fifth the inference cost.Ungated full-text self-reflection corrects only 14.66% of actions; posterior selection remains bounded by candidate quality and diversity.Source: §4.2 · p. 10
• Code2Image prior perception reaches 70.39 overall Step Acc., surpassing PASS@9 at 66.04, while requiring evaluation of an average 35.18 elements per screenshot.The passage reports especially large gains for Click and Input, alongside element-list and efficiency constraints.Source: §4.2 · p. 10, Table 7 · p. 10

4.3 Does More Sampling Help? Test-Time Scaling Analysis

• Code2Image gains track Pass@k as sampling grows and remain unsaturated at k = 16, whereas text feedback becomes harder to distinguish across larger candidate pools.More sampling can partly overcome low-entropy proposal limits, but does not produce the same trend for text-based feedback.Source: §4.3 · pp. 10–11, Figure 8 · p. 11

RQ2: How should mobile world models assist agents at test time?

• Mobile agents' low-entropy action distributions limit posterior critics, making world models more effective as prior perception than universal post-hoc verifiers.Test-time scaling gains remain bounded by initial confidence and inference budgets.Source: RQ2: How should mobile world models… · p. 11

5 Can GUI Agents Learn from Mobile World Model Imagination?

• Offline performance falls from 40.2 to 28.7 after fine-tuning, alongside click accuracy falling from 39.4 to 20.7 and predicted coordinates concentrating near x=499.The authors attribute the decline to a distributional gap: rendered pages simplify layouts and center interactive elements relative to real Android screenshots.Source: §5 · p. 11, Figure 10 · p. 12
• World-model trajectories raise AndroidWorld online success from 40.1 to 45.0, but prolonged training lowers it to 41.6.The authors hypothesize that imaginary interaction improves planning; they attribute the later decline likely to overfitting to trajectories collected once by the base model.Source: §5 · p. 11, Figure 9 · p. 12

RQ3: Can Agents Learn from mobile world model Imagination?

• Imagined trajectories provide transferable interaction experience rather than preserving the source distribution, and can improve generalization when behaviors remain actionable and useful.The authors identify iterative, more diverse collection as a way to mitigate distribution shift.Source: §5 · p. 11, RQ3: Can Agents Learn from mobile… · p. 12

6 Related Work

• Mobile GUI agents are described as vision-language-action models, alongside reviews of current agent frameworks and reinforcement-learning multimodal reasoning agents.The passage cites mobile GUI agents [5] [29], frameworks [31] [32], and RL-based agents [33] [34] [35] [36].Source: §6 · p. 12
• Textual world models have been applied at test time in mobile GUI tasks [9].Source: §6 · p. 12
• Diffusion models and rendering-code approaches aim to reconstruct GUI interfaces and provide VLAs with richer multimodal information [10] [37] [11] [12].Source: §6 · p. 12

7 Conclusion

• The study introduces a trajectory-annotated mobile GUI world-model dataset and evaluates delta text, full text, renderable code, and diffusion-based image prediction.Its conclusion reports that guidance gains depend on interaction mode, model confidence, modality, and inference cost.Source: §7 · p. 12

A.1 Mobile GUI Agents

• Reactive mobile agents that act from history and current observations lack foresight to anticipate unrecoverable navigation deviations in dynamic environments.The paper connects this limitation to compounding errors in long-horizon tasks.Source: §A.1 · p. 17

A.2 GUI World Models

• GUI world models predict future states from current observations and proposed actions, using text-level or pixel-level representations.The paper identifies how these models can effectively guide GUI agents as a neglected question.Source: §A.2 · p. 17

B.1 Full Results on AndroidWorld

• Delta text raises Gemini 3-Flash overall_SR from 50.57 to 66.59 and GPT-5.4 overall success rate from 56.03 to 63.79.Purely reactive m3a is insufficient for most agents on medium- and long-horizon tasks.Source: §B.1 · pp. 17–18
• Code2Image lowers Qwen3-VL-32B overall success rate from its 50.86 baseline to 49.57, as discontinuous rendered transitions can signal false task progress.The paper notes this risk in tasks requiring frequent, fine-grained state confirmation, such as text input and modification.Source: §B.1 · p. 18
• Larger models can propose more diverse, potentially correct actions, while smaller models often repeat actions and undermine heuristic search.The paper also reports that Gemini 3-Flash as judge can assess actions despite hallucinatory explicit world-model predictions.Source: §B.1 · pp. 18–19
• MA3 regresses with world-model verification: Delta Text scores 55.17, Full Text 51.94, and Code2Image 51.07.The paper traces this verification pollution to distorted predicted states that cause the judge to penalize valid actions.Source: §B.1 · p. 19

B.2 Entropy And Step Accuracy

• Entropy-accuracy correlations are -0.1354 on AITZ and -0.3558 on MMBench, and higher-entropy ranges show more frequent action changes.The figure compares accuracy across entropy ranges and self-reflection change rates.Source: Table 8 · p. 19, Figure 12 · p. 20
• High-entropy decisions offer more room for posterior resampling or correction, whereas low-entropy decisions limit opportunities for world-model feedback to change the action.The paper reports that GUI action distributions remain low-entropy overall, limiting test-time guidance gains.Source: §B.2 · pp. 19–20

B.3 World Model Inference Token Statistics

• HTML render-code completions average 5361.23 tokens, versus 121.68 for delta descriptions and 368.15 for full descriptions.Long-tail HTML can exceed a small model's output budget and fail to render when truncated, motivating filtering or prioritizing annotations small models can fully express.Source: §B.3 · p. 20

C Evaluation Metrics of Next UI Prediction

• The evaluation uses four specialized metrics across Functional Logic and Visual Quality, with a unified VLM-as-a-Judge framework.The metrics are intended to provide a fairer, more granular comparison.Source: C Evaluation Metrics of Next UI… · p. 20

C.1 Functional Logic

• Action Adherence (Sad) checks whether a predicted next state is a logically valid consequence of the action, penalizing updates that contradict the interaction.For example, it penalizes predicting the same page after clicking “Back.”Source: §C.1 · p. 21
• Action Identifiability (Sid) measures whether an observer can infer the action type from the visual transition, supporting the Selector’s verification of simulated outcomes.It uses an inverse-dynamics VLM to classify the action from the visual difference.Source: §C.1 · p. 21

C.2 Visual Quality

• Element Alignment (Sele) checks whether key interactive elements appear at the correct relative coordinates, allowing semantic placeholders when they occupy the correct screen area.Source: §C.2 · p. 21
• Layout Integrity (Slay) evaluates global layout and penalizes issues such as CSS collapse, overlapping containers, and misalignment.Source: §C.2 · p. 21

HTML Judge Prompt

• The HTML, text, and delta-text judges use task goal and action history alongside predicted outcomes, prioritizing task progress over prediction realism alone.The text judge treats prediction quality as supporting evidence; the delta-text judge evaluates incremental changes for progress toward the goal.Source: HTML Judge Prompt · p. 25, TEXT Judge Prompt · p. 26, DELTA_TEXT Reward Prompt · p. 27

E.1 AITZ Downstream Task Case Study

• In the AITZ example, HTML-based world-model feedback helps repair two actions whose predicted coordinates were incompatible with the intended interactions.When multiple candidate actions could complete the task, the example selects the one with the higher world-model score.Source: §E.1 · p. 28, Figure 13 · p. 28

E.2 AndroidControl Delta-Text Case Study

• The section includes AndroidControl case studies using delta-text and text-based world-model feedback, but the supplied captions report no outcomes.Source: Figure 14 · p. 28, Figure 15 · p. 29

E.4 AndroidWorld Text Case Study

• In an AndroidWorld file-moving case, contradictory world-model predictions led the Judge to score the same dialog-dismissal action −0.8 or 0.9.One prediction hallucinated that the dialog remained; the other predicted it disappeared.Source: §E.4 · p. 29

E.5 AndroidWorld Code2Image Case Study

• HTML-rendered predictions failed to simulate native text input and DOM updates in a file-renaming case, producing corrupted and overlapping strings.The distorted feedback could mislead candidate evaluation.Source: §E.5 · pp. 29–30, Figure 17 · p. 30

E.6 Small-Scale Model Generation Case Study

• A small vision-language model proposed the same click on UI element 11 for all three candidates, limiting action-space exploration and recovery through candidate selection.The natural-language rationales differed, but the executable actions did not.Source: §E.6 · p. 30, Figure 18 · p. 30

E.7 Case studies for diffusion steps

• A 40-step diffusion prediction took about one and a half minutes, while intermediate images retained traces of the input screenshot and struggled with substantial page transitions.Source: §E.7 · p. 31, Figure 19 · p. 31, Figure 20 · p. 32

E.8 Case studies for training with world model imagination

• World-model renderings simplified layouts and shifted predicted click locations toward the center, producing a centering bias in collected trajectories that degraded click accuracy after fine-tuning.The examples include a missing Lap button and omitted account-page settings that shifted target coordinates.Source: §E.8 · pp. 32–33, Figure 28 · p. 37
• Figures 21–26 caption six Code2Image case studies: search for the arts, view Moon-tab details, save a story, swipe to Romeo and Juliet or more details, and enter “Literature art”.Source: Figure 21 · p. 33, Figure 22 · p. 33, Figure 23 · p. 34

F Ethics Statement

• The ethics statement says the dataset was refined to remove privacy-compromising elements and that human evaluation used a randomized selection of IT professionals.It describes the panel as gender-balanced and educationally diverse.Source: F Ethics Statement · p. 38

G Reproducibility statement

• All models and datasets used in the paper are open-source.Source: G Reproducibility statement · p. 38
• The full experimental setup is detailed in the settings part.Source: G Reproducibility statement · p. 38
• Unless noted, experiments use the same settings.Source: G Reproducibility statement · p. 38

H Limitations

• Page-rendered models may miss app-specific layouts and styles; the authors say this does not affect downstream tasks because those styles carry little screenshot information.They suggest more reliable modeling through a method similar to RAG.Source: H Limitations · p. 38
• The paper analyzes individual modalities rather than combined methods, leaving multimodal combinations as a direction for future research.The authors note that combining modalities could compensate for their respective shortcomings.Source: H Limitations · p. 38

I Broader Impact

• World models may reduce unnecessary interactions and help agents avoid obviously incorrect or risky actions, while improving sample efficiency.These are stated as potential positive impacts of anticipating action consequences before execution.Source: I Broader Impact · p. 38
• Deployment risks include unintended automation, private-information exposure, and high-risk actions; the authors call for confirmation, safety constraints, privacy protections, and failure-case evaluation.Inaccurate predictions may mislead downstream agents when used without safeguards.Source: I Broader Impact · p. 38

• The checklist says experimental settings are described and the work’s limitations are discussed in Appendix H.Source: §2 · pp. 39–40
• The full code and processed data were not publicly released at submission, although the authors planned release upon publication.The authors report providing experimental settings, data-construction procedures, and evaluation protocols.Source: §2 · p. 41
• Error bars were not reported for all experiments because repeated training and evaluation were costly; fixed protocols and test splits were used instead.Source: §2 · p. 41
• The paper omits complete compute-resource information, including exact worker types, memory usage, and execution time for every experiment.The authors say model configurations, training data, and evaluation protocols are provided.Source: §2 · p. 42