Source-linked AI summary
MobileDreamer: Generative Sketch World Model for GUI Agent
Yilin Cao, Yufeng Zhong, Zhixiong Zeng, Siran Dai, Liming Zheng, Jing Huang, Haibo Qiu, Peng Shi, Wenji Mao
TL;DR
Mobile GUI agents often react to the current screen, making long-horizon planning difficult, while useful world models must preserve spatial information efficiently. MobileDreamer combines an order-invariant textual sketch world model with rollout imagination for multi-step action selection. On Android World, it improves task success by 5.25% and accurately forecasts key GUI elements.
Problem
Most mobile GUI agents remain reactive, and existing state-prediction approaches struggle to combine spatial awareness with computational efficiency for long-horizon planning.
Method
MobileDreamer uses an order-invariant textual sketch world model and recursively rolls out predicted trajectories through a tree-of-prediction for action selection.
Results
MobileDreamer improves Android World task success by 5.25% and accurately forecasts key GUI elements.
Takeaways & Limitations
The framework provides an efficient lookahead approach for mobile GUI agents by combining spatially informed state forecasting with proactive action selection.
Takeaways & Limitations
Performance may be less stable on visually noisy, uncommon-font, icon-dominant, or fine-grained visual tasks because sketches rely on OCR and UI parsing.
Abstract
from arXiv · showhide
Mobile GUI agents have shown strong potential in real-world automation and practical applications. However, most existing agents remain reactive, making decisions mainly from current screen, which limits their performance on long-horizon tasks. Building a world model from repeated interactions enables forecasting action outcomes and supports better decision making for mobile GUI agents. This is challenging because the model must predict post-action states with spatial awareness while remaining efficient enough for practical deployment. In this paper, we propose MobileDreamer, an efficient world-model-based lookahead framework to equip the GUI agents based on the future imagination provided by the world model. It consists of textual sketch world model and rollout imagination for GUI agent. Textual sketch world model forecasts post-action states through a learning process to transform digital images into key task-related sketches, and designs a novel order-invariant learning strategy to preserve the spatial information of GUI elements. The rollout imagination strategy for GUI agent optimizes the action-selection process by leveraging the prediction capability of world model. Experiments on Android World show that MobileDreamer achieves state-of-the-art performance and improves task success by 5.25%. World model evaluations further verify that our textual sketch modeling accurately forecasts key GUI elements.
1 Introduction
Mobile GUI agents are often reactive, limiting long-horizon planning because they do not anticipate downstream interface changes. MobileDreamer addresses this with a textual sketch world model and rollout imagination for proactive action selection.
- Motivation: Reactive mobile GUI agents often make short-sighted decisions in complex long-horizon tasks because they lack downstream interface predictions.World models can forecast post-action states, helping agents make more forward-looking decisions.
- Motivation: Text-based state prediction lacks spatial information, while image-based prediction is computationally expensive and includes task-irrelevant visual details.These trade-offs make practical multi-step GUI planning difficult.
- Approach: MobileDreamer combines a lightweight textual sketch world model with rollout imagination to predict long-term outcomes before acting.The framework forecasts post-action states and recursively feeds predicted trajectories into action selection.
- Approach: Its order-invariant learning objective preserves essential spatial information while remaining robust to changes in GUI element order and small position shifts.The objective uses element matching with an IoU-aware cost.
- Results: Experiments show significant improvements in future-state forecasting and Android World task success over several baseline GUI agents.The evaluation also reports accurate forecasting of key GUI elements.
- Approach: The rollout strategy uses a multi-step tree-of-prediction to improve proactive action selection beyond single-step greedy decisions.Predicted future trajectories are recursively supplied to the GUI agent as forecast evidence.
2 Related Work
Related work spans world models, GUI representations, and mobile-agent planning. Existing approaches use textual or pixel-level predictions, but commonly face spatial, computational, or multi-step lookahead limitations.
- World Models: World models predict environment dynamics to help intelligent agents anticipate future states for decision-making across embodied AI, autonomous driving, and games.GUI applications include natural-language transition and action-outcome prediction.
- GUI World Models: Early GUI world models represent interface changes with natural language, transition-focused abstractions, or natural-language triplets.Examples include WEBDREAMER, WMA, and MobileWorld.
- GUI World Models: Pixel-level next-frame prediction captures visual information but incurs high computational costs and struggles with fine-grained detail reconstruction.These costs limit practical use for GUI planning.
- Planning: Existing GUI world-model approaches generally lack effective multi-step lookahead and rely on single-step greedy selection.This restricts agents to considering only one step of possible outcomes.
- GUI Agents: GUI agents have evolved from structured HTML or accessibility-tree inputs toward vision-based methods that process screenshots directly.Vision-based methods demonstrate superior generalization, while UGround focuses on locating target elements on current screens.
- Mobile Agents: Despite advances such as synthetic trajectories and exploration videos, mobile agents remain largely reactive and do not anticipate future consequences.This limitation is identified across recent mobile-agent approaches.
3 Method
MobileDreamer combines an action-conditioned textual sketch world model with rollout imagination to forecast GUI states and support multi-step action selection. It represents task-relevant UI elements structurally, uses order-invariant matching for robust prediction, and feeds predicted action paths back to the agent.
- Framework: MobileDreamer has two stages: TSWM forecasts post-action GUI states, while rollout imagination expands candidate actions into a multi-step prediction tree.The agent uses this tree to select actions based on simulated downstream utility.
- Textual Sketch World Model: TSWM predicts only planning-relevant GUI information in a structured textual format instead of reconstructing full screenshots.The representation retains each element’s label, text, and bounding-box position to balance efficiency with spatial description.
- Order-Invariant Learning: Order-invariant learning matches predicted and ground-truth elements one-to-one with an optimal-transport cost over geometry and content.This addresses order-sensitive token supervision and reduces sensitivity to small text or bounding-box perturbations.
- Order-Invariant Learning: The matching objective preserves element-level structure under action-conditioned transitions and makes forecasts more robust to element reordering and small perturbations.The final objective combines the matching loss with token-level cross-entropy.
- Rollout Imagination: Rollout imagination evaluates M candidate actions by recursively forecasting successor states to a specified depth d, forming short predicted trajectories.The branching factor is controlled by M, and each root-to-leaf path represents a candidate future.
- Rollout Imagination: The GUI agent receives the real screenshot plus a textual prediction tree, enabling explicit comparison of alternative actions by their anticipated states.The predicted trajectory is fed back into subsequent action generation for longer-term planning.
4 Experiments
Experiments evaluate textual sketch forecasting and MobileDreamer on Android World, including baseline comparisons and component ablations. Results show accurate element-level predictions and improved task success, with order-invariant learning and recursive prediction contributing to performance.
- World-model evaluation: The textual sketch world model substantially improves spatial and textual future-state forecasting over prompt-based LLM predictors and Qwen3-8B.Evaluation uses mIoU, text similarity, precision, recall, and F1-score for element-level predictions.
- Android World: Android World evaluates reproducible task execution across 116 navigation tasks in 20 real-world Android apps using benchmark-verified Success Rate.Tasks include standardized initialization, success checking, and teardown procedures.
- Main results: MobileDreamer consistently improves task success across evaluated backbones and surpasses basic LLM agents and prior text-based or image-based world-model baselines.The framework introduces predicted trajectories into action selection and uses recursive tree-of-prediction feedback.
- Key-component ablation: Removing the sketch world model and tree-of-prediction reduces the system to a reactive baseline, while adding these components improves long-horizon action selection.Order-invariant learning further improves robust next-state sketches, and tree-of-prediction achieves the best ablation performance.
- Rollout-imagination ablation: Increasing rollout depth from one to two steps improves performance, whereas deeper recursive feedback can introduce noisy information and diminishing returns.The number of candidate actions also helps up to a moderate branching factor.
- Textual-sketch ablation: SFT substantially improves next-state forecasting over the base model, while MobileDreamer further strengthens prediction through order-invariant learning.The objective improves robustness to element reordering and small spatial-position noise.
5 Conclusion
MobileDreamer is an efficient sketch world model-based lookahead framework for mobile GUI agents. Its textual sketch predictions and recursive rollout imagination improve post-action forecasting and proactive action selection on Android World.
- Conclusion: MobileDreamer combines a lightweight textual sketch world model with order-invariant learning and recursive rollout imagination for proactive GUI action selection.The framework feeds predicted trajectories back into action selection.
- Conclusion: Textual sketch modeling enables more effective post-action state predictions for GUI agents on Android World.The conclusion identifies improved future-state prediction as an experimental outcome.
Limitations
MobileDreamer depends on OCR and UI element parsing to construct textual sketches, which limits reliability when textual cues or fine-grained visual signals are difficult to recover.
- Representation scope: OCR-based textual sketches may be less stable on visually noisy screens, uncommon fonts, and icon-dominant layouts.The authors plan to improve sketch extraction and add complementary visual cues.
- Representation scope: The sketch representation can be less informative for precise text reading or distinguishing subtle UI appearance differences.These cases require fine-grained visual signals beyond the captured layout and text cues.
Ethical Considerations
All experiments use virtual environments, publicly available datasets, and synthetic data without accessing real user systems or personal information.
- Data and environment: The experiments are conducted entirely in virtual environments without access to real user systems.The study uses publicly available datasets and synthetic data.
- Data and environment: No personal information is involved in the datasets or experiments.
A OCR Reliability Analysis
OCR-based sketch extraction is reliable for most elements, but missed elements remain a limitation that can affect predicted lookahead references.
- Error Sources: OCR errors can produce misread, hallucinated, or missed UI elements in extracted sketches.Annotators assessed correctness by comparing extracted text and locations with screenshots, while also marking expected but absent elements.
- Evaluation Protocol: Screenshot-level error rates are not mutually exclusive and therefore do not sum to 100%.Each rate measures the fraction of screenshots containing at least one error of the corresponding type.
- Findings: High element-level Precision and F1 suggest that OCR-based sketch extraction is reliable in most cases.The evaluation reports element-level Precision, Recall, and F1 from manual annotations of 300 screenshots.
- Findings: Missing elements remain a limitation, although the agent still receives the real current screenshot as its primary observation.Predicted future sketches serve as an additional lookahead reference, so OCR noise affects that reference rather than replacing the current screenshot.
B Additional Results under the M3A-style Agent Pipeline
MobileDreamer consistently improves task success when added to a stronger M3A-style GUI-agent pipeline, though absolute rates are not directly comparable with the minimal backbone.
- Setup: The M3A-style pipeline adds ReAct-style reasoning and self-reflection to summarize progress and improve subsequent decisions.The pipeline is also used by UGround for Android World results with GPT-4 and GPT-4o.
- Setup: Absolute success rates are not directly comparable because the main experiments use a more minimal backbone without M3A-style selection or reflection.Additional experiments therefore test MobileDreamer on top of the stronger M3A-style pipeline.
- Results: On GPT-4, MobileDreamer improves the M3A baseline from 26.4% to 31.6%.These results come from Android World experiments using the M3A-style pipeline.
- Results: On GPT-4o, MobileDreamer improves the M3A baseline from 39.4% to 45.1%.The gains are reported under the same stronger agent pipeline.
C Step Efficiency Analysis
On 64 overlapping successful tasks, MobileDreamer uses fewer interaction steps than the baseline, indicating more efficient completion when both methods succeed.
- Evaluation Setup: The analysis compares average steps on 64 tasks that succeed with and without the textual sketch world model.Measurements use Claude-Sonnet-4.5 and exclude tasks not successful in both settings.
- Results: 19.4% fewer steps: MobileDreamer averages 8.7 steps versus 10.8 for the baseline.The comparison is restricted to tasks completed by both methods.
D Inference Cost Analysis
Tree-of-Prediction improves success rate as rollout search expands, while compact textual sketches keep the additional per-step latency manageable relative to greedy selection.
- Cost Measurement: Inference cost is measured as end-to-end mean wall-clock time per decision step across rollout depths and candidate-action counts.World-model inference is parallelized across nodes at the same rollout depth, with overhead reported relative to greedy selection.
- Trade-off: Tree-of-Prediction yields consistent success-rate improvements with additional per-step overhead.The analysis uses Claude-Sonnet-4.5 and varies rollout depth and the number of candidate actions per node.
- Best Setting: 65.78% success rate: the best setting improves SR from 60.53% to 65.78% at 1.90× total per-step latency.This setting uses depth 2 and 3 actions per node.
- Trade-off: The textual sketch world model keeps rollout overhead manageable by predicting future states in a compact text-sketch space.The reported latency comparison is against greedy selection.