MobileWorldBench: Towards Semantic World Modeling For Mobile Agents

1. Introduction

• Pixel-space prediction must infer action semantics, future layout and content, then render text and icons, although exact article content is often unnecessary for decisions.For example, subscribing or navigating to sports news does not require rendering the exact articles that appear after scrolling.Source: §1 · pp. 1–2
• The paper formalizes semantic world modeling by representing state changes as structured textual descriptions rather than high-dimensional pixel predictions.It presents this as a way to make GUI world modeling more efficient and generalizable.Source: §1 · p. 2
• MobileWorldBench evaluates VLMs’ predictions of future states from the current screen and a user action, using generation and yes-no question-answering tasks.The generation descriptions are compared with ground-truth next-state screenshots, while QA uses an accuracy metric.Source: §1 · p. 2
• MobileWorld provides training data pairing current states and user actions with future states represented as screenshots, QA pairs, and natural-language transition descriptions.The dataset is constructed from existing trajectories, with advanced VLMs generating QA pairs and annotating transitions.Source: §1 · p. 2
• Mobile agents using semantic world models achieve a +7.4% increase in success rate on AndroidWorld.The reported evaluation finetunes an open-source VLM using MobileWorld.Source: §1 · p. 2

2.1. World Modeling

• World-modeling approaches range from pixel-level video generation to latent feature prediction and semantic prediction of future-state descriptions.The cited examples include pixel-level models, V-JEPA’s latent-space predictions, and SWM and WMA’s semantic approaches.Source: §2.1 · p. 2

2.2. Mobile Agents

• Mobile-agent systems use UI structure or visual inputs, and commonly select actions through labeled UI elements or predicted coordinates in reactive loops.Training-based approaches include supervised finetuning and reinforcement learning.Source: §2.2 · p. 3

2.3. GUI Datasets & Benchmarks

• Unlike benchmarks centered on GUI understanding or decision-making, MobileWorld and MobileWorldBench focus on predicting future states from current observations and user actions.The related work also describes a model-based policy that combines a semantic world model with an action proposal model and value model.Source: §2.3 · p. 3, Figure 3 · p. 3

3.1. Semantic GUI World Modeling

• Semantic world modeling factors future-screen prediction into predicting state changes from the current screen and action, then rendering the next state from those changes and the prior screen.The model represents changes as a latent variable and uses text descriptions and questions about the next state to assess or use its semantics.Source: §3.1 · pp. 3–4

3.2.2. Data Generation and Filtering Pipeline

• MobileWorldBench is built by curating trajectories, annotating actions and state changes, generating QA candidates, and filtering them with model checks and human review.The pipeline’s final human verification checks correctness and relevance.Source: Figure 4 · p. 4

3.2. MobileWorldBench

• MobileWorldBench evaluates semantic state-transition predictions by measuring how well generated descriptions and next-state truth estimates capture the transition.The two evaluation tasks are Next-State-Generation and Next-State-QA.Source: §3.2 · p. 4

3.2.1. Task Definitions

• Next-State-Generation receives accuracy, relevance, and completeness ratings of 0–5 each, summed to a 0–15 score; Next-State-QA measures accuracy on yes-or-no future-state questions.Generation is judged against the current state, action, and ground-truth next state; relevance discounts accurate but trivial statements, while completeness rewards detailed coverage of relevant changes.Source: §3.2.1 · pp. 4–5

3.2.3. Coverage of Tasks, Apps, and Actions.

• Benchmark coverage spans four broad task categories, 22 apps, and common actions, while task categories do not map one-to-one onto apps.Web shopping accounts for 38.8% of tasks, whereas Chrome accounts for 54.4% of screens.Source: §3.2.3 · p. 6, Figure 6 · p. 6

3.3. MobileWorld

• MobileWorld contains 1.4M samples from human demonstrations, with 942k state-change descriptions and 543k QA pairs after filtering.Annotations use Qwen3-VL-8B for 90% of data and Qwen3-VL-235B-A22B for 10%; the training set is not human-filtered due to cost concerns.Source: §3.3 · p. 6

4.1. Main Results on MobileWorldBench

• Gemini-2.5-Pro leads Next-State-QA but performs poorly on Next-State-Generation, with 4.48 relevance and 3.49 completeness; Gemini-2.5-Flash scores 0.10 higher on generation while retaining strong QA accuracy.The authors suggest reasoning finetuning may contribute to this trade-off: Gemini-2.5-Pro favors relevance and completeness over accuracy, and can produce detailed but hallucinated descriptions.Source: §4.1 · p. 7
• Both annotation pipelines outperform existing models, and the relatively small gap between them suggests Qwen3-VL-8B-Instruct can serve as a high-quality annotator.The pretraining pipeline uses Qwen3-VL-8B-Instruct, while the finetuning pipeline uses Qwen3-VL-235B-A22B; gains are more pronounced for accuracy and completeness than relevance.Source: §4.1 · pp. 7–8
• Finetuning on MobileWorld achieves 4.19 accuracy and 12.39 overall score on Next-State-Generation, and improves performance on both benchmark tasks.The finetuned model achieves the highest accuracy and overall scores on generation.Source: §4.1 · p. 8

4.2. Online Evaluation on AndroidWorld.

• Using either world model outperforms the M3A baseline on AndroidWorld, with the finetuned Qwen3-VL-8B-Instruct world model performing best.The comparison tests M3A without a world model, with Qwen3-VL-235B-A22B as both policy and world model, and with the finetuned model as world model.Source: §4.2 · p. 8

4.3. Human Evaluation.

• Human evaluation over 3,000 randomly sampled model-output matches shows a considerable performance gain after finetuning on MobileWorld.Judges selected the more helpful output for decision-making, with access to the ground-truth next-state image; results are reported as ELO scores in Figure 7.Source: §4.3 · p. 8, Figure 7 · p. 8

• The work introduces MobileWorldBench and MobileWorld, reports substantial gains from training on MobileWorld, and demonstrates semantic world models integrated into mobile agents.Both MobileWorld and MobileWorldBench currently consist solely of human demonstrations on Android; the authors plan to extend them to iOS and other GUI environments.Source: §5 · p. 8

7.1. Dataset Statistics

• MobileWorld is presented as the first dataset focused on next-state prediction, with a larger scale than most existing datasets; GUI-World remains a GUI-understanding task despite multi-step trajectory questions.GUI-World questions can include past and future actions, unlike world modeling, where future states are not provided to the model.Source: §7.1 · p. 12
• MobileWorld's app and action distributions resemble MobileWorldBench because both draw from the same human demonstrations, though the training data contains more click operations.The authors attribute the click difference to the train/test split of the source data; Figures 8 and 9 show distributions and visual action overlays.Source: §7.1 · p. 12, Figure 8 · p. 12, Figure 9 · p. 13

7.2.1. VLM Annotation

• VLM annotators convert low-level actions into high-level descriptions and describe state changes using before-action, action-overlay, and after-action screenshots.The prompt asks for an action description and a concise account of changes between the first and third images; overlays mark click coordinates with crosses and swipes or scrolls with arrows.Source: §7.2.1 · p. 13

7.2.2. VLM Filtering

• For each transition, the process samples eight QA proposals, then filters proposals for answer accuracy and action relevance, retaining 2,458 of 4,000 candidates.Automated self-checking uses the current state, action, and ground-truth next state; a separate relevance filter removes questions about unrelated system details.Source: §7.2.1 · p. 14, §7.2.2 · pp. 14–15

7.2.3. Human Filtering

• Human reviewers check the remaining QA pairs for answer correctness, action relevance, and whether an average user could reasonably answer them, leaving 1,787 benchmark pairs.Workers were screened using a manually annotated 100-sample test set.Source: §7.2.3 · p. 15

8.1. Training Setup

• The Qwen3-VL-8B-Instruct base model is finetuned on MobileWorld for 2 epochs at a 2 × 10^-6 learning rate using 8 A6000 GPUs.Further training configurations, including training steps, optimizer setup, and image resolution, are listed in Table 5.Source: §8.1 · p. 15, Table 5 · p. 15
• Figure 10 shows annotator interfaces for correctness and relevance filtering and for detecting ambiguous responses.Source: Figure 10 · p. 16

8.2.1. Model Choice and Sampling Parameters

• For next-state generation, models use their default sampling parameters; temperature is set to 0.0 for QA, since it caused repetitions in some generation outputs.Table 6 maps closed-source model names to precise identifiers.Source: §8.2.1 · p. 16, Table 6 · p. 16

8.2.2. Automatic VLM Evaluation

• GPT-4o scores next-state descriptions on accuracy, completeness, and relevance from 1 to 5, then gives an overall score out of 15.The prompt supplies the current screenshot, action, next-state image, and reference changes.Source: §8.2.2 · pp. 16–17

8.2.3. Human Evaluation

• Human evaluators compare two model outputs instead of assigning numerical scores, and 3,000 randomly formed matches are used to compute ELO scores.The ELO setup follows LM-Arena and Gen-AI-Arena [23] [53].Source: §8.2.3 · p. 17

8.2.4. AndroidWorld

• AndroidWorld uses the official M3A baseline, while the model-based policy scores predicted next states from 1 to 10 for progress toward the goal.The prompt evaluates expected changes from proposed actions; its omitted background is copied from M3A.Source: §8.2.4 · p. 17

8.3.1. More Qualitative Examples.

• Across additional MobileWorld examples, model performance varies by sample, while the finetuned model ranks among the top performers on most samples.Examples include screenshots, actions, model outputs, and GPT judge scores.Source: §8.3.1 · p. 18

8.3.2. Model-Scaling

• Finetuning improved Qwen3-VL-2B-Instruct by +5.4 versus +4.1 for 8B, yet 2B underperformed 8B; its ground-truth-image VQA accuracy was 70.0% versus >90% for GPT4o, human annotators, and 8B.Table 7 reports qualitative next-state-QA results; the authors attribute the gap largely to the 2B model's limited GUI understanding.Source: Table 7 · p. 18, §8.3.2 · p. 18

8.3.3. Combining Semantic World Models to Pixel-Space World Models

• Accurate semantic text predictions can guide a pixel renderer to produce highly plausible next-state screenshots, treating semantic modeling as a subprocess of pixel-space world modeling.The paper leaves further pixel-world-modeling exploration to future work and focuses on the semantic level.Source: §8.3.3 · p. 18

Related-work discussion

• The authors cannot compare with ViMo [30] because its model checkpoints, training data, and evaluation data were not released.ViMo focuses on pixel-space GUI world models, including text-rendering and GUI-consistency challenges.Source: §10 · p. 18

Supplementary figures

• Figure 11 shows the human-evaluation interface for comparing two models.The interface supports the pairwise human-evaluation procedure.Source: Figure 11 · p. 19
• Figure 12 presents qualitative MobileWorld evaluation examples.The examples include screenshots, actions, model outputs, and GPT judge scores.Source: Figure 12 · p. 20
• In pixel-rendering examples, Gemini Nano Banana Pro has the best qualitative results, and text guidance produces predictions closer to ground truth.Figure 13 combines semantic-model output with pixel renders.Source: Figure 13 · p. 21