Source-linked AI summary
Latent World Action Model for GUI Agents
Seungjun Yu, Hojun Choi, Seojeong Park, Jaeyo Shin, Hyunjung Shim
TL;DR
Pixel-space GUI world models render future screens for re-perception, requiring broad appearance modeling even though action selection may depend on only some future information. This paper uses shared latent representations for action scoring and action-conditioned transition prediction, with an action-aware contrastive objective. It reports higher success rates than pixel-space world models in offline and online settings and lower inference latency.
Problem
Action selection may depend on only a subset of future GUI information, while point-wise latent prediction may preserve the current state instead of capturing action-induced change.
Source: §3.1 · p. 4, §3.2.1 · p. 5Method
A LoRA-adapted LMM shares latent representations between candidate-action scoring and action-conditioned next-state prediction, trained with an action-aware contrastive objective.
Source: §1 · p. 2Results
Across offline and online GUI settings, success rates exceed pixel-space world models by up to 31.9% and 40.6%, respectively, while inference latency falls by up to 98.9%.
Source: §1 · p. 3Takeaways & Limitations
The results support predicting decision-relevant action consequences in latent space rather than reproducing the full visual appearance of future screens.
Source: §5 · p. 10The model re-ranks candidates from a frozen base agent and cannot recover a correct action absent from that candidate set.
Source: A LIMITATIONS AND FUTURE WORK · p. 13
Abstract
from arXiv · showhide
Pixel-space world models have recently emerged as a promising approach for GUI agents, predicting future screens prior to execution and subsequently re-perceiving them for action selection. However, this two-stage paradigm faces two key challenges in GUI environments: (L1) the high computational cost of pixel reconstruction and (L2) the need to reconstruct large and diverse appearance-level details beyond what is required for action selection. In practice, this screen-generation objective can induce an identity shortcut problem, where future-state prediction largely preserves the current state rather than capturing action-induced changes. In this work, we reformulate GUI world modeling by shifting the prediction target from full-screen reconstruction to action-induced future information in latent space. Concretely, we leverage a large multimodal model as the latent world model, whose shared representations support two dedicated heads for both action selection and predictive transition modeling. The action head directly translates these latent representations into action selection, while the transition head captures action-conditioned future information. To mitigate the identity shortcut, we further introduce an action-aware contrastive objective that encourages the transition representations to capture discriminative action-induced changes. Extensive experiments across offline and online GUI settings show that our method achieves up to 31.9% and 40.6% relative gains in success rate over pixel-space world models, respectively, while substantially reducing inference latency by up to 98.9%. Our code is available at https://github.com/sjyu001/latent2world.
1 INTRODUCTION
GUI world models can simulate candidate actions before execution, but pixel-space prediction requires rendering future screens for a separate model to re-perceive. This work instead trains shared latent representations for action selection and future-state prediction, reporting gains over both base agents and pixel-space world models with lower inference latency.
1 INTRODUCTION
- The render-and-re-perceive approach predicts a visually renderable next state for each candidate action, then uses a separate model to score them.This can provide explicit spatial and visual information, such as element locations and layouts.Source: §1 · p. 1
- GUI transitions have lower visual similarity and greater variance than natural videos, broadening the future appearances pixel-space models must handle.This contrast is the stated comparison in Figure 2.Source: Figure 2 · p. 2
- A LoRA-adapted LMM jointly supports candidate-action scoring and prediction of the observed action’s next-state visual embedding.An action-aware contrastive objective aligns the predicted embedding with the next state while separating it from the current state and unrelated transitions; inference discards the transition head.Source: §1 · p. 2
- Success rates improve over base agents without world modeling by up to 11.6% and 47.0% in offline and online settings, respectively.The sentence continues across the two supplied passages.Source: §1 · pp. 2–3
- Pixel-space world models are outperformed by up to 31.9% and 40.6%, while world-model inference latency is reduced by up to 98.9%.The comparisons are across offline and online GUI settings, respectively.Source: §1 · p. 3
- The approach shifts GUI world modeling from full-screen reconstruction to action-induced future information in latent space.Its stated contributions combine action selection with action-conditioned future prediction and report gains over pixel-space models and base agents.Source: §1 · p. 3
2 RELATED WORKS
Prior mobile GUI-agent work has improved action selection through perception, reasoning, and policy optimization. GUI world-model research ranges from text and sketches to rendered screens or code, while this paper motivates latent prediction as an alternative to reconstructing broad visual detail.
2.1 MOBILE GUI AGENTS
- Mobile GUI-agent research has advanced visual grounding, interaction-history modeling, self-reflection, and reinforcement-learning-based policy optimization.The paper frames its focus as complementary: how predictive world-model information is represented and integrated to support action selection.Source: §2.1 · p. 3
2.2 GUI WORLD MODEL FOR APP AGENTS
- GUI world models span text descriptions, spatial sketches, synthesized screenshots, and renderable HTML, each representing future states in different forms.The paper motivates latent future prediction because explicitly materializing GUI states requires modeling broad appearance details and computational cost beyond action selection needs.Source: §2.2 · p. 3
3 METHODOLOGY
The method replaces full-screen prediction with action-induced future information in a shared LMM latent space, supporting action selection and transition prediction. Its contrastive objective is designed to distinguish action-specific futures from the current state and competing outcomes.
3 METHODOLOGY
- The framework shifts GUI world modeling from full-screen reconstruction to action-induced future information in a shared latent space.Shared representations support both action selection and predictive transition modeling.Source: §3 · pp. 3–4
3.1 LATENT WORLD-ACTION MODEL FOR GUI AGENTS
- An LMM produces action-specific latent representations that a lightweight action head maps to candidate-action scores.Predictive supervision shapes the same shared representation space without explicit screen reconstruction.Source: §3.1 · p. 4
3.2.1 IDENTITY SHORTCUT IN GUI WORLD MODELING
- Across transition magnitudes, pixel-space shortcut severity rises and success rates fall, while the latent model attenuates both trends.Transition magnitude is grouped using average absolute pixel change between grayscale 64×64 screenshots.Source: Figure 4 · p. 5, §3.2.1 · p. 6
3.1.1 LATENT REPRESENTATIONS FOR GUI ACTION SELECTION
- Candidate-specific hidden representations are scored directly for action decisions, without reconstructing future screens.The candidates are placed in one randomly permuted prompt, and the resulting distribution is supervised with the annotated action.Source: §3.1.1 · p. 5
3.2 PREDICTIVE LATENT MODELING OF GUI TRANSITIONS
- A transition head predicts the annotated action’s next-state representation from the shared latent features, providing future-state supervision without screen reconstruction.Because the shared representation also scores candidates, prediction requires it to encode which candidate is correct and what it leads to.Source: §3.2 · p. 5
3.2.2 ACTION-AWARE CONTRASTIVE LEARNING FOR LATENT GUI WORLD MODELING
- The contrastive objective aligns predicted futures with observed next states while separating them from the current state and competing futures.The current-state latent and a FIFO bank of latents from other transitions supply negatives, encouraging action-specific future representations.Source: §3.2.2 · p. 6
4 EXPERIMENTS
Experiments cover offline and online GUI interaction, including in-distribution mobile tasks and out-of-distribution cross-app navigation. The latent model improves action selection and task success, while pixel-space world models show weaker results and higher costs.
4 EXPERIMENTS
- Evaluation spans offline AndroidControl and AITZ, out-of-distribution GUI-Odyssey, and online AndroidWorld, using step-level action metrics and task-level success and efficiency measures.GUI-Odyssey tests cross-app, longer-horizon navigation; online evaluation also reports Tokens/Task, Mean Steps, and Run Time.Source: §4 · pp. 6–7
4.1 OFFLINE EVALUATION
- Pixel-space world models reduce offline SR versus base agents in every setting, by up to 8.6 points; latent prediction adds modest inference overhead.Compared with base agents, the latent method increases Run Time by 0.08–0.46 and Tokens/Task by 61–675; Code2World uses roughly 3.5–5.9× more tokens.Source: §4.1 · pp. 7–8
4.1 OFFLINE EVALUATION; 4.2 ONLINE EVALUATION
- Offline SR improves in seven of eight backbone–dataset settings (up to +4.8 points), while AndroidWorld Overall SR is highest across all five backbones (+4.5 to +12.5 points).These results are reported on AndroidControl/AITZ and AndroidWorld, respectively.Source: Table 1 · p. 7, §4.1 · p. 7, Table 2 · p. 8
4.2 ONLINE EVALUATION
- Code2World requires 2.1–4.1× base-agent Run Time and Diffusion2Image 4.5–18.2×, while the latent model has lower Run Time than both across backbones.On GPT-5.6 and Gemini-3.6-Flash, the latent method also reduces Mean Steps and total Tokens/Task versus the base agents.Source: §4.2 · p. 8
4.3 ABLATION STUDIES
- On out-of-distribution GUI-Odyssey, the latent approach achieved the highest SR on GUI-Owl-7B and Qwen3-VL-8B, improving over corresponding base agents under cross-app, longer-horizon distribution shift; both pixel-space models fell below base agents.Source: §4.3 · p. 8
- On AndroidWorld, action rescoring raises SR from 27.68 to 36.36, transition prediction adds +1.82 points, and contrastive learning adds +2.00, reaching 40.18 (+12.50 versus base).The contrastive gain is attributed to distinguishing action-induced futures from the current state and competing futures.Source: Table 4 · p. 9, §4.3 · p. 9
- After current-state cluster means are subtracted, t-SNE shows the clearest action-specific structure for the latent model; Base and Diffusion remain mixed, while Code2World partly separates.The analysis is qualitative evidence that predicted representations capture action-induced changes beyond current-screen correlations.Source: Figure 5 · p. 9, §4.3 · p. 9
5 CONCLUSION
The paper reframes GUI world modeling around action-induced future information rather than full-screen reconstruction. It jointly learns action evaluation and predictive transitions, reporting performance gains and faster inference than pixel-space world models.
5 CONCLUSION
- GUI world modeling is reframed from full-screen reconstruction to predicting action-induced future information relevant to action selection.Source: §5 · p. 9
- The latent world-action formulation jointly learns action evaluation and predictive transitions with action-aware contrastive learning in a shared space.Source: §5 · p. 9
- The authors conclude that future prediction need not reproduce a screen’s appearance, but should capture decision-relevant action consequences without costly screen generation and reperception.Source: §5 · p. 10
A LIMITATIONS AND FUTURE WORK
The appendix identifies limits in candidate coverage, selective invocation, and one-step prediction, then gives implementation details for the baseline and proposed model. It also outlines extensions for broader candidate coverage, longer-horizon reasoning, interpretability, and evaluation beyond mobile GUIs.
A LIMITATIONS AND FUTURE WORK
- The model cannot recover a correct action missing from the frozen base agent’s candidate set, so performance depends on proposal quality and coverage.The authors suggest more diverse sampling, adaptive candidate-set sizes, or improved candidate generation when proposals are narrow or redundant.Source: A LIMITATIONS AND FUTURE WORK · p. 13
- High-confidence base-agent errors can bypass rescoring, and terminal decisions such as completing or abandoning a task are not rescored.A learned, uncertainty-aware invocation policy is proposed as a possible alternative to the fixed confidence threshold.Source: A LIMITATIONS AND FUTURE WORK · p. 13
- Training predicts only the immediate action-conditioned next state; the model does not explicitly reason over longer-horizon consequences or reconstruct interpretable future screens.The transition head is discarded at inference; multi-step rollouts and lightweight decoders or probing modules are proposed extensions.Source: A LIMITATIONS AND FUTURE WORK · p. 13
- Future work proposes combining stronger candidate generation, learned invocation policies, and multi-step latent dynamics, and testing generalization beyond mobile GUIs.The authors identify evaluation across web and desktop interfaces and training with candidates from multiple base agents as ways to assess broader generalization.Source: A LIMITATIONS AND FUTURE WORK · pp. 13–14
B EXPERIMENTAL DETAILS
- The experimental appendix reports Ubuntu 22.04, eight NVIDIA RTX A6000 GPUs, dual AMD EPYC 7413 CPUs, and a hyperparameter table for training and inference.Training used approximately 31 GiB of GPU memory per device; evaluation used approximately 22 GiB for the judge and 26 GiB for the Qwen3-VL-8B agent.Source: §B.1 · p. 14, §B.2.4 · p. 16, Table A1 · p. 16
B.2.1 WORLD-MODEL BASELINE PROTOCOL FOR GUI AGENTS
- The render-and-re-perceive baseline proposes candidate actions, predicts a future state for each, then scores those predicted states to select an action.Predicted states can be screenshots or renderable HTML, generated through image synthesis or code generation, respectively.Source: §B.2.1 · p. 14
B.2.2 MODEL ARCHITECTURE OF THE PROPOSED METHOD
- The proposed model uses a shared LMM representation for candidate-action scoring and next-state latent prediction, with frozen screenshot embeddings and memory-bank negatives supervising transitions.The candidate actions come from the frozen base agent; a FIFO bank supplies additional negative future-state representations for contrastive learning.Source: §B.2.2 · p. 15
B.2.3 JOINT TRAINING AND INFERENCE
- Joint training connects action selection with action-conditioned transition supervision, while inference discards the transition head and selectively rescores low-confidence decisions.Candidate scores come directly from the shared backbone and action head, without generating or re-perceiving screens.Source: §B.2.3 · p. 16
B.2.5 ANALYSIS SETTINGS
The analysis characterizes GUI appearance changes and defines how the paper measures identity shortcuts and action-specific transition representations. It compares pixel-change residuals across visual domains and probes representations by action type.
GUI Transition Analysis
- GUI transitions produce substantially larger post-warp residuals than natural videos, indicating that pixel transport explains less of their changes.Figure A1 contrasts small residuals for natural video transitions with larger residuals for action-induced GUI transitions.Source: Figure A1 · p. 17
- 2.4× larger residual magnitude and 2.8× higher residual-coverage AUC distinguish GUI transitions from natural videos in the comparison.The cross-domain analysis also reports that GUI transitions have larger and more spatially extensive appearance changes than the other evaluated domains.Source: §B.2.5 · p. 17, Figure A2 · p. 17
Identity-Shortcut Analysis
- An identity shortcut is counted when the ground-truth change exceeds the predicted change while the prediction remains within 0.08 pixel difference of the current screen.Transitions are grouped by magnitude using thresholds of 0.08 and 0.24, and shortcut severity is the fraction meeting the criterion in each group.Source: §B.2.5 · p. 18
Action-Specific Representation Analysis
- The representation probe samples 900 AndroidControl transitions across click, scroll, and text input, then visualizes action-conditioned residual representations with t-SNE.Current-state representations are clustered first to reduce confounding from current-state similarity; the analysis uses 52 qualifying clusters.Source: §B.2.5 · p. 18
B.3 DATASET DETAILS
The appendix describes three mobile GUI datasets for offline, online, and out-of-distribution evaluation. AndroidWorld measures end-to-end task completion through execution, while GUI-Odyssey tests transfer to long-horizon cross-app navigation.
B.3 DATASET DETAILS
- AndroidControl contains 15,283 demonstrations across 14,548 tasks and 833 Android apps, and is used for offline in-distribution step-level evaluation.Its trajectories include screenshots, accessibility trees, instructions, actions, and both high-level goals and low-level step instructions.Source: §B.3 · p. 18
- AITZ contains 18,643 screen–action pairs with action-level annotations and chain-of-action-thought labels, and serves as an additional offline in-distribution benchmark.Its scenarios include general app use, Google applications, installations, single-app interactions, and web shopping.Source: §B.3 · p. 18
- AndroidWorld evaluates online execution on 116 parameterized tasks across 20 real-world Android apps, measuring end-to-end task completion.Because predicted actions are executed, action-selection errors and improvements can propagate through subsequent trajectory states.Source: §B.3 · pp. 18–19
- GUI-Odyssey is used without adaptation for out-of-distribution evaluation on long-horizon cross-application navigation.Its trajectories span multiple apps and differ from the predominantly single-app in-distribution settings.Source: §B.3 · p. 19
B.4 EVALUATION DETAILS
- The evaluation appendix covers offline, online, out-of-distribution, and efficiency protocols and metrics.The supplied dataset discussion distinguishes the first three settings through AndroidControl and AITZ, AndroidWorld, and GUI-Odyssey, respectively.Source: §B.3 · pp. 18–19, §B.4 · p. 19
B.4.1 OFFLINE EVALUATION
Offline evaluation on AndroidControl and AITZ measures action type, click grounding, and step-level success, with success requiring the action and its argument to be correct.
B.4.1 OFFLINE EVALUATION
- Offline step-level success requires both the action type and its argument to be correct, with task-specific checks for clicks, text input, and scrolling.Click grounding is correct within a radius equal to 14% of screen width, while step-level click success requires the coordinate to fall within the target element’s bounding box.Source: §B.4.1 · p. 19
- Type measures action-type agreement, while Grounding measures click-coordinate accuracy within the stated screen-width radius.Grounding is evaluated only for click actions; all methods use the same examples and candidate-action sets for each base GUI agent.Source: §B.4.1 · p. 19
B.4.2 ONLINE EVALUATION
Online evaluation measures task success and interaction efficiency in AndroidWorld under controlled rollouts, while GUI-Odyssey tests transfer to longer-horizon, cross-application tasks without adaptation.
B.4.2 ONLINE EVALUATION
- AndroidWorld reports success rate by difficulty and overall, plus Mean Steps, Tokens/Task, and Run Time for interactive rollouts.Each of 116 task templates uses one instance with parameters sampled from fixed seed 30; methods share goals and a maximum budget of 10× task complexity.Source: §B.4.2 · p. 19
- Online rollouts reset and tear down each task to restore consistent initial conditions across methods.Source: §B.4.2 · p. 19
B.4.3 OUT-OF-DISTRIBUTION EVALUATION
- GUI-Odyssey evaluates out-of-distribution transfer on 160 held-out episodes (1,598 steps), with no training, validation, or hyperparameter selection on its data.The model is trained only on AndroidControl and AITZ and receives no GUI-Odyssey fine-tuning or adaptation.Source: §B.4.3 · p. 19
B.4.4 EFFICIENCY EVALUATION
The section defines efficiency measures across settings and gives pseudocode for training the latent transition judge and gating its use during inference.
B.4.4 EFFICIENCY EVALUATION
- Tokens/Task counts input and output tokens across agent, world-model, and scorer calls; Run Time measures end-to-end task time, while Mean Steps counts online environment interactions.Offline Tokens/Task is reported per step, and offline Run Time sums proposer, world-model, and scorer latencies; diffusion image-generation tokens are omitted.Source: §B.4.4 · p. 20
B.5 PSEUDOCODE
- Training shuffles candidate order, learns candidate-action scores and a transition embedding, and optimizes their combined loss using the current screen and a FIFO bank as negatives.The transition head predicts the next-screen embedding; the bank stores screen pairs and drops its oldest pair when it exceeds size M.Source: §B.5 · p. 20
- At inference, the judge rescales candidates only when the gate opens; otherwise, the agent’s top-ranked action is executed.The gate bypasses judging when there are too few candidates, the top action is terminal, the top two actions match, or only one candidate meets the confidence threshold.Source: §B.5 · p. 20
C ADDITIONAL QUALITATIVE RESULTS
The qualitative results contrast action choices in AndroidWorld trajectories with pixel-space world-model errors, then document the prompts used for agents, candidates, baselines, and scoring.
C.1 GUI NAVIGATION TRAJECTORIES
- In AndroidWorld episodes, the model finds and deletes recipes in 21 steps, reaches brightness in five steps, and completes a Wi-Fi toggle that Base leaves on.The figures compare the same task instance and seed; orange frames mark where the gate replaces the agent’s top-ranked candidate.Source: Figure A3 · p. 21, §C.1 · p. 21, Figure A4 · p. 22
C.2 ADDITIONAL IDENTITY-SHORTCUT EXAMPLES
- Diffusion predictions preserve the current screen despite major transitions, while Code2World produces plausible HTML that does not match the action’s real outcome.Source: §C.2 · pp. 24, 26
D PROMPTS
- The released-code prompts use runtime-filled fields, retain the unmodified M3A prompt, and add candidate-generation and judge-scoring prompts.Source: D PROMPTS · p. 28
D.1 GUI-AGENT PROMPT
- The M3A agent prompt supplies the goal, screenshot, action history, and UI details, then requests a valid next action in the specified JSON format.Source: §D.1 · p. 28
D.2 CANDIDATE-GENERATION PROMPT
- Candidate-generation prompts request three distinct single-step alternatives ordered best-first, with confidence scores.Online candidates use a text format; offline candidates use a JSON object.Source: §D.2 · p. 29
D.3 WORLD-MODEL BASELINE PROMPTS
- Baseline prompts condition Code2World on the current screenshot, goal, and action to generate resulting-state HTML, while diffusion receives action descriptions and targets.The Code2World prompt specifies action-dependent expectations for clicks, scrolling, text entry, and navigation.Source: §D.3 · p. 30
D.4 EVALUATION AND SCORING PROMPTS
- Selectors share the same scoring prompt, but only Code2World and diffusion supply rendered future screens; our judge uses blank placeholders and reads candidate scores from index-token logits.The prompt asks which candidate best advances the goal, and the logits are read without generating text.Source: §D.4 · p. 31