Computer-Using World Model

Yiming Guan; Rui Yu; John Zhang; Lu Wang; Chaoyun Zhang; Liqun Li; Bo Qiao; Si Qin; He Huang; Fangkai Yang; Pu Zhao; Lukas Wutschitz; Samuel Kessler; Huseyin A Inan; Robert Sim; Saravan Rajmohan; Qingwei Lin; Dongmei Zhang

Source-linked AI summary

Computer-Using World Model

Yiming Guan, Rui Yu, John Zhang, Lu Wang, Chaoyun Zhang, Liqun Li, Bo Qiao, Si Qin, He Huang, Fangkai Yang, Pu Zhao, Lukas Wutschitz, Samuel Kessler, Huseyin A Inan, Robert Sim, Saravan Rajmohan, Qingwei Lin, Dongmei Zhang

arXiv:2602.17365v1Feb 19, 2026cs.SE

TL;DR

Desktop agents need counterfactual reasoning because UI actions can be costly to reverse and a single mistake can derail artifact-preserving workflows. CUWM predicts textual action-induced changes and renders their visual consequences, then uses these simulations for test-time action search. Across Microsoft Office tasks, the approach improves decision quality and execution robustness, although combining text and image predictions often degrades performance.

Problem
Desktop software is deterministic but real UI execution is costly and unsafe to reverse, limiting counterfactual exploration for computer-using agents.
Method
CUWM factorizes UI dynamics into textual transition prediction followed by visual state realization, trains on offline Office transitions, and uses the model for test-time action search.
Results
Across Microsoft Office tasks, world-model-guided test-time simulation improves decision quality and execution robustness without modifying agent policies.
Takeaways & Limitations
CUWM enables safer planning and more reliable automation by simulating UI consequences before agents interact with live software.
Takeaways & Limitations
Combining textual and image predictions degraded agent performance across most configurations, likely reflecting cross-modal conflict or accumulated prediction noise.

Abstract

from arXiv · show

Agents operating in complex software environments benefit from reasoning about the consequences of their actions, as even a single incorrect user interface (UI) operation can derail long, artifact-preserving workflows. This challenge is particularly acute for computer-using scenarios, where real execution does not support counterfactual exploration, making large-scale trial-and-error learning and planning impractical despite the environment being fully digital and deterministic. We introduce the Computer-Using World Model (CUWM), a world model for desktop software that predicts the next UI state given the current state and a candidate action. CUWM adopts a two-stage factorization of UI dynamics: it first predicts a textual description of agent-relevant state changes, and then realizes these changes visually to synthesize the next screenshot. CUWM is trained on offline UI transitions collected from agents interacting with real Microsoft Office applications, and further refined with a lightweight reinforcement learning stage that aligns textual transition predictions with the structural requirements of computer-using environments. We evaluate CUWM via test-time action search, where a frozen agent uses the world model to simulate and compare candidate actions before execution. Across a range of Office tasks, world-model-guided test-time scaling improves decision quality and execution robustness.

1 INTRODUCTION

Computer-using agents need counterfactual reasoning because desktop actions are costly to reverse and can damage long-lived artifacts. CUWM addresses this need with a two-stage world model for Microsoft Office that supports test-time simulation and improves reliable decision-making.

Desktop UI actions are costly and difficult to reverse, so a single mistake can corrupt artifacts or derail long workflows.Although software is deterministic, latency, limited context-dependent undo, and irreversible edits make real rollouts unsafe for counterfactual exploration.
CUWM models real desktop software by predicting action-induced textual UI changes and then rendering the next screenshot.The separation focuses modeling on structurally salient transitions while retaining pixel-level state generation.
The model is trained on offline transitions from agents interacting with Microsoft Office applications and refined with reinforcement learning aligned to UI structure.The refinement encourages concise transition descriptions relevant to software interfaces.
Test-time action search uses CUWM to simulate candidate actions before a frozen agent executes one, adding computation without further training or risky exploration.The evaluation measures both agent performance and the fidelity of predicted transitions and screenshots.
CUWM is presented as the first computer-using world model explicitly modeling UI transitions for test-time planning in productivity applications.The demonstrated domain includes Word, Excel, and PowerPoint.
Test-time simulation of UI consequences can substantially improve reliable decision-making in software systems.

2 RELATED WORK

Prior world-model research spans latent dynamics, textual or semantic transitions, and visual prediction, but these approaches have mainly targeted games, robotics, web agents, or mobile environments.

Implicit World Models: Implicit world models encode dynamics in latent representations for planning and value prediction rather than explicit state reconstruction.Examples include World Models, PlaNet, Dreamer, and MuZero, which have been effective in games and robotics.
Implicit World Models: Latent models are not designed for interpretable UI semantics, limiting their applicability to computer-using agents.
Textual and Semantic World Models: Recent explicit textual or semantic world models primarily support web and mobile agents, while related work also studies implicit language-model world modeling.

3 METHOD

CUWM models sparse desktop UI dynamics by separating semantic transition prediction from visual realization. It is trained on offline application trajectories, refined for structurally important descriptions, and used to simulate candidate actions before execution.

3.1 TWO-STAGE WORLD MODEL ARCHITECTURE: CUWM first predicts a structured textual description of the localized, decision-relevant change induced by an action.Stage 1 uses the current screenshot, textual context, and action to produce a transition description.
3.1 TWO-STAGE WORLD MODEL ARCHITECTURE: CUWM then conditions visual synthesis on the current UI and predicted transition to render the next screenshot while preserving unchanged regions.Stage 2 delegates image realization to a diffusion-based image-editing model.
3.1 TWO-STAGE WORLD MODEL ARCHITECTURE: Desktop UI transitions are typically localized, compositional, and causally aligned with the triggering action, making monolithic pixel prediction inefficient.Most interface regions remain unchanged while small updates such as selections, dialogs, or cursor movements matter for decisions.
Training: Supervised learning on GUI-360 trajectories grounds both stages in real Office behavior and initializes the model for subsequent refinement.The trajectories contain screenshot states and text-described GUI/API actions, while GPT-5 supplies concise transition annotations.
Training: Reinforcement learning refines textual transitions using an LLM judge and length penalty targeting correctness across UI structural aspects and concise descriptions.The judge evaluates features such as ribbon state, editing area, and side panes; the penalty discourages unsupported or noisy changes.
3.4 WORLD-MODEL-GUIDED TEST-TIME ACTION SEARCH: World-model-guided action search simulates each candidate action with CUWM, then lets an unchanged agent select one based on predicted outcomes.This think-then-act procedure adds test-time computation without modifying the agent policy.

4 EXPERIMENTS

CUWM is evaluated for transition fidelity, visual realization, and downstream agent performance across Microsoft Office applications. Its textual and visual components improve key evaluation outcomes, while combining text and image predictions can reduce agent performance.

Evaluation setup: Evaluation covers world-model fidelity and agent-level performance across Word, Excel, and PowerPoint using test-time action search.The study assesses textual transitions, screenshots, and task completion across multiple agent backbones and model configurations.
Textual state transitions: Textual transition scores improve from Base to SFT and further to SFT+RL under the LLM-as-a-Judge evaluation.The judge scores consistency across application state, executed actions, and major UI components.
Textual state transitions: SFT+RL achieves the highest Action Consistency Score across GPT-4.1-mini and Gemini-2.0-Flash, and this improvement translates into downstream agent gains.Action consistency measures agreement between actions selected from the real UI screenshot and the predicted textual transition.
Visual state realization: CUWM achieves the best visual-realization performance across Word, Excel, and PowerPoint, including improved preservation of rendered UI text.Text Perception Score generally increases over training epochs, and joint fine-tuning improves text preservation during UI transitions.
Agent performance: 4% for GPT-4o and 8% for Qwen3-VL-8B are the task-completion gains from CUWM with image-only input, which outperforms text-based and image-generation baselines across agents.The comparisons include None, textual-only, visual-only, combined CUWM configurations, Qwen-Image-Edit-2509, and GPT-Image-1.5.
Agent performance: Combining text and image predictions degrades agent performance across most configurations, potentially because of cross-modal conflict or accumulated prediction noise.The paper identifies this as a limitation of current VLMs’ integrated multimodal reasoning capacity.

5 CONCLUSION

CUWM is a two-stage world model for desktop productivity software that supports test-time action search. Across Microsoft Office tasks, it improves decision quality and execution robustness without modifying agent policies.

CUWM factorizes UI dynamics into textual transition prediction followed by visual state realization for desktop productivity software.The design captures structurally salient UI changes while remaining compatible with pixel-level agent interaction and offline training with lightweight refinement.
Across a range of Microsoft Office tasks, CUWM serves as an effective test-time simulator for computer-using agents.World-model-guided action search improves decision quality and execution robustness without modifying agent policies.
These gains hold even in deterministic software environments, highlighting the value of test-time simulation for reliable computer use.

A.1 DATASET SUMMARIZATION

The CUWM dataset is built from GUI-360 trajectories spanning Word, Excel, and PowerPoint. Samples pair current screenshots, actions, and subsequent screenshots, with successful trajectories selected for training and evaluation.

GUI-360 supplies frame-by-frame screenshots, action commands, and task completion signals across Word, Excel, and PowerPoint trajectories.
Each sample pairs a current UI screenshot with its executed action and subsequent UI screenshot as (st, at, st+1).
2,876 samples were collected for training and 339 for evaluation from successful, continuous trajectories with complete standardized action annotations.The dataset is described as an initial collection that can be scaled up in future work.

A.2.1 OVERVIEW OF THE TRAINING PIPELINE

CUWM predicts UI dynamics in two stages: textual transition prediction from the current screenshot and action, followed by visual synthesis of the next screenshot. The stages are trained sequentially with supervised initialization before later refinement.

Stage 1 predicts a concise textual transition ∆t from the current UI screenshot st and action at.Stage 2 synthesizes the next screenshot ŝt+1 from the current UI and predicted transition.
The visual realization model conditions next-state synthesis on the current UI and the predicted textual transition.
Both stages receive supervised fine-tuning initialization before subsequent training refinement.

A.2.2 STAGE 1: QWEN2.5-VL SUPERVISED TRAINING (TEXTUAL TRANSITIONS)

Stage 1 uses Qwen2.5-VL to generate textual descriptions of action-induced UI transitions from screenshots and natural-language actions. Training applies autoregressive cross-entropy with LoRA fine-tuning.

Qwen2.5-VL maps the current UI screenshot st and natural-language action at to a textual transition description ∆t.The model is trained to generate the target transition text ∆GTt.
Stage 1 optimizes autoregressive cross-entropy loss, LSFT = −log p(∆GTt | st, at), on the target transition text.

A.2.3 STAGE 2: QWEN-IMAGE-EDIT SUPERVISED TRAINING (VISUAL REALIZATION)

Stage 2 uses Qwen-Image-Edit to generate the next UI screenshot from the current screenshot and predicted textual transition. Supervised training uses pixel-wise reconstruction against the ground-truth next-state screenshot, with LoRA applied to the DiT backbone.

Visual state realization: Qwen-Image-Edit generates the next UI screenshot from the current screenshot and predicted textual transition.The model is conditioned on both inputs through ˆst+1 = fimage(st, ∆t).
Supervised training: The visual realization model is fine-tuned with a pixel-wise mean squared error reconstruction objective.The loss compares the generated screenshot with the ground-truth next-state screenshot.
Supervised training: LoRA fine-tuning is applied only to Qwen-Image-Edit’s DiT backbone.The Stage 2 hyperparameters are summarized in Table 8.

A.2.4 STAGE 1 RL REFINEMENT: QWEN2.5-VL + GRPO

The Stage 1 textual transition model is refined with GRPO to favor semantically faithful, concise descriptions, and evaluation examines resulting visual fidelity, action consistency, and downstream task accuracy.

RL refinement: GRPO refines the SFT-initialized model because faithful initialization does not ensure that textual transitions capture decision-critical UI structures.The model generates transition descriptions conditioned on the current screenshot and action.
RL refinement: GRPO samples K = 5 candidate descriptions and optimizes their relative preference without requiring a separate value network.Sampling uses temperature 1.0 and top-p 1.0.
RL refinement: The composite reward combines an LLM-as-a-Judge semantic consistency score with a soft length penalty controlled by β.The judge evaluates predefined UI structural aspects, while the length penalty discourages overly long or short descriptions.
Training curves: PSNR and SSIM increase while LPIPS and FID decrease across training, and Text Perception Score rises steadily across applications.Word achieves the highest scores, followed by Excel, while PowerPoint is the most challenging domain.
Evaluation: Action Consistency Score measures whether textual transitions preserve UI information needed for an agent to choose the same action as with the actual screenshot.The protocol compares a frozen policy’s oracle action from visual context with its action based only on the world model’s textual output.
Downstream evaluation: RL-refined CUWM achieves higher downstream accuracy than SFT on CUWM-generated images for both GPT-4.1-mini and Gemini-2.0-Flash agents.The reported accuracies are 0.4317 and 0.4700, respectively; approximately 35% of evaluation samples lack the ground-truth action in the candidate set.