PaperlayerView exact revision on arXiv

Source-linked AI summary

Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents

Jiaming Zhang, Xuan Wang, Fuyao Zhang, Yang Cao, Lingjuan Lyu, Wei Yang Bryan Lim

arXiv:2610.11942v1cs.CV

TL;DR

GUI-agent safety depends on whether a screen follows the action that produced it, yet existing GUI world models render predictions in forms that require a second model to judge against observations. LGWM predicts the next screen directly in representation space, and its training-free score reaches 0.987 AUC on a diagnostic designed to defeat screen-only detectors.

  • Problem

    GUI integrity depends on the action-screen transition, but rendered world-model predictions require a second model to judge whether they match observed screens.

    Source: §1 · p. 1

  • Method

    LGWM is an action-conditioned, decoder-free predictor of the next screen’s representation, using residual magnitude to detect violations and residual direction to assess harm.

    Source: §1 · p. 2

  • Results

    0.987 AUC on RSWT-BENCH for the training-free score, with the residual head reaching 0.953 AUC at separating harmful from benign violations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Representation-space prediction makes verification a vector comparison, and its residual can also serve as a reliability monitor for missed actions, layout changes, or unexpected modals.

    Source: §1 · p. 2, §6 · p. 9

    RSWT-BENCH contains only 222 donor-paired credential-screen pairs, because each screen must also occur as a legitimate outcome of its own transition.

    Source: §3 · p. 6

Abstract

from arXiv · show

A login screen that appears after a tap on Sign in is expected; the same screen after a tap on View order is an attack. For GUI agents, safety is therefore a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated by reusing a legitimate one. Judging a transition requires an expectation of what should have followed the action. Existing GUI world models provide one, but they output it as text, code, or images, so checking it against the observed screen requires a second model to judge the two. We argue that a world model meant for verification should instead predict in the space in which observations are encoded, and present LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly, trained without semantic annotation on 1.85M real GUI transitions. Verification reduces to a vector comparison, and the same signal reveals whether a mismatch is harmful. We evaluate on RSWT-BENCH, a diagnostic where each credential screen appears under both a legitimate and a hijacked transition, so detectors that see only the screen are at chance by construction. The training-free score reaches 0.987 AUC at 17 ms per decision, on par with the strongest closed-source VLMs and about ten AUC points above generative GUI world models at over three orders of magnitude lower latency. The residual direction reaches 0.953 AUC at separating harmful from benign violations, where prompted VLMs are near chance. Further analyses show that the prediction is a usable future state rather than an anomaly score. World models have mostly served as simulators or planners; our results point to a third role, verification, for which predicting in representation space is the natural design.

1 INTRODUCTION

GUI safety depends on whether an outcome fits the action that produced it, not on the screen alone. LGWM predicts the next screen in representation space, enabling direct verification and residual-based assessment of harm.

1 INTRODUCTION

  • A pixel-identical credential screen is legitimate after Sign in but hostile after View order, making GUI integrity a property of transitions rather than screens.Third-party interfaces can redirect what happens after an action while leaving every screen plausible.Source: §1 · p. 1
  • Existing GUI world models render predictions as text, screenshots, or code, so comparing them with observed screens requires a second model to judge agreement.Rendering takes one to two minutes per transition in current generative GUI world models, before that judge is run.Source: §1 · p. 1
  • LGWM predicts the next screen’s representation from the current screen and structured action, allowing a cosine comparison without a decoder, judge, or attack labels.The prediction is trained on real transitions without semantic supervision and compared with the observed encoding in 17 ms.Source: §1 · p. 2
  • Residual magnitude detects violations of learned dynamics without training, while a linear head on residual direction separates harmful violations from benign ones.The distinction matters because push notifications or server-side layout changes can surprise a normal-dynamics model without harming anyone.Source: §1 · p. 2
  • 0.987 AUC on RSWT-BENCH puts the training-free score on par with the strongest closed-source VLMs and about ten points above generative GUI world models; the residual head reaches 0.953 AUC.The comparison with rendered GUI world models comes at more than three orders of magnitude lower latency.Source: §1 · p. 2
  • LGWM binds touch coordinates to visual patches with a shared positional basis and focuses training on action-induced changes through a change-weighted objective.Its 261M-parameter model is reported on RSWT-BENCH as on par with frontier closed-source VLMs without task training.Source: §1 · pp. 2–3

2 LGWM: PREDICTING GUI FUTURES IN REPRESENTATION SPACE

LGWM predicts GUI futures directly in the observation encoder’s representation space, conditioning on the current screen and action. Its residual supports separate readouts for whether a transition violated expectation and whether the violation is harmful.

2 LGWM: PREDICTING GUI FUTURES IN REPRESENTATION SPACE

  • LGWM uses a decoder-free world model to compare the expected future with the observed outcome, making transition integrity measurable in representation space.The design adapts a joint-embedding predictive architecture to verification rather than simulation or planning.Source: §2 · p. 3, Figure 2 · p. 4

2.1 ACTION-CONDITIONED LATENT PREDICTION

  • The screen encoder and action encoder feed a Transformer that predicts a full spatial map of the next screen’s representation.Touch coordinates and visual patches share a positional basis, and an EMA copy of the screen encoder supplies the target.Source: §2.1 · p. 3

2.2 LEARNING SPARSE GUI TRANSITIONS

  • Self-supervised training on 1.85 million transitions weights changed patches and adds anti-collapse and inverse-dynamics objectives plus short rollouts.The change mask uses no semantic labels; rollout training feeds the predictor’s own outputs back as context.Source: §2.2 · p. 4

2.3 RESIDUAL READOUTS: SURPRISE DETECTS, DIRECTION INTERPRETS

  • At runtime, residual magnitude gives a training-free consistency score, while residual direction feeds a linear head that separates harmful from benign mismatches.The magnitude flags outcomes unlike learned normal GUI dynamics; direction is used to interpret the kind of mismatch.Source: §2.3 · p. 5

3 RSWT-BENCH: RIGHT SCREEN, WRONG TRANSITION

RSWT-BENCH tests whether a detector uses transition context rather than credential-screen appearance. Donor pairing makes the outcome-only detector’s expected AUC exactly 0.5, while the benchmark’s strict construction limits it to 222 pairs.

3 RSWT-BENCH: RIGHT SCREEN, WRONG TRANSITION

  • Every outcome-only detector has expected RSWT AUC 0.5 because donor-paired examples share pixel-identical screens with opposite labels.Within each pair, equal scores yield one-half under the standard tie convention.Source: §3 · p. 5
  • RSWT-BENCH constructs hijacks by replacing a real transition’s outcome with a credential screen that also occurs as a legitimate outcome elsewhere.The test set contains 966 transitions around 222 donor pairs.Source: §3 · p. 5
  • Controls test benign mismatches and surprising benign transitions; a blind three-annotator audit agrees with constructive labels at 96.3 ± 0.8%.The secondary metrics compare benign mismatches with paired safe transitions and with legitimate credential transitions.Source: §3 · p. 5
  • The strict requirement that each credential screen also be a legitimate outcome limits the benchmark to 222 donor pairs.RSWT confidence intervals use donor-paired bootstrap resampling, with leading systems’ 95% interval half-widths near 0.01.Source: §3 · p. 6

4 EXPERIMENTS

The experiments test whether LGWM detects wrong transitions and whether its predictions represent action-conditioned future states. On RSWT-BENCH, its training-free integrity score performs comparably to leading closed-source systems, while ablations and forecast tests examine what the model learns and which components matter.

4.1 SETUP

  • The experiments use 1,845,659 transitions from five GUI datasets, with official evaluation partitions excluded; prompted baselines receive the same screen-action-screen triplet and zero-shot prompt.The model trains for 120,000 updates; 172.86M parameters execute at runtime.Source: §4.1 · p. 6

4.2 THE DECISIVE TEST: IDENTIFYING WRONG TRANSITIONS

  • 0.987 RSWT AUC: LGWM’s training-free integrity score is statistically indistinguishable from the three strongest closed-source systems and exceeds generative GUI world models at 0.885 and 0.870 AUC.Generative models require at least 92.9 seconds per transition before a separate judge; Table 1 reports the benchmark results.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • 17.1 ms per decision: the complete integrity decision costs 0.408 TFLOPs and 1.09 GB peak memory, enabling checks at every agent step.Compared with Qwen3-VL-8B, this is 27× fewer FLOPs, 16× less memory, and 64× lower latency.Source: §4.2 · p. 7
  • 0.953 Harm AUC: the residual direction separates hijacked from benign expectation violations, while the integrity score reaches 0.513 and is stronger on wrong-transition detection at 0.987 versus 0.930.The two residual readouts capture different properties: whether expectation was violated and what the violation consists of.Source: §4.2 · p. 7

4.3 THE VERIFIER IS A WORLD MODEL

  • LGWM predicts a usable, action-conditioned future rather than copying the present, replaying training outcomes, or producing only a one-step anomaly statistic.Its prediction is more similar to the future than the current screen (0.618 vs. 0.245); it ranks the true future first 68.3% of the time versus 44.6% for memorization, tracks counterfactual actions (win rates 0.891, 0.800, and 0.618), and at horizon ten reaches 0.562 similarity versus 0.455 for persistence.Source: §4.3 · p. 8

4.4 DESIGN ABLATIONS

  • Without rollout training, ten-step forecast agreement drops 10% relative, from 0.612 to 0.553, although one-step prediction remains functional.The ablations use ViT-S at 15k updates; each uses one seed and a smaller scale than the main model.Source: §4.4 · pp. 8–9
  • Without inverse dynamics, the absolute copy score falls 32%, from 0.286 to 0.193, and predictions rely more on copying the current screen.Inverse dynamics is intended to preserve which action caused a transition.Source: §4.4 · p. 8
  • Without VICReg, effective prediction-covariance rank falls 49%, from 43.7 to 22.5, leaving the integrity score unable to flag transitions when representations collapse.VICReg is the variance-covariance regularization component.Source: §4.4 · p. 9

5 RELATED WORK

Prior GUI safety monitors focus on current screens, while hidden-transition attacks can keep screens benign. LGWM instead compares an action-conditioned prediction with the observed outcome, using the residual’s magnitude and direction for detection and interpretation.

5 RELATED WORK

  • Existing GUI world models render futures as text, code, or images, so checking transitions requires a judge; rendering remains useful for human inspection.The paper describes these interfaces as complementary to prediction in representation space.Source: §5 · p. 9
  • LGWM applies latent-planner comparisons to observed outcomes rather than goal encodings, addressing GUI actions and sparse transitions that prior lineages did not need to handle.Prior latent planners compare predicted representations with goal-image encodings [Zhou et al., 2024; Assran et al., 2025].Source: §5 · p. 9
  • Prediction-error monitoring here is action-conditioned and shares the encoded observation’s space; its magnitude detects violations and its direction supports interpretation, unlike prior scalar-only uses.The paper contrasts its approach with classical exploration and out-of-distribution detection [Pathak et al., 2017; Burda et al., 2019].Source: §5 · p. 9
  • GUI safety evaluations target current-step risks, whereas hidden-transition attacks redirect outcomes while screens remain benign; RSWT-BENCH makes outcome-only detection chance-level by construction.The paper identifies RSWT-BENCH as the first GUI benchmark with this property.Source: §5 · p. 9
  • Unlike SeerGuard, LGWM’s integrity signal is geometric, post-hoc, and training-free, with harm interpretation isolated in a linear head; it checks the environment’s response, not the agent’s reasoning.SeerGuard predicts consequences before execution, renders them as text, and learns risk judgment jointly [Yu et al., 2026].Source: §5 · p. 9

6 DISCUSSION AND CONCLUSION

The discussion presents representation-space prediction as a practical verification interface for GUI agents: a fast latent gate can check transitions, while rendering remains available for human inspection. It frames verification as a third role for world models alongside simulation and planning.

6 DISCUSSION AND CONCLUSION

  • A latent gate can verify every step at 17 ms, with a renderer or VLM judge reserved for the flagged minority.The paper describes the interfaces as complementary.Source: §6 · p. 9
  • The residual also flags missed actions, layout changes, and unexpected modals, making the verifier a reliability monitor beyond attack detection.The score is not specific to attacks.Source: §6 · p. 9
  • A cosine gap from a 261M-parameter self-supervised model falls in the band set by frontier prompted-reasoning systems, suggesting the key challenge is having the right expectation.The paper presents this as evidence about transition integrity, not as a quantified measure of adversarial difficulty.Source: §6 · p. 9
  • The paper frames verification as a third world-model role for untrusted environments, alongside simulation and planning, with prediction for machines and rendering for people.The discussion says expectation-based monitoring requires an attacker to forge an outcome in the agent’s representation space, but that difficulty remains unquantified.Source: §6 · p. 9
Loading 2610.11942v1…