PaperlayerView exact revision on arXiv

Source-linked AI summary

Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training

Yongxin Ning, Runliang Niu, Qianli Xing, Zhiyi Duan, Qingzu He, Pan Wang, Qi Wang

arXiv:2610.05861v1cs.CVcs.AI

TL;DR

GUI-agent training needs massive, high-fidelity visual-action trajectories, but they are difficult to acquire and existing text- or HTML-based world models omit pixel-level details. Infinite-Dreamer uses VLM-described delta-text to condition an image-editing model that synthesizes screenshot transitions and trajectories. Infinite-Actor improves benchmark performance, including AndroidWorld Pass@1 gains across scales and nearly doubled MobileWorld Pass@3 success.

  • Problem

    GUI agents require massive, high-fidelity visual-action trajectories, but human demonstrations are difficult to scale and text- or HTML-based world models omit pixel-level interface details.

    Source: §1 · p. 1

  • Method

    Infinite-Dreamer uses VLM-generated structured delta-text to condition a fine-tuned image-editing model that synthesizes screenshot transitions, robustness data, and multi-step trajectories.

    Source: §1 · p. 2

  • Results

    Infinite-Actor improves AndroidWorld Pass@1 across all three model scales by up to +9.20 points and nearly doubles MobileWorld Pass@3 success.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper reports that synthetic data increases trajectory utility and helps agents generalize to unseen UI layouts and interaction patterns.

    Source: §1 · p. 3

    Synthesized data inherits AndroidControl's application coverage and language mix, so agents trained on it may generalize unevenly across regions and applications.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Graphical User Interface (GUI) agents have emerged as a promising paradigm for automating complex digital workflows across diverse applications. However, training highly capable and generalizable agents fundamentally relies on massive, high-fidelity visual-action trajectories, which are notoriously difficult to acquire. While human demonstrations are unscalable, existing GUI world models rely on text descriptions or HTML rendering, discarding crucial pixel-level visual details like icons and layout styles. To address this issue, we introduce Infinite-Dreamer, a simulation-free data synthesis method powered by a pixel-level Image Editing World Model. By conceptualizing GUI transitions as image editing tasks, we leverage Vision-Language Models (VLMs) to describe action-induced UI changes as structured delta-text. We then fine-tune an image editing backbone to controllably synthesize realistic screenshot transitions. We utilize this model to generate both single-frame visual robustness data and multi-step imaginary trajectories. To validate the effectiveness of our approach, we fine-tune the Qwen3-VL baseline solely on the synthesized data to obtain Infinite-Actor, and evaluate it on AndroidWorld, MobileWorld, and AndroidControl-Curated benchmarks. Infinite-Actor consistently outperforms the Qwen3-VL baselines across scales: Infinite-Actor-8B improves AndroidWorld Pass@1 by +4.45 and nearly doubles the MobileWorld Pass@3 success rate, while Infinite-Actor-2B improves Pass@1 by +9.05. Code is available at https://github.com/swaydy-n/Infinite-Dreamer.

1 INTRODUCTION

GUI agents need massive, high-fidelity visual-action data, but acquiring coherent trajectories is difficult and existing text- or HTML-based world models miss pixel-level interface detail. Infinite-Dreamer uses image editing to synthesize screenshot transitions and trajectories; the paper reports downstream gains in generalization and benchmark performance.

1 INTRODUCTION

  • Manual demonstrations are costly and difficult to scale, while automated exploration can produce short, repetitive, disjointed sequences that fail to sustain complex workflows.The authors identify scalable synthesis of coherent trajectories as a core training challenge.Source: §1 · pp. 1–2
  • Text- and HTML-based GUI world models struggle to represent fine-grained visual features such as icons, banners, and dynamic animations.The paper motivates pixel-level screenshots as a more direct representation of the visual state users see.Source: §1 · p. 2
  • Infinite-Dreamer fine-tunes an image-editing model to transform pre-action screenshots into post-action screenshots conditioned on VLM-generated delta-text.The delta-text describes UI changes; the pipeline uses the model for both single-screenshot robustness data and multi-step imaginary trajectories.Source: §1 · p. 2
  • Blending synthetic and real-world data improves generalization to unseen UI layouts and interaction patterns over agents trained purely on real demonstrations.The introduction presents this as increased utility from existing trajectories.Source: §1 · p. 3
  • Infinite-Actor improves AndroidWorld Pass@1 across all three model scales, by up to +9.20 points, and nearly doubles MobileWorld Pass@3 success.The paper also reports that Infinite-Actor is trained using newly synthesized data.Source: §1 · p. 3

2 RELATED WORK

Prior GUI-agent data work collects trajectories through environment exploration, reuses human-created artifacts, or trains agents in synthetic environments. Related world models and image editors provide visual simulation or editing capabilities, but the paper distinguishes screenshot editing from those approaches.

2 RELATED WORK

  • GUI-agent data methods include active environment exploration, harvesting tutorials or videos, and training in synthetic environments with verifiable rewards.Examples include reasoning-guided exploration, tutorial replay, video mining, and auto-synthesized apps.Source: §2 · p. 3
  • GUI world models use text or HTML rendering, train visual simulators from scratch, or apply sketch-based abstractions and recurrent state tracking.The paper contrasts these lines with its direct editing of screenshots to synthesize pixel-level trajectories.Source: §2 · p. 3
  • Image-editing models modify reference images under text conditioning, but the paper says they lack procedural knowledge of app operation needed to serve as genuine GUI world models.The cited examples include FLUX.1, FLUX.2, and Qwen-Image-Edit.Source: §2 · p. 3

3 METHODOLOGY

Infinite-Dreamer represents GUI transitions with structured delta-text and uses an image-editing world model to generate visual variants and imagined trajectories for GUI-agent training.

3.1 PROBLEM FORMULATION

  • Because action labels lack visual specifications, the formulation factors transition prediction into delta-text generation and image editing conditioned on the current screenshot and delta-text.Source: §3.1 · p. 4

3.2 DELTA-TEXT EXTRACTION

  • A VLM extractor turns real screenshot-action transitions into delta-text recording the trigger, added or removed UI elements, and optional next-step affordances; synthetic text descriptions later provide the same interface.Source: §3.2 · pp. 4–5

3.3 IMAGE-EDITING WORLD MODEL TRAINING

  • The editor is fine-tuned on paired screenshots with the current screen as source and delta-text as instruction; coordinate-free diffs require it to infer layout reflow, occlusions, and persistence in untouched regions.Source: §3.3 · p. 5

3.4 SYNTHESIS REGIMES

  • Single-frame augmentation changes screenshot appearance while approximately preserving layout and displayed text, keeping the paired task instruction valid.Source: §3.4 · p. 5
  • An App Designer plans multi-step tasks and per-step UI descriptions; a text-only extractor authors delta-texts, and the world model rolls out screenshots autoregressively from a real seed.Source: §3.4 · p. 5
  • A VLM scores each synthesized step by confidence in the intended next-action function call, and steps below threshold η are discarded from the trajectory.Source: §3.4 · p. 5

3.5 GUI AGENT TRAINING

  • The GUI agent is fine-tuned with a supervised objective over selected real, augmented, and imagined data sources; Infinite-Actor is trained solely on synthesis data and paired with same-scale Qwen3-VL.Source: §3.5 · p. 5, Table 1 · p. 6

4 EXPERIMENT

Experiments assess world-model synthesis quality and the downstream GUI agent on online task-success and offline step-level benchmarks. They also examine data-source contributions, task difficulty, rollout length, and delta-text conditioning.

4.1 EXPERIMENTAL SETUP

  • The world-model evaluation uses AndroidControl validation, while Infinite-Actor is trained solely on synthesized data and tested on online and offline benchmarks.The online benchmarks measure task success; AC-Curated measures step-level action and grounding performance.Source: §4.1 · p. 6

4.2 WORLD MODEL QUALITY

  • Infinite-Dreamer-9B leads on all four Table 2 synthesis metrics, while its 4B variant surpasses the unfine-tuned FLUX.2-Klein-4B backbone.The metrics are Accuracy, Completeness, Contextual alignment, and Visual preservation; higher is better.Source: Table 2 · p. 7, §4.2 · p. 7
  • At 10 steps, Infinite-Dreamer-9B scores 48.84, versus 40.96 for FireRed-Image-Edit-1.0 and 29.61 for unfine-tuned FLUX.2-Klein-9B.Figure 4 shows a representative six-step rollout that the caption describes as preserving layout, text fidelity, and visual coherence.Source: §4.2 · p. 7, Figure 4 · p. 9

4.3 GUI AGENT PERFORMANCE

  • Infinite-Actor-8B reaches 43.82 Pass@1 and 62.07 Pass@3 on AndroidWorld, versus Qwen3-VL-8B’s 39.37 and 56.90.On MobileWorld, Infinite-Actor-8B reaches 21.93 Pass@3 versus 11.40 for Qwen3-VL-8B; Infinite-Actor-2B improves over Qwen3-VL-2B by +9.05 Pass@1 and +19.83 Pass@3.Source: §4.3 · pp. 7–8

4.4 ABLATION STUDY

  • Combining real, imagined, and augmented data yields 48.28 AndroidWorld Pass@1, versus 42.67 with real data alone.The analysis assigns augmentation a robustness role for surface-level UI variation and multi-step imagination a role in expanding state-transition coverage.Source: Table 3 · p. 8, §4.4 · p. 8
  • Removing delta-text drops synthesis Accuracy from 68.13 to 21.56, while text-only rollout specifications can omit visual changes and compound distortions.The training delta-texts come from real image pairs, but rollout delta-texts are authored text-only; a quality filter prunes some such steps before training.Source: Table 4 · p. 8, §4.4 · p. 9

4.3 GUI AGENT PERFORMANCE

  • On AC-Curated, Infinite-Actor-8B action-type accuracy is 73.30 on Hard and 82.40 on Easy, but 4B Hard Box Acc falls from 77.00 to 64.70.The reported coordinate displacement is larger on Hard than Easy, and offline Box Acc requires the predicted center to fall inside the annotation.Source: §4.3 · p. 8

4.4 ABLATION STUDY

  • Mixed training improves Easy (+4.9) and Medium (+8.3) AndroidWorld tasks but trails real-4B by 2.7 points on Hard (10.5 versus 13.2).Successful mixed-model episodes extend past 50 steps, while real-4B stops near 30 and base-4B near 25; Figure 3 covers task difficulty and episode-length analyses.Source: Figure 3 · p. 7, §4.4 · pp. 8–9

5 CONCLUSION

Infinite-Dreamer uses an image-editing world model to synthesize screenshot transitions and multi-step rollouts for GUI-agent training. Experiments report improved generalization and task-level gains, alongside risks tied to source-data coverage and pixel-level dual use.

5 CONCLUSION

  • The framework conditions screenshot-transition generation and multi-step rollouts on structured delta-text.The paper presents Infinite-Dreamer as a simulation-free data-synthesis framework using an image-editing world model.Source: §5 · p. 9
  • Experiments report that Infinite-Dreamer synthesis enhances agent generalization, with large task-level gains on AndroidWorld and MobileWorld.The conclusion also describes a broader direction in which world models expand state coverage while agents identify transitions relevant to long-horizon competence.Source: §5 · pp. 9–10
  • Synthesized data inherits AndroidControl's application and language distribution, and pixel-level GUI transition prediction is dual-use.The study restricts evaluation to standard mobile-assistant benchmarks and releases screenshots and generation scripts, but not a device-control harness.Source: ETHICS STATEMENT · p. 10

A EXPERIMENTAL DETAILS

A.1 WORLD MODEL TRAINING SETTINGS

  • The image-editing world model uses FLUX.2-Klein 4B and 9B backbones, both fine-tuned on paired UI screenshots with delta-text conditioning; all training configurations are summarized in Table 5.Source: §A.1 · p. 13

A.3 GUI AGENT TRAINING SETTINGS

  • Infinite-Actor is fine-tuned from Qwen3-VL 2B, 4B, and 8B checkpoints solely on single-frame augmentations and quality-filtered multi-step imagination trajectories.The training uses no real trajectories, and its configurations are summarized in Table 8.Source: §A.3 · p. 13

A.4 REPRODUCIBILITY DETAILS

The appendix documents reproducibility choices, the composition of the synthesized dataset, evaluation protocols, and baseline implementations. It also summarizes task-level benchmark comparisons between Infinite-Actor and zero-shot Qwen3-VL.

A.4 REPRODUCIBILITY DETAILS

  • Experiments use one hyperparameter setting per model, selected by preliminary AndroidControl validation, a fixed seed of 42, and one training run per configuration.Pass@3 uses three independent inference attempts; world-model and agent configurations are provided in Tables 5 and 8.Source: §A.4 · p. 13, Table 5 · p. 14, Table 8 · p. 15

A.5 SYNTHESIZED DATA

  • The Infinite-Dreamer synthesis set contains appearance-only screenshot augmentations and quality-filtered, autoregressive multi-step imagination trajectories.Samples are stored as screenshot, task-instruction, and mobile-action triples at 672 × 1536 resolution; the subsets are regenerated deterministically from the documented prompts and configurations.Source: §A.5 · pp. 14–15

A.6 EVALUATION PROTOCOLS

  • AndroidWorld and MobileWorld use Pass@1 and Pass@3 task-success rates, while AndroidControl-Curated uses action-type accuracy, coordinate-grounding accuracy, and step-level success rate.Pass@3 counts a task as successful if any of three independent attempts completes it; coordinate accuracy is evaluated on coordinate-requiring steps.Source: §A.6 · pp. 15–16
  • Infinite-Actor models consistently outperform zero-shot Qwen3-VL counterparts on task-level benchmarks across 2B, 4B, and 8B scales.Figure 5 summarizes six metrics: AndroidWorld Pass@1 and Pass@3, MobileWorld Pass@1 and Pass@3, and AndroidControl-Curated Easy and Hard Step SR.Source: Figure 5 · p. 16

A.7 BASELINE IMPLEMENTATIONS

  • Baseline implementations use official APIs for proprietary models, public checkpoints and a shared prompt for open-source VLMs, and official default settings for image-editing models.The image-editing baselines use the same delta-text prompts as Infinite-Dreamer.Source: §A.7 · p. 16

A.8 ADDITIONAL EXPERIMENTAL RESULTS

The additional experiments show task-level advantages for Infinite-Actor and a qualitative benefit from delta-text conditioning. The section also distinguishes the image-editing model from a conventional renderer through spatial inference and preservation of unaffected screen regions.

A.8 ADDITIONAL EXPERIMENTAL RESULTS

  • Infinite-Actor consistently outperforms zero-shot Qwen3-VL on task-level benchmarks across model scales, while delta-text conditioning yields a transition closer to the target UI.Figure 5 summarizes six evaluation metrics; Figure 6 contrasts the delta-text-conditioned output with a variant that produces an unrelated screenshot.Source: §A.8 · p. 16, Figure 6 · p. 17

A.9 CONCEPTUAL DISENTANGLEMENT: WORLD MODEL VS. CONDITIONAL GUI RENDERER

  • Delta-text specifies semantic intent rather than exact coordinates, so the image-editing model infers element placement, typography, reflow, and background dimming from visual context.The paper contrasts this with conventional renderers that require explicit specifications for visual elements.Source: §A.9 · p. 17
  • The model updates action-relevant screen regions while preserving unrelated interface areas, such as status bars, navigation icons, and background cards.This is presented as state persistence and causal invariance, unlike rendering the full interface anew.Source: §A.9 · p. 17
  • The framework separates symbolic transition dynamics from visual dynamics, assigning task-level intent and screenshot realization to distinct components.The paper describes this hierarchical formulation as separating high-level task logic from low-level pixel synthesis.Source: §A.9 · p. 17

B GUI AGENT ARCHITECTURE

This section describes the screenshot-based GUI agent loop and the delta-text templates used to support synthesis from screenshot transitions. It covers single-frame edits and multi-step GUI imagination, including trajectory planning, filtering, and synthesis-quality evaluation.

B GUI AGENT ARCHITECTURE

  • The agent observes screenshots, selects and executes one action at a time, and appends each resulting screenshot with updated task progress for the next step.It interacts with Android through ADB; no separate outcome-summarization phase is required.Source: B GUI AGENT ARCHITECTURE · p. 18

B.1 ACTION SPACE DEFINITION

  • The action space defines 15 touchscreen action types, while deployment uses eight compact tool-call actions that rename or merge several raw actions.The deployment vocabulary includes click, long press, swipe, type, answer, system button, wait, and terminate.Source: §B.1 · p. 18, §B.3 · p. 19

B.2 ACTION-SELECTION DECISION LOOP

  • At each step, the vision-language model receives the screenshot, goal, and task-progress string, then returns a Thought, Action description, and JSON tool call.Coordinates are normalized from the model’s input resolution to the device’s native resolution before execution via ADB; the agent waits for UI stabilization and updates the trajectory.Source: §B.2 · pp. 18–19

B.3 ACTION-SELECTION PROMPT TEMPLATE

  • The action-selection prompt requires output in Thought → Action → tool-call order and supplies the user query, task progress, and current screenshot.It specifies the eight-action mobile-use interface and instructs the agent to use terminate when finishing.Source: §B.3 · pp. 19, 21

C DELTA-TEXT EXTRACTION TEMPLATES

  • Two delta-text templates share fields for the trigger operation, added and reduced elements, and elements needed for the next operation, but differ in inputs and UI context.The image-in template supports world-model training; the text-in template supports imagined agent training data.Source: C DELTA-TEXT EXTRACTION TEMPLATES · p. 21

C.1 IMAGE-IN EXTRACTION FOR WORLD-MODEL TRAINING

  • The image-in template uses consecutive screenshots and current and next actions to produce six fields, including current- and next-page elements, for pairing with screenshots in world-model training.Its fields also describe the trigger, added and reduced elements, and UI elements required for the next operation.Source: §C.1 · p. 21

C.2 TEXT-IN EXTRACTION FOR TRAJECTORY IMAGINATION

  • The text-in template uses textual UI-element descriptions to extract the four core delta fields, producing delta-texts that condition world-model rollouts for multi-step trajectory imagination.The descriptions are designed by the App Designer and used to construct synthesized agent training data.Source: §C.2 · pp. 23–24
  • The trajectory-imagination prompt asks for the first operation’s trigger, added and reduced UI elements, and elements and coordinates needed for the next operation.It specifies normalized screen coordinates ranging from 0 to 1, with the upper-left as origin and coordinates rounded to four decimal places.Source: §C.2 · pp. 25–26

D.1 SINGLE-FRAME AUGMENTATION PROMPT

  • Single-frame augmentation generates 3–5 appearance-only edits that preserve layout and text content, changing visual elements such as colors, icons, controls, backgrounds, or themes.Because the edits change only the visual skin, the paired task instruction remains valid without modification.Source: §D.1 · p. 26

D.2 MULTI-STEP IMAGINATION (APP DESIGNER) PROMPT

  • Given a task, difficulty, target step count, and initial UI elements, the App Designer imagines coherent GUI pages with per-step elements, goals, and operations, then ends with a status.Delta-texts for world-model rollouts are extracted from consecutive designed UI-element descriptions.Source: §D.2 · p. 28
  • The app-designer prompt requests coherent GUI pages and stepwise task plans that describe UI elements, goals, and operations in a structured format ending with a completion status.Source: §D.2 · pp. 30–31

E QUALITY FILTERING DETAILS

  • A VLM retains actionable synthesized trajectories by predicting the reference action type and confidence from task context and screenshot, dropping steps below η = 90 and their trajectories.Source: E QUALITY FILTERING DETAILS · p. 31

E.1 SYNTHESIS-QUALITY EVALUATION TEMPLATE

  • GLM-4.7v scores each generated transition against before and ground-truth after screenshots and the editing instruction for accuracy, completeness, contextual alignment, and visual preservation on a 0–10 scale.Source: §E.1 · p. 34
  • The template evaluates completeness and emulator contextual understanding on 0–10 scales, with 10 indicating full completeness or perfect contextual understanding.It also calls for visual-consistency scoring of the “after” screenshot on a 0–10 scale, with 10 indicating perfect consistency.Source: §E.1 · p. 35
  • The requested JSON output contains reasoned analysis and metric fields for accuracy, completeness, contextual understanding, and visual consistency.The format specifies the four metric names but the supplied passages do not provide their scoring instructions in full.Source: §E.1 · p. 35

E.2 ERROR PROPAGATION DYNAMICS IN RECURSIVE ROLLOUTS

Recursive rollout quality declines with horizon, and accumulated visual and specification errors explain why later generated states become less reliable. Qualitative comparisons show Infinite-Dreamer edits better preserve intended UI changes and layout than the listed baselines.

E.2 ERROR PROPAGATION DYNAMICS IN RECURSIVE ROLLOUTS

  • 70.71 at one step falls to 48.84 at ten steps, showing progressive rollout-quality decline over extended horizons.The section formalizes this error compounding and analyzes extraction-template shift and quality filtering.Source: §E.2 · p. 35

1. Semantic Specification Shift (esem

  • Underspecified text-only transition descriptions, unlike image-pair-derived training delta-text, can omit secondary effects and distort generated frames.Examples include background dimming and keyboard dismissal.Source: §1 · p. 36

2. Visual Memory Amplification (evis

  • Once a generated canvas leaves the clean-training screenshot manifold, visual errors feed into later steps and can be amplified by the world model.The model can misread font blurring or boundary shifts as intentional UI structures.Source: §2 · p. 36
  • 9.3% cumulative survival at step 10, down from 84.6% at step 1, while per-step validity stays within 78.4–84.9%.The filter removes trajectories when any step fails; the at-risk set also shrinks as tasks terminate.Source: Table 11 · p. 36, §2 · p. 36

F.1 SINGLE-STEP EDITING COMPARISON

  • In three single-step examples, Infinite-Dreamer-9B more accurately follows edit instructions while preserving UI structure and unmodified regions than the listed editors.The examples span a plant-care screen, recipe-filter page, and schedule-meeting form.Source: §F.1 · p. 37, Figure 7 · p. 37

F.2 MULTI-STEP ROLLOUT COMPARISON

In five-step LeafSnap rollouts, Infinite-Dreamer tracks intended UI states more faithfully than the compared image-editing baselines. Its examples retain UI text and reminder state while baselines accumulate visual errors.

F.2 MULTI-STEP ROLLOUT COMPARISON

  • Across five LeafSnap steps, Infinite-Dreamer tracks ground-truth UI transitions, while baselines accumulate errors.The comparisons include Infinite-Dreamer-4B and -9B alongside general-purpose and image-editing baselines.Source: §F.2 · p. 37, Figure 8 · p. 38
  • Infinite-Dreamer preserves plant names, UI text, and the toggled reminder state; baselines show corrupted text, missing or duplicated elements, and inconsistent backgrounds.The section also presents additional visualizations of single-frame augmentation and multi-step imagination.Source: §F.2 · pp. 37–38, G MORE VISUALIZATIONS · p. 38

G.1 SINGLE-FRAME AUGMENTATION RESULTS

Single-frame augmentation produces visually varied screenshot variants while preserving source layout and text. The examples span app categories and appearance changes, and require no new task labels.

G.1 SINGLE-FRAME AUGMENTATION RESULTS

  • Augmented screenshots vary themes, colors, backgrounds, icons, and keyboard styling while reproducing every source button, list item, and text label verbatim.The examples span recipe editing, dialing, display settings, file management, scheduling, messaging, route planning, and battery settings.Source: §G.1 · pp. 38–39
  • These augmentations improve robustness to surface-level UI variation without requiring new task labels.Source: §G.1 · p. 39

G.2 MULTI-STEP IMAGINATION RESULTS

Multi-step imagination generates complete rollouts from seed screenshots by conditioning each step on the preceding screenshot and a delta-text action label. The examples show dialogs, text entry, and modal dismissal while retaining surrounding UI chrome.

G.2 MULTI-STEP IMAGINATION RESULTS

  • Three independent imagined rollouts begin from initial states and unfold over five steps, with each transition conditioned on the preceding screenshot and a delta-text action label.Example actions include creating a folder, entering text, and sending.Source: §G.2 · p. 39
  • A five-step comparison shows Infinite-Dreamer-9B closely following the ground-truth layout and text throughout the trajectory.Source: Figure 8 · p. 40
  • Across the examples, the model opens dialogs, types into text fields, and dismisses modals while preserving surrounding UI chrome.Source: §G.2 · p. 40

G.3 DETAILED FAILURE MODE ANALYSIS, DIAGNOSTICS, AND MITIGATIONS

Infinite-Actor improves end-to-end task success but shows coordinate-grounding regressions and a Hard-task regression when synthetic data is mixed with real demonstrations. Hard-task failures include repeated or ineffective actions and false completion; proposed explanations remain unconfirmed.

Detailed Failure Mode Analysis

  • AndroidWorld Hard performance regresses by 2.7 points when synthetic data is mixed with real demonstrations, alongside coordinate-grounding regressions on AC-Curated.The analysis identifies these as anomalies warranting further investigation.Source: §G.3 · p. 40

Measured Failure Outcomes on AndroidWorld Hard Tasks

  • Six of 17 failed episodes include at least three repeated actions, seven include an action followed by an unchanged screenshot, and three include clicks with no screen change.Repetition or oscillation rises to 35.3% in Hard failures, compared with 19.0% in Easy and 21.7% in Medium; false completion rises from 14.3% and 30.4% to 35.3%.Source: §G.3 · pp. 40, 43

Single-frame augmentation examples

  • Figure 9 shows visually distinct theme, color, and icon or keyboard variants while widget positions and text remain fixed.The examples span multiple app categories.Source: Figure 9 · p. 41

Multi-step imagination examples

  • Figure 10 illustrates autoregressive trajectories in which actions introduce widgets or text while surrounding UI chrome remains stable.Each row is an independent trajectory from an initial screenshot through Step 5.Source: Figure 10 · p. 42

Measured Failure Outcomes on AndroidWorld Hard Tasks

  • 35.3% of failed Hard episodes show false completion, and 35.3% show repetition or oscillation; 17 of 19 evaluated Hard-task episodes fail.The 17 failures span budget exhaustion, failed verification, false completion, or stopping mid-interaction; none ends with an explicit infeasible verdict.Source: §G.3 · pp. 40, 43, Table 12 · p. 43

From Measured Outcomes to Mechanisms

  • Four candidate mechanisms are proposed, but the logs cannot measure dense-UI confusion or transient-dialog inconsistency, and controlled studies are needed to confirm the explanations.Other candidates are coordinate displacement and long-horizon causal noise; only 9.3% of trajectories reaching step 10 survive quality filtering intact.Source: §G.3 · p. 43

Representative failure cases

  • Figure 11 illustrates long-horizon drift through accumulating visual inconsistencies and dense-UI errors such as merged widgets or misplaced insertions and removals.Examples include garbled text, shifted buttons, and spurious elements.Source: Figure 11 · p. 44

Constructive Mitigation Strategies

  • Bidirectional rollout checks are proposed to screen long-horizon trajectories, while post-condition verification is proposed to reduce false completion.The suggested checks include testing whether reverse actions reconstruct the prior state and verifying task post-conditions before accepting termination.Source: §G.3 · pp. 43–44
Loading 2610.05861v1…