Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

• Generalized GUI agents require diverse, precisely documented interaction trajectories, while Internet videos present quality-filtering, missing-annotation, and coordinate-grounding challenges.Existing manually annotated trajectories and simulated environments (Shvo et al., 2021; Lee et al., 2024) entail substantial annotation costs and limited scalability.Source: §1 · p. 1
• Video2GUI uses coarse-to-fine filtering to select high-quality GUI instructional videos efficiently for automated trajectory extraction.Low-cost screening uses titles, descriptions, and keywords to eliminate unrelated videos before storage and computation. A visual scoring model then evaluates visual fidelity, instructional completeness, and task relevance.Source: §1 · pp. 1–2
• Trajectory annotation identifies task instructions, operation timestamps, action details, and reasoning before grounding actions to exact coordinates in extracted screenshots.Screenshot-based grounding mitigates difficulties caused by video frame compression (Mu et al., 2024).Source: §1 · p. 2
• WildGUI pre-training improves Qwen2.5-VL and Mimo-VL across GUI grounding and offline/online agent benchmarks, matching or surpassing state-of-the-art performance.The training approach also includes post-training on curated open-source datasets to refine task-specific performance.Source: §1 · p. 2
• 12.7 million trajectories and 124.5 million screenshots span over 1,500 applications and websites in WildGUI, extracted from 500 million video metadata entries.The dataset covers website, mobile, and desktop platforms.Source: §1 · p. 2, Table 1 · p. 2
• The authors plan to release both WildGUI and the Video2GUI pipeline to support scalable GUI-agent training and evaluation research.Source: §1 · p. 2

2. Formulation of GUI Agent

• GUI interaction is modeled as a POMDP defined by user instructions U, environment states S, actions A, observations O, and state-transition function T.T: S × A → S models how actions change GUI states. Instructions and actions are typically expressed in natural language, while observations provide multimodal feedback such as screenshots.Source: §2 · p. 2
• At each step, policy π_θ selects an action conditioned on the interaction history, including the user instruction, previous actions, and observations.The agent receives observation o_{t−1} and selects a_t according to π_θ(· | e_{t−1}).Source: §2 · p. 2
• Each action pairs an operation type with parameters, changes the environment state, and yields feedback until task completion or the step limit.Action a_t = (τ_t, b_t) specifies a type such as click, type, or scroll and parameters such as coordinates, target elements, or input text. The resulting trajectory is e_n = (u, a_1, o_1, …, a_n, o_n), with length n.Source: §2 · pp. 2–3

3.4. Agent Training

• Two-stage agent training combines WildGUI pretraining on grounding, action prediction, and trajectory modeling with supervised post-training on curated high-quality open-source datasets.Continual pretraining runs for one epoch over approximately 200 billion tokens; trajectory modeling predicts actions autoregressively from screenshot sequences and interaction histories, with loss restricted to text tokens. Post-training uses three epochs totaling approximately 15 billion tokens.Source: §3 · p. 3, §3.4 · p. 5

3.1. Coarse-to-Fine Video Filtering

• Approximately 20 million candidates remain after a lightweight metadata classifier filters metadata collected from over 500 million YouTube videos.A larger language model supplies relevance labels for 10K samples, which are balanced through upsampling to train the classifier. Held-out human annotations validate alignment of the automated labels with human judgment.Source: §3.1 · pp. 3–4
• 4.16 million videos, approximately 300,000 hours, survive content-based quality scoring of 20 million coarsely filtered candidates.A lightweight omnimodal scorer evaluates each video’s first minute for topic relevance, instruction clarity, and screen recording quality. Its training annotations comprise dimension-wise scores and rationales for approximately 200 hours of sampled videos.Source: §3.1 · p. 4

3.2. Trajectory Extraction

• Trajectory extraction uses Gemini-3-Pro on segments of at most four minutes, carrying previous extraction results forward as textual memory for tasks spanning segment boundaries.The output comprises task instructions, timestamped actions, action details and rationales, and visually grounded low-level instructions for individual steps.Source: §3 · p. 3, §3.2 · p. 4

3.3. Action Spatial Grounding

• Over 95% of 200 manually checked actions were accurately parameterized using high-resolution frames at t-0.5s, t, and t+0.5s.Gemini-3-Pro uses each action’s low-level instruction to predict coordinates or bounding boxes, selecting the first frame yielding valid grounding. The 0.5-second offset approximates the average duration of one GUI action.Source: §3.3 · pp. 4–5

4.3. Main Results

• 71.4 and 91.8 step success rates on AndroidControl-High and AndroidControl-Low improve WildGUI-pretrained Mimo-VL-7B over its respective 65.6 and 87.9 baselines.On CAGUI, type accuracy rises from 74.2 to 90.3 and step success rate from 55.2 to 71.0; Qwen2.5-VL also improves. These results support improved atomic action execution and generalization across languages and application scenarios.Source: Table 3 · p. 6, §4.3 · pp. 6–7

4.1. Experiment Setup

• Training combines WildGUI pre-training and curated-data refinement, using up to 4,096 input visual tokens for fine-grained details and a maximum sequence length of 32,768 for long-horizon interaction trajectories.Stage 1 trains for 24,000 steps; Stage 2 refines the model for three epochs, corresponding to 2,000 steps. The maximum sequence length applies across both stages.Source: §4.1 · pp. 5–6, §4.3 · p. 7

4.2. Evaluation

• Evaluation covers GUI grounding, offline mobile planning and execution, Chinese-language interfaces, and online desktop and mobile environments.Grounding uses OSWorld-G (Xie et al., 2025) and ScreenSpot-Pro; offline evaluation uses AndroidControl and CAGUI (Zhang et al., 2025b). Online evaluation uses OSWorld (Xie et al., 2024) and AndroidWorld (Rawles et al., 2024).Source: §4.2 · p. 6

4.3. Main Results

• 67.6 average on OSWorld-G makes WildGUI-pretrained Mimo-VL-7B state of the art, exceeding Qwen3-VL-32B’s 60.6 and Seed1.5-VL’s 62.9.Improvements extend across all sub-metrics, including element recognition and layout understanding.Source: §4.3 · p. 6
• 56.9 on ScreenSpot-Pro places WildGUI-pretrained Mimo-VL-7B above Qwen3-VL-32B’s 54.9 but below Seed1.5-VL’s 60.9.WildGUI pre-training also raises Qwen2.5-VL-7B from 26.8 to 41.2, demonstrating grounding improvements across model architectures.Source: §4.3 · p. 6
• 31.9% AndroidWorld success with the full two-stage Mimo-VL-7B pipeline exceeds the base model’s 16.4% and Stage2-only baseline’s 23.3%.On OSWorld, the full pipeline reaches 12.3%, compared with 10.4% for Stage2-only training. These gains demonstrate transfer from offline WildGUI pre-training to dynamic, open-ended online environments.Source: Figure 2 · p. 7, §4.3 · p. 7

5.1. Scaling Effects

• 56.9% on ScreenSpot-Pro and 67.6% on OSWorld-G were peak accuracies at 200 billion pretraining tokens, exceeding the Stage-2 Only baseline.Across the 0–200 billion-token study, accuracy increased from approximately 41% and 55%, respectively. The model surpassed the baseline post-trained exclusively on open-source data around 50 billion tokens and showed no evident saturation.Source: Figure 3 · p. 7, §5.1 · p. 7

5.2. Ablation Studies

• 31.9 → 24.1 on AndroidWorld after removing Ltraj contrasts with competitive static-task performance on ScreenSpot-Pro and CAGUI.The Mimo-VL-7B ablation supports trajectory modeling for long-horizon planning: it enables tracking goal states and maintaining awareness of historical interactions during multi-step execution.Source: §5.2 · pp. 7–8
• 6.0 on AndroidWorld without Stage 2 accompanied severe degradation across all evaluated metrics, demonstrating the importance of post-training.WildGUI pretraining provides broad GUI knowledge, but alignment through high-quality second-stage data remains necessary for complex instruction-following scenarios.Source: §5.2 · pp. 7–8
• 56.9 → 49.8 on ScreenSpot-Pro after removing Lground demonstrates the importance of explicit grounding supervision for accurate action grounding.This ablation was conducted on Mimo-VL-7B.Source: §5.2 · pp. 7–8

5.3. Data Quality Check

• 4.62 was WildGUI’s overall trajectory-quality score, exceeding TongUI’s 3.35 and VideoAgentTrek’s 4.05 in expert evaluations.Ratings assessed action, timestamp, and coordinate accuracy; platform and task diversity; and real-world task relevance. Inter-rater agreement was Krippendorff’s α = 0.84.Source: §5.3 · p. 8
• 1.22 → 2.12 → 4.45 was the progression in average video-quality ratings through metadata filtering and then video scoring.Five independent experts rated 300 sampled data points on a 1–5 scale, assessing video task relevance and screen-recording quality. Each evaluator passed a 20-sample qualification trial with ≥0.85 accuracy.Source: §5.3 · p. 8

6.1. GUI Agents

• GUI agents have progressed from structured HTML or accessibility inputs toward vision-based interaction using screenshots and natural-language instructions.Structured-input approaches include Deng et al. (2023), Gur et al. (2023), Yang et al. (2023), He et al. (2024), and Lai et al. (2024). Vision-based approaches include Hong et al. (2024), Zhang & Zhang (2024), and Zhang et al. (2024b).Source: §6.1 · p. 8
• Recent GUI agents improve training and agent design but remain dependent on large-scale, diverse data that is prohibitively expensive to collect manually.UI-TARS (Qin et al., 2025) uses multi-stage post-training to improve perception, action, and reasoning. ARPO (Lu et al., 2025b) uses end-to-end reinforcement learning with a replay buffer that reuses successful interactions.Source: §6.1 · p. 8

6.2. Data Collection for GUI Agents

• Web trajectory harvesting reduces annotation costs, but existing approaches remain platform-specific and rely on keyword retrieval that limits coverage and diversity.Web harvesting is explored by Lu et al. (2025a) and Zhang et al. (2025a). Platform-specific examples include mobile (Jang et al., 2025) and desktop environments (Song et al., 2025).Source: §6.2 · pp. 8–9
• Human annotation and model-based synthesis provide GUI supervision, but existing datasets remain limited in scale and diversity.Collection methods include human annotation (Deng et al., 2023) and model-based synthesis (Lu et al., 2025c). Supervision covers planning, action sequences, and interaction targets (Cheng et al., 2024; Lu et al., 2024).Source: §6.2 · p. 8
• Video2GUI uses top-down filtering and annotation of raw YouTube videos to build a large-scale, diverse GUI dataset beyond prior collection methods.The pipeline contrasts with existing approaches whose keyword-based retrieval constrains coverage and diversity.Source: §6.2 · pp. 8–9

7. Conclusion

• WildGUI pre-training consistently improves Qwen2.5-VL and Mimo-VL across GUI grounding and agentic tasks, enhancing model generalization.The authors plan to release both WildGUI and the Video2GUI pipeline.Source: §7 · p. 9
• 12 million trajectories across 1,500+ applications and websites form WildGUI, constructed automatically from unlabeled Internet videos through coarse-to-fine filtering and VLM-driven extraction.The authors describe WildGUI as the largest GUI pre-training dataset to date.Source: §7 · p. 9

A. Meta Info Classification Details

• DeepSeek (Liu et al., 2024) supplies binary GUI-tutorial labels and brief reasoning in JSON, enabling automatic parsing and iterative annotation-prompt refinement.Annotation receives the complete video metadata; the reasoning helps refine the prompt design.Source: A. Meta Info Classification Details · p. 12
• The metadata classifier identifies GUI-operation tutorials using only titles, descriptions, keywords, channel names, video categories, and available subtitles.Subtitles are included only when present.Source: A. Meta Info Classification Details · p. 12
• Qwen2.5-7B uses a lightweight binary head on its final hidden layer to output class probabilities instead of generating classification results and rationales.Training uses cross-entropy loss and the minimal prompt “Classify based on the following content” to prioritize inference efficiency.Source: A. Meta Info Classification Details · p. 12
• 10K randomly sampled videos yield approximately 400 positives and over 9K negatives; positive upsampling balances the classifier’s training set.The videos are annotated using DeepSeek. Qwen2.5-7B is trained for 3 epochs with a learning rate of 2 × 10^-5.Source: A. Meta Info Classification Details · p. 12

B. Video Scoring Details

• Gemini-3-Pro supplies quality supervision for a Qwen2.5-Omni scorer with three regression heads trained using mean squared error.Annotation uses each video's first minute, or its entirety if shorter, and returns scores with reasoning in JSON. Training uses 200 annotated hours, score-balancing upsampling, three epochs, and a learning rate of 2 × 10^-5.Source: B. Video Scoring Details · pp. 12–13
• 4.16 million videos, approximately 300,000 hours, remain from 20 million candidates after requiring scores of at least 4.2 on all three quality dimensions.The dimensions are topic relevance, instruction clarity, and recording quality; thresholds were established through manual verification.Source: B. Video Scoring Details · p. 13

C. Trajectory Extraction Details

• Two distinct action spaces cover mobile and desktop platforms, following prior work (Xie et al., 2024; Zhang et al., 2025b).Only videos shorter than 12 minutes are selected for annotation, yielding 300,000 hours of annotated content.Source: C. Trajectory Extraction Details · p. 13
• 4-minute clips receive structured annotations, with preceding segments' annotations supplied as context for coherent annotation of later segments.Annotations cover the user task instruction, dense caption, task plan, platform, application type, website name, and action trajectory.Source: C. Trajectory Extraction Details · p. 13
• Action annotations include timestamps, types, grounding instructions, rationales, parameters, and resulting interface changes to support learning GUI state prediction and long-horizon planning.Before trajectory extraction, videos are segmented into consecutive clips corresponding to individual user operations; annotations also explain why interface changes occur.Source: C. Trajectory Extraction Details · p. 13

D. Action Spatial Grounding Details

• Three frames at t, t - 0.5s, and t + 0.5s provide sequential grounding attempts, with actions discarded if all attempts fail.Gemini-3-Pro checks whether each frame permits grounding before returning action coordinates, addressing temporal misalignment around frequent GUI operations.Source: D. Action Spatial Grounding Details · p. 13

E. Statistics

• WildGUI spans varied platforms and application categories, providing broad coverage of user tasks across usage scenarios.Statistics examine platforms, functional application and website categories, video durations, trajectory lengths, and action types.Source: E. Statistics · pp. 13, 15

F. Agent Training

• Cleaner, more precise human supervision consolidates the agent’s policy and enhances performance on domain-specific downstream tasks.These benefits are attributed to the post-training stage; this passage provides no quantitative results.Source: F. Agent Training · p. 15
• Nine GUI interaction datasets provide diverse data for post-training.Sources include Rico, SeeClickWeb (Cheng et al., 2024), WebUI (Wu et al., 2023), OS-Atlas (Wu et al., 2024), AITW, AITZ (Zhang et al., 2024b), AndroidControl, AMEX (Chai et al., 2025), and GUI-Odyssey (Lu et al., 2025c).Source: F. Agent Training · p. 15
• Prompts 6, 7, and 8 present the prompt templates used during training.Source: F. Agent Training · p. 15

G. API cost for Video2GUI pipeline

• $0.0763 per sample is the approximate total API cost of running the default Video2GUI pipeline end-to-end.Spatial grounding adds approximately $0.011 per sample; video quality scoring uses self-deployed Qwen2.5-Omni and incurs negligible cost.Source: G. API cost for Video2GUI pipeline · p. 15
• $0.0653 per sample makes trajectory extraction the dominant API expense.Each sample consumes approximately 15,908 input tokens, 1,338 output tokens, and 1,452 thinking tokens.Source: G. API cost for Video2GUI pipeline · p. 15
• 15–20% lower annotation quality is reported for the functional open-source alternative Qwen3.5-397B-A17B, primarily in temporal alignment and spatial grounding.The pipeline supports both closed-source and open-source backbones, but defaults to Gemini-3-Pro for its strong long-context video understanding and spatial grounding quality.Source: G. API cost for Video2GUI pipeline · p. 15
• Dataset construction is a one-time cost: released WildGUI data and the reproducible Video2GUI pipeline mean downstream users need not rerun annotation.Source: G. API cost for Video2GUI pipeline · pp. 15, 17

GUI grounding

• GUI grounding uses OSWorld-G (Xie et al., 2025) for Linux capabilities and ScreenSpot-Pro for high-resolution professional interfaces.OSWorld-G contains 564 samples covering text matching, element recognition, layout understanding, and precise manipulation. ScreenSpot-Pro contains 1,581 expert-annotated tasks across 23 professional applications and three operating systems, emphasizing industrial software and multi-window interfaces.Source: H. Evaluation · p. 17

Offline agent evaluation

• Offline mobile evaluation uses AndroidControl and CAGUI (Zhang et al., 2025b), with CAGUI assessing Chinese-language interfaces.AndroidControl-High tests global reasoning and long-horizon planning, while AndroidControl-Low tests fine-grained step-level action execution. Evaluation uses Type (action type accuracy) and Step-wise Success Rate (SR).Source: H. Evaluation · p. 17

Online agent evaluation

• Online evaluation uses OSWorld (Xie et al., 2024) and AndroidWorld (Rawles et al., 2024) to assess agents in interactive environments.OSWorld comprises 369 real-world tasks spanning web applications, desktop software, and OS-level operations. AndroidWorld provides a fully functional Android environment with 116 programmatic tasks across 20 real-world apps, dynamically constructed using parameterized natural language instructions.Source: H. Evaluation · p. 17

Trajectory Examples

• Figures 8, 9, 10, and 11 visualize examples of WildGUI trajectory data.Source: §I · p. 17

GUI Relevance Filtering

• GUI relevance screening uses textual metadata alone to identify software interactions and application demonstrations or tutorials.Excluded content includes gaming unless specifically teaching UI/settings, static presentations without active software operation, and physical hardware reviews rather than screen interfaces.Source: §I · p. 18

Filtering Example

• 0.95 confidence accompanies a positive GUI-content classification for the Excel automation example, justified by software-specific tutorial wording and tutorial/code tags.This is an illustrative filtering output, not a benchmark result.Source: §I · p. 18

Video Quality Evaluation

• 1 to 5 ratings assess topic relevance, instruction clarity, and recording quality using textual metadata, descriptions, and transcripts.The criteria emphasize function-oriented GUI teaching on PCs, smartphones, and tablets; clear, logical instructions; and legible, stable screen captures.Source: §I · p. 19

Shot Splitting

• Shot splitting starts each shot at an atomic action and ends it when the next action begins, retaining the resulting screen state.Analysis must remain within the provided video segment, without generating content beyond its end.Source: §I · pp. 20–21

Task Annotations

• Task annotations group atomic shots into high-level subtasks, use first-person reasoning linking prior history to current decisions, and conclude completed tasks with finish actions.Each action also includes a natural-language instruction for precisely locating its UI element.Source: §I · pp. 20–21

Field Definitions for JSON (Part 2)

• Sub-task JSON combines intent, causally detailed captions, high-level plans, and chronological actions aligned with individual video shots.Each action's type and parameters must match the task platform's action space.Source: §I · p. 22
• Action reasoning combines trajectory history with current observations, while screen-change predictions require justified mechanisms and evidence.The predicted next shot must follow from the reasoning, with every detail justifiable.Source: §I · p. 22

Platform-Specific Action Spaces

• Platform-specific schemas distinguish PC and mobile actions, while grounding instructions must uniquely identify the target UI element.Action types determine parameter structures; pixel coordinates use [y, x] ordering.Source: §I · pp. 22–23

I. Examples

• Segment analysis preserves global context and annotation consistency, uses absolute video timestamps, and continues unfinished sub-tasks under their existing indices.Completed sub-tasks advance to the next available index. Output must read seamlessly without mentioning the supplied annotation history.Source: §I · p. 24
• The grounding prompt supplies a grounding instruction, action type, and screenshot, then begins with a feasibility check of the screenshot and instruction.The supplied excerpt does not specify the second step.Source: §I · p. 24

Grounding feasibility

• Grounding checks target visibility, explains infeasible actions, and otherwise predicts each requested target’s center point and bounding box.Coordinates are relative, ranging from 0–1000, with formats <point>y x</point> and <bbox>y1 x1 y2 x2</bbox>.Source: §I · pp. 24–26

Prompt 6: Prompt for GUI Grounding

• GUI grounding prompts locate command-matching components and return JSON containing either points or bounding boxes, labeled with the whole command.The two variants use the keys point and bbox_2d, respectively.Source: §I · p. 26

Prompt 7: Prompt for GUI Action Prediction

• Action prediction prompts use a screenshot, goal, action history, and action space to request the next action, either directly or with explicit reasoning.The direct variant returns JSON actions; the reasoning variant adds Thought and Action fields before the JSON.Source: §I · p. 26

Prompt 8: Prompt for GUI Trajectory Modeling

• Trajectory modeling prompts request the next action from a screenshot and goal, offering direct JSON output or a reasoning-then-action format.The reasoning variant includes Thought and Action fields followed by JSON actions.Source: §I · p. 26