PaperlayerView exact revision on arXiv

Source-linked AI summary

Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

Weimin Xiong, Shuhao Gu, Bowen Ye, Zihao Yue, Lei Li, Feifan Song, Sujian Li, Hao Tian

arXiv:2605.14747v1cs.CLcs.AIcs.CVcs.LG

TL;DR

GUI agents need large-scale, diverse interaction trajectories, but existing data collection is limited in scale, diversity, and platform coverage. Video2GUI automatically extracts trajectories from unlabeled Internet videos to build WildGUI, and pre-training on it consistently improves GUI grounding and agentic-task performance.

  • Problem

    Existing GUI-agent datasets remain limited in scale, diversity, and platform coverage, despite diverse trajectories being important for generalized agents.

  • Method

    Video2GUI uses coarse-to-fine video filtering, VLM-driven trajectory extraction, action-space grounding, and two-stage training to construct and use GUI trajectories.

  • Results

    Pre-training Qwen2.5-VL and Mimo-VL on WildGUI consistently improves performance across multiple GUI grounding and agent benchmarks.

  • Takeaways & Limitations

    Scaling training with diverse offline video data provides a promising pathway toward generalized GUI agents.

  • Takeaways & Limitations

    Action spatial grounding assumes a fixed temporal offset of 0.5 seconds, roughly matching the average duration of a single GUI action.

Abstract

from arXiv · show

Recent advances in multimodal large language models have driven growing interest in graphical user interface (GUI) agents, yet their generalization remains constrained by the scarcity of large-scale training data spanning diverse real-world applications. Existing datasets rely heavily on costly manual annotations and are typically confined to narrow domains. To address this challenge, we propose Video2GUI, a fully automated framework that extracts grounded GUI interaction trajectories directly from unlabeled Internet videos. Video2GUI employs a coarse-to-fine filtering strategy to identify high-quality GUI tutorial videos and convert them into structured agent trajectories. Applying this pipeline to 500 million video metadata entries, we construct WildGUI, a large-scale dataset containing 12 million interaction trajectories spanning over 1,500 applications and websites. Pre-training Qwen2.5-VL and Mimo-VL on WildGUI yields consistent improvements of 5-20% across multiple GUI grounding and action benchmarks, matching or surpassing state-of-the-art performance. We will release both the WildGUI dataset and the Video2GUI pipeline to support future research of GUI agents.

1. Introduction

Video2GUI addresses the scarcity and limited scalability of diverse GUI interaction data by automatically extracting grounded trajectories from unlabeled Internet videos. The resulting WildGUI dataset spans millions of trajectories across more than 1,500 applications and websites, and improves GUI agent pretraining performance.

  • Motivation: Generalized GUI agents require large-scale, diverse trajectory data documenting authentic interactions across web, desktop, and mobile platforms.Such data capture visual interface states and actions including clicking, typing, and dragging.
  • Motivation: Existing training data primarily depends on manual annotation or simulated environments, incurring substantial costs and limiting scalability and generalization.These limitations constrain performance on unseen interfaces and tasks.
  • Video2GUI: Video2GUI is a fully automated, scalable framework that filters high-quality GUI instructional videos and extracts accurate interaction trajectories using coarse-to-fine processing.It combines low-cost metadata screening with fine-grained visual evaluation of fidelity, completeness, and task relevance.
  • WildGUI: 12.7 million GUI operation trajectories and 124.5 million screenshots are extracted from 500 million video metadata entries, spanning more than 1,500 applications and websites.WildGUI is constructed from unlabeled web videos and provides coverage across website, mobile, and desktop platforms.
  • Results and release: Pre-training Mimo-VL and Qwen2.5-VL on WildGUI significantly improves performance across multiple GUI grounding and agent benchmarks.The experiments report strong effectiveness and generalization, and the authors will release WildGUI and the Video2GUI pipeline.

2. Formulation of GUI Agent

The GUI agent is formulated as a POMDP in which high-level instructions, environment states, atomic GUI actions, and observations define the interaction setting. At each step, the policy uses the interaction history to select parameterized actions, receives execution feedback, and continues until completion or a step-budget limit.

  • POMDP formulation: The GUI interaction is modeled as a POMDP with user instructions, environment states, atomic GUI operations, observations, and state transitions.The transition function T : S × A → S models how actions change GUI states.
  • Policy and history: At time t, policy πθ selects action at from observation ot−1 and interaction history et−1.The history is et−1 = (u, a1, o1, . . . , at−1, ot−1).
  • Action representation: Each action at consists of an action type τt and parameters bt, such as click coordinates, target UI elements, or input text.Action types include click, type, and scroll.
  • Interaction trajectory: Executing an action produces the next state and feedback observation, forming a perception–action loop that yields trajectory en.The loop ends when the task is completed or the maximum step budget is reached, with en = (u, a1, o1, . . . , an, on) of length n.

3. Video2GUI

Video2GUI converts unlabeled Internet videos into structured GUI interaction trajectories through progressive filtering, trajectory extraction, and spatial grounding. It then uses WildGUI in a two-stage training strategy combining continual pre-training with supervised post-training.

  • Framework: Video2GUI uses progressive filtering, trajectory extraction, and action-space grounding to convert Internet videos into structured GUI interaction trajectories.The pipeline derives task instructions, timestamps, action details, rationales, and precise screen coordinates.
  • Trajectory Extraction: Trajectory extraction transforms high-quality tutorial videos into instruction–trajectory pairs, with each action associated with a precise timestamp.Gemini-3-Pro processes videos in segments of at most 4 minutes using sliding windows and historical context memory.
  • Action Spatial Grounding: Action spatial grounding retrieves three high-resolution frames around each action timestamp and maps the extracted action to a precise screen location.The frame triplet uses offsets of -0.5 seconds, 0, and +0.5 seconds, matching the average duration of a GUI action.
  • Action Spatial Grounding: Over 95% of 200 manually verified sampled actions are accurately parameterized by selecting the first candidate frame with valid spatial grounding.The strategy evaluates three candidate frames and keeps the first one that produces a valid grounding result.
  • Training Strategy: Video2GUI applies two-stage training: continual pre-training on WildGUI for GUI grounding, action prediction, and trajectory modeling, followed by supervised post-training.Continual pre-training spans approximately 200 billion tokens for one epoch, while post-training uses curated open-source datasets for three epochs and approximately 15 billion tokens.

4. Experiments

Experiments evaluate WildGUI-pretrained models on GUI grounding, offline agent, and online agent benchmarks. The models show strong grounding, consistent offline gains, and improved online generalization from offline interaction data.

  • Evaluation Setup: Evaluation spans GUI grounding, offline mobile and Chinese-interface agent tasks, and online OSWorld and AndroidWorld environments.Grounding uses OSWorld-G and ScreenSpot-Pro; offline evaluation uses AndroidControl and CAGUI.
  • GUI Grounding Evaluation: 67.6 average on OSWorld-G makes WildGUI-pretrained Mimo-VL-7B state of the art, exceeding Qwen3-VL-32B at 60.6 and Seed1.5-VL at 62.9.The model also improves element recognition and layout understanding across sub-metrics.
  • Offline GUI Agent Evaluation: 71.4 step success rate on AndroidControl-High and 91.8 on AndroidControl-Low are achieved by WildGUI-pretrained Mimo-VL-7B.The reported gains span high-level planning and low-level execution tasks.
  • Offline GUI Agent Evaluation: Offline evaluation results indicate stronger atomic action execution and generalization across languages and diverse application scenarios.This conclusion follows the reported AndroidControl and CAGUI gains.
  • Online GUI Agent Evaluation: WildGUI improves Mimo-VL-7B performance on OSWorld and AndroidWorld when Stage1 pre-training is added to Stage2 post-training.Despite using entirely offline GUI interaction data, the gains indicate generalization to online, open-ended environments.

5. Analysis

The analysis examines how pretraining scale affects downstream accuracy, how objectives and two-stage training contribute to GUI-agent performance, and how dataset quality is evaluated by experts.

  • Scaling Study: 56.9% accuracy at 200 billion tokens on ScreenSpot-Pro marks the peak of a positive relationship between pretraining scale and downstream performance.The study varies pretraining tokens from 0 to 200 billion and compares against the Stage-2 Only baseline on ScreenSpot-Pro and OSWorld-G.
  • Ablation Study: 56.9 →49.8 on ScreenSpot-Pro follows removal of grounding supervision, showing that Lground is essential for accurate action grounding.The ablation study is conducted on Mimo-VL-7B; trajectory loss supports tracking goal states and historical interactions during multi-step execution.
  • Ablation Study: 6.0 on AndroidWorld under w/o Stage 2 accompanies catastrophic performance drops across all metrics, supporting the necessity of post-training.The study also evaluates removals of specific pretraining loss components, denoted w/o Lx.
  • User Study: Five expert participants rated 300 sampled data points on a 1 to 5 scale to assess video quality and overall dataset quality.Participants were CS master’s or Ph.D. candidates experienced in VLM-based GUI agents, uninvolved in the project, and required to achieve ≥0.85 accuracy on a 20-sample qualification trial.

6. Related Work

GUI agents have progressed from structured-interface methods to vision-based and increasingly capable end-to-end systems. However, training data remains limited by costly collection, narrow platform coverage, and poor scalability, motivating Video2GUI’s scalable filtering and annotation pipeline.

  • GUI Agents: GUI agents evolved from relying on HTML or accessibility trees to vision-based methods enabled by multimodal large language models.Recent work further improves GUI agents through training-strategy and agent-design innovations.
  • GUI Data: Large-scale GUI-agent training data commonly relies on human annotation or model-based synthesis, while existing approaches often remain confined to mobile or desktop platforms.Keyword-based retrieval further limits their scalability, coverage, and diversity.
  • Video2GUI: Video2GUI directly filters and annotates raw YouTube videos through a scalable, top-down pipeline to build GUI data beyond prior methods.This approach targets the coverage and diversity constraints imposed by existing collection schemes.

7. Conclusion · A. Meta Info Classification Details

Video2GUI addresses GUI-agent training data scarcity by automatically extracting interaction trajectories from unlabeled Internet videos and constructing WildGUI, a large-scale pretraining dataset. Its metadata classifier uses textual video information, DeepSeek supervision, and an efficient Qwen2.5-7B binary classification design with positive-sample upsampling.

  • 7. Conclusion: Video2GUI synthesizes high-quality GUI interaction trajectories from unlabeled Internet videos through a fully automated framework.The framework uses coarse-to-fine filtering and VLM-driven trajectory extraction.
  • 7. Conclusion: 12 million trajectories across 1,500+ applications and websites make WildGUI the largest GUI pre-training dataset to date.The dataset is constructed to address data scarcity in GUI agent training.
  • A. Meta Info Classification Details: The metadata classifier determines whether videos are GUI tutorials using titles, descriptions, keywords, channel names, categories, and available subtitles.It operates solely on textual information and uses DeepSeek to annotate training data for high-quality supervision.
  • A. Meta Info Classification Details: Qwen2.5-7B uses a lightweight binary classification head over its final hidden layer to output class probabilities directly.A minimal classification prompt replaces complete generated results and rationales; optimization uses cross-entropy loss.
  • A. Meta Info Classification Details: 10K videos annotated with DeepSeek yield approximately 400 positive samples and over 9K negative samples.The classifier predicts the probability that each video is a GUI tutorial.
  • A. Meta Info Classification Details: Positive-sample upsampling constructs a balanced training set to mitigate the severe class imbalance.The Qwen2.5-7B classifier is trained for 3 epochs, although the supplied passage truncates the remaining training details.

B. Video Scoring Details … E. Statistics

The paper details automated video scoring, trajectory extraction, spatial grounding, and statistical analysis for constructing WildGUI. The pipeline uses multidimensional quality assessment, structured segment annotations, timestamp-based frame grounding, and distributional analyses to support broad task coverage.

  • B. Video Scoring Details: The video scoring model evaluates tutorial quality across three dimensions using video content as input.Gemini-3-Pro annotates training data to provide high-quality supervision.
  • B. Video Scoring Details: 200 hours of training data are annotated, and the scorer is trained for 3 epochs at a learning rate of 2 × 10−5.The test set contains 100 videos, with evaluation results reported in Figure 5.
  • C. Trajectory Extraction Details: 300,000 hours of annotated content are obtained by selecting filtered videos under 12 minutes and using distinct mobile and desktop action spaces.The action spaces follow prior work and are detailed in Table 6.
  • C. Trajectory Extraction Details: Videos are segmented into 4-minute clips annotated with task instructions, dense captions, task plans, platforms, and application types.These annotations describe both intended tasks and their execution environments.
  • C. Trajectory Extraction Details: Each action trajectory records timestamps, action types, grounding instructions, rationales, parameters, and resulting interface changes.The strategy is intended to help pretrained models predict GUI states from screenshots and actions.
  • D. Action Spatial Grounding Details: Spatial grounding extracts frames at t, t −0.5s, and t + 0.5s, then uses Gemini-3-Pro to determine whether the action can be grounded.When grounding is feasible, the model’s grounding result is used for the action coordinates.
  • E. Statistics: WildGUI statistics analyze platform, application, website, video-duration, trajectory-length, and action-type distributions.Figures 6 and 7 show broad platform and application-category coverage across diverse user tasks and usage scenarios.

F. Agent Training

Agent post-training combines diverse GUI interaction datasets with cleaner, more precise human supervision to consolidate the agent’s policy and enhance performance.

  • Training data: Post-training uses GUI interaction datasets spanning Rico, SeeClickWeb, WebUI, OS-Atlas, AITW, AITZ, AndroidControl, AMEX, and GUI-Odyssey.The collection includes datasets from diverse GUI interaction settings.
  • Supervision: Cleaner and more precise human supervision signals are used during post-training.The passage identifies supervision quality as a defining feature of this training stage.
  • Training objective: This post-training stage consolidates the agent’s policy and enhances performance on domain-specific tasks.The supplied passage truncates immediately after “domain-,” so the specific performance domain is not stated.

G. API cost for Video2GUI pipeline

Video2GUI supports both closed-source and open-source backbones, using Gemini-3-Pro as the default annotator. End-to-end cost is dominated by trajectory extraction, at roughly $0.0653 per sample.

  • Backbone and cost: Gemini-3-Pro is the default annotator because of its long-context video understanding and spatial grounding quality.The pipeline supports both closed-source and open-source backbones.
  • Backbone and cost: 15,908 input tokens, 1,338 output tokens, and 1,452 thinking tokens yield roughly $0.0653 per sample for trajectory extraction.These figures summarize the full pipeline’s per-sample API cost, with trajectory extraction dominating expenses.

H. Evaluation

The evaluation covers GUI grounding, offline mobile-agent performance, and online interactive-agent performance. It uses benchmarks spanning Linux, mobile, desktop, web, and OS-level environments.

  • Evaluation scope: Three evaluation categories assess GUI grounding, offline agent capabilities, and online agent performance in interactive environments.The benchmark suite is designed to provide a comprehensive assessment of model capabilities.
  • GUI grounding: 564 OSWorld-G samples test instruction-to-UI-element grounding across text matching, element recognition, layout understanding, and precise Linux manipulation.ScreenSpot-Pro complements OSWorld-G with high-resolution professional scenarios.
  • Offline agent evaluation: AndroidControl-High evaluates global reasoning and long-horizon planning, while AndroidControl-Low measures fine-grained step-level action execution in mobile environments.CAGUI is also used for offline mobile-agent evaluation, with Type and another standard metric reported.
  • Online agent evaluation: 369 OSWorld tasks span web applications, desktop software, and OS-level operations, while AndroidWorld includes 116 programmatic tasks across 20 real-world apps.AndroidWorld dynamically constructs tasks from parameterized natural-language instructions.

I. Examples

The section visualizes several WildGUI trajectory examples and illustrates the metadata-based criteria used to identify GUI operation videos. It also shows example outputs for relevance classification and trajectory annotation.

  • Examples: Several WildGUI trajectory examples are visualized in Figures 8–11.The examples are presented as visualizations of the dataset’s trajectory data.
  • GUI Content Filtering: Relevant videos feature screen recordings, software interaction, or application demonstrations and tutorials.Examples include interacting with graphical user interfaces and walkthroughs of software such as Photoshop, Excel, VS Code, web browsers, and operating-system settings.
  • Filtering Example: The filtering examples classify an Excel automation tutorial as GUI content because its title and tags indicate a software walkthrough.The example output sets is_gui_content to true and cites the title’s “How to” and “Excel” references plus “tutorial” and “code” tags.
  • Trajectory Annotation: Trajectory annotation outputs require granular shots split at atomic actions and a JSON list whose items represent complete high-level subtasks.A shot spans an atomic action and the resulting screen state until the next atomic action, while each subtask concludes with a finish action.
Loading 2605.14747v1…