PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos

Dunjie Lu, Yiheng Xu, Junli Wang, Haoyuan Wu, Xinyuan Wang, Zekun Wang, Junlin Yang, Hongjin Su, Jixuan Chen, Junda Chen, Yuchen Mao, Jingren Zhou, Junyang Lin, Binyuan Hui, Tao Yu

arXiv:2510.19488v1cs.CLcs.AIcs.LG

TL;DR

Computer-use agents need extensive GUI trajectories, but manual annotation is too expensive and tutorial videos lack explicit action labels. VideoAgentTrek uses Video2Action to mine structured trajectories from unlabeled videos, then applies continued pretraining and supervised fine-tuning. On OSWorld-Verified, task success rises from 9.3% to 15.8%, while AgentNetBench step accuracy rises from 64.1% to 69.3%.

  • Problem

    Computer-use agents require large-scale GUI interaction trajectories with precise action parameters, but manually annotating them is prohibitively expensive and tutorial videos lack structured labels.

    Source: §1 · p. 1

  • Method

    VideoAgentTrek uses Video2Action to detect and localize GUI events, extract action parameters, and convert unlabeled videos into structured trajectories for continued pretraining followed by supervised fine-tuning.

    Source: §1 · p. 1, §2.3.2 · p. 5

  • Results

    15.8% OSWorld-Verified task success versus 9.3% for SFT-only baselines, a 70% relative improvement, and AgentNetBench step accuracy rising from 64.1% to 69.3%.

    Source: §1 · p. 2, §3.1.2 · p. 6, §6 · p. 8

  • Takeaways & Limitations

    Unlabeled internet videos can provide effective, scalable supervision for computer-use agents when processed through learned inverse dynamics.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

Training computer-use agents requires massive amounts of GUI interaction data, but manually annotating action trajectories at scale is prohibitively expensive. We present VideoAgentTrek, a scalable pipeline that automatically mines training data from publicly available screen-recorded videos at web scale, eliminating the need for manual annotation. Our approach addresses a key challenge: raw videos contain implicit demonstrations but lack explicit action labels. To solve this, we develop Video2Action, an inverse dynamics module (IDM) with two components: (1) a video grounding model that detects and localizes GUI actions with precise temporal boundaries and context, and (2) an action-content recognizer that extracts structured parameters like click coordinates and typed text with high fidelity. Applied to 39,000 YouTube tutorial videos, our pipeline generates 1.52 million interaction steps automatically. We leverage this data through continued pretraining followed by supervised fine-tuning. On OSWorld-Verified, our approach improves task success rates from 9.3% (SFT-only baseline) to 15.8%, a 70% relative improvement. On AgentNetBench, step accuracy increases from 64.1% to 69.3%. Our results demonstrate that passive internet videos can be transformed into high-quality supervision for computer-use agents, providing a scalable alternative to expensive manual annotation.

1 Introduction

VideoAgentTrek addresses the data bottleneck in computer-use agents by converting unlabeled screen-recorded tutorials into structured GUI trajectories through learned inverse dynamics. The resulting data supports continued pretraining and supervised fine-tuning, improving benchmark performance.

  • Manual annotation is prohibitively expensive, while screen-recorded tutorials contain implicit demonstrations of clicks, typed text, and interface responses without structured action labels.VideoAgentTrek targets this gap by converting passive recordings into active training data.Source: §1 · pp. 1–2
  • Video2Action detects GUI events with precise temporal boundaries and extracts parameters such as click coordinates and typed text from raw videos.Its inverse-dynamics pipeline reconstructs structured interaction trajectories without ground-truth labels.Source: §1 · pp. 1–2
  • VideoAgentTrek automatically extracts 1.52 million interaction steps from 39,000 YouTube videos for large-scale computer-use pretraining.The mined trajectories span hundreds of applications across Windows, macOS, and web platforms.Source: §1 · p. 1
  • The pipeline combines video collection, Video2Action trajectory extraction, continued pretraining, and supervised fine-tuning to train computer-use agents.SCREENFILTER filters GUI footage before action extraction, and the mined trajectories are used in both training stages.Source: §1 · p. 1, Figure 1 · p. 2
  • 15.8% task success on OSWorld-Verified versus 9.3% for SFT-only baselines demonstrates a 70% relative improvement.The approach also improves AgentNetBench step accuracy from 64.1% to 69.3%.Source: §1 · p. 2

2 VideoAgentTrek

VideoAgentTrek turns publicly available screen recordings into structured computer-use supervision through scalable video collection, VIDEO2ACTION extraction, and two-stage agent training. The pipeline combines dense temporal action detection, parameter recognition, and inner-monologue generation to produce training trajectories from web videos.

2.2 VIDEO2ACTION: Inverse Dynamics Module

  • VIDEO2ACTION converts raw video into ReAct training tuples by detecting tightly bounded GUI events, inferring action parameters, and generating brief inner monologues.The module recovers typed segments, parameters such as click coordinates or typed strings, and intent-oriented rationales.Source: §2 · p. 2, Figure 4 · p. 4, §2.2.3 · p. 5

2.1.1 Video Candidate Auto-Discovery

  • 55,000 candidate videos spanning roughly 10,000 hours are discovered through keyword, channel, and tag expansion before filtering.High-quality channels are expanded when at least 80% of sampled videos pass validation, deliberately favoring recall before SCREENFILTER processing.Source: §2.1.1 · p. 3, Figure 2 · p. 3

2.1.2 Video Preprocessing with SCREENFILTER

  • SCREENFILTER retains GUI-focused segments by requiring cursor presence in at least 80% of frames for six seconds or more, extracting 7,377 verified interaction hours.A two-second merge gap provides temporal smoothing, and the filter addresses non-interactive or presentation-heavy retrieved content.Source: §2.1.2 · p. 3

2.1.3 VideoAgentTrek Data Analysis

  • The collected corpus is predominantly clear and tutorial-oriented, with tutorials comprising 69.6% of audited videos and domain coverage biased toward OS-level and professional use cases.About 97% of videos are 720p or higher; the domain distribution is approximately 36% OS-level and 19% professional.Source: §2.1.3 · p. 3, Figure 3 · p. 3

2.2.1 Action Event Detection

  • The action detector predicts multiple typed GUI events with start and end timestamps directly from an unlabeled video, while the recognizer maps each detected segment to an action and its parameters.The detector uses video grounding for ordered event spans; parameterization produces outputs such as click coordinates or typed content and is evaluated through blinded manual review on unseen tutorials.Source: §2.2.1 · p. 4, §2.2.2 · p. 4

2.3.1 Agentic Data Collection

  • 39,000 processed videos yield approximately 1.52 million ReAct steps and about 26 billion training tokens for downstream training.Each video produces an average of 39 steps; the resulting trajectories include screenshots, thoughts, actions, and parameters.Source: §2.3.1 · p. 5

2.3.2 Training strategy.

  • Training uses one stage of interleaved video-text pretraining on 26B tokens followed by eight billion tokens of supervised fine-tuning on clean human-annotated trajectories.The schedule first stabilizes grounding with broad, imperfect supervision and then consolidates policy on curated data.Source: §2.3.2 · p. 5

2.3 Computer Use Model Pretraining

  • 15.8% OSWorld-Verified task success is reported for VideoAgentTrek, with experimental results also covering AgentNetBench.Figure 5 presents the OSWorld-Verified and AgentNetBench evaluation scope, while the supplied prose establishes the OSWorld result.Source: Figure 5 · p. 6

3 Experiments

The experiments evaluate VideoAgentTrek on online and offline computer-use benchmarks and separately assess whether VIDEO2ACTION can recover GUI events and action parameters from video. Video pretraining improves benchmark performance, while detector and identifier evaluations show stronger results for visually salient pointer actions than subtle typing or drag actions.

3.1.2 Main Results.

  • 69.3% AgentNetBench step success exceeds the SFT-only baseline of 64.1% by 5.2 points and the base model’s 38.5% by 30.8 points.The benchmark covers offline tasks across applications and websites on Windows and macOS.Source: §3.1.2 · p. 6, §3.1.1 · p. 6
  • 14.13% OSWorld-Verified task success improves over 9.3% SFT-only training by 4.83 points and more than triples the 4.5% base-model result.OSWorld-Verified evaluates 369 human-crafted Ubuntu desktop tasks in dynamic online environments.Source: §3.1.2 · p. 6, §3.1.1 · p. 6
  • 15.78% OSWorld-Verified success at 50 steps rises from 14.13% at 20 steps, whereas SFT-only performance does not improve with additional steps.The result indicates that the video-pretrained model can use extra test-time exploration budget.Source: §3.1.2 · p. 6

3.2.1 Action Event Detection

  • The action-event detector reaches 0.88 overall precision and 0.71 recall, with pointer-centric actions more reliably localized than keystroke-only actions.Evaluation uses 20,282 annotated GUI events from 23 held-out hours, plus manual review on 10 unseen tutorials; the manual recovery rate is about 70%.Source: §3.2.1 · p. 7, Table 1 · p. 7

3.2.2 Action Identification

  • Action-parameter accuracy is highest for clicks and scrolls, moderate for typing, and weakest for drag or press actions because subtle visual cues are harder to interpret.A blinded manual assessment reviews 500 predictions by checking whether predicted parameters plausibly explain observed screen transitions.Source: §3.2.2 · p. 7, Table 2 · p. 7

4 Analysis

Performance scales with the amount of video pretraining data, and longer mined trajectories improve the agent’s ability to use larger action budgets. These effects connect data scale and long-horizon supervision with stronger benchmark performance and test-time scaling.

4.1 Effectiveness of Data Scaling

  • 69.3% AgentNetBench step success and 15.7% OSWorld-Verified SR@50 are reached with 100% video tokens, versus 64.1% and 9.3% with none.The intermediate 50% setting reaches 68.1% on AgentNetBench and 13.3% on OSWorld-Verified.Source: §4.1 · p. 7, Figure 6 · p. 7

4.2 Improving Long Horizon planning

  • 42.1% of trajectories exceed 20 steps, 14.5% contain at least 50 steps, and average length is 39.25 steps.Cross-dataset comparisons indicate that the corpus emphasizes complex multi-step workflows rather than brief interactions.Source: §4.2 · p. 7
  • 15.78% task success at 50 steps versus 14.13% at 20 steps shows a 1.65-point gain, while Stage2-only remains flat at 9.3%.The analysis attributes this differential to learning from extended trajectories that support subgoal decomposition, persistence, exploration, and error correction.Source: §4.2 · pp. 7–8

5 Related Work

Prior work obtains computer-use trajectories through human annotation, programmatic synthesis, or web-scale mining, but each approach has important limitations. VideoAgentTrek builds on unlabeled-video learning and precise temporal grounding to recover GUI-specific action supervision.

Generating agent trajectories

  • Human annotation provides precise labels but is costly and narrow, while programmatic synthesis scales with exact parameters but may miss real-world UI variability.Web-scale mining offers diverse trajectories but typically lacks precise temporal boundaries or action parameters.Source: §5 · p. 8

Precise Event Grounding in Video

  • General video-grounding systems describe when events occur but generally target semantic interpretation rather than millisecond GUI timing and parameter extraction.The paper positions GUI reconstruction as requiring both precise temporal localization and structured interaction parameters.Source: §5 · p. 8

Learning from unlabeled video to act in environments

  • Unlabeled-video methods such as VPT show that inverse-dynamics auto-labeling can provide effective action-training signals, while later work extends human-policy transfer to interactive environments and humanoid control.VideoAgentTrek applies this broader learning-from-video direction to computer-use trajectories.Source: §5 · p. 8

6 Conclusion

The paper concludes that public screen recordings can be converted into structured computer-use supervision without manual annotation, supporting scalable agent training. Its data-selection standards restrict the source videos to visually stable, relevant, recent, captioned English screen recordings.

6 Conclusion

  • 39,000 YouTube videos yield 1.52 million interaction steps, improving OSWorld-Verified from 9.3% to 15.8% and AgentNetBench from 64.1% to 69.3%.The conclusion presents unlabeled internet videos processed through inverse dynamics as a cost-effective alternative to manual annotation.Source: §6 · p. 8
  • VideoAgentTrek’s release includes ScreenFilter and Video2Action tools for community use in GUI automation research.The conclusion frames these tools as enabling broader use of the mined resource.Source: §6 · p. 8

A YouTube Video Quality Standards

  • The source-video standards require direct horizontal screen recordings with stable framing, limited overlays, English language, captions, and interfaces no older than five years.These criteria are intended to keep videos suitable for detailed analysis and processing.Source: A YouTube Video Quality Standards · p. 11
  • The collection excludes or limits videos with incomplete desktop views, external-camera footage, missing captions, excessive overlays, or outdated interfaces.The stated orientation, recency, and presentation requirements define practical boundaries for usable source videos.Source: A YouTube Video Quality Standards · p. 11

B SCREENFILTER Details

SCREENFILTER filters screen-recorded video for computer-use content, while the dense detector learns temporally grounded GUI actions from interaction logs.

B SCREENFILTER Details

  • SCREENFILTER achieves 89.58% F1, with 90.64% precision and 88.54% recall on held-out data.It is trained on 15,000 synthetic cursor-composited GUI images spanning Windows and macOS patterns.Source: B SCREENFILTER Details · p. 11
  • SCREENFILTER retains video segments with sustained cursor presence and processes approximately 840 hours per GPU-day.It samples at 1–2 frames per second, requiring at least 80% cursor-containing frames for six seconds and using a two-second smoothing gap.Source: B SCREENFILTER Details · p. 11

C Dense Event Detection

  • The dense detector converts OpenCUA interaction logs into prompt-free temporal supervision by mapping, merging, and filtering GUI events.Training uses screen-capture video paired with precisely timestamped mouse and keyboard interactions.Source: C Dense Event Detection · p. 11
  • Training preparation downsamples videos to 4 fps, creates non-overlapping 10-second clips, aligns typed spans, and uses two-frame temporal patches.Action labels are normalized to the paper’s taxonomy before modeling.Source: C Dense Event Detection · p. 11
  • The dense detector is trained by full-parameter supervised fine-tuning of Qwen2.5-VL-7B-Instruct.Its configuration and loss curve are presented in Figure 8.Source: C Dense Event Detection · p. 11

D Action Identification

Action Identification adapts frame sampling to action duration and supplies structured action evidence for inner-monologue generation from GUI demonstrations.

D Action Identification

  • The action-identification data comprise 512,000 screen-capture clips paired with OpenCUA interaction logs and a dynamic frame-rate policy capped at 20 frames per clip.The policy is designed to preserve short, fast interactions.Source: D Action Identification · pp. 11–12

E Inner Monologue Generation

  • Training examples for action parametrization and dense event detection are visualized in Figures 5, 7, and 9.The supplied passages identify these figures as example training samples.Source: Figure 7 · p. 12, Figure 9 · p. 13

D Action Identification

  • Frame sampling varies from 30 fps for brief clicks or scrolls to 4 fps for extended typing while keeping approximately 20 frames.Examples include f =30 at about 0.5 seconds, f ≈20 at about 1 second, and f =4 at about 5 seconds.Source: D Action Identification · p. 12

E Inner Monologue Generation

  • Inner-monologue generation receives action type, parameters, before/after keyframes, surrounding speech, and optional validation descriptions.Action content can include coordinates, bounding boxes, typed text, pressed keys, scrolls, and drag endpoints.Source: E Inner Monologue Generation · pp. 12–13
  • The generated JSON contains only an action description and a first-person thought grounded primarily in visual evidence.The thought covers intent, visible change, atomic actions, and what to verify next, while speech provides weaker contextual evidence.Source: E Inner Monologue Generation · pp. 13–14
  • The action description names the target UI element when inferable and excludes raw coordinates, code, function names, and automation tokens.The thought must remain 4–8 sentences and use first-person language.Source: E Inner Monologue Generation · pp. 13–14

F Computer Use Agent Training

Computer-use-agent training uses two stages: continued pretraining followed by supervised fine-tuning of the resulting checkpoint.

F Computer Use Agent Training

  • Stage 1 performs full-parameter continued pretraining of Qwen2.5-VL-7B-Instruct.The training data and configuration are shown in Figures 10 and 11.Source: F Computer Use Agent Training · p. 14
  • Stage 2 performs full-parameter supervised fine-tuning on the Stage-1-trained checkpoint.The training data and configuration are shown in Figures 10 and 12.Source: F Computer Use Agent Training · p. 14

G Computer Use Agent Evaluation

Evaluation uses standardized live-desktop interaction without manual intervention, and VideoAgentTrek improves both task completion and step success over SFT-only training.

G Computer Use Agent Evaluation

  • OSWorld-Verified evaluates agents on live desktops using natural-language instructions, up to five recent screenshots, and action or rationale history.The 20-step setting averages three independent runs, whereas the 50-step setting uses one run.Source: G Computer Use Agent Evaluation · p. 14
  • All models use identical inference settings and action executors, with no manual interventions during evaluation.This controls the evaluation procedure across compared models.Source: G Computer Use Agent Evaluation · p. 15
  • 0.693 overall step SR is achieved with Stage 1 + Stage 2 on AgentNet Bench, versus 0.641 with SFT-only and 0.385 for the base model.The reported trends associate video pretraining with stronger grounding and multi-action control.Source: G Computer Use Agent Evaluation · p. 15, Table 5 · p. 16
  • 15.78% Task SR at 50 steps is achieved with Stage 1 + Stage 2, versus 9.27% with SFT-only training on OSWorld-Verified.Stage 1 + Stage 2 also reaches 13.6–14.7% at 20 steps, while SFT-only remains around 9.1–9.4%.Source: G Computer Use Agent Evaluation · p. 15, Table 4 · p. 16

H VideoAgentTrek Data Analysis

VideoAgentTrek builds a large, predominantly high-resolution corpus of screen recordings and audits topical relevance using title-and-description classification. The resulting data are primarily tutorials, with action distributions and cross-dataset step counts reported for analysis.

Resolution and scale

Title/description-based content classification

  • The audit classifier assigns each video to tutorial, background, tech talk, or unrelated categories using only its title and brief description.The classifier supports corpus auditing, tag mining, and optional down-weighting rather than hard filtering.Source: H VideoAgentTrek Data Analysis · pp. 16–17

Content classification criteria

  • Tutorials are selected through signals such as step-by-step demonstrations, cursor-driven walkthroughs, UI-operation verbs, and app names paired with action verbs.Background content, tech talks, and unrelated videos are distinguished by incidental screen use, limited live demos, or off-topic content.Source: H VideoAgentTrek Data Analysis · p. 16

Limitations

  • Metadata-only classification can mislabel borderline cases, so final training sets still undergo cursor gating, license and PII checks, and downstream detector screening.Source: H VideoAgentTrek Data Analysis · p. 17

Distribution

Dataset statistics

Related papers

Loading 2510.19488v1…