PaperlayerView exact revision on arXiv

Source-linked AI summary

Watch and Learn: Learning to Use Computers from Online Videos

Chan Hee Song, Yiwen Song, Palash Goyal, Yu Su, Oriana Riva, Hamid Palangi, Tomas Pfister

arXiv:2510.04673v3cs.AIcs.CV

TL;DR

Computer-using agents need high-quality demonstrations for workflows across diverse, changing interfaces, but collecting them manually is costly and existing data can be limited. Watch & Learn recovers actions from consecutive screen states in online videos to create reusable UI trajectories. The trajectories improve agents on OSWorld and support leading 7B-scale performance on WindowsAgentArena under its 15-step limit.

  • Problem

    High-quality computer-use demonstrations are costly to collect at scale, limiting the data available for agents that must handle diverse, changing interfaces.

    Source: §1 · p. 1

  • Method

    Watch & Learn predicts actions from consecutive screen observations, then uses task-aware video retrieval to turn online demonstrations into executable trajectories.

    Source: §1 · pp. 2–3

  • Results

    W&L trajectories consistently improve OSWorld results and achieve leading performance among 7B-scale models on WindowsAgentArena under the 15-step limit.

    Source: §1 · pp. 2–3, §5 · p. 12

  • Takeaways & Limitations

    The results support web-scale human demonstration videos as a practical, scalable source of supervision that generalizes across operating systems and interface conventions.

    Source: §5 · p. 12

    W&L targets vision-only interaction, using raw screen pixels rather than internal UI representations such as HTML or accessibility trees.

    Source: §3 · pp. 4–5

Abstract

from arXiv · show

Computer-using agents (CUAs) must plan task workflows across diverse and evolving applications, yet progress is limited by the lack of large-scale, high-quality training data. Existing datasets are narrow, static, and costly to annotate, while synthetic data often yields oversimplified or misaligned behaviors. We present Watch & Learn (W&L), a framework that converts readily available Internet videos of human computer use into executable UI trajectories at scale. Instead of directly generating actions or relying on handcrafted heuristics, we cast trajectory annotation as an inverse dynamics problem that predicts user actions from consecutive screen states, which simplifies learning and generalizes across domains. Through a task-aware retrieval and labeling pipeline, W&L yields over 53K high-quality trajectories that enhance CUAs both as in-context exemplars and as supervised training data. On OSWorld, it consistently improves general-purpose and specialized CUAs, while on WindowsAgentArena it achieves state-of-the-art performance among 7B-scale models under the 15-step limit. These results show that web-scale human demonstration videos can serve as a practical and scalable foundation for advancing real-world CUAs.

1. Introduction

Watch & Learn (W&L) addresses the cost and scale limits of collecting high-quality computer-use demonstrations by recovering actions from visual transitions in online videos. Its trajectories support both in-context use and supervised training, improving CUA results on OSWorld and WindowsAgentArena.

1. Introduction

  • Manual demonstration collection is costly: AgentNet’s 22K tasks across three operating systems and 200-plus applications took six months and cost over $32,000.Scaling manual annotation to millions of trajectories would take substantial additional time and cost well over half a million dollars, even for well-funded labs.Source: §1 · p. 1
  • W&L predicts the action between consecutive screen observations, avoiding handcrafted parsing heuristics and using a one-step prediction problem that is easier to learn.The authors report that this formulation generalizes across applications.Source: §1 · p. 2
  • A task-aware retrieval and labeling pipeline turns web videos into over 53K executable trajectories, without manual annotation.The reported collection spans three operating systems and a broad range of applications and task categories.Source: §1 · p. 2
  • The trajectories serve both as in-context exemplars that supply application-specific knowledge and as supervised training data for CUAs.The paper reports gains for both uses across general-purpose and specialized agents.Source: §1 · pp. 2–3
  • W&L’s trajectories improve OSWorld performance and give fine-tuned models leading performance among 7B-scale models on WindowsAgentArena under the 15-step limit.Open-weight models see gains of up to 11 points; OSWorld in-context use improves by up to 3 percentage points.Source: §1 · pp. 2–3

2. Related Work

Prior work synthesizes computer-use data through exploration, tutorials, and self-improving agents, but these approaches face noise, brittleness, narrow task distributions, or retraining costs. W&L mines videos with inverse-dynamics action labeling, while its in-context-learning framing highlights the need to select demonstrations efficiently for long tasks.

2.1. Data Synthesis for Computer-Using Agents

  • Exploration-based, tutorial-driven, and self-improving synthesis broaden data collection, but are associated with noisy labels, brittle LLM reliance, narrow task distributions, or costly iterative retraining.The cited examples include exploration and retroactive instruction labeling, tutorial conversion, and automated task-and-reward generation.Source: §2.1 · p. 3
  • W&L also mines web videos, but infers actions from screen transitions rather than labeling videos with multimodal LLMs, producing trajectories for both training and in-context reasoning.The paper describes this as complementary to prior synthesis efforts.Source: §2.1 · p. 3

2.2. In-context Learning for Agents

  • In-context learning can adapt agents without parameter updates, but long demonstrations add computational overhead and latency, motivating efficient selection and leaving web-scale videos underexplored as examples.The cited work also notes that effectiveness depends on model scale and context-window size, especially for long-horizon tasks.Source: §2.2 · pp. 3–4

3. Method

W&L uses raw screen pixels to recover actions, label tutorial videos, and create executable trajectories without manual labeling. These trajectories can be retrieved as in-context demonstrations or aggregated as supervised training data.

3. Method

  • The method focuses on raw screen pixels and structured actions, avoiding dependencies on application-specific APIs or potentially incomplete and inconsistent UI representations.This vision-only setting is presented as a way to maximize generality and scalability.Source: §3 · pp. 4–5

3.1. Inverse Dynamics Model

  • The inverse dynamics model predicts a user action from consecutive screen observations, trained on roughly 600K transitions collected through automated interactions with live web pages.Its six atomic primitives are Click, Release, Scroll, Type, Wait, and Move; together they represent interactions such as drag-and-drop compositionally.Source: §3.1 · p. 5

3.2. Trajectory Data Generation from Videos

  • The video pipeline retrieves tutorials, filters out unusable footage, and applies the model to consecutive frames to assemble executable trajectories without manual annotation.The collection covers 69 applications across seven main categories; qualified videos are retained for training, while inference-time use keeps only the top three that pass filtering.Source: §3.2 · pp. 6–7, Table 1 · p. 7

3.3. Applications of Trajectories

  • For in-context learning, trajectories become observation-action examples augmented with reasoning traces; for fine-tuning, they become state-action sequences used to train multimodal models.The paper applies these uses to general-purpose Qwen 2.5-VL and specialized UI-TARS-1.5.Source: §3.3 · p. 7

4. Experiments and Analysis

The experiments test W&L trajectories as in-context exemplars and fine-tuning data across OSWorld and WindowsAgentArena. Results show gains across model categories, with label accuracy, trajectory content, and retrieval quality examined as potential contributors.

4.4. Main Results

  • On OSWorld, Qwen rises from 1.9 to 13.0 (+11.1); on WindowsAgentArena, UI-TARS rises from 18.1 to 24.0 (+5.9), leading 7B models.The WindowsAgentArena comparison uses a 15-step limit; on OSWorld, general-purpose multimodal models gain +1.6 to +3.0 points and Jedi gains +2.2 points.Source: Table 2 · p. 8, §4.4 · pp. 9–10, Table 3 · p. 9

4.1. Experimental Setup

  • OSWorld-Verified uses Ubuntu desktop tasks with a 50-step limit, while WindowsAgentArena tests Windows tasks under a 15-step limit.The two benchmarks are used to assess generalization across operating systems and interface distributions.Source: §4.1 · p. 8

4.2. Datasets

  • The video-derived dataset contains 53,125 trajectories from 53,125 instructional videos, spanning 69 applications.Applications cover productivity, programming, design, audio production, system utilities, and scientific/data domains.Source: §4.2 · p. 8

4.3. Models

  • General multimodal models and Jedi use retrieved trajectories as ICL exemplars; UI-TARS-1.5-7B and Qwen 2.5-VL 7B receive SFT.Jedi combines the o3 planner with the Jedi-7B grounding model.Source: §4.3 · p. 9

4.4.1. Effect of Label Accuracy on Performance

  • On the held-out action-label test set, W&L’s IDM outperforms Gemini and TongUI across six primitive action types.ActionType Accuracy measures action classification; Action Accuracy also checks argument correctness.Source: §4.4.1 · pp. 10–11, Table 4 · p. 11
  • On WindowsAgentArena, fine-tuning with TongUI-labeled data decreases success rate by 5.2 points, whereas IDM-labeled trajectories yield consistent gains, supporting more accurate and consistent supervision.IDM-derived labels remain reliable across Ubuntu-style OSWorld and Windows-style WindowsAgentArena interfaces.Source: §4.4 · p. 10, §4.4.1 · p. 11

4.4.2. Effect of Trajectories on In-Context Learning

  • On OSWorld, adding action labels improves ICL over frame-only exemplars, and adding natural-language reasoning brings further gains across tested models.The ablation compares consecutive frames, frames with predicted actions, and frames with actions plus Gemini 2.5 Flash reasoning.Source: §4.4.2 · p. 11, Table 5 · p. 12

4.4.3. Effect of Video Retrieval on In-Context Learning

  • Random retrieval neither improves nor degrades o3 relative to its base model, while task-relevant retrieval improves performance.The authors suggest retrieval quality controls the strength of the positive effect, while accurate labels keep random exemplars from introducing significant noise.Source: Table 6 · p. 12, §4.4.3 · p. 12

5. Conclusion and Future Work

W&L converts web-scale human demonstration videos into UI trajectories using a vision-only inverse dynamics model and task-aware retrieval. Its experiments support the use of over 53K automatically labeled trajectories across in-context learning and fine-tuning, with benefits across operating systems and interface conventions.

5. Conclusion and Future Work

  • W&L combines a vision-only inverse dynamics model with task-aware video retrieval to convert web-scale human demonstrations into UI trajectories.The resulting trajectories are used in both in-context learning and supervised fine-tuning.Source: §5 · p. 12
  • Experiments on OSWorld and WindowsAgentArena show that IDM-labeled trajectories provide supervision that generalizes across operating systems and interface conventions.The framework produces over 53K automatically labeled trajectories and benefits general-purpose multimodal models as well as open-source CUAs.Source: §5 · p. 12
  • Future work will explore combining or segmenting tutorials for long-horizon workflows and using trajectories in reinforcement learning.Proposed uses include demonstrations, replay buffers, and reward-modeling priors.Source: §5 · p. 12

Appendices

The supplementary material supplies additional details and clarifications omitted from the main text due to space constraints.

Appendices

  • The appendices cover limitations, dataset and implementation details, IDM architecture and training, and additional results.The supplementary material identifies these topics as Appendices A through F.Source: Appendices · p. 17

A. Limitations

The framework has two stated limitations: its IDM lacks shortcut actions, and retrieval operates on full tasks rather than execution-sized segments; the authors view both as opportunities for extension.

A. Limitations

  • The IDM omits keyboard shortcuts such as Ctrl+C and Ctrl+Z because web transition data rarely captures them, though graphical equivalents retain near-complete task coverage.Application-based collection or key-event logging could add shortcut support.Source: A. Limitations · p. 17
  • Full-task retrieval may not match an agent’s execution granularity; merging shorter tasks or splitting tutorials could enable finer retrieval and improve adaptability.These are proposed future directions, not capabilities already implemented.Source: A. Limitations · p. 17
  • The authors characterize these limitations as opportunities to enhance scalability and generality, rather than fundamental barriers.Source: A. Limitations · p. 17

B. Dataset Details

The dataset spans seven application categories selected to cover realistic computer-use domains, with applications chosen for adoption, tutorial availability, and interaction diversity.

B. Dataset Details

  • Seven categories span productivity, programming, design, screen editing, audio production, system utilities, and science and data; applications were selected for adoption, tutorial availability, and diverse interaction challenges.The collection is not restricted to these applications: additional tutorials can supply data; Table 7 groups the applications by category.Source: §B.1 · pp. 17–18, Table 7 · p. 18

C.1. State-Transition Data Collection

The pipeline builds a state-transition corpus, retrieves and filters tutorial videos, and formats demonstrations for in-context use; model and action-space choices support executable interactions.

C.1. State-Transition Data Collection

  • About 600K transitions were collected over 3.5 days using 48 workers, with sampled web actions recorded as screenshot transitions and filtered for meaningful visual change and execution failures.Collection used diverse entry points, capped visits per domain, and limited sessions to 15 actions.Source: §C.1 · pp. 18–19

C.2. Video Retrieval

  • Task descriptions and related applications guide concise, tutorial-focused YouTube search queries to improve retrieval precision and reduce unrelated results.Source: §C.2 · p. 19

C.3. Video Filtering

  • A visual classifier labels tutorial frames and scores quality; videos are retained when their average frame score exceeds 0.8, filtering noisy content while preserving tutorial coverage.Source: §C.3 · pp. 19–20

C.4. In-context Learning

  • For OSWorld in-context evaluation, three relevant trajectories are formatted as observation-thought-action examples and prepended to the agent’s existing prompt without changing its logic.Source: §C.4 · p. 20

C.5. Models

  • In-context evaluations use public Gemini 2.5 Flash, OpenAI o3, and Claude 4 Sonnet versions with deterministic decoding; IDM and CUA training follow specified optimizer and hardware recipes.IDM training uses AdamW for 15 epochs on 8×A100 GPUs; CUA fine-tuning follows the official UI-TARS-1.5 and Qwen 2.5-VL recipes.Source: §C.5 · p. 21

C.6. Action Space and Primitive Composition

  • The IDM predicts six primitive GUI actions, which a deterministic composition layer maps into executable commands; visually indistinguishable sequences resolve to the minimal consistent primitive sequence.For example, Click→Release maps to a mouse click, while Click→Move→Release supports drag interactions.Source: §C.6 · pp. 21–22

D. Inverse Dynamics Model Architecture

The IDM encodes consecutive screenshots and uses specialized heads to predict action type, coordinates, and text, with visual conditioning for text generation.

D.1. Vision Encoder

  • NaFlex SigLIP-2 processes each 1000×1000 screenshot into 1024 visual tokens, preserving spatial detail for small interface elements while avoiding fixed-resolution downsampling.The authors chose NaFlex for its effective resolution and text-heavy visual understanding despite fixed-resolution variants’ additional distillation stages.Source: §D.1 · p. 22

D.2. Transformer Backbone

  • A four-layer bidirectional transformer fuses patch features from both observations with temporal embeddings; its [CLS] output represents the action globally, while patch outputs support coordinate prediction.Source: §D.2 · pp. 22–23

D.3.1. Action Classification Head

  • A two-layer MLP predicts a probability distribution over the six primitive action types.Source: §D.3.1 · p. 23

D.3.2. Coordinate Prediction Head

  • Separate heads classify normalized x- and y-coordinates into 1000 bins, turning location prediction into classification and supporting multiple valid click locations and calibrated uncertainty.The 1000-bin discretization aligns with the 1000×1000 input resolution.Source: §D.3.2 · p. 23

D.3.3. Language Head

  • A GPT-2-based autoregressive decoder predicts typed text and uses cross-attention to condition generation on features from both screenshots.The reported category-wise gains are strongest where specialized tutorials are abundant and smaller for heavy text entry, rare actions, or fine-grained control.Source: §D.3.3 · pp. 23–24, Table 8 · p. 24

E. Inverse Dynamics Model Training

The inverse dynamics model predicts action primitives, coordinates, and typed text from consecutive screen observations, using only the applicable losses for each action. On OSWorld, performance rises sharply with more training trajectories, reaching 13.0% with the full dataset.

E.1. Model Outputs

  • From consecutive screen observations, the IDM predicts an action primitive, optional coordinates, and typed text, routing each action only to the applicable output heads.Coordinate predictions use discretized pixel bins; typed text is predicted autoregressively.Source: §E.1 · p. 24, E. Inverse Dynamics Model Training · p. 24

E.2. Overall Objective

  • The end-to-end objective sums the losses for applicable prediction heads, with classification, coordinate, and language-loss weights set to 1 by default.Loss terms are included only when the action has the corresponding supervision signal.Source: §E.2 · p. 24

E.2.1. Classification Loss

  • The classification loss trains the IDM to infer the categorical action from the observed state transition.It uses standard cross-entropy supervision for the action primitive.Source: §E.2.1 · p. 25

E.2.2. Coordinate Loss

  • For location-based actions, coordinate loss sums independent cross-entropy terms for normalized x and y coordinates discretized into 1,000 bins each.The loss is omitted for actions that do not require coordinates.Source: §E.2.2 · p. 25

E.2.3. Language Modeling Loss

  • For text-entry actions, the language loss is sequence negative log-likelihood over autoregressively predicted tokens, and it is skipped for non-text actions.Together, the losses reconstruct action primitives, spatial arguments, and typed text from UI transitions.Source: §E.2.3 · p. 25

F.1. What is the effect of data scale for supervised fine-tuning?

  • 13.0% success with the full dataset, versus 1.9% for the base model, on OSWorld with Qwen 2.5-VL.Success rates rise to 3.3% with 10k trajectories and 4.9% with 25k; the authors suggest substantial gains emerge after a critical mass of data.Source: Table 9 · p. 25, §F.1 · p. 25

F.2. Which application domains benefit most from our data?

The domain breakdown shows the largest benefits in applications where online tutorials are plentiful and align with the agent’s action space. Gains are smaller in text-heavy or fine-grained interaction domains.

F.2. Which application domains benefit most from our data?

  • The largest improvements occur in Chrome, GIMP, and VLC, where step-by-step tutorials cover specialized procedures such as browser configuration, image editing, and media-player settings.The authors report that these demonstrations enable extraction of trajectories that transfer to downstream agents.Source: §F.2 · p. 26
  • Gains are smaller in VS Code and OS tasks requiring extensive text entry or code manipulation, which the current action set captures less easily.Tutorials are also scarce for Thunderbird and LibreOffice tasks that may involve fine-grained dragging or small interface elements.Source: §F.2 · p. 26
  • Benefits are largest where web tutorials are plentiful and aligned with the agent’s action space, leaving room to extend support for text-heavy and fine-grained interactions.The domain analysis compares general-purpose models, an agentic framework, and an open-source model with and without W&L data.Source: §F.2 · p. 26

Related papers

Loading 2510.04673v3…