PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, Tao Yu

arXiv:2412.09605v2cs.CL

TL;DR

GUI agents remain suboptimal in real-world scenarios, and synthesizing grounded, multimodal trajectories is challenging. AgentTrek transforms web tutorials into verified agent trajectories; its synthesized data is reported to outperform existing datasets across multiple benchmarks.

  • Problem

    GUI agents remain suboptimal in real-world scenarios, while synthesizing trajectories is difficult because instructions, visual observations, and grounded actions must be integrated.

    Source: §1 · pp. 1–2

  • Method

    AgentTrek harvests and filters web tutorials, structures them as step-by-step agent tasks, and uses a VLM agent and evaluator to generate and verify trajectories.

    Source: §1 · p. 2

  • Results

    Across multiple benchmarks, agents trained on AgentTrek data outperform those trained on existing datasets in textual and visual browsing, with a cost-per-trajectory of $0.551.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present automatic, low-cost trajectory synthesis as a promising direction for future LLM-agent development.

    Source: §6 · p. 10

    The current task coverage spans 11 categories; adapting to website changes and broader task coverage remain future work.

    Source: §4 · p. 9

Abstract

from arXiv · show

Graphical User Interface (GUI) agents can automate complex tasks across digital environments, but their development is hindered by the scarcity of high-quality trajectory data for training. Existing approaches rely on expensive human annotation, making them unsustainable at scale. We propose AgentTrek, a scalable data synthesis pipeline that generates web agent trajectories by leveraging publicly available tutorials. Our three-stage method: (1) automatically harvests and filters tutorial-like texts from the internet using a specialized classification model, (2) transforms these texts into structured task specifications with step-by-step instructions, and (3) employs a visual-language model (VLM) agent to execute these instructions in real environments, while a VLM-based evaluator verifies trajectory correctness. The synthesized trajectories encompass multiple modalities, including text-based HTML observations with function-calling API actions, and vision-based screenshot observations with pixel-level actions. This multimodal data, enriched with chain-of-thought reasoning, enables agents to achieve state-of-the-art performance on both textual web browsing benchmarks (e.g., WebArena) and visual web grounding and browsing benchmarks (e.g., ScreenSpot Web and Multimodal Mind2Web). Furthermore, our fully automated approach significantly reduces data collection costs, achieving a cost of just $0.55 per high-quality trajectory without human annotators. Our work demonstrates that guided replay using web tutorials is a practical and scalable strategy for training advanced GUI agents, paving the way for more capable and autonomous digital assistants.

1 INTRODUCTION

High-quality GUI-agent trajectories are scarce and costly to annotate, while synthesizing grounded, multimodal interactions remains challenging. AgentTrek uses web tutorials to generate and verify trajectories for training GUI agents.

1 INTRODUCTION

  • GUI agents need multi-step trajectories joining goals, observations, reasoning, and grounded actions, but such data is scarce and costly to annotate.These trajectories support the complex, sequential decisions required for GUI interaction.Source: §1 · p. 2
  • Synthesizing GUI trajectories is challenging because instructions, visual observations, and context-specific actions must be integrated and accurately grounded.Source: §1 · p. 2
  • AgentTrek harvests tutorial-like web text, structures it into objectives and step-by-step instructions, then uses a VLM agent and evaluator to generate and verify trajectories.Source: §1 · p. 2
  • Across multiple benchmarks, AgentTrek-trained agents outperform agents trained on existing datasets in textual and visual browsing, at a cost-per-trajectory of $0.551.The reported cost is compared with traditional human-annotated data pipelines.Source: §1 · p. 2

2 METHOD

AgentTrek turns internet tutorials into structured tasks, replays them with a VLM in real web environments, evaluates the resulting trajectories, and uses them to train text- and vision-based agents. Its pipeline combines filtering and standardization with multimodal trace collection and automated quality assessment.

2.1.1 PREFILTER FUNCTION

  • 92.69% recall on 285 manually labeled samples let the prefilter reduce 20.8 billion entries to 68.8 million.It uses action, interface, and platform keywords, content-length thresholds, and tutorial-like URL patterns.Source: §2.1.1 · p. 4

2.1.2 LLM LABELER; 2.1.3 FASTTEXT FILTER

  • 88.5% F1 for GPT-4O MINI labeling and 89.5% F1 for FastText successively filtered the prefiltered corpus, leaving about 18.8 million deduplicated likely tutorials.The labeler processed 90,000 entries, which supplied FastText training data.Source: §2.1.3 · p. 4, §2.1.2 · p. 4

2.1.4 TUTORIAL STANDARDIZATION: TAG & PARAPHRASE

  • GPT-4O MINI standardizes tutorials into platform, task, prerequisites, step-by-step instructions, and expected-outcome fields.The pipeline uses gold-standard examples to guide prompts and costs approximately $0.89 per 1,000 entries.Source: §2.1.4 · pp. 4–5

2.2.1 TRAJECTORY DATA SCHEMA

  • Each trajectory combines task metadata, screenshots and video, reproducible Playwright traces, and post-processed observations, reasoning, and actions.The native trace includes DOM snapshots, HTML structure, network flow, and precise action sequences.Source: §2.2.1 · p. 5

2.2.2 GUIDED REPLAY WITH TUTORIALS

  • In BrowserGym, a VLM replays tagged tutorials on target websites using screenshots and the accessibility tree, while the system records actions, coordinates, screenshots, DOM snapshots, and reasoning.The agent uses Playwright actions; reported usage averages approximately 8,027 tokens per interaction step and 86,114 per task.Source: §2.2.2 · p. 5

2.2.3 EVALUATION OF TRAJECTORY

  • The GPT-4O evaluator scores instruction adherence and completion of key task components, providing trajectory-level and step-level assessments and identifying the earliest failure when applicable.Its validation involved human review of 1,081 trajectories, including 558 samples with detailed human-annotated justifications; the authors report that it often applies stricter criteria than human reviewers.Source: §2.2.3 · p. 6

2.3 TRAINING WITH TRAJECTORY DATA; 2.3.1 VISION-BASED WEB AGENT; 2.3.2 TEXT-BASED WEB AGENT

  • Text-based agents use accessibility-tree observations and Playwright actions, while vision-based agents act on screenshots with pixel-coordinate commands.The vision agent maps Playwright actions to pyautogui commands; the text agent uses semantic structure for interactions such as forms and nested navigation.Source: §2.3 · p. 6, §2.3.2 · p. 7, §2.3.1 · p. 7

2.3.3 MODEL ARCHITECTURE AND TRAINING

  • Qwen2-VL was fine-tuned on 10,000 AgentTrek trajectories, while Qwen2.5 models at 7B and 32B scales used 6,000 trajectories.The vision training targets visual grounding and multi-step planning; text training pairs accessibility-tree observations with Playwright actions.Source: §2.3.3 · p. 7

3 EXPERIMENTS

The experiments evaluate AgentTrek-trained agents on WebArena, ScreenSpot Web, and Multimodal Mind2Web. The reported results show gains in textual web task completion, visual grounding, and visual web tasks, with Mind2Web results strongest when AgentTrek data is combined with Mind2Web training data.

3.1 EXPERIMENTAL SETUP

  • The evaluation uses WebArena for textual web tasks, ScreenSpot Web for visual grounding, and Multimodal Mind2Web for cross-task, cross-website, and cross-domain settings.The ScreenSpot evaluation focuses on web performance to match AgentTrek's domain.Source: §3.1 · p. 7, §3 · p. 7

3.2 MAIN RESULTS

  • AgentTrek-trained models significantly outperform open-source baselines and GPT-4o on WebArena, including on unseen self-hosted websites.The authors report this as evidence of generalization to novel domains.Source: §3.2 · p. 8, Table 4 · p. 8
  • ScreenSpot performance more than doubled on both text-based and icon-based tasks after Qwen2-VL fine-tuning with AgentTrek data.The fine-tuned model also surpassed several competitive baselines.Source: §3.2 · p. 8, Table 5 · p. 8
  • On Multimodal Mind2Web, combining AgentTrek and Mind2Web training data yielded the strongest results across all metrics.The Qwen2-VL-7B baseline was excluded because of insufficient grounding capabilities.Source: §3.2 · p. 8, Table 6 · p. 8

4 ANALYSIS

AgentTrek’s analysis emphasizes dataset diversity, realism, and comprehensiveness, while examining scaling, cost, comparisons with existing datasets, and remaining challenges.

Diversity and Scale

  • 10,398 successful trajectories were collected from 23,430 filtered tutorials, spanning 127 websites and 11 task categories.Figure 8 depicts the website and domain distribution.Source: §4 · p. 9, Figure 8 · p. 10

Effectiveness of Data Scaling

  • 45.0% cross-domain step success rate at 100% data, versus 39.5% at 20%, approaches the 47.7% Mind2Web human-annotated training-split metric.The comparison is shown in Figure 7.Source: Figure 7 · p. 9, §4 · p. 9

Realism through Authentic Environments

  • 52% of tutorial-guided tasks produced effective trajectories, versus 15.78% with high-level goals alone, across 400 tasks.The authors report 208 and 63 effective trajectories, respectively.Source: §4 · p. 9

Comprehensiveness and Cost-Efficiency

  • The dataset combines DOM/HTML, AXTree, video, screenshots, and reasoning chains, averaging 12.1 steps per trajectory at $0.551 per trajectory.Table 3 provides a cost breakdown.Source: §4 · p. 9

Comparison with Existing Datasets

  • AgentTrek’s nearly 5,000 verified trajectories exceed most compared datasets in scale and step complexity, with text, vision, and reasoning modalities.The authors contrast its broader modalities with the narrower observation-action pairs of Mind2Web and WebShop.Source: §4 · p. 9

Research Challenges

  • The current dataset covers 11 task categories, and adapting to website changes and broader task coverage remains future work.The pipeline uses filtering, standardization, and VLM-based evaluation to address quality and noisy-content challenges.Source: §4 · p. 9

5 RELATED WORK

Related work covers GUI agents’ trajectory-data needs, synthetic trajectory datasets, and automated evaluation; AgentTrek is positioned across these areas.

5.1 LLM-BASED AGENTS

  • GUI agents still struggle with sequential decisions, and AgentTrek addresses the shortage of specialized trajectory data with multimodal trajectories.The discussion situates this need alongside web, desktop, and mobile agents.Source: §5.1 · p. 10

5.2 AGENT TRAJECTORY DATA

  • AgentTrek offers a fully automated, diverse, realistic trajectory pipeline at $0.551 per trajectory, addressing scale and adaptability limits of human-annotated datasets.The discussion contrasts it with synthetic-generation efforts including BAGEL and NNetNav.Source: §5.2 · p. 10

5.3 AUTOMATED EVALUATION OF DIGITAL AGENTS

  • AgentTrek uses GPT-4o as a VLM evaluator to assess trajectories from task descriptions, actions, and reasoning chains.The discussion frames this as scalable, transparent quality control.Source: §5.3 · p. 10

6 CONCLUSION

The conclusion presents AgentTrek as an efficient pipeline for generating comprehensive, cost-effective trajectory data and a diverse dataset validated through model training and evaluation.

6 CONCLUSION

  • AgentTrek automatically generates comprehensive, cost-effective agent trajectory data.The conclusion describes it as an efficient pipeline.Source: §6 · p. 10
  • The authors present a large, diverse dataset generated by the pipeline and validate it by training models and evaluating their performance.Source: §6 · p. 10
  • The authors characterize automatic, low-cost trajectory synthesis as a promising direction for future LLM-agent development.They describe AgentTrek as a strong standard for agent data generation.Source: §6 · p. 10

A CALCULATION OF OTHER TRAJECTORY DATASETS •

The appendix lists source materials and calculation bases for statistics on several trajectory datasets.

A CALCULATION OF OTHER TRAJECTORY DATASETS •

  • Dataset statistics draw on WebLINX tables and training-set information, dataset repositories, and published calculations for RUSS, ScreenAgent, Mind2Web, WebShop, and WonderBread.For WebLINX, the calculation excludes “say” actions, yielding 18,249 non-say actions across 969 demos.Source: A CALCULATION OF OTHER TRAJECTORY… · p. 15

B ANALYSIS OF THE EFFECTIVENESS OF TUTORIALS

The section identifies tutorial guidance that supports replay and reports the cost of producing verified trajectories. It attributes the cost calculation to the web-tutorial ratio, replay success rate, and phase costs.

B ANALYSIS OF THE EFFECTIVENESS OF TUTORIALS

  • Tutorials support replay by providing a target URL, human-informed steps, and paths through multi-level menus.These factors are identified as contributors to improved replay effectiveness.Source: B ANALYSIS OF THE EFFECTIVENESS OF… · p. 15

C COST DETAILS

  • The verified-effective-trajectory cost calculation uses a 0.275 web-tutorial ratio and a 39.9% Replay Success Rate.The cost expression also lists web ratio, replay price, and evaluation price.Source: C COST DETAILS · p. 15
  • $550.75 is the cost per 1,000 verified effective trajectories.The cost breakdown is presented in Table 7.Source: C COST DETAILS · p. 15, Table 7 · p. 15

D EVALUATOR ALIGNMENT

The section compares the automatic trajectory evaluator with human and other automatic evaluators, and introduces a mapping between Playwright and PyAutoGUI action spaces.

D EVALUATOR ALIGNMENT

  • The evaluator-alignment section compares metrics between human and automatic evaluators.The section labels its metric as trajectory evaluator accuracy.Source: D EVALUATOR ALIGNMENT · p. 15
  • The Web Tutorials VLM Evaluator scores 84.0%, versus 80.6% for GPT-4V, 82.1% for Captioner + GPT-4, and 74.4% for Captioner + Mixtral.Figure 9 presents evaluator performance on a human-annotated validation set across scenarios.Source: D EVALUATOR ALIGNMENT · p. 15, Figure 9 · p. 16

E ACTION MAPPING

  • Table 8 maps between the Playwright and PyAutoGUI action spaces.The supplied passage identifies the table's scope but does not specify its mappings.Source: Table 8 · p. 16

F EXPERIMENTAL RESULTS ON TEXTUAL DATA

This block reports textual-data results and supplementary details on tutorial filtering, benchmark scope, replay failures, and scaling. It describes steadily improving Multimodal-Mind2Web performance as more AgentTrek data is used.

F EXPERIMENTAL RESULTS ON TEXTUAL DATA

  • On WebArena and Miniwob++, Qwen2.5-7B-Instruct fine-tuned on AgentTrek textual data ranks highest among open-source web agents and approaches GPT-4o.The WebArena evaluation uses self-hosted websites that are out-of-domain from the AgentTrek dataset.Source: F EXPERIMENTAL RESULTS ON TEXTUAL DATA · p. 16

G.1 PREFILTER FUNCTION •

  • The content must contain at least 20 common keywords, including at least 4 distinct common keywords, and at least one mandatory keyword must appear at least twice to demonstrate topic relevance.Source: §G.1 · pp. 16–17

G.2 LLM LABELER PROMPT

  • The LLM labeler prompt assesses whether a URL and its context meet the tutorial requirements.Figure 11 shows the user prompt for classifying GUI tutorials.Source: §G.2 · p. 18, Figure 11 · p. 18

H EXAMPLES OF FAILED GUIDED REPLAY TRAJECTORIES

  • One failed replay could not complete a booking because the tutorial had expired before the actual date.Figure 13 illustrates this failure case.Source: Figure 13 · p. 20

I SCALING UP AGENTTREK

  • With more than 10K AgentTrek trajectories, Multimodal-Mind2Web Step SR improves steadily from 20% to 100% training-data usage, peaking with the full dataset.The evaluation covers three splits; Table 9 reports Step SR across data amounts.Source: I SCALING UP AGENTTREK · p. 21, Table 9 · p. 21

J EVALUATION BENCHMARKS

  • The section introduces details of the evaluation benchmarks used in the work.The supplied passage does not specify further benchmark details.Source: J EVALUATION BENCHMARKS · p. 21

J.1 GUI GROUNDING EVALUATION

  • ScreenSpot evaluates visual grounding using 1.2K single-step instructions with target-element coordinates across mobile, desktop, and web platforms.It includes original-instruction and self-plan scenarios.Source: §J.1 · p. 21

J.2 OFFLINE GUI AGENT EVALUATION

The section defines Multimodal-Mind2Web metrics for offline GUI-agent planning and describes how guided replay grounds tutorial instructions in webpage elements and executes actions.

J.2 OFFLINE GUI AGENT EVALUATION

  • Multimodal-Mind2Web offline planning is measured with Element Accuracy, Operation F1, and Step Success Rate; comparison methods include HTML, HTML+Image, SoM, and image-only observations.For image-only evaluation, Qwen2VL identifies target objects from webpage screenshots by generating coordinates.Source: §J.2 · p. 21

K DETAILED DESCRIPTION OF GUIDED REPLAY

  • Guided replay uses the accessibility tree as its primary observation, with each element identified by a unique bid.The webpage also provides HTML structure and screenshots; tutorial descriptions can help identify the target element.Source: K DETAILED DESCRIPTION OF GUIDED REPLAY · p. 22
  • Actions specify the action type and target bid, which the system uses to retrieve the webpage element and execute a Playwright operation.The described action does not require fine-grained coordinates.Source: K DETAILED DESCRIPTION OF GUIDED REPLAY · p. 22
Loading 2412.09605v2…