PaperlayerView exact revision on arXiv

Source-linked AI summary

Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents

Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, Ahmed Awadallah

arXiv:2502.11357v4cs.AIcs.HC

TL;DR

Web agents need larger and more diverse trajectory data because existing datasets are difficult to scale and provide limited coverage for realistic online settings. Explorer uses exploration-driven, multi-agent synthesis with iterative refinement and verification to build such data. Its 94K-trajectory dataset spans 49K URLs at $0.28 per successful trajectory, and Explorer outperforms existing baselines on online and offline benchmarks.

  • Problem

    Existing web trajectory datasets lack scalable, diverse coverage, while human annotation is expensive and many synthetic datasets rely on limited task proposals or domains.

  • Method

    Explorer explores real websites with proposer, refiner, summarizer, and verifier agents to synthesize grounded multimodal trajectories while refining task intent.

  • Results

    Explorer produces 94K trajectories spanning 49K URLs, 720K screenshots, and 33M web elements, and outperforms existing web-agent baselines on online and offline benchmarks.

  • Takeaways & Limitations

    The results highlight data scale as a key driver of web-agent performance and make larger-scale LMM-based agent research more accessible.

  • Takeaways & Limitations

    Autonomous exploration can produce incoherent tasks, closed-source LLM use incurs API costs, and login-protected websites may yield insufficient data.

Abstract

from arXiv · show

Recent success in large multimodal models (LMMs) has sparked promising applications of agents capable of autonomously completing complex web tasks. While open-source LMM agents have made significant advances in offline evaluation benchmarks, their performance still falls substantially short of human-level capabilities in more realistic online settings. A key bottleneck is the lack of diverse and large-scale trajectory-level datasets across various domains, which are expensive to collect. In this paper, we address this challenge by developing a scalable recipe to synthesize the largest and most diverse trajectory-level dataset to date, containing over 94K successful multimodal web trajectories, spanning 49K unique URLs, 720K screenshots, and 33M web elements. In particular, we leverage extensive web exploration and refinement to obtain diverse task intents. The average cost is 28 cents per successful trajectory, making it affordable to a wide range of users in the community. Leveraging this dataset, we train Explorer, a multimodal web agent, and demonstrate strong performance on both offline and online web agent benchmarks such as Mind2Web-Live, Multimodal-Mind2Web, and MiniWob++. Additionally, our experiments highlight data scaling as a key driver for improving web agent capabilities. We hope this study makes state-of-the-art LMM-based agent research at a larger scale more accessible.

1 Introduction

Existing web trajectory datasets lack the diversity and scale needed for realistic web-agent training, while human annotation is expensive to scale. Explorer addresses this gap with exploration-driven synthesis and a large multimodal dataset that supports strong benchmark performance.

  • Research gap: Existing trajectory datasets are mostly human-annotated or use synthetic data mainly for task-proposal curation, limiting scalable collection.Human annotation is expensive to scale for large and diverse training datasets.
  • Research gap: Collecting trajectory-level datasets requires diverse task intents, real-world multi-step interaction, and verification of task completion.These requirements make trajectory synthesis more challenging than task proposal generation alone.
  • Research gap: Homepage-based task proposals and tutorial-supervised trajectories overlook deeper web content, restrict domain coverage, and underrepresent information-seeking tasks.The resulting lack of diversity is associated with limited success in realistic online evaluations.
  • Explorer: Explorer uses web exploration as a core mechanism to synthesize diverse, high-quality GUI trajectories across domains and skills.Unlike static task proposals, the framework dynamically explores web environments and grounds tasks in real-world websites.
  • Explorer: 94K trajectories span 49K unique URLs, 720K screenshots, and 33M web elements, forming a large-scale multimodal GUI dataset.The dataset includes richly annotated artifacts for comprehensive web-agent training.
  • Results: Explorer-trained small language models significantly surpass existing web-agent baselines on both online and offline benchmarks, including larger models.The introduction presents this result as evidence for the dataset’s effectiveness.

2 Related Work

Prior web-agent trajectory collection relied heavily on crowdsourcing, motivating synthetic generation because human annotation is costly. Existing synthetic approaches include task-proposal generation and tutorial-based supervision.

  • Trajectory collection: Early web-agent trajectory datasets primarily relied on crowdsourcing, while later work adopted synthetic generation for large-scale collection.The shift reflects the high cost of human annotation.
  • Synthetic generation: Related synthetic approaches generate task proposals from homepages or use web tutorials as supervision for web-trajectory generation.These approaches represent two distinct strategies for constructing synthetic trajectories.

3 Data Recipe

Explorer synthesizes web trajectories by exploring real websites with interacting agents that progressively refine task intent, summarize action histories, and verify completion. The resulting dataset spans diverse domains and is generated at $0.28 per successful trajectory.

  • Pipeline: Explorer executes browser actions on real websites and collects screenshots, HTML, accessibility trees, and grounded and natural-language actions.Playwright is used to execute actions and collect metadata from an initial URL.
  • URL sources: The pipeline combines high-traffic transactional URLs with 49K popular Tranco URLs to broaden website and domain coverage.Harmful websites are filtered for safety compliance.
  • Task construction: Explorer progressively shapes low-level actions into high-level task descriptions while maintaining a coherent task intent.This bottom-up exploration strategy is designed to enhance diversity.
  • Task construction: The task proposer creates an abstract task and first action from a homepage, while the refiner updates the task after each action using the complete action history.The proposer halts when encountering CAPTCHA, login, or payment prompts.
  • Task construction: The task summarizer converts action and screenshot histories into a high-level task description that omits execution details.The summary is intended to describe what the task entails rather than how it is completed.
  • Verification: The task verifier evaluates the description and action history with trajectory screenshots and the final page’s markdown representation, discarding incoherent or misaligned trajectories.This verification stage is used to enforce data quality.
  • Dataset composition: Explorer covers services, entertainment, shopping, travel, and information, with an average of 7.7 steps per trajectory.Its dataset statistics are reported for successful trajectories.
  • Cost: $0.28 per successful trajectory makes Explorer approximately 2× more cost-effective than AgentTrek’s $0.55 per trajectory.The automated pipeline avoids training and continuously monitoring crowd workers.

4 Experiments

The experiments train small multimodal language models on filtered synthetic trajectories and evaluate them across offline, live-web, and low-level web-agent benchmarks. Mind2Web-Live evaluation uses key-node-based success measures on accessible live tasks, while other benchmarks assess element, operation, step, or averaged task performance.

  • Training Setup: ~30K filtered trajectories fine-tuned Phi-3.5V and Qwen2-VL-7B into Explorer-4B and Explorer-7B for web-agent evaluation.The training subset began with 40K trajectories and excluded trajectories containing more than two scroll actions.
  • Benchmarks: Explorer models were evaluated on Mind2Web-Live, Multimodal-Mind2Web, and MiniWob++.These benchmarks cover live websites, offline multimodal tasks, and low-level single-page interactions, respectively.
  • Multimodal-Mind2Web: Multimodal-Mind2Web reports element accuracy, operation F1, and step success rate on open-ended tasks with screenshots and HTML.The benchmark contains 2K tasks spanning 137 websites across 31 domains.
  • Mind2Web-Live: Mind2Web-Live evaluates agents on 104 live-web tasks using key-node trajectories rather than golden action sequences.The reported metrics are average step success rate, completion rate, and full task success rate.
  • MiniWob++: MiniWob++ uses zero-shot evaluation on 46 low-level webpage tasks, averaging results across four runs per task.Typical tasks include button sequences, dropdown selection, and form filling.

5 Results

Explorer models improve over pretrained counterparts and strong baselines across in-domain, Mind2Web, Multimodal-Mind2Web, and MiniWob++ evaluations. Performance also improves as the training trajectory dataset grows, while task success increases more gradually than stepwise metrics.

  • In-domain Evaluation: Fine-tuned agents significantly outperform their pretrained counterparts on 100 disjoint Explorer-generated live-web tasks.An LLM-as-a-judge verifier evaluates action correctness at the trajectory level.
  • Mind2Web-Live Results: 5.1% and 4.8% gains in average step SR and key node completion rate, respectively, are achieved by Explorer-7B over pretrained Qwen2-VL-7B.Explorer-4B gains 15.5% and 15.9% on the same metrics over its pretrained counterpart.
  • Multimodal-Mind2Web Results: Explorer significantly outperforms existing GUI agent baselines on Multimodal-Mind2Web average step success rates across three test splits.The table’s final column reports average step success rates over the three test splits.
  • Mind2Web-Live Results: 8.5% and 2.7% full task SR margins separate Explorer-4B from Mistral-7B-Instruct-0.3 and Qwen2-72B-Instruct, respectively.Phi-3.5V reaches 18.1% full task success versus GPT-3.5’s 15.4%.
  • MiniWob++ Results: Explorer-4B surpasses Synatra-CodeLlama-7B and AgentTrek-7B on zero-shot MiniWob++, despite using a 4.2B-parameter model.The reported comparison highlights generalization to new web environments.
  • Data Scaling Experiments: 25% of the training data already produces rapid gains over the pretrained model, while larger datasets yield gradual improvements across all reported metrics.Overall task success improves more gradually than stepwise metrics because it is coarser-grained.

6 Analyses

Explorer’s analyses show that exploration expands abstract proposals into many fine-grained tasks, while verifier accuracy and pipeline failures remain important quality considerations.

  • Diversity Analysis: 94K final task descriptions emerged from approximately 53K initial high-level goals, with 81K final descriptions unique.The analysis attributes this expansion to web exploration shaping proposals into fine-grained tasks with different constraints.
  • Analysis of Verifier Accuracy: 81% agreement with human judgment was obtained for the task verifier on 100 Explorer-generated trajectories.The agreement was comparable to prior work.
  • Failure Modes of Trajectory Generation: Grounding errors during refinement can misalign grounded actions with natural-language actions and propagate inaccurate task descriptions.This misalignment was identified as a failure mode in the trajectory-generation pipeline.
  • Failure Modes of Trajectory Generation: Unresponsive websites can cause failures even when the grounded action matches the task objective.Dynamic content changes and nonresponsive web elements are cited as causes during execution.
  • Failure Modes of Trajectory Generation: Summarization hallucinations add extraneous constraints or goals absent from the underlying trajectory, causing task-description misalignment.The figure identifies grounding errors during refinement and summarization hallucinations as dominant error types.
  • Failure Modes of Trajectory Generation: Technical issues and step limits also produce unsuccessful trajectories through access barriers, media problems, bot detection, or incomplete action sequences.Examples include login requirements, media playback issues, automated bot detection, and reaching the step limit.

7 Conclusion

The paper concludes that Explorer scales web-trajectory synthesis through exploration, yielding diverse domains and skills while outperforming existing baselines on online and offline benchmarks. Its experiments also emphasize data scale, with future work extending exploration to broader GUI environments.

  • 7 Conclusion: Explorer is a scalable framework for synthesizing web trajectories at large scale through thorough web exploration.The framework is designed to increase diversity in domains and skills while producing contextually grounded trajectories.
  • 7 Conclusion: Explorer outperforms existing web-agent baselines by a significant margin on both online and offline web-agent benchmarks.The conclusion reports this result without specifying individual benchmark values in the supplied passage.
  • 7 Conclusion: The results highlight data scale as a critical factor in improving web-agent performance.The conclusion identifies scaling training data as a key empirical finding.
  • 7 Conclusion: Future work targets broader GUI environments and specialized exploration for information-seeking, operational, and navigation skills.The paper specifically mentions operating systems with diverse applications as an extension target.

Limitations

The paper identifies limitations involving autonomous exploration, collection cost, proof-of-concept status, and inaccessible website content.

  • Limitations: Autonomous web exploration may occasionally produce incoherent tasks.This limits the reliability of tasks generated without additional controls.
  • Limitations: Synthetic data collection with closed-source LLMs can be costly because of API expenses.The authors identify tailor-made open-source LLMs as future work addressing this cost concern.
  • Limitations: Some website content remains inaccessible because of login requirements, resulting in insufficient data for those websites.This constrains coverage of websites requiring authentication.

Ethical Considerations

The pipeline is restricted to academic GUI-agent research and uses safeguards to avoid unauthorized interactions, transactions, and exposure to violent or explicit content.

  • Ethical Considerations: Agents automatically terminate when they encounter CAPTCHAs, login prompts, or payment requests.This safeguard prevents actual transactions or bookings during data collection.
  • Ethical Considerations: The pipeline filters out websites containing violent or explicit content.This filtering is part of the paper’s stated ethical-compliance measures.

Appendices

The appendices provide supplementary details on evaluation, cost, task complexity, prompts, trajectory examples, related work, and implementation settings.

  • The supplementary material includes appendices covering Mind2Web training and evaluation, cost analysis, task complexity, system prompts, trajectory examples, and additional related work.
  • The experiments use Nvidia H100 GPUs, with hyperparameters and training time reported for Mind2Web-Live and Multimodal-Mind2Web.
  • Mind2Web-Live results exclude nine websites because access was denied and report performance on 83 of 104 tasks across 37 remaining websites.
  • The evaluation follows prior settings by using top-50 candidate elements, including the ground-truth element, a 1280 × 720 viewport, and element accuracy, operation F1, and step SR metrics.

A.3 Ablation Studies

The ablation and error analyses examine visual-modality effects and characterize common failure modes in Mind2Web-Live evaluation.

  • Ablation Studies: The ablations compare textual-only Phi-3.5V, text-only Phi-3-mini, Qwen2-VL-7B, and LLaVA-Mistral-7B.
  • Ablation Studies: Omitting visual input causes a 4.8% performance drop for Phi-3.5V, underscoring the importance of visual modality.
  • Error Analysis: The sampled failures include task deviation, missing key steps, grounding errors, unresponsive websites, and failure to reach the correct website.
  • Error Analysis: Task deviation is the most prevalent error type in Mind2Web-Live evaluation.
  • Evaluation: Mind2Web-Live reports full task success, average step success, completion rate, and tolerance-based task success metrics.

B Cost Analysis

The appendix reports the cost and complexity of Explorer’s trajectory-synthesis pipeline, alongside the prompts, action space, and supporting benchmark context used in the system.

  • Cost Analysis: Trajectory synthesis uses GPT-4o-turbo at $2.5 per 1M tokens, with 3.6K textual tokens per proposal or refinement stage and $0.0028 per input image.
  • Cost Analysis: The cost calculation assumes an average of 7.7 steps per trajectory, including the proposal stage, with module-level costs detailed in Table B.4.
  • Cost Analysis: The average cost per raw trajectory is $0.15, the estimated success rate is 53.1%, and the average cost per successful trajectory is $0.28.
  • Task Complexity: Tasks are categorized as easy, medium, or hard using 2–4, 5–7, or 8–12 action steps, and most fall within the medium-to-high complexity range.
  • System Prompts: Explorer’s prompts predict a natural-language next action, ground it to webpage structure, and update the overall task while enforcing atomic actions from a fixed vocabulary.
Loading 2502.11357v4…