PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind2Web: Towards a Generalist Agent for the Web

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, Yu Su

arXiv:2306.06070v3cs.CL

TL;DR

Generalist web agents need datasets that cover diverse tasks on complex, real-world websites rather than restricted or simulated environments. Mind2Web supplies this data and explores MINDACT, which filters webpage elements before LLM action prediction; it achieves decent generalization to unseen websites and domains, while substantial improvement remains necessary.

  • Problem

    Existing web-agent datasets rely on simulated websites or limited website and task coverage, leaving insufficient support for developing generalist agents that follow language instructions on arbitrary websites.

  • Method

    Mind2Web combines over 2,000 open-ended tasks, crowdsourced action sequences, and real-world website data, while MINDACT filters webpage elements with a small LM before LLM action prediction.

  • Results

    MINDACT significantly outperforms prior modeling strategies and achieves decent generalization, including 52.0% step success on Cross-Task and 38.9% / 39.6% on unseen websites and domains.

  • Takeaways & Limitations

    Mind2Web provides a platform for evaluating and advancing generalist web agents across diverse, complex, real-world environments.

  • Takeaways & Limitations

    The dataset predominantly represents English-language websites used in the United States and annotators sourced through Amazon MTurk, so it may cover only a subset of possible web tasks.

Abstract

from arXiv · show

We introduce Mind2Web, the first dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website. Existing datasets for web agents either use simulated websites or only cover a limited set of websites and tasks, thus not suitable for generalist web agents. With over 2,000 open-ended tasks collected from 137 websites spanning 31 domains and crowdsourced action sequences for the tasks, Mind2Web provides three necessary ingredients for building generalist web agents: 1) diverse domains, websites, and tasks, 2) use of real-world websites instead of simulated and simplified ones, and 3) a broad spectrum of user interaction patterns. Based on Mind2Web, we conduct an initial exploration of using large language models (LLMs) for building generalist web agents. While the raw HTML of real-world websites are often too large to be fed to LLMs, we show that first filtering it with a small LM significantly improves the effectiveness and efficiency of LLMs. Our solution demonstrates a decent level of performance, even on websites or entire domains the model has never seen before, but there is still a substantial room to improve towards truly generalizable agents. We open-source our dataset, model implementation, and trained models (https://osu-nlp-group.github.io/Mind2Web) to facilitate further research on building a generalist agent for the web.

1 Introduction

Mind2Web addresses the lack of datasets for generalist web agents by combining diverse tasks and websites, real-world environments, and sophisticated interactions. It also introduces MINDACT, which filters webpage elements before an LLM predicts actions, achieving generalization while leaving substantial room for improvement.

  • Motivation: Generalist web agents must operate across unseen websites and domains, handle dynamic real-world complexity, and support diverse multi-step interactions.These requirements arise because modern websites are complex and tasks can require many actions.
  • Motivation: Existing web-agent efforts remain limited by restricted website coverage, simplifying assumptions, narrow task types, or step-by-step user instructions.The paper identifies the absence of a dataset satisfying all requirements as a central gap.
  • Mind2Web: Mind2Web provides over 2,000 tasks from 137 websites spanning 31 domains for developing and evaluating generalist web agents.Its diversity supports evaluation across tasks within websites, across websites, and across disparate domains.
  • Mind2Web: The dataset uses authentic websites with interaction traces, webpage snapshots, and network traffic instead of oversimplified simulations.This exposes agents to the complexity and uncertainty of real-world web environments.
  • Mind2Web: Mind2Web supports sophisticated interactions including clicking, selecting, and typing, expanding beyond basic search, navigation, and reading tasks.This broader interaction space is intended to capture common real-life website actions.
  • Initial LLM exploration: MINDACT first filters webpage elements with a fine-tuned small LM, then uses an LLM to select an element and predict its operation.The approach addresses the size of raw HTML documents and works with both open-source and closed-source LLMs.
  • Initial LLM exploration: MINDACT achieves decent generalization to unseen websites and domains, but substantial improvement remains necessary for truly generalist web agents.The dataset enables out-of-distribution evaluation on websites or entire domains not seen during training.

2 MIND2WEB Dataset

Mind2Web collects high-level tasks, annotated action sequences, and rich snapshots from diverse real-world websites. Its collection and representation choices target autonomous, multi-step planning and evaluation in complex web environments.

  • Dataset design: Mind2Web bridges simulation and reality by collecting interaction data directly from real-world websites across varied domains.The design aims to expose agents to environments without manual simplification.
  • Dataset components: Each dataset instance contains a high-level task description, an action sequence, and webpage snapshots representing the task environment.The snapshots include raw HTML, DOM and layout information, rendered screenshots, network traffic, and interaction traces.
  • Task definition: Task descriptions state high-level goals rather than step-by-step instructions, encouraging agents to interpret and execute tasks autonomously.This creates a more demanding planning and grounding setting than prescriptive instructions.
  • Task definition: Each action is a target-element and operation pair, supporting Click, Type, and Select Option, with values required for Type and Select Option.Action sequences may span multiple webpages.
  • Task definition: Agents receive the task description, current webpage, and prior action history, then predict the next target element and operation.
  • Data collection: Data collection uses website selection, task proposal, task demonstration, and task verification, combining author oversight with Playwright-based crowdsourcing.The authors verify descriptions, actions, and task boundaries after demonstrations.
  • Data collection: The collection covers 137 websites across 31 secondary domains selected from five top-level domains.Websites were manually selected as representative and ranked by popularity in the United States.

3 Method: MINDACT

MINDACT addresses oversized webpage HTML with a two-stage pipeline: a small LM ranks candidate elements, then an LLM predicts actions from a pruned representation. Candidate generation uses task context and DOM-element representations, while action prediction is framed as multiple-choice QA.

  • Overall pipeline: MINDACT first ranks webpage elements with a small LM, then uses an LLM to select an element and predict its action.The two stages reduce the input before large-model prediction.
  • Candidate generation: Candidate generation ranks DOM elements against the task description, current step, and preceding actions.The task query concatenates the task description with previous actions.
  • Candidate generation: Candidate representations combine element tags, textual content, salient attributes, and neighboring parent-child context.A cross-encoder produces matching scores for task-query and candidate pairs.
  • Action prediction: The top-k candidates and their neighbors form pruned webpage snippets supplied to the prediction LLM.This avoids passing the full webpage snapshot to the large model.
  • Action prediction: MINDACT converts element selection into multiple-choice QA rather than requiring the LLM to generate a complete target element.The formulation selects from a list of candidate options.

4 Experiments

Experiments evaluate candidate generation, action prediction, and generalization across tasks, websites, and domains. MINDACT performs best on familiar task settings, while unseen-environment generalization and whole-task completion remain difficult.

  • Evaluation settings: 912 tasks from 73 websites comprise TestCross-Domain, while TestCross-Website contains 177 tasks across held-out websites.These splits test generalization beyond seen domains or websites.
  • Data preprocessing: HTML cleaning reduces average webpage elements from 1,135 to 580 while retaining 94.7% target-element recall on training data.The preprocessing keeps visible, semantically meaningful elements.
  • Candidate generation: 88.9% / 85.3% / 85.7% Recall@50 is achieved by DeBERTaB on TestCross-Task, TestCross-Website, and TestCross-Domain.The top-50 results are used as the candidate pool.
  • Action prediction: 52.0% step success rate is achieved under Cross-Task, compared with 38.9% / 39.6% on unseen websites and domains.The overall task success rate remains low because one incorrect step can fail a task.
  • Generalization: Cross-Task performance exceeds Cross-Website and Cross-Domain performance by over 10% absolute step-success rate on average.The latter two settings show no clear performance distinction, suggesting website design and interaction logic are major challenges.
  • In-context learning: GPT-3.5-turbo reaches around 20% element selection accuracy despite using three in-context examples.The analysis identifies frequent selection of the None option as one possible difficulty.

5 Related Work

Mind2Web relates to autonomous web and mobile agents, web automation, large language models, grounded language understanding, and tool learning. It targets natural-language control of noisy, heterogeneous web environments and longer-horizon interaction.

  • Autonomous agents: Prior autonomous-agent work is limited by simplified simulations or narrow domains, websites, and task types.These limitations motivate broader web-agent evaluation.
  • Web automation: Mind2Web adds a natural-language interface to web automation, which often requires programming skills.The intended effect is to reduce the entry barrier for general users.
  • Large language models: Large language models are relevant because pretraining supports broad language understanding and few-shot learning.The paper explores their use for web-agent action prediction.
  • Grounded language understanding: Unlike schema-based grounded-language settings, Mind2Web grounds language in noisy and schemaless web environments.This connects the task to grounded language understanding and embodied AI.
  • Tool learning: Existing tool-learning research emphasizes short-term tool invocation, whereas Mind2Web requires longer-term planning.Web tasks require language models to take sequences of actions.

6 Limitations and Potential Societal Impact

The paper identifies limitations in data representation, multimodal and interaction modeling, offline evaluation, human-agent interaction, and deployment safety. It also outlines directions for expanding the dataset and improving real-world applicability.

  • Diversity and Representation in Data Collection: The dataset predominantly covers English-language websites used in the U.S., with annotators recruited through Amazon MTurk, limiting representation of possible web tasks.The authors suggest adding websites from other countries and languages and tasks from more diverse demographics.
  • Use of Multimodal Information: MINDACT uses textual webpage snapshots without visual information, although complete snapshots enable future multimodal interpretation.The paper presents multimodal information as a prospect for improving model performance.
  • Modeling of Interaction Dynamics: MINDACT encodes each webpage independently and provides only previous actions as history, leaving dynamic environment changes insufficiently modeled.The authors highlight cues such as dropdowns appearing after button clicks as a target for future interaction modeling.
  • Human-Agent Interaction: MIND2WEB currently assumes a single task description and autonomous execution from start to finish, excluding mid-task requirement changes or confirmation requests.An interactive or conversational extension is proposed as future work.
  • Evaluation with Offline/Online Environments: Offline evaluation can fail immediately when an action was not cached, producing false negatives even when alternative task-completion paths exist.The authors propose live evaluation on real websites with human assistance as a promising direction.
  • Safety in Deployment: Real-world deployment raises concerns about sensitive actions, transparency, user control, security-measure circumvention, and malicious use.The paper calls for cybersecurity research and preemptive protective measures.

7 Conclusion

The paper concludes by introducing MIND2WEB as a dataset for generalist web-agent research and MINDACT as an LLM-based agent for this setting. It positions the work as a platform for future research, supported by released artifacts and further development directions.

  • Conclusion: MIND2WEB is introduced as the first dataset for developing and evaluating generalist agents for the web.It is intended to support agents operating across websites and web tasks.
  • Conclusion: MINDACT is proposed as an agent that leverages large language models to tackle generalist web tasks.The conclusion presents it alongside the dataset as the paper’s central system contribution.
  • Conclusion: The paper identifies multimodal information, reinforcement learning with feedback from real websites, and specialized language models as promising future directions.These directions target web understanding and action taking.
  • Conclusion: The authors provide supplementary data-collection and experiment details, including additional results, to support use and further study of the work.The supplementary material covers crowdsourcing, implementation, evaluation, and auxiliary experiments.
  • Conclusion: The released artifacts include a code repository, training data, and test data under stated open-source or dataset licenses.The paper also states that the authors and OSU NLP group are committed to regular upkeep and updates.
  • Conclusion: The appendix documents task collection through seed-task inspiration, annotation tooling, and author review procedures.These materials cover task proposal, demonstration, and verification.

B.3 Task Demonstration

Task demonstrations use a controlled Playwright-based annotation workflow that separates element selection from operation selection. Exploration, standardized operations, disruption handling, and author verification are used to produce clean action traces.

  • Task Demonstration: The annotation tool uses side-by-side dialogue and browser windows for controlling operations and selecting webpage elements.The overall procedure is illustrated through the annotation-tool figures.
  • Task Demonstration: Annotators explore and prepare websites before recording, then reset altered values so demonstrations begin from a fresh state.Exploration is not recorded and is used to handle pop-ups and familiarize annotators with the task.
  • Task Demonstration: Each recorded action separates target-element selection from operation selection, with the tool executing the chosen operation in the browser.The workflow provides Click, Type, Hover, Press Enter, Click (Fake), and Ignore operations.
  • Task Demonstration: Pop-ups and CAPTCHAs are intentionally cleared or omitted to keep task demonstrations clean and direct.Only websites without access issues through the annotation tool are selected.
  • Task Demonstration: Click (Fake) records disruptive actions such as appointment scheduling without executing them on the website.This operation is designed to avoid interfering with normal website operation.
  • Task Demonstration: Author verification reviews task quality, individual actions, and task descriptions, with uncertain cases escalated for re-evaluation.The process can discard extraneous actions and modify inaccurate descriptions.

C Experiment Details

Evaluation uses heuristics to recognize webpage elements equivalent to the labeled ground-truth element when multiple elements produce the same effect.

  • Experiment Details: The evaluation checks ancestors of a labeled element to identify higher-level elements that are acceptable for the same action.This addresses cases such as a button and its nested text span producing identical results.

C.2 Model Implementation Details

The implementation uses DeBERTa for candidate generation and Flan-T5 for action prediction, with GPT models evaluated through three-shot in-context learning. Random grouping of webpage elements causes only small performance changes across five runs.

  • Model components: Candidate generation uses a DeBERTa-v3-base cross-encoder implementation from Sentence-Transformers.
  • Model components: Action prediction experiments use base, large, and xl Flan-T5 Seq2Seq models.
  • In-context learning: GPT-3.5-turbo and GPT-4 receive three demonstration examples for in-context learning through the OpenAI API.
  • Training details: Flan-T5XL and Flan-T5-large use 4 A100 80GB cards, while other models use single A6000 48GB cards.
  • Evaluation stability: Randomly grouping webpage elements changes final scores only slightly, based on averages and standard deviations across 5 random-seed runs.

D.2 Zero-shot Results for Flan-T5XL

Zero-shot Flan-T5XL can select some webpage elements, but performs much worse than its fine-tuned counterpart and three-shot GPT models. The comparison is reported in Table 6.

  • Zero-shot element selection: Zero-shot Flan-T5XL achieves some correct element selections but substantially underperforms the fine-tuned model and 3-shot GPT-3.5/GPT-4.
  • Interpretation: The authors attribute the zero-shot gap to Flan-T5 not being tuned for HTML and coding-related tasks.
  • Evaluation: Table 6 compares zero-shot Flan-T5XL element selection with its fine-tuned counterpart.

D.3 Results on the 50 task subsets

GPT-4 is evaluated on 50 tasks per setting because of budget constraints, and the resulting subset comparisons remain consistent with the corresponding full test sets. The section includes a prompt example for action prediction.

  • Evaluation subsets: GPT-4 is evaluated on 50 tasks for each setting because of budget constraints.
  • Evaluation subsets: On these 50-task subsets, step success rates are consistent with the respective full test-set results, and relative method and split performance remains unchanged.
  • Prompt example: Table 8 provides the GPT prompt for action prediction, showing only part of the HTML snippet.
  • Prompt example: The illustrated task asks for a mini van in Brooklyn from April 5th to April 8th for a 22-year-old renter, with the demonstrated next action being CLICK on the pick-up-date control.
Loading 2306.06070v3…