PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-4V(ision) is a Generalist Web Agent, if Grounded

Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su

arXiv:2401.01614v2cs.IRcs.AIcs.CLcs.CV

TL;DR

The paper asks whether LMMs can act as generalist web agents across real-world websites despite the difficulty of grounding textual plans into executable actions. It proposes SEEACT, which combines visual website understanding, textual action generation, and grounding, and finds strong GPT-4V performance with oracle grounding while practical grounding remains a substantial gap.

  • Problem

    The paper investigates whether LMMs can follow natural-language instructions to complete diverse tasks on any real-world website, where rendered webpages are visually complex and grounding plans into actions is difficult.

  • Method

    SEEACT uses LMMs to visually perceive webpages and generate textual action plans, then grounds those plans onto HTML elements and operations, evaluated offline and on live websites.

  • Results

    51.1% whole task success rate was achieved online with GPT-4V and oracle grounding, while the best practical grounding strategies remained 20–25% below oracle grounding.

  • Takeaways & Limitations

    GPT-4V shows promise as a generalist web agent, but effective web grounding must leverage the correspondence between HTML structure and visual elements.

  • Takeaways & Limitations

    Web agents can generate harmful actions involving privacy-sensitive access or operations such as financial transactions, requiring safety assessment and mitigation before deployment.

Abstract

from arXiv · show

The recent development on large multimodal models (LMMs), especially GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries of multimodal models beyond traditional tasks like image captioning and visual question answering. In this work, we explore the potential of LMMs like GPT-4V as a generalist web agent that can follow natural language instructions to complete tasks on any given website. We propose SEEACT, a generalist web agent that harnesses the power of LMMs for integrated visual understanding and acting on the web. We evaluate on the recent MIND2WEB benchmark. In addition to standard offline evaluation on cached websites, we enable a new online evaluation setting by developing a tool that allows running web agents on live websites. We show that GPT-4V presents a great potential for web agents -- it can successfully complete 51.1 of the tasks on live websites if we manually ground its textual plans into actions on the websites. This substantially outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2) specifically fine-tuned for web agents. However, grounding still remains a major challenge. Existing LMM grounding strategies like set-of-mark prompting turns out to be not effective for web agents, and the best grounding strategy we develop in this paper leverages both the HTML structure and visuals. Yet, there is still a substantial gap with oracle grounding, leaving ample room for further improvement. All code, data, and evaluation tools are available at https://github.com/OSU-NLP-Group/SeeAct.

1. Introduction

SEEACT investigates LMMs as generalist web agents that follow natural-language instructions across real-world websites, combining visual understanding with web actions. Its evaluations show strong GPT-4V potential under oracle grounding, while grounding remains a major challenge.

  • Motivation: LMMs face a new web challenge because rendered websites contain thousands of visually related elements, although accurate comprehension could enable numerous web applications.The cited example contrasts 423 HTML elements and 186,490 textual tokens with 1,445 visual tokens.
  • Problem: SEEACT treats generalist web agents as systems that follow natural-language instructions and complete diverse, multi-step tasks on any real-world website.Tasks may require 10+ actions across dynamically rendered webpages.
  • Approach: SEEACT integrates LMM-based visual website understanding with textual action plans that are grounded onto HTML elements and browser operations.The grounding process selects both the target element and operation, such as Click, Type, or Select.
  • Findings: 51.1% of online tasks were completed by SEEACT with GPT-4V and oracle grounding, compared with 13.3% for GPT-4 and 8.9% for FLAN-T5.This result is reported for different websites in the online evaluation setting.
  • Findings: Grounding remains a major challenge: the best strategy has a 20–30% gap with oracle grounding and outperforms image annotation strategies by up to 10%.The strongest strategy leverages both HTML text and visual information, while set-of-mark-style annotation produces severe hallucination on complex webpage screenshots.
  • Findings: In-context learning generalizes better to unseen websites, whereas supervised fine-tuning retains an advantage on websites seen during training.The paper also reports a non-negligible discrepancy between online and offline evaluation because multiple viable plans may complete the same task.

2. SeeAct

SEEACT models web interaction as generating textual actions and grounding them into executable browser events. It uses GPT-4V to combine visual understanding with several grounding strategies for identifying target webpage elements.

  • Formulation: A web agent generates executable actions from the website, task, current observation, and previous actions.The observation includes an HTML document and screenshot image, while the website status updates after each action.
  • Formulation: Each browser action is represented as (e, o, v): target element, operation, and any required input value.Examples include clicking a button or typing a date into a webpage element.
  • Action Generation and Grounding: Action Generation produces a textual action description, while Action Grounding converts it into an executable browser action.Models typically describe the intended element, operation, and value rather than directly producing the browser-event variables.
  • Action Grounding: Element Grounding is the key challenge because the intended target element must be identified from the generated textual description.Operation type and input value can be obtained through string parsing more readily than the target element.
  • Grounding Strategies: SEEACT explores element-attribute, textual-choice, and image-annotation grounding strategies, alongside oracle grounding by human identification.Textual choices use ranked HTML candidates, image annotation overlays candidate bounding boxes and labels, and element attributes describe the target’s text and type.

3. Experiments

The experiments evaluate SEEACT and comparison systems on MIND2WEB across cached and live websites. They introduce a cleaned multimodal dataset alignment and an online tool for executing agents on live sites.

  • Dataset: MIND2WEB contains over 2,000 complex tasks spanning 137 websites, 31 low-level domains, and 12 high-level domains.Its primary operations are Click, Type, and Select, with Hover and Press Enter integrated into Click.
  • Dataset: The benchmark tests generalization across new tasks, websites, and held-out domains.Cross-Task, Cross-Website, and Cross-Domain settings vary which tasks, websites, or top-level domains are unseen during training.
  • Dataset: Multimodal Mind2Web aligns HTML documents with corresponding screenshots and human-verifies element visibility and rendering.This produces a cleaned version of the dataset for multimodal action prediction.
  • Systems: SEEACT ranks the top 50 candidate elements with a DeBERTa-base cross-encoder and groups them into sets of 17 for textual-choice and image-annotation inference.Element-attribute grounding supplies no candidate element, and all three methods are tested with GPT-4V.
  • Evaluation: Evaluation compares SEEACT with text-only LLMs, BLIP-2, LLaVA-1.5, and CogAgent using element, operation, step, and task success metrics.The reported step-wise metrics use macro averages across tasks.
  • Online Evaluation: The online evaluation tool uses Playwright to transfer multimodal browser inputs to agents and convert predicted actions into browser events.Experiments restrict live evaluation to non-login tasks and monitor activities for safety.

4. Results and Analysis

SEEACT results show that GPT-4V can generalize across web tasks and settings, but grounding remains the main bottleneck. GPT-4V benefits especially from oracle grounding, while grounding methods combining textual and visual information still trail it.

  • Oracle Grounding: 61.9%, 65.0%, and 62.1% step success rates were achieved by GPT-4V with oracle grounding across three test splits.The oracle method used human annotation to identify intended actions.
  • Generalization: 8.4%, 23.9%, and 23.2% step success rate improvements over the second-best method were obtained in Cross-Task, Cross-Website, and Cross-Domain settings.The larger Cross-Website and Cross-Domain margins indicate stronger performance under broader generalization settings.
  • Grounding: Grounding remains a major bottleneck: the proposed methods show a noticeable gap from oracle grounding, while textual-choice grounding performs best among them.Textual-choice grounding is comparable to supervised fine-tuning and substantially outperforms text-only LLMs.
  • Online Evaluation: Over 20% whole task success rate separated GPT-4V with textual-choice grounding from GPT-4 and FLAN-T5-XL across all three settings.Oracle grounding further raised whole task success rate to 51.1% in online evaluation.
  • Error Analysis: 54% of errors were attributed to GPT-4V visual illusion, in which it misinterprets or fabricates image content during grounding.A described target element may lack the expected bounding box or label, yet the model can falsely assume one exists.

5. Related Work

Prior web agents mainly rely on raw HTML, while LMM research has advanced visual-language understanding and grounding methods. SEEACT builds on these directions by combining multimodal webpage understanding with web action grounding.

  • Web Agent: Raw HTML-based web agents face massive, noisy inputs that can be costly or infeasible to provide directly to language models.MindAct addresses this by ranking HTML elements and selecting only the top elements as context.
  • Large Multimodal Models: GPT-4V and Gemini represent major progress in large multimodal models for vision-and-language understanding and reasoning.Open-source models remain behind GPT-4V but offer greater controllability and ease of fine-tuning.
  • Large Multimodal Models: CogAgent enhances webpage understanding using HTML-screenshot pairs and a high-resolution image encoder, while Ferret supports visual referring and grounding.
  • Visual Grounding: Visual prompting methods augment fine-grained grounding by overlaying marks such as numbers, masks, or bounding boxes onto images.Related approaches also represent bounding-box locations with textual tokens or train on image-annotated data.

6. Conclusion

SEEACT applies large multimodal models to web interaction and finds promise for generalist agents, while fine-grained grounding remains a major challenge. The conclusion also emphasizes online evaluation because live web interactions and possible action paths vary.

  • SEEACT integrates visual understanding and web action using large multimodal models such as GPT-4V.
  • 50% of tasks on live websites were completed with oracle grounding, indicating promise for generalist web agents.
  • The strongest explored grounding strategies still trail oracle grounding by 20-25%, leaving fine-grained visual grounding as a major challenge.
  • Online evaluation is important because offline and online results differ substantially as multiple valid plans can complete the same task.The conclusion links this discrepancy to the dynamic nature of web interactions.

7. Impact Statements

Generalist web agents could automate routine tasks, improve user experiences, and support accessibility, but deployment raises privacy, sensitive-operation, and harmful-action concerns.

  • Generalist web agents may automate routine web tasks, enhance user experiences, and promote web accessibility.
  • Real-world deployment raises privacy concerns involving personal profiles and risks around sensitive operations such as financial transactions or form submissions.
  • Online evaluation revealed that web agents may generate harmful actions on the web.

Table of Content:

The contents list organizes supplementary material covering offline methods, markup ablations, online experiments, prompts, grounding errors, planning, textual-choice challenges, reasoning requirements, and path variation.

  • Appendix A covers offline experiment method details, while Appendix B reports the markup type ablation study.
  • Appendices C and D cover online experiment details and offline experiment prompts.
  • Appendices E through I address image-annotation grounding errors, planning capability, textual-choice grounding challenges, knowledge and reasoning requirements, and path variation with error correction.

A. Offline Experiments Method Details

The offline experiments compare fine-tuned and in-context web-agent baselines with SEEACT models using different multimodal and language-model configurations.

  • Baselines: FLAN-T5 is fine-tuned on Mind2Web actions to generate parsed action sequences.It uses a left-to-right language-modeling objective with ground-truth actions.
  • Baselines: BLIP-2-T5 jointly fine-tunes its language model and bridging module while keeping the vision encoder frozen.The vision encoder is CLIP-pretrained ViT-L/14 at 2,048-pixel image resolution.
  • Baselines: GPT-3.5 and GPT-4 are evaluated as text-only LLMs with three-shot in-context learning using Mind2Web’s multiple-choice formulation.The experiments use GPT-3.5-turbo-0613 and GPT-4turbo-1106-preview.
  • SEEACT models: SEEACT experiments include GPT-4-vision-preview, Gemini Pro Vision, and LLaVA-1.5.Gemini Pro Vision’s single-turn constraint is handled by merging the two turns used for other models.
  • Additional model: CogAgent is evaluated with an unfine-tuned cogagent-chat-hf checkpoint.The checkpoint has not been fine-tuned on Mind2Web.

B. Markup Type Ablation Study

The study tests image-annotation markup choices and documents the online evaluation setup and model variants used to compare grounding approaches.

  • Markup ablation: Image-annotation grounding is ablated across numerical, single-digit, and two-digit labels placed at two positions.The experiments select a relatively better markup type using the results in Table 5.
  • Online evaluation: The online evaluator uses Playwright to load live webpages, represent interactive elements textually, and execute agent-generated operations.Human monitoring assesses task completion, while login, final submissions, and potentially harmful actions are prohibited.
  • Baselines: MindAct follows its original offline settings, including the Click, Type, and Select action space.The evaluation reuses Mind2Web scripts for webpage, element, and option processing.
  • Grounding settings: SEEACTOracle uses human annotators to implement the model’s intended actions while automatically generating action history.Current-view screenshots are used and scrolling intentions are allowed.
  • Grounding settings: SEEACTChoice uses a ranker and top-50 candidates grouped into three option sets, with PRESS ENTER and TERMINATE controls.These controls allow the model to confirm or stop the process.
  • Evaluation caveat: Pop-up ads were manually closed because MindAct cannot automatically handle them, whereas SEEACT can suggest closing ads through visual analysis.This creates a practical difference in live-web operation.

E. Error Examples for Grounding via Image Annotation

The error examples show that image-annotation grounding fails through hallucinated labels, spatial mislinking, ambiguity among identical elements, and knowledge demands.

  • Error categories: Image-annotation grounding exhibits significant hallucination errors across several recurring categories.The following examples classify these failures rather than presenting a single isolated mistake.
  • Spatial linking: The model often links a bounding box to an adjacent label because relative spatial understanding is difficult in dense webpage layouts.Figures 12 and 13 show incorrect associations with neighboring indices.
  • Capabilities: GPT-4V can generate full multistep plans and use screenshot analysis to understand remaining work during a task.Figures 14 and 15 illustrate planning across later pages and recovering information missing from textual history.
  • Textual-choice grounding: Textual choices perform best among the three grounding approaches but remain vulnerable to similar or identical elements.The model tends to select the first seemingly matching choice, as with repeated “Schedule” buttons.

I. Path Variation and Awareness of Error Correction

Live-web execution can diverge from benchmark paths because multiple routes exist, yet the agent can plan broadly, exploit direct alternatives, and correct errors during exploration.

  • Path variation: Multiple valid action paths can complete the same task, so an agent’s route may differ from the dataset’s ground truth.Figure 19 shows a more direct route to the target page than the annotated path.
  • Error correction: Live-web histories may contain redundant, failed, erroneous, or exploratory actions, requiring self-correction to finish the task.The final path can deviate substantially from ground truth while still reaching completion.
  • Planning: GPT-4V can generate comprehensive plans that include later actions on pages not currently visible.Its screenshot analysis also recovers information absent from textual history to choose the next action.
  • Grounding ambiguity: Identical “Schedule” buttons make choice-based grounding unable to distinguish candidates, with the model tending to select the first option.This ambiguity arises even when the options appear identical.
  • Knowledge requirements: Some tasks require world knowledge, including Dublin’s district and airport IATA codes.GPT-4V correctly supplies the Los Cabos International Airport code in one example.
  • Error correction: A webpage error can cause the model to abandon planned later steps and prioritize immediate correction.Figure 20 shows this behavior after an invalid phone number error.
Loading 2401.01614v2…