Source-linked AI summary
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, Dong Yu
TL;DR
Existing web agents often use only one input modality and are evaluated in simplified or static settings, motivating a more realistic end-to-end approach. WebVoyager is a multimodal web agent that interacts with real-world websites, supported by a 15-website benchmark and GPT-4V-based automatic evaluation. It achieves a 59.1% Task Success Rate on the new benchmark, while the evaluator reaches strong agreement with human judgments.
Problem
Existing web agents typically handle one input modality and are evaluated in simplified web simulators or static snapshots rather than end-to-end real-world browsing.
Method
WebVoyager uses an LMM to interact with real websites through screenshots and textual web-element content, while GPT-4V evaluates saved navigation trajectories and final results automatically.
Results
59.1% Task Success Rate was achieved by WebVoyager on the new benchmark, compared with 30.8% for GPT-4 (All Tools) and 40.1% for the text-only setting.
Takeaways & Limitations
WebVoyager demonstrates the promise of combining visual and textual signals for end-to-end web agents, while GPT-4V provides a promising automatic evaluator for online agents.
Takeaways & Limitations
The agent does not support all human browsing actions, including webpage dragging, and real-world deployment requires substantial safety checks.
Abstract
from arXiv · showhide
The rapid advancement of large language models (LLMs) has led to a new era marked by the development of autonomous applications in real-world scenarios, which drives innovation in creating advanced web agents. Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. To bridge this gap, we introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. Moreover, we establish a new benchmark by compiling real-world tasks from 15 popular websites and introduce an automatic evaluation protocol leveraging multimodal understanding abilities of GPT-4V to evaluate open-ended web agents. We show that WebVoyager achieves a 59.1% task success rate on our benchmark, significantly surpassing the performance of both GPT-4 (All Tools) and the WebVoyager (text-only) setups, underscoring the exceptional capability of WebVoyager. The proposed automatic evaluation metric achieves 85.3% agreement with human judgment, indicating its effectiveness in providing reliable and accurate assessments of web agents.
1 Introduction
WebVoyager addresses limitations in prior web agents by combining visual and textual webpage information for end-to-end interaction with real-world websites. It introduces a benchmark and GPT-4V-based evaluation protocol, achieving a 59.1% Task Success Rate and 85.3% agreement with human judgment.
- Motivation: Existing web agents commonly rely on text-based representations, while rendered webpages make visual analysis important for understanding structured user interfaces.Prior approaches also face challenges processing complex and verbose HTML texts.
- WebVoyager: WebVoyager uses screenshots and textual content from interactive web elements to select actions such as clicking, typing, and scrolling without intermediate human intervention.Interactive elements are marked on screenshots to facilitate action decisions.
- Evaluation: 85.3% agreement with human judges was achieved by the GPT-4V-based automatic evaluation protocol for online web-agent trajectories.The protocol evaluates saved screenshots together with final results and was designed to accommodate varied end-to-end strategies.
- Benchmark: 643 web tasks from 15 commonly accessed websites form the newly collected evaluation dataset, alongside evaluations on GAIA and SeeAct tasks.The comparisons include GPT-4 (All Tools) and a text-only WebVoyager setting.
- Results: 59.1% Task Success Rate was achieved by WebVoyager, compared with 30.8% for GPT-4 (All Tools) and 40.1% for the text-only setting.These results are reported on the new benchmark.
2 Related Work
Related work includes text-based, multimodal, and prompted web agents, often operating in simplified environments or using single-modality inputs. WebVoyager instead targets multimodal task completion on real-world websites without additional candidate-selection modules.
- Autonomous Web Navigation: Earlier web-agent research used web simulators with simplified websites to study autonomous navigation in controlled environments.More realistic benchmarks such as Mind2Web and WebArena later expanded evaluation settings.
- Text-Based Agents: Text-based agents simplify HTML or use language models to generate actions, while prompting-based approaches construct agents without necessarily fine-tuning task-specific models.Examples include WebGPT, WebAgent, and several prompting-based systems.
- Multimodal Agents: Multimodal systems have combined screenshots with HTML text or relied solely on screenshots to predict web actions.WebGUM uses T5 with a Vision Transformer, whereas PIX2ACT uses screenshots alone.
- WebVoyager: WebVoyager combines multimodal interaction with real-world websites and does not require the additional candidate-element selection module used by the best SeeAct agent.This distinguishes its approach from prior single-modality or simplified-environment systems.
3 WebVoyager
WebVoyager operates an online browser environment where an LMM repeatedly observes screenshots and textual signals, generates actions, and executes them until completion or a step limit. Its interaction design emphasizes visual input, marked interactive elements, and common browsing actions.
- Interaction Formulation: At each step, the agent receives screenshots, auxiliary text, and interaction history, then generates an action that the browser executes.The environment returns a new observation after execution, continuing until termination or the maximum step count.
- Browsing Environment: WebVoyager browses the open web online through Selenium, exposing challenges such as floating ads, pop-ups, and constantly changing content.The environment avoids locally hosted websites to better reflect real-world interaction.
- Interaction Formulation: Context clipping retains only the three most recent observations to reduce confusion during longer browsing episodes.The agent is also prompted to produce a thought before its action code.
- Observation Space: Screenshots serve as the primary input, while bounding boxes and numerical labels identify interactive webpage elements for action prediction.This avoids processing verbose DOM or accessibility-tree representations and uses the webpage structure to locate elements.
- Action Space: The action space implements common mouse and keyboard operations, including click, input, and scroll, expressed through a concise labeled action format.The format maps numerical screenshot labels to the elements requiring interaction.
4 Benchmark for WebVoyager
The benchmark evaluates web agents on diverse real-world websites using a self-instruct and human-verification process. It contains 643 tasks, supports open-ended or changing answers through answer categories, and exhibits low pairwise task repetition.
- Website Selection: 15 representative websites spanning daily-life domains were selected, excluding sites requiring login or CAPTCHA because of technical limitations.Google Search also serves as a universal starting point for accessing other websites.
- Data Construction: The dataset combines self-instruction with human verification to generate and refine tasks for the selected websites.Seed tasks are sampled, GPT-4 Turbo generates additional tasks, and generated items are manually checked before further expansion.
- Data Quality: 99.68% of the 206,403 task pairs have pairwise similarity below 0.6, supporting low repetition among the 643 generated questions.Pairs with higher similarity were manually checked and judged acceptable.
- Answer Annotation: Answers are categorized as “Golden” or “Possible” because open-ended, multi-answer, and real-time tasks may lack a fixed stable response.“Golden” answers list comprehensive responses considered stable in the short term, while “Possible” answers provide partial or flexible coverage.
5 Experiment
Experiments evaluate WebVoyager on web-browsing tasks using human and multimodal automatic judgments, comparing it with GPT-4 (All Tools) and text-only baselines. Results show stronger performance overall, while revealing modality-, evaluator-, and navigation-related limitations.
- Results: WebVoyager outperforms both baselines on most website tasks, but trails text-only on Allrecipes and is similar on Github, ESPN, Cambridge Dictionary, and Wolfram Alpha.The authors attribute these cases primarily to dense text being difficult to recognize from screenshots.
- Automatic Evaluation: GPT-4V agreement with human judgments improves with more screenshots and reaches Kappa 0.7, matching the reported agreement among human annotators.The automatic evaluator receives the task, WebVoyager’s responses, and the last k screenshots or the full trajectory.
- Automatic Evaluation: Automatic evaluation varies by evaluator: Claude-3-Opus reaches Kappa 0.6 with humans, while GPT-4o reaches 0.72 and shows leniency toward task performance.GPT-4V is described as relatively strict, and both GPT-4V and GPT-4o judge Claude-3-Opus as performing worst.
- Limitations: Failure modes include running out of steps, visual grounding errors, and prompt misalignment during longer or complex trajectories.Open-source multimodal models are also constrained by low image resolutions that can make small text unrecognizable.
6 Conclusion
WebVoyager is an LMM-powered agent for completing real-world web tasks end-to-end, using both visual and textual signals. Evaluations show it outperforming several baselines, while GPT-4V supports automatic online-agent evaluation.
- WebVoyager completes real-world web tasks end-to-end by interacting with websites.
- The agent leverages both visual and textual signals to outperform several baselines in evaluation.
- GPT-4V is proposed as an evaluator for online web agents through an automatic evaluation protocol.
Limitations
The paper identifies limitations in WebVoyager’s supported actions and file-format coverage, and highlights safety risks for real-world deployment. These constraints motivate further development and substantial safety checks.
- WebVoyager does not support all human browsing actions, including drag operations whose degree is not a finite set.
- The agent currently analyzes basic text and PDF files but lacks support for many formats, especially videos.
- Deployment requires substantial safety checks because the agent might download malicious content, expose private information, or generate harmful website activity.
Ethics Statement
The experiments restrict WebVoyager to non-login tasks and monitor its online activities under ethical guidelines. Evaluation queries are manually inspected to ensure they are harmless.
- The web agent is restricted to non-login tasks to comply with website terms of service and user agreements.
- The agent’s online activities are closely monitored to identify and prevent potentially harmful consequences.
- All evaluation task queries undergo manual inspection to ensure they are harmless and ethically sound.
B Prompt for Auto Evaluation
Figure 8 illustrates GPT-4V’s automatic evaluation of web tasks using the task instruction, trajectory screenshots, and WebVoyager responses. The evaluator marks each task as successful or unsuccessful with temperature set to 0.
- GPT-4V evaluates web tasks using the task instruction, trajectory screenshots, and WebVoyager responses.
- The evaluator marks whether the web agent successfully completed each task.
- GPT-4V’s temperature is set to 0 to reduce randomness during evaluation.
C Action Space
WebVoyager uses a structured action space to navigate webpages, interact with elements, wait for loading, recover from dead ends, and submit answers.
- Basic actions: Click, input, scroll, wait, and back actions provide the core operations for web navigation.Input replaces existing text and presses ENTER automatically, while scrolling can target a specific scrollable region.
- Recovery: Google lets the agent restart through a search engine when it becomes stuck on a website.The implementation adopts Google Search for this recovery action.
- Completion: ANSWER concludes the iteration once the task questions are resolved and provides content matching the task requirements.Its format is ANSWER; [Content].
D Additional Trajectories
The paper presents WebVoyager’s visual web-navigation protocol and illustrates successful trajectories across multiple websites, including multilingual settings.
- Successful trajectories: The paper provides screenshot-and-action trajectories for successfully completed tasks across the remaining websites.Examples include Allrecipes, Amazon, ArXiv, BBC News, Booking, Cambridge Dictionary, Coursera, ESPN, GitHub, Google Maps, Google Flights, Google Search, Hugging Face, and Wolfram Alpha.
- Related work: Related vision-based agents explore visual reasoning, screenshot-based interaction, web simulation, or smartphone-app operation.The cited systems include VisualWebArena, SeeClick, WebVLN, GPT-4V Wonderland, and AppAgent.
- Navigation protocol: WebVoyager completes tasks by analyzing screenshots and text observations, selecting labeled webpage elements, and issuing structured actions.The system prompt is designed for generic web navigation rather than website-specific instructions.
- Multimodal foundation: Large multimodal models provide a foundation for constructing multimodal agents through strong benchmark performance.The paper situates WebVoyager within recent progress on unified image-and-language models.
F Error Cases
The paper documents both successful WebVoyager browsing trajectories and representative failures, including visual grounding, hallucination, and prompt-misalignment issues.
- Error cases: A Google Flights visual-grounding error selects December 22nd instead of January 22nd and fails to correct the date.The agent attempts a modification later in the trajectory but remains unsuccessful.
- Evaluation context: The evaluation setup uses task instructions and result screenshots as evidence for judging web-agent outcomes.The evaluator receives natural-language requirements and visual proof of the performed actions or intermediate states.
- Error cases: On Coursera, a hallucination-related failure identifies only a quiz in module 1 when the task asks for the course’s number of quizzes.The identified quiz is not the optimal answer and does not satisfy the task requirements.
- Task coverage: The trajectories span information retrieval, recommendations, travel planning, price lookup, quizzes, course identification, and mathematical simplification.Additional examples cover Allrecipes, Booking, GitHub, Google Flights, Google Search, Hugging Face, and multilingual browsing.
- Successful cases: Successful trajectories show WebVoyager retrieving answers from diverse real-world websites and tasks.Examples include a 4.7-star green Xbox controller on Amazon, a 6 out of 6 quiz score, 30 NBA teams, and a simplified polynomial expression.