PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization

Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, Dong Yu

arXiv:2410.19609v1cs.CLcs.AI

TL;DR

Existing self-improving agents were largely text-only and used synthetic environments, leaving multimodal real-world web navigation without built-in reward signals as an open challenge. OpenWebVoyager trains an open-source agent through imitation learning and repeated real-web exploration with GPT-4o feedback on trajectories. Task Success Rate increased across the reported WebVoyager and Mind2Web test sets over four phases.

  • Problem

    Prior self-improving agents focused on text-only tasks in synthetic environments, while real-world webpages require visual inputs and often lack built-in reward signals.

    Source: §1 · p. 1

  • Method

    OpenWebVoyager first learns from successful imitation-learning trajectories, then iteratively explores real websites and trains on trajectories GPT-4o judges correct.

    Source: §1 · pp. 1–2

  • Results

    Across WebVoyager and Mind2Web test sets, Task Success Rate increased over four phases; on the Mind2Web cross-task set it rose from 6.3% to 19.6%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework relies primarily on agent self-exploration and GPT-4o supervision, reducing human intervention while allowing continuous expansion.

    Source: §5 · p. 9

    The approach considers clicking, typing, and scrolling, but not advanced actions such as dragging or zooming.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The rapid development of large language and multimodal models has sparked significant interest in using proprietary models, such as GPT-4o, to develop autonomous agents capable of handling real-world scenarios like web navigation. Although recent open-source efforts have tried to equip agents with the ability to explore environments and continuously improve over time, they are building text-only agents in synthetic environments where the reward signals are clearly defined. Such agents struggle to generalize to realistic settings that require multimodal perception abilities and lack ground-truth signals. In this paper, we introduce an open-source framework designed to facilitate the development of multimodal web agent that can autonomously conduct real-world exploration and improve itself. We first train the base model with imitation learning to gain the basic abilities. We then let the agent explore the open web and collect feedback on its trajectories. After that, it further improves its policy by learning from well-performing trajectories judged by another general-purpose model. This exploration-feedback-optimization cycle can continue for several iterations. Experimental results show that our web agent successfully improves itself after each iteration, demonstrating strong performance across multiple test sets.

1 Introduction

Prior self-improving agents were text-only and trained in synthetic environments, while real webpages require visual inputs and lack built-in rewards. OpenWebVoyager pairs initial imitation learning with repeated real-web exploration and GPT-4o trajectory feedback, with task success rising across its reported test sets.

1 Introduction

  • Earlier self-improvement studies used text-only agents in synthetic environments with defined rewards, whereas real webpages require visual inputs and often lack built-in rewards.Prompt-engineered agents based on closed-source models also face limitations to further improvement, motivating open-source approaches.Source: §1 · p. 1
  • OpenWebVoyager addresses multimodal real-world web navigation with an initial imitation-learning phase followed by iterative exploration, feedback, and policy optimization.GPT-4o evaluates trajectory correctness; successful trajectories are merged with imitation-learning data for the next training round.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Experiments use Idefics2-8B-Instruct, 48 websites, one imitation-learning phase, and three exploration-feedback-optimization cycles, evaluating Task Success Rate on WebVoyager and Mind2Web.Self-instruct generates new web queries for each phase.Source: §1 · p. 2
  • Task Success Rate rose from 19.9% to 25.8% on WebVoyager, 6.3% to 19.6% on Mind2Web cross-task, and 6.6% to 10.4% on Mind2Web cross-web.The cross-web set contains unseen websites; the authors characterize its improvement as slight and suggest some generalization.Source: §1 · p. 2

2 Related Work

Prior work developed multimodal web agents with varied visual-grounding methods and separately explored iterative self-improvement. This paper situates its approach at their intersection: using multimodal web signals for self-improvement.

2.1 Multimodal Web Agents

  • Multimodal agents use varied webpage-grounding methods, but open-source vision-language models have struggled because pretraining rarely covers interleaved image-text agent trajectories.Examples include indexed boxes, Set-of-Mark annotations, neighboring-element context, and dot matrices for grounding webpage elements.Source: §2.1 · pp. 2–3

2.2 Self-Improving Web Agents

3 Method

OpenWebVoyager builds an open-source multimodal web agent that first learns from demonstrations, then explores websites and improves using feedback on its trajectories. The method formulates web navigation as a multimodal decision process and specifies how tasks, trajectories, and training data are collected and used.

3 Method

  • OpenWebVoyager first learns WebVoyager-4o navigation trajectories through imitation learning, then iteratively explores and improves using GPT-4o feedback.The method describes an initial imitation-learning phase followed by multiple exploration-feedback-optimization cycles.Source: §3 · p. 3

3.1 Task Formulation

  • Web navigation is formulated as a partially observable process in which an LMM uses screenshots and accessibility trees to produce thoughts and actions.Browser operations transition the page to the next state, and a trajectory ends after T navigation iterations.Source: §3.1 · p. 3

3.2 OpenWebVoyager Overview

  • The agent uses Idefics2 with unmarked screenshots and accessibility trees; its context shows the latest three screenshots but retains only the latest tree.The environment omits Set-of-Mark screenshot labels because open-source LMMs have difficulty grounding numerical tags.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Web Task Queries Collection

  • The authors select 48 popular websites for imitation-learning queries and generate 480 new exploration queries per cycle, with 10 queries per website.Exploration queries are generated with self-instruct to be similar to, but not duplicates of, existing queries.Source: §3.3 · p. 4

3.4 Imitation Learning

  • WebVoyager-4o generates imitation-learning trajectories; unfinished trajectories are filtered out, while finished successful and unsuccessful trajectories are retained for Idefics2 training.Unfinished tasks are resampled once to improve query use and reduce failures due to sampling randomness.Source: §3.4 · pp. 4–5
  • For training, the context retains the latest k images and the current page’s accessibility tree, while preserving each step’s thoughts and actions and omitting the system prompt.The paper says imitation learning teaches the agent the basic operation logic and response format, so the prompt is no longer needed.Source: §3.4 · pp. 4–5

3.5 Iterative Optimization

  • In each optimization cycle, the agent explores with newly generated queries, GPT-4o accepts successful trajectories, and the accepted data is mixed with imitation-learning data.The resulting mixed trajectory set is used in subsequent optimization, initialized from the imitation-trained agent.Source: §3.5 · p. 5

4 Experiment

Across the evaluated test sets, task success generally improved with iterative optimization, while gains on unseen websites were less prominent and unstable. The experiments also examine difficulty-guided sampling and changes in trajectory length and navigation behavior.

4.1 Dataset and Metric

  • Training used 48 websites and 1,516 imitation-learning queries, while evaluation covered 643 WebVoyager queries, 112 cross-task queries, and 106 unseen-site queries.The primary metric was GPT-4o-evaluated Task Success Rate.Source: §4.1 · pp. 5–6

4.3 Main Results

  • Task success generally improved on WebVoyager and Mind2Web cross-task tests as optimization progressed, while gains on unseen websites were less prominent and unstable.The comparison covers the WebVoyager test set and Mind2Web cross-task and cross-website sets.Source: Table 1 · p. 6, Table 2 · p. 6, Figure 3 · p. 7

4.2 Experimental Details

  • The agent used Idefics2, was fine-tuned with at most three screenshots and one accessibility tree, and completed three exploration-feedback-optimization cycles.Exploration used temperature 1.2 and sampled up to five trajectories per query; GPT-4o evaluated feedback.Source: §4.2 · pp. 6–7

4.3 Main Results

  • Difficulty-guided sampling added trajectories for hard websites, but optimization was unstable; adding GPT-4o-sampled trajectories produced some overall performance enhancement.The alternative added trajectories sampled by the agent itself; the paper notes sampling randomness and gives Google Flights, Booking, and GitHub as examples.Source: §4.3 · p. 7

4.4 Discussion

  • As optimization progressed, successful tasks on familiar websites took fewer interaction steps; shorter trajectories may increase hallucinated answers, and high F@1 alongside low S@1 indicates mistaken completion.F@1 is first-exploration finish rate and S@K is success within K explorations.Source: §4.4 · pp. 7–8, Table 3 · p. 8
  • Agents increasingly used search-engine restarts, and successful restart trajectories made up a rising share of all successful trajectories.The paper says this addresses some navigation failure issues.Source: §4.4 · p. 8
  • Learning-rate and global-batch-size adjustments had little significance compared with webpage-navigation challenges and sampling randomness.Source: §4.4 · p. 8

5 Conclusion

OpenWebVoyager builds a multimodal web agent through imitation learning followed by iterative exploration, feedback, and optimization. Its cycles use GPT-4o supervision to select trajectories for further learning while reducing human intervention.

5 Conclusion

  • OpenWebVoyager uses an Idefics2 backbone trained first through imitation learning, then through iterative exploration, feedback, and optimization.The framework targets web operation in complex, dynamic online environments and supports multi-image understanding.Source: §5 · pp. 8–9
  • During each cycle, the agent explores new web tasks and retains trajectories GPT-4o judges correct for further learning and policy optimization.The process relies primarily on the agent’s self-exploration and GPT-4o supervision, reducing human intervention and allowing continued expansion.Source: §5 · p. 9
  • The framework is intended to help the agent adapt to complex and dynamic online web environments through multi-image understanding.Source: §5 · p. 9

Limitations

The approach is limited by its action set, backbone scale, and reliance on accessibility trees rather than direct screenshot-based planning.

Limitations

  • The simulated environment includes clicking, typing, and scrolling, but not advanced actions such as dragging or zooming.Source: Limitations · p. 9
  • The 8B-parameter Idefics2 backbone may limit navigation on unseen-domain websites and responses to complex user queries.Low performance on complex websites may also reduce exploration efficiency, yielding minimal improvement and a time-consuming optimization process.Source: Limitations · p. 9
  • The model primarily relies on accessibility trees; the authors identify stronger visual grounding and multi-image reasoning as needed for screenshot-based planning.Source: Limitations · p. 9

Ethics Statement

The experiments used human supervision and GPT-4 monitoring, with tasks selected to avoid sensitive content and consequential transactions. The environment combines screenshots with accessibility trees and uses basic web actions.

Ethics Statement

  • Human supervision and GPT-4 monitoring were used; no trajectory violations were detected, and some sensitive advertisement or news tasks were filtered.Tasks excluded private information, bookings, and payments, and sampling was slow to avoid pressuring websites.Source: Ethics Statement · p. 9

A Environment and Prompts

  • The environment uses screenshots and accessibility trees as observations, with clicks, inputs, and scrolling as its basic actions.Selenium locates elements using their saved accessibility-tree position information.Source: A Environment and Prompts · p. 11
  • The agent retains WebVoyager’s error-reflection mechanism, which prompts it to revise its thought and action when an action fails.Source: A Environment and Prompts · p. 11

B Algorithm

The algorithm begins with imitation learning and proceeds through exploration-feedback-optimization cycles. GPT-4o samples expert trajectories for initial training and evaluates exploration trajectories, with API cost shaping the balance between the phases.

B Algorithm

  • In imitation learning, GPT-4o samples expert trajectories through WebVoyager, while in later cycles it evaluates agent trajectories using one API call per trajectory.Initial expert sampling requires many OpenAI API calls.Source: B Algorithm · p. 11
  • The algorithm balances costly initial sampling against the possibility that a stronger base model could improve exploration efficiency.If imitation-learning gains are not evident, additional GPT-4o calls for that phase may not be cost-effective, and agent-led exploration with GPT-4o supervision may be preferable.Source: B Algorithm · p. 11
  • The algorithm takes an LMM agent, GPT-4o agent and evaluator, and separate query sets for imitation learning and exploration-feedback-optimization, and outputs a fine-tuned agent.Source: B Algorithm · p. 11

C Details of Datasets

Training and exploration-feedback-optimization use queries from 48 websites, while evaluation draws on WebVoyager and Mind2Web test sets.

C Details of Datasets

  • 48 websites are used for imitation learning and exploration-feedback-optimization: 15 from WebVoyager and 37 from Mind2Web, with four duplicates.The inference set includes all WebVoyager test queries and selected Mind2Web cross-task and cross-website queries.Source: C Details of Datasets · p. 11
  • Mind2Web cross-task and WebVoyager test websites were seen during training, whereas Mind2Web cross-website test websites were unseen.Some task time information was updated for testing, but task expressions were unchanged.Source: C Details of Datasets · p. 11
  • The test set combines all WebVoyager queries with selected Mind2Web cross-task and cross-website queries.The Mind2Web selection includes both learned and unlearned websites.Source: C Details of Datasets · p. 11

Queries preparation for Imitation Learning

Imitation-learning queries were assembled from four sources to diversify trajectories, with sources spanning existing datasets, generated and human-written tasks, and user queries.

Queries preparation for Imitation Learning

D Example Trajectories

Example trajectories show successful navigation through direct browsing and search-engine recovery, alongside a case where the agent hallucinates an answer after failing to find a suitable recipe.

D Example Trajectories

  • On Allrecipes, the agent failed to find a qualifying cookie recipe but answered with “Classic Chocolate Chip Cookies,” including a hallucinated cook time.The authors suggest it may have treated “Classic” in the accessibility tree as a recipe despite the lack of relevance.Source: D Example Trajectories · p. 12, Figure 7 · p. 17
  • The Google Flights example reports an approximately $750 average round-trip fare from Chicago to London for October 20–23, 2024.The agent navigates directly on the Flights webpage using screenshots and accessibility-tree text.Source: D Example Trajectories · p. 12, Figure 5 · p. 15
  • After mistakenly assuming GitHub required login, the agent restarts from Google Search and finds electron/electron’s 33.0.0-beta.4 release dated Sep 25, 2024.This is presented as a successful recovery trajectory.Source: D Example Trajectories · p. 12, Figure 6 · p. 16

Related papers

Loading 2410.19609v1…