PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Web Navigation with Instruction-Finetuned Foundation Models

Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, Izzeddin Gur

arXiv:2305.11854v4cs.LGcs.AIstat.ML

TL;DR

Autonomous web navigation has relied on costly online exploration and task-specific designs, motivating safer offline learning with stronger generalization. WebGUM combines an instruction-finetuned language model with visual and HTML inputs, and reports substantially stronger performance across web-navigation benchmarks while releasing a much larger multimodal demonstration corpus.

  • Problem

    Prior web-navigation approaches depend on massive online exploration, while available offline data is limited and often lacks visual features needed for spatial perception.

  • Method

    WebGUM jointly finetunes an instruction-finetuned language model and vision encoder to process HTML, screenshots, temporal context, and local visual information for offline action prediction.

  • Results

    WebGUM improves MiniWoB success from 48.4% to 94.2%, outperforms PaLM-540B on WebShop, and transfers positively to real-world Mind2Web planning tasks.

  • Takeaways & Limitations

    Offline training with multimodal foundation models provides a practical route to stronger web navigation while requiring fewer online interactions.

  • Takeaways & Limitations

    The released 347K-episode dataset remains far from the internet-scale data needed for generalist models, and deployment to real-world web navigation is left for future work.

Abstract

from arXiv · show

The progress of autonomous web navigation has been hindered by the dependence on billions of exploratory interactions via online reinforcement learning, and domain-specific model designs that make it difficult to leverage generalization from rich out-of-domain data. In this work, we study data-driven offline training for web agents with vision-language foundation models. We propose an instruction-following multimodal agent, WebGUM, that observes both webpage screenshots and HTML pages and outputs web navigation actions, such as click and type. WebGUM is trained by jointly finetuning an instruction-finetuned language model and a vision encoder with temporal and local perception on a large corpus of demonstrations. We empirically demonstrate this recipe improves the agent's ability of grounded multimodal perception, HTML comprehension, and multi-step reasoning, outperforming prior works by a significant margin. On the MiniWoB, we improve over the previous best offline methods by more than 45.8%, even outperforming online-finetuned SoTA, humans, and GPT-4-based agent. On the WebShop benchmark, our 3-billion-parameter model achieves superior performance to the existing SoTA, PaLM-540B. Furthermore, WebGUM exhibits strong positive transfer to the real-world planning tasks on the Mind2Web. We also collect 347K high-quality demonstrations using our trained models, 38 times larger than prior work, and make them available to promote future research in this direction.

1 INTRODUCTION

Web navigation involves sequential interaction with web interfaces, but prior online-RL approaches require costly trial and error while offline methods have lagged. WebGUM addresses this with a multimodal, instruction-following foundation-model agent trained offline on HTML, screenshots, and demonstrations.

  • Web navigation agents follow user instructions through sequential interface actions such as clicking, typing, form filling, retrieval, and email handling.
  • Prior work commonly used online reinforcement learning with task-specific models, while offline training is safer but had lower performance.Online failures can have practical consequences, such as account freezes or misdirected emails.
  • WebGUM combines a language model and vision transformer to process HTML and screenshots with local and temporal perception.Its multimodal tokens support grounded spatial understanding for web navigation.
  • 45.8%: WebGUM outperforms previous best offline approaches on MiniWoB++ and also exceeds existing online-RL approaches, humans, and private-LLM agents.The reported advantages include multimodal perception, HTML understanding, and multi-step reasoning.
  • 347K: the authors collect and release multimodal expert demonstrations, reported as 38 times larger than the existing unimodal dataset.The dataset is intended to support future offline web-agent research.

2 RELATED WORK

Related work spans simulated web-navigation benchmarks and language-model agents, while WebGUM emphasizes multimodal finetuning with domain-specific data. Its MiniWoB++ comparison is framed against offline, online-RL, human, and LLM-based systems.

  • MiniWoB++ evaluates autonomous agents on simulated websites ranging from primitive actions to complex multi-step tasks such as email sending and flight booking.
  • Private language-model agents use few-shot demonstrations, self-improvement, code generation, or structured prompts to approach RL-finetuned and human performance.
  • WebGUM focuses on multimodality and finetuning with domain-specific data, achieving competitive performance with PaLM-540B using only 3 billion parameters.
  • Table 1 reports average success rates on MiniWoB++, comparing WebGUM with offline methods, online-RL systems, humans, and GPT-4-based agents.The table caption notes that “+” denotes extra billions of frames required during online RL.

3 PRELIMINARIES

The paper formulates web navigation as deterministic sequential decision making over webpages, actions, instructions, transitions, and success criteria. States contain both raw HTML and screenshots, while actions are constrained click or type functions.

  • Autonomous web navigation is modeled with state space S, action space A, deterministic transitions, instruction space G, and binary episodic success criteria.
  • An episode succeeds when the instruction is satisfied and fails after an invalid action or wrong terminal state.
  • Each webpage state consists of raw HTML represented as text and a screenshot represented as an image.
  • Actions use the form function(selector, text), where function is click or type and selector is an integer identifying an element.The text argument supplies input for type actions.
  • MiniWoB includes both primitive tasks such as button clicking and text entry and multi-step tasks such as forwarding an email.

4 WEBGUM

WebGUM combines T5 and a vision transformer to encode HTML and screenshot observations, using temporal and local visual tokens for multimodal action prediction. The design is supported by instruction finetuning and large-scale multimodal data collection, with performance improving as data and model capacity grow.

  • 4.1 MULTIMODAL TRANSFORMER MODELS WITH TEMPORAL AND LOCAL PERCEPTION: WebGUM combines a T5 encoder-decoder with a vision transformer that maps screenshots into image tokens, while the decoder predicts actions in text.The T5 encoder consumes visual and HTML tokens jointly.
  • 4.1 MULTIMODAL TRANSFORMER MODELS WITH TEMPORAL AND LOCAL PERCEPTION: Image inputs improve tasks involving dynamic page transitions or visual concepts, including book-flight and click-shape.The figure measures absolute improvement as SR of WebGUM(HTML+Image) minus SR of WebGUM(HTML).
  • 4.1 MULTIMODAL TRANSFORMER MODELS WITH TEMPORAL AND LOCAL PERCEPTION: Temporal tokens use recent screenshot history, while local tokens represent image patches to capture multi-step context and website spatial structure.The history uses H = 2 steps, and local encoding uses one token per patch.
  • 4.2 INSTRUCTION-FINETUNED LARGE LANGUAGE MODELS: Instruction-finetuned Flan-T5 is used because web navigation is inherently instruction-following and may benefit from improved alignment and reasoning.
  • 4.3 LARGE-SCALE DATA COLLECTION WITH LANGUAGE MODEL AGENTS: The available human MiniWoB++ dataset has 12K DOM-only episodes, motivating a larger multimodal dataset that includes screenshots.
  • 4.3 LARGE-SCALE DATA COLLECTION WITH LANGUAGE MODEL AGENTS: The collection pipeline uses finetuned language-model policies and additional agents to build a multimodal dataset containing HTML and screenshots at each step.The resulting multitask dataset contains 401K episodes, including 347K plus 54K episodes.
  • 4.3 LARGE-SCALE DATA COLLECTION WITH LANGUAGE MODEL AGENTS: Larger datasets and models yield higher success rates, while combining temporal and local visual tokens is critical for performance.The figure uses a logarithmic x-axis for dataset and model-size results.

5 RESULTS

WebGUM achieves strong offline web-navigation performance through multimodal perception, instruction-finetuned models, scaling, HTML comprehension, and multi-step reasoning. It also transfers effectively from simulated MiniWoB++ tasks to WebShop and real-world Mind2Web action prediction.

  • Overall results: 94.2% success rate on MiniWoB++ exceeds WebN-T5 by over 45.8% and CC-Net by 0.7%, despite fully offline training and fewer data.WebGUM also surpasses humans and recent LLM-based agents.
  • Temporal and local visual perception: Temporal and local visual tokens together reach 66.1%, outperforming temporal-only tokens at 64.2% and local-only tokens at 64.0%.Different pretrained ViT choices have marginal effects compared with visual-token design.
  • Scaling dataset and model size: WebGUM improves as dataset size increases, while a Base model trained on only 2.8K episodes already reaches 55.7%/66.1%.The 2.8K-episode Base model surpasses prior supervised-learning results of 49.8%/55.6%.
  • HTML comprehension: WebGUM with HTML and image inputs outperforms prior systems on six unseen compositional MiniWoB++ task combinations.The result indicates stronger HTML reading and transfer to unseen compositions.
  • HTML comprehension: +56.2% multimodal and +33.4% unimodal performance over prior finetuned LLMs under distracted HTML attributes.The perturbations add HTML at the top or bottom or insert irrelevant coordinate attributes.
  • Multi-step reasoning: 45.0% success on WebShop outperforms simple baselines and ReAct using prompted PaLM-540B, with WebGUM using 3 billion parameters.The model can compare products with backtracking and select options matching the instruction.
  • Real-world transfer: WebGUM transferred from MiniWoB++ achieves superior performance to MindAct-Large/XL and GPT-4 across Mind2Web cross-task, website, and domain categories.The transfer uses further finetuning on Mind2Web training data.

6 DISCUSSION AND LIMITATION

The paper demonstrates practical offline web navigation and positive transfer to Mind2Web, but deployment-scale multimodal foundation models and broader human-level generalization remain future work.

  • Discussion and limitation: Scaling multimodal foundation models to deployment for real-world web navigation remains future work.The paper reports positive transfer to Mind2Web but does not present deployment-scale multimodal navigation.
  • Discussion and limitation: The 347K-episode multimodal dataset is still far from the internet-scale data needed for generalist models.Broader human-level generalization across diverse real websites and instructions remains unresolved.

7 CONCLUSION

WebGUM is an instruction-following visual-language foundation model for autonomous web navigation, combining HTML and visual inputs with offline training. It improves simulated navigation performance and transfers positively to real-world action prediction, while deployment on the open Internet remains safety-constrained.

  • WebGUM learns web navigation with an instruction-following visual-language foundation model.Its broader design incorporates multimodal perception and foundation-model inductive biases for navigation.
  • WebGUM improves MiniWoB success rates from 48.4% to 94.2% over previous offline-trained state of the art.The paper attributes gains to temporal and local visual tokens, HTML comprehension, and multi-step reasoning.
  • WebGUM outperforms PaLM-540B on WebShop and shows strong positive transfer to real-world action prediction tasks in Mind2Web.
  • The work scales MiniWoB into 347K multimodal expert demonstrations, about 38 times larger than the prior dataset.
  • Deployment on the real-world Internet requires careful attention to security and safety because WebGUM is evaluated only in realistic web simulators.The paper gives account freezing and misdirected email as examples of potential consequences.

D DETAILS ON DATASET AND MODEL SIZE SCALING

WebGUM benefits from scaling both its dataset and model, while instruction-finetuned language models improve HTML comprehension. Even a small dataset and base-sized model surpass prior supervised state of the art.

  • Larger datasets and models produce higher WebGUM success rates for both HTML-only and multimodal models.
  • 55.7% success is achieved with 2.8K HTML episodes and a Base-size model, surpassing the previous supervised state of the art at 48.4%.The 2.8K-episode dataset is about 25% of the previous dataset, and the Base model has about 7.3% of its parameters.
  • Model scaling may matter more than dataset scaling because low-capacity models can cap performance at a lower level.
  • Flan-T5 achieves better HTML-comprehension performance than T5 after both are finetuned on the training dataset.

F DATASET DETAILS

The multimodal MiniWoB++ dataset is built from successful trajectories generated by finetuned language-model policies and additional private-LLM demonstrations. It contains about 347K episodes overall.

  • The dataset uses successful trajectories from a public finetuned-LLM policy run for 10,000 episodes per task.Only successful trajectories are retained to maintain dataset quality.
  • Additional demonstrations are collected with Synapse, a private-LLM-based agent.
  • The resulting multimodal dataset contains about 347K episodes in total.

G PER-TASK PERFORMANCE OF MINIWOB++

The MiniWoB++ evaluation reports per-task success rates across 56 tasks and measures the gain from adding images to HTML. Visual inputs are especially useful for dynamic transitions and tasks requiring global page context.

  • MiniWoB++ evaluation averages success rates over 56 tasks, with 100 evaluation episodes per task.
  • Table 8 reports per-task MiniWoB++ success rates using baselines from Gur et al. and Zheng et al.
  • Image-modality improvement is computed as WebGUM(HTML+Image) success rate minus WebGUM(HTML) success rate.
  • Visual inputs help WebGUM on multi-step tasks with dynamic page transitions, such as booking flights or using search engines.
  • Visual inputs also help tasks requiring global page context, such as tic-tac-toe or click-shape.

H COMPOSITIONAL EVALUATION ON MINIWOB++

The compositional MiniWoB++ evaluation combines primitive click tasks into ordered multi-step sequences, testing whether agents can execute composed instructions successfully.

  • Compositional task construction: Six compositional tasks combine two or three click tasks, including links, buttons, checkboxes, and dialogs.The task names specify the primitive-task combinations evaluated.
  • Compositional task construction: Agents must complete each constituent click action in the order encoded by the task name.For example, a three-step task requires clicking the proper link, button, and dialog sequentially.
  • Evaluation protocol: The evaluation measures whether agents can resolve instructions formed by simply combining original task instructions.This setup tests multi-step composition rather than isolated primitive actions.
  • Evaluation protocol: Per-task average success rates are reported across all six compositional MiniWoB++ tasks.Table 9 provides the task-level breakdown for the evaluation.

K EVALUATION ON WEBSHOP

On WebShop, WebGUM processes noisy HTML to search for, compare, and select products matching natural-language requirements. Its 3-billion-parameter model achieves 45.0% success, exceeding both conventional baselines and the prompted PaLM-540B ReAct agent.

  • Benchmark and inputs: WebShop simulates online shopping with real-world product data and instructions specifying item features and price constraints.Agents must search, compare, and choose products that satisfy the instruction.
  • Benchmark and inputs: WebGUM uses noisy HTML text inputs because WebShop does not provide screenshots of rendered websites.The model converts raw search and click actions into dictionary-like representations and finetunes Flan-T5-XL with about 1K human demonstrations.
  • Results: 45.0% success is achieved by WebGUM on WebShop, outperforming imitation-learning baselines, IL plus RL-finetuning, and prompted ReAct using PaLM-540B.WebGUM has 3 billion parameters, while ReAct uses PaLM-540B with one-shot prompting and reasoning annotations.
  • Results: The WebShop evaluation selects a score threshold of 50 to balance dataset coverage and proficiency.Table 12 reports average score and success rate under different thresholds.
  • Qualitative behavior: Successful WebGUM episodes include product search, option selection, browsing, comparison, and backtracking through next and previous controls.These examples illustrate multi-step interaction with product pages.
Loading 2305.11854v4…