PaperlayerView exact revision on arXiv

Source-linked AI summary

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Xing Han Lù, Zdeněk Kasner, Siva Reddy

arXiv:2402.05930v2cs.CLcs.CVcs.LG

TL;DR

Conversational web navigation asks agents to complete real-world browser tasks through multi-turn dialogue, a capability limited by website-specific plugins and large webpage representations. The paper introduces WEBLINX and Dense Markup Ranking to benchmark and support this task using pruned DOMs, screenshots, and action history. Smaller finetuned decoders achieve the strongest reported results, but finetuned models still struggle on unseen settings.

  • Problem

    Conversational assistants’ website plugins are website-specific and incomplete, motivating direct browser navigation that retains conversational capabilities.

  • Method

    The paper introduces WEBLINX and Dense Markup Ranking, which prunes HTML by ranking elements against action history before combining the result with instructions and screenshots.

  • Results

    Smaller finetuned decoders outperform multimodal LLMs and zero-shot models, while all finetuned models struggle to generalize to novel scenarios.

  • Takeaways & Limitations

    WEBLINX provides a benchmark for training and evaluating dialogue-enabled navigation agents across realistic websites and unseen settings.

  • Takeaways & Limitations

    The benchmark contains only static demonstrations, so it cannot meaningfully evaluate models’ behavior on alternative trajectories.

Abstract

from arXiv · show

We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve real-world tasks in a multi-turn dialogue fashion. To support this problem, we introduce WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers a broad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller finetuned decoders surpass the best zero-shot LLMs (including GPT-4V), but also larger finetuned multimodal models which were explicitly pretrained on screenshots. However, all finetuned models struggle to generalize to unseen websites. Our findings highlight the need for large multimodal models that can generalize to novel settings. Our code, data and models are available for research: https://mcgill-nlp.github.io/weblinx

1 Introduction

The paper defines conversational web navigation as completing real-world browser tasks through multi-turn dialogue and introduces WEBLINX, a large expert-annotated benchmark for studying this problem. It also proposes HTML pruning through Dense Markup Ranking and finds that smaller finetuned decoders outperform larger multimodal or zero-shot models, although generalization remains difficult.

  • Problem: Conversational web navigation requires an agent to complete real-world browser tasks while communicating with a user through multi-turn dialogue.The task is motivated by the goal of navigating websites directly in the user’s browser while retaining conversational capabilities.
  • Benchmark: WEBLINX contains 2337 expert demonstrations across 155 real-world websites, with over 100K recorded actions and utterances linked to DOM trees, screenshots, and video frames.The benchmark supports training and evaluating dialogue-enabled navigation agents on realistic scenarios, including new websites, categories, and geographies.
  • Method: Dense Markup Ranking compares HTML elements with the full action history to rank relevant elements and prune irrelevant DOM content.The resulting compact representation is combined with action history, detailed instructions, and screenshots for model inputs.
  • Evaluation: The paper introduces action-specific evaluation metrics that distinguish among action types, such as clicking and producing navigator utterances.The benchmark evaluates models using a suite of metrics tailored to different action behaviors.
  • Results: Smaller text-only decoders outperform multimodal LLMs, while all finetuned models struggle to generalize to novel websites and other unseen settings.The experiments compare 19 models across 8 architectures, including zero-shot and finetuned text-only, image-to-text, and multimodal systems.

2 Related Work

Prior web-navigation research spans simulated and real-world task agents, but WEBLINX focuses on multi-turn dialogue over diverse, noisy websites. Its website-representation approach selects relevant DOM elements and can combine them with screenshots.

  • Web Navigation Agents: Simulated environments such as MiniWoB++ enable reinforcement-learning agents through well-defined tasks and flexible task primitives.
  • Web Navigation Agents: WebShop improves realism with e-commerce tasks, but relies on clean HTML and simple visual representations rather than noisy, dynamic websites.
  • Web Navigation Agents: WEBLINX extends autonomous web agents by modeling user-steered, multi-turn dialogue for precise goals on real-world websites.
  • Website Representations: Website-representation methods include rule-based compression, accessibility trees, graph embeddings, and model-based approaches; WEBLINX uses a dense markup ranker to select relevant DOM elements and optionally screenshots.
  • Conversational Interfaces: Earlier dialogue-centric datasets covered narrower settings, whereas WEBLINX contains 2337 demonstrations spanning diverse real-world tasks with longer interactions.

3 WEBLINX

WEBLINX is a benchmark of conversational web navigation built from expert demonstrations on real-world websites. Each demonstration records multi-turn browser interactions, states, actions, dialogue history, and evaluation splits for in-domain and out-of-domain testing.

  • Dataset: WEBLINX contains 2337 demonstrations averaging 43 turns across 155 real-world websites, organized into 8 categories and 50 subcategories.
  • Demonstration Framework: Each demonstration is a sequence of website states and actions, with every action following one of five core intents.
  • Data Collection: Expert annotator pairs collect demonstrations through chat, with an instructor communicating tasks and a navigator controlling the browser.
  • Evaluation Splits: The benchmark includes TRAIN and VALID splits, TESTIID for in-domain generalization, and four out-of-domain splits for distinct evaluation scenarios.
  • State Representation: A state may include candidate elements, the DOM tree, a browser screenshot, instructor utterance, viewport size, and interaction history, but some items may be unavailable at a turn.
  • State Representation: Interaction history is represented by the five most recent actions and five utterances because models have limited input length.
  • Action Representation: Predicted actions use a predefined textual structure containing an intent and argument so they can be parsed and executed in a browser.

4 Evaluation Framework

WEBLINX evaluates turn-level action prediction because conversational objectives evolve during interaction rather than being fully specified initially. Metrics separately assess intent, element overlap, and text similarity.

  • Evaluation Metrics: Task success rate is unsuitable because WEBLINX objectives evolve across turns, so evaluation uses automatic turn-level metrics.
  • Intent Match: Intent Match equals 1 when predicted and reference intents agree and 0 otherwise, measuring action-type recognition without argument correctness.
  • Element Similarity: IoU rewards visual overlap, penalizes substantial size mismatches, and assigns 0 when predicted and reference elements do not overlap.
  • Element Similarity: Element-group actions—click, textinput, and submit—use intersection over union to measure predicted-versus-reference bounding-box overlap.
  • Text Similarity: Text-group actions—load, say, and textinput—use F1-based similarity, with character n-gram matching for text and segmented matching for URLs.
  • Turn-Level Score: Turn-level scores combine intent correctness with element IoU or text F1, producing intermediate values when arguments partially match.

5 Methods

The methods filter webpage representations before predicting structured browser actions, then compare models across modalities and training regimes. Dense Markup Ranking improves candidate-selection efficiency, while evaluation includes both in-domain and out-of-domain settings.

  • Candidate Selection: Candidate selection filters DOM elements before models predict actions, using the task state, selected candidates, and remaining contextual information.
  • Candidate Selection: A prior cross-encoder candidate selector takes 916ms on average, creating processing-time concerns when combined with network latency and LLM inference.
  • Dense Markup Ranking: Dense Markup Ranking is 5 times faster than the prior approach, using simplified element representations, a dual encoder, and similarity-based learning.
  • Input Representation: The truncation strategy exploits hierarchical input structure to reduce information loss when candidate selection still leaves sequences too long.
  • Modeling Actions: Models are categorized as text-only, image-to-text, or multimodal according to whether they receive DOM and history, screenshots, or both.
  • Modeling Actions: The experiments compare zero-shot and WEBLINX-finetuned text-only, proprietary, image-to-text, and multimodal models using aggregated intent, element, text, and overall metrics.
  • Evaluation Settings: Out-of-domain evaluation includes splits designed to test generalization beyond familiar websites and categories, with TESTCAT identified as especially difficult for finetuned LLaMA-2-13B.

6 Experimental Results

Experiments compare model modalities, training settings, representations, and generalization across conversational web navigation. Finetuned text-only models often outperform larger or multimodal alternatives, while all finetuned models struggle on unseen websites.

  • Impact of representation for text-only models: DMR-based input representation improves text-only performance, with Flan-T5 outperforming MindAct among 3B-parameter variants.MindAct was finetuned for a related task but not multi-turn dialogue, whereas Flan-T5 used DMR-based input despite lacking navigation-action training.
  • Impact of representation for text-only models: Sheared-LLaMA and LLaMA-2 outperform Flan-T5 and MindAct, while Sheared-LLaMA performs equally well to LLaMA-2 13B.
  • Image-to-text vs. multimodal models: Fuyu-8B generally outperforms Pix2Act for image-to-text versus multimodal comparisons, but Pix2Act leads on intent matching and text prediction.The passage attributes Fuyu's overall advantage possibly to text input and greater parameter count.
  • Comparing multimodal with chat-based models: Fuyu-8B is outperformed by chat-based text-only LLaMA models, indicating screenshot-pretrained multimodal models remain behind instruction-finetuned chat models.
  • Comparison with proprietary models: In zero-shot settings, GPT-3.5T and GPT-4T outperform open-source LLaMA-2, while finetuned GPT-3.5F is outperformed by Sheared-LLaMA and LLaMA-2.GPT-4V and GPT-4T achieve similar performance, suggesting current multimodal models may not effectively use screenshots for action prediction.
  • Generalization capabilities: All finetuned models show a major TESTOOD–TESTIID gap and struggle to generalize to unseen websites, with unseen subcategories harder than new websites within known categories.LLaMA-13B performs poorly on TESTCAT, such as adapting restaurant-seat booking to medical-appointment booking.
  • Qualitative assessment: Qualitative cases show GPT-4V and finetuned LLaMA-2 make different errors across clicks, text entry, and dialogue responses despite the latter's stronger benchmark performance.GPT-4V selects wrong tabs, fields, or writing styles; LLaMA-2 can click irrelevant elements, choose wrong action types, omit titles, or miss follow-up questions.

7 Discussion

The discussion finds that finetuned text-only decoders perform strongly and DMR improves performance, but models remain limited in out-of-domain generalization and the benchmark cannot test alternative trajectories.

  • Larger finetuned multimodal models can surpass smaller image-only models but remain behind finetuned text-only models.
  • DMR-based representations improve performance, while zero-shot models are consistently surpassed by finetuned counterparts.
  • Qualitative assessments show that even the best zero-shot models can make simple and unjustified errors.
  • Static demonstrations prevent meaningful evaluation of models on alternative trajectories, although they support training on diverse real websites without recreating them.
  • Text-only models cannot draw on a canvas or describe images, reflecting inherent architectural limitations addressed by future multimodal-specific contributions.

8 Conclusion

WEBLINX provides an expert-built benchmark and DMR-based framework for conversational web navigation. The best finetuned chat-based decoders still struggle with out-of-domain generalization, motivating broader multimodal and training approaches.

  • WEBLINX is a large-scale expert-built benchmark covering diverse conversational web-navigation demonstrations on real-world websites.
  • The framework defines the task, data representation, and evaluation metrics, and introduces DMR to summarize webpages.
  • Chat-based decoder models finetuned on WEBLINX achieve the best results but still struggle to generalize to out-of-domain splits.
  • Future directions include multimodal architectures that efficiently combine visual input with structured information and evaluations spanning more complex websites and browser events.
  • Additional proposed directions include OS-level interaction tasks, reward-based methods such as RLHF and DPO, and self-experience or grounded synthesis.

Impact Statement

The paper discusses societal risks and safeguards for conversational web navigation, including job displacement, malicious use, and unintended actions. It emphasizes human supervision and responsible deployment.

  • Automating vs. Elevating Users: Human instructors remain part of the framework, which aims to automate difficult, repetitive, and error-prone steps rather than users’ higher-level work.The stated purpose is to let users focus on reliably solving high-level problems.
  • Malicious Usage and Mitigation: Web navigation models could enable malicious activities such as spam creation and impersonation for fraud at scale.The authors propose using released models and data to research mitigations, including red teaming.
  • Unintended Actions: Misinterpreted instructions can cause harmful unintended actions, such as booking the wrong flight and incurring financial loss.The paper recommends human supervision, continued multi-turn dialogue, exhaustive testing, and safeguards before deployment.
  • Data Collection: The dataset uses publicly accessible, safe websites and specially created accounts without identity references.Expert annotators were trained and paid fair wages relative to their countries of employment.

A.1 Supplementary Statistics

The supplementary material details WEBLINX’s action space, dataset statistics, split construction, input processing, and the efficiency of Dense Markup Ranking. It also explains how dialogue and website coverage shape the benchmark.

  • Dataset Statistics: Say, click, and load dominate the intent statistics, while load and hover occur less often for task- and recording-specific reasons.New links are needed mainly when moving to another website, and unnecessary hover events were removed.
  • Action Space: WEBLINX includes 10 browser actions and 1 chat action, with arguments represented as strings, integers, elements, or browser tab ids.The complete action space is documented in Table 6, including restrictions on agent-selected speakers and tab actions.
  • Input Processing: Dense Markup Ranking selects relevant DOM candidates using similarity scores, then supplies them to the action prediction stage.The method ranks candidates and chooses the top-k elements for prediction.
  • Input Processing: The input pipeline can restrict history to a window of w = 5, retaining recent actions and the first and last instructor utterances.This preserves the initial request while emphasizing recent instruction updates within context limits.
  • Dense Markup Ranking: 186 ms per turn was required by DMR-MiniLM versus 916 ms for M2W-DeBERTa when selecting candidates across 24,418 active turns.DMR-MiniLM processed the full training set in 4,545 seconds, compared with 22,385 seconds for M2W-DeBERTa.

B.8 Input Samples

The input samples show how conversational history, screenshots, pruned HTML, candidate elements, and action formats are assembled for web-navigation models. They culminate in structured action selection for a search task.

  • Model Inputs: Screenshot-oriented samples embed viewport and dialogue information as header text above the image while highlighting the target action.The examples illustrate multimodal or image-to-text input construction.
  • Pruned Page Representation: The model receives pruned HTML containing candidate elements such as a search input, a submit button, navigation controls, and page headings.Candidates include tags, paths, bounding boxes, attributes, text, and unique identifiers.
  • Conversation and Task Context: Sample inputs combine instructor and navigator dialogue with a loaded Encyclopedia page and a request to search for biotechnology.The history includes greetings, opening the website, loading its URL, and the latest search instruction.
  • Action Output: The action interface supports operations including change, click, load, navigator speech, scroll, submit, and text input.The model must replace placeholders with appropriate values and return only the selected action format.

B.8.3 SAMPLE INPUT FOR CHAT-BASED MODELS (LLAMA, GPT)

Chat-based model samples format recent dialogue, viewport information, candidate DOM elements, and prior actions into structured prompts. The accompanying experiments compare representations, modalities, model size, and generalization.

  • Input Format: The sample action vocabulary includes change, click, load, navigator speech, scroll, submit, and text input.The prompt instructs the model to select one correctly formatted action without additional explanation.
  • Input Format: Chat-model prompts retain the first and last four user utterances, the viewport size, the last five turns, and ranked candidate elements.Candidate descriptions include tags, paths, bounding boxes, attributes, and child structure.
  • Representation Comparison: Flan-T5 with OTR outperforms MindAct in overall and group-level performance, with the gap increasing for larger models.The authors interpret this as evidence that constructing the state and history representation matters as models scale.
  • Multimodal Comparisons: Fuyu-8B outperforms both Pix2Act variants on the element group and achieves similar text-group and intent-match performance.The comparison concerns image-only baselines and larger multimodal models.
  • Model Size and Finetuning: Finetuning yields considerable improvements over zero-shot models, while increasing decoder size produces only small additional gains among finetuned models.A 2.7B finetuned model can surpass the best zero-shot model, whereas LLaMA-2-13B only slightly exceeds the 2.7B variant.
  • Generalization: Finetuned models show a sharp performance decrease on new scenarios, underscoring the difficulty of generalization.The authors identify OTR, multimodality, and finetuning as important for stronger overall performance.

C.5 Extended Qualitative Assessment

The qualitative assessment compares GPT-4V with finetuned LLaMA models across click, textinput, submit, and conversational response scenarios. It also measures agreement with alternative human trajectories and tests whether prompts containing action descriptions and examples help zero-shot models.

  • Assessing click: GPT-4V can select incorrect links or fail to recognize the current step in multi-step navigation scenarios.The click assessment includes choosing the wrong time-specific news link and failing to update an already-open location setting.
  • Assessing textinput: GPT-4V sometimes targets the wrong field or performs an illogical text-entry action, while LLaMA correctly follows the requested input.Examples include entering a recipient when the recipient is already specified and typing an email address into a password field.
  • Assessing submit: GPT-4V and GPT-4T can read a filled form yet predict text entry or repetition instead of pressing the required submit button.LLaMA-2.7B correctly predicts submission in the restaurant-booking example.
  • Assessing say: GPT-4V may produce unhelpful or pragmatically mismatched responses, including incorrect links, refusals, or wording that differs from the reference.The assessment distinguishes semantic variation from pragmatic correctness in conversational replies.
  • Comparison with human performance: Human-agreement evaluation collected 402 annotations across 134 validation turns; LLaMA-2-13B reached 65% and zero-shot GPT-4V 31% of the original navigator’s overall score.The comparison uses the closest alternative annotations and is limited to a validation subset.
  • Augmenting non-finetuned models with in-context examples: Prompts augmented with action descriptions and examples were evaluated against zero-shot prompts, with results reported for the TESTIID split.Examples were truncated to fit the common 2048-token limit; the associated table reports no substantial differences.

D Additional Result Tables

The additional results section provides split-specific model scores, grouped results, and evaluation details for the benchmark. It also documents the recording protocol, supported browser actions, task constraints, and operational guidance used to collect demonstrations.

  • Additional result tables: Split-specific results are reported for in-domain, out-of-domain, TESTCAT, TESTGEO, TESTVIS, and TESTWEB evaluations.The tables include intent match to identify failures in predicting the correct intent, alongside grouped results.
  • Additional result tables: Grouped tables report Element Group, Text Group, and overall results, separating zero-shot from finetuned models across the benchmark splits.TESTIID and TESTOOD appear together, as do TESTCAT with TESTGEO and TESTVIS with TESTWEB.
  • Demonstration protocol: Each demonstration records a sequence of website states and actions, with actions drawn from five core intents.The demonstrations represent real-time interactions between an instructor and a navigator controlling a browser.
  • Demonstration protocol: The collection protocol uses Chrome, Zoom, screen sharing, video recording, chat instructions, browser actions, and saved artifacts including metadata, events, screenshots, and HTML snapshots.The navigator performs browser actions according to instructor instructions and submits the resulting recording.
  • Task constraints: Supported browser actions include URL navigation, clicking, text input, and scrolling, while opening new tabs and drag-and-drop are prohibited.The protocol also requires stopping before real-world actions such as booking or purchasing.
  • Task guidance: Collection guidance emphasizes waiting for page loads, avoiding unnecessary actions, asking only relevant questions, and allowing intermediate details when tasks are initially unspecified.Demonstrations should be finalized before changing the real-world state.
Loading 2402.05930v2…