WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

• Conversational web navigation requires agents to complete real-world browser tasks while communicating with users through multi-turn dialogue.The problem supports applications including voice-controlled navigation, accessibility, and reducing repetitive work, while testing interaction and generalization abilities.Source: §1 · pp. 1–2
• WEBLINX contains 2,337 expert demonstrations across 155 real-world websites, with over 100K actions and utterances linked to DOM trees, screenshots, and video frames.The benchmark supports training dialogue-enabled navigation agents and testing adaptation to new websites, categories, geographies, and no-visual-access settings.Source: §1 · p. 2
• Dense Markup Ranking prunes large HTML pages by ranking elements against action history, then combines selected DOM content with instructions, screenshots, and dialogue context.The approach addresses the difficulty of processing pages containing thousands of elements in real time.Source: §1 · p. 2
• The paper introduces action-specific metrics that distinguish evaluation criteria for different actions, such as clicking versus navigator utterances.These metrics account for minor response differences and overlapping-element selections that can make correct actions appear incorrect.Source: §1 · pp. 2–3
• Smaller finetuned models outperform zero-shot multimodal LLMs, including GPT-4V (OpenAI, 2023a), but all finetuned models struggle with unseen websites and other novel settings.Sheared-LLaMA (Xia et al., 2023) outperforms the much larger screenshot-pretrained Fuyu, while models still make simple interaction errors.Source: §1 · pp. 2–3

2.1 Web Navigation Agents

• Earlier web agents commonly focused on single tasks in simulated environments such as MiniWoB++, where reinforcement learning reached human-level performance but transferred poorly to realistic settings.WebShop improved realism with over 12K human-written instructions, yet models still relied on clean HTML and simple visual representations.Source: §2.1 · p. 3
• WEBLINX targets multi-turn dialogue on real-world websites, extending autonomous navigation benchmarks and earlier exploratory dialogue-based agents toward user-steerable, precise goals.Prior resources include MIND2WEB, WebArena, and VisualWebArena, while WEBLINX generalizes simulated instructors, semantic-parsing agents, and iterative tool resolution.Source: §2.1 · p. 3

2.2 Website Representations

• WEBLINX represents websites with a dense markup ranker that selects relevant DOM elements and can combine them with high-resolution browser screenshots.This builds on prior rule-based, accessibility-tree, graph-embedding, model-based, and visual feature-extraction approaches for simplifying website representations.Source: §2.2 · pp. 3–4

2.3 Conversational Interfaces

• Compared with prior conversational interfaces, WEBLINX covers 2337 demonstrations across diverse real-world tasks with considerably longer dialogues involving dynamic topic switching.RUSS (Xu et al., 2021) provided 80 domain-specific-language-annotated demonstrations, while Dialog2API (Shu et al., 2022) and META-GUI (Sun et al., 2022) addressed API services and mobile apps rather than general websites.Source: §2.3 · p. 4

3 WEBLINX

• WEBLINX contains 2,337 demonstrations averaging 43 turns across 155 real-world websites from 15 geographic areas, organized into 8 categories and 50 subcategories.Source: §3 · p. 4
• Each demonstration is a sequence of website states and actions, with every action following one of 5 core intents.The benchmark includes TRAIN, VALID, and TESTIID splits for in-domain generalization, plus 4 out-of-domain splits for varied evaluation scenarios.Source: §3 · p. 4
• The benchmark records instructor–navigator dialogues in which navigators control browsers, with demonstrations collected by 8 trained expert annotators and independently validated.An instructor communicates with a navigator through chat, while only the navigator operates the browser.Source: §3 · p. 4

3.1 Representing actions and states for modeling

• At each turn, models may use candidate elements, the DOM tree, a browser screenshot, the instructor’s utterance, viewport size, and interaction history to predict an action.States can omit some components, especially early in a demonstration when dialogue is still defining the objective.Source: §3.1 · p. 4
• Because of input-length limits, interaction history is restricted to the five previous actions and five previous utterances, excluding past element and screenshot representations.Generated actions use a predefined textual structure that can be parsed and executed in a browser.Source: §3.1 · p. 5

4.1 Metrics

• Task success rate is inappropriate for WebLINX’s evolving multi-turn objectives, so evaluation estimates predicted–reference action similarity at each turn.The framework follows established dialogue-system evaluation approaches rather than judging only whether a final state was reached.Source: §4.1 · p. 5
• Intent Match measures whether predicted and reference actions share the same intent, while IoU measures overlap between their element arguments.IoU applies to click, textinput, and submit actions; it rewards visual overlap, penalizes size mismatch, and assigns zero to non-overlapping elements.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 Turn-level score and overall score

• Text arguments use chrF or segmented URLF, scaled by Intent Match, with textinput combining element IoU and text F1 in its turn score.The overall score is the micro-average of turn-level scores across element-group and text-group intents.Source: §4.1 · p. 5, §4.2 · p. 5

5.1 Dense Markup Ranking (DMR) for Candidate Selection and Input Representation

• DMR ranks candidate HTML elements five times faster than the prior approach, trading slightly lower recall for real-time processing.It reduces overhead through simplified element representations, dual encoders, and similarity-based learning, using a finetuned MiniLM variant.Source: §5.1 · pp. 5–6
• The representation combines selected candidates with state information and truncates hierarchically, incorporating HTML attributes, viewport size, XML Path, and candidate bounding boxes.This design aims to reduce information loss when DOM content or action history exceeds the model’s input limit.Source: §5.1 · p. 6, §5.2 · p. 6

5.2 Modeling Actions

• Action models predict structured action strings from text-only, image-to-text, or multimodal inputs, with experiments covering 19 zero-shot and finetuned models.Text-only models receive instructions, pruned DOM, candidate descriptions, and history; image-to-text models use screenshots directly; multimodal models combine screenshots with textual webpage representations.Source: §5.2 · p. 6

6.1 Overview of Results

• DMR-based input representation improves 3B Flan-T5 over MindAct, while both LLaMA-based models outperform Flan-T5 and MindAct despite Sheared-LLaMA being smaller than Flan-T5.Sheared-LLaMA performs equally well compared with LLaMA-2 13B.Source: §6.1 · p. 7
• Fuyu-8B outperforms 1.3B Pix2Act overall but trails it on intent matching and text prediction, while chat-based text-only LLaMA models outperform Fuyu-8B.The comparison attributes Fuyu’s overall advantage partly to text-input access and its larger parameter count, while indicating screenshot-finetuned multimodal models remain behind instruction-finetuned chat models.Source: §6.1 · p. 7
• Proprietary GPT-3.5T and GPT-4T outperform open-source LLaMA-2 models zero-shot, but finetuned Sheared-LLaMA and LLaMA-2 surpass GPT-3.5F, while GPT-4V and GPT-4T perform similarly.The similar GPT-4V and GPT-4T results suggest existing multimodal models may not effectively use screenshots for action prediction.Source: §6.1 · p. 7
• Finetuned models perform well on familiar websites but struggle on unseen websites, with unseen subcategories harder than new websites within known categories.LLaMA-13B performs poorly on TESTCAT, indicating difficulty transferring skills such as restaurant booking to medical appointments.Source: §6.1 · p. 7

6.2 Qualitative Assessment

• GPT-4V often selects incorrect or suboptimal click targets, whereas finetuned LLaMA-2 addresses those cases but can still click irrelevant elements.The qualitative assessment covers click actions in examples where intent is correctly predicted.Source: §6.2 · p. 7
• For text input and conversational responses, GPT-4V confuses fields, repeats or omits content, uses a different writing style, shares irrelevant links, and refuses feasible assistance, while LLaMA-2 succeeds in some cases but still misclassifies actions, omits titles, or misses follow-up questions.LLaMA-2 learned annotator writing style but retained several textinput and say-action failures.Source: §6.2 · pp. 7–8

7.1 Experimental Findings

• Finetuned larger multimodal models surpass smaller image-only models but remain behind finetuned text-only models, while DMR-based representations improve performance.On out-of-domain splits, text-only decoders perform close to smaller variants, but finetuned models consistently surpass zero-shot models.Source: §7.1 · p. 8
• Qualitative assessments find that even the best zero-shot models make simple, unjustified errors, highlighting the need for better generalization to unseen scenarios.Such generalization is presented as necessary for agents intended to work in the real world.Source: §7.1 · p. 8

7.2 Limitations

• Because the benchmark contains only static demonstrations, it cannot meaningfully evaluate model behavior on alternative trajectories.The static approach nevertheless enables training on diverse real websites without recreating them from scratch.Source: §7.2 · p. 8
• The evaluated architectures have inherent modality limitations: text-only models cannot draw on a canvas or describe images, while future multimodal-specific technical contributions may address these constraints.The examples illustrate why some tasks require capabilities beyond text-only processing.Source: §7.2 · pp. 8–9

Conclusion

• WEBLINX combines an expert-built benchmark, task framework, evaluation metrics, and dense markup ranker for conversational navigation on real-world websites.Source: §8 · p. 9
• Finetuned chat-based decoder models achieve the best results, but still struggle to generalize to out-of-domain splits.Source: §8 · p. 9

Future directions

• Future work should design multimodal architectures that efficiently process visual input together with structured information.Source: §8 · p. 9
• Models should be evaluated across broader scenarios, including complex websites and advanced browser events.Source: §8 · p. 9
• The framework could expand beyond browser tasks to OS-level interactions.Source: §8 · p. 9
• Future training could leverage reward-based methods such as RLHF (Christiano et al., 2017) and DPO (Rafailov et al., 2023), alongside selfexperience and grounded synthesis.Source: §8 · p. 9

Impact Statement

• Conversational web navigation keeps a human instructor in the loop, aiming to automate difficult, repetitive, and error-prone steps rather than replace users’ higher-level problem solving.This design is presented as a distinction from fully autonomous navigation, which could automate knowledge work and contribute to job losses.Source: Impact Statement · p. 9
• Web navigation agents could enable malicious activities at scale, including spam generation and impersonation for fraud, while existing tools such as Selenium already partially automate such activity [7].The paper argues that releasing models and data can support mitigation research, including red teaming and systems robust to malicious agents.Source: Impact Statement · p. 9
• Making the models and data accessible can help researchers study defenses against malicious agents, even though malicious actors may independently build such systems using commercial services.The proposed mitigation pathway includes incorporating the models into red teaming procedures.Source: Impact Statement · p. 9
• The models should remain under human supervision with multi-turn dialogue enabled, undergo exhaustive testing and safeguards, and not be deployed beyond research.Misinterpreted instructions could trigger harmful unintended actions, such as booking the wrong flight and causing financial loss.Source: Impact Statement · p. 9
• WEBLINX was collected with trained expert annotators who were familiarized with the task and project purpose and paid fair wages relative to their countries of employment.The dataset uses publicly accessible, safe websites and accounts created specifically for collection.Source: Impact Statement · p. 9
• The dataset avoids references to account holders’ identities to preserve privacy.The accounts appearing in the dataset were specifically created for data collection rather than taken from annotators’ personal identities.Source: Impact Statement · p. 9

A.1 Supplementary Statistics

• WEBLINX emphasizes say, click, and load actions, while load and hover occur less often because many tasks stay on one page and unnecessary hovering is removed.The benchmark exposes 10 browser actions and one chat action, with arguments including strings, integers, elements, and browser-tab identifiers.Source: §A.1 · p. 19, Figure 5 · p. 19, Table 6 · p. 19
• Most demonstrations contain 40–50 turns, but TESTVIS has substantially fewer because non-visual instructors cannot request screen-dependent follow-up actions such as applying visible filters.AI tools are used for tasks such as summarizing news, while models learn general actions rather than tool-specific behavior.Source: §A.1 · pp. 19–20

A.2 Categories and Subcategories

• The dataset organizes demonstrations into 8 categories and 50 subcategories across 155 URL sub-domains, with TESTCAT using infrequent subcategories to test generalization.A demonstration may contain multiple subcategories when it uses multiple websites.Source: §A.2 · p. 20, Table 10 · p. 20

A.3 Input Processing Details

• Model inputs combine state, candidate elements, screenshots, and restricted history according to modality, with history limited to w = 5 recent actions or utterances.Candidate selection ranks DOM elements and presents the top-k candidates; text-only, image-to-text, and multimodal models receive different formatted representations.Source: §A.3 · pp. 20, 22

A.4 Output Processing Details / A.5 Data Collection Details / A.6 Actions and Intents / A.7 Websites overview

• Demonstrations are recorded with screenshots, DOM trees, bounding boxes, browser actions, and chat, then curated for ordering, typographical accuracy, alignment, and action validity across diverse safe websites.The evaluated intents are click, load, say, submit, and textinput; change and scroll support training but are not evaluated reliably.Source: §A.4 · p. 22, §A.5 · p. 23, §A.6 · p. 23

B.1 Optimal Text Representation (OTR) / B.2 Strategic Truncation

• Strategic truncation preserves important structure by allocating token limits across rendered sub-components and reducing longer components until the target total, such as L = 2048.OTR adds DOM attributes, viewport size, XPath, bounding boxes, and unique candidate identifiers before truncation.Source: §B.2 · p. 27, §B.1 · p. 27

B.4 Technical Aspects of Dense Markup Ranking (DMR)

• MiniLM achieves better overall retrieval results than other retrievers, approaches MindAct’s DeBERTa cross-encoder, and is substantially more computationally efficient, so it supplies all downstream candidates.DMR ranks candidates using cosine similarity between encoded state representations and candidate elements, selecting the top-k results for action prediction.Source: §B.4 · pp. 28–29, Table 13 · p. 29

B.4.1 EMPIRICAL SPEED IMPROVEMENTS

• 186 ms per turn versus 916 ms makes DMR-MiniLM substantially faster than M2W-DeBERTa for candidate selection, supporting real-time navigation use cases.Across 24,418 active turns, DMR-MiniLM processed the training set in 4,545 seconds versus 22,385 seconds for M2W-DeBERTa.Source: §B.4.1 · p. 29

B.5.1 TEMPLATE FOR PIX2ACT

• Pix2Act receives viewport information, the first and previous user utterances, and previous actions through a dedicated header template.The template is described in Appendix B.5.1.Source: §B.5.1 · p. 29

B.5.2 TEMPLATE FOR CHAT-BASED MODELS (LLAMA, GPT)

• Chat-based models receive pruned HTML, recent dialogue, viewport dimensions, previous actions, and ten candidate elements, then output one formatted browser action.Permitted actions include changing values, clicking, loading URLs, speaking, scrolling, submitting, and text input.Source: §B.5.2 · p. 29

B.5.3 TEMPLATE FOR INSTRUCTION-BASED MODELS (FLAN, FUYU, MINDACT)

• Instruction-based models use the same action vocabulary and candidate-element representation while formatting recent user–assistant turns and requiring only the selected action.Their templates provide the first and last four utterances, a bounded turn history, viewport size, and ten candidates.Source: §B.5.3 · p. 30

B.7 Hyperparameters

• The experiments used fixed seeds and one finetuning run per hyperparameter set, with maximum output tokens 256, bf16 precision, AdamW, and FSDP only for models with 7B+ parameters.Source: §B.6 · p. 30, §B.7 · p. 31

B.6 Model Implementation

• The implementation evaluates text-only, screenshot-based, decoder-only, multimodal, and API models, adapting inputs and starting checkpoints to each architecture.MindAct uses DMR’s top k = 10 candidates instead of 13-step elimination over k = 50; Pix2Act predicts coordinates from a screenshot; LLaMA-2, Sheared-LLAMA, GPT, GPT-4V, and Fuyu are also explored.Source: §B.6 · pp. 30–31
• Fuyu is finetuned as an 8B unified image–text decoder pretrained on high-resolution images and capable of locating objects on real websites.Its architecture uses linear projections of image patches rather than a separate image encoder.Source: §B.6 · p. 31

B.8 Input Samples

• The input samples span MindAct, chat-based models, instruction-based models, and Pix2Act, with the latter using embedded header text above its screenshot.Appendix B.8.1 covers MindAct, Appendix B.8.2 instruction-based models, Appendix B.8.3 chat-based models, and Figure 7 Pix2Act.Source: §B.8 · p. 32, Figure 7 · p. 32

B.8.1 SAMPLE INPUT FOR MINDACT

• MindAct receives available webpage HTML, truncated dialogue utterances, recent actions, and top candidate elements before selecting the next action.The sample exposes HTML elements, the first and last four utterances, only the last five actions, and ranked candidate elements.Source: §B.8.1 · p. 32
• The MindAct example ends with a strict instruction to output only the correctly formatted action, without explanation.Its dialogue includes opening an encyclopedia website and a subsequent request to search for biotechnology.Source: §B.8.1 · p. 32

B.8.2 SAMPLE INPUT FOR INSTRUCTION-BASED MODELS (FLAN, FUYU)

• The instruction-based HTML sample exposes a rendered encyclopedia page with searchable inputs, navigation controls, descriptive text, and associated web-task identifiers.The representation includes both structural HTML and element metadata used to identify possible actions.Source: §B.8.2 · p. 33
• Instruction-based models receive pruned HTML and must predict an executable action from a fixed set including change, click, load, say, scroll, submit, and text_input.The permitted actions replace bracketed values with appropriate strings, identifiers, URLs, coordinates, or text.Source: §B.8.2 · p. 33
• The instruction-based sample combines recent dialogue context with viewport dimensions, a five-turn history limit, and top candidate elements represented by UIDs, tags, paths, and bounding boxes.Candidate entries also include attributes or visible text, such as the search input, form, navigation link, and close control.Source: §B.8.2 · p. 33
• The instruction-based prompt concludes with a multi-turn example and asks the assistant to select the best action in the prescribed format.The example includes greeting, loading Encyclopedia.com, and the user request to search for biotechnology.Source: §B.8.2 · p. 33

B.8.3 SAMPLE INPUT FOR CHAT-BASED MODELS (LLAMA, GPT)

• B.8.3 SAMPLE INPUT FOR CHAT-BASED MODELS (LLAMA, GPT): Chat-based models receive pruned HTML, recent dialogue history, viewport context, candidate elements, and a constrained action vocabulary.The example exposes candidate UIDs, tags, attributes, coordinates, and text for actions such as clicking, loading, scrolling, submitting, and text input.Source: §B.8.3 · p. 33

B.9 Output Sample

• B.9 Output Sample: Table 15 presents model outputs for formatted text inputs and Figure 8 inputs used by multimodal models.The accompanying example shows a target action highlighted, while GPT-4V and GPT-4T may input text already present instead of submitting the button.Source: §B.9 · p. 34, Figure 8 · p. 34, Figure 9 · p. 34

C.1 Comparison of Mind2Web representation with OTR

• C.1 Comparison of Mind2Web representation with OTR: Flan-T5 with OTR outperforms MindAct overall and across individual groups, with larger gaps for larger models.The result indicates that carefully constructing P(st, a1:t−1) becomes increasingly important as parameter count grows.Source: Table 16 · p. 36, §C.1 · p. 36

C.2 Comparison of image-only baseline with multimodal models

• C.2 Comparison of image-only baseline with multimodal models: Fuyu-8B outperforms Pix2Act overall, while GPT-4V is consistently worse than Fuyu-8B and generally behind Pix2Act.The comparison supports finetuning, larger models, and richer candidate-inclusive textual information; GPT-4V is the only model not finetuned.Source: Table 17 · p. 36, §C.2 · p. 36

C.3 Assessing impact of model size for text-only decoders

• C.3 Assessing impact of model size for text-only decoders: Zero-shot performance rises with model size, whereas finetuning yields substantial gains and makes even 2.7B models surpass GPT-4T on scenarios.LLaMA-2-13B only slightly exceeds the 2.7B finetuned variant, while finetuned GPT-3.5T trails the smallest LLaMA model, possibly because API hyperparameters are limited.Source: §C.3 · pp. 36–37, Table 18 · p. 37

C.4 Generalization capabilities of evaluated models

• C.4 Generalization capabilities of evaluated models: Out-of-domain performance drops sharply after finetuning, yet OTR, multimodality, and finetuning remain beneficial; decoder-only models are strongest, while multimodal models trail text-only counterparts.The Sheared-LLaMA-2.7B–LLaMA-2-13B gap narrows out of domain, suggesting greater robustness for Sheared-LLaMA; GPT-4T and GPT-4V also become closer zero-shot.Source: §C.4 · p. 37

C.5 Extended Qualitative Assessment

• Finetuned LLaMA-2-13B generally selected correct click actions, while GPT-4V confused links with different time frames, repeated completed steps, and sometimes chose less efficient actions; LLaMA also failed on inert headings.Figure 10 compares four click scenarios, including cases where GPT-4V was correct and LLaMA was wrong.Source: §C.5 · p. 37, Figure 10 · p. 38
• For text input, GPT-4V sometimes entered recipients instead of subjects or usernames instead of passwords, while both models missed later steps and earlier information in multi-step tasks.LLaMA correctly handled the subject and password examples, but both models failed to change a translation language after entering text and to retain an earlier email title.Source: §C.5 · pp. 38–39, Figure 11 · p. 39
• LLaMA-2.7B correctly submitted a completed restaurant form, whereas GPT-4T typed a date and GPT-4V repeated existing text despite reading the field correctly.The example illustrates a mismatch between visual reading and logically appropriate action.Source: §C.5 · p. 38
• LLaMA-2-13B learned navigator-specific acknowledgment style and usually produced short correct utterances, while GPT-4V sometimes gave unhelpful, refusing, or semantically different responses.Average say-intent lengths were 58.29 characters for LLaMA on validation and in-domain test data and 60.41 for GPT-4V; the semantically different follow-up could still be pragmatically valid.Source: Table 19 · p. 39, §C.5 · p. 40

C.6 Comparison with human performance

• The human-agreement evaluation compared model actions with the closest alternative annotations using the paper’s proposed per-turn metrics.Three annotators predicted best actions on a validation subset, yielding 402 annotations across 134 turns.Source: §C.6 · p. 40
• 65% of the original human navigator’s overall agreement score was achieved by finetuned LLaMA-2-13B, versus 31% by zero-shot GPT-4V on 402 annotations across 134 validation turns.The comparison used three annotators and a subset of the validation set; annotating all test splits was estimated to take about 10 months.Source: §C.6 · p. 40, Table 20 · p. 40

C.7 Augmenting non-finetuned models with in-context examples

• Prompts augmented with an action description and a concrete training example were tested against zero-shot prompts on TESTIID, but no substantial differences were observed.Examples were truncated to fit the common 2048-token limit while preserving a few demonstrations for each action type.Source: §C.7 · p. 41, Table 21 · p. 41

D Additional Result Tables

• The supplementary results report intent-match scores and full or grouped evaluations across in-domain, out-of-domain, TESTCAT, TESTGEO, TESTVIS, and TESTWEB splits.Tables 22–27 provide split-specific results, while Tables 28–30 group Element Group, Text Group, and overall scores; zero-shot and finetuned results are separated.Source: D Additional Result Tables · p. 43, Table 22 · p. 43, Table 23 · p. 43

Project Information

• The project collects demonstrations in which an instructor gives natural-language instructions and a navigator systematically follows them in a web browser.The interaction is conducted by two people using Google Chrome, Zoom, and an internet connection.Source: Project Information · p. 48, Ingredients · p. 48

Ingredients

• Preparation requires installing the Chrome extension, optionally creating a dedicated browser profile, and configuring Zoom recording and chat settings.The extension is loaded unpacked in Chrome, while Zoom is configured for chat saving, local recording, and automatic recording.Source: Ingredients · p. 48

Updating the extension

• Annotators should regularly compare their installed extension with the current version and reinstall it when a newer release exists.The current version appears at the top of the document, while the installed version is shown at chrome://extensions/.Source: Updating the extension · p. 48

Demonstrations

• Demonstrations use a Zoom call in which the navigator shares only the Chrome window, starts recording, and receives task instructions through chat while operating the browser.The instructor may provide intermediate instructions or answer questions, and the navigator may ask for missing details before saving and submitting the recording.Source: Demonstrations · p. 49

Recording

• Each submitted recording combines a ZIP archive of metadata, events, screenshots, and HTML snapshots with the saved Zoom chat history.The recording is submitted through the web interface after the call and browser interaction are complete.Source: Recording · p. 49

Actions

• The navigator can navigate to URLs, click elements, enter text, and scroll, but should not open new tabs or use drag-and-drop.A tab opened automatically by a webpage is permitted, whereas manually opening a new tab is not.Source: Actions · pp. 49–50

Tasks

• Tasks may begin underspecified and require intermediate clarification, but demonstrations must stop before actions that change the real world, such as booking, purchasing, or submitting.Annotators are encouraged to cover simple through complex multi-turn tasks while waiting for pages and actions to complete and minimizing unnecessary interactions.Source: Tasks · p. 50