PaperlayerView exact revision on arXiv

Source-linked AI summary

WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents

Michael Lutz, Arth Bohra, Manvel Saroyan, Artem Harutyunyan, Giovanni Campagna

arXiv:2404.05902v1cs.CLcs.AI

TL;DR

Web agents remain less successful than experienced people, and existing approaches face challenges generalizing across websites. WILBUR combines backtracking with ranked and synthesized demonstrations, supported by an automatically scored curriculum. On WebVoyager, it improves on text-only state of the art and approaches a strong multimodal model.

  • Problem

    Even the best web agents remain far less successful than experienced people, while learning how specific websites work is difficult to achieve through pretraining alone.

    Source: §1 · p. 1

  • Method

    WILBUR uses action verification and backtracking, ranks retrieved demonstrations, synthesizes instructions from prior actions, and builds execution data through an LLM-scored autocurriculum.

    Source: §1 · p. 2

  • Results

    53% on WebVoyager is 8% higher than the previous text-only state of the art, and WILBUR is within 5% of a strong multimodal model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The WebVoyager results show that text-only agents can be effective at web navigation, with WILBUR surpassing the multimodal state of the art on five listed websites.

    Source: §5 · p. 9

    Anti-scraping can rate-limit WILBUR and present an empty page, preventing it from progressing on sites such as ArXiv.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

In the realm of web agent research, achieving both generalization and accuracy remains a challenging problem. Due to high variance in website structure, existing approaches often fail. Moreover, existing fine-tuning and in-context learning techniques fail to generalize across multiple websites. We introduce Wilbur, an approach that uses a differentiable ranking model and a novel instruction synthesis technique to optimally populate a black-box large language model's prompt with task demonstrations from previous runs. To maximize end-to-end success rates, we also propose an intelligent backtracking mechanism that learns and recovers from its mistakes. Finally, we show that our ranking model can be trained on data from a generative auto-curriculum which samples representative goals from an LLM, runs the agent, and automatically evaluates it, with no manual annotation. Wilbur achieves state-of-the-art results on the WebVoyager benchmark, beating text-only models by 8% overall, and up to 36% on certain websites. On the same benchmark, Wilbur is within 5% of a strong multi-modal model despite only receiving textual inputs, and further analysis reveals a substantial number of failures are due to engineering challenges of operating the web.

1 Introduction

Web agents remain far less successful than experienced people, and the paper argues that they must learn how specific websites work. WILBUR combines backtracking, retrieved and synthesized demonstrations, and an automatically scored curriculum; on WebVoyager it outperforms prior text-only results.

1.1 Contributions

  • WILBUR verifies each action and, when verification fails, returns to a previous successful state while storing the failure for future steps.The approach frames web tasks as graph exploration, enabling recovery from delayed mistakes.Source: §1 · p. 2, §1.1 · p. 3

1 Introduction

  • WILBUR retrieves both goal-conditioned task demonstrations and website-conditioned page demonstrations as complementary sources of guidance.The demonstrations are drawn from previous agent executions and can support similar tasks or similar pages.Source: §1 · p. 2, §1.1 · p. 3
  • A trained ranking model selects useful demonstrations, while synthesized instructions distill patterns from successful and unsuccessful actions to fit the limited prompt.The ranker predicts whether actions will lead to successful execution; the prompt combines selected examples with concise instructions.Source: §1 · p. 2, §1.1 · p. 3

1.1 Contributions

  • An LLM-generated autocurriculum proposes plausible goals and automatically scores executions, producing successful and unsuccessful trajectories for future demonstrations and instruction synthesis.The paper presents this as a way to bootstrap new websites and tasks without human feedback.Source: §1 · p. 2, §1.1 · p. 3

1 Introduction

  • 53% on WebVoyager is 8% higher than the previous text-only state of the art, and WILBUR is within 5% of a strong multimodal model.The paper also reports a 6% improvement from backtracking and loop detection, and an additional 12% from adaptive in-context learning and autocurriculum, without annotated data.Source: §1 · p. 2

2 Related Work

Prior web agents use modular action and verification pipelines, but retrying does not recover mistakes that become apparent later. Work on demonstrations and in-context learning also leaves open how to select and summarize useful examples within limited prompts.

2 Related Work

  • Common web-agent architectures divide work among actors, retrievers, planners, and verifiers, but retrying cannot recover delayed mistakes; prior backtracking used a fixed WebShop state space.Web-agent research spans specialized and general benchmarks across diverse websites.Source: §2 · p. 3
  • Context limits led many web agents to use zero-shot prompts or fine-tuning; WILBUR instead summarizes positive and negative demonstrations and separates website- from goal-conditioned examples.Earlier page summarization omitted prior demonstrations, while another approach used only a few positive experiences for a subtask.Source: §2 · p. 3
  • In broader in-context learning, prior methods used fixed examples or embedding similarity, whereas WILBUR trains a model to predict each example’s suitability.Earlier instruction synthesis was evaluated on classification and did not use negative examples.Source: §2 · p. 3

3 The WILBUR Agent

WILBUR treats web interaction as partially observed and combines retrieved demonstrations, synthesized learnings, reflection-driven backtracking, and an auto-curriculum that supplies training trajectories.

3.1 Problem Statement

  • The agent is framed as a partially observable, multi-step process: goals drive actions, while observations expose visible DOM and extracted text rather than backend state.Source: §3.1 · p. 4

3.2 WILBUR During Inference Time

  • WILBUR retrieves full trajectories for planning and isolated action examples for interaction, including negative demonstrations that teach pitfalls.Source: §3.2 · pp. 4–5
  • Because DOM similarity alone can mislead across websites, an MLP reranks retrieved action demonstrations by predicted execution success before successful examples enter the actor prompt.Source: §3.2 · pp. 5–6
  • A synthesizer distills multiple demonstrations into actionable insights, helping the actor use their patterns, strategies, and pitfalls despite prompt-window limits.Source: §3.2 · p. 6
  • A rule-based state comparison and reflection LM judge whether actions match the plan; failed actions can trigger feedback-guided backtracking, limited to a recent navigable URL.Source: §3.2 · p. 6

3.3 Learning websites with WILBUR

  • The first auto-curriculum run collects trajectories scored by an execution-evaluation LM; a demonstration-conditioned follow-up supplies action-success labels to train the knowledge model.Source: §3.3 · p. 7
  • The auto-curriculum samples achievable goals from website DOMs and conditions follow-up goals on earlier ones to make them more challenging.Source: §3.3 · p. 7

4 Evaluation

WILBUR is evaluated on WebVoyager, a benchmark of real websites, and outperforms the text-only state of the art. The evaluation also compares its components and identifies operational failure modes, including limitations with widgets, page actions, and anti-scraping.

4.1 Experimental Setup

  • WebVoyager evaluates task success on 643 goals across 15 real websites, with end-of-task scoring by GPT4V using a screenshot.WILBUR itself uses no screenshots to predict actions.Source: §4.1 · p. 7

4.2 Results

  • WILBUR outperforms the text-only state of the art by 8% and is within 5% of the multimodal model, despite using text-only inputs.It improves Booking.com results from around 2–4% to 39%.Source: Table 1 · p. 8, §4.2 · pp. 8–9
  • Adding task demonstrations raises the ablation result to 50%, and the fine-tuned retrieval model adds a further 3% overall.Backtracking alone brings the zero-shot baseline close to the state of the art.Source: §4.2 · p. 9

4.3 Error analysis

  • The error analysis attributes many practical web-agent failures to engineering challenges rather than the model.The identified issues include difficulty with complex widgets, performing actions, and anti-scraping techniques.Source: §4.3 · p. 9
  • Date selectors on Google Flights and Booking.com are a major error source, with the agent often getting stuck and eventually failing.The authors suggest built-in functions for common complex widgets as future work.Source: §4.3 · p. 9
  • Even when it predicts the correct action, WILBUR can fail to perform it; its main-DOM-only design also excludes frames and shadow DOM.The authors report effects on success rates for Apple and GitHub.Source: §4.3 · p. 9
  • Anti-scraping can rate-limit WILBUR on ArXiv, present an empty page, and prevent progress beyond the first execution step.The passage notes proxies and slower execution as known workarounds.Source: §4.3 · p. 9

5 Conclusion and Future Work

WILBUR combines experience-based backtracking with demonstration selection and instruction synthesis, and reaches 53% on WebVoyager. The conclusion also identifies engineering failures and costly inference as remaining concerns.

5 Conclusion and Future Work

  • Backtracking makes mistakes non-fatal, while selected positive demonstrations and synthesized instructions incorporate both successful and unsuccessful experience.The approach is described as leveraging its own experience and improving automatically over time.Source: §5 · p. 9
  • 53% on WebVoyager gives WILBUR a text-only state-of-the-art result, within 5% of the multimodal state of the art.It also surpasses the multimodal model on Apple, ArXiv, Cambridge Dictionary, BBC News, and Wolfram Alpha.Source: §5 · p. 9
  • The error analysis attributes a large class of failures to agent engineering rather than model failure, with web complexity implicated in these practical issues.The authors expect these issues eventually to be overcome.Source: §5 · pp. 9–10
  • WILBUR still requires expensive model predictions at inference time, although previously generated CSS-selector skills could in principle be reused.The authors identify direct reuse of previously generated simple and complex skills as future work.Source: §5 · p. 10

A DOM Formatting

The DOM representation retains interactive and structural content while omitting formatting elements.

A DOM Formatting

  • The formatted DOM includes leaves of interactive elements and structural elements such as headings, links, and paragraphs, but excludes formatting elements.Each element is represented with its index, tag, accessibility properties, content, and links.Source: A DOM Formatting · p. 13
  • Each included DOM element carries its index, tag name, accessibility properties, content, and links.Accessibility properties include role, alt text, and ARIA label.Source: A DOM Formatting · p. 13
  • The paper provides an example snippet of the formatted DOM in Figure 3.The example illustrates the representation used for page content.Source: A DOM Formatting · p. 13

B DSL Definition

WILBUR acts through a restricted Python-like DSL that refers to page elements by numeric index and provides built-in operations for interacting with and extracting web content.

B DSL Definition

  • The actor predicts Python-like DSL code limited to assignments and function calls, without control constructs.Multiple function calls may be emitted in one step and execute sequentially.Source: B DSL Definition · p. 13
  • Numeric element references are converted to CSS selectors before execution.This is how the DSL's predicted element indices are resolved for page actions.Source: B DSL Definition · p. 13
  • Built-ins support extracting text and links, iterating over list elements, entering text, pressing Enter, clicking, and navigating back.List mode scopes repeated DSL calls to the list element.Source: B DSL Definition · p. 13

C Training Algorithm

The training algorithm builds demonstration banks by generating website goals, running WILBUR, evaluating executions, and then training the knowledge model.

C Training Algorithm

  • For each website, the algorithm generates plausible goals and runs WILBUR on each goal from the initial observation state.It records each execution and its automatic evaluation for updating demonstration banks.Source: C Training Algorithm · p. 13
  • The algorithm generates follow-up goals conditioned on earlier goals, then runs and evaluates WILBUR on those goals as well.The resulting trajectories also feed demonstration updates.Source: C Training Algorithm · p. 13
  • After processing the goals, the algorithm trains the knowledge model using the accumulated goal and action demonstrations.The algorithm labels these demonstration vectors as knowledge and action demonstrations.Source: C Training Algorithm · p. 13

D Computation Cost of WILBUR

Table 2 reports actor LLM calls on successful WebVoyager runs; the text describes how in-context examples, instruction synthesis, and demonstration ranking affect steps to success.

D Computation Cost of WILBUR

Loading 2404.05902v1…