PaperlayerView exact revision on arXiv

Source-linked AI summary

NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild

Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, Christopher D. Manning

arXiv:2410.02907v2cs.CL

TL;DR

Human demonstrations are costly to collect, and interaction-based search over website tasks is large. NNetNav samples interactions, retroactively labels them, and prunes trajectories without meaningful sub-task labels; its fine-tuned model outperforms zero-shot GPT-4 on WebArena and WebVoyager.

  • Problem

    Human demonstrations for browser agents are costly to collect, while searching the large space of meaningful website interactions is challenging.

    Source: §1 · p. 2

  • Method

    NNetNav explores websites, retrospectively labels trajectories with instructions, and prunes episodes whose prefixes cannot receive meaningful sub-task labels.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    16.3% on WebArena and 35.2% on WebVoyager: fine-tuned Llama-3.1-8B outperforms zero-shot GPT-4 by 2 and 1.7 points, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    NNetNav opens avenues for open-ended workflow discovery on unknown web interfaces without human supervision.

    Source: §1 · p. 2

    Live-site exploration risks website overload, unintended interactions, and bias propagation; the authors prohibit login and content submission on live websites.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

We introduce NNetNav, a method for unsupervised interaction with websites that generates synthetic demonstrations for training browser agents. Given any website, NNetNav produces these demonstrations by retroactively labeling action sequences from an exploration policy. Most work on training browser agents has relied on expensive human supervision, and the limited prior work on such interaction-based techniques has failed to provide effective search through the exponentially large space of exploration. In contrast, NNetNav exploits the hierarchical structure of language instructions to make this search more tractable: Complex instructions are typically decomposable into simpler sub-tasks, allowing NNetNav to automatically prune interaction episodes when an intermediate trajectory cannot be annotated with a meaningful sub-task. \texttt{LLama-3.1-8b} finetuned on 10k NNetNav self-generated demonstrations obtains over 16\% success rate on WebArena, and 35\% on WebVoyager, an improvement of 15pts and 31pts respectively over zero-shot \texttt{LLama-3.1-8b}, outperforming zero-shot GPT-4 and reaching the state-of-the-art among unsupervised methods, for both benchmarks.

1. Introduction

Zero-shot language models struggle with grounded browser tasks because website interactions vary, while covering every use case with human demonstrations is costly. NNetNav generates demonstrations through website interaction and uses meaningful language sub-tasks to prune exploration.

1. Introduction

  • NNetNav addresses costly human demonstration collection by generating synthetic training demonstrations from website interaction.Its exploration policy produces trajectories that a language-model labeler retrospectively annotates with instructions.Source: §1 · p. 2
  • NNetNav prunes exploration when trajectory prefixes cannot be labeled with meaningful sub-tasks, using instruction hierarchy to constrain the search.The paper says this structure also produces complex, hierarchical instructions.Source: §1 · p. 2
  • 16.3% on WebArena and 35.2% on WebVoyager: the fine-tuned Llama-3.1-8B outperforms zero-shot GPT-4 by 2 and 1.7 points, respectively.The paper reports state-of-the-art performance among unsupervised methods on WebArena and says its WebVoyager result beats all known open methods to the authors’ knowledge.Source: §1 · p. 2
  • The dataset contains over 10k demonstrations, about 100k state-action transitions, collected from 20 websites, including 15 live websites.The collection also includes five self-hosted WebArena websites.Source: §1 · p. 2

2. Background

Browser instruction following is modeled as sequential action selection from observations and trajectory history. Earlier synthetic-demonstration methods generate instructions before trajectories, but those instructions may be infeasible or limited in complexity and visibility.

2. Background

  • A browser episode is a sequence of observations and actions, and the agent selects actions from the current observation, past trajectory, and instruction.The paper represents observations as flattened DOM or accessibility trees, with keyboard and mouse commands as actions.Source: §2 · p. 2
  • Prior synthetic-demonstration methods sample instructions first, use a zero-shot agent to produce trajectories, and filter demonstrations with reward models.The filtering may use a ground-truth reward function or a language-model outcome reward function.Source: §2 · pp. 2–3
  • Instruction-first methods can produce plausible but infeasible tasks, miss entities embedded deep in a site, and offer little control over instruction complexity.The paper contrasts plausible instructions with feasible ones, noting that a task may fit a site's genre without being executable there.Source: §2 · p. 3

3. Our Approach

NNetNav samples website interactions first and retrospectively labels trajectories as instructions. Its language-model components summarize state changes, label trajectories, assess instruction fit, prune unpromising episodes, and post-process actions for the inferred instructions.

3. Our Approach

  • NNetNav samples website interactions before assigning instructions, making each retrospectively labeled instruction feasible for the sampled interaction.The exploration policy generates long trajectories, which are later labeled and post-processed.Source: §3 · p. 3

3.1. LM Components

  • The exploration policy conditions actions on a website persona, available actions, the current observation, and action history.Personas are used to simulate diverse user behaviors and increase trajectory diversity.Source: §3.1 · p. 3
  • A language model summarizes each action’s state change, which supplies the trajectory labeler with descriptions for generating an instruction.The summary describes the change from the current observation to the next one.Source: §3.1 · p. 4
  • An outcome reward model scores whether the observed state changes correspond to the generated instruction.The model returns a binary reward for the instruction and state-change sequence.Source: §3.1 · p. 4

3.2. Sampling Demonstrations via Interactions

  • At set time steps, NNetNav labels the partial trajectory as a sub-task and prunes the episode if the outcome reward model gives it no reward.A rewarded prefix is stored as a demonstration, and exploration continues; one episode can yield multiple demonstrations.Source: §3.2 · p. 4
  • Because exploratory actions may not suit the retroactively generated instruction, NNetNav generates a corresponding action for each instruction, observation, and prior-action context.The agent policy supplies the post-hoc action for each state in the demonstration set.Source: §3.2 · p. 4
  • Optional best-of-K sampling scores the original and additional trajectories for each instruction, then pairs the instruction with the highest-scoring trajectory.The outcome reward model scores candidates, with ties broken arbitrarily.Source: §3.2 · p. 4

4. Main Experiments

NNetNav’s demonstrations cover diverse websites and task complexities, and fine-tuned models achieve strong results on both benchmarks. Transfer and failure analyses reveal limits involving website domain, navigation efficiency, and structured information.

4.1. Collecting Demonstrations in the Wild

  • Over 10,000 demonstrations from 20 websites show diverse intent distributions (PPL 13.5 for self-hosted sites; 16.2 for live sites) and substantial numbers of complex demonstrations.The dataset includes 15 live websites and five self-hosted WebArena websites.Source: §4.1 · p. 4

4.2. Finetuning: Details and Results

  • On WebArena, Llama8B-NNetNav-WA and Llama8B-NNetNav-All outperform zero-shot GPT-4o; on WebVoyager, Live and All surpass zero-shot GPT-4o and prior 34B OSS results.The paper reports state-of-the-art performance among unsupervised methods on WebArena and among open-source methods on WebVoyager.Source: Table 2 · p. 5, §4.2 · p. 5

4.3. Cross-Website Transfer

  • In-domain data improves 10/13 live and 3/5 WebArena sites; overall it helps 13/18, while adding WebArena data lowers live-site average performance by 1.3 points.Notable gains occur on ESPN, Apple, and GitHub.Source: §4.3 · pp. 5–6, Table 3 · p. 6

4.4. Error Analysis

  • Agents perform strongly in element interaction and search, but struggle with navigation efficiency, redundant actions, and retrieving structured web content.The error analysis also identifies difficulty discovering information through multiple queries and piecing together results.Source: §4.4 · p. 6, Figure 3 · p. 6

5. Controlled Experiments

Controlled experiments compare NNetNav-generated data with instruction-first data and examine pruning efficiency and same-model self-training.

5.1. Experimental Settings

  • Controlled small-scale experiments compare NNetNav with baselines on WebArena and eight complex MiniWoB++ tasks, using gpt-4o-mini-2024-07-18 and generating 50 WebArena episodes per website and 80 MiniWoB++ episodes per interface.Pruning is applied every 4 time-steps on both benchmarks.Source: §5 · p. 6, §5.1 · p. 7
  • The study compares zero-shot prompting, instruction-first fine-tuning, NNetNav fine-tuning, and an ablation that regenerates trajectories from NNetNav instructions.The ablation isolates the contribution of NNetNav trajectories by retaining its instructions but prompting the same large model to generate new trajectories.Source: §5.1 · p. 7

5.2. Results

  • NNetNav fine-tuning improves MiniWoB++ mean reward from 28% to 48% and WebArena task SR from 1% to 7.2%, beating instruction-first by 12 and 1.2 points, respectively.The controlled evaluation reports results averaged across domains.Source: Table 4 · p. 7, §5.2 · p. 7
  • Using 302 demonstrations generated by Llama-3.1-8b to fine-tune the same model improved WebArena performance from 1% to 5.3%.The data were collected over 200 episodes per website.Source: Table 5 · p. 7, §5.2 · pp. 7–8
  • More than 60% of WebArena episodes were pruned after 16 actions, while 65% of MiniWoB++ episodes were pruned after four actions.The MiniWoB++ early pruning identified episodes whose initial actions resulted in execution failures.Source: §5.2 · p. 7, Figure 4 · p. 8

6. Related Work

Prior browser-agent training commonly uses human demonstrations or synthetic instruction-first data, while earlier interaction-first exploration faced scalability challenges. NNetNav uses language to filter interactions and prune exploration.

Language Conditioned Digital Assistants

  • Earlier approaches to language-conditioned digital assistants often combined expert demonstrations with shaped reward functions [Misra et al., 2017].NNetNav instead learns from synthetic demonstrations derived through language-model exploration of websites.Source: §6 · p. 8

Linguistic Priors for Exploration

  • NNetNav uses natural language both to filter meaningful interactions and to prune the potentially exponential search over them.This contrasts with prior language-prior methods that guide exploration with learned state/action associations, goal generators, or intermediate-state goals.Source: §6 · p. 8

Training Data for LLM browser agents

7. Conclusion

NNetNav generates synthetic browser-agent demonstrations by exploring websites and relabeling trajectories into instructions, using language-based pruning to manage the interaction search space. The resulting 10k demonstrations support strong unsupervised benchmark results.

7. Conclusion

  • NNetNav first explores websites to generate trajectories, then retrospectively labels them as instructions and prunes exploration when trajectory prefixes lack meaningful sub-tasks.The paper reports that 10k demonstrations from 15 live and 5 self-hosted websites fine-tuned Llama-3.1-8b to state-of-the-art results among unsupervised methods on WebArena and WebVoyager, surpassing zero-shot GPT-4 by 1.7 to 2.2 points.Source: §7 · p. 8
  • The demonstrations cover 15 live websites and 5 self-hosted websites and are used to fine-tune Llama-3.1-8b.NNetNav opens the possibility of scaling training data for generalist web agents across a broad range of interfaces without human intervention.Source: §7 · p. 8

Impact Statement

Unsupervised exploration on live websites carries risks including website overload, unintended interactions, and bias propagation.

Impact Statement

  • Live-site exploration is constrained to 10 parallel instances, a 0.5-second delay between actions, and no login or content submission.The paper recommends monitoring agents and auditing data for bias before deployment.Source: Impact Statement · p. 9

A. Prompts for LM components

The appendix documents prompt templates for language-model components used in browser interaction across MiniWoB++ and WebArena/live websites, covering exploration, trajectory and completion assessment, agent guidance, and required stop actions. These prompts specify component roles, inputs and outputs, action constraints, and— for WebArena/live exploration—browser navigation and safety bounds.

A.1. MiniWoB++

  • The available interfaces include fill(bid, value), select_option(bid, options), click(bid, button, modifiers), and drag_and_drop(from_bid, to_bid); only a single action can be provided at once.Source: §A.1 · p. 12
  • The MiniWoB++ exploration prompt conditions actions on the current DOM and interaction history, and asks for at least 4 and at most 20 actions unless exploration stops early.It directs the agent to use scrolling for elements outside the visible DOM and to stop generating actions after issuing stop.Source: §A.1 · pp. 12–13
  • MiniWoB++ ΔLM describes browser-state changes by comparing the initial and final DOM given the action, returning a one- or two-sentence description in a fixed format.The required output format is “State change: <your-answer>”.Source: §A.1 · p. 13
  • MiniWoB++ trajectory labeling infers the user’s instruction from an interaction trajectory, while the reward prompt scores completion from 1 to 5 and reserves 5 for error-free performance.The reward rubric gives 4 for very minor errors or more than 70% completion, and 3 or below for little progress or major errors.Source: §A.1 · pp. 13–14
  • The MiniWoB++ base agent receives the DOM, user objective, and previous action, and must choose a valid single action, scrolling when needed and stopping when the objective is achieved.Its prompt also requires step-by-step reasoning and action output in a code block.Source: §A.1 · p. 14

A.2. Prompts for WebArena and Live Websites

  • WebArena and live-site prompts provide page operations, tab management, URL navigation, and stop actions, with the WebArena homepage offered as a directory of sites.Actions include click, type, hover, press, scroll, new_tab, tab_focus, close_tab, goto, go_back, and go_forward.Source: §A.2 · p. 14
  • The WebArena exploration prompt conditions one-at-a-time actions on the accessibility tree, URL, open tabs, previous action, trajectory, and person description, with a 4-to-40-action range and early stopping allowed.The prompt requires valid actions in a specified format and directs the agent not to generate anything after stop.Source: §A.2 · pp. 15–16
  • Live-site exploration adds safety and site-specific rules, including terminating at CAPTCHAs or login prompts and avoiding account creation, disabled links, and PDF viewing on arXiv.The rules also specify booking dates, suggest searching for locations on Google Maps, and recommend dismissing pop-ups with options such as “Maybe later.”Source: §A.2 · pp. 15–16
  • The instruction-inference prompt asks the model to guess the user’s instruction from a trajectory and explain why that trajectory achieves it.It supplies example tasks and requires output in the form “Instruction: <your-answer>.”Source: §A.2 · p. 17
  • 5 is reserved for error-free task completion, 4 for minor errors or over 70% completion, and 3 or below for very little progress.The reward prompt asks the scorer to compare the trajectory with the instruction and return a score on a 1-to-5 scale.Source: §A.2 · p. 17
  • The base-agent prompt supplies the objective, accessibility tree, current URL, open tabs, and prior actions, and requires one valid action at a time.It also instructs the agent to reason step by step, follow the required action format, and issue stop when it believes the objective is achieved.Source: §A.2 · p. 17
  • The base-agent prompt forbids goto actions to URLs other than the homepage and requires the action to follow the prescribed format.It asks the agent to end its reasoning with the specified next-action phrase and stop generating after issuing stop.Source: §A.2 · p. 17
  • Because WebArena and WebVoyager require a final [stop] action, NNetNav uses a prompt to append that token to its demonstrations.The prompt infers the stop action using the instruction and final webpage, placing the answer in the stop action when the task requests information and otherwise using no arguments.Source: §A.2 · pp. 17–18
  • The stop-action prompt reconstructs a missing final action from the user instruction and final webpage state, with reasoning and a formatted stop output.Examples include stopping with an answer for information-seeking tasks and without arguments for tasks that change website state.Source: §A.2 · p. 18

B. Processing Demonstrations for SFT

For supervised fine-tuning, NNetNav converts demonstrations into training instances according to the inputs used by the policy. MiniWoB++ uses the previous action, while WebArena and WebVoyager use the full action history.

B. Processing Demonstrations for SFT

C. Training Details

Llama-3.1-8B fine-tuning runs use a batch size of 128 × 20000, train for two epochs, and use a learning rate of 2e-5.

C. Training Details

  • Llama-3.1-8B fine-tuning uses a batch size of 128 × 20000, two epochs, and a learning rate of 2e-5 warmed up linearly over 3% of training steps.Training uses four 80GB H100 GPUs and DeepSpeed ZeRO-3.Source: C. Training Details · p. 18

D. Distribution of Intents in NNetNav demonstrations and Examples

NNetNav demonstrations span varied website intents, with examples described as hierarchical, concrete, and plausible. Live-website demonstrations are limited by the prevention of model logins, unlike WebArena demonstrations.

D. Distribution of Intents in NNetNav demonstrations and Examples

  • Live-website intents include recipes, restaurant reviews, and product pricing, while WebArena examples include tasks requiring login, such as forum posting and order management.The figure presents the top 10 intents per website for live sites and WebArena sites.Source: Figure 5 · p. 19
  • Example NNetNav-WebArena demonstrations are described as hierarchical, grounded in concrete features and entities, and plausible by design.Source: Table 6 · p. 20

Related papers

Loading 2410.02907v2…