PaperlayerView exact revision on arXiv

Source-linked AI summary

WALT: Web Agents that Learn Tools

Viraj Prabhu, Yutong Dai, Matthew Fernandez, Jing Gu, Krithika Ramakrishnan, Yanqi Luo, Silvio Savarese, Caiming Xiong, Junnan Li, Zeyuan Chen, Ran Xu

arXiv:2510.01524v1cs.CVcs.AIcs.LG

TL;DR

Web agents often rely on brittle step-by-step UI interactions, while humans use website functionality such as search and filtering. WALT reverse-engineers that functionality into reusable tools; it reports state-of-the-art success rates on VisualWebArena and WebArena, with fewer LLM interventions.

  • Problem

    Existing skill-discovery approaches mine successful trajectories or hypothesize automations, and their resulting UI action sequences can be brittle to dynamic elements and design changes.

    Source: §1 · p. 2

  • Method

    WALT reverse-engineers website-provided functionality into validated, reusable high-level tools rather than agent-induced skills or UI action sequences.

    Source: §1 · p. 2

  • Results

    WALT reports state-of-the-art success rates of 52.9% on VisualWebArena and 50.1% on WebArena.

    Source: §1 · p. 2

  • Takeaways & Limitations

    WALT's tool abstraction provides a practical path toward safe, auditable automation through explicit tool contracts, examples, and validation traces.

    Source: §5 · p. 9

    Tool discovery incurs per-website exploration and validation costs, and broader external validity beyond the two evaluated research benchmarks remains to be tested.

    Source: §5 · p. 9

Abstract

from arXiv · show

Web agents promise to automate complex browser tasks, but current methods remain brittle -- relying on step-by-step UI interactions and heavy LLM reasoning that break under dynamic layouts and long horizons. Humans, by contrast, exploit website-provided functionality through high-level operations like search, filter, and sort. We introduce WALT (Web Agents that Learn Tools), a framework that reverse-engineers latent website functionality into reusable invocable tools. Rather than hypothesizing ad-hoc skills, WALT exposes robust implementations of automations already designed into websites -- spanning discovery (search, filter, sort), communication (post, comment, upvote), and content management (create, edit, delete). Tools abstract away low-level execution: instead of reasoning about how to click and type, agents simply call search(query) or create(listing). This shifts the computational burden from fragile step-by-step reasoning to reliable tool invocation. On VisualWebArena and WebArena, WALT achieves higher success with fewer steps and less LLM-dependent reasoning, establishing a robust and generalizable paradigm for browser automation.

1 INTRODUCTION

WALT reframes browser automation around reusable tools that expose website-provided functionality rather than brittle UI action sequences. It reports state-of-the-art success rates on two benchmarks and gains in success and efficiency from its proposed components.

1 INTRODUCTION

  • Prior skill-discovery methods mine successful trajectories or ask agents to hypothesize automations, while both can yield brittle UI action sequences.Dynamic elements and design changes make these sequences sensitive to interface variation.Source: §1 · p. 2
  • WALT reverse-engineers website-provided functionality into validated, deterministic high-level tools instead of agent-induced skills or workflows.Its demonstrate-generate-validate process maps interaction traces to structured tools, prioritizing deterministic actions while permitting agentic steps for dynamic elements.Source: §1 · p. 2
  • WALT shifts runtime effort from reasoning through interface mechanics to calling tools such as search, filter, and sort, with discovery and optimization performed offline.The motivating example contrasts agents' low-level search, filter, and sort steps with a human's functionality-focused description of the task.Source: §1 · pp. 1–2
  • WALT reports state-of-the-art success rates of 52.9% on VisualWebArena and 50.1% on WebArena, with ablations showing 10%-30% success gains and 1.3-1.4x fewer steps.The discovered tools span search and filtering, content creation and management, and communication or social interactions.Source: §1 · p. 2

2 RELATED WORK

Related work advances web agents through perception, planning, reasoning, and action execution, while API use and skill discovery offer other routes to higher-level actions. WALT instead learns website-specific tools through systematic exploration.

Web Agents

  • Web-agent research spans perception, planning, reasoning, and action execution; WALT targets planning and execution with reusable tools mined offline.The paper characterizes screenshot-only pixel-space actions as potentially more brittle to layout change.Source: §2 · pp. 2–3

Benchmarks for Web Agents

  • WebArena and VisualWebArena provide realistic simulated tasks, with VisualWebArena adding visually grounded tasks that require multimodal understanding.The paper focuses its evaluation on these two benchmarks.Source: §2 · p. 3

API-using Web Agents

  • Unlike API-using approaches, WALT does not assume API documentation and instead reverse-engineers website functionality into tools with validated schemas and URL-level operations.The paper notes that API documentation may be undocumented or proprietary.Source: §2 · p. 3

Skill Discovery for Web Agents

  • Prior skill-discovery methods typically mine successful executions and compose primitive actions, whereas WALT systematically explores site functionality and stress-tests tools for reliability and modularity.WALT also uses schema validation, selector stabilization, URL reverse-engineering, and targeted agentic fallbacks.Source: §2 · p. 3

3 APPROACH

WALT expands browser agents' action space with callable tools built from website functionality. Its pipeline discovers candidates, constructs and validates tools, and iteratively optimizes them for correctness, efficiency, and determinism.

3 APPROACH

  • WALT treats websites as sources of structured functionality and backs each callable tool with an action script that favors deterministic URL or DOM operations while allowing targeted agentic steps.The pipeline comprises strategic discovery of tool candidates followed by construction and validation.Source: §3 · p. 3

3.1 PROBLEM FORMULATION

  • The formulation adds discovered high-level actions Atools to primitive browser actions Aprim so agents can execute tasks more efficiently and reliably.A tool maps structured input parameters to a target outcome and is exposed as an atomic action after validation.Source: §3.1 · p. 3
  • Discovery explores user-facing site sections and proposes reusable tool candidates; construction uses browser traces and a separate agent to create and validate executable tools.The overview describes stabilized interaction traces, action scripts, and end-to-end testing before registration.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 ALGORITHM DESIGN

  • Tool construction seeks to minimize failure rate, primitive step count, and the fraction of steps requiring LLM-dependent reasoning, iterating with feedback until validation succeeds or attempts run out.These objectives measure correctness, efficiency, and determinism, respectively.Source: §3.2 · p. 4
  • Construction records stable DOM selectors with vetted alternatives, then maps traces into scripts for interaction, navigation, extraction, and dynamic agentic steps.The pipeline also attempts to replace eligible multi-step UI sequences with URL operations and induces validated input schemas.Source: §3.2 · pp. 4–5
  • Registered tools are tested end to end, and structured failure feedback refines selectors, schemas, or scripts; runtime agentic fallback handles unanticipated failures.The system also adds a multimodal DOM parser and an external verification tool.Source: §3.2 · p. 5
  • WALT pairs grounded browser interaction with schema-checked, URL-optimized executors to deliver tools across discovery, content, and communication with fewer LLM calls.The paper describes the resulting tools as running faster as well.Source: §3.2 · p. 5

4 EXPERIMENTS

WALT is evaluated on VisualWebArena and WebArena against prior agents, with ablations and analyses examining its performance, efficiency, and tool use. It achieves leading success rates on both benchmarks while improving efficiency over a no-tool control.

4.1 OVERVIEW

  • WALT improves prior state-of-the-art web-agent performance on VisualWebArena and WebArena while reducing action steps, with ablations and analyses examining its components and successes.The evaluation compares WALT with prior methods on both benchmarks.Source: §4.1 · p. 5

4.2 BENCHMARKS

  • VisualWebArena contains 910 visually grounded tasks across three websites, while WebArena contains 812 tasks spanning five websites and binary evaluator-based success.Evaluators include string, URL, HTML, and image-content checks.Source: §4.2 · p. 6

4.3 IMPLEMENTATION DETAILS

  • The base agent combines a GPT-5 planner and GPT-5-mini browser executor, using screenshot and indexed-element observations, retrieval, verification, and a multimodal DOM parser.Agents run for at most 30 steps and replan every 15.Source: §4.3 · p. 6

4.4 BASELINES

  • Comparisons cover skill-based, test-time search and planning, API-using, computer-use, and verification agents, alongside benchmark baselines and human performance.Skill-based comparators include SkillWeaver, AWM, and ASI on WebArena.Source: §4.4 · p. 6

4.5 MAIN RESULTS

  • WALT reports the best VisualWebArena average (52.9%) and leads WebArena on 5 of 6 splits, tying on the sixth and beating ASI by 9 points.The WebArena comparison spans success rates across domains.Source: §4.5 · pp. 6–7, Table 1 · p. 7
  • On VisualWebArena, tools improve success by up to 30.7% relative and efficiency by 1.4x versus an identical-architecture no-tool control.The lower control success rates indicate gains are not explained by the stronger underlying GPT-5 model alone.Source: §4.5 · p. 7, Figure 3 · p. 7
  • On Classifieds, autonomous WALT reaches 64.1% success versus 66.0% with manually demonstrated tools, using 5% fewer steps.Multimodal DOM parsing adds 2.6% and external verification adds 3.3%, but verification costs extra checks; all components use 21.3% fewer steps than baseline policies.Source: §4.5 · p. 7, Table 2 · p. 8
  • On Classifieds, only 3 of 9 discovered tools include agentic steps; successful rollouts solve varied tasks in 2–5 calls, while failures expose compound visual constraints and unavailable rating actions.Search listings is invoked 262 times, and tools achieve nearly perfect invocation success rates.Source: §4.5 · pp. 7–8, Figure 4 · p. 8, Figure 5 · p. 9

5 DISCUSSION

WALT reframes browser automation around validated tools reverse-engineered from website functionality, rather than brittle, agent-imagined UI sequences. The discussion identifies operational and evaluation limits while outlining directions for extending and maintaining the approach.

5 DISCUSSION

  • WALT exposes website functionality as tools with validated schemas and failsafes, and reports state-of-the-art benchmark performance with fewer LLM interventions.Tools may combine UI interaction, extraction, agent, and navigation steps.Source: §5 · p. 9
  • Offline discovery and validation cost time per website, and tool quality depends on exploration coverage and the functionality each site exposes.Highly dynamic interfaces, A/B experiments, CAPTCHAs, and anti-automation can reduce determinism or block URL promotion; schemas and selectors may also drift, and complex editors or uploads may need agentic steps.Source: §5 · p. 9
  • Future directions include online patching for selector and schema drift, extracting canonical web patterns, and integrating official APIs or external MCP servers.The authors also describe explicit contracts, examples, and validation traces as supporting safer, auditable automation that is easier to monitor, share, and maintain.Source: §5 · p. 9
  • The experiments use public benchmark testbeds and no human subjects, though browser automation could be misused for scraping or spam.Source: §5 · p. 10
  • The paper reports pipeline, benchmark, and implementation details, with appendix pseudocode and ablations; code is planned for public release.Source: §5 · p. 10

A.1 ANALYSIS

The appendix analyzes tool-construction attempts, classified-task performance, and qualitative trajectories. It reports that most tools are validated on the first attempt, while difficulty and complex reasoning remain sources of failure.

A.1 ANALYSIS

  • Most tools are discovered on the first attempt, while several nuanced functionalities require up to four attempts to obtain a validated implementation.Source: §A.1 · p. 12, Figure 6 · p. 13
  • An external verifier corroborates agent assessments in the classifieds analysis, addressing agreement bias in which agents may rationalize failed trajectories as successful.Source: §A.1 · p. 12
  • WALT’s failure rate stays below 50% even on the most difficult visual, reasoning, and overall task categories, though failures increase with difficulty.Source: §A.1 · p. 13
  • Qualitative rollouts show challenging visual grounding and cross-website reasoning, but continued difficulty with tasks requiring very complex multi-step reasoning.Source: §A.1 · p. 13, Figure 7 · p. 13

A.2 IMPLEMENTATION DETAILS

The implementation appendix describes the baseline browser agent and notes that heuristic active-page detection can make reported accuracy a lower bound.

A.2 IMPLEMENTATION DETAILS

  • The baseline uses Claude Computer-Use in a dedicated desktop environment, with Chrome task pages, screenshot observations, and pyautogui OS-level actions.Source: §A.2 · p. 13
  • The implementation uses claude-4-sonnet-20250514 with thinking mode, resizes screenshots to 1280×720, and caps tasks at 30 steps.Source: §A.2 · p. 13
  • Heuristic active-page detection may identify the current page incorrectly in edge cases, so reported accuracies are lower bounds rather than exact measurements.Source: §A.2 · p. 14

A.3 USE OF LARGE LANGUAGE MODELS

The appendix documents the prompts and algorithmic workflow used to explore website functionality and turn recorded browser actions into reusable tools. The workflow prioritizes deterministic, parameterized steps, optimizes eligible sequences into URL operations, and validates tools through testing and refinement.

A.3 USE OF LARGE LANGUAGE MODELS

  • The authors used Claude-4-Sonnet and GPT-5 to polish the paper’s writing through proofreading, revision, and compression.Source: §A.3 · p. 14
  • When feasible, UI sequences for search, filtering, sorting, or pagination are replaced with URL operations; candidate tools are tested and refined after validation errors.The algorithm stabilizes selectors, infers input schemas and test inputs, registers tools, and retries failed candidates within an attempt budget.Source: §A.3 · pp. 15, 17
  • The discovery agent systematically explores logged-in site areas for user-facing functions, prioritizing search, content management, communication, and organization.It is instructed to document only functioning tools and explore diverse functionality.Source: §A.3 · pp. 15–16
  • The exploration findings inform a goal-oriented, reusable, composable API specification, with one clear user goal per function rather than UI mechanics.Source: §A.3 · pp. 15–16
  • The creation agent converts recorded browser steps into parameterized tools by extracting values, identifying inputs, favoring deterministic actions, and skipping unnecessary clicks when direct navigation works.Source: §A.3 · p. 16
  • Tool construction classifies steps as navigation, extraction, deterministic interaction, or agentic interaction, reserving agentic reasoning for essential dynamic content.The design principles call for sequential, single-purpose, parameter-driven tools containing only essential steps.Source: §A.3 · p. 17
Loading 2510.01524v1…