PaperlayerView exact revision on arXiv

Source-linked AI summary

WebXSkill: Skill Learning for Autonomous Web Agents

Zhaoyang Wang, Qianhui Wu, Xuchao Zhang, Chaoyun Zhang, Wenlin Yao, Fazle Elahi Faisal, Baolin Peng, Si Qin, Suman Nath, Qingwei Lin, Chetan Bansal, Dongmei Zhang, Saravan Rajmohan, Jianfeng Gao, Huaxiu Yao

arXiv:2604.13318v2cs.AIcs.CL

TL;DR

Autonomous web agents remain brittle on long-horizon workflows because textual skills cannot execute and code-based skills lack step-level guidance. WebXSkill bridges this grounding gap with extracted, graph-organized executable skills and grounded or guided deployment, improving benchmark task success while favoring different modes by model capability.

  • Problem

    Web agents must re-plan recurring workflows, while textual skills lack direct execution and code-based skills lack step-level guidance for adaptation.

  • Method

    WebXSkill extracts parameterized skills from synthetic trajectories, organizes them in a URL-based graph, and deploys them through grounded or guided execution.

  • Results

    WebXSkill improves task success over strong baselines on WebArena, WebVoyager, and Online-Mind2Web; grounded mode reaches 86.1% on WebVoyager.

  • Takeaways & Limitations

    Deployment mode should be selected according to model capabilities: stronger models benefit from grounded execution, while weaker models benefit from guided use.

  • Takeaways & Limitations

    Skill-library coverage inherits blind spots from synthetic trajectories, leaving poorly covered sites with sparser libraries.

Abstract

from arXiv · show

Autonomous web agents powered by large language models (LLMs) remain brittle on long-horizon browser workflows. A key bottleneck is a grounding gap in existing skill formulations: textual workflow skills provide natural language guidance but cannot be directly executed, while code-based skills execute without giving the agent step-level guidance for adaptation or recovery. We introduce WebXSkill, a framework that bridges this gap with executable skills, each pairing a parameterized action program with step-level natural-language guidance. WebXSkill operates in three stages: skill extraction mines reusable action subsequences from readily available synthetic agent trajectories and abstracts them into parameterized skills, skill organization indexes them into a URL-based graph for context-aware retrieval, and skill deployment exposes two complementary modes, grounded mode for fully automated execution and guided mode where skills serve as step-by-step instructions the agent follows with its native planning. WebXSkill demonstrates consistent improvements on WebArena, WebVoyager, and Online-Mind2Web. We further find that better skill deployment mode depends on a model's plan and execution capability. The code is available at https://github.com/aiming-lab/WebXSkill.

1 Introduction

Autonomous web agents struggle with recurring long-horizon workflows because procedural knowledge is discarded and existing skills separate guidance from execution. WEBXSKILL bridges this gap with readable executable skills, organized retrieval, and complementary deployment modes, improving task success across benchmarks.

  • Web agents must often re-derive recurring multi-page action sequences because procedural guidance from completed trajectories is discarded.
  • Textual skills guide planning but cannot execute, whereas code-based skills execute without exposing step-level guidance for adaptation.
  • WEBXSKILL pairs parameterized browser-action programs with step-level natural-language guidance that remains readable to the agent.
  • The framework extracts skills from synthetic trajectories, organizes them in a skill graph, and deploys them in grounded or guided mode.
  • WEBXSKILL improves task success over strong baselines on WebArena, WebVoyager, and Online-Mind2Web, with deployment mode selected according to model capabilities.

2 Related Work

Research on web-agent skills increasingly provides reusable abstractions for fragile long-horizon interaction, but methods differ in execution, guidance, acquisition, and retrieval.

  • Web agents operate over low-level browser actions, making long-horizon tasks fragile and motivating reusable tool and skill abstractions.
  • Skill-based methods provide reusable higher-level knowledge for web agents alongside planning, collaboration, memory, and reinforcement-learning approaches.
  • Existing methods differ along executability, step-level guidance, acquisition strategy, and context-aware skill retrieval.
  • Some executable-skill methods expose directly invocable tools but hide internal logic or acquire skills from test-time trajectories, creating transparency or leakage concerns.

3 Method

WEBXSKILL builds a reusable skill library from agent trajectories, retrieves skills using page context, and separates readable skill guidance from grounded or guided execution.

  • Overview: WEBXSKILL augments web agents with reusable, readable action programs organized through skill extraction, organization, and deployment.
  • Skill Extraction: Skill extraction uses an LLM to identify reusable action subsequences and abstract concrete values into parameterized skills from public trajectories without evaluation-derived data.
  • Skill Extraction: Curation deduplicates candidates, while test-environment execution verifies action sequences and discards skills that fail to run.
  • Skill Organization: A URL-based skill graph groups skills by generalized page patterns, while element-presence and embedding checks retrieve executable or cross-site candidates.
  • Skill Deployment: Grounded mode auto-executes matched action sequences, whereas guided mode exposes step-level guidance for agents to adapt their own actions when states differ.
  • Skill Deployment: Grounded mode favors stronger models, while guided mode offers greater robustness for weaker models that benefit from explicit procedural guidance.

4 Experiments

WEBXSKILL is evaluated across WebArena, WebVoyager, and Online-Mind2Web using strong multimodal models and multiple deployment configurations. Results show gains from its skill organization and deployment design, including cross-site transfer and model-dependent preferences between grounded and guided modes.

  • Experimental Setup: WEBXSKILL is evaluated on WebArena, WebVoyager, and Online-Mind2Web with GPT-5 and Qwen-3.5-122B, alongside broader open-source model comparisons.The evaluation uses benchmark-specific baselines and additional re-deployments of SkillWeaver and WALT within the WEBXSKILL framework.
  • Main Results: Both WEBXSKILL deployment modes outperform all baselines on WebArena overall, while GPT-5 slightly favors grounded mode and Qwen3.5-122B favors guided mode.Re-deploying SkillWeaver and WALT within the WEBXSKILL framework also recovers and can surpass MAP performance.
  • Main Results: Grounded mode reaches 86.1% on WebVoyager, improving over Vanilla by 14.2 points and MAP by 11.7; guided mode improves over MAP by 4.8 points on Online-Mind2Web.Grounded mode is the best method under all three judges on Online-Mind2Web.
  • Skill Transfer across Websites: WebArena skills transferred in guided mode reach 85.1% on WebVoyager without target-site programs, gaining 10.7 points over MAP and yielding an 8.1% relative improvement on Online-Mind2Web.The reported transfer relies on step-level guidance rather than fixed action scripts tied to specific websites.
  • Deployment Mode Preference: Deployment preferences vary by model: GPT-5 and Qwen3.6-35B benefit more from grounded execution, whereas Qwen3.5-122B and Qwen3.5-9B benefit more from guided use.The same learned procedure can therefore be exposed either as an executable tool or as step-by-step guidance.
  • Efficiency & Skill Usage: Grounded mode achieves a 70.8% Usage Rate, 16.5% Invocation Rate, and 9.3 average steps, while guided mode uses slightly more steps in exchange for adaptability.Guided-mode Usage Rate is nearly identical on successful and failed tasks, at 68.9% versus 68.8%.

5 Analysis

The analysis examines skill-library scaling, retrieval, component contributions, and failure patterns. Results show that broad synthetic experience and context-aware organization improve coverage and success, while many residual failures arise from agent decisions rather than skill execution.

  • Skill coverage: WEBXSKILL builds 100+ skills per site across all ten categories, with no category exceeding 24%.SkillWeaver concentrates over 60% of skills in retrieval, whereas WALT has 41 skills in total.
  • Skill scaling: WEBXSKILL scales from 105 to 591 verified skills as the extraction corpus expands, while task success rises from 62.8% to 69.5%.UR also increases from 26.6% to 70.8%, with the final 25% of sources adding 117 skills and 18.9 UR points.
  • Retrieval: 69.5% task success from the URL-based skill graph exceeds DOM matching at 65.6% and flat-library search at 59.1%.The graph first narrows candidates by page context and then filters them against visible elements.
  • Generality: Removing 196 near-matching synthetic sources retains 548 verified skills and achieves 68.8% task success, close to the full corpus’s 69.5%.This result supports usefulness across varied task instructions sharing reusable browser operations.
  • Ablations: The full lifecycle matters: verification contributes most, while contextual organization and step-level guidance provide substantial additional gains.Re-deploying SkillWeaver and WALT skills through WEBXSKILL raises success by 22.3 and 19.4 points, respectively.
  • Failure analysis: Failure rates vary from 96% execution success on Shopping to 52% on CMS, while post-skill reasoning accounts for 38% of failures.Wrong answer extraction is dominant among failed tasks, and 80% of CMS overlay-dismissal cases still complete despite a trailing error.

6 Conclusion

WEBXSKILL bridges the grounding gap by exposing readable action programs for both automated execution and adaptive guided use. Across benchmark tasks, it consistently outperforms strong baselines, with deployment mode depending on model capability.

  • Conclusion: WEBXSKILL pairs executable action programs with step-level guidance, enabling grounded execution and guided use.Synthetic-trajectory extraction and URL-based organization provide a rich, retrievable skill library.
  • Conclusion: WEBXSKILL consistently outperforms strong baselines across benchmark tasks, while stronger models favor grounded execution and weaker models favor guided use.The framework exposes reusable procedural knowledge through readable and executable skills.

Limitations

The framework’s skill coverage depends on synthetic trajectory coverage, and its URL-based graph is primarily designed for in-site retrieval. Guided mode partially relaxes the resulting cross-site transfer constraint.

  • Synthetic coverage: Synthetic skill extraction inherits upstream synthesis blind spots, so poorly covered sites yield sparser skill libraries.The approach is scalable and avoids test-data leakage, but its coverage remains bounded by the synthesis corpus.
  • Transferability: The URL-based skill graph is primarily designed for in-site retrieval, while code-based skills remain difficult to transfer across sites because they are coupled to site-specific DOM structures.Guided mode partially relaxes this constraint by exposing skills as procedural guidance rather than fixed scripts.

Use of AI Assistants

The authors used coding agents to assist with implementation and debugging, and AI assistants to polish the paper’s writing.

  • Use of AI Assistants: Coding agents assisted with code implementation and debugging, while AI assistants helped polish the paper’s writing.The authors state that they reviewed and edited the content as needed.

A Implementation Details

WebXSkill extracts reusable, parameterized skills from synthetic trajectories, organizes them for URL-aware retrieval, and deploys them either as executable tools or step-by-step guidance.

  • Skill extraction: 2,500 and 600 synthetic tasks supply WebArena and WebVoyager trajectories for extracting reusable action subsequences from successful and failed runs.The extraction process uses categorized exploration across target websites and benchmarks.
  • Skill extraction: Trajectory formatting records URLs, reasoning, action types, target-element descriptions, and parameters before LLM-based skill extraction.The structured representation follows a fixed JSON schema for every benchmark.
  • Skill organization: Online deduplication applies exact-name, same-site Jaccard, and library-wide embedding similarity checks before retaining or updating skills.The library-wide stage compares candidates with text-embedding-3-small and retrieves the top 20 similar existing skills.
  • Skill representation: Each skill combines a semantic signature with parameterized action steps containing natural-language guidance, element references, and step-specific parameters.Parameters use the {{param_name}} syntax to abstract values.
  • Skill organization: The skill graph indexes generalized URL patterns, ranks exact and wildcard matches by specificity, removes duplicate names, and surfaces up to 20 relevant skills per page.Element-based heuristics further filter skills whose target elements are absent from the current page.
  • Skill deployment: Grounded mode exposes skills as callable fg_* tools that execute low-level actions sequentially, while guided mode presents skills as instructions followed with native browser actions.Grounded execution can fall back to native actions after a failed skill call; guided mode injects step-level guidance at runtime.

A.6 Web Agent Evaluation

The evaluation tests WebXSkill across self-hosted and live-web settings, using trajectory inspection and multiple models or judges to address reproducibility and reliability concerns.

  • Evaluation settings: WebArena is commonly used because its high-quality verification and self-hosted environments support reproducible attribution of performance differences.The evaluation also extends beyond this single benchmark to test generalization.
  • Guided deployment: Guided-mode runtime injection lists available skills and provides activated step-by-step guidance for use with native browser actions.The system prompt frames skills as workflow guides for common webpage tasks.
  • Evaluation settings: WebVoyager adds more than 10 live websites, with unstable sites excluded after manual inspection for reCAPTCHA or task-accessibility problems.This filtering addresses reliability concerns in live-site evaluation.
  • Evaluation settings: Online-Mind2Web evaluation uses GPT-4.1, GPT-5.1, and WebJudge-7B on 300 tasks.The supplied passage identifies the models and task count used for this benchmark.
  • Case studies: WebArena case studies show GPT-5 using WEBXSKILL in both grounded and guided modes, with invoked or activated steps highlighted in the trajectories.The examples include task descriptions, selected steps, screenshots, and agent actions.

B.1 Grounded Mode

In a Reddit posting case, grounded mode compresses a multi-step submission workflow into a three-step skill-assisted interaction, despite an initial formatting error.

  • Grounded mode: 3 steps complete a Reddit forum posting task in grounded mode using a skill that automates the multi-step submission workflow.The task creates a Harry Potter virtual-meetup post in the books subreddit.
  • Grounded mode: 8–10 steps would typically be required without the skill to find the forum, open the submission form, fill multiple fields, and submit.The skill therefore compresses navigation, form completion, and submission into a shorter interaction.

B.2 Guided Mode

In a CMS shipping-address task, guided mode supplies a five-step workflow while allowing the agent to adapt when the page state changes or the guidance is incomplete.

  • Guided mode: 5-step guidance supports editing an order’s shipping address, including filling the street, city, and zip fields and saving the form.The agent activates edit_shipping_address_for_order after earlier navigation steps.
  • Guided mode: The agent independently adapts when the actual page state changes or the skill does not cover all required fields.Guided mode therefore preserves native planning alongside workflow instructions.
Loading 2604.13318v2…