AutoTailor: Automatic, User-Aligned Capability Selection and Adaptation for Web Agents

1 Introduction

• Web-agent capability spaces remain difficult to make both efficient and user-aligned: manual tool construction is costly, while automated discovery may yield insufficient coverage or redundant, misaligned behaviors.The section frames this as an open problem for web agents using reusable tools.Source: §1 · pp. 2–3
• AutoTailor builds a compact MCP capability set by converting trajectories into parameterized APIs and filtering them for quality and likely usefulness.At runtime, it logs task outcomes and API use, detects recurring coverage gaps, retrieves candidates, adds useful APIs, and prunes persistently unused ones.Source: §1 · p. 3
• The framework combines broad offline capability discovery with usage-driven online adaptation to keep the deployed tool set compact while tracking observed task needs.The paper presents this as a portfolio balancing functional coverage, deployment efficiency, and alignment with user needs.Source: §1 · p. 3
• 90.6% correctness versus 87.5% for ReAct alone, with 57.8% lower average total request-token cost and 29.4% lower latency, using a 33-API set.Offline filtering reduces 1,283 APIs to 87; without ReAct, the 33-API set achieves 60.1% correctness while reducing request-token usage by 94.9%.Source: §1 · p. 3

2.1 Web trajectory generation

• Trajectory-generation methods broaden or improve the reliability of web-interaction corpora; AutoTailor instead takes such a corpus as input and selects which behaviors become deployed tools.The cited methods vary in exploration, task synthesis, and trajectory quality control.Source: §2.1 · p. 4

2.2 Tool and skill synthesis

• Tool- and skill-synthesis systems turn interactions into reusable capabilities, but leave open how to maintain an effective inventory from a large candidate pool.AutoTailor filters and reselects tools to keep the inventory compact and useful as demand changes.Source: §2.2 · p. 4

2.3 Meta-agent optimization

• Meta-agent optimization adapts prompts, programs, or harnesses; AutoTailor instead changes deployed tool-set membership using observed requests, usage, fallback calls, and failures.The paper frames capability inventory optimization as selecting tools offline and adapting that set as demand changes.Source: §2.3 · pp. 4–5

3 System Architecture

• The architecture transforms a potentially redundant, non-user-aligned API pool into a compact, demand-responsive inventory through offline selection and online reselection.The active tool space is reduced while coverage can be restored as new needs emerge.Source: §3 · p. 5

3.1 Offline Selection Module

• Trajectory crawling collects diverse multi-step web interactions, with each trajectory represented as a task description, context, and ordered browser actions.The crawler explores available interactions on a target website; the illustrated trajectory includes navigation, search, item selection, and cart interaction.Source: §3.1 · p. 5, Figure 2 · p. 6
• A code-generation model turns each trajectory into a reusable asynchronous Playwright function by replacing scenario-specific values with semantic parameters.Function names, signatures, and docstrings specify the task, arguments, execution steps, and return value.Source: §3.1 · p. 6
• The Quality Filter removes overly specific, low-level, or fine-grained APIs and deduplicates functional equivalents to form the quality-filtered candidate pool.It uses semantic clustering followed by LLM verification of high-similarity pairs, retaining the more complete trajectory for verified duplicates.Source: §3.1 · pp. 5–6
• The Usage Likelihood Filter keeps APIs scored 4 or 5 and retains the best-scoring API from otherwise uncovered clusters when its score is at least 3.The selected programs are exposed as executable MCP tools with descriptions and input schemas derived from their function metadata.Source: §3.1 · pp. 6–7

3.1.1 Online Dynamic Reselection (DR)

• Online reselection logs task intents, tool calls, outcomes, and usage, then clusters normalized uncovered-task intents to detect recurring capability gaps.When triggered, the system retrieves up to five candidate APIs per unresolved gap cluster and adds suitable candidates to the active set.Source: §3.1.1 · pp. 7–8
• Pruning requires sustained non-use: previously invoked APIs are protected, new APIs receive a grace period, and the active set cannot fall below a minimum size.In the evaluation, unused APIs become eligible for pruning after 50 tasks, newly added APIs receive a 50-task grace period, and at least 10 APIs are retained.Source: §3.1.1 · p. 8

4.1 Resulting API sets

• The API set shrinks from 1,283 initial APIs to 215 after QF, 87 after ULF, and 33 after Dynamic Reselection.QF filters granularity and duplicates; ULF filters by usage likelihood; Dynamic Reselection adapts to observed task demand, usage, and coverage gaps.Source: §4.1 · p. 8

4.2 Dynamic Calibration Protocol

• Calibration randomized the 106 tasks into a simulated interaction sequence, then froze the resulting SDR set for evaluation; it assumes task interests and relative frequencies remain approximately stable.The evaluation does not report per-step calibration accuracy, trigger counts, task-family statistics, or intermediate API-set measurements.Source: §4.2 · p. 8

4.4 Results

• 90.6% correctness and 72.45 s all-task latency: the dynamically reselected set also uses 98.46k tooling request tokens and 1.32m total request-token cost with ReAct fallback.These are the best correctness and lowest all-task latency reported for the evaluated configurations.Source: Table 1 · p. 9, §4.4 · p. 10

4.3 Evaluation on WebArena-Postmill

• The benchmark evaluates four trajectory-derived API sets on all 106 Postmill tasks, with and without ReAct; ReAct-only serves as a reference condition.The ReAct-enabled setting measures end-to-end performance, while the no-ReAct setting isolates synthesized APIs' functional coverage and execution efficiency.Source: §4.3 · p. 9

4.4 Results

• 60.1% correctness: the frozen SDR set nearly matches Sinit (60.4%) and SQF (59.9%) without ReAct, using 142.86k tooling request tokens and 79.50 s average latency.The authors report that SDR recovers much of the direct API coverage lost through static filtering.Source: Table 2 · p. 10, §4.4 · p. 11

5.1 Web Agent Capability Management As a First-Class Component

• AutoTailor treats capability-set composition as a complement to planning and reasoning, and proposes shared profiles for users with similar task distributions alongside individual specialization.The authors characterize the framework as an early exploration of meta-agent capability management.Source: §5.1 · p. 11

5.2 Evaluation Realism Through Real World Usage

• The templated WebArena evaluation simulates repeated interactions but does not capture noisier, less regular, context-dependent real requests or changing user interests.The reported adaptation benefit is evidence for a stable simulated task distribution, not a complete characterization of production behavior.Source: §5.2 · p. 11

5.3 Verification for Compact Capability Sets

• Compact reusable APIs could be inspected, tested, or formally verified offline, shifting part of the reliability burden away from repeated online reasoning.The discussion presents this as a potential complement to dynamic capability management, not an evaluated result.Source: §5.3 · pp. 11–12

• Task granularity is how well a trajectory corresponds to a single, meaningful, self-contained user task.A well-formed function achieves a clear goal, avoids unnecessary intermediate steps, and cannot be decomposed further without losing usefulness.Source: §A.1 · pp. 13–14
• The evaluator considers goal completeness, step minimality, and focus, including whether unrelated subtasks are present.Over-scoped trajectories combine loosely related tasks or unnecessary actions; under-scoped ones may miss key steps or fail to reach a meaningful outcome.Source: §A.1 · pp. 14–15
• For a user-ask trajectory ending with generate_answer, the requested answer is treated as successfully retrieved, not incomplete solely because no answer-generation steps are shown.Source: §A.1 · p. 14
• The rubric rates web trajectories from 1–5 on task granularity and coherence; a score of 5 indicates severe over-scoping, such as combining distinct tasks or including actions beyond one clear goal.Such trajectories can be split into meaningful sub-functions. When granularity is too low, a function may be merely atomic, such as only navigating to a page, without accomplishing a meaningful goal.Source: §A.1 · pp. 13–14

• Two trajectories are functional duplicates when they accomplish the same user task in essentially the same way.Source: §A.2 · p. 15
• The comparison ignores superficial wording differences, reordered equivalent steps, and differences in specific input values.Source: §A.2 · p. 15
• The evaluator receives each trajectory’s description, actions, and code, then returns a duplicate judgment with brief reasoning.Source: §A.2 · pp. 15–16

• Usage likelihood is the estimated chance that a real user would want to perform the trajectory’s task on the specified website.Source: §A.3 · p. 16
• The scale distinguishes core popular goals, frequent natural tasks, occasional goals, niche goals, and rare or contrived intentions.Examples include searching for a topic as a core goal, saving a post as occasional, and changing a rarely used display preference as rare.Source: §A.3 · p. 16
• The rating is based on the website description and parameterized trajectory description, with one integer from 1–5 and brief reasoning.Source: §A.3 · pp. 16–17

• Candidate APIs are selected only when appropriate parameter values let them genuinely accomplish one or more unresolved task templates.Templates mask parameter values, while concrete examples ground their structural intent.Source: §A.4 · p. 18
• The selection is capped at the specified maximum, ordered by preference, and uses API identifiers exactly as listed in the catalog.Source: §A.4 · p. 18
• The prompt supplies unresolved task templates and a candidate API catalog, then requests solving API identifiers with brief reasoning; it allows an empty list when none apply.Source: §A.4 · p. 18