Source-linked AI summary
ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data
Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, Ameet Talwalkar
TL;DR
Web agents often rely on proprietary general-purpose LLMs that are not optimized for web contexts. ScribeAgent fine-tunes open models on production-scale workflows and achieves strong results on Mind2Web and WebArena, though Mind2Web evaluation has limitations.
Problem
Leading web agents rely on general-purpose proprietary LLMs that are not optimized for HTML and navigation, and their proprietary nature obstructs adaptation through continual training.
Source: §1 · p. 1Method
ScribeAgent fine-tunes open-source LLMs on production-scale real-world workflows to generate the next action directly from website DOM and action history.
Source: §1 · p. 2Results
5–10% higher step success rate across Mind2Web test sets for ScribeAgent-Large; on WebArena, ScribeAgent-Small reaches 51.3% task success from 45.7%, and Large reaches 53%.
Source: §1 · p. 2Takeaways & Limitations
The results highlight the potential of specialized fine-tuning with production-scale data to improve web-agent capabilities and reduce inference costs with smaller open-source models.
Source: §1 · p. 3Mind2Web evaluation can underestimate performance by marking functionally identical deeper elements incorrect, and its synthetic trajectories differ from the training-data distribution.
Source: §4.2 · p. 10
Abstract
from arXiv · showhide
Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. Most of these agents rely on general-purpose, proprietary models like GPT-4 and focus on designing better prompts to improve their planning abilities. However, general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. This simple yet effective approach shows substantial gains over prompting-based agents on existing benchmarks -- ScribeAgent achieves state-of-the-art direct generation performance on Mind2Web and improves the task success rate by 7.3% over the previous best text-only web agents on WebArena. We further perform detailed ablation studies on various fine-tuning design choices and provide insights into LLM selection, training recipes, context window optimization, and effect of dataset sizes.
1 INTRODUCTION
The introduction identifies a gap in prompt-based agents built on general-purpose proprietary LLMs and presents workflow-data fine-tuning of open models as an alternative with benchmark gains.
1 INTRODUCTION
- Existing benchmark performance remains below human levels, while leading agents rely on proprietary general-purpose LLMs not optimized for HTML or navigation and difficult to adapt through continual training.The introduction cites WebPilot (Zhang et al., 2024), AWM (Wang et al., 2024b), and SteP as examples.Source: §1 · p. 1
- ScribeAgent fine-tunes open-source LLMs on real-user workflows with DOM observations and documented actions, using a dataset spanning over 250 domains and more than 6 billion tokens.The dataset also covers over 10,000 subdomains and varied task objectives, difficulty, and length.Source: §1 · p. 2
- 5–10% higher step success rate across Mind2Web test sets; on WebArena, ScribeAgent-Small reaches 51.3% task success from 45.7%, while ScribeAgent-Large reaches 53%.ScribeAgent-Large also achieves state-of-the-art direct-generation performance on Mind2Web.Source: §1 · p. 2
- The introduction identifies direct fine-tuning on HTML-DOM, HTML preprocessing, backbone and context-window choices, and dataset size as sources of empirical insight.It reports that fine-tuning improves agent performance as dataset size increases.Source: §1 · p. 2
- Specialized fine-tuning is presented as a way to improve agent capabilities and reduce inference costs through smaller open-source models, with extensions to search, memory, planning, and multimodal systems proposed.The proprietary workflow dataset is restricted due to privacy concerns, though preprocessing, training, and inference code are released.Source: §1 · pp. 2–3
2 RELATED WORK
Related work spans prompting-based agents, fine-tuned web agents, HTML-input question answering, and multimodal systems; ScribeAgent is positioned as a workflow-data-trained text agent.
2 RELATED WORK
- Prompting-based agents use self-feedback, demonstrations, evaluators, synthesized instructions, and search, but their performance remains dependent on the underlying LLM.Examples include self-feedback (Sun et al., 2023), in-context demonstrations (Fu et al., 2024; Zheng et al., 2024; Wang et al., 2024b), and search (Koh et al., 2024; Zhang et al., 2024).Source: §2 · p. 3
- MindAct (Deng et al., 2023) filters web elements before selection, while WebAgent (Gur et al., 2023) processes HTML-5 before generating page-control code.Both differ from ScribeAgent’s direct next-step generation described in the paper.Source: §2 · p. 3
- AutoWebGLM (Lai et al., 2024) combines curriculum learning, reinforcement learning, and rejection sampling; NNetnav fine-tunes from synthetic demonstrations.The passage contrasts these procedures with fine-tuning a single LLM using sufficient high-quality workflow data.Source: §2 · p. 3
- Earlier HTML-input fine-tuning work focused on question answering and did not generate action sequences from user objectives.The cited examples are Gur et al. (2022), Nakano et al. (2022), and Liu et al. (2023).Source: §2 · p. 3
- Multimodal agents combine screenshots with HTML, whereas this version of ScribeAgent uses text inputs and leaves multimodal development for future work.The authors attribute this scope to the lack of effective visual preprocessing schemes.Source: §2 · p. 3
3 METHOD
The method frames web tasks as objective-guided action loops and trains specialized agents from annotated production workflows. It preprocesses DOMs and next-action examples, then studies model, context-window, and dataset-size choices.
3.1 GENERAL SETUP
- Web tasks are modeled as objective-guided sequential decisions: the agent uses the current observation and action history to choose an action, which changes the environment state.The loop ends when the agent signals stop or a task termination condition is met.Source: §3.1 · p. 4
3.2.1 COLLECTING PRODUCTION-SCALE DATA
- Scribe workflows span everyday web applications and pair a high-level objective with each step’s URL, raw HTML-DOM, action description and type, and target CSS selector.The data are collected through Scribe from user-recorded interactions; workflows with invalid selectors and non-English workflows are removed, yielding data from over 250 domains and 10,000 subdomains, averaging 11 steps and about 6 billion tokens.Source: §3.2.1 · pp. 4–5
3.2.2 PREPROCESSING
- DOM pruning removes metadata, CSS, JavaScript, comments, whitespace, and unhelpful attributes while retaining useful tags and attributes, addressing DOMs that can exceed open-source models’ context windows.Attributes with character-to-token ratios below 2 are removed, and each retained tag receives a unique ID.Source: §3.2.2 · p. 5
3.2.3 FINE-TUNING WITH LORA
- Supervised fine-tuning uses 1,200 workflows for testing and the remaining workflows for training, with each example labeled by one next step rather than all remaining task steps.The agent is trained to generate the five-line action format; LoRA is used to reduce fine-tuning cost because updating more parameters had not shown significant performance gain.Source: §3.2.3 · p. 6
3.2.4 EXPLORING THE DESIGN SPACE
- Qwen models achieve stronger prediction quality as model size grows, while larger models cost more to train and serve; Qwen2 7B balances accuracy and efficiency, and Qwen2.5 32B offers stronger performance.The comparison covers different LLMs fine-tuned on 1B workflow tokens and evaluated on the proprietary test split.Source: Table 1 · p. 6, §3.2.4 · p. 7
- On both Qwen2 and Qwen2.5, scaling context from 32K to 65K increases Exact Match by approximately 2% but decreases Calibrated Exact Match by approximately 2.5%, while inference time rises approximately fourfold.The authors hypothesize the Calibrated Exact Match drop may relate to rotary position embedding and choosing among twice as many targets.Source: §3.2.4 · p. 7, Table 2 · p. 7
- Doubling dataset size yields roughly a 2% performance boost for Qwen2 7B on a log-linear scale.Table 3 uses Qwen2-7B-Instruct and a 32K context window for all settings.Source: §3.2.4 · p. 7, Table 3 · p. 7
4 RESULTS
ScribeAgent is evaluated on proprietary workflows, Mind2Web, and WebArena, with results spanning next-step prediction and end-to-end task completion. It outperforms general-purpose baselines in the reported comparisons, though Mind2Web task-success results are qualified by evaluation and data-distribution issues.
4.1 PROPRIETARY DATASET
- On the full 1,200-workflow test set, ScribeAgent significantly outperforms GPT-4o and GPT-4o mini, while fine-tuning boosts non-fine-tuned Qwen2 performance by nearly 6×.Table 4 compares ScribeAgent with non-fine-tuned general-purpose baselines, with inputs truncated at 32K tokens.Source: §4.1 · p. 8, Table 4 · p. 8
- Across CRM, e-commerce, productivity, and social websites, ScribeAgent-Small beats general-purpose baselines, despite a 6% gap between its best- and worst-performing domains.The comparison is shown as Exact Match across website types.Source: §4.1 · p. 8, Figure 2 · p. 8
- On 500 sampled proprietary workflows, o1-preview performs best among general-purpose baselines, but ScribeAgent still outperforms it by a wide margin.ScribeAgent-Small has 7B parameters and ScribeAgent-Large 32B; neither requires additional inference scaling.Source: §4.1 · p. 8, Table 5 · p. 8
4.2 MIND2WEB
- In Mind2Web multi-stage evaluation, ScribeAgent-Large has the best overall zero-shot performance, but its task success rates remain unsatisfactory, mainly due to training-data distribution differences.Element accuracy and step success rate are competitive with the best fine-tuned baseline, HTML-T5-XL, on cross-website and cross-domain tasks.Source: §4.2 · pp. 9–10
- ScribeAgent-Large direct generation outperforms all existing Mind2Web baselines, with step success rates 2-3 times higher than fine-tuned Flan-T5 and 5-10% higher than Synapse.Because ScribeAgent was not trained on Mind2Web, its performance is similar across Cross-Task, Cross-Website, and Cross-Domain test sets.Source: Table 6 · p. 9, §4.2 · p. 10
- Mind2Web’s exact element-ID evaluation can mark functionally equivalent or hierarchically deeper targets incorrect, and its synthetic trajectories differ from ScribeAgent’s training data.The reported action mismatch includes ScribeAgent clicking before typing where Mind2Web expects immediate typing.Source: §4.2 · p. 10
4.3 END-TO-END TASK EXECUTION ON WEBARENA
- 7.3% higher total task success rate than the previous-best GPT-4-Turbo-based AgentOccam; ScribeAgent augmented with GPT-4o leads text-only baselines in 4 of 5 WebArena categories.Reddit and GitLab results are stronger than those on other domains; the table caption reports testing ScribeAgent-Small because of evaluation costs.Source: Table 7 · p. 11, §4.3 · p. 11
- WebArena ablations show that using ScribeAgent in the four-stage system consistently outperforms using GPT-4o alone, and training-token increases and the Qwen2 7B backbone track higher task success in the tested subset.The ablation reports that the GPT-4o-only setup is less effective than existing agents such as WebPilot; the backbone and token trends were tested on 158 tasks.Source: §4.3 · pp. 11–12, Table 8 · p. 11, Table 9 · p. 12
5 CONCLUSION
The paper reports that ScribeAgent consistently outperforms methods prompting proprietary models across evaluation settings and benchmarks. It also identifies long DOM contexts and the absence of planning as limitations and directions for future work.
5 CONCLUSION
- ScribeAgent consistently outperforms methods prompting proprietary models across various evaluation settings and benchmarks.The paper also reports empirical insights into data processing and model fine-tuning.Source: §5 · p. 12
- Long DOMs challenge LLM adaptation, and ScribeAgent currently outputs the next action without incorporating planning.The authors aim to compare multiple DOM chunks, potentially using memory, and suggest MCTS as a possible approach to online planning and exploration.Source: §5 · p. 12
- The authors also aim to expand ScribeAgent to handle multimodal inputs and multilingual content.They describe this as a long-term goal.Source: §5 · p. 12
A.1.2 TOKENIZER PRUNING
The tokenizer-pruning analysis examines how tokenizer-based filtering affects DOM content length and false positives. It reports a trade-off between reduction and precision, alongside differences in how tokenizers handle URLs.
A.1.2 TOKENIZER PRUNING
- The rationale assumes typical English words contain more than two characters, using character and token counts to identify likely random strings.The described threshold rule is intended to remove nonsensical strings composed solely of single-character tokens.Source: §A.1.2 · p. 17
- The pruning rule targets tag attributes with a character-to-token ratio below 2, aiming to remove strings made of single-character tokens.In implementation, the technique is applied only to attributes longer than 32 characters, with shorter attributes treated more leniently.Source: §A.1.2 · p. 17
- The analysis varies thresholds from 1.5 to 2.5 across Qwen2, Mistral, and Llama tokenizers, using 1,000 sampled DOMs from the proprietary test dataset.It examines false positives, average character and token counts before and after pruning, and frequently pruned tags and attributes.Source: §A.1.2 · p. 17
- Greater reductions in content length tend to result in higher false-positive rates in tokenizer pruning.The cited result is presented as a precision-versus-context-reduction trade-off in Table 10.Source: §A.1.2 · p. 17, Table 10 · p. 17
- Threshold 2 achieves the most balanced trade-off among the tested pruning thresholds.The authors say this aligns with their intuition.Source: §A.1.2 · p. 18
- Qwen and Mistral tokenizers tend to prune href attributes, whereas Llama preserves them and better tokenizes URLs.The authors use Qwen for preprocessing to align with ScribeAgent’s backbone, while noting Llama as a possible alternative for URL recognition and shorter token sequences.Source: §A.1.2 · p. 18
A.2 EXAMPLE PROMPT AND LABEL FOR SCRIBEAGENT
The example illustrates a ScribeAgent task to grant another user delegation access in Gmail settings, with a URL, a processed-DOM placeholder, and labeled action steps.
A.2 EXAMPLE PROMPT AND LABEL FOR SCRIBEAGENT
- The example’s objective is to grant delegation access to another user in Gmail settings.It supplies a Gmail URL and a processed-DOM placeholder as context.Source: §A.2 · p. 18
- The first labeled action is clicking “See all settings,” identified as a mouse click on a button.The example gives the target node and HTML button.Source: §A.2 · p. 18
- The next steps click the “Accounts” tab and then “Add another account.”The example labels the respective targets as a tab and a link.Source: §A.2 · p. 18
A.3 OPENAI PROMPTS
The appendix shows an OpenAI prompt for describing a current webpage action, followed by generated description examples. The prompt uses the objective, prior actions, current action, and screenshot to ground a concise description.
A.3.1 DATA PREPARATION
- The prompt asks for a concise description of the current action on the webpage.It frames the task as navigating a page to achieve an objective.Source: §A.3.1 · p. 18
- The objective and previous steps provide grounding, while the current action and its screenshot are emphasized as most useful.The prompt includes all four as context.Source: §A.3.1 · p. 18
- The prompt directs the model to inspect the part of the webpage marked by a red circle, where the user clicked.It asks the model to use relevant image information to ground the action description.Source: §A.3.1 · pp. 18–19
- The requested response is one concise, coherent sentence that starts with an action verb.The prompt gives “Click on the ‘SUBMIT’ button” as an example format.Source: §A.3.1 · p. 19
- The prompt disallows JSON and markdown formatting.It asks for a single sentence.Source: §A.3.1 · p. 19
- The appendix presents examples of action descriptions generated using GPT-4o.The listed examples include clicking Submit, typing an item name, and double-clicking highlighted text.Source: §A.3.1 · p. 19
A.3.2 PROPRIETARY BENCHMARK BASELINES
The OpenAI baseline prompt frames web navigation as producing one next action from the user’s objective, page URL, HTML, and action history. It constrains that action to a specified format.
A.3.2 PROPRIETARY BENCHMARK BASELINES
- The prompt combines the task objective, current URL, HTML, and action history as input for next-step generation.The prompt is prepended to each input, followed by those task-specific details.Source: §A.3.2 · p. 19
- The agent may click, type a character sequence, or press a key combination.Source: §A.3.2 · p. 19
- The response must contain a single next step in the prescribed Description, Action, Node, and Target format.The prompt prohibits additional punctuation and markdown formatting.Source: §A.3.2 · p. 19
A.4.1 PREPROCESSING
The appendix describes data preprocessing, refinements to Mind2Web evaluation, and a multi-stage WebArena agent workflow. It reports gains from refined evaluation while noting a remaining distribution gap.
A.4.1 PREPROCESSING
- Mind2Web DOMs are reprocessed and mapped to backend labels; DOMs over 32K tokens are chunked, with five actions sampled per chunk and the most frequent selected.Source: §A.4.1 · p. 19
A.4.2 REFINED EVALUATION
- Refined evaluation yields significant gains in both settings; label relaxation bridges part of the distribution gap, and the Mind2Web ranker covers most of those gains.Source: Table 11 · p. 20, §A.4.2 · p. 20
- Mind2Web evaluation relaxes labels to include children and grandchildren, matches target attributes as well as node IDs, and converts input-field clicks to typing actions.Source: §A.4.2 · p. 20
A.5.1 FOUR-STAGE PIPELINE
- The WebArena workflow expands intent, preprocesses DOM chunks and generates action candidates for GPT-4o selection, then maps the selected action into accessibility-tree format.Source: §A.5.1 · pp. 20–21
- GPT-4o checks whether the task is complete and returns an answer for information-seeking tasks; the action is then sent to the environment.Source: §A.5.1 · p. 22
A.5.2 SCROLLING ACTIONS AND COMBOBOX SELECTION
- Because training captures the full DOM, it has no scrolling; the WebArena system inserts scrolling when a target is outside the viewport and types combobox choices to surface them.Source: §A.5.2 · p. 22
A.5.3 GPT-4O-ONLY SETTING
- In the GPT-4o-only setting, the prompt requests a single next action with its description, action type, node, and HTML target.Source: §A.5.3 · p. 23
A.5.4 RESULTS
- Example trajectories include a multi-step wish-list update and an airport-address lookup that returns the address as the answer.Source: §A.5.4 · pp. 23, 25