Source-linked AI summary
AutoWebGLM: A Large Language Model-based Web Navigating Agent
Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, Jie Tang
TL;DR
AutoWebGLM addresses the difficulty of reliable real-world web navigation by combining webpage simplification, operational action modeling, and data-driven training. Built on ChatGLM3-6B, it combines curriculum learning, reinforcement learning, and rejection sampling finetuning, and is reported to reach a practically usable level for real-world web tasks. Its scope remains constrained by reduced performance on unfamiliar websites and instability in real web environments.
Problem
Existing web agents face practical limitations from complex HTML, diverse webpage actions, and open-domain task difficulty, while high-quality complex browsing data is scarce.
Method
AutoWebGLM combines simplified HTML observations and a complete action space with model-assisted and manual browsing data, curriculum learning, reinforcement learning, and rejection sampling finetuning.
Results
AutoWebGLM achieves performance comparable to advanced LLM-based agents and reaches a practically usable level for real-world web tasks.
Takeaways & Limitations
The system provides an open language-model-based agent and a bilingual web browsing benchmark for evaluating autonomous navigation across real-world tasks.
Takeaways & Limitations
Performance and efficiency may decrease on unfamiliar websites or websites with unique operating logic, while real web-environment stability is not guaranteed.
Abstract
from arXiv · showhide
Large language models (LLMs) have fueled many intelligent web agents, but most existing ones perform far from satisfying in real-world web navigation tasks due to three factors: (1) the complexity of HTML text data (2) versatility of actions on webpages, and (3) task difficulty due to the open-domain nature of the web. In light of these challenges, we develop the open AutoWebGLM based on ChatGLM3-6B. AutoWebGLM can serve as a powerful automated web navigation agent that outperform GPT-4. Inspired by human browsing patterns, we first design an HTML simplification algorithm to represent webpages with vital information preserved succinctly. We then employ a hybrid human-AI method to build web browsing data for curriculum training. Finally, we bootstrap the model by reinforcement learning and rejection sampling to further facilitate webpage comprehension, browser operations, and efficient task decomposition by itself. For comprehensive evaluation, we establish a bilingual benchmark -- AutoWebBench -- for real-world web navigation tasks. We evaluate AutoWebGLM across diverse web navigation benchmarks, demonstrating its potential to tackle challenging tasks in real environments. Related code, model, and data are released at \url{https://github.com/THUDM/AutoWebGLM}.
1 INTRODUCTION
AutoWebGLM addresses limitations in practical web agents by combining an open ChatGLM3-6B-based agent with data construction, learning, and browsing-operation strategies. The system is presented as practically usable for real-world web tasks and evaluated with a bilingual browsing benchmark.
- Existing web agents are restricted by the absence of a universal action space and the diversity of real-world websites.
- AutoWebGLM is built on the open ChatGLM3-6B model to construct webpage navigation agents.
- The training pipeline combines supervised learning, reinforcement learning, curriculum learning, and rejection sampling finetuning for web browsing.
- The authors deploy AutoWebGLM as a Chrome extension that performs operations across websites to complete user tasks.
- AutoWebBench is introduced as a bilingual English-and-Chinese webpage browsing evaluation dataset for regionally varied websites.
- A six-billion-parameter AutoWebGLM achieves performance comparable to advanced LLM-based agents and reaches a practically usable level for real-world web tasks.
2 RELATED WORK
Related work spans web-agent architectures, language models, web-navigation benchmarks, prompt-based data construction, and rejection sampling finetuning. These lines of work motivate AutoWebGLM’s focus on practical operation, structured webpage understanding, and scalable training data.
- Web browsing agents combine language models for decision-making with HTML parsers for environment observation and require suitable evaluation criteria.
- Smaller open models such as LLaMA-2-7B and ChatGLM3-6B are pursued because large language models are costly to deploy while retaining strong capabilities in some domains.
- Web-navigation benchmarks progress from simulated component interaction in MiniWoB++ to complex browsing in Mind2Web and real-website task completion in WebArena.
- Earlier WebGPT and WebGLM systems primarily used web environments for question answering, whereas newer work emphasizes complex operations and interactive tasks.
- Prompt-based data construction uses language models to generate synthetic training data, including diverse and complex instructions through Evol-Instruct.
- Rejection Sampling Finetuning collects accurate reasoning paths from a supervised model as augmented finetuning data and is reported to improve web-page browsing capabilities.
3 AUTOWEBGLM AS A WEB AGENT
AutoWebGLM models browsing as sequential decision-making over webpage states, histories, and actions. Its framework simplifies observations, represents operational actions, and iterates between language-model decisions and automated browser execution.
- 3.1 Problem Setup: Web browsing is formulated as sequential decision-making in which a policy selects actions from the current state and history.
- 3.1 Problem Setup: The state contains HTML, URL, and window position, while the action space includes operations such as clicking, scrolling, and typing.
- 3.1 Problem Setup: History is updated from the previous history, previous action, and current state before the policy selects the next action.
- 3.2 The AutoWebGLM Framework: HTML simplification and OCR transform webpage HTML and screenshots into a concise representation with operable elements marked for interaction.
- 3.2 The AutoWebGLM Framework: The interaction framework organizes simplified HTML and other observations, while an automated web program executes the language model’s predicted actions.
- 3.2.1 Observation space.: The observation space includes task description, simplified HTML, current location, and past operation records to support webpage comprehension and operation.
- 3.2.1 Observation space.: Window position and page size provide spatial context, while previous actions help prevent ineffective repetition loops.
- 3.2.1 Observation space.: The action space is designed around operational possibilities encountered during real task execution and is intended to be complete and self-consistent.
4 BUILDING AUTOWEBGLM
AutoWebGLM is built through human-AI hybrid data construction and staged training that progresses from webpage understanding and simple operations to complex-task planning, reinforcement learning, and self-sampling. The resulting model acquires capabilities for independent web browsing while addressing limitations of supervised training.
- Data Construction: Human-AI hybrid data construction produces training data for webpage recognition, simple operations, and complex web tasks.Complex-task samples include real-world tasks, operation sequences, and step intents.
- Web Recognition: Web recognition data uses parsed and simplified HTML to train understanding of webpage structures and interactive components.The process records component positions and sizes, then generates simplified HTML and natural-language recognition questions and responses.
- Simple Task Operation: Simple-task data trains single-step operations such as clicking links, filling forms, and navigating sections, with rule-based construction used to ensure executability.The dataset is split by operation type to cover basic web interactions.
- Complex Task Operation: Complex-task data trains planning and reasoning by pairing real-world browsing tasks with manually captured operation sequences and step intents.About 20 feasible tasks per website were manually selected and labeled from an initial set of 50 designed tasks.
- Training Procedure: Training proceeds from supervised fine-tuning and curriculum learning to complex-task planning, self-sampling, and reinforcement learning.The curriculum begins with webpage interpretation and simple operations, then advances to task decomposition and execution based on webpage state and prior actions.
- Training Procedure: After supervised training, AutoWebGLM can independently execute browser operations from user instructions, while later training targets more robust behavior.The supervised model already demonstrates essential web-task completion and browser operation capabilities.
- Training Procedure: Self-sampling reinforcement learning addresses hallucinations caused when supervised models overlook webpage state and preceding operation sequences.The model samples each complex-task example 20 times and uses successful and unsuccessful outputs to construct contrastive training data.
5 EXPERIMENTS
AutoWebGLM is evaluated across bilingual and established web-navigation benchmarks, with ablations examining data and training strategies. Results indicate benefits from complex-task data, joint simple-task training, DPO, and RFT, while case studies identify recurring execution errors.
- Benchmark Evaluation: AutoWebGLM is evaluated on AutoWebBench, Mind2Web, MiniWoB++, and WebArena using web-navigation performance measures.AutoWebBench is split into Chinese, English, in-domain, and out-of-domain subsets and uses Step Success Rate.
- Benchmark Evaluation: The experiments also measure execution efficiency across system components because speed is critical to user experience.These measurements are reported in the system execution-efficiency results.
- Ablation Study: Complex-task data significantly improves performance, whereas simple-task data alone yields only slight improvement but helps when combined with complex-task data.The ablation attributes the stronger effect of complex data to its closer alignment with real-world scenarios.
- Ablation Study: Training exclusively on complex tasks causes basic operational errors, while adding simple-task data can mitigate this problem.The finding supports combining simple and complex task data rather than using complex tasks alone.
- Ablation Study: DPO improves over SFT by facilitating learning from mistakes, while RFT supports bootstrap enhancement within different domains.The reported RFT effect is domain-specific improvement through practice.
- Case Study and Error Analysis: Case studies report satisfactory results in most scenarios, but error analysis identifies hallucinations, poor graphical recognition, task-context misinterpretation, and pop-up interruptions.These errors are described as relatively infrequent but important for further refinement.
6 FUTURE DIRECTION
The paper identifies multimodal input and stronger reasoning or self-checking as future directions for web navigation. These directions address weaknesses with advanced visual applications, unfamiliar websites, and unstable real-world environments.
- Multimodal Input: HTML input performs satisfactorily in many scenarios but falters on advanced applications such as maps, animations, and video browsing.The limitation motivates adding visual information beyond HTML.
- Multimodal Input: Image input helps interpret images, icons, and special effects but makes numerals and extensive web text harder to understand.The paper proposes combining HTML and webpage screenshots to combine the strengths of both modalities.
- Reasoning and Self-check Techniques: Efficiency and success rate may decrease on unfamiliar websites or sites with unique operating logic.The paper identifies this as a boundary of current web-navigation performance.
- Reasoning and Self-check Techniques: Self-check mechanisms that confirm current state and verify operation effects are proposed to improve robustness in unstable web environments.The paper also suggests reasoning strategies that use previous browsing experiences.
7 CONCLUSION
The conclusion presents AutoWebGLM as a web-navigation agent designed to address HTML complexity and open-domain browsing through simplified webpages and staged training. It also introduces a bilingual benchmark for future evaluation.
- Conclusion: AutoWebGLM is presented as an agent with robust performance across autonomous web-navigation benchmarks.The conclusion describes the model as addressing limitations of existing LLM-based web agents.
- Conclusion: The system simplifies HTML and uses curriculum learning, reinforcement learning, and rejection sampling finetuning to improve webpage comprehension and browser-operation learning.These methods are presented as the paper’s strategy for handling web-navigation challenges.
- Conclusion: The paper introduces a bilingual web-browsing benchmark as a foundation for future research.The conclusion identifies the benchmark as a distinct contribution.
A IMPLEMENTATION DETAILS OF AUTOWEBGLM
The implementation prompt supplies AutoWebGLM with a simplified HTML webpage, task description, browser context, and interaction functions. The model outputs one commented command to act on the current webpage.
- Prompt and Inputs: AutoWebGLM receives a simplified HTML webpage and a task description as inputs for web-navigation tasks.The prompt frames the model as an assistant whose goal is to complete the task.
- Action Output: The prompt exposes a click function that operates on a webpage element identified by its element_id.The function documentation specifies that the element id selects the element to click.
- Prompt and Inputs: The prompt includes previous commands, current viewport information, open tabs, and the task description as browser context.These fields provide the current interaction state needed for deciding the next action.
- Action Output: The model is instructed to output one command for interacting with the current webpage and add a brief reasoning comment.The command is selected from provided browser-interaction functions.
C DATA CONSTRUCTION PROMPT
The data-construction prompts generate task-operation pairs and infer user intent from recorded action traces. They constrain outputs to structured, one-step tasks or stepwise intent descriptions tied to the webpage and overall task.
- Task-operation generation: The task generator creates a one-step task that a specified operation can solve using the webpage HTML.The required output includes a generated task, the corresponding operation, and its intention or reasoning.
- Generation constraints: Generated tasks must avoid excessive simplicity, and type operations are restricted to input and textarea elements.The prompt also supplies the user’s overall task as context for generation.
- Task and trace intent: The prompts ask the model to infer each action’s intent from an annotated user action trace.The response must describe intents from the user’s first-person perspective and contain one line per action.
D ANNOTATION DETAILS
The annotation process used 20 annotators over one month to record and screen web-browsing tasks with a Chrome plugin.
- Annotation procedure: 20 annotators spent one month recording browsing actions in Google Chrome with the project plugin installed.They first checked whether target-website descriptions matched the assigned tasks.
- Task screening: Annotators screened tasks for clarity, relevance, achievability, complexity, and subjectivity before proceeding.Tasks that failed these criteria were skipped.
- Action recording: For accepted tasks, annotators carefully recorded each action step while completing the task.
E FULL RESULTS OF MINIWOB++
This section reports per-task average success rates for 56 MiniWoB++ tasks.
- MiniWoB++ results: Table 8 presents per-task average success rates across 56 tasks from MiniWoB++.