Source-linked AI summary
A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models
Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S. Yu, Qing Li
TL;DR
Repetitive, time-consuming web tasks motivate autonomous WebAgents, while LFMs offer human-like comprehension and reasoning for following user instructions. This survey synthesizes WebAgent research across architectures, training, and trustworthiness, concluding that the field remains early and that fairness and explainability need further study.
Problem
Many web tasks are repetitive and time-consuming, raising whether LFMs can power agents that automatically handle them according to user instructions.
Method
The survey reviews representative WebAgent methods across architectures, training, and trustworthiness, including safety, privacy, and generalizability.
Results
The survey provides a comprehensive overview of recent WebAgent developments and discusses promising future research directions.
Takeaways & Limitations
WebAgent research offers a structured basis for understanding autonomous web-task systems and identifying directions for further investigation.
Takeaways & Limitations
Fairness and explainability remain in early development, with limited literature on these dimensions for WebAgent techniques.
Abstract
from arXiv · showhide
With the advancement of web techniques, they have significantly revolutionized various aspects of people's lives. Despite the importance of the web, many tasks performed on it are repetitive and time-consuming, negatively impacting overall quality of life. To efficiently handle these tedious daily tasks, one of the most promising approaches is to advance autonomous agents based on Artificial Intelligence (AI) techniques, referred to as AI Agents, as they can operate continuously without fatigue or performance degradation. In the context of the web, leveraging AI Agents -- termed WebAgents -- to automatically assist people in handling tedious daily tasks can dramatically enhance productivity and efficiency. Recently, Large Foundation Models (LFMs) containing billions of parameters have exhibited human-like language understanding and reasoning capabilities, showing proficiency in performing various complex tasks. This naturally raises the question: `Can LFMs be utilized to develop powerful AI Agents that automatically handle web tasks, providing significant convenience to users?' To fully explore the potential of LFMs, extensive research has emerged on WebAgents designed to complete daily web tasks according to user instructions, significantly enhancing the convenience of daily human life. In this survey, we comprehensively review existing research studies on WebAgents across three key aspects: architectures, training, and trustworthiness. Additionally, several promising directions for future research are explored to provide deeper insights.
1 INTRODUCTION
WebAgents address repetitive, time-consuming web tasks by combining autonomous AI-agent behavior with the human-like comprehension and reasoning capabilities of LFMs. This survey organizes WebAgent research around architectures, training, and trustworthiness.
- Many everyday web activities, including account registration, form filling, and product purchasing, require repetitive information entry, comparison, and decision-making.
- AI Agents can continuously execute tasks without fatigue or performance degradation, supporting reliable automation of repetitive workflows.
- LFMs with billions of parameters exhibit human-like comprehension and reasoning capabilities after training on massive data.
- LFM-empowered WebAgents follow natural-language instructions to perform multi-step web tasks such as retrieving an email address, composing a message, and sending it.
- The survey reviews WebAgents through three perspectives: architectures, training, and trustworthiness.
- Unlike related surveys centered on operating-system agents, planning mechanisms, or broad autonomous agents, this work focuses specifically on WebAgents.
2 BACKGROUND
WebAgents build on AI-agent paradigms by using LFMs to generate actions for web environments, iteratively updating website states until user tasks are completed. Their background spans reinforcement-learning agents and newer LFM-based agents.
- AI Agents are broadly categorized into reinforcement-learning-based agents and LFM-empowered agents.
- Reinforcement-learning agents learn policies through environmental interaction and reward signals, with value-function and policy optimization as major approaches.
- LFM-based agents have attracted attention because their human-like intelligence and open-world knowledge support applications across complex environments.
- For a website and user instruction, a WebAgent generates executable actions and interacts with the environment through those actions.
- WebAgents repeatedly generate and execute actions, updating the website state after each operation until the user-given task is completed.
3 WEBAGENT ARCHITECTURES
WebAgent architectures are organized as three consecutive processes: perception, planning and reasoning, and execution. Together, these processes connect environmental observation to action and interaction.
- WebAgents fulfill user commands through perception, planning and reasoning, and execution.Perception observes the environment; planning and reasoning interpret the task and predict actions; execution performs those actions.
3.1 Perception
WebAgent perception must interpret dynamic web environments before action generation, using textual metadata, screenshots, or both. Multimodal approaches combine complementary data sources for broader environmental understanding.
- WebAgents must perceive external environments and reason behaviorally over dynamic states in addition to processing user instructions.For example, opening YouTube requires locating the browser address bar before entering the URL.
- Existing WebAgent perception methods use text-based, screenshot-based, or multimodal environmental inputs.
- Text-based WebAgents typically use webpage metadata such as HTML and accessibility trees to represent the environment.
- Textual representations can be verbose, environment-dependent, visually misaligned with graphical interfaces, and associated with weaker generalization and greater computational overhead.
- Multimodal WebAgents combine textual metadata and screenshots to exploit their complementary strengths for environmental perception.
3.2 Planning & Reasoning
WebAgents plan and reason over user instructions and environmental observations to generate actions, using explicit or implicit planning, reactive or strategic reasoning, and short- or long-term memory.
- Task Planning: Task planning determines an efficient action sequence and is categorized as explicit or implicit according to whether task decomposition is explicit.Explicit methods decompose requests into sub-tasks, whereas implicit methods directly use instructions and observations to generate actions progressively.
- Task Planning: Explicit planners can decompose complex requests, describe screenshots, generate function-call actions, and reflect on whether to proceed, retry, or reformulate.ScreenAgent exemplifies this structured workflow, while OS-Copilot also retrieves relevant external information during planning.
- Action Reasoning: Action reasoning is categorized as reactive or strategic according to whether additional exploration or in-context information supplements direct next-action generation.Strategic methods may simulate candidate-action outcomes before execution, while reactive methods generate actions directly from prompts containing observations and instructions.
- Memory Utilization: Short-term memory stores previous actions and webpage states, helping agents avoid redundant operations and improve task-completion efficiency.AutoWebGLM and LLMPA incorporate action histories or generated action descriptions into prompts for predicting the next action.
- Memory Utilization: Long-term memory stores persistent external information, including prior task trajectories and knowledge acquired through online search, which can enhance task success rates.Agent S retrieves online knowledge and successful or failed trajectory summaries, while Synapse retrieves similar trajectories as exemplars.
3.3 Execution
Execution completes WebAgent tasks through grounding and interaction: agents locate target webpage elements, then perform generated actions using browser-based or tool-based methods.
- Execution Overview: Execution comprises grounding, which locates webpage elements, and interacting, which performs actions on the selected elements.These steps are repeated as WebAgents execute actions to complete complex user tasks.
- Grounding: Direct grounding selects an element from a screenshot or the full HTML, while inferential grounding uses auxiliary modules to locate the target.Ponder & Press uses a general-purpose MLLM to translate instructions and a GUI-specific MLLM to identify elements visually.
- Interacting: Web browsing-based and tool-based methods represent two broad approaches for interacting with webpages.Tool-based methods use APIs, and API-calling agents can bypass GUI interactions to improve efficiency and adaptability for online tasks.
- Interacting: AgentOccam simplifies the action space by removing redundant, low-utility, and embodiment-dependent actions while adding high-level commands such as [Note] and [Stop].The simplification is described as enhancing WebAgent interaction efficiency.
4 TRAINING OF WEBAGENTS
WebAgent training depends on both data and training strategies: data supplies diverse web-specific examples, while strategies determine how capabilities are acquired and refined.
- Training Framework: The training framework focuses on constructing training data and applying strategies that acquire and refine WebAgent capabilities.The survey presents these as the two fundamental aspects of WebAgent training.
4.1 Data
WebAgent training data is constructed through preprocessing and augmentation, which improve usability, quality, quantity, and diversity across heterogeneous web environments.
- Data Construction: Data preprocessing refines and structures training data, while data augmentation increases dataset quantity and diversity.These are the two essential steps in WebAgent training-data construction.
- Data Pre-processing: Preprocessing addresses heterogeneous modalities and platform formats through filtering, curation, mixture, and format alignment.Format alignment resolves naming conflicts such as mobile “tap” versus PC “click”; OS-Atlas aligns cross-platform action spaces.
- Data Augmentation: Data augmentation uses collection or synthesis to obtain large volumes of diverse data for perception, reasoning, and execution.Collection draws from public datasets or real-world scenarios, whereas synthesis generates web-relevant data with LLMs or VLMs when real samples are insufficient or costly.
- Data Collection: Lexi collects 114k UI images paired with functional captions from open websites spanning diverse applications, platforms, and UI characteristics.The example illustrates collection-based augmentation using curated real-world web data.
- Data Synthesis: Synatra converts procedural knowledge, environment knowledge, and ungrounded observations into direct demonstrations at scale.This synthesis process increases the richness of information in WebAgent training data.
4.2 Training Strategies
WebAgent training strategies are organized into four categories, addressing GUI comprehension, task-oriented capabilities, and adaptation to dynamic web environments.
- Existing WebAgent training strategies comprise training-free methods, GUI comprehension training, task-specific fine-tuning, and post-training.The categories differ in learning paradigms, data usage, and optimization objectives.
- Training-free: Training-free methods adapt LFMs into WebAgents through prompts that guide web-task execution using screen descriptions, previous actions, and outcomes.The approach also generates descriptions of the next step and its potential results.
- GUI Comprehension Training: GUI comprehension training targets general-purpose LFMs’ weaknesses in screen understanding and OCR, which can cause misinterpretation of webpage interface elements.These weaknesses are especially relevant to visually and textually dense pages such as e-commerce product pages.
- GUI Comprehension Training: Layout-guided contrastive learning enhances GUI comprehension by modeling individual UI-element semantics according to visual organization.
- GUI Comprehension Training: Additional training components, such as screenshot region summarization, are used to strengthen GUI understanding.Spotlight extracts essential screenshot regions using VLM encoding, while ScreenAI employs Pix2Struct’s patching strategy.
- Task-specific Fine-tuning: Task-specific fine-tuning equips WebAgents with planning, reasoning, and interaction skills for web-oriented tasks.
- Post-training: Post-training uses reinforcement learning to help WebAgents adapt to changing interfaces and user requirements through real-time environmental feedback.AutoGLM exemplifies continuous self-evolving learning through autonomous interactions with web environments.
5 TRUSTWORTHY WEBAGENTS
Trustworthy WebAgents must address security, privacy, and generalizability risks arising from complex, adversarial, and changing web environments.
- WebAgents face inherent risks and challenges as they become integrated with web systems across applications such as e-commerce, healthcare, and education.
- Security: Robust WebAgents must withstand noisy changes and adversarial attacks, including black-box attacks and malicious instructions embedded in public webpages.
- Security: Benchmarks such as ARE, ST-WebAgentBench, and RedCode provide structured evaluations of WebAgent safety and trustworthiness.ARE includes 200 targeted adversarial tasks in a realistic VisualWebArena threat model.
- Safety Methods: Policy composition and multi-agent communication are proposed to help WebAgents manage action control and verify task accuracy.
- Privacy: WebAgent interactions can expose personal, financial, and proprietary information when agents encounter compromised websites.MEXTRA examines private-information extraction from agent memory, while EIA studies attacks targeting personal information or entire user requests.
- Privacy: PrivacyLens evaluates privacy leakage across multiple levels by expanding privacy norms into vignettes and agent trajectories.
- Generalizability: Out-of-distribution mismatch challenges WebAgent generalizability because training and testing data may come from different distributions.Mind2Web addresses this with over 2,000 open-ended tasks from 137 websites across 31 domains and crowd-sourced action sequences.
- Other Trustworthiness Dimensions: Fairness and explainability remain important but underdeveloped dimensions of trustworthy WebAgents.The survey identifies both as future research directions requiring dedicated efforts.
6 FUTURE DIRECTIONS
Future WebAgent research should strengthen trustworthy behavior, evaluation, personalization, and adaptation to specialized professional domains.
- 6.1 Trustworthy WebAgents: Fairness and explainability are under-explored compared with capability enhancement and require further study in WebAgent research.Fairness concerns unbiased perception, reasoning, and execution, while explainability concerns justifying actions in high-stakes settings.
- 6.2 Datasets and Benchmarks of WebAgents: Comprehensive benchmarks evaluate WebAgent navigation, automation, personalization, and interaction with dynamic online interfaces, but often cover limited functionality facets.PersonalWAB contains 1,000 user profiles and over 40,000 real-world web behaviors; Webcanvas evaluates real-time online interactions.
- 6.3 Personalized WebAgents: WebAgents remain limited in personalization because billion-scale models and expensive training costs can produce generic actions misaligned with individual users.
- 6.3 Personalized WebAgents: Retrieval-Augmented Generation and memory-based approaches are among the emerging strategies for improving WebAgent personalization.
- Domain-specific WebAgents: Applying LFM-empowered WebAgents to education and healthcare remains largely unexplored despite substantial needs and potential benefits.Adapting general-purpose WebAgents to domain-specific contexts is identified as a promising direction.
7 CONCLUSION
The survey reviews WebAgents as autonomous systems for repetitive web tasks and organizes the field around architectures, training, and trustworthiness. It also identifies future research directions for this emerging area.
- The survey summarizes WebAgent research from the perspectives of architectures, training, and trustworthiness, then discusses potential future research directions.