Source-linked AI summary
The BrowserGym Ecosystem for Web Agent Research
Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados, Alexandre Lacoste
TL;DR
Web-agent research needs standardized evaluation for complex browser tasks that are repetitive, time-consuming, and difficult to compare reproducibly. BrowserGym and AgentLab unify benchmarks, agent tooling, and experiments; across six benchmarks, Claude-3.5-Sonnet achieved 39.1% on WorkArena L2 versus GPT-4o’s 8.5%, while reproducibility remains challenged by live-web variability.
Problem
Web tasks are often repetitive, time-consuming, and multi-step, motivating assistants that can execute browser actions for users.
Method
The BrowserGym ecosystem unifies web benchmarks through a common interface and adds AgentLab for building, evaluating, analyzing, and reproducing web-agent experiments.
Results
Claude-3.5-Sonnet reached 39.1% on WorkArena L2, compared with 8.5% for GPT-4o, in the ecosystem’s large-scale multi-benchmark evaluation.
Takeaways & Limitations
BrowserGym and AgentLab standardize web-agent research while supporting extensible benchmarks, experiment management, reproducibility features, and trace analysis.
Takeaways & Limitations
Localization, software, browser, and dynamic webpage differences introduce stochasticity that can make benchmark results inconsistent and difficult to reproduce.
Abstract
from arXiv · showhide
The BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents, particularly those leveraging automation and Large Language Models (LLMs). Many existing benchmarks suffer from fragmentation and inconsistent evaluation methodologies, making it challenging to achieve reliable comparisons and reproducible results. In an earlier work, Drouin et al. (2024) introduced BrowserGym which aims to solve this by providing a unified, gym-like environment with well-defined observation and action spaces, facilitating standardized evaluation across diverse benchmarks. We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature and includes AgentLab, a complementary framework that aids in agent creation, testing, and analysis. Our proposed ecosystem offers flexibility for integrating new benchmarks while ensuring consistent evaluation and comprehensive experiment management. As a supporting evidence, we conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks made available in BrowserGym. Among other findings, our results highlight a large discrepancy between OpenAI and Anthropic's latests models, with Claude-3.5-Sonnet leading the way on almost all benchmarks, except on vision-related tasks where GPT-4o is superior. Despite these advancements, our results emphasize that building robust and efficient web agents remains a significant challenge, due to the inherent complexity of real-world web environments and the limitations of current models.
1 Introduction
Web-agent research is hindered by fragmented benchmarks and evaluation practices. The BrowserGym ecosystem addresses this with unified benchmarks, AgentLab tooling, and a large multi-benchmark model comparison.
- Web assistants target repetitive multi-step browser tasks, with potential benefits for user focus, accessibility, and visual verification.
- Fragmented benchmark code and inconsistent evaluation practices hinder fair comparison and make setup, data adaptation, and agent integration cumbersome.
- The ecosystem supports creating agents, adding benchmarks, and comparing backbone models across web-agent tasks.
- BrowserGym exposes diverse web-agent benchmarks through one unified interface, while AgentLab supports reproducible, large-scale experimentation and agent development.
- 39.1% success on WorkArena L2 for Claude-3.5-Sonnet exceeded GPT-4o's 8.5%, establishing the experiment's highlighted benchmark comparison.
2 Background and Related Works
Web-agent research spans LLM-based agents, increasingly realistic web benchmarks, and broader UI-agent settings. Existing work motivates standardization through shared evaluation, ecosystem growth, and reproducible experimentation.
- Recent web-agent research evaluates LLMs and VLMs across increasingly diverse benchmarks, from controlled interactions to live websites and enterprise workflows.
- Broader frameworks extend evaluation across web, desktop, and mobile interfaces, while ST-WebAgentBench targets organizational safety compliance.
- Web-agent implementations either manipulate HTML elements directly or use vision-language models to ground screenshots to web elements.
- The literature identifies a need for standardized evaluation with statistical power, open-source ecosystem growth, and reproducible experiments at scale.
3 BrowserGym
BrowserGym provides a unified interface for web agents, combining chat and browser interaction with rich observations, executable actions, and extensible task definitions.
- BrowserGym exposes a unified chat-and-browser environment where agents navigate pages, perform UI actions, extract information, and respond to users.
- Its interaction loop follows a POMDP abstraction and the standard OpenAI Gym API, with a Python interface implemented through Gymnasium, Chromium, and Playwright.
- 3.1 Extensive Observation Space: Observations combine goals or chat history, open-page metadata, structured DOM and accessibility-tree representations, screenshots, and per-element identifiers and visual properties.BrowserGym supplies BrowserGym IDs, bounding boxes, visibility ratios, and Set-of-Marks indicators alongside page representations.
- 3.1 Extensive Observation Space: BrowserGym returns errors from failed actions in the next observation, giving agents immediate feedback for self-correction.
- The action space supports unrestricted Python execution as well as safer custom or high-level action mappings that translate constrained inputs into Playwright code.Raw code is expressive but can expose safety risks, whereas high-level primitives provide more control over interaction behavior.
- 3.3 Extensibility: create your own task: New tasks can be implemented through setup and validation logic, with setup preparing the browser and returning text- or image-containing goals.
4 Unification of Web Agent Benchmarks
BrowserGym unifies diverse web-agent benchmarks behind a common task and evaluation interface. This reduces adaptation effort, supports cross-benchmark comparisons, and accommodates varied task settings and backend requirements.
- Six popular benchmarks are exposed as BrowserGym tasks executable through the same code base and observation/action API.Agents implemented for one benchmark can therefore be evaluated on another without benchmark-specific interface changes.
- A shared interface can accelerate compatible benchmark development by allowing new evaluations to reuse existing web-agent implementations.
- Cross-benchmark evaluation provides wider statistical signals than single-benchmark testing for comparing agents and design choices.
- The unified collection spans synthetic UI tasks, replica websites, enterprise workflows, open-web search, and human interaction traces.
- Benchmark metadata records task attributes, train/test splits, and optional dependencies, while suggested action sets, seeds, and step limits preserve benchmark-specific evaluation practices.
- BrowserGym automates backend resets for benchmarks such as WebArena and VisualWebArena, restoring clean server states between agent evaluations.
5 AgentLab
AgentLab provides tools for running, analyzing, reproducing, and extending BrowserGym experiments. Its workflow supports parallel studies while addressing failures, dynamic environments, task dependencies, and agent development.
- Launching experiments: AgentLab organizes benchmark evaluations through Study objects that launch parallel experiments, save reproducibility information, and relaunch failed tasks.Study.run can relaunch failed tasks up to three times; multiprocessing backends support scalable execution.
- Launching experiments: Task dependencies in (Visual)WebArena constrain parallelism to 2–4 tasks, while make_study enforces sequential agent execution with instance resets.These dependencies prevent one task from corrupting the benchmark instance used by others.
- AgentXRay: AgentXRay displays profiles, goals, observations, actions, and prompts from logged traces for step-by-step inspection of agent decisions.The interface is designed for deeper analysis of individual task behavior.
- Reproducibility: Dynamic software, APIs, websites, and stochastic tasks complicate reproducibility, motivating version tracking, journals, replay tools, and reproduced-result ranges.Software versions, silently changing commercial LLMs, live websites, and task randomness can alter evaluation outcomes.
- Reproducibility: AgentLab records benchmark and software metadata, maintains a reproducibility journal, supports leaderboard comparisons, and can replay action sequences for visual prompt diffs.These features help isolate system effects and monitor benchmark changes over time.
- Extensibility: BrowserGym and AgentLab expose a minimalist agent API, while BaseModelArgs and AbstractChatModel let users configure and plug models into existing agents.AgentLab includes ready-made integrations for OpenAI, Azure OpenAI, and OpenRouter models.
6 Experiments
The experiments use AgentLab’s GenericAgent to evaluate six modern language models across BrowserGym benchmarks, revealing strong cross-benchmark variation in model performance. Claude performs best on most benchmarks, while general-purpose evaluation exposes weaknesses on information-retrieval tasks.
- 6.1 Setup: AgentLab supports large-scale experiment management, while GenericAgent combines modular prompting, multiple observation formats, memory, self-criticism, and retry handling.
- 6.1 Setup: The evaluation reports task success rates and standard errors across benchmarks, with multimodal inputs restricted to models evaluated on VisualWebArena.
- 6.2 Results: Claude achieves the best performance across most benchmarks, with o1-Mini generally ranking second.
- 6.2 Results: Llama-3.1 70B performs close to GPT-4o Mini, while Llama-3.1 405B significantly surpasses GPT-4o Mini on numerous benchmarks.
- 6.2 Results: GPT-4o improves from 23.5% to 31.4% on WebArena and from 3.8% to 8.5% on WorkArena L2 versus earlier results.The paper attributes this change to possible checkpoint improvements or benchmark exposure, but presents both as explanations rather than established causes.
- 6.2 Results: The agent reaches only 6.9% on AssistantBench, below the leaderboard’s 27% score, because it prioritizes broad applicability over information-retrieval specialization.
- 6.3 Errors: The error analysis is qualitative, although full experiment traces are released through AgentXRay for inspection.
7 Discussion
The discussion presents BrowserGym and AgentLab as complementary packages for standardized, reproducible web-agent evaluation and development. It also identifies practical barriers, including environmental nondeterminism, open-web safety and access constraints, concurrency conflicts, and interaction latency.
- 7 Discussion: BrowserGym unifies web-agent benchmarks under one interface, while AgentLab provides agent implementations, model APIs, reproducibility features, trace analysis, and a leaderboard.
- 7 Discussion: A unified codebase enables large-scale experiments across six benchmarks that would be cumbersome with fragmented benchmark implementations.
- 7.1 Limitations: Reproducibility remains limited by stochasticity, localization, browser and operating-system differences, rendering variation, and dynamic web content.
- 7.1 Limitations: Open-web benchmarks face safety risks and robot detection through CAPTCHAs, rate limits, and behavior analysis, with AssistantBench specifically affected.
- 7.1 Limitations: Concurrent agents can interfere when modifying shared database records, so some benchmarks permit only one agent at a time and become slower to evaluate.
- 7.1 Limitations: The browser sync loop can bottleneck tasks requiring rapid sequential actions, reducing efficiency on complex interactions.
- 7.2 Future Work: Future work targets safer environments, real-time interaction, smaller efficient models, stronger visual understanding, inference-time scaling, and reuse of logged traces for fine-tuning.
8 Broader Impact
UI-based agents could transform computer work, but their broader deployment introduces safety, economic, advertising, and policy challenges that remain unresolved.
- More general UI-based agents could automate computer tasks at greater speed, lower cost, and broader expertise, although current benchmarks show this sophistication has not yet been achieved.The paper frames this as a potential future benefit rather than an established capability.
- Prompt-injection attacks could leak sensitive company information, authorize unwanted transactions, or cause other unintended harm as agents become more sophisticated.
- Autonomous interactions with dynamic advertising content could disrupt how ad impressions, clicks, and conversions are measured and monetized.
- The paper argues that society should establish robust policies and frameworks to maximize benefits while minimizing risks and harm.
B Detailed Benchmark Descriptions
BrowserGym covers benchmarks spanning single-page tasks, realistic multi-domain websites, visual interaction, open-domain browsing, and supervised one-step action prediction, while AgentLab provides configurable prompting and agent implementation tools.
- MiniWoB: MiniWoB tasks operate on single HTML pages with JavaScript-based logic and validation, enabling native parallelization without trajectory collisions.
- WebArena and VisualWebArena: WebArena and VisualWebArena use containerized multi-domain websites, with VisualWebArena adding image-based goals and visual matching during validation.Both require environment resets before evaluation, and VisualWebArena includes an additional domain beyond two retained from WebArena.
- WorkArena: WorkArena validates frontend actions and backend state through JavaScript checks and database queries while supporting large-scale parallel evaluation.
- AssistantBench: AssistantBench contains 214 realistic open-domain tasks requiring browsing across more than 525 pages from 258 websites, with both general-assistant and domain-specific subsets.Its test set contains 181 tasks, alongside a 33-task development set.
- WebLINX: WebLINX converts human interaction traces into one-step BrowserGym prediction tasks scored by partial action matching rather than environment trajectories.
- AgentLab and agent implementation: AgentLab agents combine configurable observation and action flags with dynamic prompt components that shrink until prompts fit the token limit, using a unified LLM/VLM API.The framework represents goals, observations, action descriptions, and concrete examples as separate prompt components.
E Benchmark setting
The experiments use benchmark-specific evaluation protocols: seeded tasks for MiniWoB and WorkArena L1, complete benchmark suites for WebArena variants, curricula for WorkArena L2/L3, and fixed test sets for AssistantBench and WebLINX.
- MiniWoB and WorkArena L1 use 5 and 10 seeds per task, respectively, during evaluation.
- WebArena and VisualWebArena are evaluated as complete benchmark suites because they do not provide task seeding.
- WorkArena L2 and L3 use preset curricula to balance task types in each experiment.
- AssistantBench and WebLINX are evaluated using their provided test sets.
F Costs and API Usage
The paper reports experiment costs and token counts across models, with the main cross-model totals excluding VisualWebArena and Llama costs subject to provider and price variation.
- The Llama models were run through OpenRouter, whose provider composition means their prices may have changed since the experiments were conducted.
- Experiment costs and token counts are summarized across models, with totals aggregated over all benchmarks except VisualWebArena.
G Experiment runtime and hardware
The paper reports runtime and environment metrics for benchmark experiments, distinguishing total execution time from environment-only time and noting the effects of API speed and parallelization.
- G Experiment runtime and hardware: Runtime reporting separates cumulative experiment duration, step duration, environment runtime, environment runtime per step, and total environment steps.The metrics distinguish agent-inclusive experiment timing from environment-only timing and count executed steps.
- G Experiment runtime and hardware: API speed and congestion can create substantial discrepancies between total runtime and environment runtime.Slower or overcrowded APIs are identified as a primary source of variation.
- G Experiment runtime and hardware: Experiments used up to 20 parallel jobs, so reported cumulative durations can substantially exceed the actual wall-clock completion time.Parallelization was adjusted according to each benchmark’s computational requirements.
- G Experiment runtime and hardware: Although experiments ran on large compute clusters, the benchmarks could also run locally on standard laptops without significant performance degradation.The main constraints arose from benchmark environments; WebArena used Azure VMs with 8 CPUs and 32GB RAM.
H Results analysis
A task-level comparison shows Claude 3.5 Sonnet completing a challenging WorkArena L2 laptop-ordering task, while GPT-4o fails after an incorrect quantity interaction and does not recover.
- H Results analysis: Claude 3.5 Sonnet completes the challenging WorkArena L2 laptop-ordering task, whereas GPT-4o fails on the same task.The comparison uses the workarena.servicenow.navigate-and-order-standard-laptop-l2 task, which requires ordering six configured laptops.
- H Results analysis: Claude overcomes a checkbox interaction limitation by switching from clicking the checkbox to clicking its label, then completes the task.The agent identifies that the label intercepts pointer events and uses the label’s AXTree identifier instead.
- H Results analysis: GPT-4o attempts to click a menu item instead of using the select_option action, causing the action to fail.The failed interaction occurs while selecting the laptop quantity.
- H Results analysis: After the failed action, GPT-4o continues as though the quantity were six and does not notice that only one laptop is being ordered.The agent proceeds to configure software without correcting the mistaken quantity, leading to task failure.
- H Results analysis: The trajectory figures document Claude’s successful execution and GPT-4o’s failed final steps for the same task and seed.Figure 12 covers Claude’s trajectory, while Figure 13 covers GPT-4o’s failed steps.