Beyond Browsing: API-Based Web Agents

• APIs let machines communicate directly with web-service backends using machine-friendly formats, but whether agents can effectively use them for real-world online tasks was previously unstudied.Source: §1 · p. 1
• API-Based Agents bypass web GUIs by directly interacting with web services through API calls.Source: §1 · p. 1
• Hybrid Agents dynamically interleave API calls and browsing because some websites have limited API support.Source: §1 · pp. 1–2
• 38.9% performance for the Hybrid Agent exceeded the API-Based Agent’s 29.2% and the Browsing Agent’s 14.8% on WebArena.Source: Figure 1 · p. 2
• The API-Based Agent outperformed browsing by around 15% on average, while the Hybrid Agent further improved accuracy by 10% over API-only interaction.Source: §1 · p. 2
• The results suggest that APIs are often preferable or complementary to direct interaction with human-designed graphical interfaces.Source: §1 · p. 2

2.1 The Web Browsing Task

• WebArena simulates diverse real-world activities across GitLab, Reddit, e-commerce sites, and mapping services, evaluating output correctness, website-state changes, or navigation success.Source: §2.1 · p. 2

2.2 A Baseline Web Browsing Agent

• The Browsing Agent operates through an accessibility tree with simulated clicks, form inputs, navigation, and action history.Source: §2.2 · pp. 2–3
• The text-only benchmark choice reflects limited API support for image handling in VisualWebArena.Source: §2.2 · p. 2
• Accessibility-tree agents struggle with complex GUIs, dynamically loaded content, and information not immediately visible in the tree.Source: §2.2 · p. 3
• 15 browsing steps failed to retrieve SaptakS’s commits because the relevant entries required repeated scrolling through many other commits.Source: §2.2 · p. 3, Figure 2 · p. 3

3.1 APIs and API Documentation

• APIs use standardized HTTP requests such as GET and POST to return structured data, including JSON objects, through documented endpoints.Source: §3.1 · p. 3

3.2 Obtaining APIs for Agents

• API acquisition is classified by whether websites provide sufficient APIs and documentation, APIs without documentation, or insufficient APIs.Source: §3.2 · p. 4
• When documentation is missing, the method extracts API calls from frontend or backend code and uses GPT-4o to generate documentation with parameters, outputs, and examples.Source: §3.2 · p. 4
• When APIs are insufficient, custom APIs may be created; this was necessary for one of five WebArena websites.Source: §3.2 · p. 4

3.3 Using APIs in Agents

• Small API sets are supplied through full documentation in the prompt, while larger sets use staged retrieval to limit prompt size.Source: §3.3 · pp. 4–5
• For large API sets, the agent first receives endpoint summaries and can then retrieve detailed documentation with get_api_documentation.Source: §3.3 · p. 5

4 Hybrid Browsing+API Calling Agents

• Limited or poorly documented APIs can force agents to rely on browsing, motivating a hybrid approach.Source: §4 · p. 5
• The Hybrid Agent dynamically chooses API calls, web browsing actions, or clarification at each step within a fixed task budget.Source: §4 · p. 5
• The agent is intended to use well-documented APIs efficiently and rely more on browsing when API support or documentation is limited.Source: §4 · p. 5
• Its prompt combines API-call descriptions, detailed browser-action specifications, and the history of prior steps.Source: §4 · p. 5

5.1 Dataset Description

• WebArena evaluates agents across Gitlab, Map, Shopping, Shopping Admin, Reddit, and multi-site tasks representing varied online interactions.The benchmark is used to assess both API calling and web browsing across diverse web settings.Source: §5.1 · pp. 5–6

5.2 API Statistics for WebArena Sites

• API support is categorized as good, medium, or poor according to endpoint availability, functionality, and documentation.These properties are presented as important to agent efficiency and flexibility.Source: §5.2 · p. 6, Table 1 · p. 6

5.2.1 Good API Support

• Gitlab provides 988 endpoints and Map provides 53 well-documented endpoints covering most essential use cases.Gitlab offers broad functional coverage, while Map has fewer but well-documented endpoints.Source: §5.2.1 · p. 6

5.2.2 Medium API Support

• Shopping and Shopping Admin share 556 APIs with detailed documentation, but missing wishlist functionality still requires browsing.API calling is therefore useful for common shopping tasks without being exhaustive.Source: §5.2.2 · p. 6

5.2.3 Poor API Support

• Reddit has only 31 endpoints, no documentation, and missing functions such as post search, making it the least API-friendly site.These gaps motivate combining API calls with browsing for Reddit tasks.Source: §5.2.3 · p. 6

5.3 API Implementation Details

• For small API sets, the authors provide full documentation directly, while larger API sets use a two-stage documentation retrieval process.Map documentation comes from public materials, Reddit documentation is generated with GPT-4o, and Gitlab, Shopping, and Shopping Admin use existing README or OpenAPI documentation.Source: §5.3.1 · p. 6, §5.3.2 · p. 6

5.4 Evaluation Framework

• OpenHands supplies the evaluation framework, with browser-based BrowsingAgent and CodeAct-based API execution as foundations for the proposed agents.The framework records prior actions and supports browser interaction through Chromium and Playwright.Source: §5.4 · pp. 6–7
• The API-Based Agent adds API-specific instructions to CodeAct, while the Hybrid Agent can freely transition between API calls and Playwright browsing using current and prior state.All three agents use GPT-4o as the base language model.Source: §5.4 · p. 7

6.1 Main Results

• The API-Based Agent achieved higher scores than the Browsing Agent on most websites, while browsing performed best on Shopping Admin and Map but struggled elsewhere.The API-Based Agent’s specialized API-calling design enabled task completion without browsing.Source: §6.1 · p. 7
• The Hybrid Agent outperformed both Browsing and API-Based Agents on all websites by interleaving API calls with browsing when endpoints were unavailable or incomplete.It used both browsing and API actions in 77.7% of WebArena tasks, with API-only and API-plus-browsing choices showing higher accuracy.Source: §6.1 · p. 7, Table 3 · p. 8

6.2 Does API Quality Matter?

• High-quality, well-documented APIs improved agent performance, whereas incomplete functionality or ambiguous documentation caused missing tasks, parameter errors, and lower effectiveness.Gitlab and Map had the strongest API support and highest completion accuracies; Reddit initially had limited API availability.Source: §6.2 · p. 8
• Adding 13 APIs to Reddit’s initial 18, including post voting, markedly improved the API-Based Agent’s performance.The intervention addressed missing forum functionality and linked API availability with average agent performance.Source: §6.2 · p. 8, Table 4 · p. 8

6.3 Error Analysis

• Among 100 sampled tasks, 33% were solved with API calling alone, 50% were unsolvable solely through APIs, and 66% of API-solvable tasks were completed correctly.Another 6% were incorrect because of task understanding and 11% because of API-calling errors such as malformed or incorrect inputs.Source: §6.3 · p. 8, Figure 4 · p. 8
• API-based execution required an average of 2.1 API calls for API-solvable tasks, while the agent averaged 7.8 total steps largely for documentation, error recovery, and output verification.Source: §6.3 · p. 8

6.4 Case Studies

• Case studies showed the Hybrid Agent succeeding where API-only and browsing-only approaches failed, while the API-Based Agent excelled when structured API retrieval was simpler than web navigation.The Hybrid Agent handled a review-deletion task by combining interfaces; the API-Based Agent retrieved GitLab contributor information through a dedicated endpoint.Source: §6.4 · p. 9

• API-Based Agents outperform Browsing Agents, especially on websites with good API support, while Hybrid Agents empirically outperform agents using only one interface.Source: §7 · p. 9

API Availability

• Inconsistent API availability and coverage can leave API-Based Agents unable to complete tasks requiring unsupported functionality.Even GitLab may lack support for retrieving a user’s official username from a displayed name.Source: §8 · p. 10

• Automatic API mining and related methods could generate APIs for unsupported tasks, reducing manual creation and expanding agent applicability.Source: §8 · p. 10

Incorporating APIs

• API-Based Agents require manually integrating API documentation into each website’s action space, increasing complexity and limiting scalability compared with browsing-only agents.Source: §8 · p. 10

Evaluation Benchmark

• The evaluation uses only WebArena, whose number and variety of tasks may be limited despite its realistic and diverse challenges.Other benchmarks offer alternative evaluation platforms, but several lack API-calling support or image interaction relevant to this study.Source: §8 · p. 10

Web Navigation Agents

• Web navigation agents use browsing actions and are effective when tasks require human-like interaction with user interfaces.WebArena provides complex, realistic tasks for evaluating such agents.Source: §A.1 · p. 13
• Hybrid Agents extend browsing-only work by integrating structured API calls with web navigation to enhance task performance.Source: §A.1 · p. 13

Code Generation Agents and Tool Usage

• API-using and code-generation agents target structured data exchanges and can perform operations more efficiently than traditional web navigation agents.Source: §A.1 · p. 13
• Hybrid Agents dynamically switch between APIs and browsing, enabling them to outperform API-only agents on tasks requiring web navigation.Source: §A.1 · p. 13
• The paper presents the first comparative study of API and Browsing Agents on the same websites and finds APIs often more efficient when available.Source: §A.1 · p. 13

A.2 WebArena Tasks

• WebArena reproduces commonly used websites with real-world data imported through open-source frameworks.Source: §A.2 · p. 14
• WebArena tasks span GitLab, maps, shopping, shopping administration, Reddit, and multi-website interactions.Source: §A.2 · p. 14
• GitLab tasks cover project management and version control, including issues, merge requests, and repository creation.Source: §A.2 · p. 14
• Map tasks require navigation, trip planning, and distance queries using map-based data.Source: §A.2 · p. 14
• Shopping tasks include product search, cart additions, transactions, delivery-address changes, and store administration.Source: §A.2 · p. 14
• Reddit tasks involve posting, voting, and retrieving information from discussion threads, while multi-website tasks require simultaneous interaction across two sites.Source: §A.2 · p. 14

A.3 Obtaining APIs of WebArena Websites

• WebArena API access spans GitLab’s 988 REST endpoints, Map’s 53 endpoints, Shopping’s 556 Adobe Commerce endpoints, and Reddit’s 31 self-implemented endpoints.Most WebArena tasks are covered by GitLab’s APIs; Map combines Nominatim, OSRM, and OpenStreetMap APIs, while Shopping supports common tasks such as purchasing products and searching categories.Source: §A.3 · pp. 14–15

A.4 Additional Analysis

• API-Based and Hybrid Agents achieve higher accuracy than Browsing Agents despite higher costs, while the API-Based Agent uses the fewest steps and the Hybrid Agent the most.Browsing is cheaper because its prompts and action space are smaller; API prompts can be much longer, such as GitLab’s 988 APIs, while shorter Reddit documentation lowers cost.Source: §A.4 · pp. 15–16, Figure 7 · p. 15, Figure 8 · p. 15

A.5 API-Based Agent Prompt

• The API-Based Agent prompt enables Python requests-based API calls and instructs agents to inspect responses, handle headers and pagination, and retrieve detailed endpoint documentation before use.The prompt also directs agents to resolve ambiguity independently, avoid clarification requests, and return concise task-completion responses.Source: §A.5 · pp. 16–17

A.6 Hybrid Agent Prompt

• The Hybrid Agent prompt combines API calling with browser interaction, requiring agents to try APIs first and browse when APIs are unavailable or to verify API-derived results.It provides both Python execution and browser commands for navigation, scrolling, form entry, selection, clicking, keyboard input, dragging, and file upload.Source: §A.6 · pp. 17–19
• Before making an API call, agents must retrieve its detailed documentation, including input parameters and example JSON outputs.The prompt illustrates this with get_api_documentation for a GitLab commits endpoint.Source: §A.6 · p. 20
• The prompt requires agents to try APIs first and use web browsing when no useful API is available.This establishes an API-first strategy with browsing as a fallback.Source: §A.6 · p. 20
• After using APIs, agents must browse to content that can verify whether the API-derived results are correct.This adds an explicit web-based verification step to API execution.Source: §A.6 · p. 20