Source-linked AI summary
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, Jonathan Berant
TL;DR
The paper asks whether language models and web agents can solve realistic, time-consuming tasks requiring open-web research. It introduces AssistantBench and SPA, finding that current systems remain challenged, while SPA and an ensemble with a closed-book model perform best overall. The analysis identifies open-web navigation as a major challenge.
Problem
Existing evaluations do not fully measure whether agents can handle realistic, time-consuming web tasks, while current models have limited access to web information and may hallucinate.
Method
The paper introduces AssistantBench, a 214-task benchmark, and SPA, a web agent with planning and memory components for multihop information-seeking questions.
Results
No model exceeded 26 points in accuracy; SPA outperformed SEEACT, and an ensemble of SPA with a closed-book model achieved the best overall performance.
Takeaways & Limitations
AssistantBench shows that open-web navigation remains a significant challenge for current systems, while combining web interaction with closed-book modeling yields the best reported performance.
Takeaways & Limitations
The benchmark excludes some realistic time-dependent tasks because their answers cannot be automatically verified, and the reported models rely on GPT4-Turbo and Claude-3.5-Sonnet.
Abstract
from arXiv · showhide
Language agents, built on top of language models (LMs), are systems that can interact with complex environments, such as the open web. In this work, we examine whether such agents can perform realistic and time-consuming tasks on the web, e.g., monitoring real-estate markets or locating relevant nearby businesses. We introduce AssistantBench, a challenging new benchmark consisting of 214 realistic tasks that can be automatically evaluated, covering different scenarios and domains. We find that AssistantBench exposes the limitations of current systems, including language models and retrieval-augmented language models, as no model reaches an accuracy of more than 26 points. While closed-book LMs perform well in terms of accuracy, they exhibit low precision and tend to hallucinate facts. State-of-the-art web agents reach a score of near zero. Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly outperforms previous agents, and an ensemble of SPA and closed-book models reaches the best overall performance. Moreover, we analyze failures of current systems and highlight that open web navigation remains a major challenge.
1 Introduction
AssistantBench evaluates whether web agents can solve realistic, time-consuming information-seeking tasks that exceed the capabilities of systems relying only on parametric knowledge or retrieved evidence. Across 214 diverse tasks, current systems remain limited, while SPA and an ensemble with a closed-book model improve performance.
- Benchmark: AssistantBench contains 214 realistic, time-consuming tasks requiring agents to browse the open web and dynamically interact with relevant pages.The tasks are based on human information needs and span diverse scenarios and domains.
- Results: 26 points was the highest accuracy achieved by any evaluated model on AssistantBench.The benchmark evaluates closed-book models, retrieval-augmented models, and web agents.
- Results: SPA outperformed SEEACT by about 7 points, answered twice as many questions, and achieved 5 points higher precision.An ensemble combining SPA with a closed-book model achieved the best overall performance.
- Error analysis: Web-agent errors most often involved incorrect navigation or loops, while closed-book and retrieval-augmented models commonly hallucinated facts or failed to retrieve relevant information.Navigation errors accounted for 60% and 37% of errors for SEEACT and SPA, respectively; hallucinated facts accounted for 85% of closed-book-model errors, and retrieval failures for 80% of retrieval-augmented-model errors.
- Approach: SPA combines planning and memory components to address multihop information-seeking questions with a web agent.The system is introduced as a new agent alongside the benchmark.
2 Background
AssistantBench extends web-agent evaluation beyond simplified single-site or sandbox settings by targeting realistic, time-consuming tasks requiring multiple websites and broad domain coverage. Its design aims to measure agents’ ability to assist humans on challenging open-web tasks.
- Prior evaluation: Previous web-agent benchmarks commonly evaluated interaction with a single website or operation within a sandbox environment.AssistantBench is positioned as a broader evaluation setting than these environments.
- Benchmark scope: AssistantBench focuses on realistic, time-consuming tasks that require multiple websites and cover a wide range of domains.The benchmark is presented as a measuring stick for web agents’ ability to assist humans.
3 ASSISTANTBENCH
AssistantBench is constructed from human, crowdworker, and domain-expert tasks, filtered for realistic, time-consuming, automatically verifiable web use. Its 214 tasks span diverse domains and websites, with answers evaluated through structured string, numeric, and dictionary metrics.
- Task criteria: Tasks were selected to be realistic, take at least several minutes, and have closed-form answers that can be automatically verified.The benchmark excludes answers likely to change quickly and includes manual verification by two authors.
- Data collection: The dataset combines seed tasks from participants, crowdworker-expanded variants, and professional tasks contributed by domain experts.These are the three main construction stages described for AssistantBench.
- Task distribution: 214 tasks span diverse users, domains, and websites, with answers distributed across 525 webpages on 258 websites.The benchmark includes 42 expert tasks and covers more than 15 domains in the expert set.
- Automatic Evaluation: Automatic evaluation supports string, numeric, and dictionary answers using F1-based comparisons and an order-of-magnitude score for numbers.Dictionary outputs are compared by matching values under identical keys, using the metric appropriate to each value type.
- Automatic Evaluation: The benchmark measures answer rate, precision, and exact match in addition to its primary answer-scoring procedures.Answer rate captures whether a model generated an answer, while precision measures accuracy among non-abstained answers.
4 SPA: See-Plan-Act
SPA extends SEEACT with planning, memory, and open-web navigation actions to handle tasks requiring planning and reasoning.
- SPA is introduced as a web agent built to solve tasks in ASSISTANTBENCH.
- SEEACT examines screenshots, describes the next action, and grounds it to an HTML element.
- SPA equips SEEACT with planning and memory components that support execution re-planning and information transfer between steps.These components are implemented through prompting.
5 Experiments
The experiments evaluate language models, retrieval-augmented models, and web agents on ASSISTANTBENCH and FANOUTQA. SPA generally outperforms SEEACT, while closed-book models achieve stronger accuracy on ASSISTANTBENCH and SPA-based ensembles perform best in reported comparisons.
- Experimental setup: The evaluation uses closed-book, retrieval-augmented, and web-agent baselines, including zero-shot, one-shot, and search-based prompting configurations.Web agents are limited to 30 execution steps, and SPA operates over multiple websites.
- Experimental setup: Fallback ensembles answer with CB-1S when a web agent abstains, combining web navigation with closed-book answering.RALM-1S→CB is one example of this fallback design.
- Experimental setup: FANOUTQA supplements ASSISTANTBENCH with multi-page information-seeking tasks, retaining 31 dictionary-answer tasks that CB-1S does not solve perfectly.The transferred tasks are used as an additional development set for evaluating SPA.
- ASSISTANTBENCH: No system exceeds 25% accuracy on ASSISTANTBENCH with GPT-4-T, while SPA outperforms SEEACT in answer rate and precision.Closed-book models achieve higher accuracy than retrieval-augmented models and web agents, mainly because they answer more often.
- ASSISTANTBENCH: SPA→CB achieves the best reported accuracy of 26.4 with Claude-3.5-Sonnet, while SPA surpasses SEEACT by 10 points with better precision and answer rates.The development-set results show similar trends.
- FANOUTQA: SPA outperforms SEEACT by 22.5 points on FANOUTQA, answers 45% more often, and reaches the exact reference answer on 3/31 tasks.SPA also has higher or similar precision relative to the other evaluated models.
6 Analysis
The analysis shows that long web interactions, navigation failures, retrieval failures, and hallucinations constrain current systems. Performance is especially weak on harder tasks and on tasks requiring sustained web navigation.
- Trajectory length: SPA and SEEACT reach near-zero accuracy below five or above fifteen actions, peaking at around ten steps.Because ASSISTANTBENCH tasks require long web interactions, this trajectory-length pattern helps explain their difficulty.
- Web-agent errors: Answer-generation failures account for 80% of SPA errors and 97% of SEEACT errors, with navigation errors the leading cause for both agents.Navigation errors constitute 36.7% of SPA errors and 63.6% of SEEACT errors.
- Model errors: Closed-book models hallucinate in 85% of incorrect answers and provide outdated answers in the remaining 15%.When they abstain, they often propose a solution plan instead of answering.
- Model errors: Retrieval failures cause 80% of retrieval-augmented-model errors, including tool-related failures, partial information, and irrelevant retrieval.Tool-related failures occur in 38.5% of retrieval errors and partial-information failures in 11.5%.
- Chatbot analysis: CHATGPT errs on more than 90% of development-set tasks, commonly over-relying on search results or hallucinating during code execution.The code-interpreter hallucination failure occurs in about 15% of cases.
- Task time-dependence: Static and stable task splits remain below 21 accuracy points for every model, supporting their use as evaluation resources in coming years.Across splits, CB-1S remains strongest and SPA outperforms SEEACT.
7 Related Work
Related work covers benchmarks for web browsing, multimodal web understanding, and broader agent interaction. The paper focuses on prompted foundation-model web agents while distinguishing them from specialized interaction models and tool-using agents.
- Web-agent benchmarks: Existing benchmarks study web browsing with dialogue, annotation instructions, and multimodal webpage understanding such as OCR.Other datasets evaluate interaction with operating systems and mobile applications.
- Web agents: Prior web-agent research includes prompted foundation models and specialized models trained for web interaction.The paper’s focus is prompted foundation models because of their recent success.
- Broader agent research: Related agent research also examines predefined tools, code generation, games, embodied robots, and broader evaluations of language-model agents.These areas extend beyond direct web interaction.
8 Conclusion
ASSISTANTBENCH introduces a challenging benchmark of realistic, time-consuming web tasks and SPA, an improved web agent. Results show that all evaluated models struggle, while SPA and an ensemble with a closed-book model improve performance.
- ASSISTANTBENCH contains 214 realistic, time-consuming tasks spanning diverse scenarios and domains over more than 525 pages and 258 websites.
- No evaluated model reaches more than 26 accuracy points, and closed-book models have the best accuracy but low precision and factual hallucinations.
- SPA improves over current web agents, while an ensemble combining SPA with a closed-book model achieves the best overall performance.
- The benchmark’s analysis identifies web navigation as a significant challenge for current systems.
9 Limitations
The benchmark excludes some realistic tasks because their answers are time-dependent or difficult to verify automatically, and its task collection remains limited in scale and participant coverage.
- Some realistic time-dependent tasks were excluded because their answers cannot be automatically verified.
- Future work could extend the benchmark to time-dependent tasks using specialized evaluation models in a multimodal web setting.
- The task collection is limited to participants the authors could contact, although future work could further scale the collection method.
- ASSISTANTBENCH is smaller than some recent web-agent benchmarks, despite the authors’ argument that small, high-quality benchmarks can remain useful.
- The reported models rely on GPT4-Turbo and Claude-3.5-Sonnet, limiting reproducibility and leaving open-source web-agent performance less represented.
10 Ethical Implications and Broader Impact
Web agents could assist with useful tasks but also create social, safety, privacy, transactional, and environmental risks. Safe deployment therefore remains important as their capabilities improve.
- Web agents have potential to assist humans with useful tasks such as those represented in ASSISTANTBENCH.
- Web agents may affect the job market or help malicious users spread fake news across the web.
- Safe deployment must prevent unintended personal-data sharing and undesired transactions, while accounting for high model-call costs.
A Appendix
The appendix provides additional technical details and examples related to the paper’s benchmark and methods.
- The appendix introduces additional examples and technical details.
A.1 Detailed Comparison to Recent Benchmarks
The appendix details how AssistantBench tasks and evaluation settings were constructed, alongside implementation choices and analyses of model behavior. It also documents recurring navigation failures and hallucinations in evaluated systems.
- Experimental setup: The evaluation compares closed-book, retrieval-augmented, and web-agent systems using bounded retrieval and web-execution procedures.Retrieval-augmented models use Google Search with a ten-search limit, while SPA adds planning, memory, and open-web actions such as SEARCH and GOBACK.
- Web-agent implementation: SPA’s implementation uses screenshots and Playwright-based browsing, making runs vulnerable to technical failures such as failed screenshots and crashes.The agents operate over the open web and rely on multimodal model inputs from rendered pages.
- A.7 FANOUTQA: 72 seed tasks were expanded and filtered into 31 FANOUTQA development tasks after removing memorized or ambiguous cases.The broader benchmark construction involved task validation, template-based expansion, and manual refinement.
- Failure analysis: Web-navigation errors dominate agent failures, while closed-book models frequently hallucinate facts in their answers.Incorrect trajectories and loops account for 60% of SEEACT errors and 37% of SPA errors, while hallucinated facts account for 85% of closed-book errors.