PaperlayerView exact revision on arXiv

Source-linked AI summary

BEARCUBS: A benchmark for computer-using web agents

Yixiao Song, Katherine Thai, Chau Minh Pham, Yapei Chang, Mazin Nadaf, Mohit Iyyer

arXiv:2503.07919v3cs.AIcs.CLcs.LG

TL;DR

Web agents can interact with live webpages, but evaluating their real-world browsing capabilities remains challenging. BEARCUBS is a 111-question benchmark built around live web content and diverse multimodal interactions. Humans outperform the tested agents, with ChatGPT Agent the strongest computer-using system reported.

  • Problem

    Existing web-agent benchmarks rely on simulated environments, approach performance saturation, or cover only limited multimodal abilities.

    Source: §1 · pp. 1–2

  • Method

    BEARCUBS evaluates information-seeking through live-web questions with short answers and human-validated trajectories, filtering multimodal questions that admit text-based workarounds.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Humans achieve 84.7% accuracy, compared with 65.8% for ChatGPT Agent and 23.4% for Operator; humans outperform agents on text and multimodal tasks.

    Source: §1 · p. 2, §5 · p. 6

  • Takeaways & Limitations

    The authors identify trajectory transparency, source credibility, multimodal capabilities, and planning as directions for future agent development.

    Source: §8 · p. 10

    Thirteen multimodal questions were removed after Deep Research found text-based workarounds, underscoring the need to filter such questions.

    Source: §2 · p. 3

Abstract

from arXiv · show

Modern web agents possess computer use abilities that allow them to interact with webpages by sending commands to a virtual keyboard and mouse. While such agents have considerable potential to assist human users with complex tasks, evaluating their capabilities in real-world settings poses a major challenge. To this end, we introduce BEARCUBS, a "smallbut mighty" benchmark of 111 information-seeking questions designed to evaluate a web agent's ability to search, browse, and identify factual information from the web. Unlike prior web agent benchmarks, solving BEARCUBS requires (1) accessing live web content rather than synthetic or simulated pages, which captures the unpredictability of real-world web interactions; and (2) performing a broad range of multimodal interactions (e.g., video understanding, 3D navigation) that cannot be bypassed via text-based workarounds. Each question in BEARCUBS has a corresponding short, unambiguous answer and a human-validated browsing trajectory, allowing for transparent evaluation of agent performance and strategies. A human study confirms that BEARCUBS questions are solvable but non-trivial (84.7% human accuracy), revealing domain knowledge gaps and overlooked details as common failure points. We find that ChatGPT Agent significantly outperforms other computer-using agents with an overall accuracy of 65.8% (compared to e.g., Operator's 23.4%), showcasing substantial progress in tasks involving real computer use, such as playing web games and navigating 3D environments. Nevertheless, closing the gap to human performance requires improvements in areas like fine control, complex data filtering, and execution speed. To facilitate future research, BEARCUBS will be updated periodically to replace invalid or contaminated questions, keeping the benchmark fresh for future generations of web agents.

1 Introduction

BEARCUBS evaluates whether computer-using agents can find factual information through real-world web interactions, including multimodal tasks. Its results show that agents still trail human performance.

1 Introduction

  • BEARCUBS contains 111 question-answer pairs designed to test web agents in multimodal online environments.The benchmark asks agents to search, browse, and identify factual information from the web.Source: §1 · p. 1
  • Existing benchmarks use synthetic or simulated environments, approach performance saturation, or cover only limited multimodal abilities.Some can be solved from HTML alone, while others focus on specific capabilities such as map navigation or image processing.Source: §1 · pp. 1–2
  • Each BEARCUBS question has a short, unambiguous answer and a human-validated browsing trajectory for comparing agent paths.The authors filter multimodal questions that can be solved through text-based workarounds and plan to replace invalid or contaminated questions.Source: §1 · p. 2
  • Humans reach 84.7% accuracy, while ChatGPT Agent achieves 65.8% and Operator 23.4% on BEARCUBS.The authors also report that Deep Research, which lacks computer-use capabilities, reaches 36.0% accuracy through guessing.Source: §1 · p. 2

2 Challenges of evaluating modern web agents

Evaluating computer-using agents on the live web presents challenges in maintaining valid questions, preventing workarounds, testing diverse interactions, and running evaluations efficiently.

Web contamination

  • Live-web benchmark validity requires ongoing maintenance because published questions may become indexed or contaminated.The authors plan periodic additions and removals to preserve BEARCUBS’s relevance; public release does not fully remove contamination risk, while rerunning agents is costly and time-consuming.Source: §2 · p. 2

Workarounds

  • A BEARCUBS example shows ChatGPT Agent succeeding where Operator fails to click through to the critical document.The figure illustrates how a missed interaction can prevent an agent from reaching the source needed to answer.Source: Figure 1 · p. 3
  • Thirteen multimodal questions were removed after Deep Research found text-based workarounds, despite efforts to make questions adversarial to Google Search.The authors recommend rigorously filtering questions that can bypass the intended interaction mode.Source: §2 · p. 3

Maximizing interaction diversity

  • Only 58.2% of freelancer-written questions were accepted as the benchmark sought diverse interactions across tasks and domains.The authors describe this diversity as requiring greater creativity from question writers.Source: §2 · p. 3

Evaluation is slow

  • Manual evaluation took near five minutes per question on average, so computer-using agents generally faced a 15-minute limit.The authors cite lack of API access and agents getting stuck in repetitive loops as evaluation obstacles.Source: §2 · p. 3

3 Building the BEARCUBS benchmark

BEARCUBS is a live-web benchmark of 111 information-seeking questions, designed around concise answers, diverse interactions, and human-validated paths.

Dataset overview

  • BEARCUBS includes 111 information-seeking questions requiring live website interaction, with text-based and multimodal questions.Multimodal tasks include image, video, audio, or real-time interaction; the benchmark is intended to be updated with new questions and filtered for contamination.Source: §3 · pp. 3–4

Criteria for valid QA pairs

  • Questions are designed to provide sufficient but minimal information, have concise and unambiguous answers, resist Google Search, and use publicly accessible answers..Source: §3 · p. 4

Collection and validation process

  • Each question has a gold answer, a human-validated trajectory, and a list of visited websites, with quality control for quality and workaround prevention.Authors wrote 65 questions, while trained freelancers contributed the rest.Source: §3 · p. 4

Prioritizing reliable interactions

  • Most questions specify an intended source, such as a particular book or video, so agents can be assessed on retrieving information from that source.Filtering out questions with workarounds was a top priority during benchmark creation.Source: §3 · p. 4

Diverse interaction types

  • BEARCUBS spans text tasks such as database navigation and multimodal tasks such as interpreting videos and virtual tours.The text-based questions test handling text tasks, while multimodal questions assess adaptability to real-world dynamics.Source: §3 · p. 4

Website diversity

  • 108 unique top-level URLs appear in viable trajectories, which average 6.1 steps and 3.4 webpages per question.These counts use human-written trajectories, and the URL count excludes Google Search visits.Source: Table 1 · p. 4, §3 · p. 5

4 Experiments

The experiments measure human performance and benchmark seven commercial web agents, using answer, timing, and trajectory records to assess performance and behavior.

4.1 Measuring human performance

  • The human study asks browser users unfamiliar with each question to solve it, while recognizing that valid answer paths may still be difficult to find.Annotators report answers, time, dead ends, challenges, and perceived difficulty; they may abandon after 15 minutes.Source: §4.1 · p. 5

4.2 Benchmarking web agents

  • The agent benchmark includes seven commercial agents, spanning advanced search systems and computer-using agents.Five baselines test whether zero-shot prompting or simple search augmentation can solve BEARCUBS; the setup also includes Perplexity sonar-pro.Source: §4.2 · pp. 5–6
  • Models exhibit low accuracy variance on BEARCUBS, supporting the benchmark’s reported reliability..Source: §4.2 · p. 5
  • For computer-using agents, the setup minimizes user intervention and records answers, elapsed time, and solving trajectories.Sessions end when an agent answers, abstains, or enters a dead loop without progress.Source: §4.2 · p. 6
  • Agents’ answers count as correct only when they unambiguously entail the gold answer; tentative guesses are not accepted.The agents are manually run and evaluated because their setups and agent-user interactions vary.Source: §4.2 · p. 6

5 Results

Humans outperform agents overall, although ChatGPT Agent ranks first among computer-using agents. Agent results also show low multimodal accuracy and difficulty with complex filtering, while human errors often concern details or domain knowledge.

5.1 Human performance results and analysis

  • Humans achieved 84.7% accuracy despite rating 50.5% of questions moderate-to-high difficulty, averaging 1.5 dead ends and 4 minutes 46 seconds per question.Source: §5.1 · p. 6

5 Results

  • 65.8% accuracy for ChatGPT Agent remained below humans’ 84.7%, while exceeding Operator by 42.4 percentage points and OpenAI Deep Research by 29.8 points.Source: §5 · p. 6, Table 2 · p. 7

5.1 Human performance results and analysis

  • Human errors most often involved overlooked question or answer details, followed by topic understanding; complex data filtering and domain knowledge also made questions challenging.Source: §5.1 · p. 7

5.2 Agent performance results

  • Agents found correct answers faster than humans but had significantly lower accuracy; unanswered Proxy and Operator questions increased notably when runtime exceeded 10 minutes.Source: §5.2 · pp. 7–8
  • 8.1% overall accuracy made DeepSeek-R1 the strongest zero-shot baseline, while Perplexity sonar-pro led search-augmented models at 5.4%, below DeepSeek-R1.Source: §5.2 · p. 7
  • 54.5% multimodal accuracy for ChatGPT Agent remained poor despite gains in computer-use skills, while annotators generally found multimodal questions easier.Source: §5.2 · p. 8
  • 76.8% text-based accuracy for ChatGPT Agent exceeded OpenAI Deep Research’s 60.7%, but ChatGPT Agent struggled with complex interactive-database filtering.Source: §5.2 · p. 8
  • 36.0% accuracy placed OpenAI Deep Research second, largely through text questions; 37.5% of its correct answers relied on secondary sources or were ungrounded.Source: §5.2 · p. 8

6 Discussion

BEARCUBS exposes challenges in agent transparency, source credibility, multimodal interaction, and planning. ChatGPT Agent approaches human-level performance on text-based questions, but multimodal performance and interaction skills remain areas for improvement.

6 Discussion

  • Trajectory reports vary from 69.8 steps per question for Grok 3 DeepSearch to 6.2 for Proxy, while Deep Research provides only top-level URLs.Both excessive detail and vague or overly concise reports make failure points harder to evaluate.Source: §6 · pp. 8–9
  • 38.5% of OpenAI Deep Research’s correct answers rely on an unreliable source or no source, showing why evaluation should assess credibility as well as correctness.Operator also disregards a source specified in a question in favor of an alternative.Source: §6 · p. 9
  • Brief summaries or obscured URLs may be an intentional anti-distillation strategy, though the authors note poor agent performance undermines its capability-preserving purpose.The authors present this explanation as plausible, not certain.Source: §6 · p. 9
  • ChatGPT Agent approaches human-level performance on text-based questions, but agents still struggle with multimodal interactions and restricted content access.The authors recommend improving mouse and keyboard control and strategies for restricted content.Source: §6 · p. 10
  • Agents repeat failed actions and visit irrelevant pages; the authors link unfocused execution plans to irrelevant information and impaired decision-making.They suggest more structured planning could improve search efficiency and decision-making.Source: §6 · p. 10

7 Related work

Related benchmarks assess low-level web operations, text-based or multimodal web tasks, and agent capabilities beyond web use. BEARCUBS is closest to realistic real-web evaluation, with a broader emphasis on multimodal interaction than AssistantBench.

7 Related work

  • WebSuite and WebGames assess fundamental web operations and offer granular insight into basic web UI skills, while focusing on low-level tasks.The cited works are WebSuite (Li & Waldo, 2024) and WebGames (Thomas et al., 2025).Source: §7 · p. 10
  • Web-agent benchmarks range from text-based information sources to multimodal tasks; AssistantBench uses the real web but intentionally limits interactions such as video understanding.BEARCUBS emphasizes diverse multimodal capabilities.Source: §7 · p. 10
  • Non-web evaluations cover scientific discovery, software engineering, open-ended real-computer tasks, safety, human-agent interaction, and hard-to-find information.Examples include ScienceAgentBench, SWE-Bench, OSWORLD, ST-WebAgentBench, CowPilot, and Wei et al. (2025).Source: §7 · p. 10

8 Conclusion

BEARCUBS evaluates factual information seeking on the real web through multimodal interactions. The authors find that agents lag behind humans, particularly on multimodal tasks, and identify benchmark scope and trajectory comparability as limitations.

8 Conclusion

  • BEARCUBS evaluates web agents’ ability to find factual information on the real web through multimodal interactions, and the authors report that agents lag behind humans, especially on multimodal tasks.The paper highlights trajectory transparency, source credibility, multimodal capability, and planning as development directions.Source: §8 · p. 10

A Limitations

  • Each BEARCUBS question has one short answer, limiting coverage of realistic questions with no answer, multiple answers, or long-form responses.The authors state that such questions would also call for credible sources and source-quality evaluation.Source: A Limitations · p. 14
  • The benchmark’s limited size constrains its multilingual focus, and inconsistent trajectory detail makes direct agent comparisons challenging.The authors encourage systematic studies across cultures and languages and improved agent transparency.Source: A Limitations · pp. 14–15

B Data creation workflow

The workflow selects questions for concise, unambiguous answers, resistance to search or text-only workarounds, and public accessibility, then validates questions and trajectories. The appendices also describe human annotator recruitment and point to baseline settings.

B Data creation workflow

  • Questions are designed to have unique, concise answers, resist Google Search snippets and text-only workarounds for multimodal tasks, and use publicly accessible sources without login.Answers cannot be lists or sets, and paraphrases are not accepted as correct.Source: B Data creation workflow · p. 15
  • At least two authors review each question against the four criteria, its viable trajectory, and visited links; questions with text-only workarounds are removed.The figure caption also says multimodal questions are verified to require multimodal interaction.Source: B Data creation workflow · p. 15, Figure 3 · p. 15

C Human annotator recruitment

  • Human annotators include native-speaker volunteers for Arabic and Chinese, and hired annotators for Hindi, German, and Finnish.Annotators receive $2.5 per question and a $1 bonus for each correct answer.Source: C Human annotator recruitment · p. 15

D Baseline prompt

  • Table 4 provides the hyperparameters and prompt for the Google-search-augmented baselines.The supplied text identifies the table’s scope but does not give its settings.Source: D Baseline prompt · p. 16, Table 4 · p. 16

E Human performance results

The human study reports 84.7% accuracy on BEARCUBS, with annotators generally finding answers smoothly and quickly. Table 5 also reports dead ends, time, and perceived difficulty.

E Human performance results

  • Humans achieved 84.7% accuracy on BEARCUBS, finding answers generally smoothly with 1.5 dead ends on average and quickly, in an average completion time of 4 minutes 46 seconds.Source: Table 5 · p. 16
  • Annotators averaged 1.5 dead ends per question and took 4 minutes 46 seconds on average.Table 5 reports dead ends and time alongside perceived difficulty.Source: Table 5 · p. 16, E Human performance results · p. 16
  • Table 5 marks questions abandoned after 15 minutes as “None” and reports annotators’ subjective perceived difficulty.Source: Table 5 · p. 16, E Human performance results · p. 16

F Human error examples

The appendix details human error patterns, evaluator settings, and agent performance analyses. It also discusses BEARCUBS’s size and stability, answer sourcing, and agent-specific behaviors.

G Details of automatic agent answer evaluator

F Human error examples

  • 14 human questions were answered incorrectly and 3 were abandoned; missing details and lack of topic knowledge were the two most frequent error reasons.The table provides three examples with attributed error reasons and explanations.Source: Table 6 · p. 17

H Detailed agent performance results

I Justifying size of BEARCUBS

  • The authors report that high-quality question writing was difficult: freelancers’ accepted-question quality declined, and unintended shortcuts led to discarded questions.They report that Deep Research often found unintended shortcuts, especially for multimodal questions.Source: I Justifying size of BEARCUBS · p. 17
  • The authors defend 111 questions as a meaningful benchmark size, citing other small benchmarks and the complex retrieval, reasoning, and synthesis each question typically requires.They also report minimal accuracy standard deviation across four DeepSeek R1 trials, with and without Serper.Source: I Justifying size of BEARCUBS · pp. 17–18, Table 10 · p. 20

J Model correct answer source proportions

K Agent-specific behavior

  • Grok 3 mirrors an input’s non-English language in its search and reasoning trajectory and never abstains; Computer Use sometimes declares tasks impossible without trying.The authors say Grok 3’s always-answer behavior needs user study to assess its desirability.Source: K Agent-specific behavior · p. 20
Loading 2503.07919v3…