HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents

1 INTRODUCTION

• Existing browsing evaluations often isolate subsets of real-world search challenges, while HyperBrowseComp requires agents to infer what and where to search and connect evidence across multiple steps.Relevant entities, relationships, sources, and modalities may be implicit; agents must aggregate and compare candidate answers or use multi-hop retrieval to verify a result.Source: §1 · pp. 1–2
• A German example requires identifying a video moment, grounding its location, and using geographic clues to find information about a nearby bank.The route crosses video, map, and financial-report evidence.Source: §1 · p. 2, Figure 1 · p. 2
• HyperBrowseComp contains 423 manually authored questions in 13 languages, designed to locate obscure but publicly verifiable open-web information.Many questions require reasoning across heterogeneous sources and extracting decisive evidence from non-textual content.Source: §1 · pp. 2–3
• The evaluation covers five web-search-enabled frontier LLMs and compares built-in, Exa, and OWL search integrations on accuracy, cost, and failure cases.Source: §1 · p. 3

Web-Browsing and Search Benchmarks

• Knowledge benchmarks primarily test domain knowledge and closed-ended reasoning, while browsing benchmarks target information seeking, tool use, realistic web tasks, or persistent search.Examples include GPQA and Humanity’s Last Exam (Rein et al., 2024; Phan et al., 2025), GAIA (Mialon et al., 2024), WebArena (Zhou et al., 2024), BrowseComp (Wei et al., 2025), BrowseComp-Plus (Chen et al., 2025), and SealQA (Pham et al., 2025).Source: §2 · p. 3
• HyperBrowseComp retains BrowseComp’s hard-to-find, verifiable answers while combining multilingual and multimodal questions on the open internet rather than a fixed corpus.This setting is described as closer to what browsing agents realistically encounter than BrowseComp-Plus’s static corpus.Source: §2 · p. 3

Multilingual and Multimodal Search

• BrowseComp-ZH and K-BrowseComp adapt browsing to Chinese and Korean, while XBCP keeps questions and answers in English and translates supporting documents.HyperBrowseComp instead authors questions independently in their respective languages, preserving language-specific phrasing, terminology, cultural references, and web environments.Source: §2 · pp. 3–4
• Multimodal benchmarks examine text-image, image-video, visual multi-hop, or noisy video-and-audio evidence; HyperBrowseComp combines multilingual questions with a broader range of web sources.Prior examples include MMSearch, MM-BrowseComp (Li et al., 2025), BrowseComp-V 3 (Zhang et al., 2026), and MERRIN (Wang et al., 2026).Source: §2 · p. 4

3.1 DESIGN PRINCIPLES

• Each question targets one concise, distinguishable factual answer, keeping evaluation unambiguous while making discovery—not answer interpretation—the source of difficulty.Source: §3.1 · p. 4
• Native or highly proficient speakers authored questions without an English seed set, preserving language-specific terminology, cultural context, and search behavior.Source: §3.1 · p. 4
• Questions may require evidence from images, video, audio, scanned documents, books, tables, figures, or maps, and the decisive modality may not be stated in the question.Challenging text-only questions are also retained; agents may need to discover the relevant evidence medium before finding the answer.Source: §3.1 · p. 4
• Questions target obscure but public information, with evidence that may be poorly indexed, buried in long sources, or accessible only at a particular point in a source or clue chain.Source: §3.1 · p. 4

3.2 HYPERBROWSECOMP DATA COLLECTION

• Authors start from a verifiable answer and evidence, submit exact evidence locations and a solution trace, then a second annotator checks correctness, uniqueness, accessibility, and ambiguity.Validators can request non-leading fingerprints to disambiguate answers and can ask that questions be made more difficult.Source: §3.2 · p. 5
• A seven-model no-internet audit identified questions answerable from model knowledge; applying the exclusion rule removed 77 questions and left 423.Questions answered correctly by at least two models were excluded, as were highly specific questions answered by one model.Source: §3.2 · pp. 5–6
• The final dataset contains 423 questions in 13 languages; approximately 64.3% require a non-text modality, and video/YouTube is the most frequent source category at 39.0%.It spans 12 primary domains plus Other; modalities can overlap.Source: Figure 2 · p. 5, §3.2 · p. 6

4.1 EXPERIMENTAL SETUP

• The study evaluates five models with built-in search, Exa, and OWL; built-in and Exa use ReAct up to 25 research steps, while OWL uses a specialized multi-agent workforce.Binary correctness is scored by an LLM judge; judge sensitivity tests showed at most 0.80 percentage points of aggregate variance.Source: §4.1 · p. 6, §3.7 · pp. 6–7

4.2 MODEL PERFORMANCE

• 31.68% accuracy for Gemini 3.7 Flash led the evaluated models, followed by 25.53% for Gemini 3.1 Pro Preview.The authors describe the scores as leaving substantial headroom for improvement.Source: Table 2 · p. 7, §4.2 · p. 7
• Replacing built-in search with Exa improves GPT-5.6 Sol but severely degrades both Gemini variants; OWL has 93 terminal runtime failures counted as incorrect.Even on completed OWL attempts, Gemini 3.7 Flash reaches 22.73% accuracy, below its built-in-search result.Source: §4.2 · p. 7
• 1.32–2.20 million tokens per question for Exa configurations contrasts with about 462 thousand for OWL, which uses 133 model calls per attempt.Gemini’s internal search is described as both cheaper and better-performing than Exa or OWL.Source: §4.2 · pp. 7–8

4.3 HUMAN EVALUATION

• Human and Gemini performance was comparable on the 30-question sample, but they were jointly correct on only 8; language difficulty patterns also differed.Across all attempts, including give-ups, mean elapsed time was 89.5 minutes and median time was 54.6 minutes.Source: §4.3 · p. 8

4.4 WHAT MAKES A QUESTION DIFFICULT?

• 244 of 423 questions had no correct answer across five native-search models, while 49 of the 179 solved questions were answered by exactly one model.These outcomes indicate shared failures alongside complementary successes.Source: §4.4 · p. 8
• Shared failures were associated with visual modalities, Games and Food & Drink, German and Indonesian queries, and wording such as “magazine,” “page,” and “photograph.”The patterns are descriptive, and the analysis does not isolate retrieval, source-access, interpretation, or reasoning failures; difficulty may vary across language splits.Source: §4.4 · p. 8, Figure 3 · p. 9

• 31.68% accuracy was achieved by the strongest evaluated configuration, while 57.68% of questions received no correct answer from any of the five models evaluated with native search.Source: §5 · p. 9
• The benchmark contains 423 manually authored questions across 13 languages, 8 modalities, and 13 primary domains, grounded in linguistic and cultural environments.Source: §5 · p. 9
• Questions require agents to locate and connect evidence across webpages, images, videos, audio, scanned documents, maps, and other sources.Source: §5 · p. 9
• Independent validation and a multi-model no-internet audit aim to ensure the benchmark measures difficult evidence discovery rather than parametric recall.Source: §5 · p. 9
• The findings motivate further research on agents that interpret implicit, language- and culture-specific clues and reliably discover, connect, and verify heterogeneous evidence.Source: §5 · p. 9

• HYPERBROWSECOMP may not represent the natural distribution of everyday search queries, and live-web rankings, page contents, geographic availability, and tool behavior can change.The authors preserve traces to mitigate variability.Source: LIMITATIONS AND ETHICAL CONSIDERATIONS · p. 9
• The selected languages do not cover all linguistic communities, and searchable information differs substantially in amount and quality across languages.Source: LIMITATIONS AND ETHICAL CONSIDERATIONS · p. 9
• High computational costs and restrictive API access limited the primary study to five representative frontier models.A single run can require days of execution and hundreds of millions of tokens.Source: LIMITATIONS AND ETHICAL CONSIDERATIONS · p. 9
• The evaluation compares models across provider-native search and Exa with an OWL-based harness, and reported scores may change with a different harness.Exa was tested on only a subset of models; broader harness-design comparisons are left for future work.Source: LIMITATIONS AND ETHICAL CONSIDERATIONS · p. 10

• The authors document experimental setup, data sources, evaluation procedures, and implementation details, with appendices listing prompts, tool schemas, model and API identifiers, harness parameters, records, grading, and cost.Source: REPRODUCIBILITY STATEMENT · p. 10
• Live webpages, search rankings, and provider tools may change, so exact reruns can differ; preserved outputs and traces support auditing of reported results.Source: REPRODUCIBILITY STATEMENT · p. 10
• The authors state that the full benchmark data, annotation guideline, and code will be public upon paper release.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

• Generative AI tools were used for manuscript language editing and proofreading, and the authors reviewed the AI-assisted work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A.1 DATA CREATION GUIDELINES

• Questions were written by native or near-native speakers who had lived from birth or a young age in cities where the languages are widely spoken.Annotators received a guideline presentation and slides for reference.Source: §A.1 · p. 15
• Annotators completed sample questions after training, and a core team member validated the samples and gave feedback before approving them to proceed.Source: §A.1 · p. 15
• Validators were selected from high-quality annotators, followed the same protocol, and reviewed questions in two stages: general validation followed by core-team review.Source: §A.1 · p. 15

A.2 HUMAN EVALUATION GUIDELINES

• Annotators were tested, limited to one hour per question, allowed to search the internet, and prohibited from using AI assistants.The Thai annotator was a top computer-science undergraduate; Indonesian and Vietnamese annotators were top business undergraduates who scored above 90% on the screening exam.Source: §A.2 · p. 15

B.1 AGENT AND GRADING PROMPTS

• ReAct evaluations use Inspect AI’s standard loop, while built-in search, Exa, and no-tools share a prompt structure with different retrieval instructions.Prompts ask for evidence-based reasoning and a shortest correct answer; ReAct is capped at 25 research steps before a finalization prompt is appended.Source: §B.1 · p. 15, §B.1.1 · pp. 15–16, §B.2 · p. 17

B.1.2 OWL WORKFORCE PROMPTS

• OWL uses a CAMEL workforce whose specialists divide task decomposition, web and multimodal inspection, and evidence reconciliation.Its tool budget is 50 external calls within a 2,400-second limit, with research tools closing 120 seconds before the deadline.Source: §B.1.2 · p. 16

B.1.5 GRADING PROMPT

• The grader extracts a final answer and judges it against the reference answer, treating an unparseable binary judgment as incorrect.The grading call uses the answering model in a fresh context containing the question, complete response, and accepted reference answers.Source: §B.1.5 · p. 17, §B.3 · p. 18

B.2 MODELS, AGENT HARNESSES, AND TOOL CONFIGURATIONS

• Built-in search and Exa run in a shared ReAct loop, whereas OWL uses a separate CAMEL workforce with browser and multimodal tools.Exa returns up to five results per query and truncates fetched pages at 20,000 characters; Tables 4 and 5 summarize harness configurations and exposed tools.Source: §B.2 · pp. 17–18, Table 4 · p. 18, Table 5 · p. 18

B.3 SCORING AND REPORTED METRICS

• The evaluation tracks token use, agent turns, end-to-end time, and execution or grading errors, and requires reporting attempted, scored, retried, and failed samples.Terminal failures remain visible so accuracy calculated over successfully scored samples does not conceal differences in failure rates.Source: §B.3 · p. 18

B.4 JUDGE SENSITIVITY ANALYSIS

• Three fixed judges—Gemini 3.7 Flash, GLM-4.7, and Kimi K2 Thinking—re-scored responses using the original binary grading prompt.The same question, complete response, and accepted reference answers were supplied for each judgment.Source: §B.4 · p. 19
• Agreement exceeded 98.7%, and accuracy shifts were at most 0.80 percentage points across evaluators.The authors conclude that self-judging introduces negligible bias for short, unambiguous targets and aggregate rankings remain robust to evaluator choice.Source: §B.4 · p. 19, Table 6 · p. 19

C.1 DATA COLLECTION

• Of 500 submitted questions, 121 were answered correctly by at least one offline model; excluding all 54 answered by at least two and 23 further questions yielded 423 retained questions.The 54 model counts overlap and do not imply that the same model combination succeeded on each question.Source: §C.1 · p. 19
• For questions answered correctly by only one model, reviewers used a qualitative specificity screen, excluding answers whose exact guess remained plausible in context.Specificity depends on required precision or unusual detail; a proper name, number, or long clue chain alone does not establish it.Source: §C.1 · p. 19
• The retained questions were classified by domain from question text and by modality from the question and proof cell, without opening linked sources.Modality labels allow overlapping source and task categories and separately assess whether non-text interpretation is required, not required, or uncertain.Source: §C.1 · p. 19, Figure 6 · p. 22, Figure 5
• Annotation rules classify the requested subject rather than incidental clues, and absence of a modality label does not establish that a question is text-only.Uncertain modality judgments and insufficient evidence require review.Source: §C.1 · pp. 19, 21, 22

C.2 PERFORMANCE AND RESOURCE USE BY LANGUAGE AND MODALITY

• Appendix C.2 summarizes accuracy and total token use and elapsed time by language and modality, with languages partitioning the questions and multi-label modality groups overlapping.OWL token and time accounting uses the workforce trace and separate grading call, while terminal workforce failures remain in the denominator as incorrect.Source: §C.2 · p. 23, Table 10 · p. 24, Table 11 · p. 24

C.3 WHAT MAKES A QUESTION DIFFICULT?

• The analysis defines shared failure as five recorded incorrect answers and covers provider-native-search runs across the benchmark’s language, domain, and modality breakdowns.It uses five model runs and excludes Exa runs and the seven-model no-internet audit; two of the 244 shared-failure questions have unresolved model outcomes.Source: §C.3 · p. 25
• Figures 7–9 present shared-failure statistics across all 13 languages, 12 domains, and modality-related labels, with the overall rate at 244/423 (57.68%).Language question sets differ, modality labels overlap, and the displayed associations are descriptive rather than causal.Source: §C.3 · p. 25, Figure 7 · p. 26, Figure 8 · p. 26
• Among terms with positive TF-IDF differences, “issue” had 19/25 shared failures (76.0%), “video” 35/47 (74.5%), and “page” 30/39 (76.9%).Figure 10 ranks terms by mean TF-IDF difference; its dots show shared-failure rates, not TF-IDF scores, and the associations do not establish causes.Source: §C.3 · p. 25, Figure 10 · p. 28