PaperlayerView exact revision on arXiv

Source-linked AI summary

BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese

Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, Yining Hua

arXiv:2504.19314v2cs.CL

TL;DR

Direct evaluation of LLMs’ web-browsing competence remains limited, and English browsing benchmarks do not adequately evaluate the Chinese web. BrowseComp-ZH provides a purpose-built multi-hop benchmark; most standalone LLMs achieve limited accuracy, while multi-turn retrieval products outperform purely parametric models, with DeepResearch at 42.9%.

  • Problem

    Direct evaluation of LLM web-browsing competence remains limited, while English benchmarks do not capture the linguistic and infrastructural complexities of the Chinese web.

    Source: §1 · p. 2

  • Method

    BrowseComp-ZH builds Chinese multi-hop questions from concise factual answers and uses three-engine screening plus human verification to promote difficulty and answer uniqueness.

    Source: §1 · p. 3, §5 · p. 9

  • Results

    42.9% accuracy for DeepResearch; most standalone LLMs achieve limited accuracy, while multi-turn retrieval products outperform purely parametric models.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    LLMs still struggle to reconcile retrieved evidence with internal representations, and retrieval without robust post-retrieval reasoning may hinder performance.

    Source: §4.6 · p. 9

    The dataset is relatively small, and changing web facts may undermine answer stability and reproducibility despite validation.

    Source: §5 · p. 9

Abstract

from arXiv · show

As large language models (LLMs) evolve into tool-using agents, the ability to browse the web in real-time has become a critical yardstick for measuring their reasoning and retrieval competence. Existing benchmarks such as BrowseComp concentrate on English and overlook the linguistic, infrastructural, and censorship-related complexities of other major information ecosystems -- most notably Chinese. To address this gap, we introduce BrowseComp-ZH, a high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents on the Chinese web. BrowseComp-ZH consists of 289 multi-hop questions spanning 11 diverse domains. Each question is reverse-engineered from a short, objective, and easily verifiable answer (e.g., a date, number, or proper noun). A two-stage quality control protocol is applied to strive for high question difficulty and answer uniqueness. We benchmark over 20 state-of-the-art language models and agentic search systems on our proposed BrowseComp-ZH. Despite their strong conversational and retrieval capabilities, most models struggle severely: a large number achieve accuracy rates below 10%, and only a handful exceed 20%. Even the best-performing system, OpenAI's DeepResearch, reaches just 42.9%. These results demonstrate the considerable difficulty of BrowseComp-ZH, where success demands not only effective retrieval strategies, but also sophisticated reasoning and information reconciliation -- capabilities that current models still struggle to master. Our dataset, construction guidelines, and benchmark results have been publicly released at https://github.com/PALIN2018/BrowseComp-ZH.

1 Introduction

BrowseComp-ZH fills the gap in direct evaluation of Chinese-web browsing by introducing a natively constructed benchmark of difficult, verifiable multi-hop questions and evaluating LLMs and search agents on it.

Introduction

  • Although Wei et al. (2025) introduced an English browsing benchmark, translated queries fail to reflect Chinese-web search pathways and linguistic challenges, motivating native construction.The Chinese web has fragmented platform-specific content, inconsistent naming, and linguistic features that can undermine keyword-based or linear retrieval.Source: §1 · p. 2
  • Expert annotators reverse-design multi-constraint queries from factual answers; three-engine first-page screening and human verification of agent-proposed answers enforce retrievability and answer uniqueness.All queries, evidence chains, and browsing steps are authored in Chinese and localized.Source: §1 · pp. 2–3
  • The dataset contains 289 multi-constraint questions across 11 domains and benchmarks more than 20 systems, including open models, APIs, and search agents.Questions require multi-step reasoning and integration of online information.Source: §1 · p. 3
  • Naive models often stay below 10%, while DeepSeek-R1 surpasses DeepSeek-V3 by 14.5% and Claude-3.7-Sonnet outperforms Claude-3.5 by 12.2%.The reported pattern holds despite differences in parameter scale or training data size for naive models.Source: §1 · p. 3
  • DeepResearch reaches 42.9% accuracy and Doubao Deep Search 26.0%, showing that some retrieval-and-reasoning agents outperform other system designs.The paper reports that well-orchestrated retrieval and reasoning pipelines can substantially improve performance.Source: §1 · p. 3
  • Adding web search lowers DeepSeek-R1 accuracy from 23.2% without web access to 7.6% with web search enabled.The result shows that retrieval integration does not benefit every system.Source: §1 · p. 3

2 Related Work

Prior work evaluates tool-using LLMs and retrieval, but existing English and Chinese benchmarks leave gaps in complex web-browsing assessment. BrowseComp-ZH addresses these gaps with native-Chinese, validated tasks and broad agent coverage.

Related Work

3 The BrowseComp-ZH Dataset

BrowseComp-ZH was built by reverse-engineering challenging Chinese-web questions from verifiable factual answers and refining them through two-stage quality control. The resulting benchmark contains 289 validated questions across 11 domains, with detailed questions and concise answers.

3.1 Dataset Construction

  • Ten expert contributors used an answer-first process to create questions from objective, independently verifiable, specific facts across 11 predefined domains.Topics included Film & TV, Technology, Art, History, Sports, Music, Geography, Policy & Law, Medicine, Video Games, and Academic Research.Source: §3.1 · p. 4
  • Questions combine multiple constraints, are revised when readily searchable, and include an authoritative source URL linking the constraints to the answer.Annotators tested queries across Baidu, Bing, and Google; GPT-4o and DeepSeek web-search checks were also used to increase complexity when answers were retrieved easily.Source: §3.1 · pp. 4–5

3.2 Quality Control

  • Two-stage quality control reduced 480 preliminary samples to 289 validated questions, filtering 76 simple and 115 ambiguous samples.The final questions were judged to retain high difficulty and answer verifiability.Source: §3.2 · p. 5, §3.1 · p. 5

3.3 Data Statistics

  • Film & TV (15.6%), Art (13.8%), and Geography (12.8%) are the largest topic categories, while Academic Papers (2.4%) and Policy & Law (3.5%) are among the smallest.The dataset spans 11 topic domains; Music accounts for 11.1%, History 10.0%, and Medicine 9.0%.Source: §3.3 · p. 5, Figure 2 · p. 6
  • Questions predominantly span 60–90 characters, while answers typically span 5–10 characters.The distributions reflect information-dense questions paired with short, precise answers.Source: §3.3 · pp. 5–6, Figure 3 · p. 6

4 Benchmarks

BrowseComp-ZH benchmarks diverse models and AI search products, with DeepResearch achieving the highest reported accuracy at 42.9%. The results also show higher performance from reasoning-enhanced models and multi-round search, while enabling search can hurt some systems.

4.1 Models

  • The evaluation covered open-source and closed-source models alongside mainstream AI search products, including DeepSeek, Qwen, GPT, Claude, Gemini, DeepResearch, Grok, Perplexity, Doubao, Kimi, and Yuanbao.The evaluated roster also included O1, O4-mini, Qwen2.5-MAX, and other listed systems.Source: §4.1 · pp. 6–7

4.2 Setting

  • For configurable model decoding, the study used temperature 0.6 and top-p 0.95; AI search products were tested through human-operated GUI interactions.For O1 and Claude-3.7, inference used fixed temperature and top-p because those parameters were not user-configurable. Model answers were extracted with regular expressions and scored using GPT-4o; human annotators extracted and verified search-product answers because those systems were less instruction-following.Source: §4.2 · p. 7

4.3 Metrics

  • The study reports accuracy and calibration error, computing ECE as a weighted average of the absolute gaps between bin-level mean confidence and empirical accuracy across five probability bins.The bins span [0–0.2), [0.2–0.4), [0.4–0.6), [0.6–0.8), and [0.8–1.0].Source: §4.3 · p. 7

4.4 Performance

  • 42.9% accuracy made OpenAI’s DeepResearch the top system, ahead of O1 at 29.1% and Gemini-2.5-Pro at 27.3%.AI search products with retrieval mechanisms outperformed closed-source APIs, which in turn outperformed open-source models; several systems scored below 10%, including Qwen2.5-72B-Instruct (6.6%), GPT-4o (6.2%), and Claude-3.5-Sonnet (5.5%).Source: §4.4 · pp. 7–8, Table 1 · p. 8

4.5 Analysis of Reasoning Ability of LLM

  • DeepSeek-R1 (23.2% accuracy) outperformed DeepSeek-V3 (8.7%), and Claude-3.7-Sonnet (17.7%) outperformed Claude-3.5-Sonnet (5.5%).The same pattern held for Gemini-2.0-Flash versus Gemini-2.5-Pro; for Doubao, Deep Search reached 26.0% versus 18.7% for Standard, a nearly 8% absolute gain.Source: §4.5 · p. 8

4.6 Analysis of AI Search Products

  • Multi-round retrieval systems had higher accuracy than single-retrieval systems: DeepResearch scored 42.9%, Doubao (Deep Search) 26.0%, and Perplexity 22.6%, versus Kimi 8.0%, Yuanbao 12.2%, and DeepSeek 7.6%.DeepSeek-R1 accuracy fell from 23.2% without web access to 7.6% with web search enabled. The authors hypothesize that less reliable retrieved content may override the model’s more accurate internal knowledge.Source: §4.6 · pp. 8–9

4.7 Calibration Analysis

  • DeepSeek-R1’s calibration error rose from 59% in the direct-answer setting to 65% with search enabled.The evaluation required confidence estimates alongside predictions and found that integrating search increased calibration errors, consistent with observations reported in BrowseComp.Source: §4.7 · p. 9

5 Conclusion and Discussion

BrowseComp-ZH is a 289-question benchmark for evaluating LLM web browsing and reasoning in Chinese, built with multi-hop questions and two-stage quality control. Evaluations show that reasoning-focused models and iterative-retrieval search products outperform standalone LLMs, while dataset size and answer stability remain limitations.

5 Conclusion and Discussion

  • The benchmark introduces challenging Chinese-web questions requiring multi-hop retrieval, information filtering, and logical reasoning, with three-engine validation and human verification to improve difficulty and answer uniqueness.It contains 289 samples across 11 topics and supports evaluations of over 20 models and AI search products.Source: §5 · p. 9
  • DeepResearch reached 42.9%, while O1 scored 29.1% and Gemini-2.5-Pro 27.3%; iterative-retrieval search products outperformed purely parametric models.The authors attribute the difficulty partly to fragmented information and inconsistent indexing in the Chinese web environment.Source: §5 · p. 9
  • The dataset is relatively small, answer uniqueness cannot be fully guaranteed, and evolving web facts threaten answer stability and reproducibility.Source: §5 · p. 9
  • Future work will broaden question coverage, analyze reasoning and search strategies, study model failures through case studies, and explore browsing improvements such as reinforcement-learning post-training.Source: §5 · p. 9

A Technical Appendices and Supplementary Material · A.1 Instructions for model prediction

For model prediction, the evaluation followed BrowseComp instructions and added a prompt asking models to answer from intrinsic knowledge if they might refuse due to lacking search capabilities.

A.1 Instructions for model prediction

  • The evaluation followed BrowseComp prediction instructions and prompted open- and closed-source models to use intrinsic knowledge if they might refuse because they lacked search capabilities.Source: §A.1 · p. 13

A.2 Instruction for grading

The grading setup follows BrowseComp’s prompt and uses GPT-4o to evaluate answers.

A.2 Instruction for grading

  • They use the BrowseComp grading prompt and GPT-4o for grading.Source: §A.2 · p. 13

Related papers

Loading 2504.19314v2…