Source-linked AI summary
ScholarSearch: Benchmarking Scholar Searching Ability of LLMs
Junting Zhou, Wang Li, Yiyan Liao, Nengyuan Zhang, Tingjia Miao, Zhihui Qi, Yuhan Wu, Tong Yang
TL;DR
Existing benchmarks do not fully evaluate the complex, real-time information retrieval required for academic research. ScholarSearch provides challenging academic questions designed to require external searching. The reported results show browsing improves performance, while current search-enabled models remain insufficient for consistently answering these queries.
Problem
Existing benchmarks inadequately measure complex academic information retrieval that requires real-time, multi-step searching beyond pre-existing knowledge.
Source: Introduction · p. 1Method
ScholarSearch is a 223-question benchmark curated to require multi-hop external searches for answers that cannot be derived from model knowledge alone.
Source: Introduction · p. 1, Problem Formalization · p. 3Results
The findings indicate reasoning alone does not solve ScholarSearch queries, browsing improves performance, and current search-enabled models remain insufficient for consistently addressing complex academic problems.
Source: Conclusion · p. 5Takeaways & Limitations
ScholarSearch offers a benchmark for evaluating LLM information retrieval capabilities in academic research settings.
Source: Conclusion · p. 5Even search-enabled models remain below perfect accuracy on complex academic queries; gpt-4o-search-preview achieved 18.83%.
Source: Current Search Models Are Still… · p. 5
Abstract
from arXiv · showhide
Large Language Models (LLMs)' search capabilities have garnered significant attention. Existing benchmarks, such as OpenAI's BrowseComp, primarily focus on general search scenarios and fail to adequately address the specific demands of academic search. These demands include deeper literature tracing and organization, professional support for academic databases, the ability to navigate long-tail academic knowledge, and ensuring academic rigor. Here, we proposed ScholarSearch, the first dataset specifically designed to evaluate the complex information retrieval capabilities of Large Language Models (LLMs) in academic research. ScholarSearch possesses the following key characteristics: Academic Practicality, where question content closely mirrors real academic learning and research environments, avoiding deliberately misleading models; High Difficulty, with answers that are challenging for single models (e.g., Grok DeepSearch or Gemini Deep Research) to provide directly, often requiring at least three deep searches to derive; Concise Evaluation, where limiting conditions ensure answers are as unique as possible, accompanied by clear sources and brief solution explanations, greatly facilitating subsequent audit and verification, surpassing the current lack of analyzed search datasets both domestically and internationally; and Broad Coverage, as the dataset spans at least 15 different academic disciplines. Through ScholarSearch, we expect to more precisely measure and promote the performance improvement of LLMs in complex academic information retrieval tasks. The data is available at: https://huggingface.co/datasets/PKU-DS-LAB/ScholarSearch
Introduction
Existing benchmarks do not adequately test the deep, real-time information retrieval required for academic research. ScholarSearch is designed to evaluate that capability with challenging, verifiable academic questions.
Introduction
- Academic and browsing benchmarks alike miss the iterative, source-critical searching needed to measure deep research in academic settings.Academic benchmarks emphasize pre-existing knowledge, while browsing benchmarks tend to use more straightforward, nonacademic queries.Source: Introduction · p. 1
- ScholarSearch introduces 223 academic questions designed to require iterative, multi-hop searches rather than a single query or stored model knowledge.Its collection process uses search-augmented models and precise answer verification and source attribution.Source: Introduction · p. 1
- The dataset’s collection pipeline filters student-sourced questions for difficulty, then has a specialized team review uniqueness, source access, and academic correctness.Source: Figure 1 · p. 2
- ScholarSearch is presented as a challenging framework for evaluating academic information retrieval and examining current models’ strengths and weaknesses.The authors intend it to support future advances in AI systems for academic applications.Source: Introduction · p. 2
Related Work
Prior academic benchmarks primarily test knowledge and reasoning over curated content, while browsing benchmarks test general web interaction. Deep-search systems and benchmarks extend browsing toward multi-step information seeking, but the related work describes a remaining need for academic depth.
Related Work
- Academic benchmarks such as MMLU, GPQA, SuperGPQA, and MMLU-Pro assess knowledge and reasoning over curated content without requiring active, real-time search.Source: Related Work · p. 2
- Browse benchmarks evaluate general web navigation, retrieval, or task completion, often without the depth needed for complex information seeking.Source: Related Work · pp. 2–3
- Deep-search systems and agentic approaches aim to acquire and synthesize external information; BrowseComp tests multi-step web retrieval with 1,266 questions.Source: Related Work · p. 3
Approach
The framework defines deep research as retrieving facts unavailable from an LLM’s static knowledge and present in an external corpus. Its dataset pipeline recruits students to create difficult questions and subjects selected items to specialist review.
Problem Formalization
- A separate discriminator model compares each final answer with a ground-truth reference, avoiding inspection of sources or reasoning steps during correctness validation.Source: Problem Formalization · p. 3
- A query qualifies as deep research only when its answer cannot be derived from the LLM’s internal knowledge and essential information exists solely in an external corpus.Source: Problem Formalization · p. 3
Data Collection
- Trained undergraduate and graduate volunteers select public online materials and formulate academic questions that require web searches.Source: Data Collection · p. 3
- Questions must resist standard Grok 3 reasoning, and at least one of Grok 3 DeepSearch or Gemini 2.5 Pro Deep Research must fail to answer correctly.Source: Data Collection · p. 3
- Specialist review checks that answers are singular, sources publicly accessible, and academic value and correctness supported by those sources.Source: Data Collection · p. 3
Experiments
The experiments evaluate LLM reasoning and browsing on ScholarSearch, with the reported findings highlighting browsing’s role in accuracy. The dataset contains curated questions contributed by students across disciplines and records four fields per item.
Statistics of ScholarSearch
- ScholarSearch contains 223 curated questions, each with a question, answer, explanation, and domain.Source: Statistics of ScholarSearch · p. 3
- Students from varied academic levels and faculties contributed questions intended to reflect practical academic search scenarios across multiple disciplines.Source: Statistics of ScholarSearch · p. 3
Testing Inference Accuracy on ScholarSearch
- The reported findings identify browsing capabilities as important for improving accuracy on complex academic information retrieval tasks.Source: Testing Inference Accuracy on… · p. 3
- Question creators manually annotated each item’s discipline, and the dataset includes an example organized by question, answer, explanation, and domain.Source: Figure 2 · p. 4, Figure 3 · p. 4
Discussion
ScholarSearch questions require information beyond static model knowledge, and browsing improves accuracy, but current search-enabled models still struggle with complex academic queries.
Question from ScholarSearch Cannot Be Solved by Reasoning Alone
- 7.175% for gpt-4.1, 2.242% for gpt-4o-mini, and 8.520% for deepseek-r1-0528 show low accuracy without browsing.The questions often require up-to-date, specific information and multi-hop reasoning across sources.Source: Question from ScholarSearch Cannot Be… · p. 4
The Impact of Browsing Capabilities
- 18.83% for gpt-4o-search-preview and 10.31% for gpt-4o-mini-search-preview illustrate the accuracy achieved by browsing-enabled models.Among models without search, Social Sciences & Humanities accuracy was significantly higher than Science & Engineering; with search, the rates were relatively close.Source: The Impact of Browsing Capabilities · pp. 4–5
Current Search Models Are Still Insufficient for Solving Complex Academic Problems
- Search-enabled models remain insufficient for consistently answering queries requiring deep research, specialized knowledge, and multi-hop reasoning across diverse sources.The reported gpt-4o-search-preview accuracy was 18.83%.Source: Current Search Models Are Still… · p. 5
Conclusion
The paper introduces ScholarSearch, a curated benchmark for complex academic information retrieval, and reports that browsing improves performance while leaving substantial challenges for current models.
Conclusion
- ScholarSearch is a benchmark designed to evaluate LLMs’ complex information retrieval capabilities in academic research settings.It emphasizes multi-hop, deep searches and real-time information acquisition beyond static knowledge bases, using expert curation and LLM-based filtering.Source: Conclusion · p. 5
- Academic queries in ScholarSearch cannot be solved by reasoning alone, browsing improves performance, and state-of-the-art search-enabled models remain insufficient.The paper identifies advanced synthesis, contextual understanding, and answer-accuracy verification as unmet needs.Source: Conclusion · p. 5
- The authors propose expanding ScholarSearch’s scale and diversity, exploring multimodal academic search, and developing metrics for synthesized information quality and source attribution.They describe the dataset as a realistic, challenging benchmark for search-augmented LLMs.Source: Conclusion · p. 5
Core Contributor
The authors acknowledge contributors who supported problem-solving and evaluation.
Core Contributor
- The paper thanks contributors for support with problem-solving and evaluation.The acknowledgments list contributors from multiple Peking University departments and the Central Academy of Fine Arts.Source: Core Contributor · pp. 5–6
Evaluation Prompt
The appendix presents the evaluation prompt used to assess whether model outputs are correct.
Evaluation Prompt
- The authors use a prompt to evaluate whether model outputs are correct.Figure 4 is identified as the Evaluation Prompt.Source: Evaluation Prompt · p. 7, Figure 4 · p. 7
Examples of ScholarSearch
Figures 5–7 are labeled as three examples of ScholarSearch. The question shown asks for the sexagenary year associated with a historical episode in Wang Yangming’s life.
Examples of ScholarSearch
- Figure 5 is labeled as Example 1 of the ScholarSearch dataset.Its question asks which sexagenary year Wang Yangming read a passage from Zhu Xi’s memorial and felt an unresolved separation between physical principle and inner mind.Source: Figure 5 · p. 7, Examples of ScholarSearch · p. 7
- Figure 6 is labeled as Example 2 of the ScholarSearch dataset.Source: Figure 6 · p. 8
- Figure 7 is labeled as Example 3 of the ScholarSearch dataset.Source: Figure 7 · p. 8