PaperlayerView exact revision on arXiv

Source-linked AI summary

PaSa: An LLM Agent for Comprehensive Academic Paper Search

Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, Weinan E

arXiv:2501.10120v2cs.IRcs.LG

TL;DR

Complex academic searches require more than retrieving documents: they demand fine-grained coverage, paper reading, and citation exploration. PaSa is an LLM agent trained with reinforcement learning on AutoScholarQuery and evaluated with RealScholarQuery, where PaSa-7B outperforms the reported search and LLM-agent baselines. The paper’s evidence is concentrated in machine learning and primarily uses 7B-parameter models.

  • Problem

    Academic search systems often fall short on complex queries requiring long-tail knowledge, comprehensive survey coverage, and fine-grained understanding.

  • Method

    PaSa uses Crawler and Selector LLM agents for search, citation expansion, paper assessment, and reinforcement-learning optimization with AutoScholarQuery.

  • Results

    37.78% and 39.90% are PaSa-7B’s improvements over Google with GPT-4o on RealScholarQuery Recall@20 and Recall@50, respectively.

  • Takeaways & Limitations

    Despite training solely on synthetic data, PaSa achieves reported real-world performance that significantly outperforms the evaluated baselines.

  • Takeaways & Limitations

    Experiments primarily focus on machine learning and 7B-parameter LLMs, leaving performance in other scientific fields and larger models unexplored.

Abstract

from arXiv · show

We introduce PaSa, an advanced Paper Search agent powered by large language models. PaSa can autonomously make a series of decisions, including invoking search tools, reading papers, and selecting relevant references, to ultimately obtain comprehensive and accurate results for complex scholar queries. We optimize PaSa using reinforcement learning with a synthetic dataset, AutoScholarQuery, which includes 35k fine-grained academic queries and corresponding papers sourced from top-tier AI conference publications. Additionally, we develop RealScholarQuery, a benchmark collecting real-world academic queries to assess PaSa performance in more realistic scenarios. Despite being trained on synthetic data, PaSa significantly outperforms existing baselines on RealScholarQuery, including Google, Google Scholar, Google with GPT-4o for paraphrased queries, ChatGPT (search-enabled GPT-4o), GPT-o1, and PaSa-GPT-4o (PaSa implemented by prompting GPT-4o). Notably, PaSa-7B surpasses the best Google-based baseline, Google with GPT-4o, by 37.78% in recall@20 and 39.90% in recall@50, and exceeds PaSa-GPT-4o by 30.36% in recall and 4.25% in precision. Model, datasets, and code are available at https://github.com/bytedance/pasa.

1 Introduction

PaSa is an LLM-based academic search agent designed for complex queries requiring comprehensive retrieval, paper reading, and citation exploration. It is trained with reinforcement learning and evaluated on synthetic and real-world benchmarks, where it outperforms the reported baselines.

  • Motivation: Academic paper search requires long-tail knowledge, survey-level coverage, and fine-grained query understanding beyond general search systems.Researchers often need to read relevant papers and inspect citations, making literature surveys time-consuming.
  • Architecture: PaSa uses Crawler and Selector agents to collect papers through search and citation expansion, then judge each paper against the user query.The Crawler maintains a growing paper queue, while the Selector reads queued papers and evaluates their relevance.
  • Training: PaSa is optimized with reinforcement learning in AGILE, using AutoScholarQuery and a session-level PPO method for sparse rewards and long trajectories.Crawler trajectories may involve hundreds of papers, motivating specialized training for the paper-search setting.
  • Datasets: PaSa introduces AutoScholarQuery and RealScholarQuery, respectively providing synthetic fine-grained queries from top-tier AI papers and 50 real-world annotated queries.These datasets support training and more realistic evaluation of complex academic search.
  • Results: 37.78% and 39.90% are PaSa-7B’s improvements over Google with GPT-4o on RealScholarQuery Recall@20 and Recall@50.PaSa-7B also surpasses PaSa-GPT-4o by 30.36% in recall and 4.25% in precision.

2 Related Work

Related work positions PaSa at the intersection of LLM-assisted academic retrieval and tool-using LLM agents. Unlike query-refinement approaches alone, PaSa targets the broader survey workflow through search, reading, and citation navigation.

  • LLMs in Scientific Discovery: Prior LLM-based information-retrieval studies primarily improve retrieval quality by refining or reformulating search queries.Academic surveys additionally involve reading relevant papers and checking citations.
  • LLM Agents: LLM agents combine language models with memory, tool use, and planning for tasks including web operations, software development, and scientific experimentation.PaSa applies this agent paradigm to academic paper search.
  • LLM Agents: PaSa uses a session-level PPO algorithm to address sparse rewards and long trajectories specific to paper search.These challenges arise because complete searches can involve many papers and actions.

3 Datasets

The paper develops two datasets for complex academic search: synthetic AutoScholarQuery for AI-focused training and evaluation, and RealScholarQuery for realistic benchmark testing. Their construction combines generated or user-provided queries with paper retrieval and expert annotation.

  • 3.1 AutoScholarQuery: AutoScholarQuery is a synthetic, high-quality dataset of AI-focused academic queries and related papers.Its instances pair fine-grained scholarly queries with papers collected from related-work references.
  • 3.1 AutoScholarQuery: GPT-4o generated queries from related-work sections of papers published at ICLR 2023, ICML 2023, NeurIPS 2023, ACL 2024, and CVPR 2024.Only papers retrievable on arXiv were retained, using arxiv_id as the article identifier.
  • 3.1 AutoScholarQuery: AutoScholarQuery is split into 33,551 training, 1,000 development, and 1,000 testing instances, with each split derived from distinct source papers.Each instance contains a query, associated paper set, and query date.
  • 3.1 AutoScholarQuery: Manual review found 94.0% of sampled queries qualified, and 93.7% of corresponding papers among qualified queries were relevant and appropriate.The sample contained 100 query-paper pairs reviewed by three authors.
  • 3.2 RealScholarQuery: RealScholarQuery contains 50 fine-grained real-world research queries collected from AI researchers and filtered for overly broad topics.Relevant papers were gathered through multiple retrieval methods and finalized by professional annotators.
  • 3.2 RealScholarQuery: Each RealScholarQuery query required annotators to review 76 candidate papers on average and yielded 15.82 answer papers on average.The 50th percentile of answer counts was 9, while the 90th percentile was 37; annotation averaged $304 per query.

4 Methodology

PaSa combines a Crawler that expands candidate coverage through search and citation networks with a Selector that judges query relevance. Its training uses imitation learning and reinforcement learning with session-level rewards to address long trajectories and sparse supervision.

  • 4.1 Overview: PaSa uses Crawler and Selector agents to retrieve papers, explore citations, and assess whether papers satisfy the user query.The Crawler builds a paper queue through search and citation extraction, while the Selector reviews queued papers for relevance.
  • 4.2 Crawler: The Crawler invokes [Search], [Expand], and [Stop] actions that respectively retrieve papers, extract cited papers from a subsection, and reset context for the next queued paper.Search results and extracted citations are appended to the paper queue; stopping advances processing to another queued paper.
  • 4.2 Crawler: The Crawler is trained first with imitation learning and then with reinforcement learning on trajectories generated from academic search data.The reinforcement-learning reward is based on newly identified matching papers and uses action costs.
  • 4.2 Crawler: The reward indicator counts newly discovered matching papers, while the Selector supplies auxiliary judgments when the annotated dataset covers only part of the ground-truth set.This auxiliary signal mitigates sparse rewards caused by citation-based datasets containing limited relevant references.
  • 4.2 Crawler: Session-level PPO training partitions long trajectories at [Stop] actions, enabling efficient sub-trajectory sampling and incorporating rewards from later sessions.The return estimate includes in-session and across-session discounting, value estimates, and a per-token KL penalty against the initial policy.
  • 4.3 Selector: The Selector emits a True/False decision token followed by a rationale, allowing it to serve as a single-token reward model and rank search results.The rationale supports decision accuracy and user trust, while decision-rationale ordering does not affect Selector performance.

5 Experiments

PaSa-7b is evaluated against search engines and LLM baselines on synthetic and real-world benchmarks, with ablations examining citation exploration, reinforcement learning, and reward design.

  • Baselines and evaluation: PaSa-7b is compared with Google, Google Scholar, GPT-4o-assisted Google, ChatGPT, GPT-o1, and PaSa-GPT-4o on AutoScholarQuery and RealScholarQuery.PaSa-GPT-4o provides a prompted implementation for isolating the impact of reinforcement-learning training.
  • Main results: 33.80%, 38.83% and 42.64% are PaSa-7b’s improvements over Google with GPT-4o in Recall@20, Recall@50 and Recall@100 on AutoScholarQuery.PaSa-7b also improves recall by 9.64% over PaSa-GPT-4o with comparable precision.
  • Ensembling: 3.34% higher Crawler recall and 1.51% higher final recall result from running two sampled Crawler instances in the PaSa-7b-ensemble setting.The ensemble produces no significant change in precision on AutoScholarQuery.
  • Main results: 37.78%, 39.90% and 39.83% are PaSa-7b’s improvements over Google with GPT-4o in recall@20, recall@50 and recall@100 on RealScholarQuery.Against PaSa-GPT-4o, PaSa-7b improves recall by 30.36% and precision by 4.25%.
  • Selector evaluation: 85% is the Selector’s F1 score, exceeding GPT-4o by 5% and Qwen-2.5-7b by 30%, while its precision reaches 95%.The Selector functions as both the final decision-maker and an auxiliary reward model for Crawler reinforcement learning.
  • Ablation study: 22.98% and 32.21% recall decreases follow removal of citation-network expansion on AutoScholarQuery and RealScholarQuery, respectively.Reinforcement learning improves recall by 6.24% and 19.96%, while removing the auxiliary Selector reward model reduces recall by 3.76% and 9.63%.

6 Conclusion

The paper introduces PaSa for comprehensive academic search and reports that PaSa-7B outperforms the evaluated search-engine and language-model baselines on benchmarked queries.

  • 6 Conclusion: PaSa combines autonomous search, paper reading, and citation navigation to provide comprehensive and accurate results for complex academic queries.It is trained with AutoScholarQuery and evaluated on RealScholarQuery.
  • 6 Conclusion: 37.78% and 39.90% are PaSa-7B’s improvements over Google with GPT-4o in recall@20 and recall@50 on RealScholarQuery.PaSa-7B also exceeds PaSa-GPT-4o by 30.36% in recall and 4.25% in precision.

Limitations

The evaluation primarily covers machine learning and uses mostly 7B-parameter language models, leaving other scientific domains and larger-model scaling for future work.

  • Scope: The dataset collection and experiments primarily focus on machine learning, so performance in other scientific fields remains unexplored.The authors identify applicability to other domains as future work.
  • Model scale: Resource constraints led the experiments to primarily use 7B-parameter LLMs, leaving larger-model versions for future work.The authors expect scaling to larger models to produce more powerful agents.

A Quality Evaluation of AutoScholarQuery

AutoScholarQuery is built from fine-grained queries and cited papers, then quality-checked for query validity and answer-paper relevance; RealScholarQuery uses expert annotation with staged quality control.

  • Quality criteria: A qualified query must be complete, faithful to its source paper, and sufficiently unambiguous for its citations to form an answer list.An answer paper must address all or most essential query factors.
  • Quality results: 94.0% of sampled queries and 93.7% of their corresponding answer papers were judged qualified.Inaccurate citations in source papers were the primary reason for unqualified papers.
  • RealScholarQuery annotation: RealScholarQuery annotators determine whether each paper satisfies the user query, may read the entire paper, exclude surveys unless requested, and require all query conditions to be met.The benchmark uses professors from a computer-science department at a top-tier Chinese university.
  • Quality control: Annotation quality control progresses from full review to 40% and then 20% random checks, with batches re-annotated when consistency falls below 90%.Two authors conducted the quality control.

D.1 Imitation learning data generation

The imitation-learning data are organized into search and expansion sessions, with trajectories generated from AutoScholarQuery queries and papers. PPO training then combines these session types into batches of 16 trajectories per step.

  • Imitation-learning data are divided into search sessions starting from Sq and expansion sessions starting from Sq+p.
  • Search sessions prompt GPT-4o to generate queries, prepend [Search] before each query, and terminate trajectories with [Stop].The process constructs 3,011 search sessions.
  • Expansion sessions select paper subsections that reference query-relevant papers, while sampling unrelated sections with 10% probability for diversity.The resulting dataset contains 9,978 expanded session trajectories.
  • PPO training generates 4 search sessions, 6 expansion sessions, and 6 additional expanded sessions per step, totaling 16 trajectories.
  • The PPO implementation uses the hyperparameters listed in Table 12, and its return eventually converges after 200 training steps.

E Implementation Details of the Selector

The Selector training data are built from query–paper pairs labeled through GPT-4o decisions and simulated search behavior. The resulting dataset contains 19,812 labeled pairs with decision tokens and rationales.

  • The Selector dataset begins with 9,000 AutoScholarQuery training instances, for which GPT-4o generates decision tokens and rationales.
  • Figure 3 tracks return and value-function losses during PPO training using exponential-moving-average smoothing with weight 0.95.
  • Pairs with a GPT-4o decision of False are rejected, while retained pairs form positive query–paper examples.
  • A simulated partial search assigns each paper a 50% probability of entering the paper queue and labels unselected non-ground-truth papers as negatives.
  • The final training dataset contains 19,812 query–paper pairs, each with a GPT-4o-generated decision token and rationale.

F Selector Test Dataset

The Selector test dataset is constructed from 200 development queries and manually annotated query–paper pairs, while supporting experiments examine fair baseline comparisons, reward design, and prompt configurations.

  • Selector test dataset: The test-set construction starts from 200 AutoScholarQuery development queries and combines Google results with relevant-paper samples before annotation.
  • Selector test dataset: Professional annotation produces 98 positive and 102 negative query–paper examples.
  • Implementation references: The Selector relevance-evaluation prompt is documented in Table 13.
  • Fair comparison: All methods are evaluated on the same 100-sample AutoScholarQuery subset used by the ChatGPT baseline, and PaSa-7b consistently outperforms the baselines.
  • Action cost: Action costs are introduced because removing them would substantially increase unproductive actions without meaningful outcomes.
  • Action cost: The study varies the reward coefficient α and action cost c(a_t), reporting their effects in Tables 8 and 15.
  • Prompt configurations: Tables 16–20 document prompts for synthesizing AutoScholarQuery and implementing the Google, ChatGPT, GPT-o1, and PaSa-GPT-4o baselines.
Loading 2501.10120v2…