Source-linked AI summary
PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVR
James Burgess, Jan N. Hansen, Duo Peng, Yuhui Zhang, Alejandro Lozano, Min Woo Sun, Emma Lundberg, Serena Yeung-Levy
TL;DR
Existing RLVR search agents mainly target general-domain QA, leaving a need for technical search and reasoning over scientific literature. This paper releases a large biomedical corpus, factoid-QA datasets, and benchmarks, then trains RLVR search agents that outperform non-RL retrieval baselines. The study also identifies challenging scope boundaries and limitations in automatically generated QA.
Problem
Existing RLVR search agents emphasize general-domain QA, while scientific AI requires search and reasoning over technical literature and knowledge bases.
Method
The paper constructs verifiable biomedical factoid QA and trains search agents over 16 million PubMed abstracts using an RLVR environment.
Results
RL training outperforms non-RL retrieval baselines, improving over RAG by 9.6 and 5.5 points for 3B models and 14.5 and 9.3 points for 7B models on PaperSearchQA and BioASQ respectively.
Takeaways & Limitations
The released corpus, datasets, and benchmarks provide an environment for training and analyzing search agents over scientific papers.
Takeaways & Limitations
The study is restricted to factoid QA and biomedical papers, excluding other question types and domains such as chemistry, materials science, and computer science.
Abstract
from arXiv · showhide
Search agents are language models (LMs) that reason and search knowledge bases (or the web) to answer questions; recent methods supervise only the final answer accuracy using reinforcement learning with verifiable rewards (RLVR). Most RLVR search agents tackle general-domain QA, which limits their relevance to technical AI systems in science, engineering, and medicine. In this work we propose training agents to search and reason over scientific papers -- this tests technical question-answering, it is directly relevant to real scientists, and the capabilities will be crucial to future AI Scientist systems. Concretely, we release a search corpus of 16 million biomedical paper abstracts and construct a challenging factoid QA dataset called PaperSearchQA with 60k samples answerable from the corpus, along with benchmarks. We train search agents in this environment to outperform non-RL retrieval baselines; we also perform further quantitative analysis and observe interesting agent behaviors like planning, reasoning, and self-verification. Our corpus, datasets, and benchmarks are usable with the popular Search-R1 codebase for RLVR training and released on https://huggingface.co/collections/jmhb/papersearchqa. Finally, our data creation methods are scalable and easily extendable to other scientific domains.
1 Introduction
The paper targets scientific search agents because existing RLVR systems largely address general-domain QA, while scientific literature requires technical search and reasoning. It introduces a biomedical paper-search environment, datasets, benchmarks, and RL-trained agents that outperform non-RL baselines despite remaining challenging.
- Contributions: The environment trains agents to interleave reasoning and retrieval for scientific QA using RLVR, with released training data, a retrieval corpus, and benchmarks.The released components are intended to support training and evaluation of search agents.
- Motivation: Existing search-agent RLVR work emphasizes general-knowledge QA rather than technical scientific question answering.Scientific domains require technical knowledge, reasoning about complex systems, and searching specialized knowledge bases.
- Motivation: Scientific research depends on traversing large literature and databases, motivating search modules that can formulate queries, reason over evidence, and assess retrieved information.The paper positions these capabilities as relevant to future scientific AI systems.
- Contributions: The authors release a 16-million-abstract biomedical corpus and a 60k factoid-QA dataset generated from PubMed articles with biology-expert quality assurance.The data-creation methods are described as scalable and adaptable to materials science or chemistry.
- Results: RL training produces stronger performance than non-RL baselines, although overall scores remain low, making the benchmarks challenging.The analysis also reports planning through query rewriting, reasoning before retrieval, and verification after an initial answer.
2 Related Work
Prior scientific-literature QA systems commonly rely on prompting, scaffolding, or supervised fine-tuning, while this work investigates reinforcement learning for search agents over scientific papers. The paper builds on recent RL-trained search agents and uses factoid QA for verifiable evaluation.
- Search agents: Recent search agents extend reinforcement learning with web search, query decomposition, and simulated retrieval environments.Search-R1 and R1-Searcher introduced open search agents trained with final-answer rewards.
- Search agents: Earlier search and retrieval systems used supervised fine-tuning, few-shot prompting, or prompt optimization rather than reinforcement learning.The paper contrasts these approaches with RL-based training for agent search behavior.
- Scientific QA: Scientific-literature systems such as PaperQA and OpenScholar handle large full-text corpora but control agent behavior through scaffolding, prompting, or supervised fine-tuning.The authors instead explore RL because they state it promises stronger long-term generalization.
- Scientific QA: BioASQ is a long-running biomedical indexing and open-domain QA challenge whose human-created data reflects the importance of literature understanding for scientists.Its smaller scale motivates using it for search-agent evaluation.
3 Methods
The methods construct verifiable, application-relevant biomedical factoid QA from PubMed abstracts, provide retrieval resources and BioASQ evaluation data, and train agents with outcome-based RLVR. The pipeline combines expert-designed categories, LLM generation and paraphrasing, and GRPO-based search-agent optimization.
- Dataset construction: The dataset restricts questions to unambiguous single-entity factoids so reward models can verify answers by exact matching or synonyms.Questions are single-hop and do not require annotations for intermediate reasoning or retrieved documents.
- Dataset construction: Practicing scientists guide task properties, pipeline construction, and data verification to keep questions relevant to real applications.The construction is also informed by BioASQ.
- Dataset construction: Ten question categories are formed by combining expert brainstorming with LLM-proposed categories derived from sampled BioASQ questions.Human experts synthesize, merge, and discard candidate categories before they are used in data construction.
- Dataset construction: PubMed abstracts are sampled and passed to GPT-4.1 with category and QA guidance, followed by synonym generation and paraphrasing for half of the questions.Paraphrasing reduces reliance on exact keywords from abstracts.
- Dataset summary: 54,907 training samples and 5,000 test samples comprise the final PaperSearchQA dataset, with category, source-PubMed-ID, and paraphrase annotations.The dataset is released on Hugging Face Hub under CC-BY.
- Evaluation dataset: BioASQ is redistributed for evaluation using the same PubMed-abstract search corpus, with answer synonyms added to enable exact-match scoring.Its factoid subset contains 1,609 samples.
- Retrieval resources: The search corpus contains 16 million PubMed abstracts, with BM25 and e5 indexes provided for retrieval.The e5 index requires two A100 80GB GPUs during inference to avoid memory errors.
- Training algorithms: During RLVR training, agents generate reasoning and search rollouts, receive reward 1 for matching a target answer and 0 otherwise, and are optimized with GRPO.Retrieved tokens are masked during gradient computation, and a KL penalty discourages excessive divergence from a reference model.
4 Results
RLVR training improves scientific paper question-answering over non-RL baselines, while additional analyses reveal retrieval, model-scaling, data-construction, and agent-behavior patterns.
- Main results: RLVR training improves performance on PaperSearchQA and BioASQ compared with non-RL baselines, although overall scores remain low.The evaluation uses exact-match accuracy against target answers or synonyms.
- Main results: 9.6 and 5.5 points: RL improves over RAG for 3B models on PaperSearchQA and BioASQ, respectively.For 7B models, the corresponding improvements are 14.5 and 9.3 points.
- Quantitative analysis: Within 2 points: semantic e5 retrieval provides only a minor benefit over syntactic BM25 retrieval across RAG and RL experiments.The authors suggest technical keywords may remain present after paraphrasing, or that e5 may underperform in scientific domains.
- Quantitative analysis: 20.3 and 29.7 accuracy: retrieval-free 3B and 7B models retain non-negligible PaperSearchQA performance, but retrieval remains necessary because memorization is imperfect.The authors attribute this likely knowledge to PubMed abstracts appearing in pretraining mixtures.
- Quantitative analysis: 57.2 versus 44.9: Search-R1 scores higher on non-paraphrased than paraphrased PaperSearchQA questions, indicating paraphrasing sustains question difficulty.Half of the questions were paraphrased to enable this comparison.
- Qualitative results: Search traces show explicit planning, reasoning before search, and verification of parametric knowledge; later training makes explicit planning more common.After retrieval, models often answer immediately without explicit reasoning about document contents.
5 Discussion
The paper demonstrates RL training of search agents over scientific papers and outlines both the resulting insights and directions for extending scientific literature understanding.
- RL trains search agents to answer questions by reasoning and gathering knowledge from scientific papers.
- The released environment includes training datasets, benchmarks, and a corpus for scientific question answering over papers.
- The datasets are limited to single-hop factoid QA and simple retrieval over abstracts, motivating multihop, list, extended-answer, and conflicting-evidence tasks.
- Future literature-understanding agents could use citation traversal, source-reliability metrics, and checks against figures or presented data.
- The automatic data-generation pipeline could be adapted to chemistry, materials science, and computer science.
6 Conclusion
The paper argues that future AI Scientist systems will need knowledge-intensive search because scientific discovery depends on retrieving and evaluating specialized information. It presents RL-trained search agents as an approach to that capability.
- Future AI Scientist systems are envisioned to progress from well-defined tasks toward planning experiments and forming hypotheses.
- Scientific discovery requires recalling, retrieving, and evaluating specialized information from a massive corpus of human knowledge.
- The paper identifies literature understanding and RL training of search agents as essential capabilities or approaches for AI systems in science.
7 Limitations
The study’s scope is constrained by automatic LLM-generated data, a focus on factoid QA in biology and medicine, text-only inputs, and limited readiness for deployment.
- Automatic LLM-based data generation can produce factually incorrect questions and answers or infer overly general questions from single abstracts.
- The study restricts RLVR search training to factoid QA and covers biology and medicine rather than chemistry, materials science, or computer science.
- The datasets use text-only problems, whereas scientific reasoning can also involve images and other data types.
- The synthetic data-generation procedure requires access to research articles that may be protected by copyright.
- The derived agent system is a research prototype that has not undergone the thorough evaluation needed for real-world deployment.
A Dataset and Code Availability
The paper releases its data artifacts under open licenses and provides access through the Hugging Face Hub and GitHub code repository.
- The paper releases all artifacts on the Hugging Face Hub.
- The code is available through the PaperSearchQA GitHub repository.
- PaperSearchQA uses the CC-BY-4.0 license, while the search corpus and BioASQ evaluation set inherit CC-BY-2.5.
B Ethical considerations
The paper identifies risks from using LLMs to answer scientific questions from literature, including unreliable sources, selection effects, hallucinations, and automated data errors. It recommends uncertainty quantification, source-quality indicators, and community standards for appropriate use.
- Agents may retrieve and amplify outdated, retracted, or flawed studies without quality-assessment mechanisms.
- Poorly understood selection bias in retrieved papers may affect which studies scientists see.
- Hallucinated outputs and incorrect question-answering responses may harm scientific practice.
- Automated dataset generation may introduce errors into the resulting data.
- Future deployments should consider uncertainty quantification and source-quality indicators, alongside community standards for LLM use.
G Results: per-category performance
The paper reports its main results by the PaperQA2 per-category labels. Table 2 presents category-level results for comparison across systems.
- The main results are split according to PaperQA2’s per-category labels.
- Table 2 contains the paper’s main results for these category values.
- The reported category-level results use the categories defined by PaperQA2.
H Results: a note on PaperQA baseline
The PaperQA baseline comparison was configured to match the models, retrieval backends, and evaluation format used in the Search-R1 experiments. Table 2 reports per-category scores for baselines and Search-R1 training with RLVR.
- PaperQA baselines used the official codebase with matched model and system components for fair comparison.
- The retrieval backend used Search-R1’s BM25 and E5 dense retrieval over the same PubMed corpus as the paper’s experiments.
- Table 2 compares baseline and Search-R1 RLVR results using per-category scores.
- Evaluation matched Search-R1’s answer-scoring logic, with final single-entity answers placed in <answer> blocks.
I Training RLVR details
The RLVR training procedure uses GRPO to generate multiple rollouts per question, normalize rewards within each rollout group, and optimize the policy with the resulting advantages. Training scripts and hyperparameters are released in the code.
- GRPO generates multiple rollouts for each question using the current old policy.
- Advantages are computed by normalizing rewards within each group using the group mean and standard deviation.
- Full training scripts and hyperparameters are available in the released code.