Source-linked AI summary
PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
Jakub Lála, Odhran O'Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G. Rodriques, Andrew D. White
TL;DR
Scientific question answering must handle rapidly growing literature while limiting hallucinations, verification burdens, and outdated knowledge. PaperQA addresses this gap with an agentic RAG system that retrieves and evaluates evidence from full-text papers before generating cited answers. It outperformed tested LLMs and commercial tools on LitQA and matched expert-human response patterns while costing less.
Problem
Scientific literature is difficult to navigate, while LLM answers can hallucinate, require costly verification, and omit information published after training cutoffs.
Method
PaperQA uses an agentic RAG workflow that dynamically searches full-text scientific papers, gathers relevant evidence, and generates answers with references.
Results
PaperQA outperformed all tested models and commercial tools on LitQA and matched expert-human response patterns, with human-PaperQA Cramer’s V of 0.67±0.02 versus human-human V of 0.66±0.03.
Takeaways & Limitations
PaperQA supports literature-based scientific QA with retrieval beyond pre-training data, calibrated uncertainty, and performance comparable to expert researchers.
Takeaways & Limitations
PaperQA assumes the underlying research papers are correct, while journal names and citation counts are not faithful indicators of quality.
Abstract
from arXiv · showhide
Large Language Models (LLMs) generalize well across language tasks, but suffer from hallucinations and uninterpretability, making it difficult to assess their accuracy without ground-truth. Retrieval-Augmented Generation (RAG) models have been proposed to reduce hallucinations and provide provenance for how an answer was generated. Applying such models to the scientific literature may enable large-scale, systematic processing of scientific knowledge. We present PaperQA, a RAG agent for answering questions over the scientific literature. PaperQA is an agent that performs information retrieval across full-text scientific articles, assesses the relevance of sources and passages, and uses RAG to provide answers. Viewing this agent as a question answering model, we find it exceeds performance of existing LLMs and LLM agents on current science QA benchmarks. To push the field closer to how humans perform research on scientific literature, we also introduce LitQA, a more complex benchmark that requires retrieval and synthesis of information from full-text scientific papers across the literature. Finally, we demonstrate PaperQA's matches expert human researchers on LitQA.
1 INTRODUCTION
Scientific literature is expanding beyond manual navigation, while LLM-based scientific question answering remains vulnerable to hallucinations, outdated knowledge, and verification costs. PaperQA addresses these limitations by using an adaptive, modular RAG agent that retrieves papers, gathers evidence, and generates referenced answers.
- Over 5 million academic articles were published in 2022, with over 200 million articles published in total.
- Manual scientific discovery remains difficult despite keyword search, vector embeddings, and recommender systems.
- LLMs can answer scientific questions quickly and broadly but risk hallucinations, costly verification, and outdated information after training cutoffs.
- Standard RAG can reduce hallucinations and improve question answering, but its fixed linear flow restricts adaptation to diverse scientific questions.
- PaperQA modularizes RAG so an agent LLM can iteratively adjust retrieval and evidence gathering before generating answers with references.
- PaperQA beat GPT-4 by 30 points on closed-book PubMedQA, rising from 57.9% to 86.3%, and was comparable to human experts on LitQA.
2 RELATED WORKS
Prior work established LLMs, agent systems, and retrieval-augmented models for scientific tasks, but these approaches commonly face outdated knowledge, reasoning difficulties, or fixed retrieval pipelines. PaperQA builds on these lines of work by combining external tools, iterative agents, and literature retrieval.
- LLMs for Natural Sciences: LLMs have been applied to scientific extraction and question answering, including biomedical and broader scientific literature.
- LLMs for Natural Sciences: Pre-trained LLMs can perform complex scientific tasks but often struggle with outdated knowledge, reasoning problems, and real-time information.
- Agents: LLM-agent systems iteratively select external tools and can correct previous behavior while completing a task.
- PaperQA’s workflow extends these foundations with agent-controlled retrieval and evidence synthesis over scientific papers.
- Evaluating LLM Scientists: Scientific QA benchmarks commonly test known information or answers given a context, while open-ended scientific tasks often require manual evaluation.
- Retrieval-Augmented LLMs: RAG systems combine document databases, query systems, and retrieved context, but are typically organized as fixed pipelines.
- Retrieval Methods: Retrieval methods include Bag-of-Words or BM25, pre-trained embeddings, trainable encoders, and phrase or keyword searches.
3 METHOD
PaperQA decomposes scientific question answering into agent-controlled search, evidence gathering, and answer generation. Its tools support repeated retrieval, relevance assessment, map-reduce summarization, and answer revision when evidence is insufficient.
- 3.1 PAPERQA: PaperQA retrieves relevant papers, gathers their text, and synthesizes information into a final answer.
- 3.1 PAPERQA: The system decomposes RAG operations into tools, allowing searches to be repeated with different keywords when evidence is insufficient.
- 3.1 PAPERQA: Map summarization gathers evidence from multiple sources before a reduce step produces the answer and intermediate evidence.
- 3.1 PAPERQA: LLM-generated relevance scores complement vector-embedding distances when ranking text chunks against the question.
- 3.2 TOOLS: Search queries scientific literature, parses retrieved papers into overlapping 4,000-character chunks, embeds them, and stores them in a vector database.
- 3.2 TOOLS: Gather evidence uses vector search and maximal marginal relevance, then summarizes retrieved chunks and collects the highest-scoring results.
- 3.2 TOOLS: The gather-evidence map step filters irrelevant context and can mitigate PDF parsing errors while supporting concurrent processing.
- 3.2 TOOLS: The answer stage combines retrieved context with information from an ask LLM before prompting an answer LLM to produce the response.
4 THE LITQA DATASET
LitQA is designed to test whether systems can retrieve and synthesize findings from recent full-text biomedical papers rather than rely on information already available to language models. It contains expert-written multiple-choice questions based on findings absent from abstracts, with human-reviewed distractors and expert comparison procedures.
- Existing scientific QA benchmarks do not adequately test answering from retrieved information, motivating the LitQA dataset.
- Dataset description: LitQA contains 50 expert-assembled multiple-choice questions from the biomedical domain.
- Dataset description: The dataset includes questions with two through seven answer choices, including 5 Yes/No questions and 23 questions with four possible answers.
- Data collection: Questions were written from papers published after September 2021 and target novel findings not presented in the abstract.
- Data collection: Distractors were created by researchers or an LLM, and each question was independently reviewed by at least one co-author.
- Human performance: Five biomedical researchers answered LitQA with internet access, three minutes per question, paper citations, and an unsure option.
5 EXPERIMENTS
The experiments evaluate PaperQA against LLMs, agents, commercial retrieval tools, humans, ablations, and citation-hallucination tests across LitQA and standard QA benchmarks. PaperQA generally outperforms comparison systems, matches expert-human agreement on LitQA, and produced no hallucinated citations in the reported test.
- 5.2 RESULTS: PaperQA outperformed competing LLMs, AutoGPT, and commercial scientific-research tools on LitQA, with performance comparable to expert humans.It also had the lowest rate of incorrectly answered questions among the evaluated tools.
- 5.2 RESULTS: PaperQA cost $0.18 per question and took about 2.4 hours to answer all questions, comparable to humans given 2.5 hours.The reported hourly cost was $3.75, excluding negligible operating costs.
- 5.2 RESULTS: Average human–PaperQA agreement was Cramer’s V = 0.67±0.02, matching average human–human agreement of 0.66±0.03 on LitQA.The comparison uses categorical correlation across response pairs.
- 5.2 RESULTS: Ablations showed major performance drops without multiple-choice options or with Semantic Scholar, while linear Vanilla RAG and no-search settings also reduced performance.The authors attribute these results to easier closed-form searching with answer options and the value of repeated agent tool use.
- 5.2 RESULTS: PaperQA produced no hallucinated citations across repeated tests, although the authors observed that citing a secondary source mentioned in a primary source could create risk.Citation evaluation covered paper existence, citation-detail accuracy, and relevance to the answer.
- 5.2 RESULTS: On standard QA benchmarks, PaperQA outperformed GPT-4 across all datasets and matched GPT-4 with ground-truth context in the reported comparison.The largest improvement occurred on context-obscured PubMedQAblind, while the gap was smaller on textbook-oriented MedQA.
6 LIMITATIONS
PaperQA depends on the correctness of its source papers and uses journal name and citation count as imperfect quality signals. Its models and benchmarks also face temporal instability, while prompt optimization for its multi-agent setting remains unresolved.
- PaperQA assumes that information in underlying research papers is correct, although that assumption may not hold.
- Journal name and citation count provide signals to the model but are not faithful indicators of source quality.
- LitQA questions may gain new correct answers or become invalid as science and literature availability change.
- Benchmark users should restrict papers used for answering LitQA questions to those published by September 15, 2023.
- Prompt optimization is unresolved for multiple agents with individual prompts, making prompt effects difficult to assess and optimal prompts unlikely.
7 CONCLUSION
PaperQA is a Retrieval-Augmented Generative agent for scientific question answering that dynamically retrieves full-text papers, iterates through evidence gathering, and synthesizes referenced answers. It performs strongly against LLMs and commercial products while matching human accuracy at lower cost, and supports multiple underlying models.
- PaperQA answers scientific questions better than other LLMs and commercial products while retaining accuracy on par with human researchers at lower cost.
- Recent LLMs hallucinated citations at rates of 40-60%, whereas no hallucinated citation was found in PaperQA responses.
- LitQA contains 50 questions requiring retrieval from full-text scientific papers.
- Strong LitQA results were obtained with Claude-2, GPT-3.5, and GPT-4, showing PaperQA works independently of the underlying model.
- PaperQA’s key attributes are dynamic RAG-tool use, full-text retrieval, and iterative answer refinement through agent decision-making.
A PAPERQA IMPLEMENTATION DETAILS
PaperQA’s implementation uses separate prompts and LLM roles for asking, summarizing, and answering, with tools for searching, gathering evidence, and generating answers. The prompts specify concise expert-facing responses, iterative retrieval, and optional latent-knowledge summaries.
- PaperQA uses separate ask, summary, and answer LLM roles within its implementation.
- The system prompt requests direct, concise, highly specific answers for expert readers and definitions for ambiguous terms or acronyms.
- The search tool accepts keyword strings with optional year ranges to retrieve additional papers.
- The gather-evidence tool asks for a specific question to increase evidence and relevant-paper counts.
- The answer tool proposes an answer using evidence from papers and may fail when better or different evidence is needed.
- A separate prompt asks an LLM to provide about 50 words of background knowledge without directly answering the question.
B AUTOGPT IMPLEMENTATION DETAILS
The AutoGPT comparison and retrieval metric use distinct evaluation setups: AutoGPT has bounded tool access, while the retrieval study generates targeted questions and measures whether APIs find, access, and parse source papers. Google Scholar and Semantic Scholar show the strongest retrieval ability.
- B AUTOGPT IMPLEMENTATION DETAILS: AutoGPT is stopped after 10 searches and uses Google search, write-to-file, and read-from-file tools with GPT-4-0314.
- B AUTOGPT IMPLEMENTATION DETAILS: The retrieval metric comprises 500 questions designed to have answers in only a small number of papers.
- B AUTOGPT IMPLEMENTATION DETAILS: The metric constructs questions by searching PubMed with 20,000 unique five-keyword tuples, retaining single non-review papers, and prompting GPT-4 for abstract-answerable questions.
- B AUTOGPT IMPLEMENTATION DETAILS: Questions are searched against the top 10 papers using 20 generated keywords.
- B AUTOGPT IMPLEMENTATION DETAILS: The retrieval evaluation measures cumulative recall for finding the original paper, with Google Scholar and Semantic Scholar showing outstanding ability.
- B AUTOGPT IMPLEMENTATION DETAILS: Figure 2 plots cumulative probabilities for finding, accessing, and parsing original papers, with lighter shading representing uncertainty.
C.2 FULL-TEXT RETRIEVAL METRIC
The evaluation measures retrieval and full-text processing performance across language models and search engines. Google Scholar outperformed Semantic Scholar, and the pipeline parsed almost all accessible papers.
- Retrieval evaluation: Retrieval AUC evaluates keyword generation and search across language models and search engines, using normalized area under the curve.The metric is derived from the corresponding retrieval probability figure.
- Full-text processing: The full-text evaluation additionally measures whether the pipeline can access papers and parse their PDFs.These stages are evaluated alongside keyword generation and search.
- Search-engine comparison: Google Scholar showed superior performance to Semantic Scholar in the full-text retrieval evaluation.The authors attribute this to Google Scholar searching article text, while Semantic Scholar is limited to titles, authors, and abstracts.
- Parsing outcome: The pipeline parsed almost all papers to which it had access.The authors also report marginally better GPT-4 performance and expect improved open-access links to raise final parsing performance.
E EVALUATIONS ON STANDARD QA BENCHMARKS
PaperQA is evaluated on evidence quality and on how parametric knowledge affects answers. Its discovered evidence is competitive with ground-truth context, while conflicting background information can cause abstention and supportive information can enable answers when context is insufficient.
- Evidence quality: PaperQA-discovered evidence is competitive with PubMedQA ground-truth context and can provide complementary information that further improves results.The comparison is reported in Table 8.
- Evaluation setup: PaperQA is prompted to answer from gathered context while citing supporting sources and optionally incorporating pre-trained knowledge.The extra background information represents the language model’s parametric knowledge.
- Contradictory background: When background information contradicted the gathered context about COVID-19 vaccine efficacy, PaperQA responded that it could not answer.Without the contradictory information, it produced a cited answer supporting vaccine effectiveness with conditions related to vaccine, variant, and time.
- Insufficient context: When relevant context was insufficient, supportive background information enabled PaperQA to answer that water typically becomes plasma above 20,000 Kelvin.Without the background information, PaperQA responded that it was unable to answer.
- Parametric knowledge: These examples show that parametric knowledge shapes responses differently when it conflicts with context versus when it supplements missing information.Contradictions prompted non-answering, whereas supportive information enabled detailed responses when context was lacking.