PaperlayerView exact revision on arXiv

Source-linked AI summary

Language agents achieve superhuman synthesis of scientific knowledge

Michael D. Skarlinski, Sam Cox, Jon M. Laurent, James D. Braza, Michaela Hinks, Michael J. Hammerling, Manvitha Ponnapati, Samuel G. Rodriques, Andrew D. White

arXiv:2409.13740v2cs.CLcs.AIcs.IRphysics.soc-ph

TL;DR

Scientific research needs literature systems that are factual, detail-sensitive, and evaluated on realistic tasks rather than narrow retrieval benchmarks. The paper develops LitQA2 and PaperQA2 for rigorous human–AI comparison, finding that PaperQA2 matches or exceeds experts across literature research tasks and supports contradiction discovery at scale.

  • Problem

    LLMs hallucinate unsupported information, can mishandle difficult scientific details, and lack realistic benchmarks covering the full literature.

  • Method

    The authors create LitQA2 and develop PaperQA2 as an agentic retrieval-augmented system that iteratively searches, gathers evidence, and generates answers.

  • Results

    PaperQA2 exceeds human precision on LitQA2, matches human accuracy, produces more factual summaries than Wikipedia on average, and identifies human-validated contradictions in biology literature.

  • Takeaways & Limitations

    PaperQA2 can support scientific literature research across retrieval, summarization, and contradiction detection tasks within the evaluated scope.

  • Takeaways & Limitations

    Detected contradictions are not necessarily errors because scientific findings can update and contradict one another over time.

Abstract

from arXiv · show

Language models are known to hallucinate incorrect information, and it is unclear if they are sufficiently accurate and reliable for use in scientific research. We developed a rigorous human-AI comparison methodology to evaluate language model agents on real-world literature search tasks covering information retrieval, summarization, and contradiction detection tasks. We show that PaperQA2, a frontier language model agent optimized for improved factuality, matches or exceeds subject matter expert performance on three realistic literature research tasks without any restrictions on humans (i.e., full access to internet, search tools, and time). PaperQA2 writes cited, Wikipedia-style summaries of scientific topics that are significantly more accurate than existing, human-written Wikipedia articles. We also introduce a hard benchmark for scientific literature research called LitQA2 that guided design of PaperQA2, leading to it exceeding human performance. Finally, we apply PaperQA2 to identify contradictions within the scientific literature, an important scientific task that is challenging for humans. PaperQA2 identifies 2.34 +/- 1.99 contradictions per paper in a random subset of biology papers, of which 70% are validated by human experts. These results demonstrate that language model agents are now capable of exceeding domain experts across meaningful tasks on scientific literature.

1 Introduction

The paper addresses factuality, detail, and benchmark limitations that constrain language models’ use in scientific literature research. It proposes rigorous human–AI evaluation across retrieval, summarization, and contradiction detection, introducing PaperQA2 as a multi-task literature agent.

  • LLMs can hallucinate unsupported information and overlook or misuse details in challenging scientific reasoning tasks.
  • Existing scientific-literature benchmarks often restrict evaluation to abstracts, fixed corpora, or directly supplied relevant papers.
  • The study compares AI systems and humans on full-literature question answering, cited scientific summarization, and contradiction detection.
  • PaperQA2 exceeds PhD students and postdocs on retrieval and summarization and identifies contradictions in biology papers at scale.

2 Answering scientific questions

The authors build LitQA2 to test scientific-literature retrieval beyond abstracts, then develop PaperQA2 as an iterative retrieval-augmented agent. PaperQA2 achieves high benchmark performance, exceeds humans in precision, and matches them in accuracy.

  • LitQA2 contains 248 multiple-choice questions whose answers require retrieving information from paper bodies rather than abstracts.
  • PaperQA2 decomposes retrieval-augmented generation into revisable search, evidence gathering, candidate-answer examination, and final response generation.
  • 85.2% ± 1.1% precision and 66.0% ± 1.2% accuracy were achieved on LitQA2, with “insufficient information” selected for 21.9% ± 0.9% of answers.
  • PaperQA2 outperformed other tested RAG systems in LitQA2 precision and accuracy, while most RAG systems outperformed non-RAG frontier models.
  • Performance on the original 147 questions did not differ significantly from performance on 101 later questions, indicating generalization to unseen questions.
  • Human annotators achieved 73.8% ± 9.6% precision and 67.7% ± 11.9% accuracy, while PaperQA2 exceeded human precision but did not differ significantly in accuracy.

3 Performance analysis of PaperQA2

Performance analysis identifies agentic retrieval, citation traversal, contextual summarization, and deep reranking as important design choices, while parser changes did not improve LitQA2 retrieval metrics. The analysis also exposes trade-offs between precision, accuracy, and computational cost.

  • The non-agentic variant had significantly lower accuracy than PaperQA2, supporting iterative agentic retrieval as a beneficial design choice.
  • 1.26 ± 0.07 searches and 0.46 ± 0.02 citation traversals per question occurred in the highest-accuracy LitQA2 runs.
  • Increasing final-context size improved accuracy but reduced precision; 15 contexts produced the highest precision, whereas 5 produced the highest accuracy.
  • Removing Citation Traversal increased accuracy while significantly increasing DOI recall at all PaperQA2 stages.
  • Adding contextual summarization significantly increased retrieval accuracy, but smaller models reduced accuracy when used for that step.
  • Increasing RCS depth from 1 to 10 significantly improved accuracy, with diminishing gains from 10 to the default depth of 30.
  • Grobid parsing and larger chunks did not significantly improve LitQA2 precision, accuracy, or recall.

4 Summarizing scientific topics

The study evaluates WikiCrow, an agentic PaperQA2 system that generates cited, Wikipedia-style gene articles, against existing Wikipedia articles using expert grading. WikiCrow produced fewer unsupported citations and reasoning errors, while the authors caution that this advantage is specific to the agentic RAG setting.

  • System design: WikiCrow combines multiple PaperQA2 queries to generate cited Wikipedia-style articles about human protein-coding genes.The queries address topics including gene structure, function, interactions, and clinical significance.
  • Evaluation: The evaluation compared 240 WikiCrow articles with matched, non-stub Wikipedia articles using 375 blinded statements graded by human experts.Experts classified statements as cited and supported, missing a citation, or cited and unsupported.
  • Results: 13.5% of WikiCrow statements were cited and unsupported versus 24.9% for Wikipedia (p = 0.0075).WikiCrow also failed to cite sources at a 3.9x lower rate than the human-written articles.
  • Error analysis: WikiCrow had fewer reasoning errors than Wikipedia, 12 versus 26 (p = 0.0144), but similar attribution errors, 10 versus 16 (p = 0.21).Reasoning errors included contradictions, over-extrapolations, or unsupported information; attribution errors involved incorrect or overly broad local citations.
  • Scope: The improved factuality is specific to the agentic RAG setting, not to standalone GPT-4 generating Wikipedia articles.The authors expect standalone GPT-4 to hallucinate at high rates in that task.

5 Detecting contradictions in the literature

ContraCrow uses PaperQA2 to extract claims, search the literature for contradictions, and score them on an 11-point Likert scale. On benchmarks and a random sample of biology papers, it detected contradictions with measurable accuracy, although human validation showed that many contradictions remain contextually valid and that the system can be overconfident.

  • System design: ContraCrow extracts claims from papers, evaluates them against the literature with PaperQA2, and returns contradiction judgments plus 11-point Likert scores.The system is designed for literature-scale contradiction detection, where comparing one claim against many papers is difficult for humans.
  • Benchmark evaluation: 73% accuracy, 88% precision, and 7% false positive rate were achieved on ContraDetect at Likert threshold 8.The benchmark was constructed from LitQA2 by converting question-answer pairs into supported or contradicted declarative statements.
  • Literature-scale application: 6.85% of 3,180 claims from 93 biology papers were classified as contradicted, yielding 2.34 ± 1.99 contradictions per paper at threshold 8.The analyzed papers contained an average of 35.16 ± 21.72 claims each; scores 8, 9, and 10 accounted for 2.89%, 3.77%, and 0.19% of claims, respectively.
  • Human validation: 70% of 100 expert-evaluated claims agreed with ContraCrow, corresponding to 1.64 validated contradictions per paper and an F1 score of 0.82.Claims scored 8 were no more likely to receive human agreement than claims scored 9 or 10.
  • Interpretation: Contradictory claims can remain contextually valid because later findings may update earlier results rather than simply prove them wrong.The paper illustrates this with claims about GBP localization and changing findings over time.

6 Conclusions

The paper presents a methodology for comparing AI systems with humans on realistic scientific research tasks and reports broad performance and scalability benefits for PaperQA2 and related agents. It also notes that PaperQA2 costs $1 to $3 per query despite being more expensive than lower-accuracy commercial systems.

  • Conclusions: PaperQA2 outperforms human experts on literature questions, produces more factual summaries than Wikipedia, and identifies contradictions at scale.The authors frame these results as evidence that literature-enabled agents can exploit scientific knowledge at greater scale than humans alone.
  • Conclusions: $1 to $3 per query is the reported absolute cost of PaperQA2, although it is more expensive than lower-accuracy commercial systems.The paper presents this cost as compatible with scaling literature-enabled agents.

7 Data Availability

The study provides code, evaluator responses, benchmark questions, contradiction claims, and generated WikiCrow articles to support replication and further research.

  • Available resources: Code for reproducing or modifying the algorithms, evaluator responses, contradiction claims, LitQA questions, and WikiCrow candidate statements are available.The code is provided through the PaperQA GitHub repository, while data are included in supplementary materials.
  • Available resources: All generated WikiCrow articles are available for download from a public Google Cloud Storage bucket.The paper provides the public storage location for bulk access.

8 Methods

PaperQA experiments use configurable retrieval, parsing, model, prompting, and agent-tool settings, with a multi-step workflow that searches papers, gathers evidence, traverses citations, and generates answers. The open-source package supports reproduction, while the reported experiments used additional institutional server infrastructure and retrieval access.

  • Reproducibility boundary: The open-source package omits the authors’ full HTTP-server infrastructure, including caching, orchestration, deployment, and scaling components.The authors state that these features improve scalability, measurability, and persistence but do not affect per-query performance.
  • Retrieval boundary: Paper Search depends on users’ access to full-text repositories and licensing-compatible links, so unavailable papers are skipped.The included implementation works with local files and accessible open-access or partner links, while users must implement their own retrieval for other access patterns.
  • Configuration: Model, parser, chunking, overlap, temperature, source-count, and evidence-count parameters are exposed as nested configuration objects.Examples include max_sources, consider_sources, model identifiers, parser preferences, chunk sizes, a fixed 750-character overlap, and zero generation temperatures.
  • Agent workflow: The agent uses paper search, evidence gathering, answer generation, and citation traversal tools, with citation traversal removed only in specified ablations.The configuration exposes the ordered tool list and supports experiments that alter the agent workflow.
  • Document processing: PaperQA supports sliding-window or semantic section chunking, with section-based parsing available through the Grobid parser.Chunking algorithm and parser preferences were varied in experiments and WikiCrow generation.
  • Agent workflow: PaperQA2’s workflow searches for papers, gathers evidence, follows cited papers, re-gathers evidence, and generates an answer after sufficient evidence is collected.The agent can search again or change phrases when evidence is insufficient, and typically generates an answer after five or more evidence pieces from multiple sources.

Paper Search Tool

The Paper Search tool generates keyword-and-date queries, retrieves candidate papers, parses them into chunks, and creates hybrid embeddings for evidence retrieval. PaperQA2 then reranks retrieved chunks through contextual LLM summaries before selecting context for cited answer generation.

  • Search and parsing: The agent generates keyword-and-date search queries, which are sent to services such as Semantic Scholar to obtain candidate papers.Queries may contain phrases, sentences, keywords, and optional year ranges.
  • Search and parsing: Retrieved papers are converted to text with Grobid or PyMuPDF and split into character- or section-based chunks.Chunking can use header metadata from Grobid when section parsing is enabled.
  • Embedding retrieval: Each chunk receives a hybrid embedding that combines dense and sparse representations for subsequent matching.
  • Embedding retrieval: The Gather Evidence tool ranks chunks by cosine similarity between the query embedding and document-chunk embeddings, then selects the top-k chunks for reranking and contextual summarization.
  • Reranking and contextual summarization: Reranking and contextual summarization maps an LLM over each top-k excerpt, producing a short relevant-information summary and a relevance score out of 10.
  • Answer generation: The resulting JSON summaries are reranked, and the final answer uses selected contextual summaries; the default was 15, while 5 produced maximal accuracy at the cost of precision.Context is injected into a prompted answer that must cite only supplied sources and may refuse when information is insufficient.

Citation Traversal Tool

The Citation Traversal tool expands evidence around highly scored retrieved papers by collecting future citers and past references, filtering them by citation overlap, and limiting the resulting set. Its overlap-based filtering retains papers shared across source-paper citation neighborhoods while prioritizing future citers when necessary.

  • Traversal inputs: Citation traversal begins from papers associated with contextual summaries scoring at least 8 and searches both future citers and past references.The traversal tool is intended to provide a fine-grained search around papers containing highly scored evidence.
  • Citation acquisition: Citation metadata is collected through Semantic Scholar and Crossref APIs, with four provider-direction calls per paper and best-effort deduplication across incomplete metadata.
  • Overlap filtering: The default overlap fraction was 1/3, and traversal imposed a 12-paper limit, selecting within crowded bins using the number of future citers.
  • Overlap filtering: The algorithm obtains citation sets, computes an overlap threshold from the overlap fraction and source-set size, then filters papers by shared citation overlap.The returned set excludes papers already present among the source papers.
  • Algorithm: The paper defines the overlap threshold as θo = ⌈α × |D|⌉ in the traversal procedure, with filtering stopping when overlap falls below threshold or the output limit is reached.

8.2 LitQA

LitQA2 is a manually constructed retrieval benchmark whose human comparison allowed substantial time and ordinary search tools. PaperQA2 was evaluated across repeated runs and showed strong precision and accuracy, with similar accuracy on newly added questions and improvements over the prior PaperQA configuration.

  • Question construction: LitQA2 questions require retrieving answers from the main text of recent scientific papers, often excluding information from abstracts and allowing unanswerable items.
  • Question construction: Questions were calibrated through author review, and LitQA2 expanded from 47 LitQA questions through two stages to 248 total questions.
  • Human evaluation: Human evaluators could use internet and institutional journal-search tools, had up to a week per quiz, and were asked not to use AI tools.
  • Measurement: PaperQA2 was evaluated with three full runs, randomized answer order, and automated extraction and grading of its selected answer letter.Responses selecting insufficient information were graded as unsure except when the benchmark’s ideal answer was null.
  • Generalization: PaperQA2 performed similarly on the original 147 questions and 101 newly added questions, indicating that the first-stage optimizations generalized to unseen questions.
  • Configuration analysis: Grobid parsing and larger chunk sizes did not significantly increase precision, accuracy, or recall on LitQA2.The authors attribute this retrieval-task insensitivity to the frequent need for only a single passage from a paper.
  • Results: 36.7% accuracy and 76.5% precision were reported for the prior PaperQA configuration, with PaperQA2 showing a large improvement in both metrics, particularly accuracy.

8.3 WikiCrow

WikiCrow generates cited gene-article statements using multiple PaperQA2 queries and evaluates them against Wikipedia through blinded expert grading. The evaluation distinguishes citation presence from whether the cited literature supports the stated information, including rules for unsupported specificity and broader context.

  • Generation: WikiCrow articles are generated from five linked queries: four to PaperQA2 and one to GPT-4-Turbo, with each PaperQA2 query producing a different article section.WikiCrow differs from LitQA2 runs mainly through Grobid parsing and changes to the RCS step.
  • Evaluation design: The comparison samples statements from 300 human-gene articles, yielding 500 Wikipedia statements and a matching set of 500 WikiCrow statements.
  • Evaluation design: References are extracted from Wikipedia links and WikiCrow parenthetical citations, then matched to source DOIs or document names for statement-level evaluation.
  • Expert grading: Four evaluators graded shuffled statements while blinded to whether each came from Wikipedia or WikiCrow.Statements were judged for both accuracy and appropriate citation.
  • Scoring criteria: The grading protocol separately asks whether a statement is cited and whether its information is correct according to the cited literature.Global truth is not the scoring target when the cited source does not support the statement.
  • Scoring criteria: Highly specific claims not explicitly stated in a citation are marked incorrect as cited, while unrelated metadata is marked not applicable.Broader context can be accepted when it is undergraduate biology knowledge.
  • Expert grading: Four expert researchers graded 375 evaluations, with agreement on 31 of 40 statements reviewed by multiple evaluators.The remaining 9 overlapping evaluations were disagreements.
  • Evaluation scope: The authors used heuristics for WikiCrow accuracy because evaluating the generated summaries was complex and time-consuming.Grobid parsing and gene-name extraction were intended to address gene confusion and tabular-data errors observed during development.

8.4 Contradiction Detection Methods

ContraCrow extracts claims from papers, searches the literature with PaperQA2 for contradictions, and produces graded contradiction outputs. Human evaluation procedures assess both claim quality and agreement with model-detected contradictions.

  • ContraCrow pipeline: Claims are extracted from paper chunks split within sections, with section and paper titles retained to preserve context.Chunks are limited to 5,000 characters without overlap before claim-extraction processing.
  • ContraCrow pipeline: Each claim is independently sent to PaperQA2, which searches the literature for relevant evidence and returns contradiction reasoning with an 11-point grade.The prompt also requires a binary contradiction determination.
  • Benchmark construction: ContraDetect creates factual and contradictory statements from LitQA2 data, while a separate hand-designed set contains 42 claims intended to have no supporting evidence.Claims with discovered evidence were removed from the no-evidence dataset.
  • Evaluation datasets: The biology-paper analysis sampled 100 papers, of which 93 remained after seven failed to parse, and processed the generated claims in batches of 1,000.The sampled papers did not overlap with papers used for LitQA2 questions.
  • Human evaluation: Human contradiction validation sampled 100 high-scoring contradictions and divided them evenly among five expert evaluators without overlap.The sample contained 50 claims scored 8 and 50 scored at least 9 by ContraCrow.
  • Human evaluation: For binary comparison, ContraCrow scores of at least 8 were treated as positive, while Likert evaluations retained raw scores.A separate 30-claim detection sample included low-, medium-, and high-scoring cases, with each claim evaluated by at least two evaluators.
Loading 2409.13740v2…