Language agents achieve superhuman synthesis of scientific knowledge

1 Introduction

• Scientific use of LLMs remains uncertain because they can hallucinate, miss or misuse details in difficult reasoning, and existing benchmarks poorly represent real research or compare performance with humans.The cited limitations of benchmarks include abstract-only tasks, fixed-corpus retrieval, or providing the relevant paper directly.Source: §1 · p. 1
• The evaluation compares AI systems and humans on literature-wide question answering, cited Wikipedia-style scientific-topic summaries, and contradiction detection across claims and the literature.The authors describe this as, to their knowledge, the first robust procedure for evaluating one AI system across multiple real-world literature-search tasks.Source: §1 · pp. 1–2
• PaperQA2 exceeds PhD students and postdocs on retrieval and summarization, and its contradiction detection identifies contradictions in biology papers at scale.The introduction gives an example involving conflicting findings about the ZNF804A rs1344706 allele and schizophrenia-related brain structure [9, 10].Source: §1 · p. 2

2 Answering scientific questions

• LitQA2 contains 248 expert-generated multiple-choice questions whose answers require retrieving obscure findings from paper bodies rather than abstracts.Questions were excluded if an AI system or human annotator could answer them using an alternative source; questions could also be unanswerable, and systems could abstain. Precision measures correctness among answered questions, accuracy measures correctness across all questions, and recall tracks attribution to the correct source DOI.Source: §2 · pp. 2–3
• PaperQA2 decomposes RAG into agent tools for searching papers, gathering and contextualizing evidence, and generating answers, with citation traversal added to improve recall.Evidence gathering combines top-k dense-vector retrieval with LLM reranking and contextual summarization, enabling the agent to revise searches and examine more text per question.Source: §2 · p. 3
• PaperQA2’s 85.2% ± 1.1% precision exceeded expert annotators’ 73.8% ± 9.6%, while its 66.0% ± 1.2% accuracy did not differ significantly from humans’ 67.7% ± 11.9%.The precision difference was significant (t(8.6) = 3.49, p = 0.0036), whereas the accuracy difference was not (t(8.5) = −0.42, p = 0.66).Source: §2 · p. 3
• PaperQA2 outperformed the other tested RAG systems on LitQA2 in both precision and accuracy; all tested RAG systems except Elicit outperformed frontier non-RAG models on both metrics.PaperQA2 used “insufficient information” for 21.9% ± 0.9% of answers and parsed an average of 14.5 ± 0.6 papers per question.Source: §2 · p. 3
• PaperQA2’s accuracy did not differ significantly between the original 147 LitQA2 questions and 101 questions generated after most engineering changes, supporting generalization to unseen questions.This comparison is reported in Table 2.Source: §2 · p. 3

3 Performance analysis of PaperQA2

• The non-agentic fixed-sequence system had significantly lower accuracy, supporting the agent’s ability to revise searches after seeing retrieved papers.The accuracy difference was t(3.7) = 3.41, p = 0.015; high-accuracy runs averaged 1.26 ± 0.07 searches and 0.46 ± 0.02 citation traversals per question.Source: §3 · pp. 3–4, Figure 2 · p. 4
• 15 contexts gave the highest precision, whereas 5 gave the highest accuracy, reflecting a tradeoff between useful and distracting context.Increasing context count can improve the chance of including key evidence but also add irrelevant context that reduces precision.Source: §3 · pp. 4–5
• Ablating Citation Traversal increased accuracy and significantly increased DOI recall at all PaperQA2 stages.The accuracy increase was t(2.55) = 2.14, p = 0.069; DOI recall increased with t(3) = 3.4, p = 0.022.Source: §3 · p. 5
• Adding RCS to traditional RAG significantly increased retrieval accuracy, but effective RCS depended on model capability.GPT-3.5-Turbo (RCS) and Llama3 (70B) decreased overall accuracy relative to using no model; GPT-4-Turbo had the highest LitQA2 accuracy, while Claude-Opus25 had the highest LitQA2 precision, not significantly above Gemini-1.5-Pro or GPT-4-Turbo.Source: §3 · p. 5
• Accuracy significantly increased as RCS ranking depth grew from 1 to 10 (t(2.15) = 5.44, p = 0.014), with diminishing gains from 10 to 30.A deep (>10) RCS ranking list with a high-performing LLM was crucial to achieving human-level LitQA2 accuracy.Source: §3 · p. 5
• Grobid parsing and larger chunks did not significantly improve LitQA2 precision, accuracy, or recall.The authors suggest retrieval may be insensitive to parser changes when only one passage from a paper is needed; better parsing was crucial for extracting table data in WikiCrow.Source: §3 · p. 5

4 Summarizing scientific topics

• WikiCrow combines multiple PaperQA2 calls on gene structure, function, interactions, and clinical significance into cited Wikipedia-style articles, evaluated against matched non-stub Wikipedia articles.It generated 240 articles averaging 1219.0 ± 275.0 words, compared with 889.6 ± 715.3 words for Wikipedia; generation averaged 491.5 ± 324.0 seconds and cost $4.48 ± $1.02 per article. Experts blindly graded 375 sampled statements from paired articles as cited and supported, uncited, or cited and unsupported.Source: §4 · p. 5
• 3.5% vs. 13.6% of WikiCrow and Wikipedia statements were uncited, respectively, a 3.9x lower uncited rate for WikiCrow.The difference was significant (p < 0.001).Source: §4 · p. 5
• 13.5% vs. 24.9% of statements were “cited and unsupported” for WikiCrow versus Wikipedia, while cited-statement precision was 86.1% vs. 71.2%, respectively.WikiCrow precision is defined as cited-and-supported statements divided by all cited statements.Source: §4 · p. 5, Figure 3 · p. 6
• 12 vs. 26 reasoning errors were found in WikiCrow and Wikipedia, respectively, while attribution errors were similar at 10 vs. 16.The cited-and-unsupported category included inaccurate statements and accurate statements with inappropriate citations; reasoning errors were defined as contradictions, over-extrapolations, or claims unsupported by included citations.Source: §4 · pp. 5–6, Figure 3 · p. 6
• The finding that language models made fewer reasoning errors than Wikipedia authors or editors applies specifically to this agentic RAG setting.The passage cautions that GPT-4 generating articles on its own would still be expected to hallucinate at high rates.Source: §4 · pp. 6–7

5 Detecting contradictions in the literature

• The work’s stated novelty is detecting contradictions without a restricted corpus and evaluating them with human experts rather than against a benchmark.Source: §5 · p. 7
• ContraCrow extracts claims with LLM calls, asks PaperQA2 to find literature contradictions, and ranks answers on an 11-point Likert scale; ContraDetect pairs LitQA2-derived contradicted and supported statements.Source: §5 · p. 7
• 2.34 ± 1.99 contradictions per paper were detected across 93 biology papers, and experts validated 70%, yielding 1.64 validated contradictions per paper.The 1.64 figure is presented as a lower bound on human-validatable contradictions in biology literature.Source: §5 · p. 7, Figure 4 · p. 8
• AUC 0.842; at threshold 8, ContraCrow achieved 73% accuracy, 88% precision, and a 7% false positive rate, and answered lack-of-evidence (5) for 98% of 42 no-evidence statements.Source: §5 · p. 7, Figure 4 · p. 8
• In contradiction detection, experts agreed with each other on 75.5% ± 13.43% of claims (mean ± SD; 10 pairwise comparisons), versus 60.42% ± 5.99% agreement with ContraCrow (mean ± SD; 5 annotators), a significant difference (p = 0.015).Experts lacked access to ContraCrow’s reasoning but received the same top 15 paper chunks identified as relevant by PaperQA2.Source: §5 · pp. 8–9
• Detected contradictions do not necessarily mean a claim is wrong: findings may update over time, and contradictory claims can remain contextually valid.Source: §5 · p. 9

• PaperQA2 outperforms human experts on questions across scientific literature, produces more factual summaries than Wikipedia on average, and identifies contradictions at scale.Source: §6 · p. 9
• The methodology compares or validates AI systems against human performance on realistic research tasks; one tested expert unsolicitedly called contradiction detection their hardest task.That expert had also been tested on a large battery of other benchmarks.Source: §6 · p. 9
• PaperQA2 costs $1 to $3 per query despite being more expensive than lower-accuracy commercial systems; scaling it and agents like WikiCrow and ContraCrow could expand access to literature’s latent knowledge.Source: §6 · p. 9

7 Data Availability

• Replication code and research data—including evaluator responses, contradiction claims, LitQA questions, and WikiCrow candidate statements—are available via GitHub and the supplementary materials.All generated WikiCrow articles are available in a public Google Cloud bucket.Source: §7 · p. 9

8.1 PaperQA Implementation and Parameters

• The experiments used a featureful institutional HTTP server, but its added infrastructure did not affect per-query performance; PaperQA and experiment configurations are available for reproduction and customization.The server added authentication, caching, orchestration, storage, monitoring, and deployment features; users would need to provision and configure those assets to run the same infrastructure.Source: §8.1 · p. 12
• Paper Search is limited by full-text access and licensing: it searches by agent-specified keywords, uses papers available locally or through open-access or partner links, and skips inaccessible results.The included PaperQA implementation provides a stub for users to implement retrieval based on their own access to full-text papers.Source: §8.1 · p. 12
• PaperQA exposes nested workflow configurations and ordered tool lists; prompts can disable RCS, and the No RCS Model ablation used that option alongside WikiCrow prompts.The usual tool list included gather_evidence, paper_search, generate_answer, and citations_traversal, with citation traversal excluded in the No Cit. Trav. and No Agent runs; prompts and settings also represented the agent’s current status.Source: §8.1 · pp. 12–13, §8.1.1 · p. 13
• The answer-generation and evidence-summary LLMs were varied in model experiments, while agent orchestration was fixed to gpt-4-turbo-2024-04-09.The LLM parameters accept valid Anthropic, OpenAI, or Gemini model identifiers.Source: §8.1 · p. 12
• Answer generation used five top-ranked sources in the Answer cutoff @ 5 experiment and 15 in other runs; evidence gathering used top-k 30 for LitQA and 25 for WikiCrow.The top-k value was X in the Top-k Rank @ X experiment, and pre-gather MMR was set to 0.9 for WikiCrow and 1.0 for LitQA.Source: §8.1 · p. 13
• Parsing used PyMuPDF by default for ablations and Grobid for WikiCrow; parser, chunk size, and chunking method varied in Figure 6, while chunk overlap was fixed at 750 characters.The sections chunking method is supported only with Grobid and uses semantic document sections where possible.Source: §8.1 · p. 13

8.1.1 Tool implementations

• The agent was instructed to search for papers, gather evidence, collect papers cited in evidence, and re-gather; it should answer after five or more evidence pieces from multiple sources or after a few attempts.When evidence is insufficient, the prompt prefers searching for papers or collecting cited papers, then gathering evidence again; the answer tool output is visible to the user.Source: §8.1.1 · p. 13

Paper Search Tool

• The Paper Search tool prompts the agent to form a keyword query that may be a phrase, sentence, or general keywords, optionally ending with a year or year range.The prompt gives “machine learning 2020” and “machine learning 2010-2020” as examples.Source: Paper Search Tool · p. 14
• Candidate papers from services such as Semantic Scholar5 are parsed into text, split into chunks, and embedded using concatenated dense and sparse keyword vectors.For these experiments, the dense embedding was OpenAI text-embedding-large-3, paired with a normalized 256-dimensional modulus-encoded keyword vector.Source: Paper Search Tool · p. 14

Gather Evidence Tool

• The Gather Evidence tool embeds the user query with the same algorithm, ranks document chunks by cosine similarity, and selects the top-k for RCS.Source: Gather Evidence Tool · p. 14
• PaperQA’s Gather Evidence tool combines top-k retrieval with reranking and contextual summarization, which the authors describe as a novel combination.The authors say top-k vector retrieval is widely implemented, while combining reranking with contextual summaries provides novel benefits.Source: Gather Evidence Tool · p. 14
• For each top-k chunk, an LLM summarizes relevant information without answering directly and returns a JSON summary with an integer relevance score.The excerpt and user query are supplied as inputs to the prompt.Source: Gather Evidence Tool · p. 14
• 200–400 tokens per contextual summary, versus 2,250-token document chunks, improves token efficiency without reducing summarization efficacy across 750–3,000-token chunk sizes.Summaries are scored for relevance and passages are reranked; structured JSON or XML can preserve metadata such as gene names to reduce hallucination and confusion. Because RCS runs in parallel, it supports deeper ranking; the study used greater RCS depth and its best-performing model.Source: Gather Evidence Tool · p. 14

Generate Answer Tool

• 5 contextual summaries maximized answer accuracy, while the study’s default was 15, at a cost in precision.The answer prompt requires context-based responses, citations from valid source keys, and “I cannot answer” when the context is insufficient; answers are written in concise Wikipedia style.Source: Generate Answer Tool · p. 15

Citation Traversal Tool

• Papers with contextual-summary scores of at least 8 seed one-degree traversal to both past references and future citers.These seed papers are Dprev; the algorithm retrieves citations in the selected direction before filtering candidates.Source: Citation Traversal Tool · pp. 15–16
• Past references are fetched through Semantic Scholar and Crossref, while future citers use Semantic Scholar; four API calls are made per paper.Because provider metadata often omits titles or DOIs, records are best-effort deduplicated using casefolded titles and lowercased DOIs.Source: Citation Traversal Tool · p. 15
• For six example source DOIs, overlap bins contained 1 DOI cited by four papers, 5 cited by three, 29 cited by two, and 428 cited by one.The 12-paper limit and within-bin ranking determine which candidates are retained when a bin exceeds available space.Source: Citation Traversal Tool · p. 16
• The overlap threshold is θo = ⌈α × |Dprev|⌉; filtering keeps highest-overlap bins first, excludes source papers, and limits traversal to 12 papers.When a bin exceeds the remaining limit, candidates are prioritized by future-citer count; the data-collection overlap fraction was α = 1/3.Source: Citation Traversal Tool · p. 16

8.2.1 Question Construction and Human Evaluation

• Questions targeted recent papers and required reasoning from main-text context, with drafts checked for model difficulty and whether search made answers trivial.Authors and contracted experts created multiple-choice questions that could not be answered from the title or abstract alone; authors also reviewed early drafts and continued reviewing questions before corpus inclusion.Source: §8.2.1 · p. 16
• Human evaluators could use institutional or internet search tools and had up to a week per quiz, while being asked not to use AI tools.Nine evaluators completed two 20-question quiz rounds, providing 266 answers across 248 questions.Source: §8.2.1 · p. 17
• Human answers to the initial 147 questions in a later quiz round were excluded after Google-indexed aggregators made those questions trivially searchable.This contamination was not found to affect PaperQA2, which does not use Google web search.Source: §8.2.1 · p. 17

8.2.2 PaperQA measurement

• PaperQA2 was tested in three full runs with randomized answer order, and an evaluation LLM extracted answer letters for grading; insufficient-information answers were marked unsure except for null-answer questions.The repeated runs controlled for inference noise even at zero temperature.Source: §8.2.2 · p. 17
• PaperQA2 performed similarly in accuracy on the original LitQA questions and the 101 newly added questions, indicating no overfitting to the original data.The full benchmark combined the original 47 LitQA questions with two additions of 100 and 101 questions, for 248 total.Source: §8.2.2 · p. 17
• PaperQA2 showed large improvements over prior PaperQA in accuracy and precision, particularly accuracy; the prior system scored 36.7% accuracy and 76.5% precision.The prior configuration used GPT 3.5 Turbo, shallower ranking, no citation traversal, and no Grobid parsings.Source: §8.2.2 · p. 18

8.3 WikiCrow

• PaperQA2 generated WikiCrow article sections from four queries, while a fifth query used GPT-4-Turbo; Grobid parsing and gene-name extraction were intended to improve table reading and reduce hallucinations.The RCS step was modified to extract the gene name that is the focus of each paper chunk.Source: §8.3 · p. 19
• The evaluation matched 500 Wikipedia statements with WikiCrow statements from the same gene articles, covering 240 genes, then shuffled them for four blinded evaluators.References were reformatted with random numbers to hide statement origins.Source: §8.3 · p. 19
• Evaluators scored whether statements were cited and whether each sentence’s information was correct according to its cited literature, not whether it was globally correct.Statements containing multiple sentences were judged sentence by sentence; a single unsupported sentence could count against the statement.Source: §8.3 · p. 19
• 77.5% (31) of the 40 statements evaluated by multiple graders had agreement, while 22.5% (9) had disagreement.Four expert researchers graded 375 evaluations; statements not completed in a contiguous block from the start were excluded from analysis.Source: §8.3 · p. 20
• Compared with the prior study, this evaluation matched gene-article subsets, included statements citing multiple sources, and used blinded external evaluators.The prior study focused on single sentences with one source, which limited its sample.Source: §8.3 · p. 20
• 2/171 evaluator-flagged statements had gene-name confusion, no table-extraction errors were reported, and Grobid parsing averaged 8,903 tokens per paper versus PyMuPDF’s 16,040 across 5,363 papers.The authors attributed token savings to reduced excess whitespace and removal of reference sections.Source: §8.3 · p. 20

8.4 Contradiction Detection Methods

• Claims are split by paper section into chunks of up to 5,000 characters, extracted by an LLM, and retained only when a filtering LLM scores them at least 8/10.PaperQA2 then searches literature independently for each claim and returns reasoning plus an 11-point score; experiments used Claude 3.5 Sonnet, a 7,000-character chunk size, 250-character overlap, and a default decision boundary of 8.Source: §8.4 · p. 21

8.4.1 ContraDetect

• ContraDetect rephrased LitQA2 questions and ideal answers as factual statements and questions with distractors as incorrect statements, then evaluated them with ContraCrow.The benchmark also included 42 hand-designed no-evidence statements, screened to remove any for which ContraCrow found evidence.Source: §8.4.1 · p. 21

8.4.2 Human Evaluation

• The paper’s human-evaluation corpus comprised 93 parsed biology papers sampled from a local database, with generated claims processed in batches of 1,000.Seven of the 100 randomly selected papers failed to parse, and the corpus did not overlap with papers tied to LitQA2 questions.Source: §8.4.2 · p. 21
• For human validation, five experts assessed 100 ContraCrow-positive cases using its reasoning and cited chunks, with analysis based on agreement with its conclusion.The sample included 50 claims scored 8 and 50 scored at least 9; experts were instructed to base judgments on the provided evidence and reasoning.Source: §8.4.2 · p. 21
• For human detection, five evaluators reviewed 30 claims across low and high ContraCrow score ranges, with each claim assessed by at least two evaluators without seeing the model’s reasoning.Evaluators assigned a contradiction Likert grade and binary label, relying on the claim and provided literature chunks.Source: §8.4.2 · pp. 21, 23