PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs

Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D'arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Dan Weld, Doug Downey, Wen-tau Yih, Pang Wei Koh, Hannaneh Hajishirzi

arXiv:2411.14199v1cs.CLcs.AIcs.DLcs.IRcs.LG

TL;DR

Researchers face a growing scientific literature and language models that can produce fabricated citations or rely on outdated pretraining data. OpenScholar retrieves scientific passages and iteratively synthesizes cited answers; on ScholarQABench, it outperformed LM-only and retrieval-augmented systems, while expert assessments favored its responses over expert-written ones in some settings.

  • Problem

    Researchers must synthesize a rapidly growing scientific literature, while prior literature-synthesis evaluations were often small-scale, single-domain, or simplified multiple-choice tasks.

    Source: §1 · p. 1, §3 · p. 7

  • Method

    OpenScholar retrieves passages from a 45-million-paper datastore and uses retrieval-augmented self-feedback to iteratively refine cited answers.

    Source: §1 · p. 2, §2 · p. 4

  • Results

    OpenScholar outperformed LM-only and retrieval-augmented pipelines, and OpenScholar-GPT4o improved correctness by 12% over GPT4o alone.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Expert assessments found OpenScholar outputs more comprehensive, well-organized, and useful for literature synthesis than expert-written answers in some cases.

    Source: §1 · p. 3

    Human preference assessments may emphasize overall writing quality rather than factual correctness or citation accuracy.

    Source: LIMITATIONS · p. 17

Abstract

from arXiv · show

Scientific progress depends on researchers' ability to synthesize the growing body of literature. Can large language models (LMs) assist scientists in this task? We introduce OpenScholar, a specialized retrieval-augmented LM that answers scientific queries by identifying relevant passages from 45 million open-access papers and synthesizing citation-backed responses. To evaluate OpenScholar, we develop ScholarQABench, the first large-scale multi-domain benchmark for literature search, comprising 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience, and biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o hallucinates citations 78 to 90% of the time, OpenScholar achieves citation accuracy on par with human experts. OpenScholar's datastore, retriever, and self-feedback inference loop also improves off-the-shelf LMs: for instance, OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's 32%. We open-source all of our code, models, datastore, data and a public demo.

1 INTRODUCTION

OpenScholar pairs scientific-literature retrieval and iterative response refinement with ScholarQABench, a benchmark for evaluating open-ended literature synthesis. The introduction reports system gains on that benchmark and expert preferences for OpenScholar responses over expert-written ones.

1 INTRODUCTION

  • Researchers face a growing volume of scientific literature, while LMs can produce fabricated citations, rely on outdated pretraining data, and lack transparent attribution.In the authors’ experiments, GPT-4 fabricated citations in 78–90% of cases across computer science and biomedicine.Source: §1 · p. 1
  • OpenScholar uses a specialized scientific-paper datastore and retrievers, then synthesizes retrieved passages with citations and iteratively refines responses using natural-language feedback.The pipeline also generates training data for smaller, more efficient models.Source: §1 · pp. 2–3
  • ScholarQABench contains 2,967 literature-synthesis questions and 208 expert-written long-form responses spanning computer science, physics, biomedicine, and neuroscience.Its evaluation combines automated metrics and human assessments of citation accuracy, factual correctness, coverage, coherence, and overall quality.Source: §1 · p. 3
  • OpenScholar outperformed LM-only and retrieval-augmented pipelines, and OpenScholar-GPT4o improved correctness by 12% over GPT4o alone.OpenScholar also outperformed PaperQA2 and Perplexity Pro, with reported improvements of 6% and 10%, respectively.Source: §1 · p. 3
  • Experts preferred OpenScholar-GPT-4o and OpenScholar-8B over expert-written responses 70% and 51% of the time, respectively.The assessment involved 16 scientists comparing system outputs against 108 expert-written responses.Source: §1 · p. 3
  • The released OpenScholar ecosystem includes code, trained retrievers, an LM checkpoint, the datastore, ScholarQABench, expert-evaluation tools, and a public demo.OpenScholar-8B is described as an open retrieval-augmented LM that does not rely on proprietary LMs or retrieval systems.Source: §1 · p. 3

2 OPENSCHOLAR: OPEN RETRIEVAL-AUGMENTED LM TO SYNTHESIZING SCIENTIFIC LITERATURE

OpenScholar combines a scientific-literature datastore, trained retrieval components, and a language model to generate cited answers. Its inference-time feedback and retrieval process also supplies synthetic data for training specialized models.

2 OPENSCHOLAR: OPEN RETRIEVAL-AUGMENTED LM TO SYNTHESIZING SCIENTIFIC LITERATURE

  • OpenScholar uses a datastore of 45 million scientific papers, trained retriever and reranker components, and an LM; the pipeline also generates training data for specialized models.The datastore uses embeddings, and the LM’s initial draft is iteratively enhanced through retrieval-augmented self-feedback.Source: §2 · p. 4

2.1 OPENSCHOLAR RETRIEVAL PIPELINE

  • The retrieval pipeline draws passages from the datastore and external sources, then uses a cross-encoder reranker to select the top N relevant passages.The datastore contains 45 million papers split into 250-word blocks; external retrieval includes Semantic Scholar abstracts and web-search results.Source: §2.1 · p. 5

2.2 INFERENCE: ITERATIVE GENERATION WITH RETRIEVAL-AUGMENTED SELF-FEEDBACK

  • The LM drafts an answer and up to three natural-language feedback suggestions, retrieves more passages when feedback identifies missing information, then refines the answer and verifies its citations.Feedback is incorporated sequentially rather than regenerating after each suggestion, which the authors found effective while avoiding additional latency.Source: §2.2 · p. 6, §2.3 · p. 6

2.3 TRAINING: HIGH-QUALITY SYNTHETIC DATA GENERATION WITH INFERENCE PIPELINE

  • Training data are generated from sampled papers by creating information-seeking queries and running the OpenScholar inference pipeline; pairwise and rubric filters remove lower-quality outputs.The filtering process finds that the initial output is preferred over the iterated output around 20% of the time, often because of over-editing or increased redundancy.Source: §2.3 · pp. 6–7
  • Training mixes answer generation, feedback generation, and feedback incorporation examples with general-domain and scientific instruction data, then trains Llama 3.1 8B Instruct.Half of the mixed training data comes from scientific domains and half from general-domain data.Source: §2.3 · p. 7

3 SCHOLARQABENCH: REALISTIC LITERATURE REVIEW EVALUATION BENCHMARK ANNOTATED BY PH.D. EXPERTS

ScholarQABench addresses the difficulty of building expert-quality literature-review datasets and reliably evaluating long-form scientific answers. It combines varied tasks and disciplines with multi-paper questions and a multifaceted evaluation pipeline.

3.1 DATA CURATION

  • ScholarQABench combines expert-annotated datasets across four disciplines and task formats, including multi-paper questions requiring retrieval from the open-access full-text collection.Four datasets specifically require reasoning over multiple retrieved papers.Source: §3.1 · p. 7, §3 · p. 7

3.1.1 SINGLE-PAPER TASKS

  • Single-paper tasks adapt SciFact and PubMedQA into open retrieval, while QASA evaluates end-to-end answers to detailed questions about a target scientific paper.SciFact uses 1.4K biomedical expert-written claims; SciFact and PubMedQA discard their original evidence passages or abstracts.Source: §3.1.1 · p. 8

3.1.2 MULTI-PAPER TASKS

  • SCHOLARQA-CS contains 100 Ph.D.-expert-written questions expected to require multiple papers, with rubrics listing key answer ingredients and supporting source quotes.Questions average 4.4 key ingredients, each supported by 4.4 quotes.Source: §3.1.2 · pp. 8–9
  • For SCHOLARQA-CS rubric scoring, inter-annotator score correlation is 79.3 with the general criterion and 59.5 without it.Agreement was measured on a subset of 10 randomly sampled questions.Source: §3.1.2 · p. 9
  • Biomedicine and neuroscience contribute 2,759 expert-written literature-review questions; SCHOLARQA-MULTI adds 108 questions with expert-written, cited answers across four domains.SCHOLARQA-MULTI annotators were instructed not to use LLM-based systems.Source: §3.1.2 · p. 9

3.2 METRICS AND EVALUATION PROTOCOLS

  • Correctness uses task-specific accuracy, ROUGE-L, or expert rubrics, while citation accuracy measures citation precision, recall, and F1 across tasks.SCHOLARQA-CS rubric scores combine general criteria weighted 40% and annotation-driven criteria weighted 60%.Source: §3.2 · p. 10
  • Relevance, topic coverage, depth, and organization are scored with Prometheus v2 rubrics; human evaluation also assesses overall usefulness on SCHOLARQA-MULTI.The fine-grained evaluation is limited to a task with human-annotated reference answers.Source: §3.2 · pp. 9–10

4 EXPERIMENTS AND RESULTS

The experiments compare OpenScholar with language-model and retrieval baselines, then analyze component ablations and context size. OpenScholar performs strongly across single- and multi-paper tasks, while retrieval-free systems show citation and coverage weaknesses.

4.1 EXPERIMENTAL DETAILS

  • Evaluation compares open-weight and proprietary LMs, retrieval variants, and PaperQA2; the PaperQA2 replication lacked private or license-protected papers.Perplexity citation information could not be retrieved because its search lacked an API.Source: §4.1 · p. 10

4.2 RESULTS

  • Across ScholarQABench, OpenScholar outperforms GPT-4o, standard RAG, and specialized literature-review systems including PaperQA2.Source: Table 2 · p. 11, §4.2 · p. 11
  • On single-paper tasks, OS-8B and OS-70B outperform Llama 3.1 models in correctness and citation, while OS-70B matches or beats GPT-4o on PubMedQA and QASA.Source: §4.2 · p. 11
  • On SCHOLAR-CS, OS-GPT4o improves correctness by 12.7 points over GPT-4o and 5.3 over standard RAG; OS-8B also beats proprietary systems at much lower cost than PaperQA2.Source: §4.2 · pp. 11–12

4.3 ANALYSIS

  • Removing reranking, feedback, or attribution reduces correctness and citation accuracy; removing feedback affects GPT-4o more than trained OS-8B.Source: Figure 4 · p. 12, §4.3 · p. 13
  • Llama 3.1 8B's correctness and citation accuracy decline beyond a context-size threshold, while trained OS-8B remains strong through 20 passages.Llama 3.1 8B's citation F1 falls as low as 10, whereas OS-8B and Llama 3.1 70B maintain around 40, despite some decline.Source: Figure 4 · p. 12, §4.3 · p. 13

4.2 RESULTS

  • Without retrieval, 78–98% of cited papers are fabricated, and even real citations are often unsupported by abstracts.Source: §4.2 · p. 12
  • On SCHOLAR-Multi, retrieval raises Llama 3.1 8B's Cov score from 3.45 to 4.01 with OSDS, reflecting higher coverage than non-retrieval models.Source: §4.2 · p. 12

5 EXPERT EVALUATION

The expert evaluation compared OpenScholar and GPT-4o answers with expert-written responses using pairwise preferences and fine-grained ratings. OpenScholar variants were preferred over expert answers in more than half of cases, with coverage and answer length examined as contributing factors and possible confounds.

5 EXPERT EVALUATION

  • Sixteen experts assessed model answers against 108 expert-written responses, with each instance reviewed by 1–3 annotators.The study collected more than 400 fine-grained evaluations; pairwise inter-annotator agreement was 0.68 with ties and 0.70 when ties were merged.Source: §5 · p. 13, §5.1 · pp. 13–14

5.1 HUMAN EVALUATION DESIGN

  • Table 4 reports five-point ratings across four aspects and relative model-versus-human differences, while Figure 5 shows score distributions.Pairwise judgments allowed annotators to choose either answer or mark a tie; usefulness scores were grouped as not useful, neutral, or useful.Source: §5.1 · p. 13, Table 4 · p. 14, Figure 5 · p. 14

5.2 HUMAN EVALUATION RESULTS

  • OS-GPT4o and OS-8B beat human answers in over 50% of cases, mainly through greater coverage; their usefulness ratings were 80% and 72%, respectively.GPT-4o without retrieval won fewer than 35% of cases and had lower usefulness than humans and the OpenScholar variants.Source: §5.2 · p. 14
  • OS-8B was judged less organized or fluent than OS-GPT4o, whose longer outputs incorporated feedback more effectively and scored higher on organization.In 59 sampled preference explanations, coverage was cited in 29%, relevance in 23%, organization in 12%, and citations in 9%; annotators also flagged outdated or less representative cited papers.Source: §5.2 · pp. 14–15
  • Shortened OS-GPT4o answers averaging about 333 words were preferred or tied with expert answers in 75% of 50 sampled queries.Their average organization, coverage, and relevance scores were 4.5, 4.6, and 4.6; annotators also noted that a 300-word limit could reduce answer utility.Source: §5.2 · p. 15

6 RELATED WORK

Prior scientific LMs span specialized domains, while general-purpose LMs can perform scientific tasks but may hallucinate from memorized information. OpenScholar’s related-work context emphasizes literature synthesis, multi-paper evaluation, and open specialized systems.

Scientific LMs

  • Scientific LMs cover fields from biomedicine to astronomy, while general-purpose LMs have also shown capability in scientific tasks but can hallucinate from parametric knowledge.Examples include scientific idea generation and experimental code generation, while this work focuses on literature reviews and up-to-date research questions.Source: §6 · p. 15

LMs to assist scientists

  • Earlier literature-review systems often used proprietary black-box components and limited evaluations, whereas OpenScholar develops specialized open models and evaluates them with expert studies across three disciplines.The paper reports that OpenScholar outperforms previously introduced systems and human experts in five domains.Source: §6 · pp. 15–16

Benchmarks for scientific literature understanding

  • Many earlier scientific-literature benchmarks assume a single preselected paper, while ScholarQABench requires synthesis across papers not specified in advance.Unlike KIWI, ScholarQABench includes human-written answers and automatic evaluation pipelines; KIWI focuses more on human evaluation and uses primarily model-generated reference answers.Source: §6 · p. 16

7 CONCLUSION

The paper introduces OpenScholar and ScholarQABench for scientific literature review. OpenScholar uses open-checkpoint LMs and trained retrievers to refine outputs, and its answers were reported as more helpful than expert-generated answers in the evaluations.

7 CONCLUSION

  • OpenScholar and ScholarQABench are introduced to support navigation of the growing scientific literature-review task.ScholarQABench is presented as a standardized evaluation across multiple scientific domains.Source: §7 · p. 16
  • OpenScholar combines open-checkpoint LMs with trained retrieval models and iterative refinement to address hallucinations and citation accuracy.The conclusion reports evaluation improvements over GPT-4o and PaperQA2.Source: §7 · p. 16
  • In expert evaluations, OpenScholar-8B and OpenScholar-GPT4o achieved 51% and 70% win rates against human-generated answers.The paper reports increased answer coverage and releases its code, data, checkpoints, datastores, benchmark, and a public demo.Source: §7 · p. 16

LIMITATIONS

The paper cautions that its systems do not fully automate literature synthesis and that benchmark and evaluation findings have limits in size, scoring, domain coverage, and human assessment. It also outlines constraints on OpenScholar’s retrieval, generated content, and reproducibility.

LIMITATIONS

  • The authors do not claim that LM-based systems can fully automate scientific literature synthesis.They release ScholarQABench and OpenScholar to the community.Source: LIMITATIONS · p. 16
  • The human-written benchmark set is small, including 110 CS-LFQA examples and 108 expert-written answers, which may increase statistical variance and expertise-related bias.The authors cite the cost and time required to engage expert annotators as reasons for the limited dataset size.Source: LIMITATIONS · p. 16
  • Automatic evaluation may not capture answer quality perfectly because some rubric weights are heuristic and rubric demands can reward ancillary background rather than direct answers.Sentence-level citation precision and recall may also underestimate citation support when a sentence without a citation is supported by an adjacent one; benchmark snapshots date to July 2024 for CS and September 2024 for Multi.Source: LIMITATIONS · pp. 16–17
  • ScholarQABench has no social-science or other engineering and scientific instances, so findings may not generalize to those domains, especially where paper access is restricted.Its coverage primarily includes computer science, biomedicine, and physics.Source: LIMITATIONS · p. 17
  • OpenScholar may miss representative or relevant papers, and its 8B outputs may contain factual inaccuracies or unsupported claims.The GPT-4o version depends on a proprietary API that may change, complicating exact reproduction; inference excludes license-protected papers.Source: LIMITATIONS · p. 17
  • Human assessments may miss nuanced differences outside annotators’ immediate expertise, and findings from 108 questions across three disciplines may not generalize broadly.Usefulness and pairwise judgments may emphasize writing quality over factual correctness or citation accuracy, which the authors leave for future analysis.Source: LIMITATIONS · p. 17

B.1 GOAL OF SCHOLARQABENCH

  • ScholarQABench is designed as a realistic, reproducible, multifaceted literature-review benchmark combining adapted tasks and new expert-annotated datasets.Its multi-paper questions are intended to reflect genuine information needs and are paired with human answers or rubrics rather than model-generated references.Source: §B.1 · p. 26

B.2.1 DETAILS OF MODIFICATION OF SINGLE-PAPER TASKS

  • The modified SciFact task is open-retrieval true/false prediction over 40 million passages, excluding not-enough-information cases; PubMedQA retains yes/no items, and QASA uses full-stack QA.These modifications remove the original target abstract or restrict evaluation to the stated label/task settings.Source: §B.2.1 · p. 27

B.2.2 DETAILS OF DATA COLLECTIONS OF MULTI-PAPER TASKS

The multi-paper tasks were built from realistic scientific questions that may require several sources, with expert annotators identifying key answer ingredients and supporting text. Annotation was time-consuming even for domain experts.

B.2.2 DETAILS OF DATA COLLECTIONS OF MULTI-PAPER TASKS

  • Experts with relevant Ph.D.-level training, research experience, and publications were recruited to curate questions and rubrics across multi-paper subsets.The team included more than 20 annotators, including Ph.D. students, postdoctoral fellows, professors, and research scientists.Source: §B.2.2 · p. 27
  • Domain-specific instructions for collecting annotations were provided for SCHOLARQA-CS, SCHOLARQA-BIO, SCHOLARQA-NEURO, and SCHOLARQA-MULTI.The instructions are shown in Tables 5–7.Source: §B.2.2 · p. 27, Table 5 · p. 28, Table 6 · p. 29
  • Experts spent at least 30 minutes annotating each answer on average, with some Physics annotations taking about 5,000 seconds.The passage describes the annotation task as time-consuming and challenging even for domain experts.Source: §B.2.2 · p. 27
  • The task asked annotators to identify important answer points and supporting scientific text, without requiring them to solve questions or verify that answers existed.Realistic questions could be unanswerable in the current literature.Source: §B.2.2 · pp. 28–29

B.3.1 SCHOLARQA-CS CORR EVALUATION

The paper evaluates long-form answers with expert-annotated rubrics, uses automated evaluators for writing quality, and measures citation accuracy separately. It also describes training procedures for the scientific retriever components.

B.3.1 SCHOLARQA-CS CORR EVALUATION

  • SCHOLARQA-CS correctness combines general criteria weighted at 40% and annotation-driven criteria weighted at 60%, with must-have ingredients counted twice as much as nice-to-have ingredients.The final rubric score is a weighted sum of LLM-as-a-judge scores; citations and excerpts are explicitly emphasized.Source: §B.3.1 · pp. 29–30, Table 7 · p. 30

B.3.2 CONTENT QUALITY AND ORGANIZATION

  • Writing quality is automatically evaluated for relevance, coverage, and organization, while the evaluators can be over-optimistic and struggle to judge overall usefulness.Prometheus BGB is used for organization and coverage, and Prometheus v2 for relevance; human-written answers serve as references.Source: §B.3.2 · pp. 30–31, Table 8 · p. 30

B.3.3 CITATION ACCURACY

  • Citation accuracy is assessed with sentence-level citation precision and recall using an attribution model, excluding sentences shorter than 50 characters.The evaluation instructions are adapted from Yue et al. (2023).Source: §B.3.3 · p. 31, Table 9 · p. 31

C.1 TRAINING A SCIENTIFIC BI-ENCODER θbi

  • The scientific bi-encoder is initialized from Contriever and continually pretrained for 500,000 steps on a mixture of peS2o v2, CCNews, and Proofpile2.Training uses a batch size of 4,096 and learning rate of 0.00005.Source: §C.1 · p. 31

C.2 TRAINING A SCIENTIFIC CROSS-ENCODER θcross

  • The cross-encoder reranker is fine-tuned from BGE-large for five epochs on newly created training data.The stated learning rate is 6e-5; passages scoring above 3 are positive and those below 3 negative for selecting top passages.Source: §C.2 · p. 31

C.3 TRAINING DETAILS OF GENERATORS G

Generator training combines general-domain instruction data with task-specific scientific data, while excluding specified evaluation datasets. The retriever trained on peS2o v2 also retrieves relevant papers from peS2o v3 at test time.

C.3 TRAINING DETAILS OF GENERATORS G

  • 130,135 training instances combine Tulu general-domain instruction data and SciRIFF task-specific data, excluding PubMedQA, SciFact, and QASA evaluation data.Figure 7 depicts the training-data distribution but the supplied passage gives no distribution values.Source: §C.3 · pp. 31–32, Figure 7 · p. 32
  • OS-8B is initialized from Llama 3.1 8B Instruct, and the generators are trained for two epochs with a maximum context length of 10k.The passage also reports batch size 1, gradient accumulation 2, bf16, and AdamW; it lists learning rates of 5e-6 and 1e-4, respectively.Source: §C.3 · p. 32

D.1 COMPARISON OF PES2O V2 AND V3

  • Despite retrieval models being trained on peS2o v2, they consistently retrieve relevant papers from peS2o v3 at test time, including many papers from 2023–2024.The comparison uses the top 20 retrieved papers for the same SCHOLARQA-CS queries across the two datastores.Source: §D.1 · p. 32

D.2 MORE ANALYSIS ON EXPERT EVALUATION

The analysis compares model-generated and expert-written answers, finding differences in answer length, cited-paper counts, and citation metrics. It also reports limitations in evaluator agreement while providing rubrics and example materials for inspection.

D.2 MORE ANALYSIS ON EXPERT EVALUATION

  • Human answers are generally shorter and cite fewer papers than model-generated answers, and the authors retain the original responses for human evaluation.They analyze how these differences influence human preferences.Source: §D.2 · p. 32
  • OS-8B often matches or exceeds expert-level citation precision and recall, whereas OS-GPT4o is slightly worse in citation accuracy.PaperQA2 has higher citation precision than OS-8B or human experts, but its brief answers and fewer citations limit coverage.Source: §D.2 · p. 32

E EXAMPLES OF SCHOLARQABENCH

D.2 MORE ANALYSIS ON EXPERT EVALUATION

  • Prometheus has weak correlation and about 50% accuracy for organization and coverage, with disagreements often between adjacent scores, while rankings remain the same for relevance, coverage, and usefulness.The paper presents confusion matrices for the evaluator comparisons.Source: §D.2 · p. 33

E.1 SINGLE-PAPER TASKS

The appendix presents examples from single-paper and multi-domain tasks, alongside materials describing query annotation and human/model evaluation. It also reports that updating the datastore to peS2o v3 lets dense retrieval retrieve more recent papers.

E.1 SINGLE-PAPER TASKS

E.3 SCHOLARQA-BIO

  • Table 23 gives expert-annotated biomedical query examples and identifies each inspiring paper as help for question formulation.Source: §E.3 · p. 34, Table 23 · p. 48

E.4 SCHOLARQA-MULTI

  • Examples from different domains show expert annotations, including supporting quotes for a “Most Important” ingredient sourced from literature review and the annotator-provided source document.Source: §E.4 · pp. 35, 47, Figure 16 · p. 49, Figure 17 · p. 50
  • The appendix includes a five-point overall-helpfulness rubric, answer-length and citation-count statistics, and human–model answer comparisons across domains.Source: Table 13 · p. 36, Table 14 · p. 36, Table 17 · p. 41
  • Updating the datastore from peS2o v2 to v3, which includes papers through October 2024, lets dense retrieval retrieve more recent papers.Source: Figure 10 · p. 38

E.2 SCHOLARQA-CS

  • Table 22 presents a SCHOLARQA-CS annotation example with key ingredients and supporting quotes used as evaluation rubrics.Source: §E.2 · p. 34, Table 22 · p. 47
Loading 2411.14199v1…