LitSearch: A Retrieval Benchmark for Scientific Literature Search

1 Introduction

• Finding related work, checking whether a method has been proposed, and recalling a paper are important research tasks that can require domain expertise and reasoning through lengthy papers.The introduction frames systems recommending relevant citations as having potential to enhance researchers’ productivity.Source: §1 · p. 1
• Earlier citation-recommendation setups used text surrounding an inline citation as the query and the cited paper as the target, but such queries can be noisy, broad, or context-dependent.Examples include “Large Language Models” and references to hyperparameters.Source: §1 · p. 1
• LitSearch contains 597 literature-search questions from GPT-4 rewrites of inline citations and questions written by authors about their own papers, all manually examined or filtered.Questions are paired with one or more target papers as ground truth.Source: §1 · p. 2
• LitSearch is presented as a first dataset of realistic literature-search questions, with full-text documents averaging 6,041 words and all questions manually examined by the authors.The paper contrasts its document length with an average of 56 words for an existing benchmark example.Source: §1 · p. 2
• 74.8% recall@5: GritLM outperforms BM25 by 24.8%, GPT-4o reranking adds 4.4%, and Google Search reaches at most 42.8%.LitSearch also better differentiates retrieval-system performance than existing benchmarks.Source: §1 · p. 2

2 LitSearch

• LitSearch pairs literature-search questions with target papers and gathers questions through inline citations and authors writing about their own papers, with manual inspection and filtering.Source: §2 · p. 2

2.1 Inline-citation Questions

• Inline-citation questions are generated from sampled citation mentions, with target papers limited to the ACL Anthology and source papers unrestricted by venue.GPT-4 receives the source paragraph and cited-paper titles; questions are filtered using different word-overlap thresholds for ACL- and non-ACL-sourced items, then manually examined.Source: §2.1 · p. 3

2.2 Author-written Questions

• Authors of recent ACL 2023 and ICLR 2024 papers wrote questions intended to be answered by their own papers.The authors report that this setup better guarantees question correctness.Source: §2.2 · p. 3

2.4 Dataset Statistics

• 597 questions comprise 351 inline-citation and 246 author-written items; author-written questions have higher title-and-abstract word overlap (0.43 vs. 0.33).Source: Table 2 · p. 4, §2.4 · p. 4

2.3 Manual Filtering to Ensure High Quality

• For inline-citation questions, 26% (98/382) of ACL-sourced and 56% (253/450) of non-ACL-sourced questions were kept; among author-written questions, 89% (155) from ACL 2023 and 78% (91) from ICLR 2024 were kept.Questions fitting more than 20 corpus papers were deemed too general and excluded.Source: §2.3 · pp. 3–4

2.5 The Retrieval Corpus

• The retrieval corpus contains 64,183 ACL Anthology and ICLR papers, averaging 134 words for titles and abstracts and 6,041 words for full texts.Source: §2.5 · p. 4

3.1 Experimental Setup

• The default setup embeds titles and abstracts, reporting recall@20 for broad questions and recall@5 and recall@20 for specific questions.These cutoffs correspond to the guidelines used to annotate question specificity.Source: §3.1 · p. 4

3.2 Baselines

• Baselines include BM25 and dense models GTR, Instructor, E5, and GritLM, with GPT-4o used for vanilla and one-hop reranking.Vanilla reranking prompts GPT-4o to reorder retrieved papers; one-hop reranking adds papers cited by retrieved seed papers to the candidate list.Source: §3.2 · pp. 4–5

3.3 Results

• GritLM-7B achieves 70.8 recall@20 on broad questions and 74.8 recall@5 on specific questions, outperforming other baselines.The default retrieval setup encodes paper titles and abstracts.Source: §3.3 · p. 5, Table 3 · p. 5
• Both vanilla and one-hop GPT-4o reranking improve base retrieval, but one-hop gains are generally lower than vanilla with GritLM.On specific inline questions, vanilla reranking improves BM25 and GritLM recall@5 by 21.5% and 5.5%, respectively; the weaker one-hop gains indicate the benchmark is not easily gamed through similar citation mentions.Source: §3.3 · pp. 5–6
• Retrievers generally achieve higher recall on specific questions, which have fewer competing relevant papers than broad questions.The benchmark retains both subsets to provide a more nuanced view of retriever capabilities.Source: §3.3 · p. 6
• Inline-citation questions are harder to retrieve than author-written questions across both broad and specific subsets.The authors attribute this pattern to greater overlap between author-written questions and paper titles and abstracts, including reused terminology (Gururangan et al., 2018).Source: §3.3 · p. 6
• Adding full text does not consistently improve retrieval, and more text more often hinders performance.Substantial gains occur only for author-written broad questions with BM25 and some embedding models.Source: Table 5 · p. 6, §3.3 · p. 6
• Questions manually rated as more realistic and difficult (quality score 2) are consistently more challenging for retrieval models.The authors say including varied-quality questions improves diversity and coverage of users’ information needs.Source: §3.3 · p. 6

4.1 Does Including More Paper Content Improve Retrieval Performance?

• Including full paper text does not consistently improve retrieval and more often hinders performance, except for some author-written broad questions.The corpus averages 6,041 words per full paper, while the tested models have context limits up to 2,048 tokens; the authors suggest this mismatch may contribute to the result.Source: §4.1 · pp. 6–7

4.2 Does the Source of Inline Citation Questions Matter?

• One-hop reranking helps BM25 more on ACL-sourced than non-ACL-sourced specific questions: +11.4% versus +1.2% recall@5 over vanilla reranking.The authors posit that BM25 can exploit the ACL question-collection pipeline by retrieving source papers that cite targets; non-ACL source papers are outside the retrieval corpus. GritLM shows no similarly large one-hop gains, which the authors hypothesize is because its initial results already include targets.Source: §4.2 · p. 7

4.3 Performance of Search Engines

• In a manually run study of 80 specific questions, all three search engines have similarly low recall on inline-citation questions, while Google Search performs better on author-written questions.The authors caution that comparison with retrieval models is not apples-to-apples because the search engines use a larger retrieval corpus.Source: §4.3 · p. 7

4.4 Comparing Other Retrieval Benchmarks

• LitSearch generally agrees with existing retrieval benchmarks but differentiates retrievers more strongly: GritLM exceeds E5 by 15 nDCG@10 points on specific questions, while they perform almost the same on MSMARCO.The authors present LitSearch as an informative testbed for reflecting advances in embedding models.Source: §4.4 · p. 8, Table 8 · p. 8

Citation recommendation

• Citation recommendation datasets use either whole papers and their cited papers or inline citation mentions as queries and targets.LitSearch instead uses manually annotated natural-language literature search questions, which the authors characterize as more realistic and challenging.Source: §5 · p. 8

Retrieval benchmarks

• General retrieval benchmarks cover sources such as Wikipedia, web queries, biomedical questions, and multi-domain collections including KILT, BEIR, and MTEB.LitSearch complements these benchmarks with a literature-search question type; CiteME instead identifies references for claims in inline text and targets LLM-based agents.Source: §5 · p. 8

Retrieval systems

• Retrieval approaches range from bag-of-words methods such as BM25 to dense embeddings, with recent instruction-based encoding intended to improve versatility across tasks.The passage describes dense models as enabling semantic search without exact keyword matches.Source: §5 · p. 8

6 Conclusion

• LitSearch contains 597 manually curated literature-search questions, including inline-citation and author-written subsets.Both subsets undergo manual inspection by the authors.Source: §6 · p. 9
• Experiments find state-of-the-art instruction-finetuned embedding models outperform BM25, with further improvement from GPT-4o-based reranking.Source: §6 · p. 9
• Commercial search engines such as Google struggle with LitSearch questions, and the benchmark differentiates retrieval systems better than existing benchmarks.Source: §6 · p. 9

• Some questions remain slightly out of distribution or too easy because of high overlap with their target papers.Source: Limitations · p. 9
• Author-written questions were easier than expected because writing challenging literature-search questions is non-trivial even for experienced researchers.Source: Limitations · p. 9
• The evaluation was not exhaustive, omitted more sophisticated retrieval or reranking systems, and primarily covers English questions and research papers.Source: Limitations · p. 9

• LitSearch was manually inspected and ensured to contain no unsafe or inappropriate content.Source: Ethics Statement · p. 9
• Inline-citation questions include more highly cited target papers, while author-written questions cover only ACL 2023 and ICLR 2024 papers.Source: Ethics Statement · p. 9

A Annotator Acknowledgments

• The authors acknowledge contributors who supplied questions about their ACL 2023 and/or ICLR 2024 papers.Source: A Annotator Acknowledgments · p. 12

B Annotation Details

• The appendix provides instructions for manually inspecting questions in Table 1.Source: B Annotation Details · p. 12
• ACL 2023 and ICLR 2024 authors were recruited by email and Google Forms to write questions.Source: B Annotation Details · p. 12

• The retrieval corpus consists of ACL Anthology and ICLR papers extracted from S2ORC.Source: C Retrieval Corpus · p. 13
• ACL Anthology papers are identified in S2ORC by entries whose metadata includes an ACL Anthology ID.Source: C Retrieval Corpus · p. 13
• ICLR papers are identified using Semantic Scholar venue queries and title matching against accepted-paper titles from the official ICLR website.Source: C Retrieval Corpus · p. 13

D Retriever details

• Table 9 lists the HuggingFace checkpoints used for each dense retriever, with instructions specified for instruction-finetuned models.The instructions distinguish query encoding from encoding paper titles and abstracts; the listed models include Instructor-XL and GritLM-7B.Source: Table 9 · p. 13, D Retriever details · p. 13

E Prompts and Additional Statistics

• Tables 12 and 13 provide the GPT-4 question-generation prompt and GPT-4o reranking prompt, respectively.The reranking prompt is adapted from Sun et al. (2023a).Source: E Prompts and Additional Statistics · p. 13, Table 12 · p. 15, Table 13 · p. 16
• Table 14 provides detailed statistics for LitSearch.Source: E Prompts and Additional Statistics · p. 13, Table 14 · p. 16
• Tables 10 and 11 present the email template and Google Forms instructions used to collect author-written questions.The forms were sent to ICLR 2024 and ACL 2023 authors.Source: Table 10 · p. 13, Table 11 · p. 14