Source-linked AI summary
LitLLM: A Toolkit for Scientific Literature Review
Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam H. Laradji, Krishnamurthy DJ Dvijotham, Jason Stanley, Laurent Charlin, Christopher Pal
TL;DR
Literature reviews require accurate retrieval and contextualization, but LLMs can hallucinate. LitLLM retrieves and re-ranks papers to ground generated related work; preliminary user feedback distinguished zero-shot and plan-based outputs. The authors describe it as a research assistant that may aid paper search and review quality.
Problem
Literature reviews require factual correctness, while LLM-generated scientific content can hallucinate, including inaccurate citations and imaginary papers.
Source: §1 · p. 1Method
LitLLM summarizes a user’s abstract into search keywords, retrieves papers, re-ranks them for relevance, and generates related work with optional sentence-plan control.
Source: §3 · p. 4, §3.1 · p. 4Results
In preliminary feedback from five researchers, zero-shot generation was more informative, while plan-based generation was more accessible, tailored, succinct, and readily usable.
Source: §5 · p. 6Takeaways & Limitations
The authors see LitLLM as a research assistant that may aid relevant-paper search and improve automatically generated related-work quality.
Source: §6 · p. 6The system considers only the query paper’s and retrieved papers’ abstracts, which the authors identify as a bottleneck to effective literature-review generation.
Source: §6 · p. 6
Abstract
from arXiv · showhide
Conducting literature reviews for scientific papers is essential for understanding research, its limitations, and building on existing work. It is a tedious task which makes an automatic literature review generator appealing. Unfortunately, many existing works that generate such reviews using Large Language Models (LLMs) have significant limitations. They tend to hallucinate-generate non-factual information-and ignore the latest research they have not been trained on. To address these limitations, we propose a toolkit that operates on Retrieval Augmented Generation (RAG) principles, specialized prompting and instructing techniques with the help of LLMs. Our system first initiates a web search to retrieve relevant papers by summarizing user-provided abstracts into keywords using an off-the-shelf LLM. Authors can enhance the search by supplementing it with relevant papers or keywords, contributing to a tailored retrieval process. Second, the system re-ranks the retrieved papers based on the user-provided abstract. Finally, the related work section is generated based on the re-ranked results and the abstract. There is a substantial reduction in time and effort for literature review compared to traditional methods, establishing our toolkit as an efficient alternative. Our project page including the demo and toolkit can be accessed here: https://litllm.github.io
1 Introduction
Literature reviews require finding and contextualizing relevant work, while factual correctness is essential and LLMs can hallucinate. LitLLM addresses this with a modular, retrieval-grounded pipeline for drafting related work from a user-provided abstract.
1 Introduction
- Factual correctness is essential to literature reviews, which combine retrieving relevant papers with contextualizing proposed work against existing research.The introduction identifies hallucination as a challenge for current LLMs.Source: §1 · p. 1
- LLM-generated scientific content can include inaccurate citations and imaginary papers, as reported for Galactica (Taylor et al., 2022).The paper also notes that LLM-based review systems may generate high-quality text while remaining prone to hallucinations.Source: §1 · p. 1
- LitLLM is an interactive tool that generates a scientific paper’s literature review or related-work section from a user-provided abstract.Its stated objective is to help users navigate research papers and draft a review for a paper or project.Source: §1 · p. 3
- The system uses retrieval-augmented generation to ground related-work generation in retrieved papers, alongside multiple search techniques and sentence-based planning for controllable output.The introduction presents these as contributions for addressing hallucinations and enabling user control.Source: §1 · pp. 1, 3
2 Related Work
Related work spans LLM-based retrieval, ranking, and generation, with RAG presented as a way to draw on external knowledge. Prior studies also explore zero-shot abilities and prompt engineering across tasks.
2 Related Work
- RAG combines LLM knowledge with external databases to support accuracy and credibility on knowledge-intensive tasks and to integrate updated, domain-specific information (Gao et al., 2023).The paper situates its approach among work addressing hallucination, outdated knowledge, and opaque reasoning in LLMs (Huang et al., 2023; Gao et al., 2023; Li et al., 2024).Source: §2 · p. 3
- LLMs have been explored for relevance ranking, with proprietary models reported as competitive with supervised methods and open-source models used for zero-shot listwise reranking (Sun et al., 2023; Pradeep et al., 2023b,a).Source: §2 · p. 3
- Prior work studies zero-shot LLM abilities in ranking, instruction tuning, reasoning, generation assessment, and open-domain question answering (Hou et al., 2023; Wei et al., 2021; Crispino et al., 2023; Liusie et al., 2023; Li et al., 2022).The cited recommender-system study also reports difficulties with historical-interaction order and position bias (Hou et al., 2023).Source: §2 · pp. 3–4
- Prompt engineering is identified as a technique for enhancing LLM capabilities, with multiple strategies under study (Shi et al., 2023).Source: §2 · p. 4
3 Pipeline
LitLLM’s modular pipeline turns a research idea or abstract into a query, retrieves and re-ranks papers, then generates related work. Generation can be zero-shot or guided by a sentence plan.
3 Pipeline
- The pipeline passes a user’s abstract or research idea through keyword summarization, paper retrieval, relevance-based listwise reranking, and LLM generation of related work.The user-provided query abstract is used in reranking and generation; sentence plans can optionally control the output.Source: §3 · p. 4
3.2 Paper Re-Ranking Module
- An LLM reranker orders retrieved papers by relevance to the query abstract before those papers are used as generation context.Source: §3 · p. 4
3.1 Paper Retrieval Module
- Users can search with abstract-derived keywords, add their own keywords, or use a relevant seed paper to retrieve similar papers.User keywords are noted as particularly useful for interdisciplinary research and including recent work that may not be captured in the abstract.Source: §3.1 · p. 4
3.3 Summary Generation Module
- The generation module offers zero-shot and plan-based strategies, with sentence plans used to make output controllable.The plans specify expected output structure and guide sentence generation (Agarwal et al., 2024).Source: §3.3 · p. 5, §3.3.2 · p. 6
3.3.1 Zero-shot generation
- Zero-shot generation retrieves relevant papers and uses them as context, grounding the review and allowing it to reflect research beyond the LLM’s training knowledge.Source: §3.3.1 · p. 5
3.3.2 Plan based generation
- Long query-paper and retrieved-paper contexts hinder applying the discussed prompting techniques; plan-based prompting instead structures expected sentences and citation placement.The example plan specifies sentence and word counts and citations at particular lines.Source: §3.3.2 · p. 6
4 Implementation Details
The implementation combines a React interface with academic search APIs and LLM calls. The modular design permits different models for pipeline components and lets users sort retrieved papers by relevance, citation count, or year.
4 Implementation Details
- The interface is built with React, and paper search uses Semantic Scholar’s Academic Graph and Recommendations API plus OpenAlex’s search endpoint (Lo et al., 2020; Kinney et al., 2023).Source: §4 · p. 6
- The implementation uses GPT-3.5-turbo and GPT-4 through the OpenAI API, while the modular pipeline can use proprietary or open-source LLMs for different components.Source: §4 · p. 6
- Users can sort retrieved papers by relevance, citation count, or year, with Semantic Scholar results used as the default ordering.Source: §4 · p. 6
5 User Experience
In a preliminary study with five researchers, feedback distinguished zero-shot generation as more informative about literature overall and plan-based generation as more accessible and tailored. In an example, the top recommended paper also appeared among the original paper’s citations.
5 User Experience
- Five researchers found plan-based generation more accessible and tailored, while zero-shot generation was more informative about literature overall.The system was made available through its user interface for a preliminary study.Source: §5 · p. 6
- For a randomly chosen recent paper, the top recommended paper was also cited in the original paper.The generated query was “Multimodal Research: Image-Text Model Interaction.”Source: §5 · p. 6
- Plan-based generation produced a more succinct and readily usable literature review, while zero-shot generation provided insights into existing literature.Source: §5 · p. 6
6 Conclusion and Future Work
LitLLM generates attributed literature reviews from abstracts using RAG and re-ranking in a modular pipeline. The conclusion cautions authors to disclose use and check for possible hallucinations, while noting that abstract-only inputs constrain effective generation.
6 Conclusion and Future Work
- LitLLM uses RAG and re-ranking to generate attributed literature reviews from abstracts in a modular pipeline adaptable to other LLMs or retrieval domains.The authors mention news as an example of another domain.Source: §6 · p. 6
- Authors should disclose use of the system and remain cautious about eliminating possible hallucinations.Source: §6 · p. 6
- Using only query-paper and retrieved-paper abstracts creates a bottleneck in effective literature-review generation.The authors envision ingesting whole papers as a future extension.Source: §6 · p. 6
- The authors identify exploring academic search through multiple APIs, such as Google Scholar, as future work.Source: §6 · p. 6
Appendix
The appendix includes prompts and code related to LitLLM’s generation and retrieval workflow, as well as an example of generated related work.
Appendix
- Figure 7 presents a prompt for sentence plan-based generation.Source: Figure 7 · p. 9
- The query-search code sends a query and result limit to Semantic Scholar and returns the search results.Source: Appendix · p. 9
- The appendix also includes code to retrieve one paper’s data using its URL.Source: Appendix · p. 9
- The appendix code requests recommended papers from Semantic Scholar using an arXiv identifier and a requested result limit.Source: Appendix · p. 9
- Table 1 presents generated related work for a randomly chosen recent paper using an LLM-summarized query and retrieved papers; the caption describes plan-based output as more succinct and readily usable.Source: Table 1 · p. 10