Source-linked AI summary
RETA-LLM: A Retrieval-Augmented Large Language Model Toolkit
Jiongnan Liu, Jiajie Jin, Zihan Wang, Jiehan Cheng, Zhicheng Dou, Ji-Rong Wen
TL;DR
LLMs may hallucinate, and their stored knowledge may not cover in-domain questions. RETA-LLM provides a modular retrieval-augmented toolkit and pipeline; its enrollment-assistant case answers in-domain questions beyond the model's own knowledge.
Problem
LLMs can hallucinate, and relying on their stored world knowledge alone may not answer in-domain questions.
Source: §1 · p. 1Method
RETA-LLM combines request rewriting, document retrieval, passage extraction, answer generation, and fact checking in a toolkit and pipeline for in-domain systems.
Source: §1 · pp. 1–2, §2 · p. 2Results
Takeaways & Limitations
The pipeline lets users build in-domain LLM systems using their own repositories, with retrieval and LLM components that can be customized.
Source: §1 · p. 2, §2 · p. 3RETA-LLM remains under development, and the authors say many issues still require substantial work.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Although Large Language Models (LLMs) have demonstrated extraordinary capabilities in many domains, they still have a tendency to hallucinate and generate fictitious responses to user requests. This problem can be alleviated by augmenting LLMs with information retrieval (IR) systems (also known as retrieval-augmented LLMs). Applying this strategy, LLMs can generate more factual texts in response to user input according to the relevant content retrieved by IR systems from external corpora as references. In addition, by incorporating external knowledge, retrieval-augmented LLMs can answer in-domain questions that cannot be answered by solely relying on the world knowledge stored in parameters. To support research in this area and facilitate the development of retrieval-augmented LLM systems, we develop RETA-LLM, a {RET}reival-{A}ugmented LLM toolkit. In RETA-LLM, we create a complete pipeline to help researchers and users build their customized in-domain LLM-based systems. Compared with previous retrieval-augmented LLM systems, RETA-LLM provides more plug-and-play modules to support better interaction between IR systems and LLMs, including {request rewriting, document retrieval, passage extraction, answer generation, and fact checking} modules. Our toolkit is publicly available at https://github.com/RUC-GSAI/YuLan-IR/tree/main/RETA-LLM.
1 Introduction
LLMs can hallucinate, while retrieval augmentation grounds generation in external content and can bring private, in-domain knowledge into answers. RETA-LLM is a toolkit and pipeline for building such systems, with additional modules for improving interaction between retrieval and generation.
1 Introduction
- Retrieval-augmented LLMs retrieve relevant external content to ground responses, with reported potential to reduce hallucinations and answer in-domain queries beyond public training data.The paper says this approach has been verified to generate texts with fewer hallucinations and can use customized private data resources for in-domain questions.Source: §1 · p. 1
- RETA-LLM focuses on retrieval-augmented LLM systems and offers more plug-in modules than general LLM-augmented toolkits such as LangChain.Its typical retrieve-and-generate setup retrieves documents or passages, then generates answers using them as references.Source: §1 · p. 1
- Beyond retrieval and answer generation, RETA-LLM adds request rewriting, passage extraction, and fact checking to improve interaction between IR systems and LLMs.The toolkit also provides a ready-to-use pipeline for building in-domain systems from users' own repositories.Source: §1 · p. 2
- RETA-LLM is part of the YuLan open-source initiative and remains under development, with issues still requiring substantial work.Source: §1 · p. 2
2 RETA-LLM Framework
RETA-LLM combines five modules in a retrieval-augmented workflow, from clarifying a request and retrieving documents to checking generated answers. Its separation of IR systems and LLMs is intended to make both easier to customize.
2 RETA-LLM Framework
- RETA-LLM's five modules cover request rewriting, document retrieval, passage extraction, answer generation, and fact checking.The toolkit focuses specifically on retrieval-augmented LLMs.Source: §2 · p. 2
- The rewriting module uses the current request and conversation history to clarify incomplete follow-up questions before retrieval.For example, it can resolve a follow-up about the School of Economics using the prior question about majors in the School of Information.Source: §2 · p. 2
- The retrieval module returns the top-K documents for the revised request, with K set to 3 by default.Source: §2 · p. 3
- Passage extraction uses an LLM and sliding windows to select relevant fragments from retrieved documents, avoiding direct concatenation that may exceed input limits or truncation that may lose information.The default window size is 512 tokens and the step is 256.Source: §2 · p. 3
- After generation from retrieved references, fact checking evaluates the answer against those references and can return it or decline to answer.The framework keeps IR systems and LLMs separate so users can customize their search engines and LLMs.Source: §2 · p. 3
3 RETA-LLM Usage Pipeline
The usage pipeline converts local HTML resources, builds retrieval indexes, prepares an answering LLM, and launches a service. It supports domain adaptation and allows users to substitute their own retrieval engines or model configurations.
3 RETA-LLM Usage Pipeline
- The pipeline converts raw HTML files into JSON data before indexing them.It uses Beautiful Soup in the HTML Converter.Source: §3 · p. 3
- The Index Builder supports unsupervised domain adaptation of dense retrieval models on local documents and also provides a sparse retriever or allows users' own search engines.The authors describe its dense method as avoiding text segmentation and its potential negative impact on overall semantic information.Source: §3 · p. 3
- Users can configure supported answering models or edit the toolkit for other LLMs, then run their service using Streamlit.Templates are provided for Alpaca, YuLan-Chat, ChatGLM, and the GPT-3.5 API.Source: §3 · p. 3
4 A RETA-LLM Service Case
The authors describe an enrollment-assistant system built from university enrollment webpages using dense retrieval and YuLan-13B. They report that it answers in-domain questions beyond the model's own knowledge.
4 A RETA-LLM Service Case
- The RUC enrollment assistant uses university enrollment webpages, dense document retrieval, and YuLan-13B as its backbone LLM.Source: §4 · p. 4
- The authors report that the system answers in-domain questions that cannot be answered from the LLM's own knowledge.The case is illustrated in Figure 2.Source: §4 · p. 4
5 Conclusion and Future Work
RETA-LLM offers five independent modules and a pipeline for building in-domain retrieval-augmented LLM systems, with additional strategies and configurability planned for future work.
5 Conclusion and Future Work
- RETA-LLM provides five independent modules: request rewriting, document retrieval, passage extraction, answer generation, and fact checking.The toolkit is intended to facilitate research and development of retrieval-augmented LLMs.Source: §5 · p. 4
- The toolkit provides a pipeline to help users build in-domain LLM-based systems.Source: §5 · p. 4
- Future plans include adding active retrieval augmented generation strategies and making RETA-LLM more modular and configurable.Source: §5 · p. 4