Source-linked AI summary
Improving Language Models via Plug-and-Play Retrieval Feedback
Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, Ashish Sabharwal
TL;DR
LLMs can hallucinate and contain incomplete or outdated knowledge, while human-feedback alignment is costly and unavailable for real-time inference correction. REFEED uses automatic retrieval feedback in a plug-and-play pipeline that generates answers, retrieves evidence, and refines outputs, with diverse generation and ensembling modules. The paper reports state-of-the-art few-shot performance on three knowledge-intensive tasks and presents the approach as a practical alternative to fine-tuning.
Problem
LLMs can hallucinate and struggle with incomplete or outdated knowledge, while human-feedback fine-tuning is costly and cannot provide real-time inference feedback.
Method
REFEED generates initial outputs, retrieves relevant documents conditioned on the query and output, and uses the retrieved information to refine answers without fine-tuning.
Results
REFEED achieves state-of-the-art performance under the few-shot setting across three knowledge-intensive tasks.
Takeaways & Limitations
The plug-and-play pipeline provides automatic retrieval feedback and adds diverse generation and ensembling to improve answer reliability without expensive fine-tuning.
Takeaways & Limitations
Human-feedback methods may generalize less well to novel and unseen data when they rely exclusively on positively rated generations.
Abstract
from arXiv · showhide
Large language models (LLMs) exhibit remarkable performance across various NLP tasks. However, they often generate incorrect or hallucinated information, which hinders their practical applicability in real-world scenarios. Human feedback has been shown to effectively enhance the factuality and quality of generated content, addressing some of these limitations. However, this approach is resource-intensive, involving manual input and supervision, which can be time-consuming and expensive. Moreover, it cannot be provided during inference, further limiting its practical utility in dynamic and interactive applications. In this paper, we introduce ReFeed, a novel pipeline designed to enhance LLMs by providing automatic retrieval feedback in a plug-and-play framework without the need for expensive fine-tuning. ReFeed first generates initial outputs, then utilizes a retrieval model to acquire relevant information from large document collections, and finally incorporates the retrieved information into the in-context demonstration for output refinement, thereby addressing the limitations of LLMs in a more efficient and cost-effective manner. Experiments on four knowledge-intensive benchmark datasets demonstrate our proposed ReFeed could improve over +6.0% under zero-shot setting and +2.5% under few-shot setting, compared to baselines without using retrieval feedback.
1 Introduction
LLMs can hallucinate and retain incomplete or outdated knowledge, while human-feedback fine-tuning is costly and cannot provide real-time inference feedback. REFEED addresses these constraints with plug-and-play retrieval feedback, enhanced by diverse generation and ensembling, and reports state-of-the-art few-shot performance on three knowledge-intensive tasks.
- 1 Introduction: LLMs may hallucinate, lack complete or current knowledge, and struggle with complex reasoning and knowledge-intensive tasks.These limitations can produce untrustworthy or inaccurate outputs, especially for long-tail knowledge.
- 1 Introduction: Human-feedback approaches require costly annotation and fine-tuning, may weaken generalization through reliance on positively rated data, and cannot receive real-time inference feedback.Once fine-tuned, such models cannot facilitate immediate error correction during inference.
- 1 Introduction: REFEED generates initial outputs, retrieves relevant information using the query and outputs, and refines answers without fine-tuning.Retrieved documents can provide feedback specifically about the generated output and may leave it unchanged when no revision is needed.
- 1 Introduction: Compared with retrieve-then-read methods, REFEED retrieves documents that more directly elucidate the relationship between the query and the generated output.The initial output helps retrieval when the question alone lacks sufficient lexical or semantic overlap with supporting documents.
- 1 Introduction: Diverse initial generations let REFEED compare multiple candidates against retrieved documents, while ensembling pre- and post-feedback outputs mitigates misleading retrieval feedback.The ensemble uses perplexity ranking to combine outputs before and after feedback.
- 1 Introduction: REFEED achieves state-of-the-art performance under the few-shot setting across three challenging knowledge-intensive tasks.The tasks include single-hop question answering, multi-hop question answering, and dialogue.
2 Related Work
Knowledge-intensive NLP commonly uses retrieve-then-read systems, while human-feedback alignment methods improve selected tasks at substantial annotation and fine-tuning cost. REFEED instead uses retrieved documents as direct feedback for generated outputs and adds diversified generation and ensembling for knowledge-intensive tasks.
- 2 Related Work: Retrieve-then-read systems retrieve evidence documents and then use a reader to predict an answer, with research improving retrievers, readers, or both end-to-end.The passage identifies Wikipedia-like evidence corpora and representative sparse and dense retrieval approaches.
- 2 Related Work: Human-feedback methods rely on annotated data and positively rated generations, creating substantial cost and time demands and limiting correction of negative attributes or errors.The passage also notes reduced generalizability to novel and unseen data.
- 2 Related Work: Contemporary automated-feedback studies support retrieval feedback, but REFEED focuses on knowledge-intensive tasks where external evidence is especially important.REFEED differs by directly using retrieved documents as feedback rather than requiring ChatGPT-generated feedback from retrieved evidence.
3 Proposed Method
REFEED is a plug-and-play pipeline that generates an initial answer, retrieves supporting documents conditioned on that output, and refines the answer with retrieval feedback. It further broadens feedback through diverse answer generation and combines pre- and post-feedback outputs with perplexity-based ensembling.
- 3.1 Proposed Method: REFEED: REFEED first generates an answer, retrieves documents from a large corpus, and refines that answer using the retrieved information.The pipeline is designed to integrate retrieval feedback without costly fine-tuning.
- 3.1.1 Basic Pipeline: An ensemble combines outputs before and after retrieval feedback using perplexity ranking to reduce the risk that misleading feedback determines the final answer.The method evaluates initial and refined outputs when selecting the result.
- 3.1 Proposed Method: REFEED: Retrieval is conditioned on the initial output as well as the query, so documents provide feedback targeted to the generated answer rather than general query information.The formulation retains the top k = 10 ranked documents for refinement.
- 3.1.1 Basic Pipeline: REFEED retrieves top-10 supporting documents with a retrieval model such as BM25, then uses them to corroborate, refute, or revise the initial answer.The refinement may rephrase, expand, or change the answer based on the retrieved knowledge.
- 3.1.1 Basic Pipeline: Diverse answer generation samples multiple candidate answers and retrieves documents for those candidates, broadening the feedback available for final answer generation.Retrieved documents are merged, ranked by query-document similarity, deduplicated, and reduced to the top k = 10 before refinement.
4 Experiments
REFEED is evaluated on knowledge-intensive QA and dialogue tasks in zero-shot and few-shot settings, with comparisons against closed-book and retrieval-based baselines. It consistently improves over baselines, while ablations and case studies examine diverse generation, ensembling, chain-of-thought integration, and retrieval-feedback errors.
- 4.3.1 Zero/Few-shot Question Answering and Dialogue Evaluation: +7.7 EM points over text-davinci-003 on two open-domain QA benchmarks in zero-shot evaluation, with consistent gains also reported for multi-hop QA and dialogue.The experiments use text-davinci-003 as the backbone; Table 1 covers three zero-shot knowledge-intensive tasks.
- 4.3.1 Zero/Few-shot Question Answering and Dialogue Evaluation: REFEED achieves superior performance to methods that directly retrieve or generate documents across benchmarks and experimental settings.The reported explanation is that competing methods can struggle when queries and source passages lack lexical overlap.
- 4.3.1 Zero/Few-shot Question Answering and Dialogue Evaluation: Few-shot evaluation shows the same pattern as zero-shot evaluation, with REFEED achieving state-of-the-art performance on three knowledge-intensive tasks.Table 2 reports the few-shot comparison across the evaluated tasks.
- 4.3.2 Ablation Study on Ensemble Method and Diverse Generation: Removing ensembling lowers average performance by 0.8 EM across three QA datasets, while removing diverse generation lowers it by 1.1 EM.The ensemble selects between pre- and post-feedback outputs using log-probability, while diverse generation broadens answer and document coverage.
- 4.3.2 Ablation Study on Ensemble Method and Diverse Generation: Diverse answer generation improves answer hit ratio and increases retrieval coverage, supporting more comprehensive documents for complex questions.Figure 4 compares direct retrieval, one-answer REFEED-O, and diverse-answer REFEED-D using Recall@K.
- 4.3.3 Applying REFEED to Chain-of-thought Reader: REFEED can be integrated with chain-of-thought reasoning, improving complex HotpotQA performance over straightforward QA prompting.The pipeline retrieves documents from an initial answer, then uses retrieval feedback to refine a subsequent chain of thought.
- 4.3.4 Case Study: Case studies show retrieval feedback correcting two initially incorrect answers but misleading the model in a third example.The examples include correcting a film release date and distinguishing a composer from a vocalist; the latter yields the correct answer after feedback.
5 Conclusion
REFEED is presented as a practical plug-and-play pipeline for improving LLM performance on knowledge-intensive tasks without expensive fine-tuning. Its diverse answer generation and ensemble modules support more reliable answers, and experiments on four benchmarks demonstrate state-of-the-art few-shot performance.
- REFEED achieves state-of-the-art performance under the few-shot setting across four challenging knowledge-intensive benchmarks.
- REFEED improves LLM performance through automatic retrieval feedback and output refinement without expensive fine-tuning.
- Diverse answer generation considers multiple retrieved-document perspectives to support more reliable and accurate answers.
- An ensemble approach mitigates misleading retrieval feedback by combining outputs before and after retrieval feedback.
- The authors propose continuing to refine REFEED to expand its applicability across diverse scenarios and applications.