Source-linked AI summary
Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, Jianfeng Gao
TL;DR
LLMs’ hallucinations, stale knowledge, and inability to access some application-specific information limit their use in mission-critical tasks. LLM-AUGMENTER augments a black-box LLM with external evidence and automated feedback that iteratively revises responses. Across task-oriented dialog and Wiki question answering, it reduces hallucinations without sacrificing fluency and informativeness.
Problem
LLMs tend to hallucinate and cannot access all changing or proprietary information needed for mission-critical applications.
Method
LLM-AUGMENTER uses plug-and-play modules to retrieve and consolidate external knowledge, prompt a fixed LLM, verify responses, and revise them with automated feedback.
Results
LLM-AUGMENTER significantly reduces ChatGPT’s hallucinations without sacrificing fluency and informativeness across task-oriented dialog and open-domain question answering.
Takeaways & Limitations
External knowledge and automated feedback enable fixed LLMs to produce more grounded responses for dialog and question answering.
Takeaways & Limitations
Interactive feedback can slow user experience because ChatGPT is often queried twice for one response, and the main results use a manually designed policy without human evaluation.
Abstract
from arXiv · showhide
Large language models (LLMs), such as ChatGPT, are able to generate human-like, fluent responses for many downstream tasks, e.g., task-oriented dialog and question answering. However, applying LLMs to real-world, mission-critical applications remains challenging mainly due to their tendency to generate hallucinations and their inability to use external knowledge. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules. Our system makes the LLM generate responses grounded in external knowledge, e.g., stored in task-specific databases. It also iteratively revises LLM prompts to improve model responses using feedback generated by utility functions, e.g., the factuality score of a LLM-generated response. The effectiveness of LLM-Augmenter is empirically validated on two types of scenarios, task-oriented dialog and open-domain question answering. LLM-Augmenter significantly reduces ChatGPT's hallucinations without sacrificing the fluency and informativeness of its responses. We make the source code and models publicly available.
1 Introduction
LLMs can generate fluent and informative text but may hallucinate, become stale, or lack application-specific information. LLM-AUGMENTER addresses these challenges by augmenting a fixed LLM with external evidence and automated feedback, improving responses across dialog and Wiki question answering.
- LLMs’ lossy knowledge encoding and memory distortion can cause hallucinations that are harmful in mission-critical applications.
- LLMs cannot encode all information needed for applications, including changing real-world facts and proprietary datasets unavailable during training.
- LLM-AUGMENTER augments a fixed LLM with plug-and-play modules using external knowledge and automated feedback.
- The system retrieves and consolidates external evidence, prompts ChatGPT to generate a grounded candidate response, then verifies and revises it when needed.
- 32.3% in Usefulness and 12.9% in Humanness were the reported customer-service dialog improvements, while Wiki QA factuality increased by absolute +10% in F1.
2 LLM-AUGMENTER
LLM-AUGMENTER organizes a fixed LLM, external-knowledge processing, prompt generation, and utility-based feedback into plug-and-play modules. Its policy selects evidence acquisition, response generation, verification, and delivery actions, while utility functions score and improve candidate responses.
- The architecture uses Working Memory, Policy, Action Executor, and Utility modules to improve a fixed LLM with external knowledge and automated feedback.
- Working Memory: Working Memory stores the user query, consolidated evidence, candidate responses, utility scores, feedback, and prior dialog history.
- Policy: Policy selects evidence acquisition, candidate-response generation, or response delivery after Utility verification, maximizing expected reward.
- Action Executor: Knowledge Consolidator retrieves raw evidence through APIs, links entities to related context, prunes irrelevant information, and forms relevant evidence chains.
- Action Executor: Prompt Engine combines task instructions, query, dialog history, evidence, and available feedback into a task-specific prompt for candidate-response generation.
- Utility: Utility functions score response dimensions or rule compliance and can generate textual feedback from the query, evidence, candidate response, and dialog history.
3 Information Seeking Dialog
The information-seeking dialog experiments evaluate LLM-AUGMENTER with external knowledge, automated feedback, and rule-based or trainable policies across News Chat and Customer Service. Retrieved knowledge and feedback improve grounding, while policy choices trade off performance against knowledge-access overhead.
- 3.1 Datasets: News Chat uses Reddit-linked news documents, while Customer Service combines FAQ facts with subjective Yelp-review knowledge.BM25 retrieves from the corresponding knowledge sources, and oracle experiments also use golden knowledge.
- 3.2 System Setup: LLM-AUGMENTER augments fixed ChatGPT with modules that retrieve and consolidate evidence, generate responses, score utility, and revise prompts with feedback.Its policy selects actions such as acquiring evidence, querying the LLM, and sending responses that pass utility verification.
- 3.3 Automatic Evaluation Results: Approximately 10 and 6 KF1 points are gained from retrieved evidence on News Chat and Customer Service, respectively.Task-specific knowledge improves grounding while preserving response fluency and informativeness.
- 3.3 Automatic Evaluation Results: Automated feedback adds 3.3 KF1 points on News Chat and 7.2 on Customer Service with golden knowledge, with significant gains also observed using retrieved evidence.The Utility module evaluates grounding against consolidated or golden evidence and generates verbalized feedback.
- 3.3 Automatic Evaluation Results: The trainable T5-Base policy surpasses a random policy after 600 interactions and reaches approximately 37.5 KF1 on test data.The experiment defines reward as the KF1 utility function because no external user reward is collected.
- 3.4 Human Evaluation Results: Human evaluation on 948 Customer Service examples strongly prefers LLM-AUGMENTER over ChatGPT alone for both Usefulness and Humanness.Differences reported in Table 3 are significant at p < 0.05.
- 3.5 Ablation Study: Always-use achieves the best KF1 but adds knowledge-consolidator overhead, whereas Self-ask improves over No-knowledge and requests access for 24% of examples.The findings motivate a trainable policy that learns when external knowledge is needed.
- 3.5 Ablation Study: Combining utility functions with feedback-augmented prompting performs best, while prompting twice and reranking gives slightly higher KF1 but performs significantly worse than the combination.Always providing feedback also improves performance but requires additional prompting.
4 Wiki QA
Wiki QA tests LLM-AUGMENTER on multi-step questions requiring joint reasoning over Wikipedia tables and passages. Consolidated evidence and automated feedback improve grounded short-form answers, though performance remains below a fine-tuned model.
- 4.1 Dataset: OTT-QA contains around 40K Wikipedia-based instances requiring joint reasoning over 400K tables and 6M passages.Questions comprise 13% single-hop, 57% two-hop, and 30% multi-hop cases.
- 4.2 Experiment Setups: DPR retrieves initial tables and passages, while CORE’s linker and chainer connect, rerank, and splice them into evidence chains.The Knowledge Consolidator uses these chains because raw evidence is insufficient for multi-hop prompting.
- 4.2 Experiment Setups: The evaluation uses token-level precision, recall, and F1 against annotated short-form answers.The setup targets concise answers rather than the long-form responses preferred in conversational tasks.
- 4.3 Results: ChatGPT alone performs poorly, with hallucinated answers common and abstentions in 17% of cases.Adding retrieved or consolidated knowledge improves F1, with CORE producing stronger gains than raw DPR evidence.
- 4.3 Results: LLM-AUGMENTER still trails the state-of-the-art fine-tuned model using top-50 consolidated evidence.The authors attribute the gap to lower evidence recall and additional alignment demands for concise, faithful reasoning.
5 Related Work
Related work augments language models with external knowledge or adapts black-box model access. LLM-AUGMENTER instead assumes fixed APIs and emphasizes deciding when and what to request, interactive feedback, and self-learning utilities.
- External-knowledge augmentation: Earlier approaches augment LLMs with personalized, Wikipedia, web, or other external knowledge, but many require modifying or tuning model parameters.The paper positions its approach against methods that rely on parameter finetuning.
- Black-box LLMs: Recent work combines black-box LLMs with prompted knowledge, web information, ranker tuning, or access to different APIs.These approaches improve understanding and reasoning across various tasks.
- Positioning: LLM-AUGMENTER assumes its APIs are fixed and focuses on when and what APIs to request, interactive feedback, and utility-driven self-learning.The authors describe these emphases as complementary to work that tunes API access.
6 Limitations and Future Directions
The paper identifies latency, manually designed policies, and limited human evaluation as current constraints. Future work targets more available-model reinforcement learning, richer utility functions, human evaluation, and user-feedback training.
- Latency: Interactive feedback can significantly slow user experience because ChatGPT is often queried twice for one response.The authors suggest showing the initial response while the improved response is generated.
- Policy learning: The main results use a manually designed policy because ChatGPT’s demand and limited bandwidth constrained reinforcement-learning experiments.The policy was trained with T5-Base, while future work plans RL experiments involving ChatGPT when it becomes more available.
- Evaluation and utilities: The current version lacks human evaluation and uses groundedness-oriented KF1 as its utility function.Future work proposes finer-grained analyses for utilities such as safety and training from real-user interactions.
7 Conclusions
The paper introduces LLM-AUGMENTER as a framework that augments black-box LLMs with external knowledge and automated feedback. Experiments show improved grounding and text-generation metrics for ChatGPT, alongside end-to-end effectiveness with T5.
- 7 Conclusions: LLM-AUGMENTER augments fixed black-box LLMs with external knowledge and automated feedback.Its components are integrated into an end-to-end reinforcement-learning framework optimized with policy gradients.
- 7 Conclusions: External knowledge makes responses more grounded in information relevant to the current conversation.Automated feedback elicits follow-up corrections that produce responses ranking higher under utility functions such as KF1.
- 7 Conclusions: End-to-end T5 experiments demonstrate effectiveness, while ChatGPT experiments significantly increase KF1 and multiple text-generation metrics.The conclusion reports gains across groundedness and broader generation measures.
Ethics Statement
The paper focuses on factual integrity as a major harm of LLMs while leaving offensive-content generation outside its scope. It also cautions that external sources may compromise generated-text factuality, so users should check their relevance.
- The paper addresses factual integrity as a major harm of large language models.
- Offensive-content generation is outside the paper’s scope, although future utility functions might address it.
- External sources may compromise generated-text factuality, making source-relevance checking important.
A Appendix
The appendix provides customer-service response examples contrasting ChatGPT with LLM-AUGMENTER, alongside prompt templates for news chat, customer service, and Wiki QA. The examples highlight how retrieved external knowledge can support richer responses.
- Customer Service Examples: Table 6 contrasts prompts and example responses from different models in customer-service scenarios.
- Customer Service Examples: LLM-AUGMENTER’s Working Memory displays richer retrieved information intended to help ChatGPT generate more contentful responses.
- Prompt Templates: Tables 7–9 provide prompt templates for news chat, customer service, and Wiki QA, respectively.