Source-linked AI summary
Towards Reliable and Fluent Large Language Models: Incorporating Feedback Learning Loops in QA Systems
Dongyub Lee, Taesun Whang, Chanhee Lee, Heuiseok Lim
TL;DR
LLMs can produce coherent answers that contain hallucinations, factual inaccuracies, and citation problems, making them difficult to trust or verify. The paper trains a critic on aspect-focused data and uses its feedback in an iterative learning loop; reported results show improvements in fluency and citation while maintaining high correctness.
Problem
LLM answers can include hallucinations, factual inaccuracies, or citation problems, making their reliability difficult for users to assess.
Source: §1 · p. 1Method
The paper constructs pseudo-labeled aspect-focused data, trains a critic to assess responses, and uses its feedback in an iterative learning loop for the answer-generating LLM.
Source: §1 · p. 1, §3.1 · p. 2, §3.3 · p. 3Results
Takeaways & Limitations
The authors frame the feedback loop as a step toward more useful and trustworthy LLMs in real-world applications.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Large language models (LLMs) have emerged as versatile tools in various daily applications. However, they are fraught with issues that undermine their utility and trustworthiness. These include the incorporation of erroneous references (citation), the generation of hallucinated information (correctness), and the inclusion of superfluous or omission of crucial details (fluency). To ameliorate these concerns, this study makes several key contributions. First, we build a dataset to train a critic model capable of evaluating the citation, correctness, and fluency of responses generated by LLMs in QA systems. Second, we propose an automated feedback mechanism that leverages the critic model to offer real-time feedback on heterogeneous aspects of generated text. Third, we introduce a feedback learning loop that uses this critic model to iteratively improve the performance of the LLM responsible for response generation. Experimental results demonstrate the efficacy of our approach, showing substantial improvements in citation and fluency metrics for ChatGPT, including a 4% precision increase in citation and an approximately 8% enhancement in the MAUVE metric for fluency, while maintaining high levels of correctness.
1. INTRODUCTION
LLMs can produce coherent answers containing hallucinations, factual inaccuracies, or citation problems, making them difficult to trust and verify. The paper proposes critic-based feedback and a learning loop to improve citation, correctness, and fluency in QA responses.
1. INTRODUCTION
- LLMs can generate engaging, coherent responses that still contain hallucinations and factual inaccuracies, leaving users struggling to trust or verify answers without evidence.The introduction also identifies erroneous citations and unnecessary or omitted information as answer-quality problems.Source: §1 · p. 1
- Retrieval-augmented LLMs use retrieved passages during training and inference, but do not ensure fidelity to them or explicitly provide citations.Prior work also relied substantially on costly and challenging human evaluation.Source: §1 · p. 1
- The paper proposes pseudo-labeled training data, critic models, and a feedback learning loop to improve fluency, correctness, and citation in QA systems.The reported approach improves fluency and citation precision and recall while maintaining high correctness.Source: §1 · p. 1
2. RELATED WORK
Prior work augments language models with retrieved information or retrieval during generation, while citation and feedback remain distinct concerns. This paper focuses on feedback for fluency, correctness, and citation in end-to-end QA systems.
2. RELATED WORK
- Prior work augments language models using retrieved information: Borgeaud et al. and Izacard et al.[4] pre-train using retrieved passages, while Khandelwal et al. and Zhong et al. augment LLM outputs via kNN interpolation.Source: §2 · p. 1
- Table 1 presents examples of citation and correctness problems in LLM-generated answers, with red text marking incorrect citations and factual inaccuracies.The caption identifies the table’s scope but does not state which type of problem is more prevalent.Source: Table 1 · p. 2
- Prompting and fine-tuning approaches support on-the-fly retrieval, while other methods generate first and then retrieve documents to revise for consistency.The passage names Schick et al. [10], Yao et al., Press et al. [12], Gao et al. [13], and He et al. [14].Source: §2 · p. 2
- The paper adds feedback on fluency, correctness, and citation to end-to-end systems that retrieve, synthesize, and cite documents.The related-work passage contrasts this focus with Gao et al. [7]’s end-to-end retrieval, synthesis, and citation system.Source: §2 · p. 2
3. PROPOSED METHOD
The method constructs aspect-focused data, trains a critic to score responses and provide feedback, then uses iterative feedback learning to improve a baseline LLM.
3.1. Constructing the Heterogeneous Aspect-Focused Dataset for Critic Model Training
- The training dataset uses ChatGPT to construct critic examples from annotated ASQA answers, with negatives created by adding fluency errors, ignoring search results, or mixing or removing citations.Fluency negatives use unnecessary repetition; correctness negatives do not use the provided search results.Source: §3.1 · p. 2
3.2. Training Critic Models to Provide Feedback on Heterogeneous Aspects
- The critic is trained to score fluency, correctness, and citation using the question, documents, and generated answer, producing positive or negative feedback signals.The implementation uses gpt-j-6B initialized from an SFT model and trains with one positive and three negative answers per aspect.Source: §3.2 · p. 3
3.3. Iterative Feedback Learning for LLM Enhancement on Heterogeneous Aspects
- The iterative feedback learning method uses a learned critic to identify aspects needing improvement and refine answers from a ChatGPT baseline.The paper contrasts this automated critic feedback with prior approaches relying on manual feedback from limited examples.Source: §3.3 · p. 3
4. EXPERIMENTS
The experiments evaluate the critic and feedback-learning approach on ASQA, measuring fluency, correctness, citation quality, and response length. The IFL models outperform ChatGPT on reported citation and fluency metrics, while IFL2 has a marginal correctness advantage.
4.1. Dataset
- Evaluation used the ASQA development dataset, part of ALCE, which focuses on citation skills and does not supply training data.Source: §4.1 · p. 3
4.2. Evaluation of the Critic Model
- The critic achieved 97.28% accuracy for fluency, 98.46% for correctness, and 97.96% for citation.The passage also reports average reward scores that support its discrimination of positive and negative responses across all three aspects.Source: §4.2 · p. 3
4.3. Evaluation Metrics for Heterogeneous Aspects
- The evaluation metrics cover fluency, correctness, and citation quality; EM Recall measures the proportion of correct short answers, while citation quality uses NLI entailment.The metrics passage also describes MAUVE as measuring similarity between response and source-material text distributions.Source: §4.3 · pp. 3–4
4.4. Quantitative Results
- IFL1 outperformed ChatGPT on MAUVE (85.96 vs. 77.51), indicating higher fluency, and on citation recall (62.22 vs. 56.26) and precision (67.26 vs. 63.51).Source: §4.4 · p. 4
- IFL2 also exceeded ChatGPT on MAUVE, scoring slightly above 78.24, while IFL1 substantially impacted fluency.Source: §4.4 · p. 4
- IFL models generated shorter responses than ChatGPT while achieving higher scores across various metrics.The passage characterizes these responses as more effective but does not specify response-length values.Source: §4.4 · p. 4
5. CONCLUSION
The paper concludes that its critic model and feedback learning loop address citation, correctness, and fluency shortcomings in LLMs. The approach improves fluency and citation metrics while maintaining high correctness.
5. CONCLUSION
- The approach improves LLM responses across citation, correctness, and fluency, while maintaining high levels of correctness.The conclusion characterizes the feedback loop as effective in refining LLM capabilities in QA systems.Source: §5 · p. 4
- The conclusion reports gains in MAUVE score for fluency and precision for citation.Source: §5 · p. 4
- The work is presented as a foundational step toward improving LLM utility and trustworthiness in real-world applications.Source: §5 · p. 4