Source-linked AI summary
Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering
Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, Zhang Xiong
TL;DR
Knowledge-intensive KBQA exposes hallucinated or unavailable intermediate reasoning in LLMs. KD-CoT uses structured multi-round interaction with an external QA system, supported by a KBQA CoT collection, and improves performance over vanilla CoT ICL on WebQSP and CWQ.
Problem
LLMs often produce incorrect or unfaithful intermediate reasoning in knowledge-intensive KBQA because they hallucinate or lack access to external knowledge.
Method
KD-CoT structures CoT as multi-round interaction between an LLM and an external QA system, using a KBQA CoT collection for demonstrations and retriever feedback augmentation.
Results
KD-CoT improves Hit@1 over vanilla ICL by 8.0 on WebQSP and 5.1 on CWQ, while the feedback-augmented retriever improves retrieved-knowledge Hit scores.
Takeaways & Limitations
The framework produces more interpretable inference steps and supports CoT fine-tuning, few-shot in-context learning, and more effective external-knowledge retrieval.
Takeaways & Limitations
Final performance remains below current SOTA because the QA system may miss precise sub-answers, reasoning hallucinations persist, and the framework is costly.
Abstract
from arXiv · showhide
Equipped with Chain-of-Thought (CoT), Large language models (LLMs) have shown impressive reasoning ability in various downstream tasks. Even so, suffering from hallucinations and the inability to access external knowledge, LLMs often come with incorrect or unfaithful intermediate reasoning steps, especially in the context of answering knowledge-intensive tasks such as KBQA. To alleviate this issue, we propose a framework called Knowledge-Driven Chain-of-Thought (KD-CoT) to verify and modify reasoning traces in CoT via interaction with external knowledge, and thus overcome the hallucinations and error propagation. Concretely, we formulate the CoT rationale process of LLMs into a structured multi-round QA format. In each round, LLMs interact with a QA system that retrieves external knowledge and produce faithful reasoning traces based on retrieved precise answers. The structured CoT reasoning of LLMs is facilitated by our developed KBQA CoT collection, which serves as in-context learning demonstrations and can also be utilized as feedback augmentation to train a robust retriever. Extensive experiments on WebQSP and ComplexWebQuestion datasets demonstrate the effectiveness of proposed KD-CoT in task-solving reasoning generation, which outperforms the vanilla CoT ICL with an absolute success rate of 8.0% and 5.1%. Furthermore, our proposed feedback-augmented retriever outperforms the state-of-the-art baselines for retrieving knowledge, achieving significant improvement in Hit and recall performance. Our code and data are released on https://github.com/AdelWang/KD-CoT/tree/main.
Introduction
The paper addresses unfaithful reasoning in knowledge-intensive KBQA, where hallucinated or unavailable sub-answers propagate into later reasoning and final answers. It proposes KD-CoT, external-knowledge interaction, and a KBQA CoT collection to improve faithful reasoning and retrieval.
- LLMs can hallucinate or fail to answer sub-questions, causing erroneous subsequent reasoning and incorrect final answers.
- Directly supplying extensive context can impair understanding, while prior work has not sufficiently addressed hallucinations in complex multi-hop KBQA.
- KD-CoT uses a QA system to access external knowledge and provide high-quality answers that help LLMs verify and adjust intermediate reasoning steps.
- The paper constructs a KBQA CoT collection for in-context learning and uses it as feedback augmentation to train a robust retriever.
- 8.0 and 5.1 Hit@1 improvements over vanilla ICL are reported on WebQSP and CWQ, respectively.
Methodology
The methodology builds a CoT collection and uses KD-CoT to interact with external knowledge during multi-round reasoning. Its QA pipeline retrieves knowledge, proposes answers, verifies them, and feeds corrected intermediate answers back to the LLM.
- CoT Collection: The CoT collection starts from human-annotated demonstrations and iteratively adds LLM-generated rationales selected by similarity and checked against training answers.
- Knowledge-Driven CoT: KD-CoT structures reasoning as interactions between a prompted LLM and a QA system that accesses external knowledge.
- Knowledge-Driven CoT: For each test question, the most similar collection instance supplies a one-shot rationale, whose intermediate sub-question becomes the QA system input.
- Knowledge-Driven CoT: The retrieve-then-read module retrieves external knowledge and proposes an answer, while the verifier selects between the original and proposed sub-answers or generates a new answer.
- Knowledge-Driven CoT: Interaction is repeated for intermediate sub-questions, while the final Action is excluded because the method supervises intermediate reasoning rather than altering the ultimate answer.
- Knowledge-Driven CoT: The retriever combines Wikipedia and KB passages and augments the original question with the last reasoning sub-question from the CoT rationale.
Experiment
Experiments evaluate KD-CoT on WebQSP and CWQ, showing gains over vanilla CoT ICL, improved retrieval, and correction of previously incorrect answers. Ablations indicate that structured multi-round QA, external knowledge access, and verification each contribute to performance, while ReAct-style structure limits flexibility.
- Main Results: 13.5% and 10.6% of previously incorrect questions are corrected on WebQSP and CWQ, while 5.8% and 5.4% become incorrect.These percentages compare answers before and after the interactive framework.
- Limitations: ReAct-format CoT reduces reasoning flexibility because structural constraints can prevent sub-question extraction and lead to counted failures.The authors note that this may underestimate LLM capability.
- Retrieval Results: Feedback-augmented DPR improves Hit@100 by 3.8 points on WebQSP and 7.8 points on CWQ over prior results.Table 3 compares retrieval results using answer hit and recall metrics for top-N passages.
- Analysis & Ablation Study: Structured multi-round QA demonstrations outperform other ICL methods, whereas directly concatenating retrieved knowledge performs worst in complex multi-hop scenarios.The constructed CoT collection further improves LLM ability, and removing the retrieve-then-read pipeline degrades performance.
- Iteration Analysis: The first iteration yields the highest WebQSP performance, while the last iteration yields the highest CWQ performance.WebQSP is primarily single-hop, whereas CWQ contains more complex multi-hop questions; about half of sub-answers are modified for the next iteration.
Related Work
Prior work improves LLM reasoning, retrieval, reading, and structured decomposition for KBQA, but existing approaches leave faithful multi-hop reasoning insufficiently addressed. KD-CoT builds on these directions by using LLM-generated coherent sub-questions with external knowledge interaction.
- LLM Reasoning: CoT prompting and decoding methods improve LLM reasoning, while other studies target more accurate or efficient chains of thought.These approaches motivate continued work on reliable reasoning traces.
- External Knowledge: LLMs remain limited by training-corpus knowledge, motivating methods that let them interact dynamically with the real world for needed information.This limitation is especially relevant to knowledge-intensive question answering.
- KBQA Systems: KBQA systems commonly use retrieve-then-read pipelines, with research improving retriever efficiency, reader performance, or structured knowledge integration.The retriever extracts pertinent corpus passages and the reader produces answers from retrieved content.
- Question Decomposition: Earlier methods decompose multi-hop questions into single-hop sub-questions, but rule-based decompositions can be unnatural and model-based methods can be capacity-constrained.KD-CoT instead uses CoT to induce more comprehensible and logically coherent sub-questions.
Conclusion
The paper introduces KD-CoT to improve LLM reasoning on knowledge-intensive KBQA tasks, reporting superior performance with interpretable inference steps. It also contributes a KBQA CoT collection and a robust retriever with improved retrieved-knowledge Hit scores.
- KD-CoT improves LLM reasoning performance on knowledge-intensive KBQA tasks through interpretable inference steps.
- The KBQA CoT collection supports CoT fine-tuning and few-shot in-context learning.
- A new retriever-training approach efficiently accesses external knowledge and substantially improves retrieved-knowledge Hit scores.
Discussion & Future work
The method remains behind the current state of the art despite correcting sub-answers and efficiently accessing external knowledge. The authors attribute this gap to imprecise QA answers, residual reasoning hallucinations, and the cost of the extra LLM and verifier.
- The final LLM performance remains behind the current state of the art despite correcting generated sub-answers.
- The QA reader achieves only 73.7 Hit@1 on WebQSP, limiting precise answers for all sub-questions.
- LLMs can still hallucinate reasoning despite corrections to precise answers.
- The framework is costly because it uses an extra LLM and a verifier.
Implementation details
The implementation uses Deepspeed with specified software versions, LoRA-based fine-tuning, AdamW, LinearDecay warmup, and eight 40G Nvidia A100 GPUs. The appendix also identifies tables covering model parameters and hyperparameter settings.
- Table 5 lists utilized models and their trainable parameters.
- Table 6 lists hyperparameter settings for QA-system training and CoT fine-tuning.
- Training uses Deepspeed with pytorch==2.0.0, peft==0.2.0, and transformers==4.29.1.
- Models larger than 1B use bfloat16 precision, while Llama fine-tuning uses LoRA adapters with dimension 16.
- Optimization uses AdamW with Beta=[0.9, 0.95] and LinearDecay with a warmup ratio of 0.1.
- All experiments run on 8 × 40G Nvidia A100 GPUs.
Demonstration illustration
The demonstration illustration shows how final and composition answers are added as hints when constructing the CoT collection and removed from selected demonstrations during inference.
- Final answers and composition answers are added as “Hint” before the structured rationale to construct the CoT collection.
- The “Hint” is removed from the chosen demonstration during in-context learning inference.
- Figure 7 illustrates one example for collection construction and another for inference-time in-context learning.
Case Analysis
Figures 8 and 9 show KD-CoT correcting sub-answers so subsequent LLM reasoning becomes more dependable and accurate. The cases illustrate this process across ComplexWebQuestion and WebQSP examples.
- Correction effects: KD-CoT corrects sub-answers, making subsequent reasoning more dependable and accurate and ultimately producing correct final answers.The before-and-after cases distinguish original hallucinations from faithful reasoning after correction.
- Reasoning format: The case-analysis instruction structures inference as interleaved Thought, Action, and Answer steps.The displayed instruction introduces Action as a component of the structured reasoning process.
- Reasoning format: Example questions are decomposed into Multi_Answer_Question actions that retrieve candidate sub-answers before the model continues reasoning.The examples cover a New York University founder question and a Carrie Underwood episode question.
- Correction effects: Figures 8 and 9 contrast original LLM hallucinations with faithful reasoning after sub-answer correction for CWQ and WebQSP.Red blocks denote original hallucinations, while green blocks denote faithful reasoning after correction.