Source-linked AI summary
Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources
Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, Lidong Bing
TL;DR
LLMs can hallucinate factual content, and existing retrieval methods are limited in source selection, structured-query generation, and progressive error correction. CoK dynamically adapts heterogeneous knowledge through source-specific queries and sequential rationale correction, improving performance across knowledge-intensive domains. The framework also supports modular use with different LLMs and knowledge-source formats, while external sources may introduce noise or conflicts.
Problem
Existing retrieval-augmented methods use fixed knowledge sources, struggle with structured query generation, and lack progressive correction, limiting their handling of specialized knowledge and dependent rationales.
Method
CoK prepares rationales and knowledge domains, retrieves heterogeneous knowledge with an adaptive query generator, progressively corrects rationales, and consolidates the final answer.
Results
CoK consistently improves LLM performance on knowledge-intensive tasks across factual, medical, physical, and biological domains, outperforming CoT by 4.3% on average.
Takeaways & Limitations
CoK provides a modular approach for grounding different LLMs with structured and unstructured knowledge sources across multiple domains.
Takeaways & Limitations
External knowledge sources may contain noise or conflicts, which the paper mainly addresses by selecting authoritative sources.
Abstract
from arXiv · showhide
We present chain-of-knowledge (CoK), a novel framework that augments large language models (LLMs) by dynamically incorporating grounding information from heterogeneous sources. It results in more factual rationales and reduced hallucination in generation. Specifically, CoK consists of three stages: reasoning preparation, dynamic knowledge adapting, and answer consolidation. Given a knowledge-intensive question, CoK first prepares several preliminary rationales and answers while identifying the relevant knowledge domains. If there is no majority consensus among the answers from samples, CoK corrects the rationales step by step by adapting knowledge from the identified domains. These corrected rationales can plausibly serve as a better foundation for the final answer consolidation. Unlike prior studies that primarily use unstructured data, CoK also leverages structured knowledge sources such as Wikidata and tables that provide more reliable factual information. To access both unstructured and structured knowledge sources in the dynamic knowledge adapting stage, we propose an adaptive query generator that allows the generation of queries for various types of query languages, including SPARQL, SQL, and natural sentences. Moreover, to minimize error propagation between rationales, CoK corrects the rationales progressively using preceding corrected rationales to generate and correct subsequent rationales. Extensive experiments show that CoK consistently improves the performance of LLMs on knowledge-intensive tasks across different domains.
1 INTRODUCTION
LLMs can generate plausible but factually incorrect text, while existing retrieval methods are limited by fixed sources, weak structured-query generation, and non-progressive correction. CoK addresses these issues by adapting heterogeneous knowledge dynamically, generating source-specific queries, and correcting rationales progressively.
- LLMs may confidently generate plausible but factually incorrect text, making factual knowledge control and updating difficult.
- Existing retrieval methods may miss specialized knowledge because they use a fixed knowledge source for all questions.
- Existing methods may generate ineffective structured queries because language models are mainly pretrained on natural-language sentences.
- CoK dynamically augments LLMs with heterogeneous knowledge sources and uses an adaptive query generator for source-specific queries such as SPARQL and natural sentences.
- CoK progressively corrects rationales so errors in preceding steps do not propagate into subsequent reasoning.
- CoK outperforms the CoT baseline by 4.3% on average across knowledge-intensive tasks spanning factual, medical, physical, and biological domains.
2 THE CHAIN-OF-KNOWLEDGE FRAMEWORK
CoK prepares rationales and relevant domains, retrieves knowledge with source-appropriate queries, progressively revises each rationale, and consolidates the corrected reasoning into a final answer.
- CoK has three stages: reasoning preparation, dynamic knowledge adapting, and answer consolidation.
- Reasoning preparation: Reasoning preparation generates preliminary rationales and answers, identifies relevant knowledge domains, and routes questions without majority consensus to knowledge adaptation.
- Dynamic knowledge adapting: Knowledge adaptation revises rationales sequentially, using preceding corrected rationales to generate and correct subsequent ones.
- Dynamic knowledge adapting: CoK retrieves knowledge by generating and executing queries matched to each source, including SPARQL, SQL, and natural-language queries.
- Answer consolidation: The final answer is consolidated from the corrected rationales after the iterative retrieval and correction process.
3 THE ADAPTIVE QUERY GENERATOR
The adaptive query generator supports heterogeneous knowledge sources by producing queries in structured or unstructured languages appropriate to each source.
- CoK uses heterogeneous sources across factual, medical, physics, and biology domains, requiring different query languages.
- AQG can be implemented with ChatGPT or a fine-tuned model, while its query language, model, and training data vary by knowledge source.
- Unstructured Query Languages: AQG generates unstructured natural-language queries for general factual and domain-specific knowledge sources.
- Structured Query Languages: AQG generates structured queries for sources such as Wikidata and tables, which can provide direct factual results.
4 EXPERIMENTS
Experiments evaluate CoK across knowledge-intensive tasks, domains, baselines, demonstration settings, and knowledge-source configurations. CoK consistently improves over CoT and retrieval-based baselines, while ablations support multiple sources and progressive knowledge adapting.
- CoK consistently outperforms CoT and CoT-SC across the evaluated datasets.
- CoK improves average accuracy over CoT by 2.6% on HotpotQA and 4.3% on FEVER in the reported 6-shot results.
- CoK achieves an average accuracy improvement of 4.9% over CoT on MedMCQA, MMLU Physics, and MMLU Biology across 3-shot and 6-shot settings.
- CoK improves over Verify-and-Edit and gains 2.0% over CoT-SC on HotpotQA, compared with ReAct’s 0.8% improvement.
5 ANALYSIS
The analysis examines why CoK needs multiple knowledge domains and sources, progressive rationale correction, and factuality evaluation. Results show gains from broader knowledge coverage and dynamic adaptation, while external sources can introduce noise or conflicts.
- Single vs. Multiple Knowledge Domains: 17.8% of MedMCQA questions identify Biology alongside Medical as a relevant knowledge domain.CoK predominantly selects one domain per dataset, but some questions require multiple domains.
- Single vs. Multiple Knowledge Domains: 1.3% improvement results when CoK adds Biology knowledge to Medical knowledge on MedMCQA.The ablation supports incorporating multiple knowledge domains for some questions.
- Single vs. Multiple Knowledge Sources: 2.1% improvement results when CoK uses both Flashcard and UpToDate instead of Flashcard alone.The analysis motivates using multiple credible sources within a domain because one source may not cover all relevant knowledge.
- Parallel vs. Dynamic Knowledge Adapting: 4.2% improvement over CoT occurs when dynamic knowledge adapting is applied.Parallel editing performs worse because errors can propagate between rationales, whereas CoK corrects rationales progressively.
- Evaluating Factuality Improvement of the Rationales: CoK improves factual accuracy over CoT-SC on HotpotQA, while CoT-SC factual accuracy decreases from rationale 1 to rationale 2.The evaluation uses ProgramFC to compare original and corrected rationales against Wikipedia.
- Evaluating Factuality Improvement of the Rationales: In human evaluation, CoK reasoning chains are consistently judged more factually consistent than CoT-SC chains.For incorrect predictions, evaluators judged CoK chains improved in factual consistency 44% of the time, and expected better answers in 73% of those cases.
6 RELATED WORK
Related work frames knowledge-intensive NLP as requiring external knowledge beyond local context and describes augmented language models as combining stronger reasoning with external resources to address hallucination.
- Knowledge-Intensive NLP: Knowledge-intensive NLP tasks require knowledge beyond local context and commonly use retriever-reader systems.A retriever selects relevant documents, after which a reader uses the extracted context to generate a response.
- Augmented Language Models: Augmented language models address traditional LLM hallucinations by adding improved reasoning capabilities and access to external resources.These enhancements can be used independently or combined in a task-specific sequence.
7 CONCLUSIONS
The conclusion presents CoK as a modular framework for progressive, knowledge-grounded generation across heterogeneous sources and query languages. It emphasizes AQG's adaptability and the framework's applicability across models, source formats, and deployment constraints.
- Conclusions: CoK enhances LLM factual correctness through progressive knowledge-grounded generation from heterogeneous sources and multiple domains.The framework is designed to support different LLMs and knowledge-source formats.
- Conclusions: AQG supports both unstructured and structured query languages and can transition between fine-tuned models and black-box LLMs.The conclusion identifies this flexibility as part of CoK's response to accurate query-generation challenges.
- Conclusions: CoK experiments on knowledge-intensive tasks demonstrate substantial improvement across the evaluated setting.The framework's modularity also addresses privacy concerns, knowledge-source reliance, and rapid information updates.
- FeTaQA: For FeTaQA, self-consistency is not used because open-ended generations may be near-equivalent without exact matches.CoK instead retrieves from external knowledge sources for every FeTaQA question.
- Query Execution: AQG queries Wikidata after entity and relation spans are replaced with linked IDs to make generated SPARQL executable.GENRE is used for entity linking before query execution.
- Query Execution: Generated natural-language queries are executed against domain-specific sources including Wikipedia, UpToDate, Physics Classroom, and CK-12 Biology.The supplied passages describe these sources as supporting factual retrieval in their respective domains.
- Query Execution: Generated SQL queries are executed on tables, and their results are consolidated with the original question for final answer generation.The original table is also provided because generated queries may be inaccurate.
D.4.2 DATA EXAMPLE
The appendix provides examples and implementation details for query generation, knowledge formatting, contrastive SPARQL training, and evaluation datasets. It illustrates how rationales are paired with generated queries, execution results, and formatted knowledge.
- Data Examples: Table 9 contains examples of generated queries, execution results, and formatted knowledge for each query language.These examples are used for rationale-level knowledge adaptation.
- Contrastive Instruction-Tuning: The SPARQL query model is trained with a contrastive autoregressive objective over input tokens, correct query tokens, and wrong query tokens.The loss applies to tokens identified as belonging to a query.
- Contrastive Instruction-Tuning: The indicator 1(x_i) equals 1 when token x_i is part of a query and 0 otherwise.This indicator selects query-token contributions in the training objective.
- Implementation: The implementation uses Llama-2 with LoRA parameter-efficient fine-tuning and 8-bit weights.Each knowledge source is trained for 3 epochs with batch size 32 and gradient accumulation of 2.
E EVALUATION DATASETS
The evaluation covers factual, medical, physics, and biology domains, using exact match for HotpotQA and assessing domain-selection quality with precision, recall, and F1.
- The evaluation datasets span factual, medical, physics, and biology domains.
- HotpotQA uses exact match as a strict evaluation metric.
- Domain selection is evaluated against manually annotated domains using precision, recall, and F1 on 50 questions per domain.
- Domain-selection F1 scores exceed 94% across all domains.
- SQL query generation is evaluated on 15,900 WikiSQL samples, while SPARQL generation uses 4,779 LC-quad and KQA-pro samples with exact match against gold queries.
G DISCUSSION OF LIMITATIONS
CoK remains bounded by the reliability and relevance of retrieved knowledge and by the reasoning capabilities of its underlying LLMs, although modularity permits component replacement.
- Unreliable or conflicting knowledge sources can make CoK generate inaccurate information, motivating the selection of authoritative sources such as Wikidata.
- CoK may not produce useful outputs when retrieval fails to find facts relevant to the question.
- Retrieval results inevitably contain noise from missing relevant data or inaccurate queries.
- The adaptive query-generator models can be swapped for models better suited to a task.
- Failure cases may stem from reasoning errors of the underlying LLMs, even when retrieved knowledge improves factual consistency.
H.2 EXAMPLES
The examples show that CoK can improve reasoning-chain factual consistency without guaranteeing a correct final answer, because LLM reasoning errors can persist.
- The examples demonstrate that improved factual consistency does not guarantee a correct final answer.
- Anne Sullivan’s birth date remains contradicted by an LLM despite repeated rationale statements that she was born in April.
- An LLM supports a French-nationality claim despite a rationale identifying Practical Magic as an American film and Alice Hoffman as American.
- An LLM supports the claim that Saturn Corporation has no other names despite repeated rationale statements identifying Saturn LLC as another name.
- In wrong predictions, humans judged CoK’s reasoning chains improved 44% of the time, and expected better answers in 73% of those instances.
I COST ANALYSIS
CoK’s selective editing keeps its cost comparable to Verify-and-Edit and substantially below ReAct while adding dynamic knowledge editing to improve performance.
- CoK’s costs are on par with Verify-and-Edit because it edits only instances below a consistency threshold.
- CoK costs around 40% of ReAct’s costs: 787 versus 1638 for HotpotQA and 329 versus 848 for FEVER.
- The additional cost comes from dynamic knowledge editing, which is reported to boost performance in the main results.
- Cost calculations use input tokens because output length is the same across methods.
- CoK and Verify-and-Edit apply editing based on the CoT-SC threshold, whereas ReAct applies its prompt to every instance.