PaperlayerView exact revision on arXiv

Source-linked AI summary

Merging Generated and Retrieved Knowledge for Open-Domain QA

Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Lu Wang

arXiv:2310.14393v1cs.CLcs.AI

TL;DR

Retrieval can lack knowledge coverage, while LLM-generated passages can hallucinate conflicts with retrieved facts. COMBO pairs retrieved and generated passages by compatibility, using silver-label-trained discriminators before a FiD reader answers. It outperforms competitive baselines on all tested datasets except HotpotQA, by up to +1.9 exact match.

  • Problem

    Retrieved passages can have incomplete knowledge coverage, while LLM-generated passages may contain hallucinations that conflict with retrieved knowledge.

    Source: §1 · p. 1

  • Method

    COMBO matches retrieved and LLM-generated passages into compatible pairs using discriminators trained on automatically mined silver compatibility labels.

    Source: §1 · p. 2

  • Results

    Up to +1.9 exact match, COMBO outperforms competitive baselines on all testbeds except HotpotQA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    COMBO is reported as empirically successful across four open-domain QA datasets.

    Source: §5 · p. 9

    Experiments use one LLM, one retriever, and a generative reader based on Fusion-in-Decoder.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Open-domain question answering (QA) systems are often built with retrieval modules. However, retrieving passages from a given source is known to suffer from insufficient knowledge coverage. Alternatively, prompting large language models (LLMs) to generate contextual passages based on their parametric knowledge has been shown to improve QA performance. Yet, LLMs tend to "hallucinate" content that conflicts with the retrieved knowledge. Based on the intuition that answers supported by both sources are more likely to be correct, we propose COMBO, a Compatibility-Oriented knowledge Merging for Better Open-domain QA framework, to effectively leverage the two sources of information. Concretely, we match LLM-generated passages with retrieved counterparts into compatible pairs, based on discriminators trained with silver compatibility labels. Then a Fusion-in-Decoder-based reader model handles passage pairs to arrive at the final answer. Experiments show that COMBO outperforms competitive baselines on three out of four tested open-domain QA benchmarks. Further analysis reveals that our proposed framework demonstrates greater efficacy in scenarios with a higher degree of knowledge conflicts.

1 Introduction

Retrieval can be factual yet incomplete, while LLM-generated passages are relevant but may hallucinate; merging them directly can expose readers to conflicts. COMBO pairs compatible passages and uses silver labels to train compatibility scorers, and outperforms competitive baselines on most tested datasets.

1 Introduction

  • Retrieval can lack knowledge coverage, while generated passages may hallucinate; directly merging both sources can confuse readers when their facts conflict.The paper notes that readers tend to favor LLM passages during conflicts.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • COMBO pairs retrieved and LLM-generated passages by compatibility, then feeds the pairs to a reader to combine factual and relevant information.The framework is designed to address conflicts between the two knowledge sources.Source: §1 · p. 2
  • Silver compatibility labels are mined automatically by checking whether removing either or both passages flips a QA model’s prediction correctness.The labels train discriminators to score passage-pair compatibility without human annotation.Source: §1 · p. 2
  • Up to +1.9 exact match, COMBO outperforms competitive baselines on all testbeds except HotpotQA.The experiments cover three single-hop datasets and one multi-hop dataset.Source: §1 · p. 2

2 Related Work

Prior work examines knowledge conflicts and evidentiality in open-domain QA, but existing approaches do not fully address conflicts involving generated passages. COMBO instead promotes compatible evidence across retrieved and generated sources.

2 Related Work

  • Earlier conflict studies used perturbed retrieved contexts, while rule-based approaches chose between parametric and retrieved knowledge using predefined rules.COMBO presents its approach as generalizable rather than relying on dataset-specific heuristics.Source: §2 · p. 2
  • LLM outputs have been used to elicit encyclopedia, commonsense, and chain-of-thought evidence for QA.The cited work includes Yu et al. (2023), which directly augments retrieval passages with generated text.Source: §2 · pp. 2–3
  • Directly augmenting retrieval passages with LLM-generated text can leave knowledge conflicts unaddressed and mislead the reader.COMBO promotes factual evidence over hallucinating information in LLM outputs with retrieval knowledge.Source: §2 · p. 3
  • Prior evidentiality-guided readers add passage-level evidence signals, but those methods are insufficient for hallucinating LLM-generated passages.COMBO incorporates evidentiality from both retrieved and LLM-generated passages.Source: §2 · p. 3

3 Method

COMBO scores and matches LLM-generated passages with retrieved passages, then gives compatibility-ranked pairs to a Fusion-in-Decoder reader. Its compatibility criteria and silver-label procedure guide the pairing toward passages that jointly support an answer.

3 Method

  • COMBO scores cross-source passage pairs, matches them for compatibility, and feeds the ranked pairs to a Fusion-in-Decoder reader.The reader processes each passage pair with the question.Source: §3 · p. 3

3.1 Defining Compatibility

  • A pair is compatible when both passages support a correct answer; discriminators assess retrieved evidence and generated-passage consistency conditional on evidential retrieval.The formulation assumes retrieved passages are factual and generated passages are relevant, though not necessarily factual.Source: §3.1 · p. 4

3.2 Mining Silver Labels

  • Silver labels avoid human annotation: retrieval evidentiality uses leave-one-out prediction changes, while pair consistency is judged from reader correctness across four input variants.A pair is labeled consistent when the prediction is correct with all passages and either passage added, but incorrect without its retrieved member.Source: §3.2 · pp. 4–5

3.3 Matching Retrieved and LLM-generated Passages into Pairs

  • Each reader input concatenates the question with a generated-retrieved pair, enabling cross-passage attention while keeping retrieved evidence in the latter position.The fixed order trains the reader to rely on factual evidence in the retrieved passage when sources conflict.Source: §3.3 · p. 5
  • Matching scores all cross-source pairs, zeros scores for non-evidential retrievals, then selects a maximum-weight matching that covers passages while maximizing compatibility.The procedure includes all passages rather than filtering incompatible pairs.Source: §3.3 · pp. 5–6

4 Experiments

The fully supervised experiments use four single- and multi-hop QA datasets, DPR or MDR retrieval, and generated passages from InstructGPT or ChatGPT. COMBO improves over direct merging on the single-hop datasets and is tested through ablations, conflict-rate analysis, passage scaling, and human evaluation.

4.2 Results and Analysis

  • COMBO improves over Direct Merging by an average of 1.3 EM scores across the single-hop QA datasets.Both methods use the same reader model and input passages.Source: §4.2 · p. 6
  • Gains are bounded by source quality: uniform compatibility defaults to random matching at extreme clean-generation or evidence-free-retrieval settings, and COMBO does not improve overall on HotpotQA.The authors report only minor improvement on HotpotQA bridge questions and call for more fine-grained modeling of compatibility between evidence hops.Source: §4.2 · p. 7
  • Removing the evidentiality discriminator or pairwise input lowers performance, supporting evidence filtering and joint reasoning over paired passages.The evidentiality discriminator filters non-evidential retrieved passages; pairwise input enables reader self-attention across generated and retrieved passage tokens.Source: §4.2 · p. 7
  • Sorting pairs by compatibility and fixing passage order help prioritize compatible information and factual evidence when sources conflict.Ablations also report that optimal matching can bring together more compatible pairs than greedy matching.Source: §4.2 · pp. 7–8
  • COMBO’s improvement over Direct Merging is greater at higher conflict rates; at high conflict, only COMBO consistently improves over retrieval-only.Conflict rate is the percentage of conflicting passage pairs.Source: Table 3 · p. 8, §4.2 · p. 8
  • As the number of generated and retrieved passages increases, compatible pairs become less common and COMBO’s gains over direct merging grow.The NaturalQuestions analysis links the increasing gains with more input passages to rising knowledge conflicts.Source: Figure 5 · p. 8
  • In an example, COMBO ranks compatible passages containing correct evidence ahead of an incompatible pair with a hallucinated answer, rectifying Direct Merging’s prediction.The example’s generated passage contains the misleading answer “George Halas.”Source: §4.2 · p. 9, Figure 6 · p. 9
  • On 150 NaturalQuestions dev examples, the discriminators achieve 78% accuracy on three-way compatibility classification.The labels distinguish compatible, conflicting, and non-evidential pairs.Source: §4.2 · p. 9, Figure 7 · p. 9

5 Conclusion and Future Work

COMBO pairs retrieved and LLM-generated passages by compatibility before feeding them to a FiD-based reader, addressing conflicts caused by hallucinations. The authors report success on four open-domain QA datasets and propose exploring LLM-based compatibility scoring for few-shot QA.

Conclusion

  • COMBO matches retrieved and LLM-generated passages into compatible pairs and feeds them to a FiD-based reader to address hallucination-related knowledge conflicts.The framework is evaluated on four open-domain QA datasets.Source: §5 · p. 9

Future Work

  • Future work would use an in-context-learning LLM to compute compatibility scores for few-shot QA instead of training a discriminator from scratch.The authors note that it remains to be examined whether, and which, LLM can discern conflicts between its parametric memory and retrieved evidence.Source: §5 · p. 9

Limitations

The evaluation is limited to open-domain QA, selected question and passage distributions, and particular generators, retrievers, and a FiD-based reader. The authors also note computational overhead and possible limits on silver-label availability for small datasets.

Scope

  • The framework is evaluated only on open-domain QA, not other knowledge-intensive tasks such as fact checking or knowledge-enhanced text generation.The authors identify those other tasks as possible future applications.Source: Limitations · p. 10

Evaluation Setting

  • Questions and input passages were not distributionally manipulated, so the reported evaluation does not establish performance in controlled settings such as infrequent-entity questions.The authors anticipate potentially larger gains in such settings and note that LLMs appear more susceptible to hallucinating about less frequent entities.Source: Limitations · p. 10

Model Coverage

  • Each dataset uses passages from only one LLM and one retriever, and evaluation is limited to a generative reader based on Fusion-in-Decoder.The tested generators are InstructGPT or ChatGPT, and the retrievers are DPR or MDR.Source: Limitations · p. 10

Computational Overhead

  • On NaturalQuestions, COMBO adds approximately 23% GPU memory and 27% training time compared with direct merging.The authors report that practitioners able to train the direct-merging model should also be able to train COMBO.Source: Limitations · p. 10

Silver Labels

  • Silver-label mining may yield too few labels on a small dataset to train a dataset-specific discriminator.The authors identify a unified discriminator for multiple datasets and tasks as a needed alternative.Source: Limitations · p. 10

Ethics Statement

Because COMBO uses LLM-generated passages, it may inherit racial and gender biases and harmful content, and it does not fully eliminate hallucinations. The authors call for care before deployment in domains such as politics, finance, and healthcare.

Risks

  • COMBO may inherit biases and harmful content from LLM-generated passages, and combining them with retrieved evidence does not fully eliminate hallucinations.The authors advise practitioners to address these risks before deployment in politics, finance, and healthcare.Source: Ethics Statement · p. 10

A.1 Analysis of Reader’s Predictions under Knowledge Conflicts

The appendix examines reader behavior under knowledge conflicts, describes retrieve- and generate-then-read pipelines, extends compatibility definitions to multi-hop QA, and tests assumptions behind those definitions.

A.1 Analysis of Reader’s Predictions under Knowledge Conflicts

  • 16% of initially correct Retrieved-passage-only predictions became wrong when LLM-generated passages were added on the conflict-focused NaturalQuestions subset.The subset included questions with more than 20% conflicting passage pairs, showing that the FiD reader did not resolve all conflicts on its own.Source: §A.1 · p. 15
  • 65% of predictions were sub-spans of at least one generated passage, and the authors hypothesize that the reader relies more on those passages because they are more question-relevant and specific.The authors frame the explanation as a hypothesis.Source: §A.1 · p. 15

A.2 Retrieve/Generate-then-read Open-domain QA Framework

  • FiD separately encodes each question-passage pair, then its decoder attends across passage representations to generate an answer; direct merging supplies retrieved and generated passages together.A HotpotQA example illustrates a FiD reader being misled by a conflict between retrieved and hallucinating generated passages.Source: §A.2 · p. 15, Figure 8 · p. 15

A.3 Extension of Compatibility Definition from Single-hop to Multi-hop QA

  • For multi-hop QA, retrieved items are passage chains; in HotpotQA, each chain is an ordered two-passage pair, and the single-hop definitions adapt by substituting chains for passages.The authors describe these definition changes as minimal and sufficient for a unified formulation and implementation.Source: §A.3 · pp. 15–16, Table 4 · p. 16

A.4 Validating Assumptions behind the Compatibility Definition

  • 94% of manually annotated NaturalQuestions generations contained plausible answers, and 51% of those plausible answers were incorrect.The authors take the incorrect answers as evidence of hallucination risk and the challenge of merging knowledge.Source: §A.4 · p. 16

B Dataset Details

The evaluation covers three single-hop open-domain QA datasets and the multi-hop HotpotQA benchmark, with additional annotation examples drawn from NaturalQuestions.

B Dataset Details

  • NaturalQuestions, TriviaQA, and WebQuestions are single-hop testbeds; HotpotQA is the multi-hop testbed.The NaturalQuestions version uses questions and short answers, while HotpotQA is evaluated in the fullwiki setting.Source: B Dataset Details · p. 16
  • HotpotQA questions require reasoning across multiple Wikipedia documents, and the study uses its fullwiki setting.The benchmark uses the official leaderboard version for its train, development, and test splits.Source: B Dataset Details · p. 16
  • Table 5 presents plausibility and hallucination annotations from 100 random NaturalQuestions development-set samples.Source: Table 5 · p. 17

C Implementation Details

Silver evidentiality labels are based on whether removing a retrieved passage changes a QA model’s prediction from correct to incorrect or vice versa.

C.1 Silver Evidentiality Label Mining

  • A retrieved passage is labeled evidential when the top-N retrieved passages yield a correct prediction but removing that passage makes it incorrect.Source: §C.1 · p. 17
  • A passage is labeled non-evidential when the full retrieved set yields an incorrect prediction but removing that passage makes it correct.This criterion treats the passage as misleading the model when included with the remaining input passages.Source: §C.1 · p. 17
  • The labeling rule estimates a passage’s evidential role by testing whether its removal flips prediction correctness.Source: §C.1 · p. 17

C.2 Model Implementations and Hyperparameters

The implementation uses generated passage sets, silver-label discriminators, and a Fusion-in-Decoder reader, with separate settings for single-hop and multi-hop QA.

C.2 Model Implementations and Hyperparameters

  • For the three single-hop datasets, the study uses 20 InstructGPT-generated documents per question; HotpotQA uses ChatGPT-generated two-document chains.Generating 10 HotpotQA chains per question cost around US$400.Source: §C.2 · pp. 17–18
  • Figure 9 illustrates conflicting-pair mining by comparing reader predictions across inputs that retain or mask the target retrieved passage and append or omit the generated passage.A pair is labeled conflicting when the all-retrieved input is correct but the masked, appended, and combined variants are incorrect.Source: Figure 9 · p. 17
  • Table 6 reports EM and F1 on four QA datasets as averages and standard deviations across three random-seed runs.Source: Table 6 · p. 18
  • The evidentiality and consistency discriminators use RoBERTa-large for single-hop QA and DeBERTa-large for multi-hop QA, trained against silver labels.The models are trained for seven epochs, with the best checkpoint selected by silver-development-set F1.Source: §C.2 · p. 18
  • The FiD reader is trained for 100,000 steps, with checkpoint selection based on development-set exact match.Pairwise inputs allow maximum lengths of 400 tokens for single-hop QA and 1,000 for multi-hop QA; single-passage inputs use 200 and 500 tokens, respectively.Source: §C.2 · p. 18
  • Table 7 reports exact match in an oracle setting with ground-truth answers available during inference and includes state-of-the-art systems.The caption notes that those systems use larger models and additional knowledge.Source: Table 7 · p. 18

D.1 More Results

The additional analyses compare compatibility-guided matching with an oracle, examine question-type sensitivity to hallucinations, and track pair distributions and reader attention.

D.1 More Results

  • COMBO’s compatibility-guided matching performs close to the same-answer oracle upper bound, suggesting its discriminators identify passages containing correct evidence.Source: §D.1 · p. 18
  • Comparisons with state-of-the-art systems are not apples-to-apples because they may use larger readers, more retrieved passages, or additional knowledge formats.Source: §D.1 · p. 19

D.2 Comparison Questions from HotpotQA are Less Sensitive to Hallucinations in LLM-generated Passages

  • HotpotQA comparison questions are less sensitive to generated-passage hallucinations than bridge questions; in the example, the reader still predicts Keith Bostic correctly despite a fabricated birth date.Source: §D.2 · p. 19

D.3 Distribution of Pairwise Relationships

  • As the numbers of generated and retrieved inputs increase, compatible-pair frequency falls, alongside larger COMBO gains over direct merging.The analysis interprets the declining compatible-pair share as evidence of more conflicts; Figure 10 plots pair-type frequency against input passage counts.Source: §D.3 · p. 19, Figure 10 · p. 19

D.4 Analyzing Attention Distributions Over Different Types of Passage Pairs

  • COMBO generally assigns higher attention scores to compatible pairs than to incompatible pairs, indicating that it prioritizes compatible information.Figure 11 shows attention-score distributions across different numbers of generated and retrieved passages.Source: §D.4 · p. 19, Figure 11 · p. 19

Related papers

Loading 2310.14393v1…