PaperlayerView exact revision on arXiv

Source-linked AI summary

Read before Generate! Faithful Long Form Question Answering with Machine Reading

Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Pascale Fung

arXiv:2203.00343v1cs.CLcs.AI

TL;DR

LFQA needs paragraph-length answers, but generating faithful responses from multiple retrieved documents remains difficult. RBG couples a reader with generation; it tops the KILT ELI5 leaderboard, improves MS MARCO automatic results, and earns stronger human ratings for relevance and factual correctness.

  • Problem

    Faithfully synthesizing multiple retrieved documents to reduce hallucinated content remains a challenge in LFQA.

    Source: §1 · p. 1

  • Method

    RBG jointly models generation and machine reading, integrating sentence-level evidence scores as salient answer-related facts and adding factual-grounding pretraining.

    Source: §1 · p. 2

  • Results

    RBG tops the public KILT ELI5 leaderboard, improves automatic evaluation results over baselines on MS MARCO, and improves relevance and factual correctness in human evaluation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Answering extractive-QA questions is proposed as a potentially efficient way to evaluate LFQA faithfulness.

    Source: §7 · pp. 8–9

Abstract

from arXiv · show

Long-form question answering (LFQA) aims to generate a paragraph-length answer for a given question. While current work on LFQA using large pre-trained model for generation are effective at producing fluent and somewhat relevant content, one primary challenge lies in how to generate a faithful answer that has less hallucinated content. We propose a new end-to-end framework that jointly models answer generation and machine reading. The key idea is to augment the generation model with fine-grained, answer-related salient information which can be viewed as an emphasis on faithful facts. State-of-the-art results on two LFQA datasets, ELI5 and MS MARCO, demonstrate the effectiveness of our method, in comparison with strong baselines on automatic and human evaluation metrics. A detailed analysis further proves the competency of our methods in generating fluent, relevant, and more faithful answers.

1 Introduction

The introduction frames faithful synthesis across retrieved documents as a central LFQA challenge, then presents RBG’s reader-guided generation and factual-grounding pretraining. It reports improvements across ELI5 and MS MARCO, including stronger human-rated relevance and factual correctness.

1 Introduction

  • LFQA generates in-depth, paragraph-length answers, which matter for everyday questions requiring multisentence explanations.The introduction describes possible uses in search, conversation agents, and explanations accompanying short-phrase answers.Source: §1 · p. 1
  • Generation must synthesize multiple retrieved documents that may be redundant, complementary, or contradictory while reducing hallucinated content.The introduction identifies handling many documents as a generation challenge and faithfulness as an additional challenge.Source: §1 · p. 1
  • Prior LFQA work focused on handling many documents, while faithfulness remained unexplored; fluent, relevant but unfaithful answers can mislead users.The paper illustrates the risk with an example in Figure 1.Source: §1 · pp. 1–2
  • RBG combines a generator and reader, using sentence-level evidence scores to guide generation toward salient answer-related facts, and adds factual-grounding pretraining.The framework is presented as an end-to-end approach to LFQA faithfulness.Source: §1 · p. 2
  • RBG tops the public KILT ELI5 leaderboard, outperforms baselines on MS MARCO automatic evaluation, and improves relevance and factual correctness in human evaluation.The MS MARCO comparison includes retrieval and non-retrieval baselines.Source: §1 · p. 2

2 A state-of-the-art LFQA system

RBG retrieves documents, scores their sentence-level evidence with a reader, and uses those scores during generation. Its system combines multi-document encoding, reader-guided generation, and retrieval-augmented pre-training.

2 A state-of-the-art LFQA system

  • RBG passes retrieved documents and the question to a reader and generator, which use sentence-level evidence scores to produce an answer.The retriever, reader, and generator form the framework’s three components.Source: §2 · p. 2, Figure 2 · p. 3

2.1 Supporting document retriever

  • DPR retrieves the top-K documents for each question, which are ranked by relevance using dense question and passage representations.The representations are 768-dimensional, and retrieval uses approximate nearest neighbors with FAISS.Source: §2.1 · p. 2

2.2 Document reader

  • Because retrieved documents can be complementary, contradictory, or redundant, the reader predicts evidence spans and sentence-level evidence probabilities.The reader uses an MRC model on each question-document pair; its span predictions are used to score sentences.Source: §2.2 · pp. 2–3
  • The reader starts from SpanBERT and is fine-tuned across six MRQA datasets, then jointly trained with the generator using golden answers for distant supervision.LFQA data lack golden answer spans, motivating a reader with generalization ability for open-domain questions.Source: §2.2 · p. 3

2.3 Generator

  • FiD-BART encodes each supporting document separately and performs cross-attention jointly in the decoder.The encoder concatenates the outputs for the K documents into henc; this design can scale to many input documents with linear computation time.Source: §2.3 · pp. 3–4

2.4 Reader-before-generator

  • The reader-before-generator approach integrates sentence evidence into the generator’s process, using a soft switch between vocabulary generation and copying from the evidence distribution.The switch is pgen; copying samples according to Prea(w).Source: §2.4 · p. 4

2.5 Pre-training

  • Retrieval-augmented recovery pre-trains RBG to recover text from a masked pseudo-query and retrieved documents, encouraging reliance on external documents for factual statements.The task masks 30% of the original text’s words and retrieves top-k documents while discarding the original text itself.Source: §2.5 · p. 4

3 Experiment Setups

Experiments use ELI5 and MS MARCO, two datasets for long-form generative question answering. Baselines range from models without explicit retrieval to retrieval-augmented and multi-document generation systems.

3.1 Datasets

  • ELI5 contains Reddit question-answer pairs, with 272,634 training examples, 1,507 development examples, and answers averaging 130 words.The experiments use the KILT version of the dataset.Source: §3.1 · p. 4
  • MS MARCO experiments use question-answer pairs from the passage-ranking track, with about 500,000 training examples and 6,980 evaluation examples.The passage-ranking pairs are described as more abstract and reliant on multi-document information than the NLG track pairs.Source: §3.1 · p. 4

3.2 Baselines

  • BART and T5 fine-tuned on question-answer pairs without explicit retrieval serve as baselines relying on models’ internal knowledge.Source: §3.2 · p. 4
  • DPR-BART retrieves and prepends the top three DPR passages, then fine-tunes BART on context-enhanced training data.Source: §3.2 · p. 4
  • RAG retrieves five passages at each generation step, while FiD-BART independently encodes passages and combines encoder outputs before decoding.RAG is fine-tuned on LFQA tasks; FiD-BART is described as the strongest baseline.Source: §3.2 · pp. 4–5

4 Experiment Results

RBG outperforms the baselines on both evaluation metrics across KILT-ELI5 and MS MARCO, and human evaluation finds advantages over FiD in fluency, relevance, and factual correctness. Ablations show lower generation performance without the reader or pre-training, with a further drop when the reader is frozen; random retrieval also reduces ROUGE-L.

4.1 Automatic Evaluation

  • RBG outperforms all baselines on both evaluation metrics across KILT-ELI5 and MS MARCO.Table 1 compares RBG with baselines on both evaluation sets.Source: Table 1 · p. 5, §4.1 · p. 5
  • RBG tops the KILT-ELI5 leaderboard on retrieval performance, generation quality, and the combined KILT R-L metric.The reported retrieval metrics are R-precision and Recall@5; generation metrics are F1 and ROUGE-L.Source: Table 2 · p. 5

4.2 Human evaluation

  • Human evaluation finds that RBG outperforms FiD in fluency, relevance, and factual correctness, with a greater advantage on factual correctness.The evaluation sampled MS MARCO questions judged better supported by Wikipedia knowledge than ELI5 questions.Source: §4.2 · pp. 5–6, Table 4 · p. 6, Table 5 · p. 6

4.3 Ablation

  • Removing the reader reduces both evaluation metrics, removing pre-training lowers performance further, and freezing the reader also lowers both metrics, especially F1.The ablations are reported on the MS MARCO evaluation set.Source: §4.3 · p. 6, Table 6 · p. 6
  • Replacing retrieved paragraphs with randomly sampled Wikipedia paragraphs significantly lowers ROUGE-L, below BART and DPR-BART.This comparison tests generation with random retrieval at inference time.Source: §4.3 · p. 6

5 Further analysis

The further analysis examines how retrieval quality, the reader, and pre-training relate to answer generation, then evaluates faithfulness through zero-shot QA and case studies.

5.1 How does retriever affect the generation quality?

  • Higher-quality retrieved documents bring higher ROUGE-L on both datasets, and K=10 is best among {5, 10, 20, 50}.More retrieved documents do not improve generation quality.Source: §5.1 · p. 6, Table 7 · p. 6

5.2 How does the reader contribute to the generation?

  • The reader's ROUGE-L advantage over the no-reader ablation grows with retrieval quality, while human ratings favor the reader in fluency, relevance, and correctness.The fine-grained comparison is between RBG without pre-training and that model with the reader.Source: §5.2 · p. 7, Figure 3 · p. 7, Table 8 · p. 7

5.3 How does pre-training help?

  • With pre-training, the model performs better in most situations, and human evaluation favors it in relevance and correctness.The authors conjecture retrieval-augmented recovery helps the model combine evidence from different retrieved documents.Source: §5.3 · p. 7, Table 9 · p. 7, Figure 4 · p. 7

5.4 Faithfulness analysis

  • The faithfulness analysis uses zero-shot generation on NaturalQuestions and HotpotQA, estimating faithfulness with short-answer recall.Short-answer recall measures whether a gold answer span is contained in the generated long answer.Source: §5.4 · p. 7
  • RBG is comparable to FiD on NQ and consistently outperforms other strong baselines on HotpotQA in short-answer recall evaluation.The authors interpret this as evidence of capability on complex questions requiring information synthesis.Source: §5.4 · p. 7, Table 10 · p. 8
  • In the case studies, the reader produces more relevant and faithful answers, but one example still calls the second-oldest person the oldest.The authors suggest the reader may overemphasize one high-scoring sentence and weaken other sentences from the same document.Source: §5.4 · pp. 7–8, Table 11 · p. 8

6 Related work

Related work covers grounded generation, open-domain question answering, and query-driven multi-document summarization.

Grounded generation

Open-domain question answering

Query driven multi-document summarization

  • Query-driven multi-document summarization generates summaries conditioned on a query and relevant documents, with prior work incorporating query relevance or QA-based evidence ranking.Su et al. (2021) incorporated an answer relevance score for source documents into generation.Source: §6 · p. 8

7 Conclusion

The paper concludes that RBG jointly models generation and machine reading for LFQA faithfulness, with experiments and analyses supporting fluent, relevant, and more faithful answers.

Framework

  • RBG is an end-to-end framework that jointly models answer generation and machine reading to address faithfulness in LFQA.Source: §7 · p. 8

Evaluation

  • Experiments on ELI5 and MS MARCO report effectiveness against strong baselines on automatic and human evaluation metrics.Source: §7 · p. 8

Faithfulness evaluation

  • The paper proposes evaluating LFQA factual correctness with extractive QA questions, which may offer an efficient way to assess faithfulness.Source: §7 · p. 9

A Implementation details

The implementation uses pre-trained generator and reader models, a specified training and selection procedure, and a multi-task DPR retriever.

A Implementation details

  • Generation is initialized from BART-large, while the reader starts from SpanBERT and is fine-tuned on MRQA for four epochs.Source: A Implementation details · p. 11
  • RBG trains for 50k gradient steps with Adam at a 5e-5 learning rate, and uses beam search with beam size 4 at inference.The stated batch size is 4, and models are selected on a validation set based on ROUGE.Source: A Implementation details · p. 11

B Document retriever model details

C Human evaluation setup and analysis

The human evaluation used scored model comparisons and assessed agreement across fluency, relevance, and factual correctness. Annotators agreed most on fluency; the authors recommend preferred ratio as the core metric for factual-related evaluation.

Basic setup

  • Each model comparison used 50 questions, three annotators per answer, and 450 model-preference judgments per evaluation aspect.The 15 participants were computer-science researchers or students who spoke and read English well.Source: C Human evaluation setup and analysis · pp. 11–12
  • Questions were stratified across five development subsets representing five answer-passage overlap levels, and model answer positions were shuffled.The sampling strategy was used to reduce position-preference bias.Source: C Human evaluation setup and analysis · p. 11

Scoring setup

  • Annotators scored fluency, relevance, and factual correctness on a 1–3 scale, checking factual claims against Wikipedia, other web pages, and the golden answer.Answers with significantly fewer factual errors received higher factual-correctness scores.Source: C Human evaluation setup and analysis · p. 12

Statistical analysis

  • Annotator agreement was highest for fluency; correctness reached moderate agreement and relevance fair agreement in the RBG-versus-FiD comparison.The authors recommend preferred ratio as the core metric for factual-related evaluation.Source: C Human evaluation setup and analysis · p. 12
  • Fluency scores had the smallest variance across the four models, while correctness scores had the largest.The authors relate the greater difficulty of consistent correctness and relevance judgments to the complicated facts in generated answers.Source: C Human evaluation setup and analysis · p. 12

D Number of retrieved documents on Generation Quality

The authors varied the number of retrieved documents and found that generation quality did not continue to improve as more documents were added. In their MS MARCO experiments, performance was best at k = 10.

D Number of retrieved documents on Generation Quality

  • k = 10 gave the best performance in this MS MARCO setting; ROUGE-L and F1 did not further improve as the number of retrieved documents increased.Table 15 presents generation performance against the number of retrieved documents.Source: D Number of retrieved documents on… · p. 12

E Case of zero-shot experiments on NQ and HotpotQA

Zero-shot examples on NQ and HotpotQA compare RBG with other answer-generation approaches. The authors describe RBG’s responses as more fluent, specific, and factually accurate.

E Case of zero-shot experiments on NQ and HotpotQA

Loading 2203.00343v1…