Source-linked AI summary
Hurdles to Progress in Long-form Question Answering
Kalpesh Krishna, Aurko Roy, Mohit Iyyer
TL;DR
LFQA research faces fundamental problems in evaluation and dataset creation that complicate meaningful modeling progress. The paper develops a retrieval-augmented system for ELI5 and finds that leaderboard success coexists with ineffective retrieval, substantial train/validation overlap, and unreliable evaluation signals. It therefore calls for more rigorous controls and metrics.
Problem
LFQA evaluation and dataset creation raise fundamental challenges that currently preclude meaningful modeling progress.
Method
The paper builds an ELI5 system using sparse Transformer attention, a REALM-style retriever, and contrastive learning with distant supervision from gold answers.
Results
The system achieves state-of-the-art performance on ELI5, but random retrieved documents barely change answer quality and analyses expose dataset-overlap and metric problems.
Takeaways & Limitations
Meaningful LFQA progress requires better dataset de-duplication, evaluation methods, and retrieval-use control experiments.
Takeaways & Limitations
The system does not ground its generations in retrieved documents, and ELI5 reflects biases prevalent on Reddit.
Abstract
from arXiv · showhide
The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress. To demonstrate these challenges, we first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. While our system tops the public leaderboard, a detailed analysis reveals several troubling trends: (1) our system's generated answers are not actually grounded in the documents that it retrieves; (2) ELI5 contains significant train / validation overlap, as at least 81% of ELI5 validation questions occur in paraphrased form in the training set; (3) ROUGE-L is not an informative metric of generated answer quality and can be easily gamed; and (4) human evaluations used for other text generation tasks are unreliable for LFQA. We offer suggestions to mitigate each of these issues, which we hope will lead to more rigorous LFQA research and meaningful progress in the future.
1 Introduction
LFQA combines document retrieval with paragraph-length answer generation, but progress on ELI5 is undermined by retrieval-grounding, dataset-overlap, and evaluation problems.
- LFQA combines searching external knowledge sources for relevant documents with generating paragraph-length answers.
- A sparse-attention system with a REALM-style retriever achieves state-of-the-art performance on ELI5.
- Retrieved documents have almost no effect on generated answer quality when replaced with randomly sampled documents.This pattern appears in both human A/B tests and automatic metrics such as ROUGE-L.
- The authors recommend reporting retrieval-control experiments alongside generation and retrieval quality.
- At least 81% of ELI5 validation questions have paraphrases in the training set, weakening the need for external retrieval.Almost all validation questions are topically similar to a training-set question.
- ROUGE-L is unreliable for LFQA: question-copying and random-answer baselines can outperform systems, while the system can exceed reference human answers despite human preferences.
2 A state-of-the-art LFQA system
The paper presents an LFQA system combining dense retrieval with a sparse-attention generator, achieving state-of-the-art ELI5 performance. Its design uses contrastive retriever learning, long-context modeling, and retrieved Wikipedia documents to condition answer generation.
- Task and motivation: LFQA combines retrieving relevant external documents with generating paragraph-length answers to open-ended questions requiring world knowledge.The ELI5 task uses questions whose answers often depend on world knowledge, motivating document conditioning.
- Retriever: The system uses a dense retriever that projects questions and candidate Wikipedia documents into a shared 128-d embedding space.C-REALM is trained with answer-based distant supervision and contrastive learning, bringing questions closer to ground-truth answers than in-batch negatives.
- Retriever: The retriever uses mini-batches of 12,288 examples and maximum inner-product search to efficiently return K = 7 documents during inference.The larger batches provide more negative examples, while ScaNN enables efficient top-document search.
- Generator: The generator is a Routing Transformer with sparse attention that models long-range dependencies across lengthy retrieved documents and questions.It is pretrained on PG-19 to encourage long and coherent generation, then fine-tuned by concatenating retrieved documents with each question and answer.
- Evaluation: The system evaluates ELI5 using retrieval metrics R-precision and Recall@5, generation metrics unigram F1 and ROUGE-L, and the combined KILT R-L score.The evaluation uses KILT validation and test subsets because the original ELI5 dataset lacks human retrieval annotations.
- Results: The system sets a new state-of-the-art on ELI5 and outperforms prior work on both generation and combined KILT scores.Four variants differ in inference-time retriever and nucleus-sampling settings; generated examples are long and coherent but repeat toward the end.
3 Analysis
The analysis finds that apparent LFQA progress on ELI5 is undermined by ungrounded retrieval, substantial train/validation overlap, and unreliable evaluation metrics and human judgments.
- 3.1 Are generations grounded in retrieval?: Randomly replacing retrieved documents leaves generation quality nearly unchanged, indicating that the system does not actually use its retrieved documents.Human A/B tests and automatic metrics support this finding.
- 3.1 Are generations grounded in retrieval?: There is almost no correlation between generation quality and unigram overlap with retrieved documents, strengthening the evidence that generations are not grounded in retrievals.The reported Spearman correlation is ρ = 0.09.
- Takeaways: The paper recommends random-retrieval controls, direct grounding measurements, stricter overlap prevention, and improved automatic metrics for future LFQA evaluation.Suggested dataset remedies include paraphrase retrieval with human validation or holding out entire genres or domains.
- 3.2 Training / Validation Overlap: 81% of validation questions have a paraphrase in the training set, while all annotated questions have a topically similar training question.This overlap can let systems retrieve and copy answers instead of generating them; the best top-K training answer reaches 28.5 ROUGE-L.
- 3.2 Training / Validation Overlap: Overlap substantially improves retrieval performance, with gains of 6.6 RPrec and 8.1 R@5, but yields only small generation gains of 0.8 F1 and 0.4 R-L.The small generation differences may be misleading because ROUGE-L is not reliable for LFQA.
- 3.3 ROUGE-L Bounds on ELI5 Performance: ROUGE-L is not selective enough to distinguish valid from invalid LFQA answers: trivial baselines can outperform strong systems, while human-preferred answers can score lower.A longest-gold-answer upper bound scores 21.2 versus the system’s 24.4, although humans prefer gold answers 68% to 14%.
4 Conclusion
The paper presents a retrieval-augmented generation system that achieves state-of-the-art performance on ELI5, but analysis exposes problems with the model, dataset, and evaluation metrics.
- The system achieves state-of-the-art performance on the ELI5 long-form question answering dataset.
- The analysis reveals issues with the model, the ELI5 dataset, and evaluation metrics.
- The authors call for solving these issues so LFQA research can make meaningful progress.
Ethical Considerations
The system shares known risks of modern text generation, including factual fabrication, misuse, and biases reflected from Reddit-based data.
- The system can fabricate facts, be misused, and reflect biases prevalent on Reddit.
- The ELI5 dataset is built from the r/ELI5 subreddit, making Reddit-related bias a relevant consideration.
- The authors conditioned generation on retrieved Wikipedia articles in an attempt to improve factual grounding and reduce fact fabrication.
A Appendices for “Hurdles to Progress in Long-form Question Answering”
The appendices describe the implementation, retrieval and generation experiments, parameter comparisons, example outputs, human evaluation setup, and ROUGE-L length analysis.
- A.1 Training & Model Details: The implementation uses TensorFlow 1.15, Tensor2Tensor, and open-source REALM and Routing Transformer codebases.
- A.1 Training & Model Details: The retriever and generator run in separate processes, with a frozen retriever and no document index refresher.
- A.1 Training & Model Details: The retriever trains for 4k steps, while the generator trains for 100k steps using Routing Transformer settings and dropout to prevent overfitting.
- A.1 Training & Model Details: Figure 3 depicts local sliding-window attention in lower layers and clustering-based global attention in upper layers.
- A.1 Training & Model Details: Pretraining on PG-19 with Routing Transformer significantly improved generation quality compared with preliminary strategies that produced short or incoherent responses.
- A.3 Number of trainable parameters: The model has a similar total parameter count to RAG and BART + DPR, despite a slightly larger generator and smaller retriever.
- A.4–A.6: The appendices include additional generations, human A/B evaluation procedures, and evidence that repeating truncated outputs raises ROUGE-L but remains below original-generation performance.
A.2 Ablation Study of C-REALM
The ablation studies examine contrastive retriever training and initialization, showing that larger minibatches and stronger pretrained initialization improve C-REALM retrieval performance.
- The retriever objective uses in-batch negative sampling, making minibatch size important for contrastive learning.
- Table 8 evaluates C-REALM validation performance across minibatch sizes against a pretrained REALM baseline.
- Table 9 compares C-REALM initialization with ICT and REALM-CCNews pretrained retrievers.
- Using a better pretrained initialization produces a large improvement in retrieval performance.
A.7 More experiments on measuring retrieval grounding of generations
Additional experiments consistently find little relationship between generation quality and overlap with retrieved documents, including when retrieval is correct. Comparisons with random retrievals likewise show very small differences, strengthening the evidence that generations are not grounded in retrievals.
- Generation quality and unigram overlap with retrieved documents show no correlation for the p = 0.6 model.Figure 4 compares ROUGE-L against reference answers with unigram overlap against retrieved documents.
- Even among instances with correct predicted retrievals, generation quality has very little correlation with retrieval overlap (Spearman ρ = 0.13).This analysis considers only cases where at least one retrieved document matched the gold annotation.
- For correctly retrieved instances, generations conditioned on C-REALM retrievals and random retrievals differ very little in quality or n-gram overlap.Table 12 compares ROUGE-L and overlap with predicted and randomly sampled retrievals.
- With p = 0.9 sampling, models conditioned on C-REALM and random retrievals again show very little difference across quality and overlap measures.The fine-grained analysis also finds little difference with and without retrieval, including noun and number overlap.