Source-linked AI summary
ELI5: Long Form Question Answering
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, Michael Auli
TL;DR
ELI5 addresses the lack of large-scale data for open-ended questions requiring long, explanatory answers grounded in multiple documents. The paper constructs such a dataset and trains a multi-task abstractive model, which outperforms several baselines but remains far below human performance. The task is limited by retrieving and using information from long web inputs while generating accurate, coherent long answers.
Problem
Existing QA datasets provide limited coverage of open-ended questions requiring diverse, multi-sentence explanations from long source documents.
Method
The paper builds ELI5 from Reddit questions and answers, supplies retrieved web evidence, and trains a Seq2Seq model jointly on language modeling, masked word prediction, and answer generation.
Results
The multi-task abstractive model outperforms conventional Seq2Seq, language modeling, and extractive baselines, while human raters prefer gold answers in at least 85.5% of comparisons.
Takeaways & Limitations
ELI5 provides a large-scale benchmark for studying retrieval, reasoning, and long coherent answer generation together.
Takeaways & Limitations
Performance remains limited by the inability to exploit long full-web text and by the interdependent demands of retrieval, reasoning, and long-output generation.
Abstract
from arXiv · showhide
We introduce the first large-scale corpus for long-form question answering, a task requiring elaborate and in-depth answers to open-ended questions. The dataset comprises 270K threads from the Reddit forum ``Explain Like I'm Five'' (ELI5) where an online community provides answers to questions which are comprehensible by five year olds. Compared to existing datasets, ELI5 comprises diverse questions requiring multi-sentence answers. We provide a large set of web documents to help answer the question. Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline. However, our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.
1 Introduction
ELI5 targets open-ended questions that require relevant information from long sources and paragraph-length explanations, unlike short extractive QA. A multi-task abstractive model outperforms conventional and extractive baselines, but remains far below human quality.
- ELI5 addresses open-ended questions requiring multi-sentence explanations rather than short extractive answers.The task combines complex, diverse questions with paragraph-length responses.
- The dataset emphasizes answers that can have several valid formulations, making exact extraction insufficient.
- Models must isolate relevant information from long web documents because no single sentence or phrase contains the complete answer.
- A multi-task abstractive Seq2Seq model outperforms conventional Seq2Seq, language modeling, and a generalized extractive BidAF baseline.Training combines language modeling, masked word prediction, and answer generation.
- 86% of raters prefer gold answers to the best model outputs, leaving substantial room for improvement.
2 Related Work
Earlier QA datasets primarily target short extractive or short abstractive answers. ELI5 instead combines open-ended, paragraph-length answers with multi-document, question-directed reasoning.
- Extractive QA datasets constrain answers to spans and commonly evaluate exact match or F1.
- Even multi-hop extractive datasets such as HotpotQA retain short spans copied from supporting evidence.
- Existing abstractive datasets generally collect short, mostly extractive responses, including about one sentence in MS MARCO.
- ELI5 frames paragraph-length answering from multiple documents as query-based multi-document summarization.
- Unlike general-topic summarization, ELI5 requires directed answers to diverse questions that may involve multiple Wikipedia concepts.
- ELI5 answers are an order of magnitude longer and its questions more open-ended than those in large-scale QA datasets.
3 Making a Long Form QA Dataset
ELI5 is built from highly rated Reddit questions and answers, enriched with Common Crawl evidence and filtered support documents. Its questions, documents, and answers are substantially longer than typical QA data, while support-document coverage remains incomplete.
- 3.1 Creating the Dataset from ELI5: The dataset selects positively rated ELI5 threads, yielding 272K questions with at least one positively rated answer.
- 3.1 Creating the Dataset from ELI5: Common Crawl supplies broad web coverage because Wikipedia was insufficient for ELI5’s diverse, open-ended topics.
- 3.1 Creating the Dataset from ELI5: Each question receives 100 retrieved web sources, which are filtered into one support document using TFIDF-selected sentences and local context.
- 3.1 Creating the Dataset from ELI5: Selecting 15 passages with one sentence of context on each side provides the best trade-off between support length and relevant-information coverage.
- 3.1 Creating the Dataset from ELI5: Validation and test questions are chosen without close TFIDF neighbors in training data to reduce memorization from similar examples.
- 3.2 Dataset Analysis: ELI5 questions are longer and more open-ended than those in other datasets, often including clarifying paragraphs and multiple question types.
- 3.2 Dataset Analysis: Support documents average 858 words across 22–60 sentences, while answers average 6.6 sentences or 130 words.
- 3.2 Dataset Analysis: 94.5% of sampled gold answers fully address their questions, but only 65% of constructed support documents contain the complete answer.92% of support documents contain information relevant to the question.
4 Evaluation Methods
ELI5 evaluation measures fluency, correctness, relevance, and generation quality using automatic metrics and human judgments. The evaluation includes model-generated and extractive answers, with examples illustrating their differences.
- ROUGE ranks generated answers against one or more references despite the diversity of valid responses.
- Generative models are evaluated with perplexity, FILL-1, and ROUGE-20% for next-token prediction, salient-word generation, and answer completion.
- Figure 3 compares extractive BidAF, a Question + Document + Answer language model, and Multi-task Seq2Seq answers to a jellyfish question.
- Human evaluation assesses fluency, correctness, support-document-based accuracy, and pairwise preferences between model answers.
5 Models
The paper benchmarks retrieval, extractive, language-model, and Seq2Seq approaches, including a multi-task Seq2Seq model for long-form answer generation.
- Retrieval models: The retrieval baseline finds the training answer with nearest-neighbor search over question embeddings, while an oracle selects support-document sentences to maximize ROUGE.The oracle is also run over the full web sources to estimate the effect of retrieval coverage.
- Extractive models: Extractive baselines select question-relevant support-document sentences, including a BidAF-based model trained on extracted spans.The selected sentences are returned in their original order, with the output count tuned on validation data.
- Language and Seq2Seq models: The Transformer language-model and Seq2Seq configurations vary whether they condition on the question, support document, and answer.These configurations test how much information the model uses from the document during generation.
- Multi-task training: The multi-task Seq2Seq model alternates among generation tasks, including language modeling, masked word prediction, and answer generation.This provides more training signal than standard Seq2Seq training, which is trained only to predict answers.
- Generation: Abstractive generation uses beam search with beam 5, disallows repeated trigrams, and tunes minimum and maximum output lengths on the validation set.The implementation also applies byte-pair encoding and a 52,863-token answer vocabulary.
6 Results
Results show that multi-task abstractive models generally outperform language-model and extractive alternatives on long-form answer generation, but remain well below human quality. Performance is constrained by evidence selection, while more data and broader source access improve the achievable result.
- Full-answer evaluation: 54.8 full-answer ROUGE-1 is achieved by an oracle over full web sources, compared with 27.4 for the support-document extractive oracle.The support-document oracle therefore leaves substantial relevant information inaccessible to extractive systems.
- Model comparisons: Abstractive methods achieve higher ROUGE than extractive methods, while Seq2Seq models generally outperform language models.The authors attribute this pattern partly to abstractive models adapting to domain shift between web sources and ELI5 answers.
- Sequential-generation metrics: 37.2 ROUGE-1 on the final 20% of answers makes multi-task Seq2Seq stronger than standard Seq2Seq and language modeling on coherence-focused generation.Language models nevertheless perform better on FILL-1 prediction, suggesting local prediction quality does not capture overall coherence.
- Human evaluation: 84% relevancy for multi-task answers exceeds 68% for extractive answers, but only 19% of multi-task answers are fully accurate versus 40% of extractive answers.The extractive model is more accurate partly because it copies human-written sentences, whereas the abstractive model can produce incorrect statements.
- Human evaluation: Human reference answers are preferred in at least 85.5% of pairwise comparisons, while multi-task abstractive answers are preferred over extractive answers in only 57%.The language model is judged worse than every other setting in at least 91% of comparisons.
- Data and evidence analysis: Less than 9% of oracle-selected sentences come from the first 20 ranked passages, and most oracles use at least one sentence from the last 100.Handling the full relevant context would require inputs tens of thousands of words long; performance also increases with training-data size.
7 Conclusion
The paper introduces ELI5 as a large-scale long-form QA dataset and finds that abstractive models are competitive with extractive models but remain far from human performance.
- ELI5 is the first large-scale dataset of open-ended questions with explanatory multi-sentence answers.
- Abstractive models generate coherent answers and are competitive with extractive models in human evaluation.
- The models remain far from human performance, partly because they cannot exploit the long full web text.
- The paper positions ELI5 as a basis for future work on extracting information from long, multi-document input and generating coherent, accurate paragraph-length answers.
A Details of Multitask Training
The multitask Seq2Seq model is trained across language modeling, masked word prediction, and answer-generation tasks, using task-specific input–target configurations and long-form generation constraints.
- The multitask Seq2Seq model uses special tokens to identify each training task.Tasks are represented as source–target pairs, with concatenated inputs separated by a special token.
- Training includes language-modeling configurations that predict question and document text, with or without the answer.
- Training includes answer-generation configurations mapping question plus document to the answer, or question to document plus answer.
- Masked word prediction replaces 15% of source words with a [MASK] token and predicts the corresponding tokens in order.
- The extractive BidAF baseline is trained with standard hyperparameters except for batch size and Adam learning rate adjustments required by GPU memory.
- Seq2Seq models use six encoder layers, six decoder layers, 16 attention heads, 1024-dimensional embeddings, dropout, and delayed gradient updates.
- Language models use six decoder layers because a deeper 12-layer Transformer was harder to optimize and performed worse.
- Generated answers are constrained to between 200 and 500 words.
C Comparison of Extractive and Abstractive Methods
An example with a poor-quality support document shows abstractive generation retaining strong ROUGE, while the extractive answer is substantially disrupted by the document quality.
- Strong ROUGE is retained by the abstractive answer even when the source document is poor quality.The extractive answer is heavily affected by the poor document and derails in topic.
D Test/Valid Similarity with Train
The analysis finds little effect of validation-question similarity to training questions on answer generation performance.
- There is very little effect of answer generation when validation questions are more similar to training questions than when they are less similar.Similarity is determined using TFIDF.
- ROUGE for full answer generation is not strongly affected by validation-question similarity to training questions.
E Variance in Human Evaluation Studies
The human fluency evaluation is stable across repeated trials, while agreement among crowdworkers varies by model. Crowdworkers agree most often for language-model answers despite those answers being less fluent than other models.
- Three trials on the same 100 test question-answer pairs produced small, statistically insignificant variation in fluency ratings.Each trial occurred on a different day.
- Around 60% of the time, all three crowdworkers agreed about answer fluency for most models.Agreement means that all workers were positive, negative, or neutral on the five-point fluency scale.
- Almost 80% of the time, all three crowdworkers agreed about language-model answer fluency.The passage reports that language-model generations were significantly less fluent than the other models.
F Examples
The examples illustrate both the dataset’s varied question-answer content and recurring generation problems. They contrast multi-task abstractive outputs with extractive outputs and show cases where overlap metrics can obscure answer quality.
- Dataset examples: About 5% of the dataset contains answers that do not answer their questions.The paper shows randomly sampled training examples and separate examples of nonresponsive answers.
- Model outputs: The multi-task Seq2Seq and extractive BidAF models are illustrated with randomly sampled test-set question-generated answer pairs.Additional Seq2Seq examples are selected to represent poor generations.
- Dataset examples: Training examples cover questions about JPEG compression, flashy PC Gamer hardware, stale Oreos, spreading cracks, game consoles, transient visual whitening, and hidden television logos.The examples include both explanatory answers and low-quality or repetitive responses.
- Model outputs: In a logo question, the extractive baseline selects a reformulated question sentence but then confuses logo types and goes off topic with a Disney analysis.The abstractive system still achieves a high ROUGE score despite low document-answer overlap.
- Model outputs: A sampled response about presidential authority repeats claims about executive power and government control without providing a coherent explanation.The example begins with the question of who controls what the president is allowed to do.