PaperlayerView exact revision on arXiv

Source-linked AI summary

ELI5: Long Form Question Answering

Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, Michael Auli

arXiv:1907.09190v1cs.CL

TL;DR

Long-form question answering requires explanations for open-ended questions, a challenge underrepresented in datasets focused on short answers. ELI5 provides a large-scale corpus with web support documents and a multitask abstractive baseline; this model outperformed conventional baselines, but reference answers were preferred in at least 85.5% of pairwise comparisons.

  • Problem

    Existing question-answering datasets emphasize extractive or unambiguous short answers, while open-ended questions requiring explanations and paragraph-length responses have received less attention.

    Source: §1 · p. 1

  • Method

    The paper builds ELI5 from highly rated forum threads, gathers web support documents, and trains an abstractive Seq2Seq model jointly on language modeling, masked-word prediction, and answer generation.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Reference answers were preferred over trained models in at least 85.5% of pairwise judgments, while the multitask abstractive model outperformed conventional Seq2Seq, language modeling, and an extractive baseline.

    Source: §1 · p. 2, §6.1 · p. 7

  • Takeaways & Limitations

    ELI5 points to future work on extracting information from long, multi-document inputs and generating more coherent, accurate paragraph-length answers.

    Source: §7 · p. 9

    Long-form QA combines interdependent retrieval, reasoning, and writing challenges, including generating long coherent outputs from several supporting facts.

    Source: §6.2 · p. 9

Abstract

from arXiv · show

We introduce the first large-scale corpus for long-form question answering, a task requiring elaborate and in-depth answers to open-ended questions. The dataset comprises 270K threads from the Reddit forum ``Explain Like I'm Five'' (ELI5) where an online community provides answers to questions which are comprehensible by five year olds. Compared to existing datasets, ELI5 comprises diverse questions requiring multi-sentence answers. We provide a large set of web documents to help answer the question. Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline. However, our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.

1 Introduction

ELI5 frames long-form question answering as finding relevant information across long, diverse sources and generating paragraph-length explanations for complex, open-ended questions. Its multi-task abstractive baseline improves on conventional Seq2Seq, language modeling, and an extractive baseline, but remains far below human answers.

Introduction

  • ELI5 targets complex, diverse open-ended questions that require isolating relevant information in long source documents and generating paragraph-length explanations.Answers span multiple sentences and may take several valid forms; short answers or extracting a word or phrase do not suffice (Rajpurkar et al., 2016).Source: §1 · p. 1
  • Answering ELI5 questions requires combining web evidence because no single sentence or phrase contains the full answer; multi-sentence-support datasets such as TriviaQA still have short answers (Joshi et al., 2017).This contrasts with earlier datasets where lexical overlap could locate the human-written answer (Weissenborn et al., 2017).Source: §1 · pp. 1–2
  • The paper benchmarks extractive, retrieval, and generative models using intermediary fill-in tasks, ROUGE for overall answer generation, and human evaluations.Evaluation is challenging for this task and for multi-sentence text generation generally.Source: §1 · p. 2
  • Raters preferred gold answers 86% of the time, although the multi-task Seq2Seq baseline outperformed conventional Seq2Seq, language modeling, and a multi-sentence BidAF extractive baseline.The model was trained on language modeling, masked word prediction, and answer generation; its performance remains limited by comprehension of long multi-document input and generation of comprehensive long outputs.Source: §1 · p. 2

2 Related Work

Prior QA datasets largely target short extractive spans or short, mostly extractive written responses, whereas ELI5 is positioned as directed, paragraph-length question answering over multiple supporting documents.

Extractive QA

  • Extractive QA benchmarks constrain answers to short input spans, with HotpotQA (Yang et al., 2018) adding multi-paragraph multi-hop reasoning.They evaluate against ground-truth spans using exact match or F1; requiring answers to be copied from evidence precludes most “how” or “why” questions.Source: §2 · p. 2

Abstractive QA

  • NarrativeQA (Kocisky et al., 2018) and CoQA (Reddy et al., 2018) responses are mostly short and extractive; MS MARCO answers average around one sentence, while TriviaQA often allows short extractive spans.MS MARCO uses short input passages, whereas TriviaQA provides longer multi-document web input.Source: §2 · p. 2

Multi-document summarization

  • ELI5 frames paragraph-length, multi-document answering as query-based summarization, requiring directed question responses rather than general-topic text.DUC tasks also involve long inputs and multi-sentence generation but have much less training data than ELI5; WikiSum writes Wikipedia articles as multi-document summaries. Table 1 characterizes ELI5 answers as an order of magnitude longer and questions as more open-ended.Source: §2 · pp. 2–3, Table 1 · p. 3

3 Making a Long Form QA Dataset

ELI5 combines highly rated open-ended questions with retrieved web support to create a dataset for long, self-contained answers. Its analysis finds that support documents often contain relevant information, though they do not always provide a complete answer.

3.1 Creating the Dataset from ELI5

  • 272K questions passed voting filters, and each thread uses its highest-voted answer as reference; 63% have at least one other valid answer.Questions and at least one answer each needed a score of at least two.Source: §3.1 · p. 3
  • Because Wikipedia proved insufficient for ELI5’s broad topics and open-ended questions, the dataset uses English pages from Common Crawl’s July 2018 archive as web sources.Pages were tokenized with Spacy, identified as English using FastText, and indexed with Apache Lucene.Source: §3.1 · p. 3
  • For each question, retrieval gathers 100 non-Reddit web sources and builds a support document by selecting question-similar sentences with local context.The heuristic uses TFIDF similarity, then concatenates selected text with markers for non-contiguous passages and document shifts.Source: §3.1 · p. 3
  • Fifteen passages with one sentence of context on either side provide the best trade-off between support-document length and likelihood of including a high-ROUGE sentence.The dataset releases all 100 Common Crawl IDs per question and a script to construct support documents.Source: §3.1 · pp. 3–4
  • To limit memorization from similar questions, validation and test examples are sampled only from questions without close TFIDF neighbors in training.The final split contains 237K training, 10K validation, and 25K test examples.Source: §3.1 · p. 4

3.2 Dataset Analysis

  • ELI5 questions are longer and more open-ended than those in other datasets, often adding clarification or assumptions; even What questions tend to require paragraph-length explanations.The paper categorizes questions by interrogative words and notes that open-ended queries are less represented in other extractive or abstractive datasets.Source: §3.2 · p. 4
  • Support documents contain 22–60 sentences and average 858 words, while ELI5 answers average 6.6 sentences or 130 words.Source: §3.2 · p. 4
  • 94.5% of sampled gold answers fully address their questions using the support document; 65% of support documents contain the complete answer, and 92% contain relevant information.The assessment covered 500 randomly sampled training question-answer pairs and required answering every part of each question.Source: §3.2 · p. 4

4 Evaluation Methods

The evaluation combines reference-based measures of answer quality with human assessments of fluency, correctness, and preference. Generative models also receive metrics designed to assess important-word prediction and answer completion under reduced answer variability.

Evaluating long-form answers

  • ROUGE-1, ROUGE-2, and ROUGE-L F1 compare model outputs with gold answers and are used to rank related models at the corpus level.The authors note that diverse valid answers pose challenges for this metric, but still find it useful for ranking.Source: §4 · p. 4

Abstractive model metrics

  • Generative models are evaluated with perplexity, FILL-1 on nouns, verbs, and adjectives in the first 2K test examples, and ROUGE-20% answer completion.ROUGE-20% generates tokens equal to 20% of the average validation answer length, conditioned on the question, support document, and first 80% of the reference; it evaluates the final 20% with ROUGE.Source: §4 · p. 4

Human evaluation

  • Crowdworkers assess fluency on a 5-point Likert scale, answer correctness, and pairwise model preference, with three evaluators per assessment.The assessments include 500 fluency ratings, 500 correctness judgments, and 1,000 preference judgments; researchers also conduct 100 accuracy evaluations using support documents.Source: §4 · p. 5

5 Models

The section describes retrieval and extractive baselines alongside Transformer language-model and Seq2Seq approaches. It also introduces multi-task Seq2Seq training that combines generation and masked-word prediction tasks.

5.1 Extractive and Retrieval Models

  • The retrieval baseline returns the answer to the nearest training question by average word embeddings, while an extractive oracle selects sentences to maximize ROUGE against the reference.The oracle uses beam search to add sentences by TFIDF similarity, then reranks candidates by ROUGE; it selects up to 10 sentences with a beam of size 10.Source: §5.1 · p. 5
  • Extractive baselines return the seven support-document sentences with highest question TFIDF or use a BidAF model to score sentences and return the five highest-scoring ones.The BidAF model is trained to predict a high-ROUGE span of up to five contiguous sentences; five test sentences performed best versus three or seven.Source: §5.1 · p. 5

5.2 Abstractive Models

  • Transformer language models use Q + D + A or Q + A, while Seq2Seq models map Q to A or Q + D to A.Special tokens separate the question, document, and answer.Source: §5.2 · pp. 5–6
  • Multi-task Seq2Seq training combines decoder language modeling of Q + D + A with varied source-target tasks and masked-word prediction on 15% of input tokens.A task marker identifies the task for each sequence.Source: §5.2 · p. 6

6 Results

Across results, abstractive Seq2Seq methods generally outperform language models in full-answer quality and can compensate for imperfect support documents, but human answers remain preferred and model accuracy is limited. The analyses also show that local prediction metrics do not reliably reflect full-answer quality, while more training data improves performance.

6.1 Overview of Model Performance

  • The BidAF extractor scores 23.5 full-answer ROUGE-1, versus a 27.4 support-document oracle and a 54.8 full-web oracle, while abstractive methods generally achieve higher ROUGE.The nearest-neighbor baseline performs similarly to returning the support document, suggesting that memorizing training answers is insufficient.Source: §6.1 · p. 6
  • The multi-task Seq2Seq model reaches language-model perplexity and achieves 37.2 ROUGE-1 on ROUGE-20%, while language models remain better at FILL-1.The results suggest Seq2Seq better maintains coherence by relying on information across many time steps.Source: §6.1 · pp. 6–7
  • Human answers rate highest in fluency, while the multi-task model is similar to the extractive model; with supporting documents, 40% of extractive and 19% of multi-task answers are fully accurate.The multi-task model is more relevant than the extractive model (84% versus 68%), but may produce incorrect statements; crowdworkers assessing accuracy do not see the support document.Source: §6.1 · p. 7, Figure 4 · p. 7
  • At least 85.5% of pairwise judgments prefer the reference over every trained model; the multi-task abstractive model ranks next, and the language model is worst in at least 91% of comparisons.All pairwise judgments are statistically significant at p < 0.05.Source: §6.1 · p. 7, Figure 5 · p. 7

6.2 Quantitative and Qualitative Analysis

  • Perplexity and FILL-1 measure local prediction and are poor indicators of full-task appropriateness; full-answer ROUGE distinguishes models of the same architecture but cannot compare abstractive with extractive systems.ROUGE-20% focuses on coherence between an answer’s beginning and end, correlates with human quality judgments, and applies only to sequential generation.Source: §6.2 · pp. 7–8
  • Language models outperform Seq2Seq on perplexity and FILL-1 but fare worse on ROUGE-20% and human evaluations; their attention focuses on nearby context, unlike multi-task Seq2Seq attention to question and document information.This supports the analysis that local context alone is insufficient for high-quality answers.Source: §6.2 · p. 8, Figure 6 · p. 8
  • Both extractive and abstractive models score better when support documents overlap more with human answers, but abstraction better compensates when document relevance is low.The support-document oracle’s ROUGE-1 is 27.4, compared with 54.8 for the full-web oracle; with 500 selected passages, less than 9% of oracle sentences come from the first 20, and most oracles use at least one from the last 100.Source: §6.1 · p. 6, §6.2 · p. 8, Figure 7 · p. 8
  • Multi-task model ROUGE increases as training data grows from 25% through 100%, and over half of questions have multiple answers that could provide additional training references.The task combines retrieval, reasoning, and long-form writing; imperfect evidence makes abstraction useful relative to extraction, while the extractive system’s fluency gap suggests a need for sequential decisions to reason over facts and generate coherent outputs.Source: §6.2 · pp. 8–9, Figure 7 · p. 8

7 Conclusion

ELI5 is introduced as the first large-scale long-form question-answering dataset, and abstractive models produce coherent answers that are competitive with extractive models in human evaluation. The models remain far from human performance, partly because they cannot exploit the long full web text.

7 Conclusion

  • Abstractive models generate coherent answers and are competitive with extractive models in human evaluation.Source: §7 · p. 9
  • ELI5 is introduced as the first large-scale long-form question-answering dataset of open-ended queries with explanatory multi-sentence answers.Source: §7 · p. 9
  • The proposed models remain far from human performance, partly because they cannot exploit the long full web text.Source: §7 · p. 9
  • ELI5 aims to inspire work on extracting information from long, multi-document inputs and generating more coherent, accurate paragraph-length answers.Source: §7 · p. 9

A Details of Multitask Training … B.2 Abstractive Models

The appendix specifies the multitask objectives, extractive baseline setup, and abstractive model configurations. It describes task-specific training inputs and targets, BidAF implementation choices, Transformer training settings, and generation-length limits.

A Details of Multitask Training

  • Task-specific tokens distinguish training tasks, including question/document/answer reconstruction and answer or document prediction from different input combinations.The specified pairs include (empty, question + document), (empty, question + document + answer), (question + document, answer), and (question, document + answer).Source: A Details of Multitask Training · p. 11
  • Masked-word prediction replaces 15% of source words with [MASK] and trains the model to predict the corresponding tokens in order.Source: A Details of Multitask Training · p. 11

B.1 Extractive BidAF

  • BidAF uses AllenNLP’s standard hyperparameters except for batch size one and an Adam learning rate of 5e−5, with code for target-span selection and input subsampling.The input is converted to the SQUAD format required by AllenNLP.Source: §B.1 · p. 11

B.2 Abstractive Models

  • Abstractive models use Adam with beta values (0.9, 0.98), label smoothing 0.1, and an inverse-square-root learning-rate schedule with 4000 warmup steps.The initial learning rate is 1e−07, rising to 0.0001.Source: §B.2 · p. 11
  • The sequence-to-sequence Transformer has 6 encoder and 6 decoder layers, FFN dimension 4096, 16 attention heads, embedding dimension 1024, and 0.1 dropout and attention dropout.Gradient updates are delayed until 32 updates have been processed.Source: §B.2 · p. 11
  • Language models use 6 decoder layers and the sequence-to-sequence settings; 12-layer models were harder to optimize and performed worse, while generation is limited to 200–500 words.Source: §B.2 · p. 11

C Comparison of Extractive and Abstractive Methods · D Test/Valid Similarity with Train

An abstractive answer can retain strong ROUGE when its source document is poor, while the extractive answer may derail in topic. Full-answer generation shows little dependence on how similar validation questions are to training questions.

C Comparison of Extractive and Abstractive Methods

  • With a poor-quality source document, the abstractive answer still has strong ROUGE, while the extractive answer is affected by the source and derails in topic.The passage describes this contrast in an example shown in Figure 13.Source: C Comparison of Extractive and… · p. 11

D Test/Valid Similarity with Train

E Variance in Human Evaluation Studies

Human fluency ratings had small, nonsignificant variation across repeated trials, while crowdworker agreement varied by model and agreement threshold.

E Variance in Human Evaluation Studies

  • Across three different-day trials of 100 randomly sampled test question-answer pairs, fluency-score standard deviations were small and not statistically significant.The same pairs were evaluated in each trial.Source: E Variance in Human Evaluation Studies · pp. 11–12
  • All three crowdworkers agreed about 60% of the time for most models and almost 80% for the language model; at least two agreed almost 100%.Agreement meant all three workers selected positive, negative, or neutral fluency judgments on a five-option scale; the language model was significantly less fluent than the other models.Source: E Variance in Human Evaluation Studies · p. 12

F Examples

The examples show randomly sampled training question-answer pairs, training answers that fail to address the question, and test-set generations from two models. They also illustrate a low-overlap case where the extractive baseline goes off topic despite achieving a high ROUGE score from the abstractive system.

F Examples

  • About 5% of the dataset is estimated to contain training answers that do not answer their questions, illustrated alongside randomly sampled training pairs.Figures 11 and 12 show the sampled question-answer pairs and nonresponsive answers, respectively.Source: F Examples · p. 12, Figure 12 · p. 13, Figure 11 · p. 13
  • Random test-set generations are shown for the multi-task Seq2Seq and extractive BidAF models, with additional poor-quality examples illustrating problems in the Seq2Seq model.Figures 14–16 provide the sampled generations and poor Seq2Seq outputs.Source: F Examples · p. 12, Figure 14 · p. 15, Figure 15 · p. 16
  • Figure 13 shows a low document-answer overlap case where the abstractive system achieves a high ROUGE score, while the extractive baseline goes off topic after selecting a sentence about reformulated logos.The baseline confuses different types of logos and continues with an unrelated Disney analysis.Source: Figure 13 · p. 14

Related papers

Loading 1907.09190v1…