Source-linked AI summary
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Mandar Joshi, Eunsol Choi, Daniel S. Weld, Luke Zettlemoyer
TL;DR
Reading-comprehension systems need datasets with complex questions, varied evidence, and cross-sentence reasoning. TriviaQA addresses this gap with independently authored questions, retrospectively collected evidence, and baseline evaluations, where the best neural model reaches 40% on TriviaQA versus 68% on SQuAD.
Problem
Reading-comprehension research needs increasingly difficult datasets that test complex questions, reasoning across sentences, and varied evidence retrieval.
Method
TriviaQA combines 95K trivia-authored question-answer pairs with roughly six independently collected Web and Wikipedia evidence documents per question.
Results
40% on TriviaQA versus 68% on SQuAD, with the neural baseline far below 79.7% human performance.
Takeaways & Limitations
TriviaQA is a challenging reading-comprehension testbed with substantial room for future research.
Takeaways & Limitations
The neural evaluation approximates answer spans by string matching and truncates evidence documents to their first 800 words.
Abstract
from arXiv · showhide
We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers. We also present two baseline algorithms: a feature-based classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension. Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that TriviaQA is a challenging testbed that is worth significant future study. Data and code available at -- http://nlp.cs.washington.edu/triviaqa/
1 Introduction
TriviaQA is introduced as a large-scale reading-comprehension dataset designed to test complex questions, lexical and syntactic variation, and multi-sentence reasoning. Its baselines remain far below human performance, establishing the dataset as a challenging resource for future study.
- Motivation: TriviaQA targets complex questions, reasoning across multiple sentences or background knowledge, and recovering facts despite lexical and syntactic variation.The dataset is designed to test these challenges simultaneously.
- Dataset: 650K+ question-answer-evidence triples combine 95K trivia-enthusiast question-answer pairs with an average of six supporting evidence documents per question.Questions were authored independently of evidence collected retrospectively from Wikipedia and the Web.
- Dataset: TriviaQA’s organically authored questions and retrospectively collected evidence distinguish it from datasets that primarily target individual reading-comprehension challenges.The paper describes this decoupling as enabling naturally occurring, full-sentence questions independent of NLP task requirements.
- Baseline experiments: 40% on TriviaQA versus 68% on SQuAD was achieved by the neural baseline, while human performance reached 79.7%.The neural model outperformed the linear classifier, but both baselines remained far below human performance.
- Additional resource: 1975 clean, human-annotated question-document-answer triples form a subset whose documents contain all facts required to answer the questions.This subset complements the automatically gathered large-scale dataset, which the paper characterizes as noisy.
2 Overview
TriviaQA frames reading comprehension as answering questions from sets of textual evidence documents, using automatically gathered Wikipedia or Web documents for distant supervision. Its supervision assumptions vary by source: redundant Web results are treated independently, while Wikipedia evidence is pooled.
- Problem Formulation: The task answers a question q using textual evidence from a document set D, with each correct answer string a appearing as a substring in some document.Unlike prior formulations using one document or a short paragraph, TriviaQA uses a set of documents as evidence.
- Data and Distant Supervision: Evidence documents are automatically gathered from Wikipedia or general Web search results, providing distant supervision because answer-string presence is assumed to indicate question-answering relevance.The automated process does not guarantee that documents contain every fact needed to answer a question.
- Data and Distant Supervision: For Web search results, redundant documents containing the correct answer are treated as independent question-answer-document data points.This setting uses |D_i| = 1 for every i, while many entries may share the same question.
- Data and Distant Supervision: For Wikipedia, where facts are generally stated once, all evidence documents are pooled and the same question is not repeated.The contrasting treatment reflects different expected redundancy between Web search results and Wikipedia evidence.
- Data and Distant Supervision: The setup could also support multi-instance learning or formulations allowing models to abstain when no evidence document contains the correct answer, but these are left for future work.The paper notes that experiments show the stated assumptions provide a strong signal for learning.
3 Dataset Collection
TriviaQA was constructed from trivia questions paired with independently collected Web-search and Wikipedia evidence, filtered to support distant supervision. The resulting dataset contains 95K question-answer pairs, 650K Web-search training examples, and 78K Wikipedia examples, alongside a larger unfiltered release.
- Question Collection: Questions came from 14 trivia and quiz-league websites, with items shorter than four tokens removed.The filtering targeted questions considered too simple or too vague.
- Evidence Collection: Evidence was collected from Bing’s top 50 search-result URLs for each question and from Wikipedia pages linked to question entities by TAGME.Trivia-site pages and URLs containing “trivia,” “question,” or “answer” were excluded from search results.
- Dataset Formation: 95K question-answer pairs remained after filtering evidence documents that lacked the correct answer string.The pairs were organized into evidence-document sets for distant supervision.
- Evidence Collection: Answers were excluded from search queries to avoid biasing the retrieved evidence.The search query consisted of the question rather than the answer.
4 Dataset Analysis
TriviaQA contains complex, diverse, and compositional questions whose answers are supported by independently collected evidence. Its question–evidence pairs exhibit substantial syntactic and lexical variation and frequently require multi-sentence reasoning.
- Question and answer analysis: The average question length is 14 tokens, indicating that many TriviaQA questions are highly compositional.Questions cover various topics of interest and concern diverse entities.
- Question and answer analysis: 73.5% of sampled questions describe fine-grained answer categories, while 15.5% hint at coarse-grained categories.Questions also often require reasoning over time frames and making comparisons.
- Question and answer analysis: 92.85% of answers are Wikipedia titles, 4.17% are numerical expressions, and the remainder are open-ended noun and verb phrases.Wikipedia answer titles span diverse topics, and many are common phrases rather than named entities.
- Evidence analysis: 79.7% of questions from the Wikipedia domain and 75.4% from the Web domain have answers in their evidence.Human annotators assessed answerability on 986 Wikipedia-domain and 1345 Web-domain dev and test questions.
- Challenging problem: 40% of questions required information scattered over multiple sentences, over three times SQuAD’s proportion, while 69% had different syntax and 41% differed lexically.Additionally, 17% required some form of world knowledge.
5 Baseline methods
The section evaluates TriviaQA difficulty using random, feature-based ranking, and neural baselines. It adapts BiDAF to TriviaQA’s missing answer spans and long, multi-document evidence setting.
- Baseline methods: The evaluation compares a random entity baseline and a simple classifier with BiDAF, a strong SQuAD reading-comprehension model.These baselines quantify dataset difficulty for current neural and other methods.
- Random entity baseline: 1.8 candidate entities per question are drawn from linked Wikipedia pages, then one absent from the question is randomly selected when possible.If no candidate is absent, the baseline selects any random candidate.
- Feature-based classifier: The classifier ranks document candidates with LambdaMART using context and Wikipedia-catalog features.Candidates are 1–5-grams in question-overlapping sentences that title Wikipedia articles.
- BiDAF adaptation: For multiple evidence documents, BiDAF predicts separately and aggregates candidate confidence scores across documents before selecting the highest-scoring answer.The modified model encodes context and questions at character, token, and phrase levels with attention between question and context.
- BiDAF adaptation: 800 words are retained because BiDAF was designed for single-paragraph evidence averaging 122 words and does not scale to TriviaQA’s long documents.Because answers lack exact spans, the method approximates them using the first answer-string match.
6 Experiments
Experiments show that TriviaQA remains challenging: BiDAF outperforms the classifier but both trail human performance, especially on longer questions and noisy, variable evidence. Error analysis attributes failures to missing or flawed evidence, compositionality, lexical variation, paraphrasing, indirect answers, and distractors.
- Experimental setup: 80%/10%/10%: QA pairs were partitioned into train, development, and test sets, with additional evaluation on verified development and test subsets.The entity classifier used 50,000 training questions, while web-domain BiDAF training used 80,000 sampled documents.
- Baseline performance: 40% vs. 23%: BiDAF outperformed the classifier on both Wikipedia and web documents, while the random entity baseline showed retrieval alone was insufficient.Oracle score denotes the upper bound on exact-match accuracy.
- Baseline performance: 79.7% and 75.4%: Both models lagged behind human baselines on Wikipedia and web domains, respectively.BiDAF accuracy declined from 50% for questions with at most five words to 32% for questions with at least 20 words.
- Error analysis: 19 examples lacked evidence, 3 had incorrect ground truth, and 3 valid answers were absent from the answer key among 100 incorrect BiDAF predictions.The analysis also identified compositionality, lexical variation, and low signal-to-noise ratio as challenges.
- Error analysis: Long, noisy documents, paraphrasing, indirect answers, and distractor entities caused substantial difficulty for BiDAF.The evidence can describe many similar roles or state the answer indirectly, as in examples involving Rebecca Front and Truman Capote.
7 Related work
Related QA datasets span cloze-style, natural-language reading comprehension, knowledge-base, and open-domain IR tasks, but differ from TriviaQA in scale, question authorship, complexity, and task focus. TriviaQA also connects to structured and quiz-bowl QA while emphasizing reading comprehension with answers present in documents.
- Dataset comparisons: Existing QA datasets are often limited in scale or biased by their construction processes, motivating comparison with TriviaQA.The paper groups existing datasets by task and summarizes the comparison in Table 1.
- Reading comprehension: Cloze-style datasets support large-scale automatic construction but require predicting missing words and do not contain natural language questions.Examples include datasets from Hill et al., Hermann et al., Paperno et al., and Onishi et al.
- Natural-language QA: MS Marco uses questions derived from search logs with crowdsourced answers, whereas TriviaQA has trivia enthusiasts provide both questions and answers.Both datasets contain independently authored questions and documents drawn from search results.
- KB and open-domain QA: Knowledge-base QA converts questions into executable logical forms but is limited by dataset scale or question complexity and by facts covered in the knowledge base.Open-domain alternatives include limited-size TREC competitions, SearchQA, WikiQA, and WebQA, while TriviaQA focuses on reading comprehension with answers present.
- Structured and quiz-bowl QA: TriviaQA’s Wikipedia-entity answers could support structured knowledge bases such as Freebase, while its examples may also serve answer-retrieval and answer-extraction stages.The paper leaves Freebase integration for future work and instead focuses on reading comprehension.
8 Conclusion and Future Work
TriviaQA is presented as a 650K question-document-evidence dataset whose trivia-authored questions are independent of evidence from Web search and Wikipedia. Its baseline results indicate a challenging testbed for future study, while also supporting benchmarks for several other question-answering tasks.
- Contribution: TriviaQA contains 650K question-document-evidence triples, with trivia-enthusiast questions authored independently of evidence documents.Its evidence spans Web search results and Wikipedia pages, which have differing levels of information redundancy.
- Result: TriviaQA is characterized as a challenging testbed that deserves significant future study.This conclusion follows from the reported results of current state-of-the-art baselines.
- Future work: TriviaQA also benchmarks IR-style question answering, QA over structured KBs, and joint modeling of KBs and text.It provides much more data for these tasks than was previously available.