SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine

1 Introduction

• Earlier large-scale question-answering datasets included open-world Freebase pairs and closed-world pairs generated from news or focused on specific question-answering aspects.The paper frames large-scale annotated data as a driver of deep-learning progress and motivates expanding it for question-answering.Source: §1 · p. 1
• Existing closed-world datasets provide relevant, well-written context, unlike general question-answering systems, which must handle noisy retrieved documents that may be irrelevant or ill-formed.The paper identifies this mismatch as a gap between existing datasets and a general system's retrieval-and-answering pipeline.Source: §1 · p. 1
• The dataset contains 140k+ Jeopardy! question-answer pairs augmented with 6.9M Google snippets, after filtering for answer presence and at least forty returned web pages.Questions begin without attached context; querying Google with each question yields realistic relevant and irrelevant snippets.Source: §1 · p. 2
• SearchQA is challenging for both humans and machines, with a significant performance gap between them.The authors present it as a resource for research toward better automated question-answering systems.Source: §1 · p. 2

Collection

• Google snippets retrieved for J! Archive question-answer pairs provide context selected by an in-production search algorithm.The dataset was designed to reflect noise from a realistic information retrieval system.Source: §2 · p. 2

Cleaning

• Cleaning removes snippets containing episode air-dates, exact questions, or quiz-related terms, checks frequently repeated URLs, and retains tuples with answers in context.URLs occurring more than 50 times were manually checked for Jeopardy! question-answer pairs; retained answers are three or fewer words long.Source: §2 · p. 2

Basic Statistics

• 140,461 question-answer pairs have 49.6±2.10 snippets per pair on average.Snippets average 37.3±11.7 tokens, answers 1.47±0.58 tokens, and the dataset contains 1,257,327 unique tokens.Source: §2 · p. 2

Meta-Data

• The public release includes Jeopardy! category, dollar value, show number, and air date, plus Google snippet URLs, titles, and related links.The authors note these metadata are included for potential future use, though not used in the paper.Source: §2 · p. 2

Training, Validation and Test Sets

• 99,820 training, 13,393 validation, and 27,248 test examples are split across non-overlapping years, with validation and test years later than training.Unigram-answer examples number 55,648, 8,672, and 17,056 in the training, validation, and test sets, respectively.Source: §2 · pp. 2–3

• SimpleQA (Bordes et al., 2015) is an open-world dataset with 100k question-answer pairs, Freebase as its knowledge base, and no explicit pair-to-entry links; its questions use simple subject-relationship form.Source: §3 · p. 3
• SearchQA is closed-world because its associated snippets suffice to answer each question; unlike Children’s Book (Hill et al., 2015) and CNN/DailyMail (Hermann et al., 2015), it starts from QA pairs rather than articles and need not use fill-in-the-blank questions.Source: §3 · p. 3
• SQuAD (Rajpurkar et al., 2016) and NEWSQA (Trischler et al., 2016) often use multiword answers not found verbatim in context and crowd-sourced questions or answers, whereas SearchQA ensures multiword answers occur in context and starts from QA pairs, not articles.Source: §3 · p. 3
• MS MARCO (Nguyen et al., 2016) uses selected real Bing queries, manually selected Bing snippets, and human answers; SearchQA differs in question source and search engine, making their comparison useful for studying those factors.Source: §3 · p. 3

4.1 Human Evaluation

• Thirteen volunteers answered randomly selected test questions through a snippet-display interface, with up to 40 minutes each; participants averaged 47.23 answers (SD 30.42).Volunteers were split between unigram-only questions and questions allowing unigram or n-gram answers.Source: §4.1 · p. 3
• Human accuracy differed significantly between unigram-only and unigram-or-n-gram groups, and F1 showed a large ASR–human gap; noisy snippets and trial exhaustion were suspected contributors.The authors suggest that snippets are often incomplete sentence excerpts and that volunteers may have become exhausted.Source: §4.1 · p. 4

4.2 Machine Baselines

• TF-IDF Max selects the highest-TF-IDF context word, exploiting snippets retrieved with the question as query, and applies only to unigram answers; vanilla ASR is a cloze-style pointer-network baseline.ASR encodes the concatenated snippets and the question with separate recurrent networks.Source: §4.2 · p. 4
• ASR performed slightly below human evaluation, while TF-IDF Max did not match ASR on the SearchQA baselines.The authors argue that improving these benchmarks is important for developing real-world question-answering systems.Source: Table 2 · p. 4, §4.2 · p. 4
• For multiword answers, n-gram ASR encodes preceding answer words into the question representation and predicts one word at a time until an appended <answer> token.During training, it uses the correct preceding answer words.Source: §4.2 · p. 4

• The latest technique, ASR, showed a meaningful gap between human and machine performance, suggesting SearchQA’s potential as a question-answering benchmark.Source: §5 · p. 4
• SearchQA was built using an in-production commercial search engine to reflect a general question-answering pipeline of information retrieval and answer synthesis.Source: §5 · p. 4
• SearchQA is publicly released with the authors’ PyTorch implementations of ASR and n-gram ASR.Source: §5 · p. 5