Source-linked AI summary
SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, Kyunghyun Cho
TL;DR
SearchQA addresses the gap between context-first question-answering datasets and general systems that retrieve noisy web content before synthesizing answers. The paper constructs the dataset from Jeopardy! question-answer pairs augmented with Google snippets, evaluates human and machine performance, and finds a meaningful gap that supports its use as a benchmark.
Problem
Existing closed-world datasets guarantee relevant, well-written contexts, whereas general question-answering systems must handle noisy retrieved documents.
Method
The paper constructs SearchQA from Jeopardy! question-answer pairs augmented with Google snippets and evaluates it with human volunteers and an attention sum reader.
Results
140,461 question-answer pairs with 49.6±2.10 snippets per pair comprise SearchQA, and evaluation finds a meaningful gap between humans and machines.
Takeaways & Limitations
SearchQA provides a challenging benchmark and resource for advancing question-answering research and building better automated systems.
Abstract
from arXiv · showhide
We publicly release a new large-scale dataset, called SearchQA, for machine comprehension, or question-answering. Unlike recently released datasets, such as DeepMind CNN/DailyMail and SQuAD, the proposed SearchQA was constructed to reflect a full pipeline of general question-answering. That is, we start not from an existing article and generate a question-answer pair, but start from an existing question-answer pair, crawled from J! Archive, and augment it with text snippets retrieved by Google. Following this approach, we built SearchQA, which consists of more than 140k question-answer pairs with each pair having 49.6 snippets on average. Each question-answer-context tuple of the SearchQA comes with additional meta-data such as the snippet's URL, which we believe will be valuable resources for future research. We conduct human evaluation as well as test two baseline methods, one simple word selection and the other deep learning based, on the SearchQA. We show that there is a meaningful gap between the human and machine performances. This suggests that the proposed dataset could well serve as a benchmark for question-answering.
V. U˘gur G¨uney
The passage identifies a senior data scientist affiliated with Driversiti and NYU’s Center for Data Science.
- He is listed as a Senior Data Scientist at Driversiti.
- Mike Higgins is identified with the Center for Data Science at NYU.
1 Introduction
SearchQA addresses a mismatch between existing question-answering datasets and a general system that must retrieve and synthesize answers from noisy web content. It constructs question-answer pairs first, augments them with Google snippets, and evaluates the resulting benchmark with humans and machines.
- A general question-answering pipeline reformulates a question, retrieves documents through a search engine, and synthesizes an answer.
- Existing closed-world datasets provide guaranteed relevant, well-written contexts, unlike general question-answering systems facing noisy retrieved documents.
- SearchQA narrows this gap by building a new closed-world question-answering dataset.
- The dataset starts with Jeopardy! question-answer pairs, augments them with Google snippets, and filters examples lacking answers in snippets or having fewer than forty returned web pages.
- 140k+ question-answer pairs and 6.9M snippets comprise the resulting dataset.
- Human and machine evaluations find SearchQA challenging, with a significant performance gap between them.
2 SearchQA
SearchQA uses Google to add realistic retrieval noise to Jeopardy!-based question-answer pairs, then cleans and releases the resulting dataset with metadata and predefined splits. The final collection contains 140,461 pairs paired with an average of 49.6 snippets.
- Collection: Google supplies realistic relevant and irrelevant snippets to make the context resemble a noisy information retrieval system.
- Cleaning: Cleaning removes snippets containing episode air-dates, exact questions, or terms such as “Jeopardy!”, “quiz” or “trivia”.
- Cleaning: Only tuples whose contexts include the answer and whose answers are three or fewer words are retained.
- Basic Statistics: 140,461 question-answer pairs remain after collection and cleaning.
- Basic Statistics: 49.6±2.10 snippets accompany each question-answer pair on average.
- Meta-Data: The public release includes Jeopardy! metadata and Google-provided URLs, titles, and related links for snippets.
- Training, Validation and Test Sets: Training, validation, and test sets contain 99,820, 13,393, and 27,248 examples, respectively, with validation and test questions drawn from later years.
3 Related Work
SearchQA is a closed-world dataset whose contexts are retrieved snippets, but whose questions originate independently from Jeopardy! question-answer pairs rather than from selected articles. This distinguishes it from prior open-world, cloze-style, and context-first datasets.
- Open-World Question-Answering: Open-world question-answering pairs questions with a knowledge database without linking each pair to a specific database entry.
- Closed-World Question-Answering: SearchQA is closed-world because each question can be answered entirely from its associated snippets.
- Closed-World Question-Answering: Unlike context-first datasets such as SQuAD and NEWSQA, SearchQA starts from question-answer pairs and does not require fill-in-a-word questions.
- Closed-World Question-Answering: SQuAD and NEWSQA often use multi-word answers that need not appear exactly in context, whereas SearchQA ensures multi-word answers appear in their corresponding context.
- MS MARCO is described as most similar to SearchQA, but differs in its question sources and search engine.
4 Experiments and Results
SearchQA is evaluated with human volunteers and machine baselines on noisy retrieved snippets, revealing substantial difficulty and a persistent human–machine gap.
- Evaluation setup: The evaluation compares human volunteers with TF-IDF Max and attention sum reader baselines on SearchQA.TF-IDF Max selects a unigram answer by highest context TF-IDF, while ASR provides trainable attention-based baselines.
- Human Evaluation: 13 volunteers answered randomly selected test questions through an interface that displayed the query and retrieved snippets.Participants had up to 40 minutes and answered 47.23 questions on average, with standard deviation 30.42.
- Human Evaluation: Human accuracy decreases as answer length increases, with a significant gap between unigram-only and mixed-length answer groups.The authors also report that snippets are often noisy excerpts rather than full sentences and that volunteer exhaustion may have affected performance.
- Machine Baselines: The vanilla ASR scores answer words by matching question and context representations, then aggregates scores for repeated words and trains on the correct answer.The n-gram ASR adds previous answer-word representations and predicts multi-word answers sequentially until an end token.
- Machine Baselines: The attention sum reader remains below human evaluation, while TF-IDF Max is not on par with ASR.Table 2 reports top-1/5 accuracies for unigram answers and F1 scores for other answers.
5 Conclusion
The paper concludes that SearchQA is a publicly released dataset built with a commercial search engine to reflect retrieval and answer synthesis, with a meaningful human–machine performance gap.
- 5 Conclusion: SearchQA is a publicly released question-answering dataset built with an in-production commercial search engine.The release includes implementations of ASR and n-gram ASR in PyTorch.
- 5 Conclusion: SearchQA reflects a hypothetical general question-answering pipeline consisting of information retrieval and answer synthesis.Human and machine evaluations indicate a meaningful gap between their performances, supporting SearchQA’s potential as a benchmark.