Source-linked AI summary
Quasar: Datasets for Question Answering by Search and Reading
Bhuwan Dhingra, Kathryn Mazaitis, William W. Cohen
TL;DR
Factoid QA over large text corpora requires both finding relevant passages and reading them, but prior datasets often isolate passage understanding. Quasar constructs two controlled datasets for these combined tasks and evaluates retrieval and reading baselines. Best baselines remain below human performance on both datasets, while the paper highlights a trade-off between search coverage and reading difficulty.
Problem
Factoid QA systems must search unstructured corpora for answer-bearing passages and then read those passages, whereas prior reading-comprehension datasets assume the relevant passage is known.
Method
The paper constructs QUASAR-S from Stack Overflow software-entity definitions and QUASAR-T from online trivia, pairing each with a restricted background corpus and evaluating search-and-reading baselines.
Results
Best baselines reach 33.6% accuracy on QUASAR-S and 28.5% F1 on QUASAR-T, while human performance is 50% and 60.6%, respectively.
Takeaways & Limitations
Quasar supports research on jointly optimizing retrieval and reading because retrieving more context improves answer coverage but typically makes comprehension harder.
Takeaways & Limitations
QUASAR-S excludes cloze questions for terms common in English, so a more sophisticated entity-recognition system could improve recall.
Abstract
from arXiv · showhide
We present two new large-scale datasets aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. The Quasar-S dataset consists of 37000 cloze-style (fill-in-the-gap) queries constructed from definitions of software entity tags on the popular website Stack Overflow. The posts and comments on the website serve as the background corpus for answering the cloze questions. The Quasar-T dataset consists of 43000 open-domain trivia questions and their answers obtained from various internet sources. ClueWeb09 serves as the background corpus for extracting these answers. We pose these datasets as a challenge for two related subtasks of factoid Question Answering: (1) searching for relevant pieces of text that include the correct answer to a query, and (2) reading the retrieved text to answer the query. We also describe a retrieval system for extracting relevant sentences and documents from the corpus given a query, and include these in the release for researchers wishing to only focus on (2). We evaluate several baselines on both datasets, ranging from simple heuristics to powerful neural models, and show that these lag behind human performance by 16.4% and 32.1% for Quasar-S and -T respectively. The datasets are available at https://github.com/bdhingra/quasar .
1 Introduction
Quasar introduces datasets that evaluate both retrieval and reading for factoid question answering over large text corpora. The datasets cover closed-domain software questions and open-domain trivia, while baseline systems remain below human performance.
- Motivation: Factoid QA over unstructured sources requires searching for relevant passages and reading them to extract an answer.Prior reading-comprehension tasks generally assume that the relevant passage is already known.
- Contribution: Quasar introduces two datasets for Question Answering by Search And Reading, each pairing factoid questions with a large background corpus.The datasets are designed to study retrieval and comprehension together.
- Datasets: QUASAR-S contains 37,362 cloze questions about software entities, while QUASAR-T contains 43,013 trivia questions with free-form answers.QUASAR-S uses Stack Overflow-derived software definitions; QUASAR-T questions come from various internet sources.
- Datasets: QUASAR-S uses Stack Overflow threads and QUASAR-T uses ClueWeb09 as restricted background corpora for answer retrieval.The corpus restriction prevents directly extracting answers from the sources used to construct the questions.
- Motivation: The datasets extend beyond passage understanding by requiring systems to find answer-containing text in large corpora.This design also targets candidate aggregation and conflict resolution across passages.
- Results: Best baselines achieve 33.6% on QUASAR-S and 28.5% on QUASAR-T, compared with human performance of 50% and 60.6%.The evaluation spans naïve heuristics through state-of-the-art machine readers.
2 Existing Datasets
Existing datasets separately emphasize open-domain answer selection or reading comprehension with the answer passage known in advance. Quasar is positioned against these limitations through controlled retrieval resources and trivia questions from online sources.
- Open-Domain QA: Early TREC-QA datasets were manually constructed and evaluated, limiting their scale to a few hundred questions.WIKIQA increased scale for answer-sentence selection but did not identify answer spans within selected sentences.
- Reading Comprehension: Reading-comprehension datasets measure understanding of a given passage while assuming the passage containing the answer is known beforehand.CNN/Daily Mail, CBT, WDW, SQuAD, and NewsQA represent this passage-focused line of work.
- Open-Domain QA: TriviaQA resembles QUASAR-T but uses commercial-search-engine documents, making retrieval variation less controlled for researchers.Quasar instead uses ClueWeb09 and supplies a larger collection of retrieved passages.
3 Dataset Construction
QUASAR constructs software cloze and trivia datasets from different sources, then builds ranked context documents and candidate answers to support separate retrieval and reading evaluation.
- Question-set construction: QUASAR-S uses Stack Overflow tag excerpts to create cloze questions by replacing each mention of another software entity with a placeholder.The preprocessing preserves software-specific tokens and prepends the head tag when the excerpt does not contain it.
- Question-set construction: A corpus-versus-Stack-Overflow incidence score filters noisy QUASAR-S clozes, excluding terms common in English and potentially reducing recall.The authors identify overloaded terms such as “can,” “swap,” and “using” as sources of noise.
- Question-set construction: QUASAR-T starts from nearly 54,000 scraped trivia questions, filters unparseable and non-free-response items, and retains 52,000 questions before later processing.The source data varied in formatting, spelling, and accuracy.
- Context retrieval: Context documents are generated by collecting a large text pool, indexing short or long pseudodocuments, and selecting N top-ranked items.For QUASAR-S, the pool contains text from at least 50 top-voted Stack Overflow threads per tag; retrieval uses phrase, token, and head-tag constraints.
- Context retrieval: QUASAR-T retrieves 100 ClueWeb09 HTML documents per question, while each record’s candidate list is guaranteed to contain the correct answer.QUASAR-S uses a fixed vocabulary of 4874 tags; QUASAR-T derives candidates from noun-phrase sequences in retrieved context and evaluates both full and answer-present subsets.
- Postprocessing: The answer-present subset isolates reading performance after two-phase search, whereas the full set evaluates QUASAR as a combined retrieval-and-reading task.The final training, validation, and test sizes are reported in Table 1.
4 Evaluation
QUASAR evaluates retrieval and reading separately and jointly using heuristic, language-model, and neural baselines, with performance measured against human testers. Results show a tension between finding answers and extracting them as context size changes.
- Evaluation: QUASAR-T evaluation uses F1 because answers are free-form spans that may have different expressions, while QUASAR-S reports average entity-prediction accuracy.QUASAR-S answers come from a fixed entity vocabulary; QUASAR-T answers require normalized span matching.
- Human Evaluation: Human evaluation recruited domain experts and non-experts, with annotations covering question ambiguity and dataset-specific categories.QUASAR-S annotations captured relations between head and answer entities; QUASAR-T annotations captured question genres and answer types.
- Results: Search accuracy increases with more context documents, but reading accuracy decreases because longer contexts make answer extraction harder.Overall accuracy therefore depends on balancing retrieval coverage against reading difficulty.
- Results: 33.6% accuracy makes the BiRNN language model the best QUASAR-S baseline.GA reaches 48.3% when the answer is present, but its 65% search accuracy lowers overall performance.
- Results: 28.5% F1 makes BiDAF the strongest QUASAR-T baseline, with neural models significantly outperforming heuristic models.BiDAF predicts multi-token answer spans, while GA extracts a single token.
- Results: 16.4% and 32.1% are the QUASAR-S and QUASAR-T gaps between the best baselines and human performance.The authors report human performance of 50% and 60.6%, respectively, and caution that these are not strict upper bounds for automatic systems.
5 Conclusion
QUASAR introduces datasets and baselines for jointly studying corpus search and answer extraction. Its results motivate improving retrieval and reading together because more retrieved context can help search while hurting reading.
- Conclusion: QUASAR supports research on searching large text corpora for relevant passages and reading those passages to extract answers.The release includes datasets, retrieved documents, and human annotations.
- Conclusion: Retrieving more context improves search performance but typically reduces reading performance.The authors therefore identify joint optimization of both components on end-task performance as a future direction.
- Conclusion: QUASAR baselines perform reasonably but lag behind human performance.This conclusion applies across the two related QA tasks evaluated in the paper.
A QUASAR-S Relation Definitions
This passage points to a table containing the definitions of all annotated QUASAR-S relations.
- A QUASAR-S Relation Definitions: Table 4 defines all relations annotated for QUASAR-S.The passage introduces the table but does not list the individual relation definitions.
B Performance Analysis
Figure 5 compares human performance with the best baseline across annotated question categories, revealing category-specific strengths and weaknesses for both datasets.
- Figure 5 compares human performance with the best-performing baseline across annotated question categories.The comparison covers both QUASAR-S and QUASAR-T categories.
- QUASAR-S: For QUASAR-S, Bi-RNN matches humans on developedwith and runs-on but underperforms on has-component and is-a.
- QUASAR-T: For QUASAR-T, BiDAF matches humans in sports but performs worse in history & religion, language, and numeric or date/time answers.