PaperlayerView exact revision on arXiv

Source-linked AI summary

QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension

Anna Rogers, Matt Gardner, Isabelle Augenstein

arXiv:2107.12708v2cs.CLcs.AI

TL;DR

QA research still faces open questions about the full range of human interactions with search engines. This paper surveys the field’s dataset dimensions, categorizes over two hundred datasets, and identifies gaps while proposing a taxonomy of reasoning skills.

  • Problem

    Studying the full range of human interactions with search engines remains an open problem at the boundary of question answering.

  • Method

    The paper surveys dataset formats, domains, and target reasoning skills, categorizing over two hundred datasets and proposing a taxonomy.

  • Results

    The survey identifies key dimensions along which current datasets vary and highlights gaps in the available resources.

  • Takeaways & Limitations

    QA for news is societally important amid increasing online misinformation, while world modeling targets spatiotemporal and social representations.

  • Takeaways & Limitations

    Evaluating generated language remains a major research problem, and freeform-format metrics have limitations.

Abstract

from arXiv · show

Alongside huge volumes of research on deep learning models in NLP in the recent years, there has been also much work on benchmark datasets needed to track modeling progress. Question answering and reading comprehension have been particularly prolific in this regard, with over 80 new datasets appearing in the past two years. This study is the largest survey of the field to date. We provide an overview of the various formats and domains of the current resources, highlighting the current lacunae for future work. We further discuss the current classifications of "skills" that question answering/reading comprehension systems are supposed to acquire, and propose a new taxonomy. The supplementary materials survey the current multilingual resources and monolingual resources for languages other than English, and we discuss the implications of over-focusing on English. The study is aimed at both practitioners looking for pointers to the wealth of existing data, and at researchers working on new resources.

1 INTRODUCTION: THE DATASET EXPLOSION

The rapid growth of QA/RC datasets has produced a diverse resource landscape that resists a single taxonomy. This survey systematically organizes these resources across formats, domains, languages, and targeted skills while identifying cross-field connections and research gaps.

  • Dataset explosion: NLP dataset development expanded rapidly, driven by crowdsourcing, divergent efforts toward natural language understanding, and the practical utility of question answering.These forces produced a breadth-first research process with limited coordination between groups.
  • Dataset explosion: QA/RC resources form a diverse potpourri that cannot be characterized by one defining feature, including datasets based on statements, questions, or both.The survey therefore treats format diversity as a central organizational problem.
  • Survey scope: The survey systematically reviews existing resources using criteria spanning question, answer, evidence, conversational, domain, language, and skill dimensions.It uses representative English resources while supplementary materials cover multilingual and non-English monolingual resources.
  • Skills taxonomy: The proposed skills taxonomy covers inference, information retrieval, world modeling, input interpretation, and multi-step reasoning.The taxonomy is presented alongside an overview of current benchmark skill classifications.
  • Cross-field connections: The criteria place QA/RC in the broader context of machine reasoning and linguistic NLP features, exposing connections to multimodal NLP and commonsense reasoning.They also map gaps for future work and support connections to approaches such as inference and entailment.
  • Scope boundary: The survey focuses exclusively on resource typology because the volume of QA/RC data makes joint coverage of datasets and modeling work impractical.Readers seeking modeling approaches are directed to existing surveys and tutorials.

2 INFORMATION-SEEKING VS PROBING QUESTIONS

QA/RC datasets distinguish information-seeking questions from probing questions, although the boundary is blurry and the two classes have different collection settings and reasoning profiles. Information-seeking data reflects real-world use but can be ambiguous and less reasoning-intensive, while probing data is crafted to test specific understanding.

  • Core distinction: Information-seeking questions arise from users seeking unknown information, whereas probing questions are written to test understanding of a supplied context.QA is more often associated with information-seeking and RC with probing, though the distinction is not absolute.
  • Collection settings: Information-seeking questions commonly come from product users and real-world sources, while probing questions are typically written by exam authors or crowd workers.Information-seeking resources often assume no given context and are rarely multiple choice; probing questions target contexts such as paragraphs or images.
  • Benchmark trade-offs: Probing datasets may be unsuitable as natural question distributions because large training sets can expose subtle spurious patterns.The survey notes that a distribution designed to test reading comprehension is not necessarily ideal for testing models.
  • Benchmark trade-offs: Real-world information-seeking data can be naturalistic but problematic as a benchmark because ambiguity and presupposition are common.Nearly half of Natural Questions are estimated to be ambiguous, motivating resources that specifically target ambiguity.
  • Reasoning profiles: Information-seeking questions tend to involve less complex reasoning than probing datasets because search users less often ask complex questions.Probing datasets can deliberately target particular parts of the long tail of question types.
  • Blurry boundary: The information-seeking and probing categories frequently overlap because identical questions can serve either information retrieval or evaluation purposes.Providing likely answer-containing documents can make information-seeking datasets resemble probing datasets, while some resources intentionally combine both elements.

3 TASK VERSUS FORMAT

Question answering can function either as a task with communicative intent or as a general output format for posing NLP problems. The distinction is useful but fuzzy, because datasets labeled QA may differ in their underlying problems and pattern diversity.

  • Almost any NLP task can be reformulated as question answering, making QA a format rather than necessarily a distinct task.Examples include machine translation and sentiment analysis recast as questions.
  • The key distinction is whether the questions contain task-relevant information or could be replaced with content-free identifiers.Sentiment questions can correspond to a few labels without requiring understanding of the question wording.
  • The task-format boundary remains fuzzy, and some QA or RC datasets can be generated from templates.The heuristic is explicitly described as non-strict, while templated resources have still enabled progress.
  • Low diversity in training patterns leads to models’ over-reliance on those patterns.The survey connects this concern to findings reported across multiple studies.
  • QA formats such as extractive answers do not determine reasoning types, because datasets using the same output format can encapsulate different problems.Extractive QA is described as an output format, not a cohesive reasoning problem.

4 FORMAT

The survey organizes QA and RC resources by the formats used for questions, answers, and evidence. This provides a framework for describing existing datasets across these dimensions.

  • Existing datasets are described along the dimensions of question, answer, and evidence formats.Questions are the inputs, answers are system outputs, and evidence is the source used to derive those outputs.

4.1 Question format

Question formats range from natural-language questions and queries to cloze statements and story-completion prompts. These formats differ in linguistic form, construction process, and the information they target.

  • 4.1.1 Natural language questions.: Natural-language questions may serve information-seeking or probing purposes and can be yes/no, wh-, tag, or declarative questions.These question forms may be paired with any answer format.
  • 4.1.2 Queries.: Queries are not necessarily well-formed questions but contain information that can be interpreted as a question.For example, “tallest mountain Scotland” can be interpreted as asking which mountain is tallest in Scotland.
  • 4.1.2 Queries.: Logical queries for tables and knowledge bases may be converted into templated natural-language questions, while search-engine queries are often messier.Current datasets using natural questions may filter out such search queries.
  • 4.1.2 Queries.: Studying the full range of human interactions with search engines remains an open problem at the boundary of question answering and information retrieval.At least one resource annotates search-engine queries for well-formedness.
  • 4.1.3 Cloze format.: Cloze statements mask spans in sentences rather than presenting linguistically formed questions, and they can target words, entities, sentences, or other spans.The masked material may be drawn from the surrounding narrative or another related text.
  • 4.1.3 Cloze format.: Cloze datasets can be created programmatically, enabling quick and inexpensive collection, but answerability and question quality may require additional filtering.Cloze prompts can test entities, common nouns, and verbs, and have resurged for evaluating masked language models.
  • 4.1.3 Cloze format.: Cloze questions do not directly target the QA task and restrict evaluation to relations within a narrow context.The masked answer may reflect a collocation, a generally known fact, or a relation unique to that context.
  • 4.1.4 Story completion.: Story-completion resources present passage endings or unfinished text for completion, often with multiple-choice answers.Unlike many automatically generated cloze resources, choice-ending datasets tend to be crowdsourced for this purpose.

4.2 Answer format

QA and reading-comprehension resources use four answer formats—extractive, multi-choice, categorical, and freeform—each balancing evaluation convenience against answer expressiveness and validity.

  • 4.2 Answer format: The survey categorizes text-based dataset outputs as extractive, multi-choice, categorical, or freeform.
  • 4.2.1 Extractive format.: Extractive QA predicts an answer span in the evidence, simplifying correctness but restricting questions to answers directly contained in text.Unanswerable questions can support document assessment before span prediction.
  • 4.2.2 Multi-choice format.: Multi-choice formats offer straightforward evaluation and controlled reasoning demands, but weak distractors can make questions nondiscriminative.Humans and models may also find different options difficult because models can exploit shallow lexical or collocational cues.
  • 4.2.3 Categorical format.: Categorical formats classify answers from a predefined set, enabling answers beyond explicit text while making unseen test answers impossible.Boolean questions are a prominent example, including BoolQ.
  • 4.2.4 Freeform format.: Freeform formats let systems generate answers, but evaluation is difficult because multiple answers may be correct and lexical metrics penalize abstraction.Existing metrics can limit question types and perform poorly for abstractive answers; representation-similarity metrics are proposed as an alternative.
  • 4.2.4 Freeform format.: Cross-format training often outperforms training solely in the target format across boolean and multi-choice questions from 20 datasets.

4.3 Evidence format

QA and reading-comprehension resources vary by evidence modality and by how much source material systems receive, spanning text, structured data, images, audio, video, and mixed inputs.

  • 4.3 Evidence format: Evidence is whatever a system must understand or use to derive an answer, and resources vary by modality, amount, and static versus conversational presentation.
  • 4.3.1 Modality.: Multimodal QA combines modalities such as audio, images, and video, while structured-text resources incorporate document layout.Tables and knowledge bases provide additional semi-structured or structured evidence settings.
  • 4.3.1 Modality.: Structured-knowledge QA requires interpreting a question as a query over a knowledge base, which may be general or domain-specific.
  • 4.3.1 Modality.: Visual and video QA cover increasingly complex inputs, but questions are often generated or probing rather than naturally information-seeking.The field lacks a settled account of what natural video QA should be.
  • 4.3.1 Modality.: Audio QA remains data-scarce, and pipelines commonly reduce it to text after speech recognition, losing prosodic cues useful for disambiguation.Public purely audio-based QA data is rare partly because releasing customer data is ethically challenging and conflicts with business interests.
  • 4.3.1 Modality.: Multimodal datasets must demonstrate that every input modality is necessary and actually used to answer the question.Otherwise systems may rely on linguistic shortcuts or salient visual objects while ignoring other inputs.
  • 4.3.2 Amount of evidence.: The amount of evidence ranges from a single predefined document to multiple, partial, or absent sources requiring retrieval.This dimension is described as a continuum rather than a strict taxonomy.

5 CONVERSATIONAL FEATURES

Conversational QA differs from static reading comprehension because dialogue history grows dynamically, affects interpretation, and may involve unequal participant knowledge or transactional goals.

  • Dialogue history: Follow-up questions depend on prior dialogue, making question order non-interchangeable and sometimes requiring conversational context for interpretation.Rewriting datasets such as CANARD and QReCC target more search-friendly versions of contextual questions.
  • Origins and settings: Conversational datasets are not necessarily spoken: CoQA, for example, was created in written form by crowd workers using prompts.
  • Information-seeking behavior: A central dataset-design challenge is ensuring questions are genuinely information-seeking rather than probing.The distinction matters because probing questions are not normally used between people outside special contexts.
  • Knowledge arrangements: Conversational QA can model equal knowledge, unequal knowledge, or dialogue-like data repurposed from existing conversational corpora.CoQA exemplifies equal knowledge, QuAC unequal knowledge, and Molweni repurposed dialogue.
  • Related dialogue tasks: Dialogue comprehension datasets test understanding of conversational scripts, whereas conversational QA uses dialogue to obtain answers from another context.
  • Task-oriented dialogue: Task-oriented dialogue extends QA with transactional operations, such as finding available flights and booking one.Resources cover domains including travel, meetings, navigation, scheduling, weather, and personal assistants.
  • Evaluation: Real information-seeking dialogue includes conversational moves beyond questions and answers, creating evaluation signals beyond correctness.Examples include informing, rejecting, promising, emotion, success, and effort.

6 DOMAINS

The survey organizes QA/RC resources primarily by source genre, while noting that “domain” is used inconsistently and that several areas remain underrepresented.

  • Classification: “Domain” is used variously for topic, genre, linguistic register, or evidence-retrieval format, limiting consistent comparisons across resources.The survey’s classification primarily treats domain as genre.
  • Fiction: Fiction remains relatively underused despite abundant public-domain material, with long-text understanding identified as a central challenge.Resources include CBT, BookTest, FairyTaleQA, and NarrativeQA, but Cloze formats retain known limitations.
  • News: News QA is societally important because of online misinformation, but public-domain data is scarce.Existing resources include CNN/Daily Mail, NewsQA, NLQuAD, QuAIL, and temporal news datasets.
  • E-commerce: E-commerce QA extends product-review resources with answerability and subjectivity annotations, notably in SubjQA.SubjQA is described as the first QA dataset to label answer subjectivity.
  • Expert materials: Expert-material resources cover narrow, specialized topics through questions answered or written by experts, including technology, NLP papers, and biomedicine.Community QA differs because volunteers provide answers with varying expertise.

7 LANGUAGES

QA/RC resources remain heavily concentrated in English, while multilingual datasets expand coverage but cannot replace expert work on individual languages.

  • Distribution: English is the default language of QA/RC research, and many languages lack substantial monolingual QA/RC data.Chinese is the second best-resourced language, while German is cited as lacking even monolingual resources.
  • Monolingual resources: Machine-translated English resources provide coverage for some languages but introduce translation artifacts and questions unlike those naturally asked by native speakers.Examples include translated SQuAD resources for Spanish, Arabic, Italian, and Korean.
  • Implications: English concentration can misrepresent QA/RC progress because string matching is easier in morphologically poor English than in languages with richer alternations and compounding.The survey links this concern to observations from TydiQA authors.
  • Multilingual resources: Multilingual resources include XQA, TydiQA, XOR QA, MLQA, XQuAD, and MKQA across diverse languages and tasks.MKQA uses professional translations into 26 languages, while TydiQA covers ten typologically diverse languages besides English.
  • Trade-offs: Large multilingual benchmarks can disadvantage small labs and necessarily miss nuance that requires in-depth, language-specific expert work.The survey notes that most large multilingual datasets were produced with major industry affiliations.

8 QA/RC “SKILLS”

The survey proposes an orthogonal taxonomy that separates QA/RC skills into complementary dimensions rather than treating each question as belonging to one reasoning category.

  • Existing taxonomies: Existing QA/RC skill taxonomies differ substantially, while datasets’ exploratory categories are too inconsistent for direct comparison.The survey identifies disagreement over intended capabilities and notes that fine-grained annotation is expensive.
  • Proposed taxonomy: The proposed taxonomy groups skills into inference, retrieval, input interpretation and manipulation, world modeling, and multi-step reasoning.These dimensions cover reasoning, information location, linguistic and numerical operations, world representation, and chains of actions.
  • World modeling and multi-step: The taxonomy distinguishes world modeling as representing spatiotemporal and social aspects of the text, and multi-step reasoning as chains of actions across dimensions.World modeling also positions the text relative to the reader and other texts.
  • Orthogonality: The dimensions are orthogonal, so one question can be characterized simultaneously by linguistic form, inference, retrievability, compositional complexity, and world modeling.Different dimensions may be more prominent or challenging for a given question.
  • Cross-dimensional analysis: Question analysis should allow multiple linguistic categories because linguistic phenomena can correlate with reasoning operations without being identical to them.The taxonomy treats questions as potentially combining skills from different dimensions.

9 DISCUSSION

The discussion argues that benchmark performance does not reliably reveal the skills models use, motivating diagnostic evaluation, better-controlled data, and faithful interpretability.

  • Required skills: Whether a reasoning skill is required depends on the question together with the entire dataset, not on the question alone.Training and testing can make dataset-level patterns part of the effective task.
  • Shortcuts: Models may exploit superficial training-data patterns, including answer-position regularities, even in few-shot or in-context settings.Such shortcuts can produce correct answers without the intended reasoning process.
  • Evaluation directions: Diagnostic tests, generalization tests, controlled training signals, and interpretability methods are proposed to assess what models actually do.The proposed directions include adversarial and minimal-pair tests, out-of-domain and cross-lingual evaluation, and explanation methods.
  • Interpretability: Faithful explanations remain difficult because explanation quality and human evaluation vary with model quality and prior belief bias.The survey treats faithfulness itself as an active research problem.
  • Annotation: Cross-dataset skill annotation remains limited because expert annotation is expensive and both experts and crowdworkers struggle to identify information needed for answers.Without mapping questions to reasoning steps and target information, system behavior cannot be assessed against intended capabilities.
  • Research gaps: Logical inference has received the least attention among the proposed taxonomy’s dimensions, despite its abstraction and specialist-knowledge demands.The survey identifies the strength of support for a conclusion as useful for reasoning under imperfect information.
  • Future datasets: Future datasets should document their limitations rather than implying that they have none.The survey frames documented limitations as useful for future evaluation and potential model certification.

10 CONCLUSION

The survey identifies key dimensions for describing QA/RC datasets and applies them to over two hundred resources, while highlighting gaps in the literature. It is intended to support both practitioners seeking data and researchers developing QA/RC resources.

  • The survey identifies key dimensions along which current QA/RC datasets vary, providing a framework for evaluating current and future resources.
  • Over two hundred datasets are categorized across format, domain, and target reasoning skills, with gaps in current literature highlighted.
  • The survey is designed to help NLP practitioners find data and researchers seeking to advance QA/RC research.
  • Quantitative analysis of the available resource types and how they are blended was outside the survey’s scope, but the proposed dimensions support such future analysis.
  • The dimensions could support diachronic meta-research on how QA/RC resources evolve over time.
Loading 2107.12708v2…