PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets

Changchang Zeng, Shaobo Li, Qin Li, Jie Hu, Jianjun Hu

arXiv:2006.11880v2cs.CLcs.AIcs.LG

TL;DR

MRC models can exceed human performance on benchmarks while remaining far from genuine human-level comprehension, and existing surveys and task classifications leave important coverage gaps. This paper surveys MRC tasks, metrics, datasets, and open issues, proposing a four-attribute task classification. It describes 47 datasets and concludes that multimodal datasets and cognitive neuroscience findings may help guide future MRC research.

  • Problem

    Existing task classifications can be ambiguous, and reviews provide limited coverage of datasets and newer multimodal MRC tasks.

    Source: §2.3.2 · pp. 9–10, §2.5.1 · p. 20

  • Method

    The paper surveys MRC tasks, evaluation metrics, and datasets, classifying 57 tasks through four attributes and analyzing metric use and dataset characteristics.

    Source: §2.3.3 · p. 11, §6 · p. 61

  • Results

    The survey describes 47 MRC datasets and concludes that high-quality multimodal datasets and cognitive neuroscience findings may help guide future MRC research.

    Source: §6 · p. 61

  • Takeaways & Limitations

    The authors argue that multimodal datasets and cognitive neuroscience findings may help construct more challenging datasets and develop MRC algorithms.

    Source: §6 · p. 61

    Because content-based task classification is subjective without established standards, the survey focuses mainly on task form.

    Source: §2.5.1 · p. 20

Abstract

from arXiv · show

Machine Reading Comprehension (MRC) is a challenging Natural Language Processing(NLP) research field with wide real-world applications. The great progress of this field in recent years is mainly due to the emergence of large-scale datasets and deep learning. At present, a lot of MRC models have already surpassed human performance on various benchmark datasets despite the obvious giant gap between existing MRC models and genuine human-level reading comprehension. This shows the need for improving existing datasets, evaluation metrics, and models to move current MRC models toward "real" understanding. To address the current lack of comprehensive survey of existing MRC tasks, evaluation metrics, and datasets, herein, (1) we analyze 57 MRC tasks and datasets and propose a more precise classification method of MRC tasks with 4 different attributes; (2) we summarized 9 evaluation metrics of MRC tasks, 7 attributes and 10 characteristics of MRC datasets; (3) We also discuss key open issues in MRC research and highlighted future research directions. In addition, we have collected, organized, and published our data on the companion website(https://mrc-datasets.github.io/) where MRC researchers could directly access each MRC dataset, papers, baseline projects, and the leaderboard.

1. Introduction

MRC research has grown with large-scale datasets and deep learning, but benchmark success still coexists with a gap from genuine human comprehension. This survey addresses the need for a systematic account of MRC tasks, metrics, datasets, and open issues.

1.1. Overview

  • MRC asks machines to answer questions about passages, and the paper illustrates its use in search engines and dialogue systems.The authors note that these applications can help users get answers quickly or reduce customer-service workload.Source: §1.1 · p. 2, Figure 1 · p. 2

1.2. History

  • Large-scale supervised datasets and attention-based neural models helped renew MRC research after a long period of limited activity.The paper reports that MRC publication counts grew rapidly after 2015.Source: §1.2 · p. 2, Figure 2 · p. 3

1.3. Motivation

  • Existing reviews and task labels leave a need for comprehensive coverage because task categories can overlap and many datasets remain little known.The survey also notes that models surpass human performance on many datasets despite a remaining gap from real human comprehension.Source: §1.3 · p. 4

1.4. Outline

  • The survey proposes a four-attribute classification of 57 MRC tasks and analyzes nine evaluation metrics and their use across task types.It also reports dataset sizes, generation methods, corpus sources, and ten dataset characteristics.Source: §1.4 · pp. 4–5
  • The survey reviews open research issues, discusses cognitive neuroscience findings, and publishes dataset resources, papers, baselines, and leaderboards online.The companion website is intended to make this information readily accessible to MRC researchers.Source: §1.4 · p. 5

2. Tasks

This section surveys MRC task formulations, treating textual question answering as the typical supervised task of answering a question from a specified passage. It classifies 57 tasks by corpus, question, answer, and answer-source form, and summarizes their distribution, focusing on task form because content-based categorization lacks established standards.

2.2.1. Multi-modal MRC vs. Textual MRC

  • Multi-modal MRC requires jointly understanding text and visual information, making it a promising direction toward human-level reading comprehension.The survey notes that illustrations can support human understanding and cites evidence that children with stronger mental imagery skills outperformed peers on story comprehension.Source: §2.2.1 · p. 6

2.2.2. Machine Reading Comprehension vs. Question Answering

  • MRC commonly uses QA, but is not identical to QA: it requires answers grounded in specified text, while MRC also includes diverse task forms and QA includes tasks without that requirement.ShARC requires background knowledge and follow-up questions before answering; RecipeQA [35] includes ordering recipe images using temporal relations between steps.Source: §2.2.2 · pp. 6–7

2.2.3. Machine Reading Comprehension vs. Other NLP tasks

  • MRC methods can transfer to other NLP tasks, NLP methods and knowledge can aid MRC, and MRC can serve as a component in open-domain QA and dialogue pipelines.Examples include SAN applied to NLP benchmarks, translation-based non-English reading comprehension, syntax-constrained attention in SG-Net [47], and MRC use in open-domain QA [36] and dialogue [8] [49].Source: §2.2.3 · pp. 8–9

2.3.1. Existing Classification Methods of MRC tasks

  • The earlier task taxonomy distinguishes cloze-style, multiple-choice, span-prediction, and free-form-answer tasks by how answers are represented or selected.Cloze tasks fill placeholders, multiple-choice tasks select from candidates, span prediction selects answer boundaries in context, and free-form answers are not restricted to a word or passage span.Source: §2.3.1 · p. 9

2.3.2. Limitations of Existing Classification Method

  • Existing cloze, multiple-choice, span-prediction, and free-form categories can overlap: 29 of the 57 collected tasks fall into this fuzzy classification.The “Who did What” task can be cloze-style, multiple-choice, and span prediction; the earlier scheme also ignores multi-modal tasks such as MovieQA, COMICS, TQA, and RecipeQA.Source: §2.3.2 · pp. 9–10

2.3.3. A new Classification Method

  • The proposed classification describes MRC tasks by corpus, question, answer, and answer-source types, and the collected 57-task sample is 89.47% textual and 10.53% multi-modal.The four attributes distinguish textual or multi-modal corpora; natural-form, cloze-style, or synthetic-form questions; natural-form or multiple-choice answers; and span or free-form answer sources. The categories classify tasks precisely, though the authors note future tasks may require updates.Source: §2.3.3 · pp. 11–12

2.4.1. Type of Corpus

  • A corpus is multi-modal when it contains text and non-text information such as images, while textual MRC uses plain-text contexts, questions, and answers.Multi-modal MRC may place visual information in the context, question, or answer; unlike typical VQA, it focuses more on natural-language understanding and reading text in context.Source: §2.4.1 · p. 13

2.4.2. Type of Questions

  • MRC questions are classified as cloze-style, natural-form, or synthetic-form; a cloze question uses a placeholder that the system fills with a suitable word, phrase, or image.Cloze questions need not be interrogative sentences and may include visual information; the supplied examples include textual and multi-modal forms [35].Source: §2.4.2 · p. 14
  • Natural questions are grammatical complete sentences, often interrogative but sometimes imperative, whereas synthetic questions are word lists that need not follow grammar.Both natural and synthetic questions may include images; natural questions may also be textual or multimodal.Source: §2.4.2 · pp. 15–16

2.4.3. Type of Answers

  • Answers are either multiple-choice options, with one or more correct, or natural words, phrases, sentences, or images; surveyed multimodal datasets used only multiple-choice answers.The survey reports finding no natural-form answers in the multimodal MRC datasets it collected.Source: §2.4.3 · pp. 16–17

2.4.4. Source of Answers

  • Span answers come from the context, while free-form answers may be any text or image and need not come from the context.The survey does not provide multimodal span-answer examples, noting such tasks exist in computer vision, including semantic segmentation and object detection.Source: §2.4.4 · p. 17

2.5. Statistics of MRC Tasks

  • Across 57 tasks, 89.47% were textual and 10.53% multimodal; natural questions predominated, while free-form answers (70.18%) exceeded spans (29.82%).Natural answers accounted for 52.63% of tasks and multiple-choice answers 47.37%; cloze questions followed natural questions, then synthetic questions.Source: §2.5 · p. 18, Figure 16 · p. 18

2.5.1. Form of Task vs. Content of Task

  • Because content-based task types depend on subjective judgments without established standards, the survey analyzes task form rather than content.The passage notes that context and question content can also determine task type, using Facebook bAbI examples.Source: §2.5.1 · p. 20, Figure 17 · p. 21

3. Evaluation Metrics

The survey reviews nine MRC evaluation metrics, their calculation and task-specific use, then compares metric use across 57 tasks. Accuracy is most common, while metric preferences vary with task type.

3.2. Accuracy; 3.3. Exact Match

  • Accuracy measures the share of questions answered correctly; Exact Match requires an answer to match a correct answer exactly, and for span prediction it equals Accuracy.When a question has multiple correct answers, matching any one of them suffices.Source: §3.2 · p. 21, §3.3 · pp. 21–22

3.4.1. Token-level Precision; 3.4.2. Question-level Precision; 3.5.1. Token-level Recall; 3.5.2. Question-level Recall

  • Token-level precision and recall measure overlap between predicted and correct answer tokens, whereas question-level versions treat each answer as one entity.The survey describes token precision using bags of tokens while ignoring punctuation and articles; question-level measures instead count shared answers as whole answers.Source: §3.4.1 · p. 22, §3.4.2 · p. 23

3.6.1. Token-level F1; 3.6.2. Question-level F1

  • Token-level F1 combines token precision and recall, and task-level averaging takes each question’s maximum F1 across its accepted answers before averaging.Question-level F1 combines question-level precision and recall.Source: §3.6.1 · p. 24, §3.6.2 · p. 25

3.7. ROUGE; 3.8. BLEU

  • ROUGE-N measures n-gram recall against reference summaries, while BLEU combines modified n-gram precision with a brevity penalty.Both metrics, first used in other NLP evaluation settings, are also used for MRC.Source: §3.7 · p. 25, §3.8 · p. 25

3.9. Meteor

  • Meteor combines precision and recall, includes synonym matching, and applies a fragmentation penalty for differences and gaps in word order.Its parameters are tuned to maximize correlation with human judgments.Source: §3.9 · pp. 25–26

3.10. HEQ

  • HEQ measures how often a system’s token-level F1 reaches or exceeds human performance, addressing cases where F1 may mislead for multiple-valid-answer conversational tasks.The survey identifies QuAC [68] as an example of a dataset using HEQ.Source: §3.10 · p. 26

3.11. Statistics of Evaluation Metrics

  • Accuracy is used by 61.40% of the 57 tasks, followed by F1 at 36.84% and Exact Match at 22.81%.The survey also reports that multi-choice tasks tend to use Accuracy, while span and free-form tasks show no special preference among metrics.Source: §3.11 · pp. 28–29, Table 3 · p. 29, Figure 20 · p. 29
  • Multi-choice tasks tend to use Accuracy because a typical question has one correct candidate, while cloze tasks tend to use Accuracy because their answers are often multi-choice.MultiRC is an exception among the surveyed multi-choice tasks because it has multiple correct candidates; span and free-form tasks show no special metric preference.Source: §3.11 · p. 29, Figure 20 · p. 29

4. Benchmark Dataset

The survey presents a timeline of MRC datasets, arranging the listed benchmarks by year from 2013 through 2019.

4. Benchmark Dataset

  • The timeline groups MRC datasets by year, with the listed benchmarks spanning 2013–2019.Examples include MCTest in 2013–2014, SQuAD 1.1 in 2016–2017, and DREAM and DROP in 2019.Source: Figure 21 · p. 30, §4 · p. 30

4.1. The Size of Datasets

The survey compares MRC datasets by question-set size and split proportions, and also describes variation in corpus units. Dataset sizes range from hundreds to millions of questions.

4.1. The Size of Datasets

  • The survey counts total questions and training, development, and test splits, noting that newer MRC datasets generally have more than 10K questions.It also reports each dataset’s training-set share.Source: §4.1 · p. 30, Table 4 · p. 30
  • WikiReading is the largest dataset at 18.87M questions, while ProPara is the smallest at 488; BookTest has the highest training-set proportion at 99.86%, and ARC Challenge the lowest at 43.20%.The development set is generally slightly smaller than the test set.Source: §4.1 · p. 32, Figure 22 · p. 33
  • MRC datasets use varied corpus units, including paragraphs and documents.The survey also tabulates corpus size and split sizes for each dataset.Source: §4.1 · p. 33, Table 5 · p. 33

4.2. The Generation Method of Datasets

The survey groups dataset generation methods into crowdsourcing, expert creation, and automated generation from a corpus.

4.2. The Generation Method of Datasets

  • Crowdsourcing produces question–answer pairs through distributed workers, often recruited through platforms such as Amazon Mechanical Turk.The survey describes crowdsourcing as a distributed problem-solving and production model.Source: §4.2 · p. 34
  • Expert-generated datasets use question–answer pairs written by people with professional knowledge; ARC [74] includes 7,787 science questions for US elementary and middle schools.Source: §4.2 · p. 35
  • Automated generation creates question–answer pairs from a corpus, as in many cloze datasets.Source: §4.2 · p. 35

4.3. The Source of Corpus

The corpus source affects the readability and complexity of MRC datasets. The survey categorizes sources across varied domains, from exams and Wikipedia to recipes and government websites.

4.3. The Source of Corpus

  • Corpus source affects the readability and complexity of MRC datasets.The survey lists sources including exam texts, Wikipedia, news articles, scientific-paper abstracts, textbooks, recipes, and government websites.Source: §4.3 · p. 35

4.4. The Type of Context

Context type can affect how MRC models are trained, motivating specialized approaches such as multi-hop and multi-document reading comprehension. The datasets span contexts from paragraphs to documents and image-containing passages.

4.4. The Type of Context

  • Context type can affect model training, prompting specialized multi-hop and multi-document reading-comprehension models.The survey identifies both multi-hop and multi-document reading comprehension as examples.Source: §4.4 · p. 35
  • MRC datasets use paragraph, multi-paragraph, document, multi-document, URL, and image-containing paragraph contexts.Table 6 organizes datasets by generation method, corpus source, and context type.Source: §4.4 · p. 35, Table 6 · p. 35
  • Examples include comics and recipes presented as paragraphs with images, and NarrativeQA using document-length movie contexts.Other listed datasets use paragraph or multi-paragraph contexts, including HotpotQA and MCScript.Source: §4.4 · p. 36

4.5. The Availability of Datasets, Leaderboards and Baselines

The survey collects dataset download links, leaderboards, and baseline projects to support researchers evaluating MRC models. It reports that download links are available for all surveyed datasets except PaperQA, while only a minority lack both leaderboards and baselines.

4.5. The Availability of Datasets, Leaderboards and Baselines

  • Only 19.3% of datasets lack both leaderboards and baseline projects.The survey reports that most datasets provide these evaluation resources and publishes links to them on its website.Source: §4.5 · p. 37
  • Download links are available for every surveyed MRC dataset except PaperQA.The survey says it collected and published dataset download links, leaderboards, and baseline projects on its website.Source: §4.5 · p. 37
  • Table 7 catalogs dataset availability alongside leaderboard and baseline availability.Its listed entries include datasets such as MultiRC, CoQA, and Natural Questions-L.Source: Table 7 · p. 37, §4.5 · p. 38

4.6. Statistical Analysis

The statistical analysis describes how MRC datasets are generated, sourced, and structured, and whether they offer leaderboards and baselines. Crowdsourcing and paragraph contexts are common, Wikipedia accounts for 19.30% of corpus sources, and 45.61% of datasets provide both leaderboards and baseline projects.

4.6. Statistical Analysis

  • 45.61% of datasets provide both leaderboards and baseline projects, while 19.3% provide neither.The analysis also reports that download links are available for all datasets except PaperQA.Source: §4.6 · p. 39, Figure 23 · p. 39
  • Crowdsourcing is the most common dataset-generation method, automated generation ranks second, and expert generation is least used because it is usually expensive.The survey associates automated generation with quickly creating large-scale MRC datasets and crowdsourcing with producing question-answer pairs requiring complex reasoning.Source: §4.6 · p. 39, Figure 23 · p. 39
  • Paragraphs are the most common context type, followed by documents, paragraphs with images, and multi-paragraph contexts.Source: §4.6 · p. 39, Figure 23 · p. 39
  • Wikipedia is the most common corpus source, but accounts for only 19.30% of sources.Source: §4.6 · p. 39, Figure 23 · p. 39

4.7. Prerequisite Skills

The survey examines prerequisite skills in MRC questions, including how often they appear in SQuAD 1.1 and MCTest. It cautions that skill definitions and classifications remain subjective and lack a standard mathematical definition.

4.7. Prerequisite Skills

  • Sugawara et al. describe 10 prerequisite skills, including enumeration, mathematical operations, coreference resolution, and logical reasoning, with frequencies reported for SQuAD 1.1 and MCTest.Table 8 covers MCTest’s MC160 development set.Source: §4.7 · p. 40, Table 8 · p. 40
  • Researchers manually annotated questions in MCTest and SQuAD 1.1 to determine the frequency of each prerequisite skill.Source: §4.7 · p. 40
  • Skill definitions and classifications are subjective and changeable, and the survey says they remain difficult to define mathematically in a standard way.The passage compares this difficulty with natural language understanding.Source: §4.7 · p. 40

4.8. Citation Analysis

The survey compares total and monthly citations for papers introducing MRC datasets as an indicator of dataset impact. SQuAD 1.1, CNN/Daily Mail, and SQuAD 2.0 lead the monthly-citation ranking, while monthly rates can differ from total counts.

4.8. Citation Analysis

  • SQuAD 1.1 [20] has the highest monthly average citations, followed by CNN/Daily Mail [53] and SQuAD 2.0 [43], which the survey identifies as widely used benchmarks.Source: §4.8 · p. 40, Table 9 · p. 41
  • The survey compares each dataset paper’s total citations with its average monthly citations, using Google Scholar except for the two PaperQA datasets.Source: §4.8 · p. 40
  • Monthly averages correlate overall with total citations, but SQuAD 2.0 [43] has high monthly attention despite its shorter publication history, while some older papers have lower monthly rates.Source: §4.8 · p. 42, Figure 24 · p. 43

4.9. Characteristics of Datasets

The survey organizes MRC datasets by characteristics that capture different demands on comprehension, from answerability and reasoning to modality, domain, and conversational context. It describes how these characteristics shape dataset scope and the skills required to answer questions.

4.9.1. Overview

  • Dataset characteristics span answerability, multi-hop and complex reasoning, modality, knowledge, scale, domain, paraphrase, open-domain, and conversational dimensions; DuoRC combines four.Source: §4.9.1 · pp. 43–44

4.9.2. MRC with Unanswerable Questions

  • Many datasets lack training examples for unanswerable questions, and models may still predict an answer when none is supported; SQuAD 2.0, MS MARCO, Natural Questions, and NewsQA include such questions.Source: §4.9.2 · p. 45

4.9.3. Multi-hop Reading Comprehension

  • Multi-hop MRC draws answers from multiple paragraphs or documents, requiring search and reasoning across potentially confusing material rather than a single passage.Source: §4.9.3 · p. 45

4.9.4. Multi-modal Reading Comprehension

  • Multimodal MRC combines text and images, addressing questions about information or precise concepts that text alone may not convey; TQA, RecipeQA, COMICS, and MovieQA are examples.Source: §4.9.4 · pp. 45–46

4.9.5. Reading Comprehension Require Commonsense or World knowledge; 4.9.6. Complex Reasoning MRC

  • Conventional MRC datasets include relatively few questions requiring commonsense, while other datasets target world knowledge and complex reasoning, including CommonSenseQA, DROP, RACE, and CLOTH.Source: §4.9.5 · p. 46, §4.9.6 · p. 46

4.9.7. Conversational Reading Comprehension

  • Conversational MRC treats prior dialogue as context for answering questions or asking meaningful follow-ups; CoQA, QuAC, DREAM, and ShARC are examples.Source: §4.9.7 · p. 46

4.9.8. Domain-specific Datasets

  • Domain-specific datasets use context from fields such as medicine or science, with examples including clinical cloze questions in CliCR and science exam questions in SciQ.Source: §4.9.8 · pp. 46–47

4.9.9. MRC with Paraphrased Paragraph; 4.9.10. Large-scale MRC Dataset; 4.9.11. MRC dataset for Open-Domain QA

4.10. Descriptions of each MRC dataset

This section surveys individual MRC datasets, distinguishing their data sources, answer formats, and targeted comprehension challenges. The datasets span extractive, reasoning-intensive, procedural, cloze, conversational, multimodal, and domain-specific tasks that test abilities such as reasoning across text, dialogue structure, and background knowledge.

4.10.2. SQuAD 2.0

  • 86% F1 on SQuAD 1.1 fell to 66% F1 on SQuAD 2.0 for the cited model, which adds more than 50,000 crowd-written unanswerable questions.Models must detect questions without correct answers; the passage reports a larger human–machine accuracy gap on SQuAD 2.0 than on SQuAD 1.1.Source: §4.10.2 · pp. 47–48

4.10.4. MS MARCO; 4.10.1. WikiQA; 4.10.3. Natural Questions; 4.10.26. Quasar; 4.10.27. SearchQA; 4.10.25. CuratedTREC

  • WikiQA, Natural Questions, and MS MARCO include unanswerable questions; MS MARCO also combines answering with passage ranking, while open-domain datasets test retrieval and reading.MS MARCO uses Bing query logs and retrieved web paragraphs; Quasar and SearchQA pair questions with retrieved text, and CuratedTREC evaluates open-domain question answering.Source: §4.10.1 · p. 47, §4.10.4 · p. 48, §4.10.3 · p. 48

4.10.10. ReCoRD; 4.10.8. MCScript; 4.10.11. CommonSenseQA; 4.10.19. NarrativeQA; 4.10.37. LAMBADA

  • ReCoRD, MCScript, CommonSenseQA, NarrativeQA, and LAMBADA emphasize commonsense, script, or broader narrative understanding rather than shallow pattern matching.ReCoRD queries and paragraphs are automatically mined from news articles; NarrativeQA requires understanding a whole book or movie script, while LAMBADA requires wider discourse context.Source: §4.10.10 · pp. 49–50, §4.10.8 · p. 49, §4.10.11 · p. 50

4.10.12. WikiReading; 4.10.22. CNN/Daily Mail; 4.10.23. BookTest; 4.10.6. Who-did-What; 4.10.24. MCTest; 4.10.35. Facebook CBT

  • Several datasets scale reading tasks across varied sources: WikiReading spans 18 million instances, while CNN/Daily Mail and BookTest use news articles and books, respectively.Who-did-What builds cloze questions from separate articles about the same events; MCTest uses fictional stories with questions designed for young children, and CBT uses a 21-sentence context for cloze prediction.Source: §4.10.6 · p. 49, §4.10.12 · p. 50, §4.10.22 · p. 52

4.10.15. COMICS; 4.10.14. MovieQA; 4.10.16. TQA; 4.10.17. RecipeQA

  • MovieQA, COMICS, TQA, and RecipeQA combine text with images or other media, requiring joint interpretation of visual and textual context.COMICS includes visual cloze and character-coherence tasks; RecipeQA also requires temporal-flow and procedural knowledge, and MovieQA draws on video clips, scripts, subtitles, and other sources.Source: §4.10.14 · p. 50, §4.10.15 · p. 51, §4.10.16 · p. 51

4.10.20. Qangaroo; 4.10.5. DuoRC; 4.10.7. ARC; 4.10.18. HotpotQA; 4.10.21. MultiRC; 4.10.34. DROP

  • DuoRC, ARC, HotpotQA, Qangaroo, MultiRC, and DROP target reasoning beyond simple lexical matching, including cross-document evidence, multi-sentence integration, and discrete operations.DuoRC minimizes vocabulary overlap between paired movie versions; ARC’s Challenge Set excludes questions answered by both retrieval and word-co-occurrence algorithms.Source: §4.10.5 · p. 48, §4.10.7 · p. 49, §4.10.18 · p. 51

4.10.29. CliCR; 4.10.36. Google MC-AFP; 4.10.9. OpenBookQA

  • CliCR reports a 20% F1 gap between humans and the best neural model, while Google MC-AFP reports 83.2% accuracy for its hybrid model against a 91% human ceiling.OpenBookQA’s authors report an approximately 50% baseline, with many state-of-the-art pretrained QA methods performing worse.Source: §4.10.9 · p. 49, §4.10.29 · p. 54, §4.10.36 · p. 55

4.10.30. PaperQA (Hong et al.,2018); 4.10.31. PaperQA (Park et al.,2018); 4.10.28. SciQ; 4.10.33. SciTail; 4.10.32. ReviewQA; 4.10.38. NewsQA

  • Other datasets focus on specific domains or materials, including science questions, medical case reports, academic papers, hotel reviews, and real news questions.The two PaperQA descriptions cover distinct datasets: one contains more than 6,000 human-generated questions on research abstracts, while the other contains over 80,000 cloze questions from papers.Source: §4.10.28 · p. 53, §4.10.30 · p. 54, §4.10.31 · p. 54

4.10.38. NewsQA

  • 0.198 F1 was the human–model performance gap on NewsQA, which tests reasoning beyond word matching through categories including inference, synthesis, and ambiguity [57].Source: §4.10.38 · p. 56

4.10.39. SQuAD 1.1; 4.10.41. TriviaQA

  • SQuAD contains over 100,000 crowd-generated questions answered by paragraph spans and evolved from SQuAD 1.1 to 2.0; TriviaQA adds over 650k question-answer pairs with evidence and cross-sentence reasoning [20] [43] [62].Source: §4.10.39 · p. 56, §4.10.41 · p. 56

4.10.40. RACE

  • 43% model performance versus 95% best human performance on RACE highlights its reasoning challenge across about 28,000 articles and 100,000 teacher-written questions [63].Source: §4.10.40 · p. 56

4.10.42. CLOTH

  • CLOTH comprises 7,131 examination passages and 99,433 questions, assessing reasoning, vocabulary, grammar, and long-text modeling [71].Source: §4.10.42 · p. 56

4.10.43. ProPara

  • ProPara requires identifying actions and tracking entity state changes across procedural text, using 488 paragraphs and 3,300 sentences.Source: §4.10.43 · p. 57

4.10.44. DREAM

  • DREAM’s 10,197 questions across 6,444 dialogues include 84% non-extractive answers, 85% requiring multi-sentence reasoning, and 34% involving common sense; dialogue structure and world knowledge improved models.Source: §4.10.44 · p. 57

4.10.45. CoQA

  • 65.1% best model F1 versus 88.8% human performance on CoQA reflects challenges including coreference and pragmatic reasoning across 127K questions in 8K dialogues [49].Source: §4.10.45 · p. 57

4.10.46. QuAC

  • QuAC’s 100K questions in 14K information-seeking dialogues feature open-ended, unanswerable, or dialogue-dependent questions, and its best model result was 20% below human F1 [68].Source: §4.10.46 · p. 57

5. Open Issues

The authors identify gaps in multimodal and commonsense resources, reasoning, robustness, interpretability, and dataset-quality evaluation. They also question how language understanding should be defined and suggest that cognitive-neuroscience findings may inform MRC research.

5.1.1. Multi-modal MRC

  • Current multimodal MRC datasets are few and often combine images and text without detailed annotations or internal connections.The authors argue that using multimodal information better is an important future research area.Source: §5.1.1 · p. 58

5.1.2. Commonsense and World Knowledge

  • Commonsense and world knowledge remain MRC bottlenecks, while existing datasets lack strict knowledge-type divisions and psychologically informed acquisition research.The authors call for better commonsense corpora and multimodal MRC datasets.Source: §5.1.2 · p. 59

5.1.3. Complex Reasoning

  • Many existing datasets use short answers and single-sentence questions, and input ablation leaves many answers correct, motivating benchmarks that require complex reasoning [5].Source: §5.1.3 · p. 59

5.1.4. Robustness

  • Adding question-overlapping distracting sentences significantly reduced sixteen models’ average SQuAD scores, indicating reliance on superficial cues and a robustness challenge.Source: §5.1.4 · p. 59

5.1.5. Interpretability

  • Because MRC tasks usually request answers without explanations, decision processes are difficult to inspect; future datasets could request reasoning steps or evidence [28].Source: §5.1.5 · pp. 59–60

5.1.6. Evaluation of the quality of MRC datasets?

  • Dataset readability is not directly tied to question difficulty, while prerequisite-skill definitions vary and their frequencies are manually counted, leaving dataset quality evaluation unresolved [5].Source: §5.1.6 · p. 60

5.2. Have we understood understanding?

  • NLP lacks a comprehensive definition of language understanding and specific metrics for evaluating MRC models’ real understanding capabilities.A cited study reports sensory-derived and cognitively-derived object knowledge representations supported by different brain systems; the authors suggest neuroscience findings may inspire MRC design.Source: §5.2 · pp. 60–61

6. Conclusions

The conclusion recaps the survey’s synthesis of MRC tasks, metrics, and datasets, alongside open issues and future research directions. It also points readers to a companion website with organized research resources.

6. Conclusions

  • The survey proposes a more adequate MRC task classification and reviews task definitions, existing classifications, and differences between textual and multimodal MRC.Source: §6 · p. 61
  • It analyzes MRC evaluation metrics and describes attributes and characteristics of datasets, including 47 datasets in detail.Source: §6 · p. 61
  • The authors argue that multimodal datasets and cognitive-neuroscience findings may help construct more challenging datasets and develop related algorithms toward human-level MRC.Source: §6 · p. 61
  • The companion website provides access to MRC datasets, papers, baseline projects, and leaderboards.Source: §6 · p. 61

Related papers

Loading 2006.11880v2…