DRCD: a Chinese Machine Reading Comprehension Dataset

1 Introduction

• Rule-based MRC is limited by costly rule maintenance and difficulty handling variations of the same questions, while machine-learning approaches require large training datasets.MRC systems answer questions about document contents.Source: §1 · p. 1
• Existing large-scale MRC datasets serve different purposes and use multiple-choice, cloze-style, span-based, or user-log answer formats.The introduction notes that multiple-choice and cloze-style formats are inappropriate for search scenarios.Source: §1 · p. 1
• DRCD targets span-based comprehension because user-log datasets mix search and comprehension abilities, while no large-scale traditional Chinese MRC dataset had been compiled.The paper reports 10,014 paragraphs from 2,108 Wikipedia articles and 33,941 question-answer pairs.Source: §1 · p. 2

2 Related Work

• Multiple-choice datasets can be adapted from exams, but human-expert data generation is expensive and may leave them insufficient for advanced data-driven MRC training.Examples include MCTest and RACE.Source: §2 · p. 2
• Cloze-style datasets are practical to build without human labeling, but single-word answers may require less reasoning and inference than span-based answers.The cited examples include English news and children’s books, and Chinese news and fairy tales.Source: §2 · p. 2
• Span-based MRC assumes each answer can be found in its reference document; SQuAD uses one evidence document per question, unlike multi-document TriviaQA.The passage describes SQuAD and TriviaQA as large-scale datasets with different question and evidence-document characteristics.Source: §2 · p. 2
• User-log datasets use real search logs: MSMARCO has 100,000 real anonymized Bing queries and real web documents from Bing’s index, while DuReader was constructed from Baidu Search and Baidu Zhidao.DuReader contains 200,000 questions, over 1 million documents, and over 420,000 human-generated answers.Source: §2 · p. 2

3 Dataset Collection

• The collection follows Rajpurkar et al. (2016) to gather Wikipedia data and proceeds through passage curation, question-answer collection, and additional answer selection.The paper directs readers to Table 1 for an example.Source: §3 · pp. 2–3
• Annotators extract paragraphs from highly ranked Chinese Wikipedia articles, then retain cleaned article text in paragraphs of 250–1,500 Chinese characters.The resulting corpus contains 10,014 paragraphs from 2,108 articles; images, tables, lists, links, and bracketed material are removed.Source: §3 · p. 3
• Annotators read the whole article and write at least 3–5 questions whose answers occur in the paragraph, phrasing questions in their own words.They are encouraged to ask harder descriptive or sentence-level questions and to specify questions to avoid multiple possible answers.Source: Question-answer collection · p. 3
• The data split contains 26,932 training questions, 3,524 development questions, and 3,485 test questions across 8,014, 1,000, and 1,000 paragraphs, respectively.One additional answer was collected for each development- and test-set question for human-performance evaluation.Source: Question-answer collection · p. 3

4 Dataset Analysis

• The authors classify questions into seven types using keyword lists constructed from a random sample of 660 questions.Table 2 presents the type distribution and sample keywords.Source: §4 · p. 3
• 70.45% of answers are entities, 18.03% are numeric, and 11.50% are descriptions under the paper’s automatic answer-type categorization.Answers longer than six characters are classified as descriptions; remaining answers are split into numeric and entity groups.Source: §4 · p. 3
• 435.8 Chinese characters is the average DRCD paragraph length, compared with 116.63 words for SQuAD documents; the authors attribute longer DRCD paragraphs to topic-based rather than period-based segmentation.Average question and answer lengths are 21.07 and 4.86 Chinese characters, respectively.Source: §4 · p. 4

• Evaluation used punctuation-insensitive F1 and exact match; F1 treated answers as bags of Chinese characters, and R-Net, QANet, and BiDAF used 300-dimensional Wikipedia-trained CBOW embeddings.Source: §5 · p. 4
• 93.30% F1 and 80.43% exact match: human performance on the DRCD test set.A different annotator answered each development- and test-set question than the annotator who constructed its paragraph and question-answer pair.Source: §5 · p. 4
• 89.59% F1 and 82.34% exact match: BERT achieves the strongest reported baseline scores.Source: §5 · p. 4
• 78.03% F1 and 65.56% exact match for QANet; 51.18% F1 and 28.08% exact match for BiDAF.Source: §5 · p. 4
• 38% F1 and 23.8% exact match for R-Net; TF-IDF scores 17% F1 and 0.05% exact match.Source: §5 · p. 4

• BERT achieves 89.59% F1 and 82.34% exact match, but its F1 remains below human performance, which the authors say indicates DRCD is still complicated.Source: §6 · p. 4
• DRCD contains 10,014 paragraphs and 40,410 question-answer pairs from 2,108 Wikipedia articles, and is intended as a source dataset for transfer learning.Source: §6 · p. 4
• The authors plan to focus on industrial data, improve annotation with community feedback, and pursue applications for search engines and question-answering systems.Source: §6 · p. 5