Source-linked AI summary
DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, Haifeng Wang
TL;DR
Real-world Chinese MRC needs data beyond conventional datasets, particularly for varied questions and open-domain documents. DuReader addresses this by releasing a large Baidu-based dataset with broad annotations and manually generated answers; human performance remains well above current systems, while later sophisticated models have shown significant improvements.
Problem
Existing MRC datasets make compromises in data sources and question types, while real-world questions include yes-no and opinion cases requiring reasoning beyond single-span extraction.
Method
DuReader builds an open-domain Chinese MRC dataset from Baidu Search and Baidu Zhidao, pairing questions with multiple full documents and manually generated answers.
Results
Human performance remains well above current state-of-the-art baseline systems, while significant improvements have followed from more sophisticated models.
Takeaways & Limitations
DuReader provides a shared benchmark and competition for developing models targeting realistic Chinese MRC questions, including opinion and yes-no questions.
Takeaways & Limitations
The baseline systems use simple paragraph selection, causing substantial degradation relative to gold paragraphs and motivating more sophisticated ranking models.
Abstract
from arXiv · showhide
This paper introduces DuReader, a new large-scale, open-domain Chinese ma- chine reading comprehension (MRC) dataset, designed to address real-world MRC. DuReader has three advantages over previous MRC datasets: (1) data sources: questions and documents are based on Baidu Search and Baidu Zhidao; answers are manually generated. (2) question types: it provides rich annotations for more question types, especially yes-no and opinion questions, that leaves more opportunity for the research community. (3) scale: it contains 200K questions, 420K answers and 1M documents; it is the largest Chinese MRC dataset so far. Experiments show that human performance is well above current state-of-the-art baseline systems, leaving plenty of room for the community to make improvements. To help the community make these improvements, both DuReader and baseline systems have been posted online. We also organize a shared competition to encourage the exploration of more models. Since the release of the task, there are significant improvements over the baselines.
1 Introduction
DuReader is introduced to advance real-world Chinese MRC through Baidu-based data, broader question annotations, and unprecedented scale. Its design supplies multiple full documents and manually generated answers for questions including yes-no and opinion cases.
- Existing MRC datasets often rely on cloze tasks, multiple-choice exams, or crowdsourced document-question pairs rather than real applications.
- DuReader annotates Entity, Description, or YesNo questions alongside Fact or Opinion labels.This includes yes-no and opinion questions, which are less suited to single-span extraction when answers require multi-document summaries.
- DuReader uses questions and documents from Baidu Search and Baidu Zhidao, with manually generated answers.
- Each question is paired with multiple full documents, making paragraph selection an important challenge before answer generation.
- 200K questions, 1M documents, and more than 420K human-summarized answers make DuReader the largest Chinese MRC dataset so far.
2 Pilot Study
A pilot study of Baidu Search queries developed a six-type taxonomy and found that opinion and yes-no questions are common. These distributions expose gaps in span-selection-focused MRC datasets.
- The pilot study manually classified questions by Fact or Opinion and by Entity, Description, or YesNo.Entity answers are entities or lists, Description answers are usually multi-sentence summaries, and YesNo answers require affirmative or negative responses with evidence.
- All six question types were common in Baidu Search logs, whereas previous MRC work emphasized fact-entity and fact-description questions.
- 33.8% of sampled queries were opinions and 15.6% were yes-no questions.About half of the yes-no questions concerned facts and about half concerned opinions.
- Span selection is appropriate for fact-entity and fact-description questions but problematic for multi-sentence, multi-document summaries, especially yes-no and opinion questions.
3 Scaling up from the Pilot to DuReader
DuReader scales a pilot collection into a 200K-question dataset by combining automated filtering, human annotation, documents from two Baidu sources, and predefined evaluation splits. Its annotations capture diverse answer types and supporting evidence, while its full documents and paraphrased answers create challenges for span-selection methods.
- Data Collection: DuReader represents each example as a 4-tuple containing a question, question type, relevant documents, and a human-produced answer set.Relevant documents were collected from Baidu Search and Baidu Zhidao, whose content differs in source and opinion relevance.
- Data Collection: 200K questions were uniformly sampled from 210K classifier-filtered question queries after starting with 1M frequent search-log queries.The classifier selected 280K question queries, which were reduced to 210K after human annotation.
- Question Type Annotation: Questions receive two-pass labels for Entity, Description, or YesNo type and for Fact or Opinion, with duplicates removed from the released dataset.The annotations preserve question diversity, including yes-no and opinion questions that prior work often underrepresented.
- Answer Annotation: Annotators write answers in their own words by reading and summarizing relevant documents, may provide multiple answers, and may return an empty answer when none is found.Experts validate worker answers, and similar multiple answers can be merged.
- Training, Development and Test Sets: 181K, 10K and 10K questions form the training, development and test sets, respectively, alongside 855K, 45K and 46K documents.The corresponding answer counts are 376K, 20K and 21K.
- DuReader is (Relatively) Challenging: 396 words is the average DuReader document length, about 5x longer than MS-MARCO, while answers average 69.6 words and questions 4.8 words.Unabridged documents contain multiple paragraphs or passages, increasing the need for paragraph selection.
4 Experiments
Experiments evaluate two span-selection baselines on DuReader and show gains over paragraph selection, but substantial gaps remain versus humans. Results also identify paragraph selection, question type, and opinion handling as central challenges.
- 4.1 Paragraph Selection: Training selects the paragraph with greatest overlap with the human answer, whereas testing selects the paragraph with greatest overlap with the question.The trained span-selection models then operate on the selected paragraphs.
- 4.1 Baseline Systems: Two state-of-the-art span-selection models, Match-LSTM and BiDAF, are implemented as DuReader baselines.Both models are evaluated after selecting one paragraph per document.
- 4.2 Results and Analysis: Current models improve over the selected-paragraph baseline but remain well below human performance, with open-domain web articles harder than question-answering-community passages.Human performance differs little between Baidu Search and Baidu Zhidao data.
- 4.2 Results and Analysis: Gold paragraphs significantly boost overall performance, showing that paragraph selection is a crucial problem for real-world MRC.Gold paragraphs also yield very high Rouge-L, while baseline models improve substantially on BLEU.
- 4.2 Results and Analysis: YesNo questions are hardest to model, while description questions receive relatively strong performance from both models and humans.Description answers are often longer and align better with BLEU or Rouge, whereas YesNo answers can be short.
- 4.3 Opinion-aware Evaluation: Opinion-aware evaluation requires an answer and an opinion label, and opinion classification improves performance under this evaluation method.Answers are matched against reference answers sharing the predicted opinion label.
- 4.4 Discussion: The dataset highlights underexplored yes-no, opinion, and multi-document questions requiring opinion recognition, cross-sentence reasoning, and multi-document summarization.These directions are presented as areas needing new methods.
- 4.4 Discussion: The baseline’s simple paragraph selection causes substantial degradation relative to gold-paragraph performance, motivating more sophisticated paragraph ranking.This limitation directly affects performance in the real-world setting.
5 A Shared Task
The shared task invites researchers to submit systems and evaluate them online, producing improvements over the paper’s baselines. A submitted team reached 51.2 ROUGE-L, compared with 39.0 for BiDAF and 57.4 for humans.
- 5 A Shared Task: The online competition lets participants submit results and evaluate system performance through a shared website.It was organized to encourage exploration of more models.
- 5 A Shared Task: 51.2 ROUGE-L was achieved by a participating team, versus 39.0 for the BiDAF baseline and 57.4 for human performance.These figures were reported when the paper was submitted.
6 Conclusion
DuReader is presented as a large Chinese MRC dataset distinguished by its real-world data sources, broad question types, and scale. The dataset is freely available, and a shared competition supports exploration of improved models.
- DuReader is distinguished by data sources based on search logs and a question-answering community, diverse question types, and being the largest Chinese MRC dataset so far.
- The dataset has been made freely available to researchers.
- A shared competition was organized to encourage exploration of more models.
- Since the task’s release, more sophisticated models have produced significant improvements.