Source-linked AI summary
A Span-Extraction Dataset for Chinese Machine Reading Comprehension
Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, Guoping Hu
TL;DR
Existing reading-comprehension datasets are mostly English, motivating a Chinese resource for testing multilingual model performance. The paper constructs a human-annotated Chinese span-extraction dataset with nearly 20,000 questions and a multi-clue challenge set; models perform well on ordinary splits but decline sharply on the challenge set while human performance remains nearly stable.
Problem
Existing reading-comprehension datasets are mostly English, leaving open whether state-of-the-art models perform similarly on datasets in different languages.
Method
The authors construct a Chinese span-extraction dataset from human-annotated Wikipedia questions and add a challenge set requiring reasoning over multiple clues.
Results
Models achieve excellent development and test F1 scores near 10 points below estimated human performance, but scores decline drastically on the challenge set while human performance remains almost unchanged.
Takeaways & Limitations
CMRC 2018 adds linguistic diversity and exposes the need for more sophisticated models that handle difficult multi-clue questions.
Takeaways & Limitations
Answers must be spans directly extracted from the passage, constraining the task to span-extraction responses.
Abstract
from arXiv · showhide
Machine Reading Comprehension (MRC) has become enormously popular recently and has attracted a lot of attention. However, the existing reading comprehension datasets are mostly in English. In this paper, we introduce a Span-Extraction dataset for Chinese machine reading comprehension to add language diversities in this area. The dataset is composed by near 20,000 real questions annotated on Wikipedia paragraphs by human experts. We also annotated a challenge set which contains the questions that need comprehensive understanding and multi-sentence inference throughout the context. We present several baseline systems as well as anonymous submissions for demonstrating the difficulties in this dataset. With the release of the dataset, we hosted the Second Evaluation Workshop on Chinese Machine Reading Comprehension (CMRC 2018). We hope the release of the dataset could further accelerate the Chinese machine reading comprehension research. Resources are available: https://github.com/ymcui/cmrc2018
1 Introduction
The paper introduces CMRC 2018, a Chinese span-extraction reading-comprehension dataset designed to broaden linguistic diversity and test models across languages. It also includes a challenge set requiring reasoning over multiple clues.
- CMRC 2018 addresses whether state-of-the-art reading-comprehension models maintain their performance on datasets in languages different from English.
- The authors propose a Chinese span-extraction reading-comprehension dataset with nearly 20,000 human-annotated questions.
- The challenge set contains carefully annotated questions requiring various clues across the passage rather than relying on a single sentence.
- BERT-based approaches achieve under 50% F1-score on the challenge set, indicating its difficulty.
- The Chinese reading-comprehension data can support cross-lingual research when studied with SQuAD and similar datasets.
2 The Proposed Dataset
CMRC 2018 frames span extraction as answering human-written questions with answer spans taken directly from passages. Its construction uses filtered Wikipedia text, human annotation, multiple answers for evaluation, and a reasoning-focused challenge set.
- In span-extraction reading comprehension, human-written questions are paired with answers that are spans directly extracted from the passage.
- Chinese Wikipedia text was preprocessed into plain text and converted from Traditional to Simplified Chinese for normalization.
- Human Annotation: Human annotators evaluated passage appropriateness before writing questions, with passages limited to 500 Chinese words and unsuitable content discarded.
- Human Annotation: Annotation encouraged diverse question forms, paraphrasing or syntactic transformation, and answers no longer than 30 characters.
- Human Annotation: Development, test, and challenge questions each have three answers produced by separate annotators who could not see the primary answer.
- Challenge Set: The challenge set retains span extraction while requiring answers to be inferred from multiple clues and avoiding uniquely identifiable named entities or categories.
3 Evaluation Metrics
The evaluation adapts exact-match and fuzzy character-level F1 metrics for Chinese answers, while estimating human performance through cross-validation over multiple reference answers.
- The paper adapts exact-match and F1 evaluation to Chinese, ignoring common punctuation and whitespace during normalization.
- Exact Match: Exact Match scores 1 only when a prediction exactly matches a ground-truth answer; otherwise, it scores 0.
- F1 measures character-level fuzzy overlap using the longest common sequence and takes the maximum score across ground-truth answers.
- Human performance is estimated by iteratively treating each of three answers as a prediction and the others as ground truths.
4 Experimental Results
The evaluation shows strong performance on standard development and test sets but substantially weaker results on the challenge set, especially for questions requiring multi-clue reasoning and exact span identification.
- Baseline Setup: The BERT baseline used a slightly modified SQuAD implementation with a 3e-5 learning rate, batch size 32, and two training epochs.Document and query maximum lengths were set to 512 and 64, respectively.
- Participant Results: Most participants obtained over 80 F1 on the test set, while exact-match scores were substantially lower than in SQuAD.The authors attribute the gap to the difficulty of determining exact span boundaries in Chinese reading comprehension.
- Challenge-Set Difficulty: Top-ranked systems achieved decent development and test scores but failed to produce satisfactory challenge-set results.Human performance was relatively similar across development, test, and challenge sets, with only slightly lower scores on the challenge set.
- Challenge-Set Difficulty: Current reading comprehension models remain relatively weak at difficult questions requiring comprehensive reasoning across several passage clues.This limitation is especially visible on the challenge set rather than the standard evaluation sets.
- Baseline Comparison: BERT-based approaches achieved competitive performance against participant submissions.The workshop participants’ systems were not based on BERT because the workshop preceded BERT’s publication.
- Baseline Comparison: BERT-based baselines were consistently higher than traditional models on the challenge set, despite traditional models scoring higher on the test set.The authors associate this pattern with the richer representations provided by BERT.
5 Conclusion
The paper introduces a human-annotated Chinese span-extraction dataset with nearly 20,000 questions and a challenge set targeting reasoning across multiple clues. Results are strong on standard sets but decline sharply on the challenge set, motivating further work on comprehensive reasoning.
- Conclusion: The dataset contains nearly 20,000 human-expert-annotated Chinese span-extraction questions.The questions are accompanied by a challenging set designed around reasoning over multiple clues.
- Conclusion: The challenge set contains questions requiring reasoning over multiple clues, while preserving the span-extraction format.This design targets comprehensive understanding rather than only straightforward span matching.
- Conclusion: Models scored near 10 F1 points below estimated human performance on the development and test sets.The conclusion characterizes these scores as excellent relative to the estimated human performance.
- Conclusion: Scores declined drastically on the challenge set while human performance remained almost unchanged from the non-challenge sets.The authors identify this gap as evidence of continuing challenges in designing more sophisticated models.
- Conclusion: The dataset is intended to add language diversity to MRC and encourage investigation of questions requiring comprehensive reasoning over multiple clues.The paper also notes that participant systems were not based on BERT because the workshop preceded BERT’s publication.
Open Challenge
The authors invite researchers to evaluate models on the hidden test and challenge sets to further test model generalization.
- Open Challenge: Researchers can submit models through the CodaLab Worksheet for evaluation on the hidden test and challenge sets.The stated purpose is to further test model generalization on the CMRC 2018 dataset.