Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension

1 Introduction

• C3 comprises dialogue and mixed-genre text tasks, with problems collected from expert-designed Chinese-as-a-second-language examinations.The mixed-genre texts are relatively formally written.Source: §1 · p. 2
• 86.8% of C3 questions require both document information and prior knowledge, and 57.3% require more general world knowledge.The analysis distinguishes linguistic, domain-specific, and general world knowledge; dialogue tasks tend to require more general world knowledge and less linguistic knowledge than relatively formal texts.Source: §1 · p. 2
• The best-performing model achieves 68.5%, versus 96.0% for human readers, with the gap especially pronounced on questions requiring prior knowledge.Superficially text-matching wrong options also play a critical role in question difficulty.Source: §1 · p. 2
• C3 is intended as a platform for studying how different types of prior knowledge can support in-depth text comprehension and crosslingual or multilingual MRC.The dataset includes multiple question types and draws on written or orally oriented texts.Source: §1 · pp. 1–2

2 Related Work

• Many English MRC tasks focused on explicit textual facts, while free-form multiple-choice benchmarks also included implicitly expressed facts, events, opinions, or emotions requiring prior knowledge.These benchmarks can evaluate a broader range of reading skills by combining prior knowledge with information in the text.Source: §2 · pp. 2–3
• Earlier Chinese MRC datasets largely used answers extracted from or abstracted from the text, whereas C3 includes varied question types and requires rich prior knowledge.The passage characterizes C3 as the first free-form multiple-choice Chinese MRC dataset of this kind.Source: §2 · p. 3
• C3 includes dialogue comprehension, which the paper says had not been studied in existing Chinese MRC tasks.Questions in 48.4% of problems require dialogue understanding.Source: §2 · p. 3
• The paper distinguishes general reading-comprehension datasets from Chinese exam QA tasks focused on domain knowledge and partly dependent on retrieval or external resources.The comparison of representative Chinese QA and MRC tasks is presented in Table 1.Source: §2 · p. 3, Table 1 · p. 3

3.1 Collection Methodology and Task Definitions

• C3 contains 13,369 documents and 19,577 questions, collected from HSK and MHK exams, split into dialogue and mixed-genre tasks, and divided into training, development, and test sets at the problem level.Each question has several options with exactly one correct answer; the split is 60% training, 20% development, and 20% test.Source: §3.1 · pp. 3–4

3.2 Data Statistics

• C3-M has a larger vocabulary and longer average documents than C3-D (180.2 vs. 76.3), but C3 documents are shorter on average than those in DuReader (396.0) and RACE (321.9).The paper cautions that C3 may not suit evaluation of comprehension on lengthy texts.Source: §3.2 · pp. 4–5

3.3 Categories of Prior Knowledge

• The analysis distinguishes linguistic, domain-specific, and general-world knowledge, with general-world subtypes including arithmetic, connotation, cause-effect, implication, and part-whole relations.Scenario knowledge includes inferred participant information and relationships, while some questions require knowledge beyond paraphrasing text.Source: §3.3 · pp. 5–6
• 49.0% vs. 30.7% of C3-M and C3-D questions require linguistic knowledge, while 50.7% vs. 64.0% require general-world knowledge; 24.3% of C3-D questions require scenario knowledge.The distributions are reported in Table 5, where a question may require more than one knowledge type.Source: Table 5 · p. 6, §3.3 · p. 7
• Two annotators labeled prior-knowledge types in sampled questions and comparison datasets; they also classified questions by how many document sentences support an answer, with Cohen’s kappa of 0.62.The annotated comparison covers C3, DuReader, RACE, and DREAM.Source: §3.3 · pp. 5, 7
• C3 and DREAM have similar prior-knowledge distributions, while DuReader questions seldom require general-world knowledge and 16.7% of its question-answer pairs lack textual support, versus 1.3% in C3.The paper suggests C3 may facilitate cross-lingual MRC studies.Source: §3.3 · p. 7

4 Approaches

• The approaches include a classical rule-based method and recent neural models for multiple-choice MRC.Source: §4 · p. 7

4.1 Distance-Based Sliding Window

• Distance-based Sliding Window ranks options using statement-document lexical similarity and the minimum token distance between question and answer-option occurrences.The method assumes shorter within-statement token distances and more informative statement tokens appearing in the document favor the correct option.Source: §4.1 · p. 7

4.2 Co-Matching

• Co-Matching treats a question and one associated answer option as two sequences and jointly models whether the document matches them; the paper adapts it to Chinese.Source: §4.2 · p. 8

4.3 Fine-Tuning Pre-Trained Language Models

• The paper fine-tunes Chinese BERT, ERNIE, BERT-wwm, and BERT-wwm-ext, scoring each answer option from a pretrained model’s pooled input representation and applying softmax across options.Inputs concatenate the document, question, and option with classifier and separator tokens.Source: §4.3 · p. 8

5.1 Experimental Settings

• Experiments used five random seeds, reported the test performance associated with the best development result, treated characters as tokens, and fine-tuned models on C3 for eight epochs.Chinese word segmentation was omitted because experiments showed it reduced performance.Source: §5.1 · p. 8

5.2 Baseline Results

• BERT-wwm-ext trails human performance by 27.5%, despite pre-trained language models outperforming the sliding-window and Co-Matching baselines by a large margin.BERT-wwm-ext reached 96.0% human performance as the comparison point.Source: §5.2 · p. 9

5.3 Discussions on Distractor Plausibility

• As maximum distractor plausibility rises, all three models’ performance drops; below −0.8 they exceed 90%, and BERT-wwm-ext’s gains over BERT mainly come on difficult questions.Plausible distractors are wrong options that superficially resemble the document, including accurate-but-incomplete or distorted interpretations.Source: §5.3 · pp. 9–10
• General world knowledge is crucial when the correct answer is less explicitly mentioned, whereas linguistic knowledge is needed when both the correct answer and most confusing distractor closely match the document.Source: §5.3 · p. 10

5.4 Discussions on Data Augmentation

• Accuracy grows roughly linearly with the logarithm of C3 training-set size, yet a substantial human-performance gap remains even assuming 105 training instances.Source: §5.4 · pp. 10–11
• 4.6% improvement (67.8% to 72.4%) resulted from adding all 94k translated RACE and DREAM instances to C3 training, with a lower learning-curve growth rate.Translated-only training reached 67.1% on C3 development, versus 67.8% with C3-only training.Source: §5.4 · p. 11
• 63.4% accuracy on C3 development with multilingual BERT trained on C3 and original English data was below the 65.7% from Chinese BERT fine-tuned on C3 alone.Source: §5.4 · p. 11

• 68.5% for the best-performing model versus 96.0% for human readers demonstrates a significant performance gap on C3.Source: §6 · p. 11
• C3 is a free-form multiple-choice Chinese MRC dataset collected from real-world language examinations and requiring linguistic, domain-specific, or general world knowledge.Source: §6 · p. 11
• The paper analyzes prior-knowledge needs and investigates distractor plausibility and data augmentation using similar English resources, identifying a need for better ways to exploit resources in other languages.Source: §6 · p. 11