Improving Machine Reading Comprehension with General Reading Strategies

• Non-extractive MRC calls for diverse reading skills and is presented as a more realistic indicator of machine-reader comprehension in settings such as exams.Source: §1 · p. 1
• Pre-training a language model with extensive general-domain knowledge is resource-demanding, motivating improvement during fine-tuning instead.The paper frames this as a practical question under limited training instances and an existing pre-trained model.Source: §1 · pp. 1–2
• Back and forth reading uses original and reverse input order; highlighting adds trainable embeddings to question- and answer-relevant tokens; self-assessment generates practice questions and candidate answers from the text.These strategies are proposed as domain-independent methods for improving MRC during fine-tuning.Source: §1 · p. 2
• The strategy-trained model improves RACE accuracy by 5.8% absolutely over the same pre-trained transformer baseline and achieves state-of-the-art results on six non-extractive MRC datasets.The six datasets cover multiple domains and a range of reading skills, including commonsense and multi-sentence reasoning.Source: §1 · p. 2

2 Task Introduction

• MRC tasks are grouped into extractive and non-extractive categories according to their expected answer types.Source: §2 · p. 2

2.1 Extractive MRC

• Extractive MRC expects a short answer span from the reference document, although free-form answers in some datasets are still often copied from the text.The passage names SQuAD and NewsQA as examples of extractive datasets and notes that answers in some free-form datasets remain mostly extractive.Source: §2.1 · p. 2

2.2 Non-Extractive MRC

• Non-extractive multiple-choice MRC allows answer options beyond text spans and can require reasoning or prior world knowledge; these datasets are often small because they are difficult to construct.The study examines seven representative datasets; most correct options are non-extractive except in ARC and MCTest.Source: §2.2 · pp. 2–3

3.1 Framework Overview

• The reader fine-tunes a pre-trained transformer on labeled examples, classifying answer-option correctness from the final transformer activation.For multiple-choice MRC, the input concatenates the document, question, and answer option with start, delimiter, and end tokens.Source: §3.1 · p. 3

3.2 Back and Forth Reading (BF)

• Back and forth reading fine-tunes separate GPTs on original and reversed input orders, then ensembles their predictions.The token order within the document, question, and option remains unchanged in the reversed sequence.Source: §3.2 · p. 3

3.3 Highlighting (HL)

• Highlighting assigns each document token one of two trainable vectors based on content-word POS tags and whether it appears in the question or answer option, then adds these vectors to document embeddings.The content-word tags include nouns, verbs, adjectives, adverbs, numerals, and foreign words.Source: §3.3 · p. 4

3.4 Self-Assessment (SA)

• Self-assessment generates practice questions from task documents before end-task fine-tuning, aiming to help the model integrate information across multiple sentences.It forms questions by removing randomly selected spans from concatenated document sentences, uses those spans as the correct option, and creates distractors from other document spans.Source: §3.4 · pp. 4–5

4.1 Experiment Settings

• The authors generate 119k practice instances from RACE training and development documents, then fine-tune on those instances, RACE, and a target dataset.Training uses one epoch on generated instances, five on RACE, and at most ten on a target task.Source: §4.1 · p. 5

4.2 Evaluation on RACE

• 65.4% vs. 59.6% accuracy on RACE: ensembling two models using all strategies and opposite input orders improves over two original fine-tuned GPT models.Single-model self-assessment and highlighting improve over the 58.7% fine-tuned GPT baseline by 1.7% and 4.5%, respectively.Source: Table 2 · p. 5, §4.2 · p. 5

4.3 Further Discussions on Strategies

• Ensembling alternative input-order pairs reaches 61.0% or 61.7% on RACE versus 59.6% for two original fine-tuned GPT models; extractive summarization was less effective.The alternative pairs are [qd $ o] with [o $ dq], and [qd $ o] with [o $ qd].Source: §4.3 · p. 6
• Highlighting question and answer-option content words reaches 63.2%, versus 60.6% with question-only highlighting and 61.0% with answer-only highlighting.Highlighting coreferential mentions does not lead to further gains.Source: §4.3 · p. 6
• Generating questions from SQuAD articles improves the fine-tuned GPT baseline to 59.7% from 58.7%, while sentence shuffling and back-translation do not improve RACE performance.The SQuAD question-generation method uses the same number of questions as the RACE-based method.Source: §4.3 · p. 6

4.4 Adaptation to Other Non-Extractive Machine Reading Comprehension Tasks

• For target-task fine-tuning, the model retains highlighting and uses back-and-forth reading, but omits self-assessment after training on RACE.For MultiRC, the reported model is selected by development-set micro-average F1; other tasks use development-set accuracy.Source: §4.4 · pp. 7–8
• 6.2% absolute improvement in average accuracy over previous SOTA (60.1%) on six target MRC datasets, using the strategies.The ensemble baseline average accuracy is 58.5%; the strategies improve on it by 7.8%.Source: §4.4 · p. 8
• 10.4% relative improvement in average accuracy (60.3%) over the 54.6% baseline when fine-tuning directly on target tasks without labeled RACE data.The largest improvements are reported on ARC, OpenBookQA, and MCTest.Source: §4.4 · p. 8

5.1 Methods for Multiple-Choice Machine Reading Comprehension

• Multiple-choice MRC methods have used attention, elimination modules, or hierarchical attention, while this work investigates strategies with a pre-trained transformer.The authors note that the cited methods seldom consider rich external knowledge beyond pre-trained word embeddings.Source: §5.1 · p. 8

5.2 Transfer Learning for Machine Reading Comprehension and Question Answering

• The paper fine-tunes on source data and then the entire model on target data; adding parameters, using an L2 loss, or fine-tuning only part of the parameters is beyond its scope.The passage frames this approach against transfer-learning methods that examine those alternatives.Source: §5.2 · p. 8

5.3 Data Augmentation for Machine Reading Comprehension Without Using External Datasets

• Unlike prior augmentation approaches based on word or sentence reordering or paraphrasing, this method generates questions and answer options without existing training questions.Generated questions can involve content from multiple sentences in a reference document.Source: §5.3 · p. 8

6 Conclusions

• 5.8% absolute improvement in RACE accuracy over the state-of-the-art performance, using the three proposed strategies.Source: §6 · p. 8
• Fine-tuning the resulting model on a new target task achieved new state-of-the-art results on six representative non-extractive MRC datasets.The datasets span multiple domains and require a diverse range of reading skills.Source: §6 · p. 8
• The three strategies are back and forth reading, highlighting, and self-assessment, intended for settings with a pre-trained model and limited training instances.Source: §6 · p. 8
• The authors conclude that results consistently indicate the strategies' effectiveness and the general applicability of the fine-tuned model incorporating them.Source: §6 · p. 8